Module 05 · Python 3.10+

Python Reading & Writing Files
open(), read(), write(), pathlib

Almost every real Python program touches files — reading config, processing CSVs, writing logs, parsing JSON. This module teaches you to do it correctly: safely, with the right encoding, and without leaking file handles.

Start with open() Jump to cheat sheet

read_write.py

with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("Hello, file!\n")
    f.write("Second line.\n")

with open("notes.txt", encoding="utf-8") as f:
    for line in f:
        print(line.rstrip())

$ python read_write.py
Hello, file!
Second line.

📁 Why File I/O Matters

Data outlives your program — files are how it does that

A program's memory disappears when it exits. Files persist. Almost every useful Python program eventually reads from or writes to some persistent store — a config file, a CSV export, a log, a JSON API response cached to disk. Getting file I/O right is the difference between a script that runs once and a tool people can rely on.

✓Configuration — .env, .toml, .json, .yaml files hold settings you shouldn't hardcode
✓Data processing — CSVs of transactions, logs, exports, surveys
✓Inter-process communication — one program writes, another reads
✓Caching — save expensive API responses to disk so you don't refetch
✓Logging — append-only files for debugging and auditing
✓Output — reports, generated HTML, images, PDFs

Run Locally Only — The Run button on this page uses Pyodide's virtual filesystem. Files are created, read, and written during a single run, but they do not persist between runs and don't touch your real disk. For real file work, copy the snippets into a .py file and run them with python on your computer.

🚪 Lesson 1 — The `open()` Function

Python's single entry point for reading and writing files

📖 The Basic Signature

open() returns a file object. Its most important arguments are the file path and the mode. The mode tells Python how you intend to use the file — whether to read, write, append, and whether to treat it as text or binary.

open(file, mode="r", buffering=-1, encoding=None,
     errors=None, newline=None, closefd=True, opener=None)

Most of these you'll never touch. The three that matter daily are mode, encoding, and newline. newline controls how line endings are translated: when reading, the default None enables universal newline mode and converts \n, \r, and \r\n all to \n; when writing, it converts \n to the platform default separator. For CSV files you'll set it to "" to disable translation entirely.

The full set of modes is small enough to memorise:

Mode	Meaning	If file exists	If file doesn't exist
`"r"`	Read text (default)	Reads from start	Raises `FileNotFoundError`
`"w"`	Write text	Truncates (empties) it	Creates it
`"a"`	Append text	Adds to end	Creates it
`"x"`	Exclusive create	Raises `FileExistsError`	Creates it
`"r+"`	Read & write	Reads and writes	Raises `FileNotFoundError`
`"rb"`, `"wb"`	Binary read/write	No text decoding — bytes in, bytes out

💡 "w" mode erases the file if it already exists. If you want to add to an existing file, use "a". If you want to fail rather than overwrite, use "x".

🛡️ Always Use `with`

You can open a file without with, but then you're responsible for closing it — and if an exception happens between open() and close(), the file handle leaks and the file may stay locked or partially written. The with statement guarantees the file is closed even if an exception is raised.

# ❌ Fragile — file may stay open if f.write() raises
f = open("data.txt", "w")
f.write("hello")
f.close()

# ✅ Safe — with guarantees close() even on error
with open("data.txt", "w", encoding="utf-8") as f:
    f.write("hello")

💡 with calls f.__exit__() when the block finishes — which closes the file. It's the try/finally pattern, packaged so cleanly you should almost never write open() without it.

🌍 Always Specify `encoding="utf-8"`

When you open a text file without specifying encoding, Python uses the platform default — utf-8 on macOS and Linux, but cp1252 on Windows. Code that works on your laptop can fail on a teammate's machine with UnicodeDecodeError purely because of this default.

with open("poem.txt", encoding="utf-8") as f:
    text = f.read()

print(f"Read {len(text)} characters")

On Python 3.15+ the default becomes UTF-8 on all platforms, but until then — and for compatibility with older interpreters — always be explicit.

⚠️ For binary files ("rb", "wb"), do not pass encoding — it's meaningless for byte streams and Python will raise a ValueError.

📖 Lesson 2 — Reading Files

Three ways to get the content out — pick the right one for the size of file

📄 `.read()` — The Whole File at Once

Simplest and fastest for small files. Returns the entire content as a single string (or bytes for binary). Once you've called .read(), the file pointer is at the end — calling it again returns an empty string.

with open("config.json", encoding="utf-8") as f:
    content = f.read()

print(content)
print(f"Total length: {len(content)} chars")

⚠️ Don't use .read() on files larger than memory. A 4 GB log file becomes a 4 GB string. Use iteration instead.

📏 `.readline()` and `.readlines()`

.readline() returns one line at a time, including the trailing \n. .readlines() reads the entire file and returns a list of lines. Both are useful in different situations.

with open("log.txt", encoding="utf-8") as f:
    first_line = f.readline()
    second_line = f.readline()

print(f"First: {first_line!r}")
print(f"Second: {second_line!r}")

💡 !r in an f-string shows the repr() of the string, which is how you can see the invisible trailing \n on each line. Very useful when debugging file reads.

🔁 Iterating Line by Line — The Memory-Safe Way

The idiomatic way to read a file of any size: iterate directly over the file object. Python reads lazily, one line at a time, so memory usage stays constant even for multi-gigabyte files.

line_count = 0
word_count = 0

with open("book.txt", encoding="utf-8") as f:
    for line in f:
        line_count += 1
        word_count += len(line.split())

print(f"Lines: {line_count}, Words: {word_count}")

💡 Always prefer for line in f over f.readlines() unless you genuinely need the whole list in memory. The iteration form is both faster and more memory-efficient.

🧹 Stripping Whitespace Safely

Every line from for line in f or .readline() includes a trailing \n. Strip it before comparing, printing, or storing. Use .rstrip("\n") to remove only the newline — .strip() removes all whitespace including leading spaces you might want to keep.

with open("names.txt", encoding="utf-8") as f:
    names = [line.rstrip("\n") for line in f if line.strip()]

print(names)

💡 The condition if line.strip() skips blank lines — a very common pattern when reading a file where you want only meaningful content.

🧮 Processing in Chunks — For Very Large Binary Files

Line iteration works for text. For very large binary files (video, datasets, archives) you iterate in fixed-size byte chunks instead. Memory stays constant and you can process a 10 GB file on a machine with 1 GB of RAM.

def checksum(path: str, chunk: int = 65536) -> int:
    total = 0
    with open(path, "rb") as f:
        while True:
            block = f.read(chunk)
            if not block:
                break
            total += sum(block)
    return total

with open("big.bin", "wb") as f:
    f.write(b"sample data for checksum" * 1000)

print(f"Checksum: {checksum('big.bin')}")

💡 A chunk size of 64 KB (65536 bytes) is a sensible default — it matches typical OS read buffers. A zero-length result means the file is exhausted.

✍️ Lesson 3 — Writing Files

Creating new files and overwriting existing ones

🖊️ `.write()` and `.writelines()`

.write() accepts a single string and returns the number of characters written. It does not add a newline automatically — you supply the \n yourself. .writelines() accepts an iterable of strings and writes them all in sequence.

with open("greeting.txt", "w", encoding="utf-8") as f:
    f.write("Hello, world!\n")
    f.write("This is the second line.\n")
    f.write("And a third.\n")

lines = ["apple\n", "banana\n", "cherry\n"]
with open("fruits.txt", "w", encoding="utf-8") as f:
    f.writelines(lines)

⚠️ If you write to an existing file in "w" mode, the previous content is completely erased before the first new byte is written. There is no undo.

🖨️ `print(..., file=f)`

Sometimes cleaner than .write() — print() automatically adds newlines and calls str() on non-string arguments. Combine with print()'s other options for quick formatted writes.

scores = [("Ana", 92), ("Bob", 85), ("Cara", 79)]

with open("scores.txt", "w", encoding="utf-8") as f:
    for name, score in scores:
        print(f"{name}: {score}", file=f)

with open("scores.txt", encoding="utf-8") as f:
    print(f.read())

➕ Lesson 4 — Appending with `"a"`

Append mode is write mode that doesn't truncate. The file is created if it doesn't exist. This is the mode you want for logs, journals, and any accumulating record.

import datetime

def log_event(message: str) -> None:
    timestamp = datetime.datetime.now().isoformat(timespec="seconds")
    with open("events.log", "a", encoding="utf-8") as f:
        f.write(f"[{timestamp}] {message}\n")

log_event("User logged in")
log_event("Report generated")

with open("events.log", encoding="utf-8") as f:
    print(f.read())

💡 For real logging in production, use the logging module instead — it handles rotation, levels, and thread safety. The append pattern here is for simple cases and scripts.

🔒 Exclusive Create with `"x"`

When you must never overwrite an existing file — for example, when generating output that a user shouldn't lose by accident — "x" mode is the right choice. It creates the file only if it doesn't already exist.

try:
    with open("report.txt", "x", encoding="utf-8") as f:
        f.write("Report content")
    print("Created new report.txt")
except FileExistsError:
    print("report.txt already exists — refusing to overwrite")

🗂️ Lesson 5 — `pathlib`: The Modern Way

Object-oriented paths that work the same on Windows, macOS, and Linux

The pathlib module (Python 3.4+) replaces string-based path manipulation with a proper Path object. This matters most when your code might run on different operating systems — path separators differ, drive letters exist only on Windows, and string concatenation breaks all of that silently.

🧩 Building Paths Safely

The / operator joins path components and handles the separator automatically. No more os.path.join() chains or manual backslash escaping.

from pathlib import Path

project = Path("myproject")
data_file = project / "data" / "input.csv"

print(f"Full path: {data_file}")
print(f"Name:      {data_file.name}")
print(f"Stem:      {data_file.stem}")
print(f"Suffix:    {data_file.suffix}")
print(f"Parent:    {data_file.parent}")

📖 Reading and Writing with `Path`

Path has shortcut methods that handle the with open(...) boilerplate for the most common cases. They're fine for small files — for large ones, use Path.open() and iterate.

from pathlib import Path

note = Path("note.txt")

note.write_text("Hello from pathlib!\n", encoding="utf-8")

content = note.read_text(encoding="utf-8")
print(content)

with note.open("a", encoding="utf-8") as f:
    f.write("Appended line\n")

print(note.read_text(encoding="utf-8"))

🔍 Checking and Manipulating Paths

Everything you'd want to know about a path, as clean method calls: does it exist, is it a file or directory, what's its size, when was it modified. No more os.path.isfile() triple-checks.

from pathlib import Path

p = Path("note.txt")
print(f"Exists:   {p.exists()}")
print(f"Is file:  {p.is_file()}")
print(f"Is dir:   {p.is_dir()}")
print(f"Size:     {p.stat().st_size} bytes")

folder = Path(".")
for py_file in folder.glob("*.py"):
    print(f"Found: {py_file.name}")

💡 .glob("*.py") matches files in the current directory. .rglob("*.py") searches recursively. .iterdir() lists everything without pattern matching.

📁 Creating and Removing Directories

Path.mkdir(parents=True, exist_ok=True) is the safe idiom for "make this folder and any missing parents, and don't complain if it's already there" — one line replaces the old try: os.makedirs(...) except FileExistsError: pass dance.

from pathlib import Path

output_dir = Path("output / "reports")
output_dir.mkdir(parents=True, exist_ok=True)

report = output_dir / "summary.txt"
report.write_text("Generated report\n", encoding="utf-8")

print(f"Created: {report}")
print(f"Exists:  {report.exists()}")

# Clean up
report.unlink()
output_dir.rmdir()

⚠️ rmdir() only

⚠️ rmdir() only works on empty directories. Use shutil.rmtree() for recursive removal — but be very careful: it does not ask for confirmation.

🆕 New `pathlib` Methods in Python 3.14+

Python 3.14 added a batch of long-requested Path methods that previously required shutil or os. They work on both files and directory trees and read far more naturally.

from pathlib import Path

src = Path("report.txt")
backup = Path("backups") / "report_backup.txt"
backup.parent.mkdir(parents=True, exist_ok=True)

# Copy — preserve_metadata=True keeps timestamps and permissions
src.copy(backup, preserve_metadata=True)

# Copy into a directory (keeps original filename)
src.copy_into("backups")

# Move (atomic on same filesystem via os.replace)
src.move(Path("archive") / "report.txt")

# Delete without error if missing
Path("temp.log").unlink(missing_ok=True)

Method	Purpose	Equivalent before 3.14
`Path.copy(target)`	Copy file/dir to a specific path	`shutil.copy2()`
`Path.copy_into(dir)`	Copy into a directory	`shutil.copy2(src, dir)`
`Path.move(target)`	Move/rename to a specific path	`shutil.move()`
`Path.move_into(dir)`	Move into a directory	`shutil.move(src, dir)`
`Path.unlink(missing_ok=True)`	Delete, ignoring absence	`try/except FileNotFoundError`

✨ New in 3.14 — On filesystems that support copy-on-write (APFS on macOS, Btrfs on Linux, ReFS on Windows), Path.copy() may use lightweight reflinks instead of physically duplicating bytes. For large files this is orders of magnitude faster.

📊 Lesson 6 — CSV and JSON

The two formats you'll touch most often, both fully covered by the standard library

🧾 CSV — Comma-Separated Values

The csv module handles edge cases you'd miss with .split(","): quoted fields containing commas, escaped quotes, and newlines inside fields. Always use it — never split CSV manually.

import csv

rows = [
    {"name": "Ana", "score": 95, "team": "red"},
    {"name": "Bob", "score": 87, "team": "blue"},
    {"name": "Cara", "score": 79, "team": "red"},
]

with open("scores.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score", "team"])
    writer.writeheader()
    writer.writerows(rows)

with open("scores.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(f"{row['name']}: {row['score']} ({row['team']})")

💡 Always pass newline="" when opening CSV files. Without it, Python's universal-newline handling can insert blank lines on some platforms, corrupting the file.

The quoting parameter controls when the writer wraps fields in quotes. The four QUOTE_* constants behave differently — and the choice affects both how the file looks and how reliably it round-trips.

Constant	Behavior	When to use
`QUOTE_MINIMAL`	Default. Quotes only fields containing the delimiter, quotechar, or newlines.	Almost always.
`QUOTE_ALL`	Quotes every field.	When downstream tools expect uniform quoting.
`QUOTE_NONNUMERIC`	Quotes all non-numeric fields. On read, unquoted fields are converted to `float`.	Mixing numeric and text columns you want typed automatically.
`QUOTE_NONE`	Never quotes. Requires `escapechar`, else raises an error.	Legacy formats that forbid quotes.

import csv

data = [["Ana", 95, "red, blue"], ["Bob", 87, "green"]]

with open("quoted.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f, quoting=csv.QUOTE_ALL)
    writer.writerows(data)

with open("quoted.csv", newline="", encoding="utf-8") as f:
    print(f.read())

🔣 JSON — JavaScript Object Notation

JSON is the standard format for structured data exchange — config files, API responses, saved game state, everything. The json module converts between Python objects and JSON text in one call, with no manual string parsing.

import json

data = {
    "user": "Glenn",
    "level": "advanced",
    "scores": [95, 88, 92],
    "active": True,
}

with open("user.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)

with open("user.json", encoding="utf-8") as f:
    loaded = json.load(f)

print(loaded)
print(f"Average: {sum(loaded['scores']) / len(loaded['scores']):.1f}")

💡 json.dump(obj, file) writes to a file; json.dumps(obj) returns a string. Same for load/loads. The s versions are "string" versions — easy to remember.

The ensure_ascii parameter is one of the most consequential. By default it's True, which escapes every non-ASCII character as a \uXXXX sequence. That produces valid JSON, but files full of Chinese, Japanese, or accented characters become unreadable. Setting ensure_ascii=False writes the characters literally — the file stays UTF-8 encoded, readable in any editor.

import json

payload = {"greeting": "Café ☕ — 中文"}

print(json.dumps(payload))
print(json.dumps(payload, ensure_ascii=False))
print(json.dumps(payload, ensure_ascii=False, indent=2))

⚠️ Handling Malformed JSON

Any JSON parsing that touches external data can fail. Wrap json.load in a try/except so a malformed file doesn't crash your whole program.

import json
from pathlib import Path

def load_config(path: Path) -> dict:
    try:
        with path.open(encoding="utf-8") as f:
            return json.load(f)
    except FileNotFoundError:
        print("No config file found — using defaults")
        return {}
    except json.JSONDecodeError as e:
        print(f"Config is malformed at line {e.lineno}: {e.msg}")
        return {}

config = load_config(Path("config.json"))
print("Config keys:", list(config))

🔐 Lesson 7 — Binary Files

Reading and writing non-text data: images, audio, PDFs, serialized objects

🖼️ Bytes, Not Text

Binary modes "rb" and "wb" skip text decoding entirely. You get bytes objects back from reads and must pass bytes to writes. This is how you handle any file that isn't valid UTF-8 text.

# Write bytes
payload = b"\x89PNG\r\n\x1a\n"  # PNG signature
with open("image.png", "wb") as f:
    f.write(payload)

# Read bytes
with open("image.png", "rb") as f:
    data = f.read()

print(f"Read {len(data)} bytes: {data[:8]!r}")

⚠️ Binary mode has no encoding argument. If you pass one, Python raises ValueError: binary mode doesn't take an encoding argument.

📦 Copying Files with `shutil`

For file-to-file operations, shutil is usually simpler than manual binary read-write. It handles buffering, permissions, and edge cases for you.

import shutil
from pathlib import Path

src = Path("image.png")
dst = Path("image_backup.png")

if src.exists():
    shutil.copy2(src, dst)
    print(f"Copied {src.name} → {dst.name}")
    print(f"Size: {dst.stat().st_size} bytes")

💡 shutil.copy2() preserves file metadata (timestamps, permissions). shutil.copy() copies only content. Prefer copy2 unless you have a reason not to.

🔢 Structuring Binary Data with `struct`

When you need to write fixed-width binary records — file formats, network protocols, save games — the struct module converts between Python values and packed bytes. Format strings use single-character codes: i for 4-byte int, d for 8-byte float, H for unsigned 2-byte int.

import struct

# Pack: int (id), float (score), 4-byte string (code)
record = struct.pack("<if4s", 42, 95.5, b"AB12")

with open("record.bin", "wb") as f:
    f.write(record)

with open("record.bin", "rb") as f:
    raw = f.read()

record_id, score, code = struct.unpack("<if4s", raw)
print(f"id={record_id} score={score} code={code.decode()}")

💡 The < prefix means little-endian. Use > for big-endian (network byte order). Pick one and stay consistent — mixing endianness silently corrupts data.

🌍 Lesson 8 — Encoding and Encoding Errors

Why UnicodeDecodeError happens and how to fix it correctly

🔤 The Encoding Problem

A text file is bytes on disk. To read it as text, Python needs to know which encoding maps those bytes to characters. If the encoding used to write the file differs from the one used to read it, non-ASCII characters get mangled or raise UnicodeDecodeError.

from pathlib import Path

text = "Café, naïve, résumé — 中文, 日本語, 한국어"
path = Path("international.txt")

path.write_text(text, encoding="utf-8")

try:
    with path.open(encoding="ascii") as f:
        content = f.read()
except UnicodeDecodeError as e:
    print(f"Failed to decode: {e.reason} at byte {e.start}")

with path.open(encoding="utf-8") as f:
    print(f.read())

🛠️ The `errors=` Parameter

Sometimes you inherit files with broken or mixed encodings and can't fix the source. The errors= parameter tells Python what to do with undecodable bytes:

from pathlib import Path

path = Path("legacy.txt")
path.write_bytes(b"Hello \xff world")  # 0xff is not valid UTF-8

with path.open(encoding="utf-8", errors="replace") as f:
    print("replace:", f.read())

with path.open(encoding="utf-8", errors="ignore") as f:
    print("ignore: ", f.read())

with path.open(encoding="utf-8", errors="backslashreplace") as f:
    print("escape: ", f.read())

💡 errors="replace" is the safest default for user-supplied files — you lose a few characters but the program keeps running. errors="ignore" silently drops bad bytes. errors="strict" (the default) raises immediately.

🧪 Detecting Encoding

For truly unknown files, the third-party charset-normalizer (a dependency of requests) or chardet libraries can guess. The standard library doesn't include automatic detection, but for UTF-8 vs. UTF-8-with-BOM the standard library is enough.

from pathlib import Path

def read_any(path: Path) -> str:
    with path.open("rb") as f:
        raw = f.read()

    if raw.startswith(b"\xef\xbb\xbf"):
        return raw[3:].decode("utf-8")
    try:
        return raw.decode("utf-8")
    except UnicodeDecodeError:
        return raw.decode("latin-1")

p = Path("sample.txt")
p.write_text("héllo", encoding="utf-8")
print(read_any(p))

💡 latin-1 is special: it maps every possible byte value to a character, so it can decode any byte sequence without error. It's a useful fallback but will produce wrong characters for genuinely UTF-8 text. Use it as a last resort, not a default.

✅ Lesson 9 — Best Practices

The rules that prevent 90% of file-handling bugs

✓Always use with — never rely on garbage collection to close files
✓Always specify encoding="utf-8" on text files
✓Always pass newline="" when reading or writing CSV
✓Prefer pathlib.Path over string paths — it's cross-platform and reads better
✓Iterate large files line by line — don't .read() them into memory
✓Catch FileNotFoundError and PermissionError — these are the two most common failure modes
✓Create output directories with parents=True — saves you from "directory not found" errors
✓Never hard-code absolute paths — build them from Path(__file__).parent or the working directory
✓Write to a temp file and rename when writing critical data — this prevents half-written files after a crash
✓Close files before renaming/moving — Windows won't let you rename an open file

🧯 Atomic Writes — When Data Must Not Be Corrupted

If your program crashes halfway through a write, you can be left with a half-written file. For critical data, write to a temporary file first, then atomically rename it into place. If the crash happens before the rename, the original file is untouched.

import tempfile, os
from pathlib import Path

def atomic_write(path: Path, content: str) -> None:
    fd, tmp_path = tempfile.mkstemp(dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            f.write(content)
        os.replace(tmp_path, path)
    except:
        os.unlink(tmp_path)
        raise

target = Path("important.json")
atomic_write(target, '{"ok": true}')
print(target.read_text(encoding="utf-8"))

💡 os.replace() is atomic on POSIX systems and on Windows when the target is on the same filesystem. This is the pattern used by every database, editor, and config-writer that can't afford corruption.

🧭 Building Portable Paths from `file`

Hard-coded paths like "/home/glenn/data.csv" work on exactly one machine. For paths relative to your script — data files, templates, resources shipped alongside the code — anchor them to __file__ so the program works from any working directory.

from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent
DATA_DIR = BASE_DIR / "data"
CONFIG = BASE_DIR / "config.json"

DATA_DIR.mkdir(parents=True, exist_ok=True)
(DATA_DIR / "cache.txt").write_text("cached\n", encoding="utf-8")

print(f"Base:   {BASE_DIR}")
print(f"Config: {CONFIG}")
print(f"Cache:  {(DATA_DIR / 'cache.txt').exists()}")

💡 .resolve() turns __file__ into an absolute, symlink-resolved path. Without it, .parent can be a relative path depending on how the script was launched — the source of endless "works on my machine" bugs.

📋 Lesson 10 — Quick Reference

Every pattern you'll need, on one page

Task	Code
Read whole file	`with open(p, encoding="utf-8") as f: text = f.read()`
Read line by line	`with open(p, encoding="utf-8") as f: for line in f: ...`
Read all lines as list	`with open(p) as f: lines = f.readlines()`
Read in byte chunks	`while (b := f.read(65536)): process(b)`
Write (overwrite)	`with open(p, "w", encoding="utf-8") as f: f.write(s)`
Append	`with open(p, "a", encoding="utf-8") as f: f.write(s)`
Create or fail	`with open(p, "x", encoding="utf-8") as f: f.write(s)`
Read bytes	`with open(p, "rb") as f: data = f.read()`
Write bytes	`with open(p, "wb") as f: f.write(data)`
pathlib read	`Path(p).read_text(encoding="utf-8")`	pathlib write	`Path(p).write_text(s, encoding="utf-8")`
Check existence	`Path(p).exists()`
Make directories	`Path(p).mkdir(parents=True, exist_ok=True)`
Glob files	`Path(d).glob("*.py")`
Recursive glob	`Path(d).rglob("*.py")`
Copy file	`shutil.copy2(src, dst)`
Copy (3.14+)	`Path(src).copy(dst, preserve_metadata=True)`
Move file	`shutil.move(src, dst)` or `Path(src).move(dst)`
Delete file	`Path(p).unlink(missing_ok=True)`
Delete tree	`shutil.rmtree(dir)`
CSV read	`csv.DictReader(f)` with `newline=""`
CSV write	`csv.DictWriter(f, fieldnames=[...])`
JSON read	`json.load(f)` or `json.loads(s)`
JSON write	`json.dump(o, f, indent=2, ensure_ascii=False)`
Binary struct pack	`struct.pack("<if4s", 42, 95.5, b"AB12")`
Atomic write	tempfile + `os.replace(tmp, target)`

📂

File Handling Overview

The broader tour — CSV, JSON, shutil, os.path, and context managers.

Open →

🛡️

Exception Handling

How to catch FileNotFoundError, PermissionError, and UnicodeDecodeError properly.

Open →

🔐

Context Managers

Why with works, and how to write your own.

Open →

❓ Frequently Asked Questions

Common sticking points about file I/O in Python

What's the difference between "w" and "a" mode?

"w" truncates the file — everything previously in it is erased before your write begins. "a" appends — existing content stays, and your new writes go at the end. If a file doesn't exist, both modes create it. Use "w" for fresh output, "a" for logs and accumulating records.

Why do I get UnicodeDecodeError when reading a file?

You're reading a file with one encoding but the bytes were written with a different one — most commonly, reading a UTF-8 file on Windows without specifying encoding="utf-8", so Python tries the platform default (cp1252). Always pass encoding="utf-8". If the file is genuinely a different encoding, pass that instead. If it's mixed or corrupted, use errors="replace" to keep going.

Do I need to call f.close() if I use `with`?

No. The with block calls f.__exit__() automatically, which closes the file — even if an exception is raised inside the block. Calling f.close() explicitly inside a with block is unnecessary and misleading. Just let the block end.

Should I use `os.path` or `pathlib`?

Use pathlib for new code. It's more readable, cross-platform by design, and its Path objects have methods for everything you'd otherwise do with os.path.*. os.path still appears in older code and third-party libraries, so you should recognise it — but prefer Path for anything new.

How do I read a file that's too large to fit in memory?

Iterate over the file object — for line in f: — instead of calling .read() or .readlines(). Python reads lazily, one line at a time, so memory usage stays constant. For very large binary files, use f.read(chunk_size) in a loop to process in blocks. For structured formats like CSV or JSON, the standard-library readers are also streaming by default when you iterate.

How do I check if a file exists before opening it?

Path("file.txt").exists(). But in most cases, checking first is the wrong pattern — between the check and the open() the file can appear or disappear (a race condition). The Pythonic approach is to just open and handle the exception: try: with open(...) as f: ... except FileNotFoundError: .... This is faster (one system call instead of two) and immune to races.

How do I write to a file and immediately read it back?

Close the file first — always. If you write with one with block and read with another, the OS flushes and closes the file between them, and the read sees the fresh content. If you try to read inside the same with block without seeking back to the start, you'll get empty content because the file pointer is at the end. Use f.seek(0) to rewind the pointer.

What is `newline=""` in `open()` for CSV files?

By default Python translates \r\n and \r to \n during text reads and writes ("universal newlines"). The CSV module does its own newline handling and needs to see the raw bytes. If you don't pass newline="", the two layers of newline translation can double up on Windows, producing blank lines between rows. Passing newline="" turns off Python's translation so the CSV module handles it alone.

What does `ensure_ascii=False` actually change in `json.dump`?

By default, json.dump escapes every non-ASCII character as a \uXXXX sequence so the output is pure ASCII. That's valid JSON, but files containing Chinese, Japanese, or accented Latin characters become an unreadable wall of escapes. Setting ensure_ascii=False writes the actual characters into the file, which is still UTF-8 encoded but human-readable. Always pair it with encoding="utf-8" on the file handle, or you'll trade one encoding problem for another.

When would I actually use `errors="ignore"` or `errors="replace"`?

When you're processing files you didn't write and can't fix — user uploads, scraped pages, legacy data exports, logs from mixed systems. errors="replace" substitutes undecodable bytes with � (U+FFFD) so the rest of the file still loads. errors="ignore" silently drops bad bytes entirely. Both are damage-control, not correctness — you accept some data loss to keep the program running. Never use them on files you control; fix the encoding instead.

Is `pathlib` slower than `os.path`?

Historically Path object creation carried a small overhead, but in modern Python (3.11+) most Path methods are thin wrappers around the same underlying os calls, and the performance gap is negligible for real programs. If you're doing millions of path operations per second in a hot loop, profile first — the difference is usually dwarfed by actual filesystem I/O. For clarity and cross-platform safety, pathlib wins.

How do I append a newline only if the file doesn't already end with one?

Read the last byte before appending. With Path: data = p.read_bytes(); needs_newline = data and not data.endswith(b"\n"); p.open("a").write(("\n" if needs_newline else "") + "new line\n"). This matters for logs and append-only files that must stay well-formed even after a crash interrupted a previous write. The check is cheap and prevents lines from gluing together.

Python Reading & Writing Filesopen(), read(), write(), pathlib

📁 Why File I/O Matters

🚪 Lesson 1 — The open() Function

📖 The Basic Signature

🛡️ Always Use with

🌍 Always Specify encoding="utf-8"

📖 Lesson 2 — Reading Files

📄 .read() — The Whole File at Once

📏 .readline() and .readlines()

🔁 Iterating Line by Line — The Memory-Safe Way

🧹 Stripping Whitespace Safely

🧮 Processing in Chunks — For Very Large Binary Files

✍️ Lesson 3 — Writing Files

🖊️ .write() and .writelines()

🖨️ print(..., file=f)

➕ Lesson 4 — Appending with "a"

🔒 Exclusive Create with "x"

🗂️ Lesson 5 — pathlib: The Modern Way

🧩 Building Paths Safely

📖 Reading and Writing with Path

🔍 Checking and Manipulating Paths

📁 Creating and Removing Directories

🆕 New pathlib Methods in Python 3.14+

📊 Lesson 6 — CSV and JSON

🧾 CSV — Comma-Separated Values

🔣 JSON — JavaScript Object Notation

⚠️ Handling Malformed JSON

🔐 Lesson 7 — Binary Files

🖼️ Bytes, Not Text

📦 Copying Files with shutil

🔢 Structuring Binary Data with struct

🌍 Lesson 8 — Encoding and Encoding Errors

🔤 The Encoding Problem

🛠️ The errors= Parameter

🧪 Detecting Encoding

✅ Lesson 9 — Best Practices

🧯 Atomic Writes — When Data Must Not Be Corrupted

🧭 Building Portable Paths from __file__

📋 Lesson 10 — Quick Reference

File Handling Overview

Exception Handling

Context Managers

❓ Frequently Asked Questions

What's the difference between "w" and "a" mode?

Why do I get UnicodeDecodeError when reading a file?

Do I need to call f.close() if I use with?

Should I use os.path or pathlib?

How do I read a file that's too large to fit in memory?

How do I check if a file exists before opening it?

How do I write to a file and immediately read it back?

What is newline="" in open() for CSV files?

What does ensure_ascii=False actually change in json.dump?

When would I actually use errors="ignore" or errors="replace"?

Is pathlib slower than os.path?

How do I append a newline only if the file doesn't already end with one?

Python Reading & Writing Files
open(), read(), write(), pathlib

🚪 Lesson 1 — The `open()` Function

🛡️ Always Use `with`

🌍 Always Specify `encoding="utf-8"`

📄 `.read()` — The Whole File at Once

📏 `.readline()` and `.readlines()`

🖊️ `.write()` and `.writelines()`

🖨️ `print(..., file=f)`

➕ Lesson 4 — Appending with `"a"`

🔒 Exclusive Create with `"x"`

🗂️ Lesson 5 — `pathlib`: The Modern Way

📖 Reading and Writing with `Path`

🆕 New `pathlib` Methods in Python 3.14+

📦 Copying Files with `shutil`

🔢 Structuring Binary Data with `struct`

🛠️ The `errors=` Parameter

🧭 Building Portable Paths from `file`

Do I need to call f.close() if I use `with`?

Should I use `os.path` or `pathlib`?

What is `newline=""` in `open()` for CSV files?

What does `ensure_ascii=False` actually change in `json.dump`?

When would I actually use `errors="ignore"` or `errors="replace"`?

Is `pathlib` slower than `os.path`?