Python Web Scraping
From zero to real data, line by line

⚖️ Is Scraping Legal? The Short Answer

Scraping public data is legal in many jurisdictions (including the US, per the hiQ v. LinkedIn and Van Buren v. United States rulings), but there are important limits:

• Always check robots.txt — a file at example.com/robots.txt that tells bots which paths they may fetch. If it says Disallow: /private/, don't scrape that path.
• Read the Terms of Service — some sites explicitly prohibit automated access. Violating ToS can get your IP banned and, in some cases, has led to civil suits.
• Don't scrape personal data — GDPR and similar laws restrict collecting names, emails, or anything that identifies a person without consent.
• Don't hammer the server — add delays between requests. A scraper that sends 100 requests per second is effectively a denial-of-service attack.
• Never bypass login walls or CAPTCHAs — that's unauthorized access in most legal interpretations.

🧰 The Tools You'll Use in This Course

🧪 Runnable Example — Hello, Web

This first example fetches a small test page and prints its status code and title. It doesn't extract anything yet — it just proves you can reach the web from Python.

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com"
response = requests.get(url)

print("Status code:", response.status_code)
print("Content type:", response.headers.get("Content-Type"))

soup = BeautifulSoup(response.text, "html.parser")
print("Page title:", soup.title.string)

• 1
  import requests — Loads the requests library into memory. requests is the standard Python tool for making HTTP calls (GET, POST, etc.). You'll install it once with pip install requests.
• 2
  from bs4 import BeautifulSoup — Imports only the BeautifulSoup class from the bs4 package. Importing just the class (instead of import bs4) means you can write BeautifulSoup(...) instead of bs4.BeautifulSoup(...).
• 3
  url = "..." — Stores the page address in a variable. Assigning it to a variable up front (rather than inline) makes it easy to change later or loop over multiple URLs.
• 4
  response = requests.get(url) — Sends an HTTP GET request to that URL. The returned response object holds the status code, headers, and body (the HTML). This is the actual network call.
• 5
  print("Status code:", response.status_code) — response.status_code is the HTTP status: 200 means success, 404 means not found, 403 means forbidden, 500 means server error. Always check this — a scraper that assumes 200 will silently fail on error pages.
• 6
  print("Content type:", response.headers.get("Content-Type")) — The headers dictionary contains metadata the server sent back. Content-Type usually reads text/html; charset=utf-8, confirming you got HTML and not, say, JSON or a PDF.
• 7
  soup = BeautifulSoup(response.text, "html.parser") — Creates a parse tree from the raw HTML string. response.text is the body as a Python string. "html.parser" is the parser — it's built into Python so needs no extra install (unlike lxml, which is faster).
• 8
  print("Page title:", soup.title.string) — soup.title grabs the first <title> tag. .string returns its text content. For this page it prints Quotes to Scrape.

🔎 The Response Object — What's Inside

Every call returns a Response object with these key attributes:

• response.status_code — the HTTP status number (200, 404, etc.).
• response.text — the body decoded as a string using the charset in the response headers.
• response.content — the body as raw bytes (use this for images, PDFs, or when text encoding is ambiguous).
• response.headers — a dict of response headers.
• response.url — the final URL after any redirects.
• response.json() — parses the body as JSON and returns a Python dict.

🧪 Runnable Example — Inspect a Response

import requests

url = "https://httpbin.org/html"
response = requests.get(url, timeout=10)

print("Status:", response.status_code)
print("Final URL:", response.url)
print("Encoding:", response.encoding)
print("Body length:", len(response.text), "chars")
print("First 200 chars:")
print(response.text[:200])

• 1
  import requests — Same as Lesson 1. This is now muscle memory.
• 3
  url = "https://httpbin.org/html" — httpbin is a free service that echoes HTTP requests back — perfect for testing without hitting a real site. /html returns a small, valid HTML page.
• 4
  response = requests.get(url, timeout=10) — The timeout=10 argument is critical: if the server doesn't respond within 10 seconds, requests raises a Timeout exception. Without a timeout, your script could hang forever. Never make a request without a timeout in production.
• 6
  print("Status:", response.status_code) — Prints 200. httpbin always returns 200 for valid paths, so this is a good sanity check that the request went through.
• 7
  print("Final URL:", response.url) — Shows the URL after any redirects. If http:// redirected to https://, this would show the final one. Useful for detecting login redirects.
• 8
  print("Encoding:", response.encoding) — Prints something like utf-8. This is the charset requests guessed from headers. If the guess is wrong (rare, but happens), call response.encoding = "utf-8" before reading .text to force it.
• 9
  print("Body length:", len(response.text), "chars") — Reports how many characters the body contains. A length of 0 or a few hundred usually indicates an error page or a block — worth checking.
• 10
  print("First 200 chars:") — A label for the next output line. Pure readability.
• 11
  print(response.text[:200]) — Slices the first 200 characters of the body. The [:200] is Python slicing — characters 0 through 199. This lets you visually confirm the HTML looks right before parsing.

📋 Common Arguments to requests.get()

requests.get(
    url,
    params={"q": "python", "page": 2},
    headers={"User-Agent": "Mozilla/5.0"},
    timeout=10,
    allow_redirects=True,
)

• →
  params — A dict that gets URL-encoded into the query string. The example above becomes ?q=python&page=2. Use this instead of manually building query strings.
• →
  headers — Extra HTTP headers to send. User-Agent is the most important — see Lesson 8.
• →
  timeout — Max seconds to wait. Float allowed (e.g. 2.5).
• →
  allow_redirects — Whether to follow 3xx redirects. Default True.

🖱️ The DevTools Workflow

1. Open the page in Chrome, Firefox, or Edge.
2. Right-click on the data you want (e.g. a product price) → Inspect.
3. The Elements panel opens, highlighting that element in the DOM tree.
4. Look at its tag — is it <span class="price">$19.99</span>? Note the tag, class, and any parent classes.
5. Check whether that class is unique. Click Ctrl+F in DevTools and search for .price to count matches.
6. If it appears many times, refine: use a parent selector like .product-card .price.

<!-- A single product card -->
<div class="product-card" data-id="42">
    <h3 class="product-title">Wireless Headphones</h3>
    <span class="product-price">$89.99</span>
    <a href="/product/42" class="product-link">View details</a>
</div>

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com"
response = requests.get(url, timeout=10)
soup = BeautifulSoup(response.text, "html.parser")

# Find the first quote block
first_quote = soup.select_one(".quote")
if first_quote:
    print(first_quote.prettify()[:600])
else:
    print("No .quote element found.")

🌳 Four Object Types You Need to Know

🔍 find vs find_all vs select

These three methods are the heart of BS4. Understanding the difference saves you hours of confusion:

• find(name, attrs) — Returns the first matching Tag, or None.
• find_all(name, attrs) — Returns a list of all matching Tags. Empty list if none found.
• select(css_selector) — Returns a list of Tags matching a CSS selector. More powerful than find_all for complex queries.
• select_one(css_selector) — Returns the first match, or None.

🧪 Runnable Example — Extract All Quotes and Authors

This is the classic "hello world" of scraping. It extracts every quote and author from the test site.

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com"
soup = BeautifulSoup(requests.get(url, timeout=10).text, "html.parser")

quotes = soup.find_all("div", class_="quote")
print(f"Found {len(quotes)} quotes")

for q in quotes[:3]:
    text = q.find("span", class_="text").get_text(strip=True)
    author = q.find("small", class_="author").get_text(strip=True)
    print(f"{text} — {author}")

• 1–2
  Imports — same as before.
• 4
  url = "https://quotes.toscrape.com" — The practice site.
• 5
  soup = BeautifulSoup(requests.get(url, timeout=10).text, "html.parser") — Combines three operations into one line: (a) requests.get() downloads the page, (b) .text gives the HTML as a string, (c) BeautifulSoup(...) parses it. If you find this hard to read, split it into three lines. Readability > brevity.
• 7
  quotes = soup.find_all("div", class_="quote") — Finds every <div> whose class attribute equals "quote". Note the trailing underscore on class_ — class is a Python keyword, so BeautifulSoup uses class_ (with underscore) to avoid the conflict. Returns a list of Tag objects.
• 8
  print(f"Found {len(quotes)} quotes") — len(quotes) tells you how many matches. This is a key diagnostic — if it prints 0, your selector is wrong. If it prints a huge number, you probably over-matched.
• 10
  for q in quotes[:3]: — Loops through the first three quote blocks. The [:3] slice is just for compact output — remove it in production to process all quotes.
• 11
  text = q.find("span", class_="text").get_text(strip=True) — Inside each quote block, find the <span class="text"> tag. .get_text(strip=True) returns the text content with leading/trailing whitespace removed. Without strip=True, you'd get lots of \n padding.
• 12
  author = q.find("small", class_="author").get_text(strip=True) — Same pattern, but for the author. Notice that q.find() (not soup.find()) scopes the search inside the current quote — it won't accidentally match an author from a different quote. This is the fundamental "scope then extract" pattern.
• 13
  print(f"{text} — {author}") — Formats each result with an em-dash separator. In an f-string, {text} and {author} are replaced with their values.

🔧 Other Ways to Navigate the Tree

# Assume you have a Tag called `tag`
tag.parent          # Immediate parent Tag
tag.parents         # Generator of all ancestors
tag.children        # Generator of direct children
tag.descendants     # Generator of ALL descendants
tag.next_sibling    # Next sibling Tag
tag.previous_sibling
tag.find_next("a")  # Next  after this tag
tag.find_parent("div") # Nearest ancestor 

⚖️ find_all vs select — Which to Use?

• 1
  soup.find_all("div", class_="card") — Simple and fast. Matches all <div class="card">.
• 2
  soup.select("div.card") — Same result, CSS syntax. Slightly slower but more flexible once you add combinators.
• 3
  soup.select("div.card > a[href]") — find_all can't express this directly; you'd have to do find_all("div", class_="card") then iterate and call .find_all("a", href=True). select does it in one call.
• 4
  soup.find_all("a", href=re.compile("^/product/")) — Passing a compiled regex as an attribute value is a find_all-only trick — very useful for matching URL patterns.

🧪 Runnable Example — Scoping with Parent Selectors

Let's scrape product cards from a listing page. We'll use select with a parent selector to scope extraction to each card, then pull out title, price, and link.

import requests
from bs4 import BeautifulSoup

url = "https://quotes.toscrape.com"
soup = BeautifulSoup(requests.get(url, timeout=10).text, "html.parser")

cards = soup.select("div.quote")
print(f"Cards found: {len(cards)}")

for card in cards[:2]:
    title  = card.select_one("span.text")
    author = card.select_one("small.author")
    tags   = [t.get_text(strip=True) for t in card.select("a.tag")]

    print("TITLE:", title.get_text(strip=True) if title else "(none)")
    print("AUTHOR:", author.get_text(strip=True) if author else "(none)")
    print("TAGS:", ", ".join(tags))
    print("-" * 50)

• 1–2
  Standard imports.
• 4–5
  Fetch and parse the page as before.
• 7
  cards = soup.select("div.quote") — Uses CSS selector syntax. div.quote means "a <div> with class quote". Same result as soup.find_all("div", class_="quote") from Lesson 4 — choose whichever reads better to you.
• 8
  print(f"Cards found: {len(cards)}") — Sanity check. Empty list here means the site changed or you were blocked.
• 10
  for card in cards[:2]: — Loop over the first two. In a real scraper, you'd loop over all cards.
• 11
  title = card.select_one("span.text") — Inside this card, find the first <span class="text">. select_one returns None if no match — that's why we check for None below.
• 12
  author = card.select_one("small.author") — Same pattern. The small.author selector means "a <small> element with class author". If the site used a <span class="author">, you'd change it to span.author.
• 13
  tags = [t.get_text(strip=True) for t in card.select("a.tag")] — A list comprehension. card.select("a.tag") returns all <a> tags with class tag. For each t, we call t.get_text(strip=True) to extract its text. The whole expression builds a list like ["change", "deep-thoughts", "thinking"].
• 15
  print("TITLE:", title.get_text(strip=True) if title else "(none)") — Ternary expression: "if title is not None, call .get_text() on it; otherwise print (none)". This prevents an AttributeError when the title is missing.
• 16
  Same pattern for author — null-safe extraction.
• 17
  print("TAGS:", ", ".join(tags)) — ", ".join(tags) joins the list of strings with , in between — producing change, deep-thoughts, thinking. If the list is empty, join returns "".
• 18
  print("-" * 50) — Prints 50 hyphens as a divider. "-" * 50 in Python repeats the string 50 times. Useful for readability in log output.

🎯 Selector Cheat Sheet

# Class
soup.select(".price")                # all elements with class="price"

# ID (unique)
soup.select("#main")                 # the element with id="main"

# Descendant
soup.select(".product-card .price")  # .price anywhere inside a .product-card

# Direct child
soup.select(".product-card > h3")    # h3 that is direct child of .product-card

# Attribute
soup.select("a[href]")                #  with any href
soup.select("a[data-id='42']")        # exact attribute match

# Pseudo-classes
soup.select("li:first-child")         # first 
 in each list
soup.select("li:nth-child(2n)")       # every other 

📦 Three Ways to Read an Attribute

• 1
  tag["href"] — Dictionary access. Raises KeyError if the attribute doesn't exist. Only use this when you're certain the attribute is present.
• 2
  tag.get("href") — Safe dictionary access. Returns None if missing. This is the version you should use 95% of the time.
• 3
  tag.get("href", "/") — Safe access with a default value. Useful when you want a fallback.
• 4
  tag.attrs — The full attribute dictionary. Useful for inspecting what's available.

🧪 Runnable Example — Absolute URLs from Relative Hrefs

A super common problem: the site uses relative URLs like href="/product/42", but you want the absolute URL https://example.com/product/42. Here's the pattern:

import requests
from bs4 import BeautifulSoup
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

base_url = "https://quotes.toscrape.com"
soup = BeautifulSoup(requests.get(base_url, timeout=10).text, "html.parser")

# Find all author links and print their absolute URLs
author_links = soup.select("div.quote span a")
print(f"Found {len(author_links)} author links")

for link in author_links[:3]:
    relative = link.get("href")
    absolute = urljoin(base_url, relative)
    print(f"Relative: {relative}")
    print(f"Absolute: {absolute}")
    print()

• 1–2
  Standard imports as before.
• 3
  from urllib.parse import urljoin — urljoin is a standard library function that correctly resolves relative URLs against a base. It handles ./, ../, absolute paths, and full URLs — everything.
• 5
  base_url = "https://quotes.toscrape.com" — The base to resolve against. It's the URL we fetched, which is the correct base for all links on that page.
• 6
  soup = BeautifulSoup(requests.get(base_url, timeout=10).text, "html.parser") — Download and parse.
• 9
  author_links = soup.select("div.quote span a") — CSS selector chain: any <a> that is a descendant of a <span> that is a descendant of a div.quote. On this site, that's the "about the author" link inside each quote block. Spaces between selectors mean "descendant of" — you can chain as many as you want.
• 10
  print(f"Found {len(author_links)} author links") — Sanity check.
• 12
  for link in author_links[:3]: — Loop through the first 3 links.
• 13
  relative = link.get("href") — Extract the href attribute with .get(), which returns None if missing (rather than raising KeyError). For this site, it returns something like "/author/Albert-Einstein".
• 14
  absolute = urljoin(base_url, relative) — urljoin combines the two. If relative is "/author/Albert-Einstein", the result is "https://quotes.toscrape.com/author/Albert-Einstein". If relative is already an absolute URL, urljoin returns it unchanged. If relative is None, this would raise TypeError — that's why we should usually add a safety check (see the note below).
• 15–16
  Print both the raw and resolved URL. This pattern is essential when scraping paginated sites, where next page links are often relative (href="?page=2").
• 17
  print() — Blank line for readability between entries.

🖼️ Extracting Images and Other Attributes

The same pattern applies to any attribute. Here's a compact reference:

# Image source
img = soup.select_one("img.product-photo")
src = img.get("src") if img else None
alt = img.get("alt", "") if img else ""

# Data attributes (used by JS frameworks)
card = soup.select_one("[data-product-id]")
product_id = card.get("data-product-id") if card else None

# datetime on  tags
time_tag = soup.select_one("time[datetime]")
published = time_tag.get("datetime") if time_tag else None

# Multiple classes — BS4 splits into a list
classes = card.get("class")  # ['product-card', 'featured']
is_featured = "featured" in (classes or [])

• 1–3
  img = soup.select_one("img.product-photo") then .get("src") — Gets the image URL. The if img else None guard prevents AttributeError if the image doesn't exist.
• 5–7
  [data-product-id] — An attribute-presence selector. Matches any element that has that attribute, regardless of its value. Modern React/Vue apps often use data-* attributes for state — they're a goldmine for scrapers.
• 9–11
  time[datetime] — Matches a <time> tag that has a datetime attribute. Semantic HTML uses this for machine-readable timestamps — much more reliable than scraping visible text.
• 13–15
  Notice: .get("class") returns a list, not a string. So "featured" in classes works, but classes == "featured" does not. The or [] guard handles the case where classes is None.

🔢 The Three Pagination Patterns

🧪 Runnable Example — Walk All Pages with the Next-Link Pattern

quotes.toscrape.com uses the next-link pattern. Each page ends with a <li class="next"><a href="/page/2/">Next</a></li>. Here's the full loop that scrapes every page until there is no next link:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

base = "https://quotes.toscrape.com"
url  = base
all_quotes = []
page = 1

while url:
    print(f"--- Page {page}: {url}")
    response = requests.get(url, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")

    for q in soup.select("div.quote"):
        text   = q.select_one("span.text")
        author = q.select_one("small.author")
        if text and author:
            all_quotes.append({
                "text":   text.get_text(strip=True),
                "author": author.get_text(strip=True),
            })

    next_link = soup.select_one("li.next > a")
    if next_link and next_link.get("href"):
        url = urljoin(base, next_link["href"])
        page += 1
    else:
        url = None

print(f"\nTotal quotes scraped: {len(all_quotes)}")
print("First:", all_quotes[0] if all_quotes else "none")

• 1–3
  Imports — urljoin is important here because the "next" links are relative (/page/2/).
• 5–7
  base = the site root, url = the current page (starts at base), all_quotes = accumulator list, page = counter for log output.
• 9
  while url: — Loops as long as url is truthy. Since we'll set url = None when there's no next page, this loop naturally terminates. This is the cleanest possible pagination loop.
• 10
  Log the current page and URL — invaluable when debugging or monitoring a long scrape.
• 11
  response = requests.get(url, timeout=10) — Fetch the current page.
• 12
  soup = BeautifulSoup(response.text, "html.parser") — Parse it.
• 14
  for q in soup.select("div.quote"): — Iterate over every quote card on this page.
• 15–16
  Extract text and author Tag objects. We keep the Tag objects (not the strings) so we can null-check them before calling .get_text().
• 17
  if text and author: — Only append when both are present. This is defensive — a malformed card shouldn't crash the whole scrape.
• 18–20
  all_quotes.append({...}) — Append a dict with the extracted strings. Now each quote is a structured record, ready to be exported to CSV or JSON later.
• 22
  next_link = soup.select_one("li.next > a") — The > combinator means "direct child". This finds an <a> that is a direct child of <li class="next">. If there's no next page, this returns None.
• 23
  if next_link and next_link.get("href"): — Two checks: does the element exist, and does it have an href? Only then is it safe to use.
• 24
  url = urljoin(base, next_link["href"]) — Resolve the relative href against the base. On page 1, this is /page/2/ → https://quotes.toscrape.com/page/2/.
• 25
  page += 1 — Increment the counter for logging.
• 26–27
  else: url = None — When there's no next link, terminate the loop. This is the natural stopping condition. Without it, you'd loop forever.
• 29
  print(f"\nTotal quotes scraped: {len(all_quotes)}") — Prints the total count. The \n adds a newline before the output.
• 30
  print("First:", all_quotes[0] if all_quotes else "none") — Safe access to the first element — if the list is empty (site blocked you, or selector broke), you get "none" instead of IndexError.

🔢 The Page-Number Pattern

Some sites just use ?page=N in the URL. No next-link needed — loop over a range:

for page_num in range(1, 11):
    url = f"https://example.com/products?page={page_num}"
    response = requests.get(url, timeout=10)
    if response.status_code != 200:
        print(f"Stopping: page {page_num} returned {response.status_code}")
        break
    soup = BeautifulSoup(response.text, "html.parser")
    if not soup.select(".product-card"):
        print(f"No products on page {page_num} — stopping.")
        break

• 1
  for page_num in range(1, 11): — Loop 1 through 10. In practice, you'd determine the max from the site (or just keep going until empty).
• 2
  url = f"https://example.com/products?page={page_num}" — Build each URL. The f-string interpolates the page number.
• 3
  Fetch. The timeout is non-negotiable.
• 4–6
  Check for non-200 status. If page 11 returns 404, break out of the loop early. Otherwise you'd waste requests on pages that don't exist.
• 7
  Parse.
• 8–10
  if not soup.select(".product-card"): — If the page has no products, stop. This is the key termination condition for sites where the total page count isn't known in advance.

🪪 Why Headers Matter — The User-Agent

By default, Python's requests library sends User-Agent: python-requests/2.31.0. Every server admin knows this string, and many sites immediately block or CAPTCHA it. Setting a realistic User-Agent makes your requests look like a browser. Note: this isn't about hiding — it's about being treated the same as a normal visitor.

🧪 Runnable Example — Requests with Custom Headers

import requests

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}

url = "https://httpbin.org/headers"
response = requests.get(url, headers=headers, timeout=10)

print("Status:", response.status_code)
print("Server saw headers:")
print(response.json()["headers"]["User-Agent"])

• 1
  Import requests.
• 3–10
  headers = {...} — A dictionary of HTTP headers. Each key is a header name, each value is its content.
• 4–7
  "User-Agent": (...) — The string is split across multiple lines using parentheses (Python's implicit line continuation inside parentheses). This is a fake Chrome-on-Windows User-Agent — the most common browser string, so it looks normal. Rotate this string across requests for large scrapes (see below).
• 8
  "Accept": ... — Tells the server what content types we accept. Browsers send this; setting it makes the request look more like one.
• 9
  "Accept-Language": ... — Language preferences. Minor, but adds realism.
• 12
  url = "https://httpbin.org/headers" — httpbin's /headers endpoint echoes back all headers it received. Perfect for testing.
• 13
  response = requests.get(url, headers=headers, timeout=10) — Sends the GET with our headers. requests merges your custom headers with its defaults (like Accept-Encoding).
• 15
  Status — should be 200.
• 16
  Label line.
• 17
  response.json()["headers"]["User-Agent"] — .json() parses the body as JSON. We index into headers then User-Agent. The printed value should match what we sent — confirming our header was transmitted.

⏱️ Rate Limiting — time.sleep() Done Right

Sending 100 requests per second will get you blocked within minutes. Even for small scrapes, add a delay between requests. Here's the pattern:

import requests, time, random

urls = [f"https://quotes.toscrape.com/page/{n}/" for n in range(1, 5)]

for url in urls:
    response = requests.get(url, timeout=10)
    print(f"{response.status_code} ← {url}")

    # Random delay between 1.5 and 3.5 seconds
    delay = random.uniform(1.5, 3.5)
    print(f"  sleeping {delay:.2f}s...")
    time.sleep(delay)

• 1
  Imports: requests, time (for sleep), and random (to vary the delay).
• 3
  urls = [...] — List comprehension building 4 page URLs. We could loop over range directly, but pre-building the list is cleaner when you might want to inspect or retry it.
• 5
  Loop over each URL.
• 6
  Send the request with a timeout.
• 7
  Log the status and URL. The arrow ← visually points from status to URL.
• 10
  delay = random.uniform(1.5, 3.5) — Why random? Because a perfectly regular 2.0-second interval is itself a fingerprint. Real users don't click in perfect rhythm. Random delays between 1.5 and 3.5 seconds mean your traffic pattern looks human.
• 11
  Log the sleep — helpful when debugging long runs.
• 12
  time.sleep(delay) — Blocks the current thread for delay seconds before the next iteration. In an async scraper, you'd use await asyncio.sleep(delay) instead — but for a simple synchronous scraper, time.sleep is exactly right.

🚨 Detecting Blocks

You'll know you've been blocked if:

• Status code is 403 (Forbidden) or 429 (Too Many Requests)
• The body contains phrases like "access denied", "unusual traffic", or "captcha"
• The body is much smaller than usual (a block page instead of the real content)
• Your selector returns 0 matches after previously returning hundreds

def is_blocked(response):
    if response.status_code in (403, 429):
        return True
    body = response.text.lower()
    for phrase in ("access denied", "unusual traffic", "captcha"):
        if phrase in body:
            return True
    return False

• 1
  def is_blocked(response): — A reusable helper. Takes a Response, returns True/False.
• 2
  Check for the two most common blocking status codes. Return True immediately if either appears.
• 4
  body = response.text.lower() — Lowercase the body once so all phrase checks become case-insensitive.
• 5–7
  Loop through a list of suspicious phrases. If any appear in the body, it's a block page. Return True.
• 8
  If we got here, no signals were triggered — the response looks legitimate.

📄 Saving to CSV — The DictWriter Pattern

CSV is the right format when your data is a flat table: one row per item, one column per field. The csv.DictWriter class is the cleanest way to write a list of dicts.

import csv

quotes = [
    {"text": "The world as we have created it...", "author": "Albert Einstein"},
    {"text": "It is our choices, Harry...",         "author": "J.K. Rowling"},
    {"text": "There are only two ways to live...",   "author": "Albert Einstein"},
]

with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["text", "author"])
    writer.writeheader()
    writer.writerows(quotes)

print("Wrote quotes.csv with", len(quotes), "rows")

• 1
  import csv — Built into Python. No pip install.
• 3–7
  quotes = [...] — A list of dicts. This is exactly the shape our scraper produces: each item is a record with named fields. If your scraper collects records into a list like this, CSV export is a one-liner.
• 9
  with open("quotes.csv", "w", newline="", encoding="utf-8") as f: — Opens a file for writing. Three critical details:
• —
    "w" — Mode "write": creates the file, or truncates an existing one. Use "a" (append) if you want to add rows to an existing CSV across runs.
• —
    newline="" — Never omit this on Windows. Without it, every row ends up double-spaced because csv writes \r\n and Python adds another \n on top. Passing "" tells Python not to translate line endings — the csv module handles them itself.
• —
    encoding="utf-8" — Explicit encoding. Without this, the default depends on your OS — and on Windows it's often a legacy codec that mangles non-ASCII characters like accented names or em-dashes. Always specify utf-8.
• 10
  writer = csv.DictWriter(f, fieldnames=["text", "author"]) — Creates a DictWriter. fieldnames is the list of columns, in order. Each dict in quotes must have these keys.
• 11
  writer.writeheader() — Writes the first row: text,author. Essential — without it, your CSV has no column labels.
• 12
  writer.writerows(quotes) — Writes every dict in quotes as a row. Each dict's values become the cell contents, correctly escaped (quotes, commas, newlines inside strings).
• 14
  print(...) — Confirms the write.

🧾 Saving to JSON — Structured Data

JSON is the right format for nested data: a quote with a list of tags, a product with a nested dimensions dict. It's also the format that JavaScript, APIs, and most modern tools speak natively.

import json

quotes = [
    {"text": "The world as we have created it...",
     "author": "Albert Einstein",
     "tags": ["change", "deep-thoughts", "thinking"]},
    {"text": "It is our choices, Harry...",
     "author": "J.K. Rowling",
     "tags": ["abilities", "choices"]},
]

with open("quotes.json", "w", encoding="utf-8") as f:
    json.dump(quotes, f, indent=2, ensure_ascii=False)

# Read back to prove round-trip
with open("quotes.json", encoding="utf-8") as f:
    loaded = json.load(f)

print("Records:", len(loaded))
print("First author:", loaded[0]["author"])

• 1
  import json — Built-in.
• 3–10
  A list of dicts where each dict has a nested tags list. CSV would struggle here (it would have to encode the list as a comma-separated string inside one cell); JSON handles nesting natively.
• 12
  with open("quotes.json", "w", encoding="utf-8") as f: — Note there is no newline="" here. That argument is only for the csv module, not for JSON.
• 13
  json.dump(quotes, f, indent=2, ensure_ascii=False) — Serialises the list to JSON:
• —
    indent=2 — Pretty-prints the JSON with 2-space indentation. Much easier to inspect manually. Omit for a compact one-line file (faster to parse programmatically, but unreadable to humans).
• —
    ensure_ascii=False — By default, json.dump escapes non-ASCII characters as \uXXXX sequences. Setting False keeps them as readable characters — essential for names, accents, em-dashes, etc. With encoding="utf-8", this is always safe.
• 16–17
  Read the file back to prove round-trip — the value we wrote comes back out exactly.
• 18
  loaded = json.load(f) — Parses the JSON from file into a Python object. json.load (file object) vs json.loads (string) is a common point of confusion — remember s = string.
• 20–21
  Print the count and the first author to prove the data made it through intact.

🧮 CSV vs JSON — Which to Use?

🛡️ The Four Layers of Scraper Resilience

🧪 Runnable Example — A Resilient Fetch-and-Extract Function

This is the pattern you should copy into every scraper. It handles each failure layer explicitly and returns None or a partial record rather than crashing.

import requests, time
from bs4 import BeautifulSoup
from requests.exceptions import RequestException, Timeout, HTTPError

def fetch(url, retries=3, backoff=2.0, timeout=10):
    for attempt in range(1, retries + 1):
        try:
            response = requests.get(url, timeout=timeout)
            response.raise_for_status()
            return response
        except HTTPError as e:
            print(f"  [{attempt}] HTTP {response.status_code} — {e}")
            if 400 <= response.status_code < 500:
                return None
        except Timeout:
            print(f"  [{attempt}] Timeout after {timeout}s")
        except RequestException as e:
            print(f"  [{attempt}] Request error: {e}")

        if attempt < retries:
            wait = backoff ** attempt
            print(f"  retrying in {wait}s...")
            time.sleep(wait)

    print(f"  Giving up on {url}")
    return None

# Usage — parse only if we actually got HTML
response = fetch("https://quotes.toscrape.com")
if response is not None:
    soup = BeautifulSoup(response.text, "html.parser")
    print("Got page, title:", soup.title.string)

• 1–3
  Imports. The requests.exceptions module has specific exception classes. Catching RequestException (the base class) catches everything network-related, while Timeout and HTTPError are more specific.
• 5
  def fetch(url, retries=3, backoff=2.0, timeout=10): — A reusable helper. Default arguments mean callers don't have to specify everything every time.
• 6
  for attempt in range(1, retries + 1): — Attempt 1, 2, 3. Not 0-based, because "attempt 0" reads badly in logs.
• 7
  try: — All network code lives inside the try. Anything that raises will jump to an except clause.
• 8
  response = requests.get(url, timeout=timeout) — The actual request. If it takes longer than timeout, a Timeout is raised.
• 9
  response.raise_for_status() — The single most important line. This raises an HTTPError for 4xx and 5xx statuses. Without it, requests happily returns a 404 as a "successful" Response — and you'd try to parse an error page as if it were your content.
• 10
  return response — Success. Exit the function immediately. No further attempts needed.
• 11–13
  except HTTPError as e: — Catches 4xx/5xx. e is the exception; response.status_code tells us which. If it's 4xx (client error — bad URL, blocked, forbidden), retrying won't help, so return None immediately. If it's 5xx (server error — temporary), fall through to the retry logic.
• 14–15
  except Timeout: — Timeouts are always worth retrying; the server may just be having a bad moment.
• 16–17
  except RequestException as e: — Connection errors, DNS failures, SSL errors. Retry, but log the error so you can diagnose later.
• 19–22
  If we're still inside the loop after an exception, wait before the next attempt. backoff ** attempt = 2^1, 2^2, 2^3 = 2s, 4s, 8s. This is exponential backoff — the standard pattern for retrying transient failures without hammering the server.
• 24–25
  If the loop finishes without returning, we've exhausted all retries. Return None — a clear signal to the caller that this URL failed permanently.
• 28–31
  Usage: call fetch(), then check the return value. If it's None, skip this URL. This is how a production scraper isolates failure — one bad URL doesn't kill the whole run.

🧩 Per-Item Error Recovery — Don't Let One Bad Row Kill the Batch

Even with network handling in place, individual items can fail extraction. A missing element, a malformed number, a typo in one card — these shouldn't stop the whole scrape. Wrap extraction in its own try/except per item:

results = []
errors = []

for card in soup.select("div.quote"):
    try:
        text   = card.select_one("span.text").get_text(strip=True)
        author = card.select_one("small.author").get_text(strip=True)
        results.append({"text": text, "author": author})
    except AttributeError as e:
        errors.append(f"Skipped card: {e}")

print(f"Success: {len(results)} | Failed: {len(errors)}")

• 1–2
  Two accumulators: results for successes, errors for failures. Tracking both is important — you want to know how many items failed, not just that the run completed.
• 4
  Loop over cards.
• 5
  try: — The entire extraction for this card is atomic. If any line in here fails, we catch it and continue with the next card.
• 6–7
  Extract both fields. If select_one returns None (element missing), .get_text() on None raises AttributeError — which we catch below.
• 8
  Append the record. This line only runs if both extractions succeeded — the record is guaranteed complete.
• 9
  except AttributeError as e: — Catch specifically AttributeError, not bare except:. This catches missing-element errors but still surfaces unexpected errors (like a KeyboardInterrupt or a bug in our own code) instead of silently swallowing them.
• 10
  Log the failure. Even though we can't fix it here, we know how many failed at the end.
• 12
  Final count — a data-quality report. "Success: 47 | Failed: 3" tells you the scrape was mostly fine but had a few bad cards. "Success: 0 | Failed: 50" tells you the selector broke entirely.

🌐 Why requests Fails on SPA Sites

Here's the difference:

🎭 Playwright — The Modern Browser Automation Tool

Playwright controls real Chrome, Firefox, or WebKit. It's the current best-in-class tool for browser automation (better than older Selenium for most tasks — faster, more reliable, better async support). Install:

# One-time install (in your terminal, not Python)
pip install playwright
playwright install chromium

🧪 Example — Scrape a JS-Rendered Page

Here's a minimal Playwright scraper that waits for content to load, extracts data, and closes the browser cleanly:

from playwright.sync_api import sync_playwright

def scrape_dynamic_page(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")

        # Wait for the data to appear in the DOM
        page.wait_for_selector(".product-card", timeout=10000)

        cards = page.query_selector_all(".product-card")
        results = []
        for card in cards:
            title = card.query_selector(".product-title")
            price = card.query_selector(".product-price")
            results.append({
                "title": title.inner_text() if title else None,
                "price": price.inner_text() if price else None,
            })

        browser.close()
        return results

# results = scrape_dynamic_page("https://example-spa.com/products")
# print(results)

• 1
  from playwright.sync_api import sync_playwright — The synchronous API. Playwright also has an async API (async_api) — use that if you're already in an asyncio project. For most scrapers, the sync API is easier to reason about.
• 3
  def scrape_dynamic_page(url): — A self-contained function. Everything is scoped inside — no global state.
• 4
  with sync_playwright() as p: — The context manager that starts Playwright and guarantees it shuts down at the end. Even if an exception is raised mid-scrape, the browser is closed cleanly.
• 5
  browser = p.chromium.launch(headless=True) — Launches a Chromium instance in headless mode (no visible
• 5
  browser = p.chromium.launch(headless=True) — Launches a Chromium instance in headless mode (no visible window). Set headless=False while developing — you'll see the browser open and can watch exactly what's happening, which is enormously helpful when a selector isn't matching.
• 6
  page = browser.new_page() — Creates a new tab. All subsequent actions happen on this page object.
• 7
  page.goto(url, wait_until="networkidle") — Navigates to the URL and waits until the network has been quiet for 500ms. Other options: "load" (DOM + resources loaded), "domcontentloaded" (DOM only), "commit" (response received). "networkidle" is the safest default for JS sites because it waits for post-load API calls to finish.
• 10
  page.wait_for_selector(".product-card", timeout=10000) — Waits up to 10 seconds for at least one .product-card to appear in the DOM. This is critical: networkidle is a good heuristic, but some apps render after that. This explicit wait is the reliable way to know the content is there.
• 12
  cards = page.query_selector_all(".product-card") — Returns a list of ElementHandle objects — the Playwright equivalent of soup.select(".product-card"). Note: query_selector_all is the sync API method; the async version is await page.query_selector_all().
• 13
  results = [] — Accumulator for the parsed records.
• 14
  for card in cards: — Loop over each card element.
• 15–16
  card.query_selector(...) — Scoped selection inside the card, identical in spirit to q.select_one(...) from BS4. The pattern carries over perfectly.
• 17–20
  Build the record. .inner_text() is Playwright's equivalent of .get_text() — it returns the rendered text, which for Playwright means text you'd see on screen (CSS display:none elements are excluded, unlike textContent). Null-safe if ... else None guards against missing elements.
• 22
  browser.close() — Cleanly close the browser. The with block would also close it, but being explicit means you can close early and start another scrape.
• 23
  return results — Return the extracted data.

⚖️ requests vs Playwright — When to Use Which

• 1
  Try requests: fetch the page, search the raw HTML for a known phrase (like the product name). If it's there, use requests + BS4.
• 2
  Try the API: open DevTools → Network tab → filter to XHR/Fetch. Many SPAs call a JSON API behind the scenes. If you can find that endpoint, hit it directly with requests — you get clean JSON instead of fragile HTML. This is the single biggest "pro tip" for modern scraping.
• 3
  Use Playwright: only if neither works — the page requires JS to render and there's no public API. Accept that it's slower and heavier.

📋 The Full Project — Pulling Every Lesson Together

Read the code, then the line-by-line breakdown below. Every line is a technique from an earlier lesson.

import requests, csv, json, time, random
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
from urllib.robotparser import RobotFileParser

BASE = "https://quotes.toscrape.com"

# 1. Respect robots.txt
def can_scrape(url):
    rp = RobotFileParser()
    rp.set_url(urljoin(url, "/robots.txt"))
    try:
        rp.read()
        return rp.can_fetch("*", url)
    except Exception:
        return False

# 2. Fetch with retries (from Lesson 10)
def fetch(url, retries=3):
    for attempt in range(1, retries + 1):
        try:
            r = requests.get(url, timeout=10,
                headers={"User-Agent": "PoeticBytes-Scraper/1.0 (+contact@example.com)"})
            r.raise_for_status()
            return r
        except requests.exceptions.RequestException as e:
            print(f"  attempt {attempt} failed: {e}")
            if attempt < retries:
                time.sleep(2 ** attempt)
    return None

# 3. Scrape with pagination, extraction, and per-item safety
def scrape_all():
    if not can_scrape(BASE):
        print("robots.txt disallows scraping — aborting.")
        return []

    records = []
    url, page = BASE, 1

    while url and page <= 20:
        print(f"[{page}] {url}")
        r = fetch(url)
        if r is None:
            break

        soup = BeautifulSoup(r.text, "html.parser")

        for card in soup.select("div.quote"):
            try:
                text   = card.select_one("span.text").get_text(strip=True)
                author = card.select_one("small.author").get_text(strip=True)
                tags   = [t.get_text(strip=True) for t in card.select("a.tag")]
                records.append({"text": text, "author": author, "tags": tags})
            except AttributeError:
                continue

        next_a = soup.select_one("li.next > a")
        url = urljoin(BASE, next_a["href"]) if next_a else None
        page += 1

        if url:
            delay = random.uniform(1.0, 2.0)
            print(f"  sleeping {delay:.2f}s")
            time.sleep(delay)

    return records

# 4. Save both formats
def save(records):
    if not records:
        print("No records to save.")
        return

    with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
        writer.writeheader()
        for rec in records:
            writer.writerow({**rec, "tags": "|".join(rec["tags"])})

    with open("quotes.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

    print(f"Saved {len(records)} records.")

if __name__ == "__main__":
    data = scrape_all()
    save(data)
    print(f"Done. Sample: {data[0] if data else '(none)'}")

• 1
  import requests, csv, json, time, random — All imports on one line. Perfectly legal Python, though some style guides prefer separate lines.
• 2–4
  BeautifulSoup and URL helpers from earlier lessons. New here: urllib.robotparser — a standard-library module that parses robots.txt.
• 6
  BASE = "https://quotes.toscrape.com" — UPPER_CASE for module-level constants. Signals "don't reassign this."
• 8–15
  can_scrape(url) — Checks robots.txt for permission. rp.set_url(urljoin(url, "/robots.txt")) builds the full robots URL. rp.read() fetches and parses it. rp.can_fetch("*", url) asks "may any user agent fetch this URL?" (our scraper is not a named bot, so "*" applies). If reading fails (no robots.txt — meaning no restrictions), we conservatively return False.
• 18–29
  fetch(url, retries=3) — Same pattern as Lesson 10, but with a specific User-Agent identifying our scraper. This is more polite than faking a browser UA for a one-off scraper — it lets site owners contact us if there's an issue. (For large-scale scraping, some sites explicitly block anything but real browsers, so rotate between "honest ID" and "browser UA" — see the note below.)
• 32–33
  if not can_scrape(BASE): ... return [] — The very first action. If robots.txt disallows, we exit immediately without making any request. This is what being an ethical scraper looks like.
• 35
  records = [] — Accumulator for all quotes across all pages.
• 36
  url, page = BASE, 1 — Tuple unpacking. Two variables set to two values in one line.
• 38
  while url and page <= 20: — The loop termination conditions: no more next-page URL, OR we hit our cap of 20 pages. The cap protects against infinite loops if the next-link is broken.
• 39
  Log which page we're on.
• 40–42
  r = fetch(url) then check for None. If fetch gave up after retries, break out of the whole loop rather than continuing with a broken page.
• 44
  soup = BeautifulSoup(...) — Parse. Then loop over each card.
• 47–54
  The per-item try/except from Lesson 10. Each record is fully built before appending, so partial records never enter our dataset. continue skips a bad card and moves to the next.
• 56–57
  Follow the next-link. if next_a else None — the ternary handles "no next page" by setting url to None, which terminates the while loop on the next iteration.
• 60–62
  Random delay between pages. random.uniform(1.0, 2.0) gives between 1 and 2 seconds, variable — much more human-like than a fixed 1.5 seconds.
• 65
  Return all records.
• 68–83
  save(records) — Writes both CSV and JSON. The CSV writes tags as a |-joined string (because CSV can't natively hold lists). The JSON preserves the list structure directly.
• 85–88
  if __name__ == "__main__": — Only runs the scrape when the file is executed directly — not when it's imported from another module. Standard Python idiom for a script's entry point.

⚖️ The Ethos — What This Project Demonstrates

• Check permission first — robots.txt read before any request.
• Identify yourself honestly — custom User-Agent with contact info.
• Be gentle — random 1–2 second delays between pages.
• Fail gracefully — retries with backoff, per-item try/except, no crash on any single error.
• Save data cleanly — both human-readable (CSV) and machine-readable (JSON).

🎓 What You've Learned

Where to go next:

• Build your own project — scrape something you care about (job listings, price comparisons, weather history) — the fastest way to actually learn.
• Learn async scraping — asyncio + aiohttp can handle 100 concurrent requests while respecting rate limits. Covered in our async lesson.
• Learn about API-first scraping — always check the Network tab first, before assuming HTML scraping is needed.
• Consider data ethics — GDPR, CCPA, and site terms matter. When in doubt, don't scrape.

🌐 Fetching — requests

import requests

r = requests.get(url, timeout=10,
    headers={"User-Agent": "Mozilla/5.0"},
    params={"q": "python", "page": 2})
r.raise_for_status()         # raises on 4xx/5xx
r.status_code                # 200
r.text                       # body as string
r.content                    # body as bytes
r.headers                    # response headers dict
r.json()                     # parsed JSON
r.url                        # final URL after redirects

🍲 Parsing — BeautifulSoup

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, "html.parser")

# Search — find returns one, find_all returns list
soup.find("h1")
soup.find("div", class_="card")
soup.find_all("a", href=True)
soup.find_all("a", href=re.compile("^/product/"))

# CSS selector style
soup.select(".card")
soup.select("div.card > h3")
soup.select_one("#main")

# Extract content
tag.get_text(strip=True)     # text, trimmed
tag["href"]                    # raises if missing
tag.get("href")                # returns None if missing
tag.get("href", "/")           # default value
tag.attrs                    # full attribute dict

# Navigation
tag.parent / tag.parents
tag.children / tag.descendants
tag.next_sibling / tag.previous_sibling
tag.find_next("a")
tag.find_parent("div")

🔗 URL Helpers

from urllib.parse import urljoin

urljoin("https://example.com", "/product/42")
# → https://example.com/product/42

urljoin("https://example.com", "../about")
# → https://example.com/about

urljoin("https://example.com", "https://other.com/x")
# → https://other.com/x   (absolute URLs unchanged)

📁 Saving Data

# CSV (flat tables)
import csv
with open("out.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["a", "b"])
    w.writeheader()
    w.writerows(rows)

# JSON (nested data)
import json
with open("out.json", "w", encoding="utf-8") as f:
    json.dump(rows, f, indent=2, ensure_ascii=False)

🛡️ Error Handling Cheat Sheet

try:
    r = requests.get(url, timeout=10)
    r.raise_for_status()
except requests.exceptions.HTTPError as e:
    # 4xx/5xx — don't retry 4xx
except requests.exceptions.Timeout:
    # retry with backoff
except requests.exceptions.RequestException as e:
    # network/DNS/SSL — retry

🎭 Playwright Cheat Sheet (run locally)

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(url, wait_until="networkidle")
    page.wait_for_selector(".card", timeout=10000)

    cards = page.query_selector_all(".card")
    for c in cards:
        title = c.query_selector(".title")
        print(title.inner_text() if title else "(none)")

    browser.close()

⚖️ Ethics Checklist

• ✅ Check robots.txt — respect Disallow paths
• ✅ Read the site's Terms of Service — don't ignore prohibition
• ✅ Never scrape personal data (GDPR)
• ✅ Never bypass login walls or CAPTCHAs
• ✅ Use a contactable User-Agent (e.g. MyBot/1.0 (+email))
• ✅ Add 1–3 second delays between requests, randomised
• ✅ Stop and reconsider if you're blocked (403/429)
• ❌ Don't impersonate a browser to bypass an anti-bot block
• ❌ Don't sell scraped personal data

🚀 Your Next Steps

• Build something real — scrape a site you care about. Job listings, price comparisons, weather history, product reviews. The only way to actually get good at scraping is to do it.
• Learn async scraping — asyncio + aiohttp lets you handle 100 concurrent requests while respecting rate limits. Essential for larger projects. See our async lesson.
• Explore the API-first approach — always check the Network tab before assuming HTML scraping is needed. Many sites have public APIs that return clean JSON — that's usually 10× easier than scraping the HTML.
• Consider data ethics — GDPR, CCPA, robots.txt, ToS. When in doubt, don't scrape. Ask the site owner if you're unsure.
• Try a larger project — a price tracker, a job board aggregator, a personal dashboard. These push you to learn scheduling, storage, and concurrency — the next tier of skills.

📚 Other Lessons That Build on This