Amazon provides one of the largest collections of product information on the internet. Titles, prices, ratings, reviews, specifications, feature bullets, and high-resolution images are all valuable for research, analytics, and catalog enrichment.
This tutorial explains how to build a safe and structured scraper in Python that collects product details and visible reviews from publicly accessible Amazon pages.
The guide covers project setup, library installation, request handling, reliable parsing, and export to CSV and Excel files.
Project Setup for an Amazon Scraper in Python
To begin your Amazon scraping project, create a dedicated working folder. This keeps your scraper structured and avoids file clutter.
Type the following in your terminal:
mkdir amazon_scraper
cd amazon_scraper
- Install all required Python libraries for your Amazon scraping project.
- These libraries support HTTP requests, HTML parsing, data processing, and file export.
- We use requests for fetching pages, BeautifulSoup for parsing content, and pandas for organizing and exporting data.
pip install requests beautifulsoup4 pandas openpyxl
We will rely on the following tools:
- requests for downloading HTML pages from Amazon
- BeautifulSoup for parsing Amazon product pages
- regular expressions for structured text extraction
- pandas for DataFrame-based processing and Excel export
- csv for lightweight file output
- random and time for safe request spacing
Step 1. Importing All Python Libraries for Amazon Scraping
The scraper begins with clean imports. Having them at the top improves readability and helps maintain a single source of truth for dependencies. The same block supports product scraping and review scraping.
import requests
from bs4 import BeautifulSoup
import re
import time
import random
import pandas as pd
These imports support every stage of Amazon HTML extraction and allow Python to handle network requests, parsing workflows, regex capture groups, export logic, and URL handling.
Step 2. Setting Request Headers and Safe Delays for Amazon Scraping
Amazon often serves incomplete pages if a request appears automated. A User Agent string gives your scraper a more realistic footprint. The Accept Language header improves the consistency of returned HTML.
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64 "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/114.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
REQUEST_DELAY_RANGE = (2, 5)
Step 3. Fetching HTML From Public Amazon Pages Safely
Fetching HTML is central to any Amazon scraper. The code below includes retry logic, timeout settings, and checks for CAPTCHA-like patterns. This ensures the scraper does not accidentally parse incomplete or restricted pages.
def fetch_page(url, retries=3, delay_range=REQUEST_DELAY_RANGE):
"""Fetch HTML from Amazon with retries."""
for _ in range(retries):
try:
r = requests.get(url, headers=HEADERS, timeout=10)
if r.status_code == 200 and "captcha" not in r.text.lower():
return r.text
except Exception:
pass
time.sleep(random.uniform(*delay_range))
print(f"[fetch_page] Failed to fetch: {url}")
return None
Step 4. Parsing Amazon Product Data With Python and BeautifulSoup
Amazon product pages contain structured regions that hold title, prices, ratings, bullets, and technical details. These elements use specific HTML tags and attributes, but Amazon changes its layout from time to time. Because of this, it is important to know how to locate the correct HTML elements yourself.
The easiest way is by using the browser’s Inspect Element tools by pressing F12. How to Use Inspect Element to Find the Parsers.
To see exactly where a product detail is located in the HTML:
- Open the Amazon product or review page in your browse
- Press F12 to open Developer Tools
- Click the Inspect icon
- Hover over the element you want to scrape, and click it. The Elements panel will highlight the precise HTML structure.
Identifying HTML Selectors for Your Parser
Typical product elements appear in predictable locations. You can confirm them through Inspect Element:
- Product title
<span id=”productTitle”> - Price
<span class=”a-price-whole”>
<span class=”a-offscreen”> - Feature bullets
located under
<div id=”feature-bullets”> - Review blocks
<li data-hook=”review”> - Review text
<span class=”cr-original-review-content”>
Make sure you include your regional variant Amazon site in the base URL.
def parse_product(html, base_url="https://www.amazon.com"):
soup = BeautifulSoup(html, "html.parser")
result = {}
# Title
title_el = soup.find("span", {"id": "productTitle"})
result["title"] = title_el.get_text(strip=True) if title_el else None
# Price
price_el = soup.find("span", class_="a-price-whole")
if not price_el:
price_el = soup.find("span", class_="a-offscreen")
result["price"] = price_el.get_text(strip=True) if price_el else None
# Rating
rating_el = soup.find("span", {"data-hook": "rating-out-of-text"})
result["rating_summary"] = rating_el.get_text(strip=True) if rating_el else None
# Review count
count_el = soup.find("span", {"id": "acrCustomerReviewText"})
result["rating_count"] = count_el.get_text(strip=True) if count_el else None
# Features
bullets = []
for li in soup.select("#feature-bullets .a-list-item"):
txt = li.get_text(strip=True)
if txt:
bullets.append(txt)
result["feature_bullets"] = bullets
# Description
desc_el = soup.find("div", {"id": "productDescription"})
result["description"] = desc_el.get_text(strip=True) if desc_el else None
# Technical details
result["technical_details"] = {}
table = soup.find("table", {"id": "productDetails_techSpec_section_1"})
if table:
for row in table.find_all("tr"):
key = row.find("th").get_text(strip=True)
val = row.find("td").get_text(strip=True)
result["technical_details"][key] = val
# Product images
result["image_urls"] = re.findall(r'"hiRes":"(https?://[^"]+)"', html)
return result
Step 5. Scraping Amazon Reviews
Amazon updated its review pages so that full review access often requires a login. Review pagination is also limited, and automated traffic is blocked frequently. A Python script can still extract visible reviews, but it must use headers and careful parsing. We can only scrape reviews visible on the product page.
def parse_visible_reviews(html):
soup = BeautifulSoup(html, "html.parser")
reviews = []
container = soup.select_one("#cm-cr-dp-review-list")
if not container:
print("❗ No visible review container found on product page.")
return reviews
review_blocks = container.select("[data-hook='review']")
print(f"Found {len(review_blocks)} visible reviews on product page.")
for li in review_blocks:
review = {}
# Reviewer name
name_el = li.select_one(".a-profile-name")
review["reviewer_name"] = name_el.get_text(strip=True) if name_el else None
# Rating
rating_el = li.select_one("i[data-hook='review-star-rating'] span.a-icon-alt")
review["rating"] = rating_el.get_text(strip=True) if rating_el else None
# Title
title_el = li.select_one("a[data-hook='review-title'] span")
review["review_title"] = title_el.get_text(strip=True) if title_el else None
# Date
date_el = li.select_one("span[data-hook='review-date']")
review["date"] = date_el.get_text(strip=True) if date_el else None
# Body text
body_el = li.select_one("span[data-hook='review-body']")
review["review_text"] = body_el.get_text(strip=True) if body_el else None
# Images
imgs = li.select("img[data-hook='review-image-tile']")
review["image_urls"] = [img.get("src") for img in imgs]
# Review ID
review["review_id"] = li.get("id")
reviews.append(review)
return reviews
Step 6. Exporting Amazon Product Data and Review Data
Scraping often requires exporting results into stable files for indexing, transformation, or machine learning analysis. This code exports the data to both CSV and Excel.
def save_products_to_files(products, csv_name="amazon_products.csv", xlsx_name="amazon_products.xlsx"):
df = pd.DataFrame(products)
df.to_csv(csv_name, index=False)
df.to_excel(xlsx_name, index=False)
def save_reviews_to_csv(reviews, filename="amazon_reviews.csv"):
df = pd.DataFrame(reviews)
df.to_csv(filename, index=False, encoding="utf-8")
def save_reviews_to_excel(reviews, filename="amazon_reviews.xlsx"):
df = pd.DataFrame(reviews)
df.to_excel(filename, index=False)
Step 7. Scrape Multiple Product URLs and Export the Results
The script below scrapes several product URLs, extracts visible reviews, and exports clean datasets for further use.
if __name__ == "__main__":
product_urls = [
"https://www.amazon.com/dp/B0DXMW74JZ",
"https://www.amazon.com/dp/B0DYQF1Q3Z",
]
products = []
all_reviews = []
for url in product_urls:
print("\nScraping product:", url)
html = fetch_page(url)
if not html:
continue
# Product info
product_data = parse_product(html)
products.append(product_data)
visible_reviews = parse_visible_reviews(html)
all_reviews.extend(visible_reviews)
if products:
save_products_to_files(products)
if all_reviews:
save_reviews_to_csv(all_reviews)
save_reviews_to_excel(all_reviews)
print("\nDone!")
This is a complete Amazon scraping workflow built entirely in Python.
The same approach can be extended to other large online retailers as well. For example, our guide on Scraping Walmart Product Data using Python shows how to handle product listings, pricing, and structured details using a similar multi-URL scraping workflow.
Want Amazon Product Data Without the Scraping Headache?
This Python script is great for small-scale Amazon scraping, but keeping it running long term requires continuous maintenance. Amazon updates its HTML structure, review layouts, price blocks, and anti-bot rules frequently. Any change in these areas can break a parser overnight.
Without proper proxy rotation, session management, and request handling, you can also run into IP blocks, throttling, or incomplete data.
Amazon’s platform evolves often, and scraping it at scale without the right infrastructure can lead to blocked requests, missing fields, or unstable results.
If you need reliable access to Amazon product data, including prices, variants, categories, reviews, and availability, ScrapeInsight can provide the solution.
Get in touch and let us handle the hard parts for you.