Costco is a major destination for shoppers looking to buy products in bulk, from groceries and appliances to electronics and home essentials. With thousands of items listed online, it has become a valuable resource for anyone interested in analyzing pricing trends, comparing products, or monitoring inventory at scale.
If you’re trying to extract product information from Costco’s site, you might be wondering where to begin. What tools are best suited for the job? How do you navigate pagination or handle structured data?
In this tutorial, we will walk you through a clear, repeatable process for scraping Costco product listings.
What We’ll Scrape from Costco
In this tutorial, we’ll extract structured product data directly from Costco’s website. Here’s a breakdown of the key fields you can capture for each item:
| Field | Description |
|---|---|
| item_product_name | Full product title as listed on the Costco page |
| item_short_description | Brief description including key features or model info |
| item_product_marketing_features | Bullet-point marketing features or highlights |
| item_product_marketing_statement | Promotional text or savings details (e.g., discount amount, validity) |
| item_number | Costco item number or product ID |
| Model_attr | Model numbers or SKUs provided by the manufacturer |
| item_review_ratings | Average rating score (e.g., 4.58) |
| item_product_review_count | Number of user reviews for the product |
| item_location_pricing_listPrice | Original list price before discounts |
| item_location_pricing_salePrice | Sale price after discounts |
| item_product_primary_image | Main product image URL |
| images | Array of swatch or alternate images (includes color variants) |
| Brand_attr | Product brand (e.g., Samsung) |
| categoryPath_ss | Category path showing where the product is listed |
| item_status_published | Whether the product is active and available online |
| item_member_only | Indicates if the product is limited to Costco members only |
| item_buyable | Availability status for purchase |
| item_location_stockStatus | Real-time stock status (e.g., in stock or backordered) |
If you’re exploring similar scraping techniques for other retailers, you might also find our blog Scrape Overstock Products Data Using Python helpful. It walks through a similar hidden API method using Python to extract structured product data.
Don’t Want to Code It Yourself?
You can follow the steps and run the script manually, or you can skip the hassle.
At ScrapeInsight, we extract accurate and ready-to-use product data from Costco at scale.
If you need bulk product data with pricing, availability, and category mapping, we can deliver it fast. Talk to us today.
How to Find Costco’s API Endpoint
Costco’s website works like many modern e-commerce platforms. It loads content dynamically using background requests instead of delivering everything as static HTML.
When you search for a product, the page pulls data from internal APIs in real time. These requests are not visible in the page source but can be captured through browser tools.
To collect product data from Costco, the first step is to identify which API is delivering it. You can do this by inspecting how the site fetches search results.
Step-by-Step Guide to Discover the API
- Go to costco.com:
Start by entering a search term like French Door Refrigerator in the search bar. - Open Browser Developer Tools:
Right-click anywhere on the page and select Inspect, or press F12 to open the developer tools panel. - Switch to the Network Tab and Filter by XHR:
Navigate to the Network tab, then click on XHR to view background data requests only. - Reload the Page After Searching:
Once the search results load, refresh the page using Ctrl+R. This triggers the actual API call that fetches product data and ensures it appears in the XHR list. - Look for a Request Related to Product Search
The product search request usually contains your search keyword in its URL. Look for an entry with www_costco_com_search?expoption and parameters like whloc=1-wh. This is the API that returns the structured product data. - Inspect and Copy the API Request URL
Click the request and review the full URL and query parameters. You can replicate this API call in your scraper and customize it with different keywords or pagination values.
How to Scrape Product Data from Costco Using Python
Once you have identified the Costco search API endpoint, you can extract structured product data using a simple Python script.
Below is a step-by-step breakdown of the scraping process using the requests module.
Step 1: Set Up Your Python Script
Start by importing the necessary modules and defining basic configurations such as the search query, pagination, and delay between requests.
import requests
import time
import json
query = "refrigerator" # Your search keyword
pages = 2 # Number of pages to scrape
items_per_page = 24 # Items shown per page
delay = 1.5 # Delay to avoid being rate-limited
output_file = "costco_products.json"
Step 2: Define the Costco Search API Endpoint
Use the URL discovered through browser DevTools. This endpoint returns product data in JSON format.
API_URL = s
"https://search.costco.com/api/apps/www_costco_com/query/www_costco_com_navigation"
Step 3: Set Request Headers
Headers simulate a real browser visit to avoid basic bot detection. These are copied from Network Tab from API url.
headers = {
"Host": "search.costco.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0",
"Accept": "application/json",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate, br, zstd",
"Referer": "https://www.costco.com/",
"Content-Type": "application/json",
"Origin": "https://www.costco.com",
"DNT": "1",
"Sec-GPC": "1",
"Connection": "keep-alive",
"Sec-Fetch-Dest": "empty",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-site",
"Pragma": "no-cache",
"Cache-Control": "no-cache"
}
Step 4: Loop Through Pages and Scrape Data
Loop through multiple pages by adjusting the start parameter, send GET requests, and extract the product data from the response.
all_items = []
for page in range(pages):
start = page * items_per_page
params = {
"expoption": "def",
"q": query,
"locale": "en-US",
"start": start,
"expand": "false",
"userLocation": "WA",
"loc": "*",
"whloc": "1-wh",
"rows": items_per_page,
"chdcategory": "true",
"chdheader": "true"
}
print(f"[INFO] Scraping page {page + 1} (start={start})...")
try:
response = requests.get(API_URL, headers=headers, params=params)
print(f"[DEBUG] Status Code: {response.status_code}")
if response.status_code != 200:
print(f"[ERROR] Page {page + 1} failed with status {response.status_code}")
break
data = response.json()
products = data.get("response", {}).get("docs", [])
if not products:
print(f"[WARNING] No products found on page {page + 1}")
break
for product in products:
all_items.append(product)
time.sleep(delay)
except Exception as e:
print(f"[ERROR] Exception on page {page + 1}: {e}")
break
Step 5: Save the Data to a JSON File
After scraping, write the complete product list to a local file for further processing or analysis.
with open(output_file, "w", encoding="utf-8") as f:
json.dump(all_items, f, indent=2, ensure_ascii=False)
print(f"\nScraping complete. {len(all_items)} items saved to {output_file}")
This Is How a Costco Product JSON Response Would Look:
{
"item_number": "1644523",
"name": "Samsung 29 cu. ft. Bespoke 4-Door French Door Smart Refrigerator with Beverage Center",
"item_short_description": "Samsung 29 cu. ft. Bespoke 4-Door French Door Smart Refrigerator with Beverage Center, White Glass",
"item_location_pricing_listPrice": 2249.99,
"item_location_pricing_salePrice": 2249.99,
"item_product_price_in_cart_only": "1",
"item_review_ratings": "4.59",
"item_product_review_count": 1849,
"item_product_primary_image": "https://bfasset.costco-static.com/.../100948386-847_stainlesssteel_1?...",
"item_product_marketing_features": [
"Beverage CenterTM - Choose from an internal dispenser or a built-in AutoFill Water Pitcher.",
"Extra-large Capacity - 29 cu. ft. of flexible storage space.",
"Dual Ice Maker with Ice BitesTM.",
"FlexZoneTM Drawer - 5 temperature settings."
],
"Online_Only_attr": ["Online Only"],
"Brand_attr": ["Samsung"],
"categoryPath_ss": [
"/appliances.html",
"/refrigerators.html",
"/french-door-refrigerators.html"
],
"item_product_marketing_statement": "Price includes $750 savings. Valid through 8/20/25",
"deliveryStatus": "in stock",
"item_location_fulfillment_restrictions_minQty": "1",
"item_location_fulfillment_restrictions_maxQty": "5"
}
Code Limitations
This script relies on Costco’s internal product search API and the current JSON structure. If the endpoint URL, query parameters, or response schema changes, the script may stop working or return incomplete data.
It does not include built-in handling for CAPTCHAs, geo-restrictions, or bot detection mechanisms. After multiple requests, the server may return blank responses or HTTP errors.
The code also does not manage proxies, headers, or session cookies. High scraping frequency without rotation can trigger IP bans or block future requests from the same network.
Internal or location-sensitive data may not be accessible without membership or authorization.
Want to expand your scraping skills beyond Costco? Check out our detailed guide: The Complete E-commerce Web Scraping Guide for 2025. It covers techniques, tools, and legal tips for scraping top e-commerce platforms.
Want Costco Product Data Without the Scraping Headache?
This Python script is useful for small-scale scraping, but keeping it working long term requires ongoing maintenance. Any change in Costco’s API structure or detection rules can break it. You also risk IP blocks and incomplete data without proper proxy management.
Costco’s platform can change frequently, and scraping it without proper precautions (like IP rotation and session handling) often leads to blocks or broken scripts.
If you need consistent access to Costco product data, including prices, inventory, and categories, ScrapeInsight can help.
Get in touch. We’ll take care of the rest.