READING How to Scrape Costco Product Details (2026 Tutorial)
On This Page

    How to Scrape Costco Product Details (2026 Tutorial)

    How to Scrape Costco Product Details

    Costco is a major destination for shoppers looking to buy products in bulk, from groceries and appliances to electronics and home essentials. With thousands of items listed online, it has become a valuable resource for anyone interested in analyzing pricing trends, comparing products, or monitoring inventory at scale.

    If you’re trying to extract product information from Costco’s site, you might be wondering where to begin. What tools are best suited for the job? How do you navigate pagination or handle structured data?

    In this tutorial, we will walk you through a clear, repeatable process for scraping Costco product listings.

    What We’ll Scrape from Costco

    In this tutorial, we’ll extract structured product data directly from Costco’s website. Here’s a breakdown of the key fields you can capture for each item:

    Field Description
    item_product_name Full product title as listed on the Costco page
    item_short_description Brief description including key features or model info
    item_product_marketing_features Bullet-point marketing features or highlights
    item_product_marketing_statement Promotional text or savings details (e.g., discount amount, validity)
    item_number Costco item number or product ID
    Model_attr Model numbers or SKUs provided by the manufacturer
    item_review_ratings Average rating score (e.g., 4.58)
    item_product_review_count Number of user reviews for the product
    item_location_pricing_listPrice Original list price before discounts
    item_location_pricing_salePrice Sale price after discounts
    item_product_primary_image Main product image URL
    images Array of swatch or alternate images (includes color variants)
    Brand_attr Product brand (e.g., Samsung)
    categoryPath_ss Category path showing where the product is listed
    item_status_published Whether the product is active and available online
    item_member_only Indicates if the product is limited to Costco members only
    item_buyable Availability status for purchase
    item_location_stockStatus Real-time stock status (e.g., in stock or backordered)

    If you’re exploring similar scraping techniques for other retailers, you might also find our blog Scrape Overstock Products Data Using Python helpful. It walks through a similar hidden API method using Python to extract structured product data.

    Don’t Want to Code It Yourself?

    You can follow the steps and run the script manually, or you can skip the hassle.
    At ScrapeInsight, we extract accurate and ready-to-use product data from Costco at scale.

    If you need bulk product data with pricing, availability, and category mapping, we can deliver it fast. Talk to us today.

    Talk to us today.

    How to Find Costco’s API Endpoint

    Costco’s website works like many modern e-commerce platforms. It loads content dynamically using background requests instead of delivering everything as static HTML.

    When you search for a product, the page pulls data from internal APIs in real time. These requests are not visible in the page source but can be captured through browser tools.

    To collect product data from Costco, the first step is to identify which API is delivering it. You can do this by inspecting how the site fetches search results.

    Step-by-Step Guide to Discover the API

    1. Go to costco.com:
      Start by entering a search term like French Door Refrigerator in the search bar.
    2. Open Browser Developer Tools:
      Right-click anywhere on the page and select Inspect, or press F12 to open the developer tools panel.
    3. Switch to the Network Tab and Filter by XHR:
      Navigate to the Network tab, then click on XHR to view background data requests only.
    4. Reload the Page After Searching:
      Once the search results load, refresh the page using Ctrl+R. This triggers the actual API call that fetches product data and ensures it appears in the XHR list.
    5. Look for a Request Related to Product Search
      The product search request usually contains your search keyword in its URL. Look for an entry with www_costco_com_search?expoption and parameters like whloc=1-wh. This is the API that returns the structured product data.
    6. Inspect and Copy the API Request URL
      Click the request and review the full URL and query parameters. You can replicate this API call in your scraper and customize it with different keywords or pagination values.

    How to Scrape Product Data from Costco Using Python

    Once you have identified the Costco search API endpoint, you can extract structured product data using a simple Python script.

    Below is a step-by-step breakdown of the scraping process using the requests module.

    Step 1: Set Up Your Python Script

    Start by importing the necessary modules and defining basic configurations such as the search query, pagination, and delay between requests.

    import requests
    import time
    import json
    query = "refrigerator"    # Your search keyword
    pages = 2                 # Number of pages to scrape
    items_per_page = 24       # Items shown per page
    delay = 1.5               # Delay to avoid being rate-limited
    output_file = "costco_products.json"

    Step 2: Define the Costco Search API Endpoint

    Use the URL discovered through browser DevTools. This endpoint returns product data in JSON format.

    
    API_URL = s
    "https://search.costco.com/api/apps/www_costco_com/query/www_costco_com_navigation"

    Step 3: Set Request Headers

    Headers simulate a real browser visit to avoid basic bot detection. These are copied from Network Tab from API url.

    
    headers = {
        "Host": "search.costco.com",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0",
        "Accept": "application/json",
        "Accept-Language": "en-US,en;q=0.5",
        "Accept-Encoding": "gzip, deflate, br, zstd",
        "Referer": "https://www.costco.com/",
        "Content-Type": "application/json",
        "Origin": "https://www.costco.com",
        "DNT": "1",
        "Sec-GPC": "1",
        "Connection": "keep-alive",
        "Sec-Fetch-Dest": "empty",
        "Sec-Fetch-Mode": "cors",
        "Sec-Fetch-Site": "same-site",
        "Pragma": "no-cache",
        "Cache-Control": "no-cache"
    }

    Step 4: Loop Through Pages and Scrape Data

    Loop through multiple pages by adjusting the start parameter, send GET requests, and extract the product data from the response.

    
    all_items = []
    
    for page in range(pages):
        start = page * items_per_page
        params = {
            "expoption": "def",
            "q": query,
            "locale": "en-US",
            "start": start,
            "expand": "false",
            "userLocation": "WA",
            "loc": "*",
            "whloc": "1-wh",
            "rows": items_per_page,
            "chdcategory": "true",
            "chdheader": "true"
        }
    
        print(f"[INFO] Scraping page {page + 1} (start={start})...")
    
        try:
            response = requests.get(API_URL, headers=headers, params=params)
            print(f"[DEBUG] Status Code: {response.status_code}")
    
            if response.status_code != 200:
                print(f"[ERROR] Page {page + 1} failed with status {response.status_code}")
                break
    
            data = response.json()
            products = data.get("response", {}).get("docs", [])
    
            if not products:
                print(f"[WARNING] No products found on page {page + 1}")
                break
    
            for product in products:
                all_items.append(product)
    
            time.sleep(delay)
    
        except Exception as e:
            print(f"[ERROR] Exception on page {page + 1}: {e}")
            break

    Step 5: Save the Data to a JSON File

    After scraping, write the complete product list to a local file for further processing or analysis.

    with open(output_file, "w", encoding="utf-8") as f:
        json.dump(all_items, f, indent=2, ensure_ascii=False)
    
    print(f"\nScraping complete. {len(all_items)} items saved to {output_file}")

    This Is How a Costco Product JSON Response Would Look:

    {
      "item_number": "1644523",
      "name": "Samsung 29 cu. ft. Bespoke 4-Door French Door Smart Refrigerator with Beverage Center",
      "item_short_description": "Samsung 29 cu. ft. Bespoke 4-Door French Door Smart Refrigerator with Beverage Center, White Glass",
      "item_location_pricing_listPrice": 2249.99,
      "item_location_pricing_salePrice": 2249.99,
      "item_product_price_in_cart_only": "1",
      "item_review_ratings": "4.59",
      "item_product_review_count": 1849,
      "item_product_primary_image": "https://bfasset.costco-static.com/.../100948386-847_stainlesssteel_1?...",
      "item_product_marketing_features": [
        "Beverage CenterTM - Choose from an internal dispenser or a built-in AutoFill Water Pitcher.",
        "Extra-large Capacity - 29 cu. ft. of flexible storage space.",
        "Dual Ice Maker with Ice BitesTM.",
        "FlexZoneTM Drawer - 5 temperature settings."
      ],
      "Online_Only_attr": ["Online Only"],
      "Brand_attr": ["Samsung"],
      "categoryPath_ss": [
        "/appliances.html",
        "/refrigerators.html",
        "/french-door-refrigerators.html"
      ],
      "item_product_marketing_statement": "Price includes $750 savings. Valid through 8/20/25",
      "deliveryStatus": "in stock",
      "item_location_fulfillment_restrictions_minQty": "1",
      "item_location_fulfillment_restrictions_maxQty": "5"
    }

    Code Limitations

    This script relies on Costco’s internal product search API and the current JSON structure. If the endpoint URL, query parameters, or response schema changes, the script may stop working or return incomplete data.

    It does not include built-in handling for CAPTCHAs, geo-restrictions, or bot detection mechanisms. After multiple requests, the server may return blank responses or HTTP errors.

    The code also does not manage proxies, headers, or session cookies. High scraping frequency without rotation can trigger IP bans or block future requests from the same network.

    ⚠️ Note
    This script accesses data from Costco’s search functionality captured via browser DevTools. Fields shown are based on publicly visible listings.
    Internal or location-sensitive data may not be accessible without membership or authorization.

    Want to expand your scraping skills beyond Costco? Check out our detailed guide: The Complete E-commerce Web Scraping Guide for 2025. It covers techniques, tools, and legal tips for scraping top e-commerce platforms.

    Want Costco Product Data Without the Scraping Headache?

    This Python script is useful for small-scale scraping, but keeping it working long term requires ongoing maintenance. Any change in Costco’s API structure or detection rules can break it. You also risk IP blocks and incomplete data without proper proxy management.

    Costco’s platform can change frequently, and scraping it without proper precautions (like IP rotation and session handling) often leads to blocks or broken scripts.

    If you need consistent access to Costco product data, including prices, inventory, and categories, ScrapeInsight can help.

    Get in touch. We’ll take care of the rest.

    Keval Kothari

    Keval Kothari

    Founder and CEO