READING How to Scrape Walmart Product Data Using Python
On This Page

    How to Scrape Walmart Product Data Using Python

    How to Scrape Walmart Product 
Data Using Python

    Walmart is one of the world’s largest retailers, offering millions of products across categories like electronics, groceries, clothing, and household essentials. With such a massive catalog, Walmart’s online store has become a valuable source of data for researchers, analysts, and businesses looking to track prices, monitor stock, analyze customer sentiment, and study product trends.

    In this blog, we’ll walk you through how to scrape Walmart product data using Python. You’ll learn how to collect information such as product details, prices, availability, reviews, and images directly from Walmart’s website.

    Follow a step-by-step process from setup to coding and saving data, so you can build your own Walmart scraper for analysis.

    Why Scrape Walmart Data?

    Walmart is one of the largest retailers in the world, with millions of products listed online. Having access to this data can be extremely valuable. For example:

    • Market research & price comparison – See how Walmart sets prices and compare them with other retailers.
    • Tracking product changes – Monitor when products go out of stock or when prices drop.
    • Competitor analysis – Businesses can analyze Walmart’s offerings to adjust their own strategies.
    • Academic or personal projects – Students and researchers can collect datasets for practice or studies.

    Manually checking products takes too much time. A Python scraper automates the process, making it faster and more reliable.

    Why Use Python to Scrape Walmart Data?

    There are many programming languages, but Python is especially popular for web scraping. Here’s why it’s a great choice:

    • Beginner-friendly – Python is simple to learn and read.
    • Powerful libraries– Tools like requests, BeautifulSoup, and pandas make scraping and data handling much easier.
    • Community support – If you face issues, you’ll find tutorials, forums, and ready-made solutions online.
    • Data analysis ready – Python works smoothly with libraries like pandas and NumPy to clean, analyze, and visualize your data.
    • Scalable– You can start small and later add advanced tools like Selenium or Scrapy if you need more power.

    In short, Python gives you everything you need to scrape Walmart effectively and turn raw HTML into useful insights.

    Preparing the Python Environment

    Let’s set up everything you need:

    1. Install Python(if you don’t have it) – Download from python.org.
    2. Install libraries – Run the following in your terminal or command prompt:
      • pip install requests beautifulsoup4
    3. Create a project folder – Keep your scraper code and output files organized. For example:
      • walmart_scraper/
        • scraper.py
        • requirements.txt
        • output.csv

    This will set up your environment and ensure you’re ready to start scraping.

    What Data Will We Extract?

    In this tutorial, our scraper will capture key product information from Walmart’s website, including:

    Data Field Description
    Product ID Unique identifier assigned to each product
    Category / Type Product classification or category
    Brand Manufacturer or brand name
    Average Rating Average customer rating score
    Short Description Brief summary of the product
    Thumbnail Image URL Link to the product’s thumbnail image
    Price Current listed price of the product
    Currency Currency used for the product price

    If you’re exploring similar scraping techniques for other retailers, you might also find our blog How to Scrape Costco Product Details (2025 Tutorial) helpful. It walks through a similar hidden API method using Python to extract structured product data.

    Understanding Walmart’s Website Structure

    Walmart’s website is built using the Next.js framework, which means product information is loaded as structured data in the background rather than being directly available in static HTML. This data is stored inside a special script tag:

    Filter by XHR or Fetch Requests

    This script holds a complete snapshot of the product page, including pricing, availability, images, and seller details. Instead of targeting fragile HTML elements, we extract data directly from this internal source, which gives us a cleaner and more dependable way to access Walmart’s product information.

    In the next section, you’ll use Chrome DevTools to visually locate this script and confirm exactly where the product data is stored before extracting it with Python.

    How to Find Walmart’s Hidden Product Data Using Chrome DevTools

    To scrape Walmart’s site efficiently, you need to locate where the site embeds product and review details on any product page. Chrome DevTools helps uncover this hidden JSON data, which is stored right in the page source.

    1. Open the Walmart Website
      Visit walmart.com and go to any product page, e.g, search for “Smartphone” and click on any listing.
    2. Open Chrome DevTools
      Press F12 (or right-click → Inspect) to launch DevTools. By default, it opens in the Elements tab.
    3. Search for the Embedded Product JSON Script
      Switch to the Elements tab and press Ctrl+F (search). Type “__NEXT_DATA__” in the search box. Chrome will highlight the script tag:
    Filter by XHR or Fetch Requests

    Extracting JSON Data from Walmart’s Website

    Now that you’ve explored Walmart’s website structure and found where product data is hidden inside the __NEXT_DATA__ JSON, let’s extract it using Python.

    Step 1: Import Libraries
    We’ll use Python’s most popular web scraping libraries, requests and BeautifulSoup,  to fetch and parse the webpage, and JSON to decode the embedded data.

    
    import requests
    from bs4 import BeautifulSoup
    import json
    

    Step 2: Send Request and Parse the Page
    Let’s start by loading a Walmart product page (you can replace the URL with any product of your choice):

    
    # URL of the Walmart product page
    url = "https://www.walmart.com/ip/AT-T-Alcatel-Volta-16GB-Suede-Gray-Prepaid-Smartphone/316232047"
    
    # Headers to mimic a browser
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }
    
    # Get the webpage
    response = requests.get(url, headers=headers)
    
    # Parse the page content using BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')
    • requests.get() fetches the HTML from the product page.
    • Headers are used to mimic a real browser and avoid blocking.
    • BeautifulSoup parses the HTML so we can easily search for tags.

    Step 3: Locate and Extract the Hidden JSON Data from the Walmart Page.
    Walmart loads most of its product data dynamically using Next.js. This data is hidden inside a “script” tag with the ID __NEXT_DATA__. In this step, we locate that script and extract the complete JSON response.

    # Locate the <script> tag containing the JSON
    json_tag = soup.find("script", {"id": "__NEXT_DATA__"})
    if json_tag:
        # Parse the JSON data
        full_json_data = json.loads(json_tag.string)
    
        # Print the entire JSON data as formatted output
        print(json.dumps(full_json_data, indent=4))
    
    

    What This Step Does:

    • Finds the hidden __NEXT_DATA__ script tag
    • Converts the raw JSON text into a Python dictionary
    • Prints the entire Walmart page data in a structured format
    • Helps us inspect where the product data is stored internally

    At this stage, we are extracting the full page JSON only for debugging and inspection. In the next step, we will drill down and extract only the required product fields such as name, price, brand, rating, and image URL.

    Step 4: Save the Extracted Walmart JSON Data to a File
    Now that we have successfully extracted the full JSON data from the Walmart product page, the next step is to save this data into a local file.

    
    # Optionally, save the JSON data to a file
        with open("full_data.json", "w", encoding="utf-8") as file:
            json.dump(full_json_data, file, indent=4, ensure_ascii=False)
    
        print("Full JSON data has been saved to 'full_data.json'.")
    else:
        print("JSON data not found on the page.")
    

    What This Step Does:

    • Saves the entire Walmart page JSON into a file named full_data.json
    • Uses UTF-8 encoding to avoid text errors
    • Prints the entire Walmart page data in a structured format
    • Displays a confirmation message after successful saving
    • Handles cases where JSON is not found

    Output Example
    Once you run the above code in Visual Studio Code, your output will look like this:

    Filter by XHR or Fetch Requests

    Note: After running the script, the extracted Walmart JSON data is saved as full_data.json in your project folder. You can open this file using any JSON Viewer such as JSONLint or JSON Formatter & Validator to easily explore the structure. These tools help you expand and navigate the data to quickly locate important product details like name, price, description, and reviews

    Step-by-Step Process for Scraping Walmart Product Data

    Now that you have successfully extracted and explored the full JSON data from Walmart’s website, it’s time to pull out only the meaningful product information we actually need. Instead of working with a massive JSON file, we’ll now focus on extracting clean, structured product data such as name, price, brand, rating, and image URL.

    Let’s break this into simple steps.

    Step 1: Load the Saved JSON File
    Since we already saved the full JSON into a file (full_data.json), we’ll now load it back into Python for structured extraction. 

    
    import json
    
    # Load the saved JSON file
    with open("full_data.json", "r", encoding="utf-8") as file:
        data = json.load(file)
    

    This step allows us to work with the extracted data without making repeated requests to Walmart.

    Step 2: Navigate to the Product Data Section
    Inside the large JSON structure, Walmart stores product details inside several nested keys. We now move through those layers to reach the product data.

    
    product_data = data['props']['pageProps']['initialData']['data']['product']
    
    

    At this point, you now have access to all key product information in a single variable.

    Step 3: Extract Required Product Fields
    Now we extract only the fields we defined earlier in our data list.

    
    product_details = {
        'Product ID': product_data.get('id'),
        'Category/Type': product_data.get('type'),
        'Product Name': product_data.get('name'),
        'Brand': product_data.get('brand'),
        'Average Rating': product_data.get('averageRating'),
        'Short Description': product_data.get('shortDescription'),
        'Thumbnail Image URL': product_data['imageInfo'].get('thumbnailUrl'),
        'Price': product_data['priceInfo']['currentPrice'].get('price'),
        'Currency': product_data['priceInfo']['currentPrice'].get('currencyUnit'),
    }
    

    This step converts messy raw JSON into clean, readable product data.

    Step 4: Display the Extracted Product Data
    Now let’s print the extracted details in a structured format:

    
    print(json.dumps(product_details, indent=4))
    This helps you instantly verify whether the correct fields were extracted.
    

    Step 5: Save Product Data to a JSON File
    After running this process, your output will look like this:

    
    with open("product_details.json", "w", encoding="utf-8") as file:
        json.dump(product_details, file, indent=4, ensure_ascii=False)
    

    You now have a lightweight JSON file containing only the important product details.

    Final Output Example
    Finally, save the clean product data into a separate file:

    Filter by XHR or Fetch Requests

    And the file extract_product_data_smartphone.py will be saved in your project folder.

    Conclusion

    Scraping product data from Walmart using Python and hidden APIs is faster, cleaner, and more reliable than traditional HTML scraping.

    By inspecting the site with Chrome DevTools, you can uncover the hidden JSON endpoints Walmart uses, and then replicate the requests using Python’s requests module.

    Whether you’re tracking prices, monitoring stock, collecting reviews, or building an internal analytics tool, this method saves time and provides structured, scalable results.

    If you want to implement this approach at scale or need a customized scraping solution, contact ScrapeInsight.

    Keval Kothari

    Keval Kothari

    Founder and CEO