READING Puppeteer – Web Scraping using Headless Chrome Node API
On This Page

    Puppeteer – Web Scraping using Headless Chrome Node API

    Puppeteer_ Web Scraping using Headless Chrome Node API
    Puppeteer is a Headless Chrome browser developed by the Google Team. A headless browser is a web browser without a graphical user interface(GUI) means that it has no visual components. Headless browsers enable you to control web pages via programming without human intervention. In programmer’s terms, Puppeteer is a Node.js library or API for headless browsing, browser automation, and web scraping, developed by the Google Chrome team.

    Browser Automation Explained

    Browser automation helps you to automate repetitive tasks and web application testing. For example, monitoring product pricing over a period of time, form submission, automatically log in to the web app, perform some task, and log out, etc. There are many libraries for browser automation and web scraping, like PhantomJS, Selenium IDE, etc. However, Puppeteer runs faster and uses less memory. Puppeteer only works with the Google Chrome browser. Puppeteer can be used for

    Use Cases of Puppeteer

    • Headless WebApp/Website Testing: Develop a function test and execute it.
    • Screen Capture: Capture web content screenshots/generate PDFs
    • Web Scraping: Retrieve and manipulate web pages with the DOM API or libraries like jQuery
    • Web Automation: Automate form submission, UI Testing, Keyboard input, etc.
    • Network Monitoring: Monitoring page loading, diagnosing performance issues.

    Puppeteer Features for Web Scraping

    Puppeteer can be used for Puppeteer provides great flexibility and features for Web Scraping. It provides all the features that a professional web scraper desires to have, like

    • Setting up the many browser options
    • Slowing down Puppeteer operations by the specified number of milliseconds.
    • Creating an individual Chromium user profile that cleans up on every run.
    • Timeout: Maximum time in milliseconds to wait for the Chrome instance to start.
    • and many more
    Our main focus here is exploring Puppeteer from a web scraper’s perspective. Let’s take some practical examples and try to learn them.

    Installing Puppeteer

    First of all, we need to install the library:
    yarn add puppeteer
    # or "npm i puppeteer"

    Example #1: Capture a screenshot of a web page

    It can be accomplished by following three simple steps: Create/Open a browser, Browser a web page, and take a screenshot. The code is for capturing a screenshot of a web page and is well-documented and self-explanatory.

    //open puppeteer library
    const puppeteer = require('puppeteer');
    
    (async () => {
    //create browser instance and launch it
      const browser = await puppeteer.launch();
    //create page instance in browser
      const page = await browser.newPage();
    //load URL into the page
      await page.goto('https://en.wikipedia.org/wiki/Main_Page');
    //capture screenshot and save it
      await page.screenshot({path: 'wikipedia.png'});
    //finally close the browser
      await browser.close();
    })();

    Example #2: Create a PDF document from a web page

    //open the puppeteer library
    const puppeteer = require('puppeteer');
    
    (async () => {
    //create browser instance and launch it
      const browser = await puppeteer.launch();
    //create page instance in browser
      const page = await browser.newPage();
    //load URL into the page
      await page.goto('https://news.google.com/', {waitUntil: 'domcontentloaded'});
    //save web page as A4 size PDF document in current directory (path can be specified)
      await page.pdf({path: 'googlenews.pdf', format: 'A4'});
    //finally close the browser
      await browser.close();
    })();

    Example #3: Browser Bing search engine and Search for web scraping

    //open the puppeteer library
    const puppeteer = require('puppeteer');
    
    (async() => {
    
    //create browser instance and launch it
      const browser = await puppeteer.launch();
    //create page instance in browser
      const page = await browser.newPage();
    //Visit the bing search engine
    await page.goto('https://www.bing.com/', {waitUntil: 'networkidle2'});
    //Wait for textbox whose name equals 'q' (textbox to search)
    await page.waitFor('input[name=q]');
    // Type search term into the search bar
    await page.type('input[name=q]', 'web scraping');
    // Click the submit button
    await page.click('input[type="submit"]');
    
    // Wait for the results to show up
    await page.waitForSelector('h2 a');
    
    // Extract the list of links from the result page
    const links = await page.evaluate(() => {
      const anchors = Array.from(document.querySelectorAll('h2 a'));
      return anchors.map(anchor => anchor.textContent);
    });
    //output the list of links to console
    console.log(links.join('\n'));
    //finally close the browser
    await browser.close();
    
    })();

    Some advanced Web Scraping Tips & Tricks:

    #1. Using Proxies with Puppeteer

    const browser = await puppeteer.launch({
      // Launch puppeteer browser using a proxy server on port 3600.
      args: [ '--proxy-server=127.0.0.1:3600' ]
    });

    You can read more about proxy settings here

    #2. Prevent/Block image loading

    const page = await browser.newPage();
    await page.setRequestInterception(true);
    page.on('request', request => {
      if (request.resourceType === 'image')
        request.abort();
      else
        request.continue();
    });
    await page.goto('https://www.yahoo.com/news/');
    Keval Kothari

    Keval Kothari

    Founder and CEO