Intercept and manipulate requests with Puppeteer and Playwright

When we browse the web, a series of HTTP requests and responses are exchanged between our browser and the pages we are visiting. There are scenarios in which it is useful to monitor or manipulate this traffic, instead of letting it happen as-is.

Request interception

With Puppeteer:

With Playwright:

We might want to intervene and filter the outgoing requests. For example, when scraping web pages, we might want to block unnecessary elements from loading in order to speed up the procedure and lower bandwidth usage.

In the following snippet we are going to abort all requests for images on our test website. We will identify them based off of their resourceType, while letting all other requests through without modification.

With Puppeteer:

With Playwright:

As a result, you will see the website logo not being loaded.

Similarly, switching the resourceType to stylesheet would result in the target website loading without any CSS styling.

Response interception

Puppeteer makes it easy for us, as for every request we can intercept we also can stub a response. This functionality is not yet available in Playwright.

Every time we load it, our test website is sending a request to its backend to fetch a list of best selling books. For our example, we are going to intercept this response and modify it to return a single book we define on the fly.

With Puppeteer:

Here is what the homepage will look like with our stubbed response:

On macOS/Linux and Windows, you can run the above examples as follows: node request-interception.js

Takeaways

  1. With Puppeteer we can easily stub HTTP responses.

Further reading

  1. Mocks Aren’t Stubs by Martin Fowler.

Originally published at https://theheadless.dev on September 3, 2020.

Head of Developer Relations @ Checkly, core theheadless.dev contributor.