Intercept and manipulate requests with Puppeteer and Playwright
When we browse the web, a series of HTTP requests and responses are exchanged between our browser and the pages we are visiting. There are scenarios in which it is useful to monitor or manipulate this traffic, instead of letting it happen as-is.
Request interception
Request interception enables us to observe which requests and responses are being exchanged as part of our script’s execution. For example, this is how we could print them out when we load our test website.
With Puppeteer:
With Playwright:
We might want to intervene and filter the outgoing requests. For example, when scraping web pages, we might want to block unnecessary elements from loading in order to speed up the procedure and lower bandwidth usage.
In the following snippet we are going to abort all requests for images on our test website. We will identify them based off of their resourceType, while letting all other requests through without modification.
With Puppeteer:
With Playwright:
As a result, you will see the website logo not being loaded.
Similarly, switching the resourceType to stylesheet would result in the target website loading without any CSS styling.
Response interception
Isolating one or more software components from their dependencies makes them easier to test. We can do so by substituting interactions with such dependencies with simulated, simplified ones. This is also known as stubbing.
Puppeteer makes it easy for us, as for every request we can intercept we also can stub a response. This functionality is not yet available in Playwright.
Every time we load it, our test website is sending a request to its backend to fetch a list of best selling books. For our example, we are going to intercept this response and modify it to return a single book we define on the fly.
With Puppeteer:
Here is what the homepage will look like with our stubbed response:
On macOS/Linux and Windows, you can run the above examples as follows: node request-interception.js
Takeaways
- Puppeteer and Playwright give us control over outgoing HTTP requests.
- With Puppeteer we can easily stub HTTP responses.
Further reading
- Official documentation on this topic from Puppeteer and Playwright.
- Mocks Aren’t Stubs by Martin Fowler.
Originally published at https://theheadless.dev on September 3, 2020.
