Ad and cookie-banner blocking

Ads, trackers and cookie banners slow pages down and get in the way of your code and your agent. Two session options remove them, and both are free on every plan:

• blockAds refuses requests to ad and tracker sites. It is off by default.
• cookieBanners answers cookie consent banners with “Reject all”. It is "reject" by default for new sessions.

Block ads and trackers

Set blockAds: true when you create a session. Requests to sites on the ad and tracker list are then refused inside the session’s own machine, before they reach a proxy or the internet, so a blocked request also costs no proxy data.

curl -X POST https://api.boxline.dev/v1/sessions \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d '{"blockAds": true}'# Later, while it runs: on or off at once.curl -X PATCH https://api.boxline.dev/v1/sessions/$SESSION_ID \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d '{"blockAds": false}'

import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URLconst s = await bx.sessions.create({ blockAds: true });await s.goto("https://example.com");await s.refresh();console.log(s.data.blockedRequests); // counted about every 30 seconds// Later, while it runs: on or off at once.await s.update({ blockAds: false });

from boxline import Boxlinebx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URLs = bx.sessions.create(block_ads=True)s.goto("https://example.com")s.refresh()print(s.blocked_requests)  # counted about every 30 seconds# Later, while it runs: on or off at once.s.update(block_ads=False)

• Updating the session (PATCH /v1/sessions/:id) turns it on or off while it runs. Turning it on also closes open connections to listed sites.
• The page APIs take it too, for that one request: fetch, screenshot, PDF, extract, and crawl (for every page of the crawl).
• So do agent runs, in their session (the run’s own session). With sessionId, that session’s own setting applies, and session is not allowed next to it.

curl -X POST https://api.boxline.dev/v1/screenshot \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d '{"url": "https://example.com", "fullPage": true, "blockAds": true}' \  -o page.pngcurl -X POST https://api.boxline.dev/v1/agent/runs \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -H "Idempotency-Key: $(uuidgen)" \  -d '{"task": "Open example.com and summarise what the page is for.", "session": {"blockAds": true}}'

import { writeFile } from "node:fs/promises";import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URLconst png = await bx.screenshot("https://example.com", { fullPage: true, blockAds: true });await writeFile("page.png", png);// fetch, pdf, extract and crawl.start take blockAds too.const run = await bx.agent.run({  task: "Open example.com and summarise what the page is for.",  session: { blockAds: true },});

from boxline import Boxlinebx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URLpng = bx.screenshot("https://example.com", full_page=True, block_ads=True)with open("page.png", "wb") as f:    f.write(png)# fetch, pdf, extract and crawl.start take block_ads too.run = bx.agent.run("Open example.com and summarise what the page is for.", session={"blockAds": True})

What was blocked

• The session’s blockedRequests counts the requests it refused. The count is kept across a stop and resume, and a recovery.
• About every 30 seconds, a lifecycle event reports new ones: { text: "ads and trackers blocked", data: { blocked, total, top: [{ domain, count }] } }. blocked is since the last report, total for the session so far, and top the most blocked list entries.
• Turning it on or off with PATCH logs “ad blocking on” or “ad blocking off”. See Events.

In the browser, a blocked HTTPS request fails (ERR_TUNNEL_CONNECTION_FAILED in the network events), and a plain HTTP request is answered 403 with X-Boxline-Proxy-Error: ad_blocked.

How it works

• A site is blocked when its host name, or any domain above it, is on the list: ads.example.com on the list blocks ads.example.com and x.ads.example.com.
• The list is built into the machine image from public lists: EasyList and EasyPrivacy (their rules that name a whole host, minus the hosts they make exceptions for) and AdAway. It holds about 91,800 domains.
• It works per connection, on host names only. Rules that depend on the page or on the URL path are left out, so it blocks whole ad and tracker sites, not single files on other sites.
• Some names are never blocked, with everything below them: Boxline’s own domains, accounts.google.com, fonts.googleapis.com, fonts.gstatic.com, amazonaws.com and cloudfront.net. A few big sites such as google.com and github.com are never listed themselves, so one bad line in a source cannot break most of the web; their ad subdomains still can be.
• With proxy.scope: "all", shell commands that go through the session’s local proxy are filtered the same way. Nothing else in the machine is: ad blocking is a convenience, not a security control.

Credits and licences

The ad and tracker list is made from these public lists. Thank you to their authors.

Because of the EasyList part, the list we derive from them is shared under the same licence, CC BY-SA 3.0.

Reject cookie banners

With cookieBanners: "reject", the default for new sessions, the session’s browser answers the consent banners of the common platforms with “Reject all” or “Necessary only” as soon as they appear, in every tab and frame. When a banner offers neither, it is hidden, and a scroll lock it set on the page is released. It never accepts.

# "reject" is already the default; "off" leaves banners alone.curl -X POST https://api.boxline.dev/v1/sessions \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d '{"cookieBanners": "off"}'# Change it while the session runs.curl -X PATCH https://api.boxline.dev/v1/sessions/$SESSION_ID \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d '{"cookieBanners": "reject"}'

import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URL// "reject" is already the default; "off" leaves banners alone.const s = await bx.sessions.create({ cookieBanners: "off" });// Change it while the session runs.await s.update({ cookieBanners: "reject" });

from boxline import Boxlinebx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URL# "reject" is already the default; "off" leaves banners alone.s = bx.sessions.create(cookie_banners="off")# Change it while the session runs.s.update(cookie_banners="reject")

• PATCH /v1/sessions/:id changes it while the session runs: "reject" also answers banners that are already open, and "off" stops at once.
• Agent runs take it in their session: session: { cookieBanners: "off" }.
• Banners are watched for the first minute of each page, late banners included. A banner that a site shows after that is left alone.
• Sessions without a browser, and sessions created before this option existed, show "off".
• The rules run in an isolated script world: the page cannot see or call them, and they only click and hide.

Each answer is a lifecycle event, { text: "cookie banner rejected" | "cookie banner hidden", data: { platform, host } }, and a change with PATCH logs “cookie banners changed”. A banner from another platform is left as it is: dismiss it with an action or a plain-English step.

{ "blockAds": true, "blockedRequests": 42, "cookieBanners": "reject", "…": "…" }