Proxies

By default, a session reaches the web from Boxline’s own cloud addresses. Some sites show different content to cloud addresses, or show content that depends on where the visitor is. A proxy changes where your traffic comes from.

You can set a proxy when you create a session or start an agent run, change it while the session runs, and use one on single requests to the fetch, screenshot, PDF, extract and crawl APIs. One proxy can serve every site, or you can pick a different proxy for each site.

Residential, datacenter or your own

Start without a proxy. Add one when a site blocks you or shows the wrong content, and try datacenter before residential.

Add a proxy to a session

Pass proxy when you create the session. The proxy is in place before the browser opens anything, so no request leaves from the platform’s own address first.

import { Boxline } from "@boxline/sdk";
import { chromium } from "playwright";

const client = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URL
const session = await client.sessions.create({
  browser: true,
  proxy: { type: "residential", country: "US", state: "us_california" },
});

try {
  const browser = await chromium.connectOverCDP(session.connectUrl!);
  const page = browser.contexts()[0].pages()[0];
  await page.goto("https://example.com");
  console.log(session.data.proxy); // the settings, never a password
} finally {
  await session.release();
}

from boxline import Boxline

bx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URL
with bx.sessions.create(
    browser=True,
    proxy={"type": "residential", "country": "US", "state": "us_california"},
) as session:
    session.goto("https://example.com")
    print(session.content()["title"])
    print(session.proxy)  # the settings, never a password
# leaving the block releases the session

curl -X POST https://api.staging.boxline.dev/v1/sessions \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "browser": true,
    "proxy": {"type": "residential", "country": "US", "state": "us_california"}
  }'

The session object shows the proxy with its defaults filled in. Without a proxy, proxy is null.

"proxy": {
  "type": "residential",
  "country": "US",
  "state": "us_california",
  "ip": "sticky",
  "scope": "browser"
}

The short form: proxy: true

proxy: true is short for { "type": "residential", "country": "US" }: a residential IP in the US, with the default ip and scope. It works everywhere proxy does.

const session = await client.sessions.create({ browser: true, proxy: true });

session = bx.sessions.create(proxy=True)

curl -X POST https://api.staging.boxline.dev/v1/sessions \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{"browser": true, "proxy": true}'

The session shows the full settings: { "type": "residential", "country": "US", "ip": "sticky", "scope": "browser" }. To change any of them, give the object instead.

Proxy options

Your own proxy

Give the server as http://host:port or https://host:port. Keep the login in environment variables, not in your code:

const session = await client.sessions.create({
  browser: true,
  proxy: {
    type: "custom",
    server: "http://proxy.example.net:8080",
    username: process.env.MY_PROXY_USER,
    password: process.env.MY_PROXY_PASSWORD,
  },
});

import os

session = bx.sessions.create(
    proxy={
        "type": "custom",
        "server": "http://proxy.example.net:8080",
        "username": os.environ["MY_PROXY_USER"],
        "password": os.environ["MY_PROXY_PASSWORD"],
    },
)

curl -X POST https://api.staging.boxline.dev/v1/sessions \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d @- <<EOF
{
  "browser": true,
  "proxy": {
    "type": "custom",
    "server": "http://proxy.example.net:8080",
    "username": "$MY_PROXY_USER",
    "password": "$MY_PROXY_PASSWORD"
  }
}
EOF

Your proxy must be on a public internet address. Location fields (country, state, city) are not accepted: the location is wherever your proxy is.

Traffic to your own proxy still goes through Boxline’s proxy service, which checks the address, blocks mail ports and counts the data. It is not a way around that service: when proxies are unavailable on the platform (503 proxy_unavailable), your own proxy is too.

Choose a location

With the platform’s proxies you choose where your traffic appears to come from. Leave the location out to get an IP from anywhere in the provider’s pool.

• A state sets the country to US for you. A state with another country is refused.
• Datacenter proxies take a country only.
• The API turns "Los Angeles" into "los_angeles". It checks the format of a name, not whether the provider has IPs there.

The smaller the place, the fewer IPs there are to choose from. Ask for a city only when the site really changes by city.

Sticky or rotating IPs

With rotating IPs, the connections of one page can come from different IPs. If a site misbehaves, try sticky.

Residential IPs belong to real home connections. The provider keeps a sticky IP for as long as it can, but one can occasionally change.

Get a new IP

POST/v1/sessions/:id/proxy/rotate

Gives the session’s proxy a new IP with the same settings, and returns the session. With a list of rules, every proxy in the list gets a new IP. New connections use the new IP at once. Connections that are already open are not cut, so a site you already have open may keep seeing the old IP until the browser opens a new connection to it.

await session.rotateProxy();

session.rotate_proxy()

curl -X POST https://api.staging.boxline.dev/v1/sessions/$SESSION_ID/proxy/rotate \
  -H "x-api-key: $BOXLINE_API_KEY"

A session without a proxy gets 400 invalid_request. With your own proxy, which IP is used is up to your proxy.

Change or remove the proxy

PATCH/v1/sessions/:id with proxy sets, changes or (with null) removes the proxy of a running or paused session. Like a new IP, the change applies to new connections at once; open connections finish as they are. A PATCH without proxy leaves it unchanged.

// Another place, or another type
await session.setProxy({ type: "residential", country: "GB", city: "london" });

// No proxy: straight out from the platform's own addresses
await session.setProxy(null);

# Another place, or another type
session.set_proxy({"type": "residential", "country": "GB", "city": "london"})

# No proxy: straight out from the platform's own addresses
session.set_proxy(None)

curl -X PATCH https://api.staging.boxline.dev/v1/sessions/$SESSION_ID \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{"proxy": {"type": "residential", "country": "GB", "city": "london"}}'

curl -X PATCH https://api.staging.boxline.dev/v1/sessions/$SESSION_ID \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{"proxy": null}'

Cookies and logins stay as they are. A site that ties a login to an IP may ask you to sign in again.

In realistic browser mode, the clock and language move with the proxy: open tabs follow within a second or two.

The shell too: scope "all"

By default (scope: "browser") only Chrome uses the proxy, and commands in the shell connect directly. With scope: "all", the shell gets HTTP_PROXY, HTTPS_PROXY and ALL_PROXY, so curl, wget, pip, npm, git and Python’s requests go out through the same proxy and IP as the browser.

const session = await client.sessions.create({
  browser: true,
  shell: true,
  proxy: { type: "datacenter", country: "DE", scope: "all" },
});
const run = await session.exec("curl -sI https://example.com | head -n 1");
console.log(run.stdout);

with bx.sessions.create(
    shell=True,
    proxy={"type": "datacenter", "country": "DE", "scope": "all"},
) as session:
    print(session.exec("curl -sI https://example.com | head -n 1")["stdout"])

curl -X POST https://api.staging.boxline.dev/v1/sessions \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "browser": true,
    "shell": true,
    "proxy": {"type": "datacenter", "country": "DE", "scope": "all"}
  }'

• Programs that ignore these variables connect directly. localhost and 127.0.0.1 are never sent through the proxy.
• The shell’s traffic counts toward your proxy data, like the browser’s.
• Shell commands you run after a change use the new setting, also when you switch from "all" back to "browser". An interactive terminal that is already open keeps its environment until you reopen it.

Different proxies for different sites

Give proxy a list of rules instead of one proxy. Each rule is a proxy, or { "type": "none" } to go straight out, plus an optional domainPattern: the sites the rule is for. For every connection, the first rule that matches the site decides where it goes.

This session sends your company’s intranet through your company’s proxy, Wikipedia straight out, and every other site through a residential IP in the US:

const session = await client.sessions.create({
  browser: true,
  proxy: [
    // The intranet: through your company's proxy
    {
      type: "custom",
      server: "http://proxy.example.com:3128",
      domainPattern: "(^|\\.)intranet\\.example\\.com$",
    },
    // Wikipedia: straight out, no proxy
    { type: "none", domainPattern: "(^|\\.)wikipedia\\.org$" },
    // Every other site: a residential IP in the US
    { type: "residential", country: "US" },
  ],
});

session = bx.sessions.create(
    proxy=[
        # The intranet: through your company's proxy
        {
            "type": "custom",
            "server": "http://proxy.example.com:3128",
            "domainPattern": r"(^|\.)intranet\.example\.com$",
        },
        # Wikipedia: straight out, no proxy
        {"type": "none", "domainPattern": r"(^|\.)wikipedia\.org$"},
        # Every other site: a residential IP in the US
        {"type": "residential", "country": "US"},
    ],
)

curl -X POST https://api.staging.boxline.dev/v1/sessions \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "browser": true,
    "proxy": [
      {
        "type": "custom",
        "server": "http://proxy.example.com:3128",
        "domainPattern": "(^|\\.)intranet\\.example\\.com$"
      },
      {"type": "none", "domainPattern": "(^|\\.)wikipedia\\.org$"},
      {"type": "residential", "country": "US"}
    ]
  }'

Your company’s proxy gets the site’s name and looks it up itself, so names that only exist inside your network work. The proxy must still be on a public address (see Your own proxy). If it needs a login, add username and password to its rule.

How matching works

• The rules are tried in order for every connection. The first match wins; later rules are not checked. Put specific rules first.
• A rule without domainPattern matches every site. Put it last, as the catch-all.
• No match means straight out, from the platform’s own addresses. A list without a catch-all only proxies the sites you name.
• The pattern is a regular expression tested against the host name only (en.wikipedia.org), not the scheme, port or path. Upper and lower case are the same.
• The pattern is not anchored: it can match anywhere in the name. Use ^ (start) and $ (end) for an exact name.
• Escape dots. A plain . matches any character, so write \.. In JSON and JavaScript strings the backslash is written twice ("\\."); in Python, use a raw string (r"\.").

Good to know

• At most 10 rules. A pattern has at most 256 characters and must be a valid regular expression; otherwise you get 400 invalid_request.
• Each rule takes the same options as a single proxy (country, ip and so on). Each proxy in the list keeps its own sticky IP, and getting a new IP gives all of them new ones.
• Rules that set scope must agree. With scope: "all", the shell follows the same rules as the browser.
• The session shows the list as you gave it, with ip filled in, and never a password.
• A single proxy with a domainPattern is a list of one rule: matching sites use the proxy, every other site goes straight out.
• Lists work wherever proxy does: new sessions, PATCH (setProxy, set_proxy), agent runs and the quick APIs.
• Straight-out connections are not proxy data. Each proxied connection counts toward the type of proxy it used. Your plan’s proxy allowance is checked only when the list has a residential or datacenter rule.

Fetch, screenshot, PDF, extract and crawl

These APIs take the same proxy option on each request. The proxy is used for that request only.

const page = await client.fetch("https://example.com", {
  format: "markdown",
  proxy: { type: "residential", country: "FR" },
});

const png = await client.screenshot("https://example.com", {
  proxy: { type: "datacenter", country: "JP" },
});

// A crawl keeps one IP from start to finish
const job = await client.crawl.start({
  url: "https://example.com",
  maxPages: 50,
  proxy: { type: "residential", country: "US" },
});

page = bx.fetch("https://example.com", proxy={"type": "residential", "country": "FR"})

png = bx.screenshot("https://example.com", proxy={"type": "datacenter", "country": "JP"})

# A crawl keeps one IP from start to finish
job = bx.crawl.start("https://example.com", max_pages=50,
                     proxy={"type": "residential", "country": "US"})

curl -X POST https://api.staging.boxline.dev/v1/fetch \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "markdown",
    "proxy": {"type": "residential", "country": "FR"}
  }'

scope has no effect here, since there is no shell. proxy: true and lists of rules work here too; in a list, each rule gets the default ip above.

All of them also take "browser": { "mode": "realistic" }, so pages are rendered with the clock and language of the proxy’s country (a crawl applies it to every page). See Browser mode.

Agent runs

POST/v1/agent/runs takes the same proxy option, for the session the run creates. With sessionId, the run uses that session’s proxy and ignores proxy. See the Agent API.

const run = await client.agent.run({
  task: "Find the price of a large pizza on example.com",
  proxy: { type: "residential", country: "US", state: "us_new_york" },
});

run = bx.agent.run(
    "Find the price of a large pizza on example.com",
    proxy={"type": "residential", "country": "US", "state": "us_new_york"},
)

curl -X POST https://api.staging.boxline.dev/v1/agent/runs \
  -H "x-api-key: $BOXLINE_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "task": "Find the price of a large pizza on example.com",
    "proxy": {"type": "residential", "country": "US", "state": "us_new_york"}
  }'

Pause, resume and move

The proxy belongs to the session, not to the machine. When a session is resumed, moved, or brought back after its machine stopped, the new machine gets the same proxy before it reopens any page. A sticky session keeps its IP as long as the provider still has it; after a long pause it may continue on a new IP from the same location. Resuming checks your plan’s monthly proxy data first, like creating a session does. See Move, pause & resume.

Billing and plan allowances

Data through the platform’s proxies is billed per GB, by type. Every byte counts, in both directions: pages, images, video, and the shell’s traffic with scope: "all". A GB is 1,000,000,000 bytes.

Current prices are on the pricing page and in GET /v1/pricing (proxies).

Your plan sets how much data you can use through the platform’s proxies each calendar month (UTC), residential and datacenter together. It is a ceiling, not free data. Find yours in GET /v1/auth/me as project.limits.proxyGbPerMonth (null means no cap). It is checked when you create a session with a proxy, set one with PATCH, resume a paused session that has one, or send a quick-API request with one.

GET /v1/usage reports proxy data separately, and its costUsd includes it:

"proxy": {
  "residentialGb": 1.284,
  "datacenterGb": 0.2,
  "customGb": 3.5,
  "costUsd": …
}

Use less data

Images and video are most of a page’s weight. When you only need the text, skip them in Playwright:

await page.route("**/*", (route) =>
  ["image", "media", "font"].includes(route.request().resourceType())
    ? route.abort()
    : route.continue(),
);

Errors

When you set a proxy

When a page does not load

When the proxy refuses a connection, the browser shows a network error: the page does not load, and the failed request appears in the session’s network events. The reason is in the X-Boxline-Proxy-Error header of the proxy’s reply. The browser does not show it, but a request from the shell does (with scope: "all"):

curl -sv -o /dev/null https://example.com 2>&1 | grep -i x-boxline-proxy-error

Security

• Your proxy’s password is stored encrypted. Only Boxline’s API and its proxy gateway can read it, never the session machine, so nothing that runs in the session (an agent included) can see it.
• It is never returned by the API. The session object, the SDKs, the event log and crawl jobs show your proxy without it.
• Use a login made for Boxline. Create a separate user on your proxy with only the access it needs, so you can change or revoke it without touching anything else. To switch a running session to a new login, set the proxy again with PATCH.
• Session URLs give access to the proxy. Anyone with a session’s connectUrl can browse through its proxy and use your data. Treat the URLs like passwords, and rotate them if one leaks (POST /v1/sessions/:id/rotate-urls).
• With scope "all", everything in the shell uses the proxy, including commands an agent runs. Use the default scope unless the shell needs it.
• Private addresses and mail ports are refused. Your own proxy must be on a public address, and outgoing mail ports are blocked through every proxy.