Start a crawl
Follows links breadth-first from a start URL in the background, 3 pages at a time, respecting robots.txt (fetched through the sandboxed browser) and crawl-delay; there is no option to ignore them. maxPages may not exceed the plan's maxCrawlPages (Free 200, paid plans 1000; 402 plan_limit). 10 per minute per project.
POST
/v1/crawlsRequest
| Method and path | POST/v1/crawls |
|---|---|
| Base URL | https://api.boxline.dev |
| Authentication | Your API key in the x-api-key header (or Authorization: Bearer). The examples read it from BOXLINE_API_KEY. |
| Safe retries | Send an Idempotency-Key header to retry without creating anything twice. See Idempotency keys. |
| Rate limit | Yes. See Rate limits for the numbers and headers. |
Headers
| Name | Type | Default | Allowed values | Description |
|---|---|---|---|---|
Idempotency-Key | string | – | – | Makes a retry safe: the same key and body within 24 hours returns the first response (with Idempotent-Replayed: true) instead of creating something twice. 1 to 255 printable ASCII characters without spaces, e.g. a UUID. Keys belong to the project (a new API key is replayed only to its caller). At most 300 new keys per minute per project (429). 1 to 255 characters. |
Body
JSON with content-type: application/json. Required.
| Name | Type | Default | Allowed values | Description |
|---|---|---|---|---|
urlrequired | string | – | – | |
proxy | boolean | null | object | object[] | – | – | true = a residential US proxy; an object = one proxy; a list = rules by site (at most 10); null or false = none. |
proxy.typerequired | string | – | residential, datacenter, custom | |
proxy.country | string | – | – | Two-letter country code (residential and datacenter). |
proxy.state | string | – | – | Residential, US states, e.g. us_california (implies US). |
proxy.city | string | – | – | Residential, e.g. los_angeles (needs country). |
proxy.ip | string | – | sticky, rotating | Default sticky for sessions and crawls, rotating for fetch, screenshot, pdf and extract. |
proxy.scope | string | browser | browser, all | all also routes the session's shell. |
proxy.server | string | – | – | custom: your proxy, http://host:port or https://host:port, on a public address. |
proxy.username | string | – | – | |
proxy.password | string | – | – | Stored encrypted; never returned. |
proxy[].domainPattern | string | – | – | Up to 256 characters. |
browser | boolean | object | – | – | true or omitted: a browser with the defaults; false: none; an object: a browser with these settings. |
browser.mode | string | standard | standard, realistic | realistic puts the clock and language on the proxy's country. It never changes what the browser reports about itself. |
browser.locale | string | – | – | BCP 47, e.g. de-DE. Wins over the mode. |
browser.timezone | string | – | – | IANA, e.g. Europe/Berlin. Wins over the mode. |
blockAds | boolean | false | – | Refuse requests to ad and tracker sites inside the machine (its local proxy), with or without a session proxy. Blocked requests are counted in blockedRequests and reported in lifecycle events. |
maxPages | integer | 20 | – | Up to the plan's maxCrawlPages (Free 200, paid 1000); 402 plan_limit beyond. At least 1. |
maxDepth | integer | 3 | – | From 0 to 10. |
sameHost | boolean | true | – | |
include | string[] | – | – | Regular expressions a URL must match. |
exclude | string[] | – | – | Regular expressions that skip a URL. |
format | string | markdown | markdown, text, html | |
waitUntil | string | load | load, domcontentloaded, networkidle, commit | |
delayMs | integer | – | – | |
timeoutMs | integer | – | – |
curl -X POST "https://api.boxline.dev/v1/crawls" \ -H "x-api-key: $BOXLINE_API_KEY" \ -H "content-type: application/json" \ -H "Idempotency-Key: $(uuidgen)" \ -d '{ "url": "https://example.com/docs", "maxPages": 50, "include": ["/docs/"] }'import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URLconst job = await bx.crawl.start({ url: "https://example.com/docs", maxPages: 50, include: ["/docs/"] });const done = await bx.crawl.wait(job.id); // polls, then returns every pageconsole.log(done.status, done.data.length);from boxline import Boxlinebx = Boxline() # reads BOXLINE_API_KEY and BOXLINE_API_URLjob = bx.crawl.start("https://example.com/docs", max_pages=50, include=["/docs/"])done = bx.crawl.wait(job["id"]) # polls, then returns every pageprint(done["status"], len(done["data"]))# Starts the crawl, waits for it, and saves one file per page in docs/.boxline crawl https://example.com/docs --limit 50 -o docsResponses
201 Created
The job (without pages yet).
Headers
RateLimit-Limit: Requests allowed in the current window of this route's limit. (Rate limits)RateLimit-Remaining: Requests left in the current window. (Rate limits)RateLimit-Reset: Seconds until the window starts again. (Rate limits)Idempotent-Replayed:truewhen this is the kept response of an earlier request with the same Idempotency-Key. (Idempotency keys)
Fields
| Name | Type | Default | Allowed values | Description |
|---|---|---|---|---|
id | string | – | – | |
status | string | – | running, completed, failed, canceled | |
url | string | – | – | |
params | object | – | – | The crawl's settings (proxy without passwords). |
pagesDone | integer | – | – | |
pagesFailed | integer | – | – | |
skippedByRobots | integer | – | – | |
error | string | null | – | – | |
createdAt | string | – | – | Format: date-time. |
finishedAt | string | null | – | – | Format: date-time. |
data | object[] | – | – | |
data[].index | integer | – | – | |
data[].url | string | – | – | |
data[].finalUrl | string | null | – | – | |
data[].status | integer | null | – | – | |
data[].title | string | null | – | – | |
data[].depth | integer | – | – | |
data[].content | string | null | – | – | |
data[].captcha | string | null | – | recaptcha, hcaptcha, turnstile, cloudflare, datadome, arkose, human | |
data[].error | string | null | – | – | |
next | string | null | – | – | Pass it back as after for the next page; null on the last page. |
{ "id": "crawl_2Gh7Kx0Lq9Zp4Nm3", "status": "running", "url": "https://example.com/", "params": {}, "pagesDone": 0, "pagesFailed": 0, "skippedByRobots": 0, "error": "…", "createdAt": "2026-09-29T10:00:00.000Z", "finishedAt": "2026-09-29T10:00:00.000Z", "data": [ { "index": 0, "url": "https://example.com/", "finalUrl": "https://example.com/", "status": 0, "title": "Example Domain", "depth": 0, "content": "…", "captcha": "recaptcha", "error": "…" } ], "next": "eyJvIjoiY3JlYXRlZF9kZXNjIiwidCI6IjIwMjYtMDktMjlUMTA6MDA6MDAuMTIzNDU2WiIsImkiOiI2YzFmIn0"}Errors
Errors have a matching HTTP status and the same body. code is stable, so check it in your code; message is for people. See Errors and request ids.
| Status | Codes | When |
|---|---|---|
| 400 Bad Request | invalid_request, invalid_json, invalid_cursor, invalid_idempotency_key, invalid_extension, variables_with_extensions, terms_not_accepted, invalid_reset_token, invalid_confirmation_code, wrong_password, no_password, invalid_next, invalid_oauth_token | The request is not valid. |
| 401 Unauthorized | unauthorized | Missing or invalid API key. |
| 402 Payment Required | feature_not_in_plan, spend_limit, out_of_credit, plan_limit, spend_limit, out_of_credit, organization, project, plan, credit | The plan does not include this, a monthly allowance or spending limit is used up, the organization has no credit left, or the plan allows no more of this. spend_limit and out_of_credit carry error.details.limit: organization, project, plan or credit. |
| 409 Conflict | session_not_running, shell_disabled, browser_disabled, nothing_saved, not_resumable, invalid_state, conflict, email_taken, identity_in_use, provider_already_linked, last_sign_in_method, not_impersonating, machine_too_old, too_many_credential_values, limit_reached, terms_version_mismatch, no_project, last_owner, organization_not_empty, project_has_sessions, already_member, invite_pending, idempotency_in_progress | The current state does not allow it, or a request with this Idempotency-Key is still being answered. |
| 422 Unprocessable Content | idempotency_mismatch | This Idempotency-Key was already used for a different request. |
| 429 Too Many Requests | rate_limited, concurrency_limit | Too many requests, or the organization's concurrent sessions (its plan's) or the project's concurrencyCap are in use. |
{ "error": { "code": "not_found", "message": "session not found", "requestId": "req_4fJx0aQm2Lk9Pz7N" }}Learn more
These docs describe a beta API. Endpoints, fields and SDK methods may change before general availability; breaking changes will be listed in the changelog.