Crawl

POST /v1/crawls starts at one page and follows its links in the background, level by level. Each page is opened in a real browser and kept as Markdown, text or HTML. You poll the crawl for its pages.

Start a crawl

curl -X POST https://api.boxline.dev/v1/crawls \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -H "Idempotency-Key: $(uuidgen)" \  -d '{"url": "https://example.com/docs", "maxPages": 50, "include": ["/docs/"]}'

const job = await bx.crawl.start({ url: "https://example.com/docs", maxPages: 50, include: ["/docs/"] });const done = await bx.crawl.wait(job.id); // polls, then returns every pagefor (const page of done.data) console.log(page.depth, page.url, page.title);

job = bx.crawl.start("https://example.com/docs", max_pages=50, include=["/docs/"])done = bx.crawl.wait(job["id"])  # polls, then returns every pagefor page in done["data"]:    print(page["depth"], page["url"], page["title"])

web_crawl_start{ "url": "https://example.com/docs", "maxPages": 50, "include": ["/docs/"] }

# Starts the crawl, waits for it, and saves one Markdown file per page in docs/.# The CLI has no include filter: it follows every link on the same host.boxline crawl https://example.com/docs --limit 50 -o docs

It also takes the options every web API shares, applied to every page:

The response (201) is the crawl job, still running, without pages yet. Send an Idempotency-Key so a retry never starts a second crawl. See Idempotency keys.

Polite by default

• It reads each site’s robots.txt and skips pages it disallows. skippedByRobots counts them. robots.txt is always obeyed: there is no option to ignore it.
• It keeps to the site’s crawl-delay (up to 5 seconds).
• It loads 3 pages at a time.

Crawl only sites you are allowed to crawl. See the Terms of Service.

Read the results

GET /v1/crawls/:id returns the job and a page of its pages, in the order they were found. Page through with limit (up to 100) and after. Use limit=0 to check progress without pages.

curl "https://api.boxline.dev/v1/crawls/$CRAWL_ID?limit=50" \  -H "x-api-key: $BOXLINE_API_KEY"

const job = await bx.crawl.get(crawlId, { limit: 50 });console.log(job.status, job.pagesDone, job.next);

job = bx.crawl.get(crawl_id, limit=50)print(job["status"], job["pagesDone"], job["next"])

web_crawl_get{ "crawlId": "crawl_2Gh7Kx0Lq9Zp4Nm3" }

{  "id": "crawl_2Gh7Kx0Lq9Zp4Nm3",  "status": "completed",  "url": "https://example.com/docs",  "params": { "maxPages": 50, "maxDepth": 3, "sameHost": true, "include": ["/docs/"] },  "pagesDone": 42,  "pagesFailed": 1,  "skippedByRobots": 3,  "error": null,  "createdAt": "2026-09-29T10:00:00.000Z",  "finishedAt": "2026-09-29T10:01:12.000Z",  "data": [    {      "index": 0,      "url": "https://example.com/docs",      "finalUrl": "https://example.com/docs/",      "status": 200,      "title": "Docs",      "depth": 0,      "content": "# Docs\n…",      "captcha": null,      "error": null    }  ],  "next": "50"}

• status is running, completed, failed or canceled.
• A page that failed has error, a message that says why: the page did not load, or the address is a PDF, which has no page text (not_a_web_page). captcha is set when a CAPTCHA was waiting on it.
• next is the cursor for the next page of pages, or null.

List and cancel

• GET /v1/crawls lists your crawls, newest first, without their pages.
• POST /v1/crawls/:id/cancel stops a crawl. The pages it already has stay.

Limits

maxPages may not be more than the most pages one crawl may ask for on your plan. A bigger maxPages is 402 plan_limit.

• The default is 20 pages. Ask for what you need.
• You can start 10 crawls per minute per project, and your plan must include crawls.
• A crawl loads 3 pages at a time, so a big one takes a while. Poll it, or subscribe to the crawl.finished webhook.

Every field is in the API reference: Start a crawl.