When to use it: You want every data table of a page as CSV, with the numbers in each column worked out from what the page shows.

Uses	`sessions.create` with `shell`, `session.goto`, `session.evaluate` (`tables.js`), `session.files`, `session.exec` (`to_csv.py`)
Needs	a plan with shell sessions
Site	Wikipedia's [List of largest cities](https://en.wikipedia.org/wiki/List_of_largest_cities) by default (`PAGE_URL` for yours). The runner uses its stand-in: two tables with a two-row header, a merged cell, footnote marks and thousands separators
Output	`output/tables/table-N.csv`, `output/summary.md` (each numeric column's count, sum, min, max, mean) and `output/result.json`
Languages	Node and Python

What the browser step takes care of, which a plain download of the HTML does not:

merged cells: a value that spans rows or columns is repeated in each cell it covers;
headers over two rows are joined per column ("Lighthouse / Name");
footnote marks ([1], [a]) and sort keys are left out;
tables a page builds with JavaScript are there too.

Then to_csv.py writes the CSV files with Python's csv module, and treats a column as numeric when almost every filled cell starts with a number ("1,204,567" → 1204567, "82.5 m" → 82.5).

The code

The example as it is in its folder. Both versions do the same job and write the same output/, so one result check covers both.

/** * Tables to CSV: every data table on a page, read in a real browser (merged cells filled in, footnote marks left * out), written as CSV files, with each numeric column's count, sum, min, max and mean computed in the session's * shell. No model: what is in the CSV is what is on the page. * *   npx tsx node/index.ts            (BOXLINE_API_KEY; PAGE_URL; a plan with shell sessions) * * Writes output/tables/table-N.csv, output/summary.md and output/result.json. */import { mkdirSync, readFileSync, writeFileSync } from "node:fs";import { dirname, join } from "node:path";import { fileURLToPath } from "node:url";import { Boxline } from "@boxline/sdk";interface Table {  index: number;  caption: string;  headers: string[];  rows: string[][];  headerRows: number;  domDataRows: number;}interface Summary {  index: number;  caption: string;  csv: string;  rows: number;  columns: number;  numeric: { column: string; count: number; sum: number; min: number; max: number; mean: number }[];}const here = dirname(fileURLToPath(import.meta.url));const out = process.env.OUTPUT_DIR ?? "output";const url = process.env.PAGE_URL ?? "https://en.wikipedia.org/wiki/List_of_largest_cities";const bx = new Boxline();const session = await bx.sessions.create({ shell: true, timeout: 600, userMetadata: { example: "tables-to-csv" } });console.log(`Session: ${session.id}`);try {  // 1. The tables, as the browser shows them: tables.js expands rowspan and colspan into a full grid.  const page = await session.goto(url);  const tables = await session.evaluate<Table[]>(readFileSync(join(here, "../tables.js"), "utf8"));  console.log(`${page.title}: ${tables.length} tables`);  if (!tables.length) throw new Error("no data table on this page");  // 2. CSV files and the numbers, in the shell (Python's csv module; no model reads the data).  await session.files.write("tables.json", JSON.stringify(tables));  await session.files.write("to_csv.py", readFileSync(join(here, "../to_csv.py")));  const r = await session.exec("python3 to_csv.py");  if (r.exitCode !== 0) throw new Error(`to_csv.py failed: ${r.stderr.trim()}`);  const summary = JSON.parse(r.stdout) as Summary[];  mkdirSync(join(out, "tables"), { recursive: true });  const md = [`# Tables on ${page.title}`, "", url, ""];  for (const t of summary) {    writeFileSync(join(out, t.csv), await session.files.read(t.csv));    console.log(`  ${t.csv}: ${t.rows} rows × ${t.columns} columns${t.caption ? ` (${t.caption})` : ""}`);    md.push(`## Table ${t.index}${t.caption ? `: ${t.caption}` : ""}`, "", `${t.rows} rows, ${t.columns} columns: \`${t.csv}\``, "");    if (t.numeric.length) {      md.push("| Column | Count | Sum | Min | Max | Mean |", "|---|---|---|---|---|---|");      for (const n of t.numeric) md.push(`| ${n.column} | ${n.count} | ${n.sum} | ${n.min} | ${n.max} | ${n.mean} |`);      md.push("");    }  }  writeFileSync(join(out, "summary.md"), `${md.join("\n")}\n`);  writeFileSync(    join(out, "result.json"),    JSON.stringify({ url, title: page.title, tables: summary.map((t) => ({ ...t, headers: tables[t.index - 1]!.headers, domDataRows: tables[t.index - 1]!.domDataRows })) }, null, 2),  );} finally {  await session.stop();}

"""Tables to CSV: every data table on a page, read in a real browser (merged cells filled in, footnote marks leftout), written as CSV files, with each numeric column's count, sum, min, max and mean computed in the session'sshell. No model: what is in the CSV is what is on the page.    python python/main.py            (BOXLINE_API_KEY; PAGE_URL; a plan with shell sessions)Writes output/tables/table-N.csv, output/summary.md and output/result.json."""import jsonimport osfrom pathlib import Pathfrom boxline import Boxlinehere = Path(__file__).resolve().parentout = Path(os.environ.get("OUTPUT_DIR", "output"))url = os.environ.get("PAGE_URL", "https://en.wikipedia.org/wiki/List_of_largest_cities")bx = Boxline()with bx.sessions.create(shell=True, timeout=600, user_metadata={"example": "tables-to-csv"}) as session:    print(f"Session: {session.id}", flush=True)    # 1. The tables, as the browser shows them: tables.js expands rowspan and colspan into a full grid.    page = session.goto(url)    tables = session.evaluate((here.parent / "tables.js").read_text())    print(f"{page['title']}: {len(tables)} tables", flush=True)    if not tables:        raise SystemExit("no data table on this page")    # 2. CSV files and the numbers, in the shell (Python's csv module; no model reads the data).    session.files.write("tables.json", json.dumps(tables))    session.files.write("to_csv.py", (here.parent / "to_csv.py").read_bytes())    r = session.exec("python3 to_csv.py")    if r["exitCode"] != 0:        raise SystemExit(f"to_csv.py failed: {r['stderr'].strip()}")    summary = json.loads(r["stdout"])    (out / "tables").mkdir(parents=True, exist_ok=True)    md = [f"# Tables on {page['title']}", "", url, ""]    for t in summary:        (out / t["csv"]).write_bytes(session.files.read(t["csv"]))        caption = f" ({t['caption']})" if t["caption"] else ""        print(f"  {t['csv']}: {t['rows']} rows × {t['columns']} columns{caption}")        md += [f"## Table {t['index']}" + (f": {t['caption']}" if t["caption"] else ""), "", f"{t['rows']} rows, {t['columns']} columns: `{t['csv']}`", ""]        if t["numeric"]:            md += ["| Column | Count | Sum | Min | Max | Mean |", "|---|---|---|---|---|---|"]            md += [f"| {n['column']} | {n['count']} | {n['sum']} | {n['min']} | {n['max']} | {n['mean']} |" for n in t["numeric"]]            md.append("")(out / "summary.md").write_text("\n".join(md) + "\n")result = {"url": url, "title": page["title"], "tables": [{**t, "headers": tables[t["index"] - 1]["headers"], "domDataRows": tables[t["index"] - 1]["domDataRows"]} for t in summary]}(out / "result.json").write_text(json.dumps(result, indent=2, ensure_ascii=False))

tables.js

// Runs in the page (session.evaluate): every data table as a full grid, merged cells (rowspan, colspan) filled in,// footnote marks left out. One expression; returns plain JSON.(() => {  const text = (cell) => {    const c = cell.cloneNode(true);    c.querySelectorAll("sup.reference, .reference, .sortkey, style, script, .mw-editsection").forEach((n) => n.remove());    return (c.innerText || c.textContent || "").replace(/\[\d+\]|\[[a-z]\]/g, "").replace(/\s+/g, " ").trim();  };  // Encyclopedia pages mark data tables; elsewhere take every table with at least two rows and two columns.  const marked = [...document.querySelectorAll("table.wikitable")];  const tables = (marked.length ? marked : [...document.querySelectorAll("table")]).filter(    (t) => !t.closest("table table") && t.rows.length >= 2 && Math.max(...[...t.rows].map((r) => r.cells.length)) >= 2,  );  return tables.slice(0, 30).map((t, index) => {    const grid = [];    const isHead = [];    [...t.rows].forEach((tr, r) => {      grid[r] ??= [];      let c = 0;      for (const cell of tr.cells) {        while (grid[r][c] !== undefined) c++;        const rs = Math.max(1, cell.rowSpan || 1);        const cs = Math.max(1, cell.colSpan || 1);        const v = text(cell);        for (let i = 0; i < rs; i++) for (let j = 0; j < cs; j++) (grid[r + i] ??= [])[c + j] = v;        c += cs;      }      isHead[r] = [...tr.cells].every((cell) => cell.tagName === "TH");    });    const width = Math.max(...grid.map((row) => row.length));    const rows = grid.map((row) => Array.from({ length: width }, (_, i) => row[i] ?? ""));    // Leading all-header rows form the header; several of them are joined per column ("Lighthouse / Name").    let headerRows = 0;    while (headerRows < rows.length - 1 && isHead[headerRows]) headerRows++;    const headers = Array.from({ length: width }, (_, col) => {      const parts = [];      for (let r = 0; r < headerRows; r++) if (!parts.includes(rows[r][col])) parts.push(rows[r][col]);      return parts.filter(Boolean).join(" / ") || `Column ${col + 1}`;    });    const data = rows.slice(headerRows).filter((row) => row.some((v) => v !== ""));    return { index: index + 1, caption: t.caption ? text(t.caption) : "", headers, rows: data, headerRows, domDataRows: data.length };  });})()

to_csv.py

"""Runs in the session's shell: tables.json (from tables.js) to one CSV per table, and the numbers in each column(count, sum, min, max, mean) computed here, not by a model. Prints a JSON summary."""import csvimport jsonimport refrom pathlib import PathNUMBER = re.compile(r"^[^\d-]*(-?\d[\d,  ]*(?:\.\d+)?)")def number(cell: str):    """The first number in a cell: '1,204,567' → 1204567, '82.5 m' → 82.5; None when it is not a number."""    m = NUMBER.match(cell.strip())    if not m:        return None    try:        return float(re.sub(r"[,  ]", "", m.group(1)))    except ValueError:        return Nonetables = json.loads(Path("tables.json").read_text())Path("tables").mkdir(exist_ok=True)summary = []for t in tables:    name = f"tables/table-{t['index']}.csv"    with open(name, "w", newline="", encoding="utf-8") as f:        w = csv.writer(f)        w.writerow(t["headers"])        w.writerows(t["rows"])    numeric = []    for col, header in enumerate(t["headers"]):        cells = [row[col] for row in t["rows"] if row[col]]        values = [v for v in (number(c) for c in cells) if v is not None]        # A numeric column: almost every filled cell is a number (a stray "n/a" is fine).        if cells and len(values) >= max(2, 0.8 * len(cells)):            numeric.append({"column": header, "count": len(values), "sum": round(sum(values), 6), "min": min(values), "max": max(values), "mean": round(sum(values) / len(values), 6)})    summary.append({"index": t["index"], "caption": t["caption"], "csv": name, "rows": len(t["rows"]), "columns": len(t["headers"]), "numeric": numeric})print(json.dumps(summary))

Inputs

Environment variables the example reads:

PAGE_URL: the page with the tables (default: the list of largest cities).

Run it

Get the examples with git clone https://github.com/Boxline-dev/examples.git and open this one: cd examples/tables-to-csv. Put your API key in BOXLINE_API_KEY (and BOXLINE_API_URL for another API), then run either version from the example’s folder. Both write to output/ and stop their sessions when they finish.

# Node 18 or newer, in the example's folderexport BOXLINE_API_KEY=…   # your API keynpm install @boxline/sdk tsxnpx tsx node/index.ts

# Python 3.9 or newer, in the example's folderexport BOXLINE_API_KEY=…   # your API keypip install boxline-sdkpython python/main.py

Then check the result, from the same folder: npx tsx check.ts output. It exits with 0 only when the result is right.

What its check proves

check.ts reads what the example wrote and passes only when the result is right, not merely when the example ran.

npx tsx check.ts output reads the CSV files back and exits with 0 only when each one has the page's header and exactly the page's number of data rows, all of the same width. On the stand-in it also compares both tables cell for cell and the sums: heights 343.5, population 1,805,880, mean area 180.9.

Source

github.com/Boxline-dev/examples/tables-to-csv on GitHub, with its README.md, check.ts and .env.example. The examples are open source. See all examples.

These docs describe a beta API. Endpoints, fields and SDK methods may change before general availability; breaking changes will be listed in the changelog.