Data collection map
What personal data a site's pages ask for, and where it goes: a starting point for a privacy review of your own site.
When to use it: You are starting a privacy review of your own site and want to know what personal data its pages ask for and where it goes.
| Uses | a session (goto page by page, evaluate with forms.js), its network log, exportCookies and files |
|---|---|
| Needs | any plan; no model |
| Site | your site; the default is books.toscrape.com; the runner uses a stand-in site |
| Output | output/map.md, output/result.json |
| Languages | Node and Python |
It visits the start page and the pages it links to on the same site (up to MAX_PAGES), one after the other in the same browser, and for each page lists its forms: the kinds of personal data their fields ask for (from each field's type, autocomplete, name and label: email, phone, name, address, birth date, payment card, password, government id) and where the form sends them, flagging another site. Across the visit it lists the third-party hosts the pages loaded from (a list of well-known trackers is marked; TRACKERS adds yours) and the cookies the site set (from the session's cookie export: HttpOnly or readable by scripts, Secure or not).
A map, not a legal assessment: use it to see what to document and what to ask your vendors.
The code
The example as it is in its folder. Both versions do the same job and write the same output/, so one result check covers both.
/** * Data collection map: what personal data a site's pages ask for and where it goes. For each page: its forms, the * kinds of personal data their fields collect (email, phone, birth date, card, …) and where each form sends it (this * site or another); and across the visit, the third-party hosts the pages load from (known trackers marked) and the * cookies the site sets. A starting point for a privacy review of your own site. * * npx tsx node/index.ts (BOXLINE_API_KEY; SITE_URL, MAX_PAGES, TRACKERS to add host names) * * forms.js runs in each page; the network log and the cookie export come from the session. Writes output/result.json * and output/map.md. */import { mkdirSync, readFileSync, writeFileSync } from "node:fs";import { dirname, join } from "node:path";import { fileURLToPath } from "node:url";import { Boxline } from "@boxline/sdk";const here = dirname(fileURLToPath(import.meta.url));const out = process.env.OUTPUT_DIR ?? "output";const start = process.env.SITE_URL ?? "https://books.toscrape.com/";const maxPages = Number(process.env.MAX_PAGES ?? 5);const forms = readFileSync(join(here, "../forms.js"), "utf8");const TRACKERS = ["google-analytics.com", "googletagmanager.com", "doubleclick.net", "facebook.net", "facebook.com", "connect.facebook.net", "segment.io", "segment.com", "hotjar.com", "mixpanel.com", "amplitude.com", "clarity.ms", "tiktok.com", "linkedin.com", "bing.com", "criteo.com", "taboola.com", "hubspot.com", ...(process.env.TRACKERS ?? "").split(",").map((t) => t.trim()).filter(Boolean)];const bx = new Boxline();type Form = { action: string; method: string; fields: { name: string; type: string; label: string; autocomplete: string | null; kind: string | null }[]; kinds: string[] };const site = (host: string) => host.split(".").slice(-2).join(".");const mine = site(new URL(start).hostname);const session = await bx.sessions.create({ timeout: 300, userMetadata: { example: "data-collection-map" } });console.log(`Session: ${session.id}`);const pages: { url: string; title: string; forms: (Form & { sendsTo: string; thirdParty: boolean })[] }[] = [];let cookies: { name: string; domain: string; httpOnly: boolean; secure: boolean; thirdParty: boolean }[] = [];let requests: { host: string; tracker: boolean; requests: number }[] = [];try { // 1. The start page and the pages it links to on the same site, one after the other in the same browser. const queue = [start]; const seen = new Set<string>(); while (queue.length && pages.length < maxPages) { const url = queue.shift()!; if (seen.has(url)) continue; seen.add(url); const opened = await session.goto(url, { waitUntil: "load" }).catch(() => null); if (!opened || (opened.status ?? 0) >= 400) continue; const found = await session.evaluate<Form[]>(forms); pages.push({ url: opened.url, title: opened.title, forms: found.map((f) => ({ ...f, sendsTo: new URL(f.action).host, thirdParty: site(new URL(f.action).hostname) !== mine })) }); const links = await session.evaluate<string[]>(`[...document.links].map((a) => a.href).filter((h) => /^https?:/.test(h))`); for (const l of links) { const u = new URL(l); u.hash = ""; if (site(u.hostname) === mine && !seen.has(u.href)) queue.push(u.href); } } // 2. Everything the pages loaded (the session's network log) and the cookies they left. await new Promise((r) => setTimeout(r, 1500)); const hosts = new Map<string, number>(); for await (const e of session.events({ types: ["network"], limit: 2000 })) { try { const h = new URL(e.url!).hostname; if (site(h) !== mine) hosts.set(h, (hosts.get(h) ?? 0) + 1); } catch { /* not an address */ } } requests = [...hosts].map(([host, n]) => ({ host, tracker: TRACKERS.some((t) => host === t || host.endsWith(`.${t}`)), requests: n })).sort((a, b) => b.requests - a.requests); // The cookie jar as a Netscape cookie file (the format curl and wget read): one tab-separated line per cookie. const exported = await session.exportCookies(); cookies = (await session.files.readText(exported.path)) .split("\n") .filter((l) => l.trim() && (!l.startsWith("#") || l.startsWith("#HttpOnly_"))) .map((l) => { const [domain, , , secure, , name] = l.split("\t"); const httpOnly = domain!.startsWith("#HttpOnly_"); const host = domain!.replace(/^#HttpOnly_/, "").replace(/^\./, ""); return { name: name!, domain: host, httpOnly, secure: secure === "TRUE", thirdParty: site(host) !== mine }; });} finally { await session.stop();}const collected = [...new Set(pages.flatMap((p) => p.forms.flatMap((f) => f.kinds)))];const offsite = pages.flatMap((p) => p.forms.filter((f) => f.thirdParty && f.kinds.length).map((f) => ({ page: p.url, sendsTo: f.sendsTo, kinds: f.kinds })));console.log(`${pages.length} pages; personal data asked for: ${collected.join(", ") || "none"}`);for (const p of pages) for (const f of p.forms) console.log(` ${p.url}\n form → ${f.method.toUpperCase()} ${f.sendsTo}${f.thirdParty ? " (another site)" : ""}: ${f.kinds.join(", ") || "no personal data"}`);console.log(`Third-party hosts: ${requests.map((r) => `${r.host}${r.tracker ? " (tracker)" : ""}`).join(", ") || "none"}`);console.log(`Cookies: ${cookies.map((c) => `${c.name} (${c.domain}${c.httpOnly ? ", HttpOnly" : ""})`).join(", ") || "none"}`);const md = [ `# Data collection map: ${start}`, "", `Personal data asked for: ${collected.join(", ") || "none"}.`, ...(offsite.length ? ["", `**Sent to another site:** ${offsite.map((o) => `${o.kinds.join(", ")} → ${o.sendsTo} (from ${o.page})`).join("; ")}`] : []), "", "## Pages and forms", "", ...pages.flatMap((p) => [`- ${p.url}`, ...p.forms.map((f) => ` - form → ${f.method.toUpperCase()} ${f.sendsTo}${f.thirdParty ? " **(another site)**" : ""}: ${f.fields.filter((x) => x.type !== "hidden").map((x) => `${x.label || x.name}${x.kind ? ` [${x.kind}]` : ""}`).join(", ")}`)]), "", "## Third-party hosts", "", ...(requests.length ? requests.map((r) => `- ${r.host}: ${r.requests} requests${r.tracker ? " (known tracker)" : ""}`) : ["None."]), "", "## Cookies", "", ...(cookies.length ? cookies.map((c) => `- ${c.name} (${c.domain}): ${c.httpOnly ? "HttpOnly" : "readable by scripts"}, ${c.secure ? "Secure" : "not Secure"}${c.thirdParty ? ", third party" : ""}`) : ["None."]), "",].join("\n");mkdirSync(out, { recursive: true });writeFileSync(join(out, "map.md"), md);writeFileSync(join(out, "result.json"), JSON.stringify({ start, pages, collected, offsite, thirdParties: requests, cookies }, null, 2));"""Data collection map: what personal data a site's pages ask for and where it goes. For each page: its forms, thekinds of personal data their fields collect (email, phone, birth date, card, …) and where each form sends it (thissite or another); and across the visit, the third-party hosts the pages load from (known trackers marked) and thecookies the site sets. A starting point for a privacy review of your own site. python python/main.py (BOXLINE_API_KEY; SITE_URL, MAX_PAGES, TRACKERS to add host names)forms.js runs in each page; the network log and the cookie export come from the session. Writes output/result.jsonand output/map.md."""import jsonimport osimport timefrom collections import Counterfrom pathlib import Pathfrom urllib.parse import urldefrag, urlparsefrom boxline import Boxline, BoxlineErrorhere = Path(__file__).resolve().parentout = Path(os.environ.get("OUTPUT_DIR", "output"))start = os.environ.get("SITE_URL", "https://books.toscrape.com/")max_pages = int(os.environ.get("MAX_PAGES", "5"))forms_js = (here.parent / "forms.js").read_text()TRACKERS = ["google-analytics.com", "googletagmanager.com", "doubleclick.net", "facebook.net", "facebook.com", "connect.facebook.net", "segment.io", "segment.com", "hotjar.com", "mixpanel.com", "amplitude.com", "clarity.ms", "tiktok.com", "linkedin.com", "bing.com", "criteo.com", "taboola.com", "hubspot.com"]TRACKERS += [t.strip() for t in os.environ.get("TRACKERS", "").split(",") if t.strip()]bx = Boxline()def site(host: str) -> str: return ".".join(host.split(".")[-2:])mine = site(urlparse(start).hostname or "")pages, cookies, requests = [], [], []with bx.sessions.create(timeout=300, user_metadata={"example": "data-collection-map"}) as session: print(f"Session: {session.id}", flush=True) # 1. The start page and the pages it links to on the same site, one after the other in the same browser. queue, seen = [start], set() while queue and len(pages) < max_pages: url = queue.pop(0) if url in seen: continue seen.add(url) try: opened = session.goto(url, wait_until="load") except BoxlineError: continue if (opened["status"] or 0) >= 400: continue found = session.evaluate(forms_js) pages.append({"url": opened["url"], "title": opened["title"], "forms": [{**f, "sendsTo": urlparse(f["action"]).netloc, "thirdParty": site(urlparse(f["action"]).hostname or "") != mine} for f in found]}) for link in session.evaluate("[...document.links].map((a) => a.href).filter((h) => /^https?:/.test(h))"): u = urldefrag(link).url if site(urlparse(u).hostname or "") == mine and u not in seen: queue.append(u) # 2. Everything the pages loaded (the session's network log) and the cookies they left. time.sleep(1.5) hosts = Counter() for e in session.events(types=["network"], limit=2000): h = urlparse(e.get("url") or "").hostname if h and site(h) != mine: hosts[h] += 1 requests = sorted(({"host": h, "tracker": any(h == t or h.endswith(f".{t}") for t in TRACKERS), "requests": n} for h, n in hosts.items()), key=lambda r: -r["requests"]) # The cookie jar as a Netscape cookie file (the format curl and wget read): one tab-separated line per cookie. exported = session.export_cookies() for line in session.files.read_text(exported["path"]).split("\n"): if not line.strip() or (line.startswith("#") and not line.startswith("#HttpOnly_")): continue domain, _, _, secure, _, name = line.split("\t")[:6] host = domain.removeprefix("#HttpOnly_").lstrip(".") cookies.append({"name": name, "domain": host, "httpOnly": domain.startswith("#HttpOnly_"), "secure": secure == "TRUE", "thirdParty": site(host) != mine})collected = sorted({k for p in pages for f in p["forms"] for k in f["kinds"]})offsite = [{"page": p["url"], "sendsTo": f["sendsTo"], "kinds": f["kinds"]} for p in pages for f in p["forms"] if f["thirdParty"] and f["kinds"]]print(f"{len(pages)} pages; personal data asked for: {', '.join(collected) or 'none'}")for p in pages: for f in p["forms"]: print(f" {p['url']}\n form → {f['method'].upper()} {f['sendsTo']}{' (another site)' if f['thirdParty'] else ''}: {', '.join(f['kinds']) or 'no personal data'}")print("Third-party hosts: " + (", ".join(f"{r['host']}{' (tracker)' if r['tracker'] else ''}" for r in requests) or "none"))print("Cookies: " + (", ".join(f"{c['name']} ({c['domain']}{', HttpOnly' if c['httpOnly'] else ''})" for c in cookies) or "none"))md = [f"# Data collection map: {start}", "", f"Personal data asked for: {', '.join(collected) or 'none'}."]if offsite: md += ["", "**Sent to another site:** " + "; ".join(f"{', '.join(o['kinds'])} → {o['sendsTo']} (from {o['page']})" for o in offsite)]md += ["", "## Pages and forms", ""]for p in pages: md.append(f"- {p['url']}") for f in p["forms"]: fields = ", ".join(f"{x['label'] or x['name']}" + (f" [{x['kind']}]" if x["kind"] else "") for x in f["fields"] if x["type"] != "hidden") md.append(f" - form → {f['method'].upper()} {f['sendsTo']}{' **(another site)**' if f['thirdParty'] else ''}: {fields}")md += ["", "## Third-party hosts", ""] + ([f"- {r['host']}: {r['requests']} requests" + (" (known tracker)" if r["tracker"] else "") for r in requests] or ["None."])md += ["", "## Cookies", ""] + ([f"- {c['name']} ({c['domain']}): {'HttpOnly' if c['httpOnly'] else 'readable by scripts'}, {'Secure' if c['secure'] else 'not Secure'}" + (", third party" if c["thirdParty"] else "") for c in cookies] or ["None."]) + [""]out.mkdir(parents=True, exist_ok=True)(out / "map.md").write_text("\n".join(md))(out / "result.json").write_text(json.dumps({"start": start, "pages": pages, "collected": collected, "offsite": offsite, "thirdParties": requests, "cookies": cookies}, indent=2, ensure_ascii=False))// Runs in the page (session.evaluate): every form, where it sends its data, and which kinds of personal data its// fields ask for (from their type, autocomplete, name and label). One expression; returns plain JSON.(() => { const KINDS = [ ["payment card", /cc-|card|cvc|cvv|expir/i], ["password", /password|passwd/i], ["government id", /ssn|social.?security|passport|national.?id|tax.?id/i], ["birth date", /bday|birth|dob/i], ["email", /email|e-mail/i], ["phone", /tel|phone|mobile/i], ["address", /street|address|postal|postcode|zip|city|country/i], ["name", /^(cc-)?name$|full.?name|given-name|family-name|first.?name|last.?name|surname/i], ]; const labelOf = (el) => (el.labels && el.labels[0] ? el.labels[0].textContent : el.getAttribute("aria-label") || el.placeholder || "").trim().replace(/\s+/g, " "); return [...document.forms].map((form) => { const fields = [...form.elements] .filter((el) => el.name && !["submit", "button", "reset", "image"].includes(el.type)) .map((el) => { const hints = `${el.type} ${el.getAttribute("autocomplete") || ""} ${el.name} ${labelOf(el)}`; const kind = el.type === "hidden" ? null : (KINDS.find(([, re]) => re.test(el.type === "email" ? "email" : el.type === "tel" ? "tel" : el.type === "password" ? "password" : hints)) || [null])[0]; return { name: el.name, type: el.type, label: labelOf(el), autocomplete: el.getAttribute("autocomplete"), kind }; }); return { action: form.action, method: (form.getAttribute("method") || "get").toLowerCase(), fields, kinds: [...new Set(fields.map((f) => f.kind).filter(Boolean))] }; });})()Inputs
Environment variables the example reads:
SITE_URL,MAX_PAGES(default 5),TRACKERS.
Run it
Get the examples with git clone https://github.com/Boxline-dev/examples.git and open this one: cd examples/data-collection-map. Put your API key in BOXLINE_API_KEY (and BOXLINE_API_URL for another API), then run either version from the example’s folder. Both write to output/ and stop their sessions when they finish.
# Node 18 or newer, in the example's folderexport BOXLINE_API_KEY=… # your API keynpm install @boxline/sdk tsxnpx tsx node/index.ts# Python 3.9 or newer, in the example's folderexport BOXLINE_API_KEY=… # your API keypip install boxline-sdkpython python/main.pyThen check the result, from the same folder: npx tsx check.ts output. It exits with 0 only when the result is right.
What its check proves
check.ts reads what the example wrote and passes only when the result is right, not merely when the example ran.
Every form has its target. On the runner's stand-in site: the sign-up and checkout pages are both read, email, phone, birth date and card data are found, the card details posted to another site are flagged, the third-party script is listed as a tracker, and the server's HttpOnly cookie and the script's cookie are both there.
Source
github.com/Boxline-dev/examples/data-collection-map on GitHub, with its README.md, check.ts and .env.example. The examples are open source. See all examples.
These docs describe a beta API. Endpoints, fields and SDK methods may change before general availability; breaking changes will be listed in the changelog.