Chat with a video
Ask questions about a video and get answers from what is said in it, each with the moments (mm:ss) to jump to.
When to use it: You want answers from what a video says, each with the moments to jump to.
| Uses | a shell session (transcript.py; yt-dlp through setup when given a video address), an agent run in that session |
|---|---|
| Needs | shell sessions and a model on the API (no model key of your own) |
| Site | Sintel (Blender Foundation, CC BY 3.0) and its official subtitles by default; the runner uses a stand-in talk |
| Output | output/answers.md, output/transcript.txt, output/result.json |
| Languages | Node and Python |
transcript.py gets the subtitles into the machine (a SubRip or WebVTT file from SUBTITLES_URL, or, with VIDEO_URL, the video's subtitles through yt-dlp, which downloads nothing else) and writes one [mm:ss] text line per caption. An agent in the same session reads that transcript in the shell and answers; every moment it cites is checked against the captions' times. Use videos you may download (yours, openly licensed ones); some sites' terms forbid downloading even subtitles. For a video without subtitles, transcribe it first (e.g. Whisper in the shell).
The code
The example as it is in its folder. Both versions do the same job and write the same output/, so one result check covers both.
/** * Chat with a video: ask questions about a video and get answers from what is said in it, each with the moments * (mm:ss) to jump to. The subtitles are fetched into a session's shell (a .srt or .vtt file, or with yt-dlp for videos * you may download), and an agent in that session answers from the transcript; every timestamp it cites is checked. * * npx tsx node/index.ts (BOXLINE_API_KEY; SUBTITLES_URL or VIDEO_URL; QUESTIONS separated by |) * * The default is Sintel (Blender Foundation, CC BY 3.0) and its official English subtitles. Writes output/result.json * and output/answers.md. */import { mkdirSync, readFileSync, writeFileSync } from "node:fs";import { dirname, join } from "node:path";import { fileURLToPath } from "node:url";import { Boxline } from "@boxline/sdk";const here = dirname(fileURLToPath(import.meta.url));const out = process.env.OUTPUT_DIR ?? "output";const video = process.env.VIDEO_URL ?? "";const subtitles = process.env.SUBTITLES_URL ?? (video ? "" : "https://durian.blender.org/wp-content/content/subtitles/sintel_en.srt");const questions = (process.env.QUESTIONS ?? "What is the girl looking for?|What does the old man warn her about?|How does it end for the dragon?").split("|").map((q) => q.trim()).filter(Boolean);const bx = new Boxline();type Answers = { answers: { question: string; answer: string; moments: string[] }[] };// yt-dlp only when it is needed (a video address rather than a subtitles file).const session = await bx.sessions.create({ browser: false, shell: true, setup: video ? ["pip install -q yt-dlp"] : [], timeout: 900, idleTimeout: 300, userMetadata: { example: "chat-with-a-video" } });console.log(`Session: ${session.id}`);let transcript = "";let result!: Answers;let runId = "";try { await session.files.write("transcript.py", readFileSync(join(here, "../transcript.py"))); const t = await session.exec("python transcript.py", { env: { SUBTITLES_URL: subtitles, VIDEO_URL: video }, timeoutMs: 300_000 }); if (t.exitCode !== 0) throw new Error(`transcript.py failed: ${(t.stderr || t.stdout).trim()}`); console.log(t.stdout.trim()); transcript = await session.files.readText("transcript.txt"); // The agent reads the transcript in the shell (grep, cat): answers come from what is said, with their moments. const started = await bx.agent.run({ sessionId: session.id, maxSteps: 15, task: `/workspace/transcript.txt is a video's transcript: each line starts with its time [mm:ss]. Answer these questions from what is said in it ` + `only; when it does not say, answer so. For each answer give the moments (mm:ss, as written in the transcript) it rests on. Do not write files.\n\n` + questions.map((q, i) => `${i + 1}. ${q}`).join("\n"), output: { type: "object", properties: { answers: { type: "array", items: { type: "object", properties: { question: { type: "string" }, answer: { type: "string" }, moments: { type: "array", items: { type: "string", description: "mm:ss" } } }, required: ["question", "answer", "moments"] } } }, required: ["answers"], }, }); console.log(`Agent run ${started.id} (${started.model}) is reading the transcript…`); runId = started.id; const run = await bx.agent.wait<Answers>(started.id); if (run.status !== "completed" || !run.result) throw new Error(`the run ${run.status}: ${run.error ?? "no answer"}`); result = run.result;} finally { await session.stop();}// Every moment must be a caption's time in the transcript.const times = new Set([...transcript.matchAll(/^\[(\d{2}:\d{2})\]/gm)].map((m) => m[1]));const unknown = result.answers.flatMap((a) => a.moments.map((m) => m.replace(/^\[|\]$/g, "")).filter((m) => !times.has(m)));for (const a of result.answers) console.log(`\nQ: ${a.question}\nA: ${a.answer}\n at ${a.moments.join(", ") || "-"}`);console.log(unknown.length ? `\nMoments not in the transcript: ${unknown.join(", ")}` : "\nEvery moment cited is a caption's time.");const md = [`# ${video || subtitles}`, "", ...result.answers.flatMap((a) => [`## ${a.question}`, "", a.answer, "", `At ${a.moments.join(", ") || "no moment"}.`, ""])].join("\n");mkdirSync(out, { recursive: true });writeFileSync(join(out, "answers.md"), md);writeFileSync(join(out, "transcript.txt"), transcript);writeFileSync(join(out, "result.json"), JSON.stringify({ video: video || null, subtitles: subtitles || null, captions: times.size, answers: result.answers, unknownMoments: unknown, agentRuns: [{ id: runId }] }, null, 2));"""Chat with a video: ask questions about a video and get answers from what is said in it, each with the moments(mm:ss) to jump to. The subtitles are fetched into a session's shell (a .srt or .vtt file, or with yt-dlp for videosyou may download), and an agent in that session answers from the transcript; every timestamp it cites is checked. python python/main.py (BOXLINE_API_KEY; SUBTITLES_URL or VIDEO_URL; QUESTIONS separated by |)The default is Sintel (Blender Foundation, CC BY 3.0) and its official English subtitles. Writes output/result.jsonand output/answers.md."""import jsonimport osimport refrom pathlib import Pathfrom boxline import Boxlinehere = Path(__file__).resolve().parentout = Path(os.environ.get("OUTPUT_DIR", "output"))video = os.environ.get("VIDEO_URL", "")subtitles = os.environ.get("SUBTITLES_URL") or ("" if video else "https://durian.blender.org/wp-content/content/subtitles/sintel_en.srt")questions = [q.strip() for q in os.environ.get("QUESTIONS", "What is the girl looking for?|What does the old man warn her about?|How does it end for the dragon?").split("|") if q.strip()]bx = Boxline()OUTPUT = { "type": "object", "properties": {"answers": {"type": "array", "items": {"type": "object", "properties": {"question": {"type": "string"}, "answer": {"type": "string"}, "moments": {"type": "array", "items": {"type": "string", "description": "mm:ss"}}}, "required": ["question", "answer", "moments"]}}}, "required": ["answers"],}# yt-dlp only when it is needed (a video address rather than a subtitles file).with bx.sessions.create(browser=False, shell=True, setup=["pip install -q yt-dlp"] if video else [], timeout=900, idle_timeout=300, user_metadata={"example": "chat-with-a-video"}) as session: print(f"Session: {session.id}", flush=True) session.files.write("transcript.py", (here.parent / "transcript.py").read_bytes()) t = session.exec("python transcript.py", env={"SUBTITLES_URL": subtitles, "VIDEO_URL": video}, timeout_ms=300_000) if t["exitCode"] != 0: raise SystemExit(f"transcript.py failed: {(t['stderr'] or t['stdout']).strip()}") print(t["stdout"].strip()) transcript = session.files.read_text("transcript.txt") # The agent reads the transcript in the shell (grep, cat): answers come from what is said, with their moments. numbered = "\n".join(f"{i}. {q}" for i, q in enumerate(questions, 1)) started = bx.agent.run( "/workspace/transcript.txt is a video's transcript: each line starts with its time [mm:ss]. Answer these questions from what is said in it " "only; when it does not say, answer so. For each answer give the moments (mm:ss, as written in the transcript) it rests on. Do not write files.\n\n" + numbered, session_id=session.id, max_steps=15, output=OUTPUT, ) print(f"Agent run {started['id']} ({started['model']}) is reading the transcript…", flush=True) run = bx.agent.wait(started["id"]) if run["status"] != "completed" or not run["result"]: raise SystemExit(f"the run {run['status']}: {run.get('error') or 'no answer'}") answers = run["result"]["answers"]# Every moment must be a caption's time in the transcript.times = set(re.findall(r"^\[(\d{2}:\d{2})\]", transcript, re.M))unknown = [m.strip("[]") for a in answers for m in a["moments"] if m.strip("[]") not in times]for a in answers: print(f"\nQ: {a['question']}\nA: {a['answer']}\n at {', '.join(a['moments']) or '-'}")print(f"\nMoments not in the transcript: {', '.join(unknown)}" if unknown else "\nEvery moment cited is a caption's time.")md = [f"# {video or subtitles}", ""]for a in answers: md += [f"## {a['question']}", "", a["answer"], "", f"At {', '.join(a['moments']) or 'no moment'}.", ""]out.mkdir(parents=True, exist_ok=True)(out / "answers.md").write_text("\n".join(md))(out / "transcript.txt").write_text(transcript)(out / "result.json").write_text(json.dumps({"video": video or None, "subtitles": subtitles or None, "captions": len(times), "answers": answers, "unknownMoments": unknown, "agentRuns": [{"id": run["id"]}]}, indent=2, ensure_ascii=False))"""Runs in the session's shell: gets a video's subtitles and writes /workspace/transcript.txt, one line per caption:"[mm:ss] text". From SUBTITLES_URL (a SubRip .srt or WebVTT .vtt file), or from VIDEO_URL with yt-dlp (subtitles only,nothing else is downloaded; for videos you may download)."""import globimport osimport reimport subprocessimport urllib.requestsubs_url = os.environ.get("SUBTITLES_URL")video_url = os.environ.get("VIDEO_URL")if subs_url: with urllib.request.urlopen(urllib.request.Request(subs_url, headers={"user-agent": "chat-with-a-video (a Boxline example)"}), timeout=60) as r: raw = r.read().decode("utf-8-sig", "replace")elif video_url: subprocess.run(["yt-dlp", "--quiet", "--skip-download", "--write-subs", "--write-auto-subs", "--sub-langs", "en.*,en", "--sub-format", "vtt/srt/best", "-o", "/workspace/video.%(ext)s", video_url], check=True) files = sorted(glob.glob("/workspace/video*.vtt") + glob.glob("/workspace/video*.srt")) if not files: raise SystemExit("this video has no English subtitles") raw = open(files[0], encoding="utf-8-sig").read()else: raise SystemExit("set SUBTITLES_URL or VIDEO_URL")lines, last = [], Nonefor block in re.split(r"\n\s*\n", raw.replace("\r", "")): m = re.search(r"(\d{1,2}):(\d{2}):(\d{2})[.,]\d{3}\s*-->", block) if not m: continue rows = block.split("\n") first = next(k for k, row in enumerate(rows) if "-->" in row) + 1 # the caption's text follows its time line text = re.sub(r"<[^>]+>", "", " ".join(row.strip() for row in rows[first:] if row.strip())).strip() # WebVTT styling tags if not text or text == last: # automatic captions repeat lines as they roll continue last = text minutes = int(m.group(1)) * 60 + int(m.group(2)) lines.append(f"[{minutes:02d}:{m.group(3)}] {text}")open("/workspace/transcript.txt", "w").write("\n".join(lines) + "\n")print(f"{len(lines)} captions, {len(' '.join(lines).split())} words")Inputs
Environment variables the example reads:
SUBTITLES_URLorVIDEO_URL,QUESTIONS(separated by|).
Run it
Get the examples with git clone https://github.com/Boxline-dev/examples.git and open this one: cd examples/chat-with-a-video. Put your API key in BOXLINE_API_KEY (and BOXLINE_API_URL for another API), then run either version from the example’s folder. Both write to output/ and stop their sessions when they finish.
# Node 18 or newer, in the example's folderexport BOXLINE_API_KEY=… # your API keynpm install @boxline/sdk tsxnpx tsx node/index.ts# Python 3.9 or newer, in the example's folderexport BOXLINE_API_KEY=… # your API keypip install boxline-sdkpython python/main.pyThen check the result, from the same folder: npx tsx check.ts output. It exits with 0 only when the result is right.
What its check proves
check.ts reads what the example wrote and passes only when the result is right, not merely when the example ran.
There is a transcript and an answer to each question, and every moment cited is a caption's time. On the runner's stand-in talk: about 50,000 bees at 00:12, the harvest in late August at 00:31, and a wind break at 00:20.
Source
github.com/Boxline-dev/examples/chat-with-a-video on GitHub, with its README.md, check.ts and .env.example. The examples are open source. See all examples.
These docs describe a beta API. Endpoints, fields and SDK methods may change before general availability; breaking changes will be listed in the changelog.