When to use it: You want an agent to solve a programming exercise, and proof that the tests pass from a fresh copy.

Uses	`sessions.create` with `browser: false, shell: true`, `session.files`, `session.exec`, `agent.run` in that session (its shell tool), `agent.wait` with an output schema
Needs	a plan with shell sessions and agent runs
Site	the exercises and test cases of [Exercism's problem-specifications](https://github.com/exercism/problem-specifications) (MIT), downloaded in the machine
Output	`output/solution.py` and `output/result.json` (the agent's report, the example's own test run, the agent's shell commands)
Languages	Node and Python

run_tests.py runs the exercise's canonical test cases without a test framework: each case calls solution.<property>(**input) and compares the return value with the expected one; an expected error means the call must raise. python3 run_tests.py --signatures prints the functions the solution needs, which go into the agent's task. It suits the exercises whose cases call one function and compare its value (most of them); exercises built around classes or several calls in a row are beyond it.

Why the second run: the agent's report is a claim. The example downloads the cases again into another folder, copies only solution.py over, runs them, and compares the agent's copies of the tests with the originals, so a solution that "passes" by changing the tests is caught.

The code

The example as it is in its folder. Both versions do the same job and write the same output/, so one result check covers both.

/** * Solve and test: an agent solves a programming exercise in a session's shell and runs the exercise's tests until they * pass; then the example runs the tests again itself, from a fresh copy, so the result does not rest on the agent's word. * The exercises and their test cases are Exercism's problem-specifications (MIT). * *   npx tsx node/index.ts            (BOXLINE_API_KEY; EXERCISE, e.g. roman-numerals; a plan with shell sessions) * * The agent works in a session without a browser: only the shell, in the machine. Writes output/solution.py and * output/result.json. */import { mkdirSync, readFileSync, writeFileSync } from "node:fs";import { dirname, join } from "node:path";import { fileURLToPath } from "node:url";import { Boxline } from "@boxline/sdk";const here = dirname(fileURLToPath(import.meta.url));const out = process.env.OUTPUT_DIR ?? "output";const exercise = process.env.EXERCISE ?? "roman-numerals";if (!/^[a-z0-9]+(-[a-z0-9]+)*$/.test(exercise)) throw new Error(`EXERCISE is a slug like roman-numerals, not "${exercise}"`);const specs = `https://raw.githubusercontent.com/exercism/problem-specifications/main/exercises/${exercise}`;const runner = readFileSync(join(here, "../run_tests.py"));const bx = new Boxline();// Downloads the exercise into a folder of the workspace (the address as a variable, not inside the command).const fetchExercise = (dir: string) =>  `mkdir -p ${dir} && cd ${dir} && curl -sSfL --max-time 60 -o instructions.md "$SPECS/instructions.md" && ` +  `curl -sSfL --max-time 60 -o canonical-data.json "$SPECS/canonical-data.json" && sha256sum canonical-data.json run_tests.py`;const session = await bx.sessions.create({ browser: false, shell: true, timeout: 900, idleTimeout: 300, userMetadata: { example: "solve-and-test" } });console.log(`Session: ${session.id}`);try {  // 1. The exercise, its test cases and the test runner go into the machine.  await session.files.write("exercise/run_tests.py", runner);  const got = await session.exec(`${fetchExercise("exercise")} && python3 run_tests.py --signatures`, { env: { SPECS: specs }, cwd: "/workspace", timeoutMs: 120_000 });  if (got.exitCode !== 0) throw new Error(`no exercise "${exercise}": ${got.stderr.trim()}`);  const lines = got.stdout.trim().split("\n");  const hashes = lines.slice(0, 2).join("\n");  const signatures = lines.slice(2);  console.log(`Exercise ${exercise}: ${signatures.join("; ")}`);  // 2. An agent solves it in the same session: it reads, writes solution.py, runs the tests, fixes, until all pass.  const started = await bx.agent.run({    sessionId: session.id,    task:      `In /workspace/exercise, instructions.md describes a programming exercise. Write the solution in Python in ` +      `/workspace/exercise/solution.py with these functions:\n${signatures.join("\n")}\n` +      `Run the tests with \`cd /workspace/exercise && python3 run_tests.py\` (it prints JSON with the failed cases) and ` +      `fix the solution until every test passes. Do not change run_tests.py or canonical-data.json. Use only Python's ` +      `standard library. When done, report the last test run's numbers and your approach in two sentences.`,    maxSteps: 25,    output: {      type: "object",      properties: { passed: { type: "integer" }, total: { type: "integer" }, approach: { type: "string" } },      required: ["passed", "total", "approach"],    },  });  console.log(`Agent run ${started.id} (${started.model}) is solving it…`);  const run = await bx.agent.wait<{ passed: number; total: number; approach: string }>(started.id);  if (run.status !== "completed" || !run.result) throw new Error(`the run ${run.status}: ${run.error ?? "no result"}`);  const toolSteps = run.steps.filter((s) => s.type === "tool");  console.log(`The agent says ${run.result.passed}/${run.result.total} pass after ${toolSteps.length} tool steps: ${run.result.approach}`);  // 3. The tests again, run by the example from a fresh download in another folder, with only solution.py copied over.  //    The agent's copies of the tests are compared with the originals too. (The agent shares the session's shell, so  //    its last `cd` would carry over: every command here names its folder with `cwd`.)  await session.files.write("check/run_tests.py", runner);  const verify = await session.exec(    `${fetchExercise("check")} >/dev/null && cp ../exercise/solution.py . && ` +      `(cd ../exercise && sha256sum canonical-data.json run_tests.py) > ../agent-hashes.txt && python3 run_tests.py`,    { env: { SPECS: specs }, cwd: "/workspace", timeoutMs: 120_000 },  );  const last = verify.stdout.trim().split("\n").at(-1) ?? "";  if (!last.startsWith("{")) throw new Error(`the tests did not run: ${verify.stderr.trim() || "no solution.py"}`);  const tests = JSON.parse(last);  const testsChanged = (await session.files.readText("agent-hashes.txt")).trim() !== hashes.trim();  console.log(`Run again by the example: ${tests.passed}/${tests.total} pass${testsChanged ? " (the agent changed the tests!)" : "; the agent left the tests as they were"}`);  mkdirSync(out, { recursive: true });  const solution = await session.files.readText("exercise/solution.py");  writeFileSync(join(out, "solution.py"), solution);  writeFileSync(    join(out, "result.json"),    JSON.stringify(      {        exercise,        source: specs,        signatures,        agent: run.result,        tests,        testsChanged,        solutionLines: solution.trim().split("\n").length,        toolSteps: toolSteps.map((s) => ({ name: s.name, input: s.input })),        agentRuns: [{ id: run.id }],      },      null,      2,    ),  );} finally {  await session.stop();}

"""Solve and test: an agent solves a programming exercise in a session's shell and runs the exercise's tests until theypass; then the example runs the tests again itself, from a fresh copy, so the result does not rest on the agent's word.The exercises and their test cases are Exercism's problem-specifications (MIT).    python python/main.py            (BOXLINE_API_KEY; EXERCISE, e.g. roman-numerals; a plan with shell sessions)The agent works in a session without a browser: only the shell, in the machine. Writes output/solution.py andoutput/result.json."""import jsonimport osimport refrom pathlib import Pathfrom boxline import Boxlinehere = Path(__file__).resolve().parentout = Path(os.environ.get("OUTPUT_DIR", "output"))exercise = os.environ.get("EXERCISE", "roman-numerals")if not re.fullmatch(r"[a-z0-9]+(-[a-z0-9]+)*", exercise):    raise SystemExit(f'EXERCISE is a slug like roman-numerals, not "{exercise}"')specs = f"https://raw.githubusercontent.com/exercism/problem-specifications/main/exercises/{exercise}"runner = (here.parent / "run_tests.py").read_bytes()bx = Boxline()def fetch_exercise(folder: str) -> str:    """Downloads the exercise into a folder of the workspace (the address as a variable, not inside the command)."""    return (        f'mkdir -p {folder} && cd {folder} && curl -sSfL --max-time 60 -o instructions.md "$SPECS/instructions.md" && '        'curl -sSfL --max-time 60 -o canonical-data.json "$SPECS/canonical-data.json" && sha256sum canonical-data.json run_tests.py'    )OUTPUT = {    "type": "object",    "properties": {"passed": {"type": "integer"}, "total": {"type": "integer"}, "approach": {"type": "string"}},    "required": ["passed", "total", "approach"],}with bx.sessions.create(browser=False, shell=True, timeout=900, idle_timeout=300, user_metadata={"example": "solve-and-test"}) as session:    print(f"Session: {session.id}", flush=True)    # 1. The exercise, its test cases and the test runner go into the machine.    session.files.write("exercise/run_tests.py", runner)    got = session.exec(f"{fetch_exercise('exercise')} && python3 run_tests.py --signatures", env={"SPECS": specs}, cwd="/workspace", timeout_ms=120_000)    if got["exitCode"] != 0:        raise SystemExit(f'no exercise "{exercise}": {got["stderr"].strip()}')    lines = got["stdout"].strip().split("\n")    hashes, signatures = "\n".join(lines[:2]), lines[2:]    print(f"Exercise {exercise}: {'; '.join(signatures)}", flush=True)    # 2. An agent solves it in the same session: it reads, writes solution.py, runs the tests, fixes, until all pass.    started = bx.agent.run(        "In /workspace/exercise, instructions.md describes a programming exercise. Write the solution in Python in "        "/workspace/exercise/solution.py with these functions:\n" + "\n".join(signatures) + "\n"        "Run the tests with `cd /workspace/exercise && python3 run_tests.py` (it prints JSON with the failed cases) and "        "fix the solution until every test passes. Do not change run_tests.py or canonical-data.json. Use only Python's "        "standard library. When done, report the last test run's numbers and your approach in two sentences.",        session_id=session.id,        max_steps=25,        output=OUTPUT,    )    print(f"Agent run {started['id']} ({started['model']}) is solving it…", flush=True)    run = bx.agent.wait(started["id"])    if run["status"] != "completed" or not run["result"]:        raise SystemExit(f"the run {run['status']}: {run.get('error') or 'no result'}")    claim = run["result"]    tool_steps = [s for s in run["steps"] if s["type"] == "tool"]    print(f"The agent says {claim['passed']}/{claim['total']} pass after {len(tool_steps)} tool steps: {claim['approach']}", flush=True)    # 3. The tests again, run by the example from a fresh download in another folder, with only solution.py copied over.    #    The agent's copies of the tests are compared with the originals too. (The agent shares the session's shell, so    #    its last `cd` would carry over: every command here names its folder with `cwd`.)    session.files.write("check/run_tests.py", runner)    verify = session.exec(        f"{fetch_exercise('check')} >/dev/null && cp ../exercise/solution.py . && "        "(cd ../exercise && sha256sum canonical-data.json run_tests.py) > ../agent-hashes.txt && python3 run_tests.py",        env={"SPECS": specs},        cwd="/workspace",        timeout_ms=120_000,    )    last = (verify["stdout"].strip().split("\n") or [""])[-1]    if not last.startswith("{"):        raise SystemExit(f"the tests did not run: {verify['stderr'].strip() or 'no solution.py'}")    tests = json.loads(last)    tests_changed = session.files.read_text("agent-hashes.txt").strip() != hashes.strip()    print(f"Run again by the example: {tests['passed']}/{tests['total']} pass" + (" (the agent changed the tests!)" if tests_changed else "; the agent left the tests as they were"))    out.mkdir(parents=True, exist_ok=True)    solution = session.files.read_text("exercise/solution.py")    (out / "solution.py").write_text(solution)    result = {        "exercise": exercise,        "source": specs,        "signatures": signatures,        "agent": claim,        "tests": tests,        "testsChanged": tests_changed,        "solutionLines": len(solution.strip().split("\n")),        "toolSteps": [{"name": s.get("name"), "input": s.get("input")} for s in tool_steps],        "agentRuns": [{"id": run["id"]}],    }    (out / "result.json").write_text(json.dumps(result, indent=2, ensure_ascii=False))

run_tests.py

"""Runs in the session's shell: the exercise's canonical test cases (Exercism problem-specifications,canonical-data.json) against solution.py, without any test framework.    python3 run_tests.py               every case; JSON {exercise, total, passed, failed: [...]}; exit 0 when all pass    python3 run_tests.py --signatures  the functions solution.py needs, with their argumentsEach case calls solution.<property>(**input) (camelCase names become snake_case) and compares the return value with"expected"; an expected {"error": "..."} means the call must raise an exception. A case another case "reimplements"is skipped, as Exercism's own test generators do."""import importlibimport jsonimport reimport sysimport tracebackdata = json.load(open("canonical-data.json"))def snake(name):    return re.sub(r"(?<!^)(?=[A-Z])", "_", name).lower()def flatten(cases, path=()):    for c in cases:        if "cases" in c:            yield from flatten(c["cases"], path + (c.get("description", ""),))        else:            yield {**c, "path": " / ".join(p for p in path + (c["description"],) if p)}cases = list(flatten(data["cases"]))replaced = {c["reimplements"] for c in cases if "reimplements" in c}cases = [c for c in cases if c.get("uuid") not in replaced]if "--signatures" in sys.argv:    seen = {}    for c in cases:        seen.setdefault(snake(c["property"]), [snake(k) for k in c.get("input", {})])    for name, args in seen.items():        print(f"def {name}({', '.join(args)}): ...")    sys.exit(0)try:    solution = importlib.import_module("solution")except Exception:    print(json.dumps({"exercise": data["exercise"], "total": len(cases), "passed": 0, "error": traceback.format_exc(limit=2)}))    sys.exit(1)failed = []for c in cases:    fn = getattr(solution, snake(c["property"]), None)    want = c["expected"]    wants_error = isinstance(want, dict) and set(want) == {"error"}    try:        if fn is None:            raise AttributeError(f"solution.py has no {snake(c['property'])}()")        got = fn(**{snake(k): v for k, v in c.get("input", {}).items()})        if isinstance(got, tuple):            got = list(got)        ok = not wants_error and got == want    except Exception as e:  # noqa: BLE001 (any exception counts as the expected error)        got = {"error": f"{type(e).__name__}: {e}"}        ok = wants_error    if not ok:        failed.append({"case": c["path"], "input": c.get("input"), "expected": want, "got": got})print(json.dumps({"exercise": data["exercise"], "total": len(cases), "passed": len(cases) - len(failed), "failed": failed[:10]}, ensure_ascii=False))sys.exit(0 if not failed else 1)

Inputs

Environment variables the example reads:

EXERCISE: the exercise's folder name in problem-specifications (default roman-numerals; also isbn-verifier, word-count, largest-series-product, …).

Run it

Get the examples with git clone https://github.com/Boxline-dev/examples.git and open this one: cd examples/solve-and-test. Put your API key in BOXLINE_API_KEY (and BOXLINE_API_URL for another API), then run either version from the example’s folder. Both write to output/ and stop their sessions when they finish.

# Node 18 or newer, in the example's folderexport BOXLINE_API_KEY=…   # your API keynpm install @boxline/sdk tsxnpx tsx node/index.ts

# Python 3.9 or newer, in the example's folderexport BOXLINE_API_KEY=…   # your API keypip install boxline-sdkpython python/main.py

Then check the result, from the same folder: npx tsx check.ts output. It exits with 0 only when the result is right.

What its check proves

check.ts reads what the example wrote and passes only when the result is right, not merely when the example ran.

npx tsx check.ts output exits with 0 only when every canonical case passes in the example's own run, the agent left run_tests.py and canonical-data.json as they were, it used the shell and ran the tests itself, and the numbers it reported are the ones the example measured.

Source

github.com/Boxline-dev/examples/solve-and-test on GitHub, with its README.md, check.ts and .env.example. The examples are open source. See all examples.

These docs describe a beta API. Endpoints, fields and SDK methods may change before general availability; breaking changes will be listed in the changelog.