These actions drive a session’s mouse and keyboard directly: move the pointer, hover, click with any button, press and release, drag, scroll at a point, and press key combinations. Use them for pages that need a real pointer, such as sliders, drag and drop, canvas apps and hover menus, and to run the moves of a computer-use model.

Coordinates are CSS pixels of the page’s viewport (the session’s viewport, 1280 by 720 by default), counted from its top-left corner.
They act on the session’s browser page only, through the DevTools protocol. Nothing reaches the machine’s desktop or operating system.
The pointer moves in straight lines between points. Nothing is randomised.

Send actions

They are actions like the others: send a list to POST /v1/sessions/:id/browser/actions and they run in order, next to the browser, until one fails. Every result has a text line saying what happened, such as “Double-clicked at (190, 112)”. See Steps for the other actions.

curl -X POST https://api.boxline.dev/v1/sessions/$SESSION_ID/browser/actions \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d @- <<'JSON'{  "actions": [    {"action": "goto", "url": "https://example.com"},    {"action": "move", "x": 640, "y": 360, "steps": 10},    {"action": "hover", "selector": "text=More information"},    {"action": "click", "x": 190, "y": 112, "count": 2},    {"action": "click", "selector": "h1", "button": "right"}  ]}JSON

import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URLconst results = await bx.sessions.actions("SESSION_ID", [  { action: "goto", url: "https://example.com" },  { action: "move", x: 640, y: 360, steps: 10 },  { action: "hover", selector: "text=More information" },  { action: "click", x: 190, y: 112, count: 2 },  { action: "click", selector: "h1", button: "right" },]);for (const r of results) console.log(r.action, r.ok ? r.text : r.error, r.value ?? "");

from boxline import Boxlinebx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URLresults = bx.sessions.actions("SESSION_ID", [    {"action": "goto", "url": "https://example.com"},    {"action": "move", "x": 640, "y": 360, "steps": 10},    {"action": "hover", "selector": "text=More information"},    {"action": "click", "x": 190, "y": 112, "count": 2},    {"action": "click", "selector": "h1", "button": "right"},])for r in results:    print(r["action"], r.get("text") if r["ok"] else r.get("error"), r.get("value", ""))

The SDKs also have helpers on a session: session.mouse.move(), moveBy(), click(), down(), up() and drag(), session.hover(), session.keyboard.key() and session.cursor(). See the Node SDK and the Python SDK.

Response (shape)

{  "results": [    { "ok": true, "action": "goto", "text": "Opened https://example.com/ (HTTP 200)", "value": { "…": "…" }, "ms": 812 },    { "ok": true, "action": "move", "text": "Moved to (640, 360) in 10 steps", "value": { "x": 640, "y": 360 }, "ms": 40 },    { "ok": true, "action": "hover", "text": "…", "ms": 35 },    { "ok": true, "action": "click", "text": "Double-clicked at (190, 112)", "ms": 60 },    { "ok": true, "action": "click", "text": "…", "ms": 48 }  ]}

The actions

Action	Body	Value
`move`	{ x, y, steps? } or { dx, dy, steps? } (from where the pointer is)	{ x, y }: where the pointer is now
`hover`	{ selector } or { x, y }	none
`click`	{ selector } or { x, y }, button? ("left", "right", "middle"), count? (1, 2 or 3), modifiers? (["Shift"])	none
`mouse_down, mouse_up`	{ button? }, where the pointer is	none
`drag`	{ from, to, steps? } (each end { x, y } or a selector), or { path: [{ x, y }, …], steps? }; button?, modifiers?	{ from: { x, y }, to: { x, y } }
`scroll`	{ x?, y?, deltaX?, deltaY? } (moves to x, y first; wheel distances in pixels, at most 50,000), modifiers?	none
`key`	{ keys: "Control+A" } or { keys: ["Control", "A"] }, holdMs? (0 to 10,000: keep them down)	none
`cursor`	none	{ x, y }: where the pointer is
`screenshot`	maxWidth? (100 to 3840), cursor? (true draws the pointer), plus fullPage?, format?, quality?	{ data, mimeType }, and width, height and scale with maxWidth or cursor

steps (1 to 100) moves in a straight line through that many evenly spaced points, for pages that follow the pointer as it moves. For a drag the default is 10; along a path it is per segment, default 1.
A path has 2 to 200 points. A drag end given as a selector is that element’s centre, scrolled into view first.
press presses one key. key takes combinations.

Drag and drop

A drag presses the button at one end, moves to the other and lets go. Drag a slider handle between two points, a card between two elements, or follow a path of points:

curl -X POST https://api.boxline.dev/v1/sessions/$SESSION_ID/browser/actions \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d @- <<'JSON'{  "actions": [    {      "action": "drag",      "from": {"x": 180, "y": 200},      "to": {"x": 400, "y": 200},      "steps": 10    },    {"action": "drag", "from": "#card-1", "to": "#done-column"},    {      "action": "drag",      "path": [{"x": 100, "y": 300}, {"x": 200, "y": 260}, {"x": 300, "y": 300}],      "steps": 5    }  ]}JSON

import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URLconst results = await bx.sessions.actions("SESSION_ID", [  {    action: "drag",    from: { x: 180, y: 200 },    to: { x: 400, y: 200 },    steps: 10,  },  { action: "drag", from: "#card-1", to: "#done-column" },  {    action: "drag",    path: [{ x: 100, y: 300 }, { x: 200, y: 260 }, { x: 300, y: 300 }],    steps: 5,  },]);for (const r of results) console.log(r.action, r.ok ? r.text : r.error, r.value ?? "");

from boxline import Boxlinebx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URLresults = bx.sessions.actions("SESSION_ID", [    {        "action": "drag",        "from": {"x": 180, "y": 200},        "to": {"x": 400, "y": 200},        "steps": 10,    },    {"action": "drag", "from": "#card-1", "to": "#done-column"},    {        "action": "drag",        "path": [{"x": 100, "y": 300}, {"x": 200, "y": 260}, {"x": 300, "y": 300}],        "steps": 5,    },])for r in results:    print(r["action"], r.get("text") if r["ok"] else r.get("error"), r.get("value", ""))

For more control, press and release the button yourself with mouse_down and mouse_up where the pointer is, and move in between. move with dx and dy moves from where the pointer is:

curl -X POST https://api.boxline.dev/v1/sessions/$SESSION_ID/browser/actions \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d @- <<'JSON'{  "actions": [    {"action": "move", "x": 300, "y": 400},    {"action": "mouse_down"},    {"action": "move", "dx": 120, "dy": 0, "steps": 12},    {"action": "mouse_up"}  ]}JSON

import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URLconst results = await bx.sessions.actions("SESSION_ID", [  { action: "move", x: 300, y: 400 },  { action: "mouse_down" },  { action: "move", dx: 120, dy: 0, steps: 12 },  { action: "mouse_up" },]);for (const r of results) console.log(r.action, r.ok ? r.text : r.error, r.value ?? "");

from boxline import Boxlinebx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URLresults = bx.sessions.actions("SESSION_ID", [    {"action": "move", "x": 300, "y": 400},    {"action": "mouse_down"},    {"action": "move", "dx": 120, "dy": 0, "steps": 12},    {"action": "mouse_up"},])for r in results:    print(r["action"], r.get("text") if r["ok"] else r.get("error"), r.get("value", ""))

Keys, scrolling and the pointer

curl -X POST https://api.boxline.dev/v1/sessions/$SESSION_ID/browser/actions \  -H "x-api-key: $BOXLINE_API_KEY" \  -H "content-type: application/json" \  -d @- <<'JSON'{  "actions": [    {"action": "click", "selector": "textarea"},    {"action": "key", "keys": "Control+A Delete"},    {"action": "type", "text": "Hello"},    {"action": "key", "keys": ["Shift", "Tab"]},    {"action": "scroll", "x": 640, "y": 400, "deltaY": 800},    {"action": "cursor"},    {"action": "screenshot", "maxWidth": 640, "cursor": true}  ]}JSON

import { Boxline } from "@boxline/sdk";const bx = new Boxline(); // reads BOXLINE_API_KEY and BOXLINE_API_URLconst results = await bx.sessions.actions("SESSION_ID", [  { action: "click", selector: "textarea" },  { action: "key", keys: "Control+A Delete" },  { action: "type", text: "Hello" },  { action: "key", keys: ["Shift", "Tab"] },  { action: "scroll", x: 640, y: 400, deltaY: 800 },  { action: "cursor" },  { action: "screenshot", maxWidth: 640, cursor: true },]);for (const r of results) console.log(r.action, r.ok ? r.text : r.error, r.value ?? "");

from boxline import Boxlinebx = Boxline()  # reads BOXLINE_API_KEY and BOXLINE_API_URLresults = bx.sessions.actions("SESSION_ID", [    {"action": "click", "selector": "textarea"},    {"action": "key", "keys": "Control+A Delete"},    {"action": "type", "text": "Hello"},    {"action": "key", "keys": ["Shift", "Tab"]},    {"action": "scroll", "x": 640, "y": 400, "deltaY": 800},    {"action": "cursor"},    {"action": "screenshot", "maxWidth": 640, "cursor": True},])for r in results:    print(r["action"], r.get("text") if r["ok"] else r.get("error"), r.get("value", ""))

Key names are Playwright’s: Enter, Tab, Escape, ArrowLeft, PageDown, F5, Control, Shift, Alt, Meta, ControlOrMeta, or one character. Common other spellings work too: ctrl, cmd, super, Return, Page_Down, ESC.
One string can hold several combinations separated by spaces, pressed one after the other: "ctrl+a Delete".
The session machines run Linux, so select-all is Control+A there. ControlOrMeta+A works everywhere.
scroll moves the pointer to x and y first, so the wheel scrolls what is under it, such as a list inside the page.

Where the pointer is

The API remembers the last place it moved the pointer to, on each tab. cursor returns it, and move with dx and dy starts from it. It is (0, 0) before the first move, and the element’s centre after a click or hover on a selector.

Screenshots for a model

maxWidth scales a screenshot down to at most that width, keeping its shape. cursor: true draws the pointer on it. With either one, the value also has width, height and scale: image pixels per CSS pixel, for example 0.5 for a 1280-wide page with maxWidth: 640. Divide a point in the image by scale to get CSS pixels. Both options work on screenshots of the viewport only; with fullPage they are a 400.

Finding coordinates

The elements action lists the page as a model sees it. Elements in view end with at (x,y), their centre in CSS pixels of the viewport, and sliders show as slider. That is a good place to pick points from.

Errors

The whole list is checked before anything in it runs. A mistake answers 400 and nothing runs:

out_of_viewport: a point outside the page, for example “action 2 (click): (5000, 10) is outside the viewport; the page is 1280×720 CSS pixels, so x must be 0–1279 and y 0–719”.
invalid_request: a bad shape, such as an unknown button or key, count not 1 to 3, steps not 1 to 100, a path of more than 200 points, or both x/y and dx/dy.

A relative move that would end outside the page fails that one action, with ok: false and code: "out_of_viewport".

Limits and safety

Screenshots (the screenshot action and computer use together) are limited to 300 per minute per session. See Rate limits.
Step variables are never filled into mouse or key actions. They only apply to plain-English steps.
A screenshot shows whatever the page shows, including personal data. Keep screenshots out of shared logs.

These docs describe a beta API. Endpoints, fields and SDK methods may change before general availability; breaking changes will be listed in the changelog.