MCP / Tool reference
fleetdeck_action
Execute one native action. Supply a stable unique requestId for each intended action. Return includes submission/completion status and screenshot images.
Parameters
sessionIdstringrequiredSession ID returned by Acquire a session. Use the key that owns that session.
"session_123"Validation pattern
^[A-Za-z0-9._:-]+$requestIdany JSONrequiredYour stable action identifier, unique within the session. Reusing it with the same command retrieves the existing action; changing the command causes IDEMPOTENCY_CONFLICT.
"tap-001"commandobjectrequiredOne strict action payload, selected by its type. Unknown fields are rejected.
View variants
Variant 1 · object
typestringrequiredDiscriminator that selects the action or content shape.
"screenshot"Unknown fields are rejected.
Variant 2 · object
typestringrequiredDiscriminator that selects the action or content shape.
"inspect"Unknown fields are rejected.
Variant 3 · object
typestringrequiredDiscriminator that selects the action or content shape.
"apps"Unknown fields are rejected.
Variant 4 · object
typestringrequiredDiscriminator that selects the action or content shape.
"tap"xnumberrequiredHorizontal position: 0 is the left edge and 1 is the right edge of the current screen.
ynumberrequiredVertical position: 0 is the top edge and 1 is the bottom edge of the current screen.
Unknown fields are rejected.
Variant 5 · object
typestringrequiredDiscriminator that selects the action or content shape.
"double_tap"xany JSONrequiredHorizontal position: 0 is the left edge and 1 is the right edge of the current screen.
yany JSONrequiredVertical position: 0 is the top edge and 1 is the bottom edge of the current screen.
Unknown fields are rejected.
Variant 6 · object
typestringrequiredDiscriminator that selects the action or content shape.
"long_press"xany JSONrequiredHorizontal position: 0 is the left edge and 1 is the right edge of the current screen.
yany JSONrequiredVertical position: 0 is the top edge and 1 is the bottom edge of the current screen.
msintegeroptionalGesture duration in milliseconds. The valid range depends on the action type.
600Unknown fields are rejected.
Variant 7 · object
typestringrequiredDiscriminator that selects the action or content shape.
"swipe"fromobjectrequiredStart point of the gesture, using normalized screen coordinates.
View child fields
xany JSONrequiredHorizontal position: 0 is the left edge and 1 is the right edge of the current screen.
yany JSONrequiredVertical position: 0 is the top edge and 1 is the bottom edge of the current screen.
Unknown fields are rejected.
toany JSONrequiredEnd point of the gesture, using normalized screen coordinates.
msintegeroptionalGesture duration in milliseconds. The valid range depends on the action type.
300Unknown fields are rejected.
Variant 8 · object
typestringrequiredDiscriminator that selects the action or content shape.
"text"valuestringrequiredText to type or write; for a clipboard read, the returned plaintext value. Writing the clipboard does not paste it.
Unknown fields are rejected.
Variant 9 · object
typestringrequiredDiscriminator that selects the action or content shape.
"key"keystringrequiredKeyboard key to send to the currently focused control.
"return""backspace""space""tab""up""down""left""right"Unknown fields are rejected.
Variant 10 · object
typestringrequiredDiscriminator that selects the action or content shape.
"button"namestringrequiredName of the resource or operation. For commands with an enum, choose one of the listed values.
"home""back""app_switcher""wake""lock""volume_up""volume_down"Unknown fields are rejected.
Variant 11 · object
typestringrequiredDiscriminator that selects the action or content shape.
"open_url"urlstringrequiredAbsolute URL or installed app scheme. javascript:, data:, file: and control characters are rejected.
Unknown fields are rejected.
Variant 12 · object
typestringrequiredDiscriminator that selects the action or content shape.
"launch_app"bundleIdstringrequiredInstalled iOS bundle ID or Android package name.
Validation pattern
^[A-Za-z0-9][A-Za-z0-9._-]*$Unknown fields are rejected.
Variant 13 · object
typestringrequiredDiscriminator that selects the action or content shape.
"kill_app"bundleIdany JSONrequiredInstalled iOS bundle ID or Android package name.
Unknown fields are rejected.
Variant 14 · object
typestringrequiredDiscriminator that selects the action or content shape.
"clipboard_get"Unknown fields are rejected.
Variant 15 · object
typestringrequiredDiscriminator that selects the action or content shape.
"set_clipboard"valueany JSONrequiredText to type or write; for a clipboard read, the returned plaintext value. Writing the clipboard does not paste it.
Unknown fields are rejected.
Variant 16 · object
typestringrequiredDiscriminator that selects the action or content shape.
"push_media"fileNamestringrequiredDestination filename with a matching media extension. Path separators are not allowed.
Validation pattern
^[A-Za-z0-9][A-Za-z0-9._ -]*$mimestringrequiredMedia MIME type. It must match the actual source bytes.
"image/jpeg""image/png""video/mp4"dataB64stringrequiredComplete padded base64 bytes without a data URL prefix. Use the complete file or the current ordered chunk.
Validation pattern
^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$Unknown fields are rejected.
Variant 17 · object
typestringrequiredDiscriminator that selects the action or content shape.
"media_chunk"uploadIdstringrequiredOne UUID shared by all chunks of the same upload.
chunkIndexintegerrequiredZero-based chunk index. Send in order, beginning at 0; it must be smaller than chunkCount.
chunkCountintegerrequiredTotal chunks in this upload. Must not exceed totalBytes; keep it constant across chunks.
totalBytesintegerrequiredDecoded byte size of the complete file. A one-chunk upload must decode to exactly this length.
fileNameany JSONrequiredDestination filename with a matching media extension. Path separators are not allowed.
mimestringrequiredMedia MIME type. It must match the actual source bytes.
"image/jpeg""image/png""video/mp4"dataB64stringrequiredComplete padded base64 bytes without a data URL prefix. Use the complete file or the current ordered chunk.
Validation pattern
^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$Unknown fields are rejected.
Variant 18 · object
typestringrequiredDiscriminator that selects the action or content shape.
"native_tool"nameany JSONrequiredName of the resource or operation. For commands with an enum, choose one of the listed values.
argumentsobjectoptionalArguments for the discovered native tool, validated against its advertised inputSchema.
{}Unknown fields are rejected.
Unknown fields are rejected.
View JSON Schema
{
"type": "object",
"properties": {
"sessionId": {
"type": "string",
"minLength": 1,
"maxLength": 128,
"pattern": "^[A-Za-z0-9._:-]+$"
},
"requestId": {
"$ref": "#/properties/sessionId"
},
"command": {
"anyOf": [
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "screenshot"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "inspect"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "apps"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "tap"
},
"x": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"y": {
"type": "number",
"minimum": 0,
"maximum": 1
}
},
"required": [
"type",
"x",
"y"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "double_tap"
},
"x": {
"$ref": "#/properties/command/anyOf/3/properties/x"
},
"y": {
"$ref": "#/properties/command/anyOf/3/properties/y"
}
},
"required": [
"type",
"x",
"y"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "long_press"
},
"x": {
"$ref": "#/properties/command/anyOf/3/properties/x"
},
"y": {
"$ref": "#/properties/command/anyOf/3/properties/y"
},
"ms": {
"type": "integer",
"minimum": 200,
"maximum": 1500,
"default": 600
}
},
"required": [
"type",
"x",
"y"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "swipe"
},
"from": {
"type": "object",
"properties": {
"x": {
"$ref": "#/properties/command/anyOf/3/properties/x"
},
"y": {
"$ref": "#/properties/command/anyOf/3/properties/y"
}
},
"required": [
"x",
"y"
],
"additionalProperties": false
},
"to": {
"$ref": "#/properties/command/anyOf/6/properties/from"
},
"ms": {
"type": "integer",
"minimum": 100,
"maximum": 1200,
"default": 300
}
},
"required": [
"type",
"from",
"to"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "text"
},
"value": {
"type": "string",
"maxLength": 16000
}
},
"required": [
"type",
"value"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "key"
},
"key": {
"type": "string",
"enum": [
"return",
"backspace",
"space",
"tab",
"up",
"down",
"left",
"right"
]
}
},
"required": [
"type",
"key"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "button"
},
"name": {
"type": "string",
"enum": [
"home",
"back",
"app_switcher",
"wake",
"lock",
"volume_up",
"volume_down"
]
}
},
"required": [
"type",
"name"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "open_url"
},
"url": {
"type": "string",
"minLength": 1,
"maxLength": 4096
}
},
"required": [
"type",
"url"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "launch_app"
},
"bundleId": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$"
}
},
"required": [
"type",
"bundleId"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "kill_app"
},
"bundleId": {
"$ref": "#/properties/command/anyOf/11/properties/bundleId"
}
},
"required": [
"type",
"bundleId"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "clipboard_get"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "set_clipboard"
},
"value": {
"$ref": "#/properties/command/anyOf/7/properties/value"
}
},
"required": [
"type",
"value"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "push_media"
},
"fileName": {
"type": "string",
"minLength": 1,
"maxLength": 160,
"pattern": "^[A-Za-z0-9][A-Za-z0-9._ -]*$"
},
"mime": {
"type": "string",
"enum": [
"image/jpeg",
"image/png",
"video/mp4"
]
},
"dataB64": {
"type": "string",
"minLength": 4,
"maxLength": 2800000,
"pattern": "^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
}
},
"required": [
"type",
"fileName",
"mime",
"dataB64"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "media_chunk"
},
"uploadId": {
"type": "string",
"format": "uuid"
},
"chunkIndex": {
"type": "integer",
"minimum": 0,
"maximum": 199
},
"chunkCount": {
"type": "integer",
"minimum": 1,
"maximum": 200
},
"totalBytes": {
"type": "integer",
"minimum": 1,
"maximum": 104857600
},
"fileName": {
"$ref": "#/properties/command/anyOf/15/properties/fileName"
},
"mime": {
"type": "string",
"enum": [
"image/jpeg",
"image/png",
"video/mp4"
]
},
"dataB64": {
"type": "string",
"minLength": 4,
"maxLength": 1400000,
"pattern": "^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
}
},
"required": [
"type",
"uploadId",
"chunkIndex",
"chunkCount",
"totalBytes",
"fileName",
"mime",
"dataB64"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "native_tool"
},
"name": {
"$ref": "#/properties/sessionId"
},
"arguments": {
"type": "object",
"additionalProperties": {},
"default": {}
}
},
"required": [
"type",
"name"
],
"additionalProperties": false
}
]
}
},
"required": [
"sessionId",
"requestId",
"command"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}Call example
const result = await client.callTool({
name: "fleetdeck_action",
arguments: {
"sessionId": "session_123",
"requestId": "tap-001",
"command": {
"type": "tap",
"x": 0.5,
"y": 0.6
}
}
});
if (result.isError) {
console.error(result.content);
} else {
console.log(result.content);
}The SDK example assumes an initialized MCP client. See connection setup for HTTP authentication, initialization and stdio configuration.
Response
The result contains content blocks and isError. The text block contains JSON with the fields below. See the corresponding REST response contract.
{
"content": [
{
"type": "text",
"text": "{\n \"id\": \"action_123\",\n \"requestId\": \"tap-001\",\n \"status\": \"completed\",\n \"kind\": \"tap\",\n \"createdAt\": \"2030-01-01T10:00:00.000Z\",\n \"finishedAt\": \"2030-01-01T10:00:00.000Z\",\n \"result\": {\n \"status\": \"submitted\",\n \"verified\": false,\n \"instruction\": \"Take a fresh screenshot to verify the intended effect before the next decision\"\n }\n}"
}
],
"isError": false
}idstringalways presentUnique identifier of this resource. Use the returned value in subsequent requests.
Validation pattern
^[A-Za-z0-9._:-]+$requestIdstringalways presentYour stable action identifier, unique within the session. Reusing it with the same command retrieves the existing action; changing the command causes IDEMPOTENCY_CONFLICT.
Validation pattern
^[A-Za-z0-9._:-]+$statusstringalways presentCurrent state. A completed action means the driver finished; verify the intended screen or app effect separately.
"queued""running""completed""failed""unknown""cancelled"kindstringalways presentAction type that produced this record.
"screenshot""inspect""apps""tap""double_tap""long_press""swipe""text""key""button""open_url""launch_app""kill_app""clipboard_get""set_clipboard""push_media""media_chunk""native_tool"createdAtstringalways presentUTC instant when the record was created.
finishedAtstring | nullalways presentUTC instant execution ended, or null while no terminal outcome is recorded.
resultany JSON | nullalways presentDriver-defined JSON or null. Known shapes are documented in ActionResult; native tool output may differ.
View child fields
statusstringoptionalCurrent state. A completed action means the driver finished; verify the intended screen or app effect separately.
"submitted""applied""staged""imported"verifiedbooleanoptionalMedia import acknowledgement is not publication verification. Other input normally returns false.
instructionstringoptionalDriver guidance for the next observation or recovery step.
contentobject[]optionalContent blocks returned by the driver. Screenshot image blocks contain base64 image bytes.
View child fields
typestringalways presentDiscriminator that selects the action or content shape.
"image"mimeTypestringalways presentMIME type of the returned image.
"image/jpeg""image/png"datastringalways presentBase64 image bytes, without a data URL prefix.
widthintegeroptionalCaptured image width in pixels.
heightintegeroptionalCaptured image height in pixels.
capturedAtstringoptionalUTC capture timestamp supplied by the relay.
coordinateSpacestringoptionalCoordinate convention for common FleetDeck actions. Normalized x/y values are relative to the current screen.
xmlstringoptionalADB UI hierarchy XML; bounds use screen pixels.
treeany JSONoptionalWDA accessibility tree; shape varies by version.
screenobjectoptionalADB {w,h} pixels; WDA {width,height} logical points.
packagesstring[]optionalAndroid package-name strings returned by ADB app inventory.
appsany JSON[]optionalPhone-agent-defined app entries. Entry fields depend on the agent version.
valuestringoptionalPlaintext clipboard value.
codestringoptionalExecution error code, distinct from an HTTP error.
errorstringoptionalHuman-readable failure explanation. Use code for programmatic handling.
Behavior
- The MCP descriptor marks this tool as potentially destructive and non-idempotent. The requestId protects action identity, not the external app effect.
- FleetDeck handler errors return isError=true with a JSON text payload. SDK input validation may return plain-text error content, and authentication or transport can fail before a tool result exists. Screenshot results place image content in MCP image blocks and omit those bytes from the parallel JSON block.
Errors
401UNAUTHORIZEDMissing, malformed, expired or revoked credential; user verification/password state can also deny access.
Recovery: Use Authorization: Bearer with a valid scoped key. For dashboard routes, sign in. Job children require an active parent job and timely renewal.
Full error reference ↗403FORBIDDENRequired scope, device grant, account membership or worker identity is missing.
Recovery: Check the endpoint’s scopes, exact device, owning key and workerId. A different key cannot access the original key’s sessions or jobs.
Full error reference ↗404NOT_FOUNDUnknown route or resource unavailable to this caller.
Recovery: Check method/path and use returned IDs from this account/key. Do not guess IDs.
Full error reference ↗402PLAN_REQUIREDDevice account has no active entitlement.
Recovery: Restore an active device entitlement in the workspace before controlling it.
Full error reference ↗409SESSION_EXPIREDSession expired, was closed or belongs to another key.
Recovery: Read the original action if needed, acquire a new session and inspect before new input. Heartbeat before the returned expiresAt; temporary child keys can have much shorter sessions than 120 seconds.
Full error reference ↗409ACTION_BUSYThis session has queued or running work.
Recovery: Poll the existing action to a terminal state before issuing another.
Full error reference ↗409IDEMPOTENCY_CONFLICTrequestId already identifies a different command.
Recovery: Retrieve that original action. Reuse its exact command only for lookup/retry; use a new ID only for a newly intended action.
Full error reference ↗429RATE_LIMITAt most 120 new actions per session in a rolling minute.
Recovery: Pause until the window clears. Poll existing actions instead of resubmitting new work. No Retry-After header is currently supplied.
Full error reference ↗422VALIDATIONA field, type, bound or unknown property failed validation.
Recovery: Read issues[].path and issues[].message. Match the endpoint schema; strict objects reject extra fields.
Full error reference ↗Client onlyCONNECTION_FAILEDThe client could not reach the configured FleetDeck origin.
Recovery: Check DNS, TLS, proxy/firewall and FLEETDECK_BASE_URL. Before replaying a mutation, query its saved requestId when connectivity returns.
Full error reference ↗Client onlyINVALID_RESPONSEThe server response was not valid FleetDeck JSON.
Recovery: Check that the base URL points to FleetDeck and that a proxy did not replace the response. Preserve the original action identifiers before retrying.
Full error reference ↗500INTERNALUnexpected service error.
Recovery: Save nonsecret request context and time for support. Query action status before repeating a mutation; never assume it did not execute.
Full error reference ↗SDK validation can return plain-text error content. Authentication and transport may fail before a tool result exists. Read the JSON-RPC and transport error reference before parsing the text as JSON.
Annotations
| Annotation | Value |
|---|---|
readOnlyHint | false |
destructiveHint | true |
idempotentHint | false |
openWorldHint | true |
