FleetDeckDocs

MCP / Tool reference

fleetdeck_action

Execute one native action. Supply a stable unique requestId for each intended action. Return includes submission/completion status and screenshot images.

Required permissionCommand-specific device scope; same session key

Parameters

sessionIdstringrequired

Session ID returned by Acquire a session. Use the key that owns that session.

min 1 charactersmax 128 characters
Example: "session_123"
Validation pattern^[A-Za-z0-9._:-]+$
requestIdany JSONrequired

Your stable action identifier, unique within the session. Reusing it with the same command retrieves the existing action; changing the command causes IDEMPOTENCY_CONFLICT.

Example: "tap-001"
commandobjectrequired

One strict action payload, selected by its type. Unknown fields are rejected.

View variants
Variant 1 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"screenshot"

Unknown fields are rejected.

Variant 2 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"inspect"

Unknown fields are rejected.

Variant 3 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"apps"

Unknown fields are rejected.

Variant 4 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"tap"
xnumberrequired

Horizontal position: 0 is the left edge and 1 is the right edge of the current screen.

min 0max 1
ynumberrequired

Vertical position: 0 is the top edge and 1 is the bottom edge of the current screen.

min 0max 1

Unknown fields are rejected.

Variant 5 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"double_tap"
xany JSONrequired

Horizontal position: 0 is the left edge and 1 is the right edge of the current screen.

yany JSONrequired

Vertical position: 0 is the top edge and 1 is the bottom edge of the current screen.

Unknown fields are rejected.

Variant 6 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"long_press"
xany JSONrequired

Horizontal position: 0 is the left edge and 1 is the right edge of the current screen.

yany JSONrequired

Vertical position: 0 is the top edge and 1 is the bottom edge of the current screen.

msintegeroptional

Gesture duration in milliseconds. The valid range depends on the action type.

min 200max 1,500default 600

Unknown fields are rejected.

Variant 7 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"swipe"
fromobjectrequired

Start point of the gesture, using normalized screen coordinates.

View child fields
xany JSONrequired

Horizontal position: 0 is the left edge and 1 is the right edge of the current screen.

yany JSONrequired

Vertical position: 0 is the top edge and 1 is the bottom edge of the current screen.

Unknown fields are rejected.

toany JSONrequired

End point of the gesture, using normalized screen coordinates.

msintegeroptional

Gesture duration in milliseconds. The valid range depends on the action type.

min 100max 1,200default 300

Unknown fields are rejected.

Variant 8 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"text"
valuestringrequired

Text to type or write; for a clipboard read, the returned plaintext value. Writing the clipboard does not paste it.

max 16,000 characters

Unknown fields are rejected.

Variant 9 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"key"
keystringrequired

Keyboard key to send to the currently focused control.

Allowed:"return""backspace""space""tab""up""down""left""right"

Unknown fields are rejected.

Variant 10 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"button"
namestringrequired

Name of the resource or operation. For commands with an enum, choose one of the listed values.

Allowed:"home""back""app_switcher""wake""lock""volume_up""volume_down"

Unknown fields are rejected.

Variant 11 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"open_url"
urlstringrequired

Absolute URL or installed app scheme. javascript:, data:, file: and control characters are rejected.

min 1 charactersmax 4,096 characters

Unknown fields are rejected.

Variant 12 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"launch_app"
bundleIdstringrequired

Installed iOS bundle ID or Android package name.

min 1 charactersmax 255 characters
Validation pattern^[A-Za-z0-9][A-Za-z0-9._-]*$

Unknown fields are rejected.

Variant 13 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"kill_app"
bundleIdany JSONrequired

Installed iOS bundle ID or Android package name.

Unknown fields are rejected.

Variant 14 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"clipboard_get"

Unknown fields are rejected.

Variant 15 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"set_clipboard"
valueany JSONrequired

Text to type or write; for a clipboard read, the returned plaintext value. Writing the clipboard does not paste it.

Unknown fields are rejected.

Variant 16 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"push_media"
fileNamestringrequired

Destination filename with a matching media extension. Path separators are not allowed.

min 1 charactersmax 160 characters
Validation pattern^[A-Za-z0-9][A-Za-z0-9._ -]*$
mimestringrequired

Media MIME type. It must match the actual source bytes.

Allowed:"image/jpeg""image/png""video/mp4"
dataB64stringrequired

Complete padded base64 bytes without a data URL prefix. Use the complete file or the current ordered chunk.

min 4 charactersmax 2,800,000 characters
Validation pattern^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$

Unknown fields are rejected.

Variant 17 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"media_chunk"
uploadIdstringrequired

One UUID shared by all chunks of the same upload.

uuid
chunkIndexintegerrequired

Zero-based chunk index. Send in order, beginning at 0; it must be smaller than chunkCount.

min 0max 199
chunkCountintegerrequired

Total chunks in this upload. Must not exceed totalBytes; keep it constant across chunks.

min 1max 200
totalBytesintegerrequired

Decoded byte size of the complete file. A one-chunk upload must decode to exactly this length.

min 1max 104,857,600
fileNameany JSONrequired

Destination filename with a matching media extension. Path separators are not allowed.

mimestringrequired

Media MIME type. It must match the actual source bytes.

Allowed:"image/jpeg""image/png""video/mp4"
dataB64stringrequired

Complete padded base64 bytes without a data URL prefix. Use the complete file or the current ordered chunk.

min 4 charactersmax 1,400,000 characters
Validation pattern^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$

Unknown fields are rejected.

Variant 18 · object
typestringrequired

Discriminator that selects the action or content shape.

Allowed:"native_tool"
nameany JSONrequired

Name of the resource or operation. For commands with an enum, choose one of the listed values.

argumentsobjectoptional

Arguments for the discovered native tool, validated against its advertised inputSchema.

default {}

Unknown fields are rejected.

Unknown fields are rejected.

View JSON Schema
json
{
  "type": "object",
  "properties": {
    "sessionId": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128,
      "pattern": "^[A-Za-z0-9._:-]+$"
    },
    "requestId": {
      "$ref": "#/properties/sessionId"
    },
    "command": {
      "anyOf": [
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "screenshot"
            }
          },
          "required": [
            "type"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "inspect"
            }
          },
          "required": [
            "type"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "apps"
            }
          },
          "required": [
            "type"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "tap"
            },
            "x": {
              "type": "number",
              "minimum": 0,
              "maximum": 1
            },
            "y": {
              "type": "number",
              "minimum": 0,
              "maximum": 1
            }
          },
          "required": [
            "type",
            "x",
            "y"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "double_tap"
            },
            "x": {
              "$ref": "#/properties/command/anyOf/3/properties/x"
            },
            "y": {
              "$ref": "#/properties/command/anyOf/3/properties/y"
            }
          },
          "required": [
            "type",
            "x",
            "y"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "long_press"
            },
            "x": {
              "$ref": "#/properties/command/anyOf/3/properties/x"
            },
            "y": {
              "$ref": "#/properties/command/anyOf/3/properties/y"
            },
            "ms": {
              "type": "integer",
              "minimum": 200,
              "maximum": 1500,
              "default": 600
            }
          },
          "required": [
            "type",
            "x",
            "y"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "swipe"
            },
            "from": {
              "type": "object",
              "properties": {
                "x": {
                  "$ref": "#/properties/command/anyOf/3/properties/x"
                },
                "y": {
                  "$ref": "#/properties/command/anyOf/3/properties/y"
                }
              },
              "required": [
                "x",
                "y"
              ],
              "additionalProperties": false
            },
            "to": {
              "$ref": "#/properties/command/anyOf/6/properties/from"
            },
            "ms": {
              "type": "integer",
              "minimum": 100,
              "maximum": 1200,
              "default": 300
            }
          },
          "required": [
            "type",
            "from",
            "to"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "text"
            },
            "value": {
              "type": "string",
              "maxLength": 16000
            }
          },
          "required": [
            "type",
            "value"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "key"
            },
            "key": {
              "type": "string",
              "enum": [
                "return",
                "backspace",
                "space",
                "tab",
                "up",
                "down",
                "left",
                "right"
              ]
            }
          },
          "required": [
            "type",
            "key"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "button"
            },
            "name": {
              "type": "string",
              "enum": [
                "home",
                "back",
                "app_switcher",
                "wake",
                "lock",
                "volume_up",
                "volume_down"
              ]
            }
          },
          "required": [
            "type",
            "name"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "open_url"
            },
            "url": {
              "type": "string",
              "minLength": 1,
              "maxLength": 4096
            }
          },
          "required": [
            "type",
            "url"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "launch_app"
            },
            "bundleId": {
              "type": "string",
              "minLength": 1,
              "maxLength": 255,
              "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$"
            }
          },
          "required": [
            "type",
            "bundleId"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "kill_app"
            },
            "bundleId": {
              "$ref": "#/properties/command/anyOf/11/properties/bundleId"
            }
          },
          "required": [
            "type",
            "bundleId"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "clipboard_get"
            }
          },
          "required": [
            "type"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "set_clipboard"
            },
            "value": {
              "$ref": "#/properties/command/anyOf/7/properties/value"
            }
          },
          "required": [
            "type",
            "value"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "push_media"
            },
            "fileName": {
              "type": "string",
              "minLength": 1,
              "maxLength": 160,
              "pattern": "^[A-Za-z0-9][A-Za-z0-9._ -]*$"
            },
            "mime": {
              "type": "string",
              "enum": [
                "image/jpeg",
                "image/png",
                "video/mp4"
              ]
            },
            "dataB64": {
              "type": "string",
              "minLength": 4,
              "maxLength": 2800000,
              "pattern": "^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
            }
          },
          "required": [
            "type",
            "fileName",
            "mime",
            "dataB64"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "media_chunk"
            },
            "uploadId": {
              "type": "string",
              "format": "uuid"
            },
            "chunkIndex": {
              "type": "integer",
              "minimum": 0,
              "maximum": 199
            },
            "chunkCount": {
              "type": "integer",
              "minimum": 1,
              "maximum": 200
            },
            "totalBytes": {
              "type": "integer",
              "minimum": 1,
              "maximum": 104857600
            },
            "fileName": {
              "$ref": "#/properties/command/anyOf/15/properties/fileName"
            },
            "mime": {
              "type": "string",
              "enum": [
                "image/jpeg",
                "image/png",
                "video/mp4"
              ]
            },
            "dataB64": {
              "type": "string",
              "minLength": 4,
              "maxLength": 1400000,
              "pattern": "^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
            }
          },
          "required": [
            "type",
            "uploadId",
            "chunkIndex",
            "chunkCount",
            "totalBytes",
            "fileName",
            "mime",
            "dataB64"
          ],
          "additionalProperties": false
        },
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "const": "native_tool"
            },
            "name": {
              "$ref": "#/properties/sessionId"
            },
            "arguments": {
              "type": "object",
              "additionalProperties": {},
              "default": {}
            }
          },
          "required": [
            "type",
            "name"
          ],
          "additionalProperties": false
        }
      ]
    }
  },
  "required": [
    "sessionId",
    "requestId",
    "command"
  ],
  "additionalProperties": false,
  "$schema": "http://json-schema.org/draft-07/schema#"
}

Call example

MCP call · SDK
const result = await client.callTool({
  name: "fleetdeck_action",
  arguments: {
    "sessionId": "session_123",
    "requestId": "tap-001",
    "command": {
      "type": "tap",
      "x": 0.5,
      "y": 0.6
    }
  }
});
if (result.isError) {
  console.error(result.content);
} else {
  console.log(result.content);
}

The SDK example assumes an initialized MCP client. See connection setup for HTTP authentication, initialization and stdio configuration.

Response

The result contains content blocks and isError. The text block contains JSON with the fields below. See the corresponding REST response contract.

Illustrative tool result
{
  "content": [
    {
      "type": "text",
      "text": "{\n  \"id\": \"action_123\",\n  \"requestId\": \"tap-001\",\n  \"status\": \"completed\",\n  \"kind\": \"tap\",\n  \"createdAt\": \"2030-01-01T10:00:00.000Z\",\n  \"finishedAt\": \"2030-01-01T10:00:00.000Z\",\n  \"result\": {\n    \"status\": \"submitted\",\n    \"verified\": false,\n    \"instruction\": \"Take a fresh screenshot to verify the intended effect before the next decision\"\n  }\n}"
    }
  ],
  "isError": false
}
idstringalways present

Unique identifier of this resource. Use the returned value in subsequent requests.

min 1 charactersmax 128 characters
Validation pattern^[A-Za-z0-9._:-]+$
requestIdstringalways present

Your stable action identifier, unique within the session. Reusing it with the same command retrieves the existing action; changing the command causes IDEMPOTENCY_CONFLICT.

min 1 charactersmax 128 characters
Validation pattern^[A-Za-z0-9._:-]+$
statusstringalways present

Current state. A completed action means the driver finished; verify the intended screen or app effect separately.

Allowed:"queued""running""completed""failed""unknown""cancelled"
kindstringalways present

Action type that produced this record.

Allowed:"screenshot""inspect""apps""tap""double_tap""long_press""swipe""text""key""button""open_url""launch_app""kill_app""clipboard_get""set_clipboard""push_media""media_chunk""native_tool"
createdAtstringalways present

UTC instant when the record was created.

date-time
finishedAtstring | nullalways present

UTC instant execution ended, or null while no terminal outcome is recorded.

date-time
resultany JSON | nullalways present

Driver-defined JSON or null. Known shapes are documented in ActionResult; native tool output may differ.

View child fields
statusstringoptional

Current state. A completed action means the driver finished; verify the intended screen or app effect separately.

Allowed:"submitted""applied""staged""imported"
verifiedbooleanoptional

Media import acknowledgement is not publication verification. Other input normally returns false.

instructionstringoptional

Driver guidance for the next observation or recovery step.

contentobject[]optional

Content blocks returned by the driver. Screenshot image blocks contain base64 image bytes.

View child fields
typestringalways present

Discriminator that selects the action or content shape.

Allowed:"image"
mimeTypestringalways present

MIME type of the returned image.

Allowed:"image/jpeg""image/png"
datastringalways present

Base64 image bytes, without a data URL prefix.

widthintegeroptional

Captured image width in pixels.

heightintegeroptional

Captured image height in pixels.

capturedAtstringoptional

UTC capture timestamp supplied by the relay.

date-time
coordinateSpacestringoptional

Coordinate convention for common FleetDeck actions. Normalized x/y values are relative to the current screen.

xmlstringoptional

ADB UI hierarchy XML; bounds use screen pixels.

treeany JSONoptional

WDA accessibility tree; shape varies by version.

screenobjectoptional

ADB {w,h} pixels; WDA {width,height} logical points.

packagesstring[]optional

Android package-name strings returned by ADB app inventory.

appsany JSON[]optional

Phone-agent-defined app entries. Entry fields depend on the agent version.

valuestringoptional

Plaintext clipboard value.

codestringoptional

Execution error code, distinct from an HTTP error.

errorstringoptional

Human-readable failure explanation. Use code for programmatic handling.

Behavior

  • The MCP descriptor marks this tool as potentially destructive and non-idempotent. The requestId protects action identity, not the external app effect.
  • FleetDeck handler errors return isError=true with a JSON text payload. SDK input validation may return plain-text error content, and authentication or transport can fail before a tool result exists. Screenshot results place image content in MCP image blocks and omit those bytes from the parallel JSON block.

Errors

401UNAUTHORIZEDMissing, malformed, expired or revoked credential; user verification/password state can also deny access.

Recovery: Use Authorization: Bearer with a valid scoped key. For dashboard routes, sign in. Job children require an active parent job and timely renewal.

Full error reference ↗
403FORBIDDENRequired scope, device grant, account membership or worker identity is missing.

Recovery: Check the endpoint’s scopes, exact device, owning key and workerId. A different key cannot access the original key’s sessions or jobs.

Full error reference ↗
404NOT_FOUNDUnknown route or resource unavailable to this caller.

Recovery: Check method/path and use returned IDs from this account/key. Do not guess IDs.

Full error reference ↗
402PLAN_REQUIREDDevice account has no active entitlement.

Recovery: Restore an active device entitlement in the workspace before controlling it.

Full error reference ↗
409SESSION_EXPIREDSession expired, was closed or belongs to another key.

Recovery: Read the original action if needed, acquire a new session and inspect before new input. Heartbeat before the returned expiresAt; temporary child keys can have much shorter sessions than 120 seconds.

Full error reference ↗
409ACTION_BUSYThis session has queued or running work.

Recovery: Poll the existing action to a terminal state before issuing another.

Full error reference ↗
409IDEMPOTENCY_CONFLICTrequestId already identifies a different command.

Recovery: Retrieve that original action. Reuse its exact command only for lookup/retry; use a new ID only for a newly intended action.

Full error reference ↗
429RATE_LIMITAt most 120 new actions per session in a rolling minute.

Recovery: Pause until the window clears. Poll existing actions instead of resubmitting new work. No Retry-After header is currently supplied.

Full error reference ↗
422VALIDATIONA field, type, bound or unknown property failed validation.

Recovery: Read issues[].path and issues[].message. Match the endpoint schema; strict objects reject extra fields.

Full error reference ↗
Client onlyCONNECTION_FAILEDThe client could not reach the configured FleetDeck origin.

Recovery: Check DNS, TLS, proxy/firewall and FLEETDECK_BASE_URL. Before replaying a mutation, query its saved requestId when connectivity returns.

Full error reference ↗
Client onlyINVALID_RESPONSEThe server response was not valid FleetDeck JSON.

Recovery: Check that the base URL points to FleetDeck and that a proxy did not replace the response. Preserve the original action identifiers before retrying.

Full error reference ↗
500INTERNALUnexpected service error.

Recovery: Save nonsecret request context and time for support. Query action status before repeating a mutation; never assume it did not execute.

Full error reference ↗

SDK validation can return plain-text error content. Authentication and transport may fail before a tool result exists. Read the JSON-RPC and transport error reference before parsing the text as JSON.

Annotations

AnnotationValue
readOnlyHintfalse
destructiveHinttrue
idempotentHintfalse
openWorldHinttrue