Vision plugins

The agent ships a vision engine: a host service that owns the camera(s), captures and normalizes frames, and publishes them on a shared frame bus that plugins read. A vision plugin does not open the camera itself. It subscribes to the normalized frame stream, runs (or asks the engine to run) inference, and publishes detections that overlays and mission logic pick up. This is a host-owned bus, not a per-plugin capture loop. The engine arbitrates the camera and the accelerator across every vision consumer, so two plugins can watch the same camera without fighting over the device or the NPU. The operator manages vision models and watches detections on the drone’s Perception page in the node’s Agent sidebar.

The frame contract

The engine downscales and converts each camera’s native format to a normalized frame, writes the pixels into a shared-memory ring (one ring per camera), and publishes a small frame descriptor on the vision.frame topic. The pixels never ride the RPC envelope. Your SDK resolves a descriptor to pixels for you by mapping the named ring read-only and reading the slot the descriptor points at. A resolved Frame carries the descriptor plus the pixel bytes. Descriptor fields:

Field	Meaning
`frame_id`	Monotonic frame counter for this camera, starting at 1.
`ts_ms`	Capture time in milliseconds.
`width`, `height`	Frame dimensions.
`format`	`"rgb24"`, `"nv12"`, or `"yuv420p"`.
`camera_id`	Source camera id, for filtering.

The Frame also gives you the pixel bytes: len(pixels) equals the descriptor’s byte_len, the valid pixel data for width x height in format. Default working resolution is 640x480; the engine downscales further per the active model’s input size. Drop policy is latest-wins: a slow consumer drops frames, it never blocks the engine. A torn or stale read (the writer recycled the slot mid-copy) is dropped silently and your callback does not fire for it.

Subscribing to frames

In Python, ctx.vision is a VisionClient. Subscribe with a callback that runs once per resolved frame:

class MyPlugin:
    async def on_start(self, ctx) -> None:
        async def on_frame(frame):
            # frame.descriptor.width, .height, .format, .camera_id, .ts_ms
            # frame.pixels is the resolved bytes for this frame
            detections = self.run(frame)
            for d in detections:
                ...
        await ctx.vision.subscribe_frames(on_frame)

subscribe_frames is gated on vision.frame.read. The callback may be sync or async. Keep it cheap: it runs on the delivery path, so offload heavy inference to a task or queue rather than blocking in the callback. The Rust SDK is the same surface: ados_sdk::vision::VisionClient, reached via ctx.vision, with subscribe_frames(camera_id, callback) and the same Frame shape. Types come from ados_sdk re-exports backed by ados_protocol::framebus, so a Python plugin and a Rust plugin read the same wire byte for byte. A Rust plugin’s entry point is ados_sdk::run_plugin::<MyPlugin, S>(...).

Running inference

There are two paths, and a plugin can use either. Both share the same accelerator, which the engine arbitrates. Engine-run. Register a model file plus its metadata with the engine. The engine loads it, runs it on the shared backend, and publishes detections itself. Your plugin only declares the model and then consumes detections like any other consumer.

from ados.sdk.vision import (
    ModelMetadata, ModelKind, ModelExecution, FrameFormat,
)

await ctx.vision.register_model(ModelMetadata(
    id="com.example.weeds",
    kind=ModelKind.DETECTION,
    execution=ModelExecution.ENGINE_RUN,
    input_width=640,
    input_height=480,
    input_format=FrameFormat.RGB24,
    output_classes=["weed", "crop"],
    model_path="/path/on/agent/weeds.onnx",
))

register_model is gated on vision.model.register. Plugin-side. Run your own model in the plugin and publish the results. You can also ask the engine to run a registered model against one frame on the shared backend with ctx.vision.infer(model_id, frame), which returns the detections without pixels crossing the RPC envelope.

async def on_frame(frame):
    boxes = self.session.run(frame)        # your own runtime
    await ctx.vision.publish_one(
        "com.example.weeds", frame,
        Detection(
            bbox=BoundingBox(x=100, y=80, width=40, height=40),
            class_label="weed",
            confidence=0.91,
        ),
    )

The accelerator is shared. A heavy or runaway model starves other vision consumers, so register honest input sizes and keep your inference cadence below your CPU and NPU budget. On boards with an NPU, load the board’s accelerator runtime (TFLite delegates, RKNN and so on) instead of running on the CPU.

Detections

A Detection has:

bbox: pixel-space box {x, y, width, height}, origin top-left, in the frame’s own resolution.
class_label: the class string.
confidence: the model’s score.
track_id: optional stable id across frames, for tracking models.

Detections are published as a DetectionBatch on the vision.detection topic, labelled by model_id plus the source camera_id, frame_id, and ts_ms so a consumer can align the boxes to the exact frame they came from.

await ctx.vision.publish_detection(DetectionBatch(
    model_id="com.example.weeds",
    camera_id=frame.descriptor.camera_id,
    frame_id=frame.descriptor.frame_id,
    ts_ms=frame.descriptor.ts_ms,
    detections=[...],
))

publish_detection (and the publish_one convenience) are gated on vision.detection.publish. Built-in consumers, the video overlay and mission logic, pick the batches up. The Rust SDK exposes the same publish_detection / publish_one on VisionClient.

Consuming detections

A plugin that acts on detections, rather than producing them, subscribes to the batches the engine publishes. The callback receives a DetectionBatch; pass camera_id to filter to one camera:

async def on_start(self, ctx) -> None:
    async def on_batch(batch) -> None:
        for d in batch.detections:
            self.track(d.class_label, d.bbox, d.track_id)

    await ctx.vision.subscribe_detections(on_batch, camera_id="uvc-0")

subscribe_detections is gated on vision.detection.subscribe.

Designating a follow target

The engine runs one single-object tracker per camera. designate_track sets the box it locks onto, overriding the auto-lock, the same way an operator’s click-to-follow does:

from ados.sdk.vision import BoundingBox

result = await ctx.vision.designate_track(
    "uvc-0",
    BoundingBox(x=310, y=200, width=60, height=90),
    class_label="person",
)
# result: {"designated": bool, "track_id": int | None, "camera_id": "uvc-0"}

designate_track is gated on vision.track.designate. Designation alone does not command the vehicle; a follow behavior that flies toward the target uses guided setpoints.

Capabilities

Declare the ones you use under agent.permissions in the manifest. Risk levels are on Permissions.

Capability	What it grants
`vision.frame.read`	Subscribe to normalized engine frames. The plugin never opens the camera itself.
`vision.model.register`	Register an inference model and run `infer` against shared frames.
`vision.model.read`	Read the delivery status and cached path of the plugin’s own declared models.
`vision.detection.publish`	Publish detections that overlays and mission logic consume.
`vision.detection.subscribe`	Receive detection batches through `ctx.vision.subscribe_detections`.
`vision.track.designate`	Set the engine’s follow target through `ctx.vision.designate_track`.
`vision.offload.advertise`	Report the perception offload link the plugin holds to a paired compute node. The agent counts the node as paired only when the advertised address answers.

The first-party Vision Navigation extension (vision-nav) is a reference for frame-bus work. It provides GPS-denied position hold from optical flow: a Rust agent half reads engine frames (vision.frame.read) and emits the flow message the configured firmware fuses over MAVLink (OPTICAL_FLOW for ArduPilot and iNav, OPTICAL_FLOW_RAD for PX4, plus DISTANCE_SENSOR when a rangefinder is wired to the companion). It supports ArduPilot 4.5+, PX4 1.14+ and iNav 7.0+. It does not inject pose through the visual-odometry helpers. Look at extensions/vision-nav/ in the ADOSExtensions repository for a real manifest.yaml, config schema and agent/GCS split. Operator setup is in Vision Navigation.

Multiple cameras

When more than one camera is attached, the engine publishes a separate ring and descriptor stream per camera. Filter to the one you want by passing camera_id:

await ctx.vision.subscribe_frames(on_frame, camera_id="uvc-0")

The Rust equivalent passes Some("uvc-0") for the camera_id argument. A None filter receives every camera’s frames; tag your detections with the frame’s camera_id so consumers know which camera each box came from.

Testing

Exercise the frame and detection path without a real host, engine, shared memory, or socket using the real test harness. The fake builds a real frame ring through the same frame-transport contract and resolves each synthetic frame the way the production client does, so your frame-handling path runs end to end; only the host and the OS shared-memory object are faked. In Python, use ados.sdk.testing.FakeVisionEngine:

from ados.sdk.testing import FakeVisionEngine
from ados.sdk.vision import FrameFormat

async def test_my_plugin():
    engine = FakeVisionEngine("uvc-0", 64, 48, FrameFormat.RGB24)
    seen = []
    engine.on_frame(lambda frame: seen.append(frame))
    engine.push_solid(0x80)        # one flat-grey frame
    await engine.deliver_all()
    assert len(seen) == 1

For an end-to-end run through the real VisionClient (the ring lives in a temp directory the client maps read-only, and the descriptor rides a real delivery event into the client’s resolver), build the engine with a file-backed ring and attach your callback:

engine = FakeVisionEngine.with_shm_dir("uvc-0", 64, 48, FrameFormat.RGB24)
await engine.attach(on_frame)
await engine.deliver_all()

Detections your plugin publishes are captured via engine.captured_detections(). The Rust harness mirrors this: ados_sdk::testing::FakeVisionEngine emits synthetic frames into the same FrameCallback your plugin registers and captures the detection batches it publishes.

Shipping models

A plugin gets its model weights onto the node in one of three ways. Bundled in the archive. Place the file under the plugin tree (for example agent/models/weeds.onnx). The archive’s SIGNATURE covers every entry, so the weights are protected by the same signature as the rest of the plugin. For an engine-run model, pass the file’s path as model_path on the model metadata. By reference. Declare the model under agent.contributes.vision.models. The agent fetches it, verifies it against the pinned sha256 and caches it. The digest sits in the signed manifest, so a fetched model is as trusted as a bundled one:

agent:
  permissions:
    - id: vision.model.read
  contributes:
    vision:
      models:
        - id: weeds-detector
          runtime: rknn          # onnx | rknn | hailo | tflite | pytorch
          board_match: rk3588    # board family this variant targets; default generic
          sha256: "<64 hex chars>"
          source: "https://models.example.com/weeds-rk3588.rknn"

The plugin reads where each model landed through the vision.read_model host method, gated on vision.model.read. It returns one entry per declared model with state (resolved, needs-model or verify-failed), model_id, runtime, path when resolved, and reason otherwise. In Rust it is vision_read_model() on the SDK client. As an install payload. agent.payloads lists files the agent downloads at install instead of shipping them in the archive. Each entry pins path (archive-relative destination), source (an https:// URL on the download allowlist), sha256 and size_bytes, and can narrow to node profiles or an arch_os. See Manifest. Do not fetch weights yourself at runtime; the plugin’s network.outbound policy blocks that by default.

Vendor binary models

Closed-source vendor binaries (for example, proprietary thermal AGC processing as a .so) ship under vendor/ per the vendor binaries doc. The plugin loads them with ctypes or cffi and calls them from its own inference path, then publishes the results as a DetectionBatch like any other plugin-side model.

Next steps

MAVLink extensions

Send results to the flight controller and inject visual odometry.

Performance and budgets

CPU, memory and accelerator limits for vision workloads.

State and storage

Where a plugin keeps data on the node.

Permissions

Every capability and its risk level.

Start here

Tutorials

Reference

SDKs

Deep dives

Test and distribute

The frame contract

Subscribing to frames

Running inference

Detections

Consuming detections

Designating a follow target

Capabilities

A worked extension: Vision Navigation

Multiple cameras

Testing

Shipping models

Vendor binary models

Next steps

MAVLink extensions

Performance and budgets

State and storage

Permissions

​The frame contract

​Subscribing to frames

​Running inference

​Detections

​Consuming detections

​Designating a follow target

​Capabilities

​A worked extension: Vision Navigation

​Multiple cameras

​Testing

​Shipping models

​Vendor binary models

​Next steps

MAVLink extensions

Performance and budgets

State and storage

Permissions

The frame contract

Subscribing to frames

Running inference

Detections

Consuming detections

Designating a follow target

Capabilities

A worked extension: Vision Navigation

Multiple cameras

Testing

Shipping models

Vendor binary models

Next steps