Video stack architecture

ADOS carries live HD video from the drone’s camera to a browser using open-source parts at every stage: V4L2 for capture, ffmpeg for encode, WFB-ng for the radio hop, MediaMTX for serving, WebRTC for delivery. On the aircraft the ados-video unit supervises the encoder and the local MediaMTX instance. On the ground station ados-groundlink owns the radio receiver and the fan-out, and its --ground-video mode (the ados-mediamtx-gs unit) runs a second MediaMTX instance that reads the radio stream and serves the browser.

The hop map

This is the list to measure against. Numbers in brackets are the hop labels used by ados diag video.

DRONE
  [1]  camera (/dev/video0, CSI or USB UVC) via V4L2
  [2]  encoder: one ffmpeg process, H.264, GOP = fps/2
       ├─ [3]  tee branch A: RTSP publish to the drone's own MediaMTX `main` path, :8554
       └─ [4]  tee branch B: RTP (RFC 6184) to UDP 5600 on loopback
  [5]  wfb_tx -p 0 -u 5600  (FEC k=8/n=12, 20 MHz)
  [6]  RTL8812EU in monitor mode, 5 GHz
GROUND
  [7]  wfb_rx -p 0 -u 5599 -l 1000   decodes and FEC-recovers
  [8]  fan-out :5599 -> :5600 (MediaMTX ingest) + :5605 (on-box display)
  [9]  ground MediaMTX reads RTP natively from udp+rtp://127.0.0.1:5600 (no ffmpeg)
  [10] ground MediaMTX `main` path
  [11] :8080/whep (ados-control, relayed to MediaMTX on 127.0.0.1:8889) -> cockpit or Mission Control

Telemetry does not share this path. MAVLink, node status and identity ride the auxiliary lane (radio ports 2 and 3, UDP 5602 and 5603), separate from video (radio port 0) and from the frequency-hop control plane (radio port 1, UDP 5803 and 5810). A dark video feed with a live telemetry link is therefore a normal combination, and it points at hops 2 to 10. main is the canonical path name on both nodes. MediaMTX serves WHEP at 127.0.0.1:8889/main/whep, on loopback only; browsers reach it through ados-control at :8080/whep. Nothing publishes to /ados, so a zero byte count on that path is not evidence of a video fault.

Stage 1: Capture

The agent scans /dev/video* at boot and supports MIPI CSI cameras and USB UVC cameras through V4L2. It sets resolution, framerate and pixel format before starting the encoder. The shipped default is 1280x720 at 30 fps. The encoder choice comes from the board’s HAL profile. Rockchip boards with a media engine can use mpph264enc through GStreamer. Boards exposing a V4L2 hardware encoder use h264_v4l2m2m through ffmpeg. Everything else falls back to libx264.

Stage 2: Encode

One process does the encode and both publishes. The default path (USB and IP cameras) is a single ffmpeg whose output is ffmpeg’s tee muxer: branch A is the RTSP publish into the local MediaMTX, branch B is the RTP copy the radio reads.

ffmpeg -fflags nobuffer -flags low_delay \
  -probesize 32 -analyzeduration 0 \
  -f v4l2 -input_format mjpeg -video_size 1280x720 -framerate 30 \
  -i /dev/video0 \
  -c:v libx264 -b:v 4000k \
  -pix_fmt yuv420p -profile:v high -level:v 4.1 \
  -preset ultrafast -tune zerolatency \
  -g 15 -bf 0 -refs 1 -threads 4 -flush_packets 1 \
  -x264-params no-mbtree=1:sync-lookahead=0:rc-lookahead=0:sliced-threads=1:slices=4:scenecut=0 \
  -bsf:v h264_mp4toannexb,dump_extra=freq=keyframe \
  -max_delay 0 -muxdelay 0 -muxpreload 0 \
  -map 0:v -f tee \
  '[f=rtsp:rtsp_transport=tcp:max_delay=0:flush_packets=1]rtsp://127.0.0.1:8554/main|[f=rtp:payload_type=96:ssrc=0xCAFE:max_delay=0:flush_packets=1]rtp://127.0.0.1:5600?pkt_size=1316'

Flag	Why it is there
`-fflags nobuffer -flags low_delay`	No input buffering, minimum encoder delay
`-probesize 32 -analyzeduration 0`	Skip format analysis on a source whose format is already known
`-preset ultrafast -tune zerolatency`	Fastest x264 path, no B-frames, no lookahead
`-profile:v high -level:v 4.1 -pix_fmt yuv420p`	`avc1.640029`, which the browser MSE player pins
`-g 15`	GOP is `fps/2`, so an IDR arrives every 500 ms at 30 fps
`-threads 4` plus `sliced-threads=1:slices=4`	Sliced threading emits a frame whole. Frame-level threading holds four frames in flight, which is about 133 ms of pure pipeline delay at 30 fps. Boards with fewer than four probed cores keep the two-thread frame-threaded form instead.
`-bsf:v ...,dump_extra=freq=keyframe`	Re-inserts SPS/PPS in-band before every IDR
`-max_delay 0 -muxdelay 0 -muxpreload 0`	ffmpeg’s defaults here are about 0.7 s of mux delay plus 0.5 s of preload, in front of every downstream hop
`-map 0:v -f tee`	One encode, two muxers. The RTP copy costs no second encode.

Parameter sets repeat before every IDR

The RTSP muxer advertises AVFMT_GLOBALHEADER, which makes libx264 put SPS and PPS only in extradata (the SDP) and never in the stream. A browser that loses sync then has nothing in-band to re-bootstrap from and freezes on its last decoded frame while the PeerConnection still reads connected. Every encode arm repeats the parameter sets in-band per IDR to close that: dump_extra=freq=keyframe on libx264 and h264_v4l2m2m, --inline on rpicam-vid, h264parse config-interval=1 on the GStreamer arms plus mpph264enc header-mode=1 at the encoder. The ground ingest re-applies the same bitstream filter, which is a no-op against a current drone and a repair against an older one.

There is no keyframe-on-demand, by design

The GOP is the recovery mechanism. A decoder that loses sync re-bootstraps at the next IDR, which is at most 500 ms away at 30 fps with -g 15. There is no PLI or keyframe-request path. The shipped encoder leaves (ffmpeg CLI, rpicam-vid, gst-launch-1.0) expose no runtime control channel an IDR could be requested through, and MediaMTX does not forward a WebRTC receiver’s PLI to an RTSP publisher. The most a request could save is the 0 to 500 ms a decoder waits, so about 250 ms expected, on an event that only happens after loss the FEC did not recover. Respawning the encoder to force an IDR costs a 1 to 2 second blackout, which is strictly worse than waiting.

H.264 is the default codec

H.264 is the shipped wire codec. H.265 saves bandwidth at the same quality, and video.camera.codec accepts it, but browser WebRTC support for H.265 is not consistent enough to ship as a default and Mission Control’s player pins avc1.640029. Test your playback path before switching.

Stage 3: WFB-ng transport

WFB-ng puts an RTL8812EU adapter in monitor mode and injects FEC-encoded packets on 5 GHz. This is not standard WiFi: no association, no handshake, no retransmission, no CSMA/CA backoff.

Property	Value
Radio mode	802.11 monitor mode (injection)
Frequency	5 GHz
Channel width	20 MHz
FEC	Reed-Solomon, `k=8` data of `n=12` total by default
Encryption	WFB-ng key exchange
Adaptive MCS	Capped at MCS 3 (`video.wfb.adaptive_mcs_max`, hard ceiling 5)

The framing between the encoder and the radio is RTP (RFC 6184), payload type 96, SSRC 0xCAFE, pkt_size 1316, on UDP 5600. RTP is the contract, not a detail: a lost RTP packet costs at most one NAL fragment, where a lost chunk of a raw byte stream corrupts everything up to the next start code. The receiver depayloads against the SDP in its MediaMTX path config (rtpSDP, shown in Stage 4).

On CSI boards the encoder is rpicam-vid or gst-launch-1.0 publishing through a shell pipeline, and there the RTP copy is still produced by a separate tap process with its own watchdog. Check which encoder is running (the encoder field in camera-state.json, or the Video row of ados status) before reasoning about hop 4.

The MCS ladder cap is a statement of what has been measured, not of what the radio can do. The rung table is arithmetic from standard 802.11n 20 MHz required-SNR figures, and nothing in the tree has measured what the vendored rtl8812eu driver actually applies. Raising the cap without a range sweep is how a link that looks perfect on a bench dies at 150 m.

Stage 4: Ground receive and serve

wfb_rx -p 0 decodes the radio packets and FEC-recovers gaps, then the fan-out forwards them to the ground MediaMTX on UDP 5600 and to the on-box display on 5605. No ffmpeg sits in between. ados-groundlink --ground-video writes a MediaMTX config whose main path has a native UDP RTP source and the SDP that describes it:

paths:
  main:
    source: udp+rtp://127.0.0.1:5600
    rtpSDP: |
      m=video 5600 RTP/AVP 96
      a=rtpmap:96 H264/90000

The same process supervises MediaMTX. It restarts MediaMTX 3 seconds after any exit, with no attempt cap. A watchdog polls every 2 seconds and restarts MediaMTX when its inbound byte count stalls for 12 seconds while the radio statistics (fresh within 10 seconds) show packets still arriving.

Endpoint	URL	Protocol
WHEP	`http://<ground-node>:8080/whep`	WebRTC (H.264 RTP), relayed by `ados-control` to MediaMTX on loopback
RTSP	`rtsp://<ground-node>:8554/main`	RTSP, for tools like VLC

MediaMTX does no transcoding. The H.264 stream passes through to WebRTC as it arrived. Both MediaMTX instances are rendered from one config generator, so the air and ground documents cannot drift apart silently. The file is written to /etc/ados/mediamtx.yml.

Do not hand-edit /etc/ados/mediamtx.yml. MediaMTX rejects an unrecognised key outright, so a typo does not degrade the node, it leaves the node with no media server at all. Change video.* in /etc/ados/config.yaml and let the agent regenerate the file.

WebRTC and ICE

The relevant MediaMTX keys, with the values the LAN profiles get:

webrtcLocalUDPAddress: ":8189"
webrtcLocalTCPAddress: ""          # LAN profiles offer no TCP ICE candidate
webrtcIPsFromInterfaces: true
webrtcIPsFromInterfacesList: []
webrtcHandshakeTimeout: 10s
webrtcSTUNGatherTimeout: 2s
webrtcICEServers2: []              # LAN profiles gather no STUN

UDP ICE is pinned to a single port, 8189, so a firewall rule is one line. STUN is configured only on the cloud-relay profile. On a LAN session the browser connects on host candidates, so every STUN entry would be pure handshake latency before the first frame. The relay profile carries five STUN servers and a TCP ICE candidate on 8189, because a relay can face a network that blocks UDP.

Stage 5: Browser decode

The browser receives H.264 RTP over WebRTC and decodes with the platform hardware decoder. Mission Control’s src/lib/video/webrtc-client.ts handles WHEP negotiation, ICE gathering, track attachment and connection health. Video reaches a browser either from the ground station over the radio hop above, or over the cloud relay described below.

Video transport modes

Mission Control’s transport switcher offers four modes:

Mode	How it works	When to use
`auto`	Tries LAN WHEP first, falls back to MQTT-signalled WebRTC	Default
`lan-whep`	WebRTC WHEP straight to the ground station’s LAN address	WiFi AP, USB tether, Ethernet
`p2p-mqtt`	WebRTC with the SDP exchange relayed over MQTT	Remote access with no port forwarding
`off`	No video	Telemetry-only monitoring

Under p2p-mqtt the browser and the agent’s ados-cloud service exchange SDP over MQTT:

ados/{deviceId}/webrtc/offer                  browser publishes {"sessionId", "sdp"}, or {"sessionId", "close": true}
ados/{deviceId}/webrtc/answer/{sessionId}     agent publishes the SDP answer, or an error document

On an offer, the agent posts the SDP to its local WHEP endpoint (http://localhost:8889/main/whep) and publishes MediaMTX’s answer on that session’s own answer topic, so concurrent viewers never see each other’s SDP. A close message, or the browser’s MQTT last will when it vanishes, ends the WHEP session. Once the exchange completes the media flows peer-to-peer through STUN-negotiated NAT traversal; the broker carries signalling only, never video. The full topic list is on Cloud infrastructure.

p2p-mqtt needs internet access on both sides. Behind symmetric NAT, which some cellular carriers use, ICE negotiation can fail outright. The transport switcher reports the failing stage.

What is measured, and what is not

One latency number in this stack has been measured, and it is not the number an operator means by “latency”. Publish-side, stamp to readback, sim bench, 2026-09-15. The SEI probe injects a timestamp after the encoder and reads it back from the drone’s own rtsp://localhost:8554/main.

Figure	Value
Mean	74 ms
Range across samples	61 to 152 ms
Samples	6
Bench	1280x720 at 30 fps, 4000 kbps synthetic RTSP source, agent binaries as installed

That number covers four things: the publish mux, the RTSP publish, MediaMTX on the drone, and the readback demux. It excludes capture, encode, the radio hop, the ground receive, the ground MediaMTX and the browser. Those are most of what an operator experiences. There is no glass-to-glass figure for ADOS, and this is not one. Do not read the 74 ms as end to end, and do not add per-stage guesses to it to manufacture a total. A real air-to-ground figure needs the same probe run on the ground node against its own rtsp://127.0.0.1:8554/main with clock-aligned nodes. A real glass-to-glass figure needs a camera-side marker and a photograph of the viewer. Neither has been done. Two things to know before quoting any reading from the probe:

It is off by default and gated on video.wfb.sei_latency, because it splices a Python stage into the byte path. Bench profiles only.
/api/video/latency reads the tap’s sidecar file with no freshness gate, so a stale reading can look live. Confirm an injector is actually in the byte path before trusting a number.

For per-hop health rather than latency, ados diag video samples the reliable cumulative counters at each hop twice and attributes a stall to the hop that stopped. That is the supported first move on a dark feed. See Diagnostics.

Recording

Recording happens inside MediaMTX, writing fMP4 segments with no extra encode pass. It is off by default; video.recording.enabled turns it on and video.recording.retention bounds how long segments are kept. The segment set on disk is the pre-roll ring, and clips are cut out of it after the fact with ados record clip. See Video pipeline.

Next steps

Video pipeline

Configure the camera, encoder and recording on a node.

Diagnostics

Run the per-hop video verdict with ados diag video.

Cloud infrastructure

The MQTT topics and the Convex status path.

Architecture

Integrations

Contribute to the code

The hop map

Stage 1: Capture

Stage 2: Encode

Parameter sets repeat before every IDR

There is no keyframe-on-demand, by design

H.264 is the default codec

Stage 3: WFB-ng transport

Stage 4: Ground receive and serve

WebRTC and ICE

Stage 5: Browser decode

Video transport modes

What is measured, and what is not

Recording

Next steps

Video pipeline

Diagnostics

Cloud infrastructure

​The hop map

​Stage 1: Capture

​Stage 2: Encode

​Parameter sets repeat before every IDR

​There is no keyframe-on-demand, by design

​H.264 is the default codec

​Stage 3: WFB-ng transport

​Stage 4: Ground receive and serve

​WebRTC and ICE

​Stage 5: Browser decode

​Video transport modes

​What is measured, and what is not

​Recording

​Next steps

Video pipeline

Diagnostics

Cloud infrastructure

The hop map

Stage 1: Capture

Stage 2: Encode

Parameter sets repeat before every IDR

There is no keyframe-on-demand, by design

H.264 is the default codec

Stage 3: WFB-ng transport

Stage 4: Ground receive and serve

WebRTC and ICE

Stage 5: Browser decode

Video transport modes

What is measured, and what is not

Recording

Next steps