Skip to main content

​
Diagnostics with ados diag

ados diag is the first thing to run on a misbehaving node. It is three thin renderers over the agent’s own REST surface, so the CLI shows exactly what Mission Control sees.
Every subcommand takes --json for scripted use.
ados diag does not appear in ados help. The advanced groups are hidden so the common path stays uncluttered; ados diag is a working diagnostic group rather than a first-run command, and it is fully supported.

​
Run the node’s own diagnostics first

A wrong diagnostic conclusion on a flight system is worse than no conclusion, so the rule is: reach for the agent’s own verdicts before hand-probing a pipeline. The reason is measurement quality. ados diag video reads only reliable cumulative and rate counters on canonical objects (the mediamtx main path, the wfb-stats sidecar, the WHEP endpoint) and samples each of them twice over a 2 second window, so a hop is judged by data actually moving between the two reads rather than by a process being alive.

​
Signals that are not primary evidence

The link verdict solves the same problem differently: channel_locked comes from the radio’s own rx_proven, and rf_unverified is its explicit “transmitting with zero confirmed reception” state. Both are derived from decoded peer traffic, frames that passed error correction and decryption, which is why they can prove delivery when a byte counter cannot. The rf_unverified condition must hold continuously for 10 seconds before the agent records an entry event, so a single missed beacon does not flap it.

​
ados diag video

The response envelope carries profile, role, canonical_path (always main), window_s, the hops array and video_dies_at. The hop set is profile-aware, because a drone is the video source and a ground station is the sink.

​
Drone hops

​
Ground-station hops

​
Verdicts

video_dies_at names the first real break. When every hop is flowing the renderer prints all hops flowing; when no hops come back at all it says so rather than inventing a verdict.
Reads /api/wfb. Two things are reported side by side and deliberately not merged. The decode verdict (link_diag) answers “can this radio decode its peer at all”: The pairing state is the lifecycle record. The two legitimately disagree: a link decoding hundreds of packets a second can sit behind a lifecycle record nothing has refreshed. Printed bare, “HEALTHY (state=stale)” reads as a surface contradicting itself, so the renderer labels which word is answering which question.

​
Packet loss and delivered frames

link_diag is not a delivery verdict. A link that decodes fine while dropping a fifth of the stream still answers healthy to “can you decode”, and that is true and beside the point. So the diagnostic leads with LOSSY once measured loss reaches 2%. A video frame needs every one of its packets, so loss compounds across the frame. At roughly 10 RTP packets per frame: That is why “2% packet loss” and “a frozen picture” describe the same link. The figure is an estimate under an independent-loss assumption and is always printed with the assumption stated; bursty loss is kinder to it.

​
Who measured the loss

A transmit-only drone has no local loss figure and never will: it injects its own downlink and a single radio in monitor mode cannot capture its own frames, so packets_received is a permanent zero. The measurement does exist, because the receiving ground station counts exactly what arrived and sends it back up the link, so the diagnostic prefers the radio’s resolved sample and labels its provenance: A number with invisible provenance is a number an operator cannot act on: “3% loss” means something different when this node’s own receiver counted it than when the far end did. And when nobody measured, the diagnostic says which measurements are missing rather than printing a bare dash next to a clean-looking link. A measured zero and an unmeasured link are different facts and are reported differently.

​
Counters

The readout carries RSSI, Channel, Decoded pkt/s, RF frames (all), Decrypt errors, Bad packets, Packets lost, FEC recovered, FEC failed, Loss and Bitrate. The three FEC and loss rows are what decide whether video actually arrives.

​
ados diag storage

This is the first check on a node that reboots on its own or has stopped booting reliably. Space, not wear, is what usually breaks these boxes: the card fills, a rewrite cannot get its scratch, a write tears, and the node will not come back. So the footprint is printed first and the wear verdict second.

​
Write rate and its provenance

The write rate is reported with a window and a source, because five seconds of kernel counter and hours of stored history answer different questions: The direct reading needs nothing but the kernel, so it is the one that still works with the Black Box store off; the stored one reaches much further back. Neither is “the” rate on its own. An unmeasured rate is never printed as a zero: the reason it is missing is printed instead.

​
Power and thermal

On a board that reports it, the throttle bitfield’s sticky “has occurred” bits are surfaced: undervoltage, frequency capped, throttled, soft temp limit. These are recorded at some point in the retained window, not necessarily now, which is exactly what makes them useful for a node that misbehaves intermittently. A board that does not report them says so.

​
Store footprint

The log store’s own live and WAL sizes are reported, along with any quarantined database. With the store off, the row reads log store disabled (journal is the record) rather than 0 B live, because “a store that exists and is empty” is a different and more alarming thing than “a store nobody asked to run”.

​
When diag is not enough

Only after the agent’s own verdicts have named a hop should you reach for anything else, and then with the blocklist above in mind. Measure cumulative counters as a delta over a 2 to 6 second window, confirm from the source which object is canonical before you probe it, and cross-check each hop with two independent reliable methods before changing a conclusion.