Drone Agent troubleshooting

Start with the agent’s own checks. They read the same counters Mission Control does and name the broken hop, so you know which section below applies.

First step: run the agent’s checks

ados                  # status page: services, MAVLink, video, network, next action
ados diag video       # which hop of the video pipeline stopped
ados diag link        # can the radio decode its peer, and how much is lost
ados diag storage     # disk space, write rate and throttle history

ados diag video answers “where does video die”: it samples each hop twice over 2 s and names the first stalled one.
ados diag link answers “is the radio link working”: healthy, searching, deaf, mis_keyed or jammed, plus measured packet loss.
ados diag storage answers “is the card full or wearing”: space first, then write rate and power or thermal throttling.

Add --json to any of them, or run ados status --json, for a support ticket or a script. See Drone Agent diagnostics for every verdict.

Read the logs

The Black Box log store is off by default, so the systemd journal is the log of record:

sudo journalctl -u ados-supervisor -n 100
sudo journalctl -u ados-video --since "10 min ago"
sudo journalctl -u 'ados-*' -p warning --since "1 hour ago"

With the Black Box store turned on, use ados logs query and ados logs tail instead.

Flight controller not detected

Symptoms: ados shows MAVLink not connected. No telemetry in Mission Control.

Check the serial connection

Verify the UART cable between the FC and the companion computer. TX goes to RX and RX to TX. Check voltage levels: most FCs use 3.3 V UART; a 5 V TTL port can damage a 3.3 V SBC input.List the serial ports:

ls /dev/tty{S,AMA,ACM,USB}*

An FC on USB usually appears as /dev/ttyACM0 or /dev/ttyUSB0.

Check baud rate

The agent sweeps common bauds automatically. To force one, set both mavlink.serial_port and mavlink.baud_rate. See What baud your drone is actually running.

Check for serial port conflicts

Another process may hold the port:

sudo fuser /dev/ttyS0

Stop that process, or point the agent at a different port.

Check the MAVLink protocol setting

The agent accepts MAVLink 1 and 2. If the port shows garbled data, confirm the FC’s SERIALn_PROTOCOL for that port is set to MAVLink.

FC not sending heartbeats

The agent waits for heartbeats to confirm the FC is alive. If none arrive, check that the FC is powered, that MAVLink is enabled on the port you wired, and that the FC is not in a boot loop (watch its LEDs).

Video frozen or not starting

Symptoms: Black video in Mission Control. ados diag video reports a stalled hop.

Check if a camera is detected

ls /dev/video*
v4l2-ctl --list-devices

No /dev/video* device means the camera is not recognised. For USB cameras check lsusb. For CSI cameras, check that the ribbon cable is seated on both ends.

Check the video service

sudo systemctl status ados-video
sudo journalctl -u ados-video -n 50

Look for errors about the encoder, camera access or mediamtx.

Check mediamtx

curl -s http://localhost:9997/v3/paths/list

With the stream up you see a path named main. If mediamtx does not answer, check its log lines:

sudo journalctl -u ados-video --grep "mediamtx" -n 20

Check cgroup resource limits

The video service has a MemoryMax cgroup limit. An OOM kill shows in the journal:

sudo journalctl -u ados-video --grep "oom|killed|cgroup" -n 20

Check the memory events counter:

cat /sys/fs/cgroup/ados.slice/ados-video.service/memory.events

A rising max count means the service is hitting its limit.

Camera permission denied

The installer adds the agent user to the video group. After a manual install you may need:

sudo usermod -aG video $(whoami)

Then log out and back in.

Radio link not working

Symptoms: No video over WFB-ng. The ground station shows no signal. ados diag link reports deaf, mis_keyed or jammed.

Check the WiFi adapter

lsusb | grep -i realtek
ados radio adapters

You should see the RTL8812EU with an injection-capable verdict. If not, check the USB connection.

Check the kernel driver

lsmod | grep 8812eu
sudo dmesg | grep -i 8812eu

If the 8812eu module is missing, rebuild it:

sudo ados radio install-driver --force

Check channel match

Open the dashboard WFB Transmit page on the drone and the WFB Receive page on the ground station, or run ados radio status on both nodes. With auto-hop on, both sides return to the home channel (149 by default) when they lose each other; a different channel or reg_domain on the two rigs keeps them apart. See Link automation.

Check key match

The drone holds tx.key and the ground station rx.key, complementary halves of one key pair created by a bind. They are not identical files. Compare the fingerprint in ados radio pair status on both nodes instead.

ados radio pair status

Different fingerprints, or mis_keyed from ados diag link, mean the two rigs come from different binds. Re-bind them; see WFB-ng pairing.

Cloud disconnected

Symptoms: The drone does not appear in Mission Control over the cloud. ados shows cloud disconnected.

Check internet connectivity

ping -c 3 example.com
ping -c 3 mqtt.example.com   # your configured MQTT broker host

If both fail, the board has no internet. Check WiFi, Ethernet or cellular.

Check pairing

Run ados. If the agent is not paired with Mission Control, follow Pairing.

Check the MQTT connection

sudo journalctl -u ados-cloud -n 50

Common causes: DNS resolution failure (check /etc/resolv.conf), TLS certificate errors from clock skew, or an API key mismatch.

Check clock synchronization

TLS needs an accurate clock. If date is off by more than a few minutes, enable NTP:

sudo timedatectl set-ntp true

Check server mode

ados status --json

The agent is local-only by default. To use the cloud relay or a self-hosted server, open dashboard Settings → Cloud (see Agent dashboard: Settings) and choose the server mode.

Agent not starting

Symptoms: ados status reports that the agent is not running.

Check systemd

sudo systemctl status ados-supervisor
sudo journalctl -u ados-supervisor -n 50

Reinstall over a broken install

Re-run the installer with --force to clear its checkpoints and reinstall:

curl -sSL https://raw.githubusercontent.com/altnautica/ADOSDroneAgent/main/scripts/install.sh | sudo bash -s -- --force

See Install the Drone Agent for every installer flag.

Check disk space

df -h /
ados diag storage

A full disk stops the agent. Recordings are stored in per-path subdirectories of /var/ados/recordings. Delete segments older than an hour and trim the journal:

sudo find /var/ados/recordings -name '*.mp4' -mmin +60 -delete
sudo journalctl --vacuum-size=100M

Then run ados diag storage again to confirm the footprint dropped.

High CPU or temperature

Check which process is using resources

top -b -n 1 -o %CPU

If the video encoder process is near the top, the board may be encoding in software. Check that your board profile lists h264_enc under hw_video_codecs (see HAL and tiers).

Thermal throttling

cat /sys/class/thermal/thermal_zone0/temp
ados diag storage

Divide the first value by 1000 for degrees Celsius. Above 80 °C the SoC throttles; ados diag storage shows whether throttling has occurred in the retained window. Add a heatsink or fan, or reduce video resolution or frame rate.

High control latency

Symptoms: Manual control feels laggy. Mission Control shows a high or climbing controlRttMs on the drone card.

Check which path the GCS is using

controlRttMs is the round-trip time of the control link, measured against the agent’s GET /api/ping. A few tens of milliseconds is a healthy LAN link. Hundreds of milliseconds points to the MQTT relay or a busy network. Seconds means the GCS is on the cloud command-queue path, which suits monitoring but not hands-on flying.For low-latency control, connect over the LAN: serve Mission Control over plain HTTP (http://localhost:4000 or http://<lan-ip>:4000) and pair the agent by IP so the GCS uses the direct WebSocket path.

Confirm the agent is reachable directly

From the GCS machine:

curl -s -o /dev/null -w "%{time_total}\n" http://192.168.1.50:8080/api/ping

Well under a second confirms a healthy direct link. If this is slow or times out while the cloud path works, fix LAN reachability to get the fast path back.

Getting help

If these steps do not solve the problem:

Run ados support-bundle on the affected node before rebooting it. It collects status, configuration, diagnostics, log records and service journals into one redacted archive, and sends nothing anywhere.
Open an issue on GitHub and attach it.
Join the Discord community for real-time help.

See Support for what the bundle contains and what makes a report diagnosable.

Next steps

Drone Agent diagnostics

Every ados diag verdict and what it measures.

Troubleshooting by area

Ground station, mesh, extensions and self-hosting problems.

Get started

Set up and use

Link, video and flight systems

Reference

Diagnose

First step: run the agent’s checks

Read the logs

Flight controller not detected

Video frozen or not starting

Radio link not working

Cloud disconnected

Agent not starting

High CPU or temperature

High control latency

Getting help

Next steps

Drone Agent diagnostics

Troubleshooting by area

​First step: run the agent’s checks

​Read the logs

​Flight controller not detected

​Video frozen or not starting

​Radio link not working

​Cloud disconnected

​Agent not starting

​High CPU or temperature

​High control latency

​Getting help

​Next steps

Drone Agent diagnostics

Troubleshooting by area

First step: run the agent’s checks

Read the logs

Flight controller not detected

Video frozen or not starting

Radio link not working

Cloud disconnected

Agent not starting

High CPU or temperature

High control latency

Getting help

Next steps