Skip to main content

​
Black Box Log Store

The Black Box is one WAL-mode SQLite database on the node that holds logs from every agent process plus telemetry history, discrete events and hardware samples. It survives a reboot and it answers when the network is down, which is exactly when journalctl over SSH is least useful. ados-logd is the sole writer. Every other reader opens it read-only.
The store ships off. logging.store.enabled defaults to false and the installer masks the unit while it is off. This is a deliberate capability regression, not an oversight: measured on a drone, the node wrote 904 KB/s with the store running and 49 KB/s with it stopped, so the store accounted for roughly 96% of everything reaching the card. Cards were filling and corrupting.While the store is off the node has no durable flight recorder and journalctl is the log of record. That is why the systemd journal is kept Storage=persistent: with the store gone it is the only thing that survives a reboot.

​
Turn it on

One config key plus a reconcile. Never a reinstall from scratch: the binary is fetched and placed either way.
Then re-run the installer so the unit is unmasked and enabled:
The reconcile is idempotent and runs on every install, so a partial state heals itself. With the key false (or absent, or malformed) the unit is stopped, disabled and masked: disable alone only drops the wants-symlinks, and the supervisor would pull the unit back in as a dependency. There is no marker that forces the store on. A second way to enable something is a second thing to check when a node misbehaves, and this one has a measured cost that makes “why is this on?” worth answering from the config alone. Confirm the posture from the storage diagnostic, which reports off rather than treating an absent store as a fault:
With the store off that prints log store disabled (journal is the record).

​
What it holds

Four record kinds, and a session model over the top. Sessions group rows into a window you can name. Three kinds: boot, flight, manual.

​
Paths and ports

Both socket paths resolve under ADOS_RUN_DIR when it is set, defaulting to /run/ados. The operator account is put in the socket’s group at install time, so on-box analysis works without sudo.

​
Query it

Every subcommand takes the same three transport flags: --host, --key and --json.
--json is the stable contract. The colorized table is the human default and is explicitly not a contract, so script against --json.

​
ados logs query

Keyset-paginated rows across one of the four tables.

​
ados logs tail

Follow new rows until interrupted. Served over SSE from /v1/tail.
tail takes the same --kind, --source, --metric, --event-kind, --level and --text filters as query, plus --replay <n>.

​
ados logs aggregate

Downsampled series for charts.
--metric is required and repeatable. --bucket is auto, 1s, 1m or 1h. --agg is avg, min, max, p50, p95, last or count.

​
ados logs status

Store health, ingest and drop rates, and the sync watermark.

​
Get data off the box

​
ados logs export

Stream a window to a file or stdout.
--format is jsonl or jsonl.zst. The same --since / --from / --to / --kind / --source / --metric / --session filters apply, plus --host and --key for an off-box export.

​
ados logs push

Export a window to the paired cloud account. This is a thin front door: it records the request and the cloud service does the export, upload and mark.
The cloud service refuses the push when the agent is in local mode, is not cloud-paired, or has cloud log push disabled. That is the correct state for an agent with nothing to sync, not an error to work around. The request file lives under the root-owned runtime dir (/run/ados/logd-push-request.json, with the outcome at logd-push-result.json). Root records it directly; a non-root operator hands the write to the running agent over loopback via POST /api/logs/push, and only falls back to the direct seam if the agent is unreachable.

​
The query API

A separate listener on its own port, dialled directly by Mission Control’s direct log tier and by ados logs. It is not reachable through either HTTP front. The same key works against :8090 as against :8080, so a tool that already holds the pairing key needs no second credential.

​
Supervision

ados-logd is registered with the supervisor so it gets the liveness check, the auto-restart, the parked retry and, most importantly, the systemctl reset-failed that clears a failed (start-limit-hit) latch. Five restarts in a minute (a store that re-quarantines, a full /var, a writer wedged against the unit’s WatchdogSec) used to leave the store permanently dead with nothing in-process able to clear it: ados logs query and ados logs tail returned nothing and every producer’s log layer dropped silently. The store is in the lean headless keep set. It is Rust, so it holds no Python dependency, and a lean flight node is exactly the node an operator cannot reach, which makes it the one that most needs its own recorder.