> ## Documentation Index
> Fetch the complete documentation index at: https://docs.altnautica.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Systemd Services

> All service units, dependencies, startup order, resource limits, and log access.

# Systemd Services

The agent runs as a collection of systemd services. Each service handles one responsibility: MAVLink, video, cloud, health, and so on. The supervisor (`ados-supervisor`, a native Rust orchestrator) manages the lifecycle of all child services through `systemctl`.

The unit names are a stable interface: you type the same `systemctl restart ados-<name>` commands regardless of which language a unit runs. Most units now exec a native Rust binary; a few stay Python by design (AI and vision, hardware-detection and bootstrap glue, the residual API, and some ground-station hardware managers). The "Runs" and "Lang" columns below show which is which.

This architecture gives you per-service restart, resource isolation via cgroups, and clean dependency ordering.

## Service list

### Core services (all profiles)

| Service            | Runs                   | Lang   | Description                                              |
| ------------------ | ---------------------- | ------ | -------------------------------------------------------- |
| `ados-supervisor`  | supervisor             | Rust   | Parent orchestrator, starts and monitors children        |
| `ados-mavlink`     | `ados-mavlink-router`  | Rust   | FC serial connection and MAVLink router (WS on `:8765`)  |
| `ados-control`     | `ados-control`         | Rust   | HTTP front on port 8080 (proxies the residual API)       |
| `ados-api`         | `ados.services.api`    | Python | Residual FastAPI behind the front, on an internal socket |
| `ados-cloud`       | `ados-cloud`           | Rust   | MQTT and Convex cloud relay                              |
| `ados-logd`        | `ados-logd`            | Rust   | Black-box logging store and query API                    |
| `ados-health`      | `ados.services.health` | Python | CPU, RAM, disk, temperature monitoring                   |
| `ados-discovery`   | mDNS                   | Python | mDNS advertisement (`_ados._tcp.local.`)                 |
| `ados-plugin-host` | plugin host            | Rust   | Plugin subprocess supervisor                             |
| `ados-peripherals` | peripherals manager    | Python | USB hot-plug, sensor manager                             |

### Drone profile services

| Service       | Runs          | Lang | Description                                          |
| ------------- | ------------- | ---- | ---------------------------------------------------- |
| `ados-video`  | `ados-video`  | Rust | Camera detection, encoding, MediaMTX                 |
| `ados-wfb`    | `ados-radio`  | Rust | WFB-ng video transmitter                             |
| `ados-vision` | `ados-vision` | Rust | Vision host (on capable boards)                      |
| `ados-gpio`   | `ados-gpio`   | Rust | GPIO output substrate (ships disabled until enabled) |

### Ground station services (ground\_station profile only)

| Service                                | Runs                                 | Lang   | Description                                 |
| -------------------------------------- | ------------------------------------ | ------ | ------------------------------------------- |
| `ados-wfb-rx`                          | `ados-groundlink`                    | Rust   | WFB-ng video receiver (relay/receiver base) |
| `ados-oled`                            | `ados-display`                       | Rust   | OLED/LCD display UI                         |
| `ados-display-probe`                   | `ados-display-probe`                 | Rust   | Display panel detection                     |
| `ados-pic`                             | `ados-pic`                           | Rust   | Panel-in-charge input arbiter               |
| `ados-input`                           | `ados-input`                         | Rust   | Bluetooth and USB gamepad input manager     |
| `ados-buttons`                         | `ados.services.ui.button_service`    | Python | GPIO button handler                         |
| `ados-uplink-router`                   | `ados-net`                           | Rust   | Internet uplink priority and data cap       |
| `ados-ethernet`                        | `ground_station.ethernet_manager`    | Python | Ethernet connection manager                 |
| `ados-wifi-client`                     | `ground_station.wifi_client_manager` | Python | WiFi client connection manager              |
| `ados-modem`                           | `ground_station.modem_manager`       | Python | Cellular modem manager                      |
| `ados-hostapd`                         | `ground_station.hostapd_manager`     | Python | WiFi hotspot (manages hostapd)              |
| `ados-mediamtx-gs`                     | `ground_station.mediamtx_manager`    | Python | MediaMTX for ground-side video relay        |
| `ados-kiosk`                           | kiosk service                        | Python | HDMI kiosk mode (Chromium + cage Wayland)   |
| `ados-dnsmasq-gs`                      | dnsmasq                              | C      | DHCP and DNS for the hotspot                |
| `ados-setup-captive`                   | captive DNS                          | Python | Captive portal DNS responder                |
| `ados-usb-gadget`                      | USB gadget                           | Python | USB CDC-NCM/RNDIS gadget for laptop tether  |
| `ados-usb-gadget-setup`                | gadget setup                         | shell  | One-shot USB gadget configuration           |
| `ados-batman`                          | `ground_station.mesh_manager`        | Python | batman-adv mesh carrier (gated)             |
| `ados-wfb-relay` / `ados-wfb-receiver` | `ados-groundlink`                    | Rust   | Mesh relay / receiver roles (gated)         |
| `ados-mesh-pairing`                    | pairing daemon                       | Python | Mesh tap-to-pair daemon (gated)             |

## Startup order

The supervisor starts first and brings up child services in dependency order:

```
ados-supervisor
  │
  ├── ados-mavlink        (starts immediately)
  ├── ados-control        (starts immediately, owns :8080)
  ├── ados-api            (starts immediately, behind the front)
  ├── ados-health         (starts immediately)
  ├── ados-logd           (starts immediately)
  ├── ados-discovery      (starts immediately)
  │
  ├── ados-video          (after camera detected)
  ├── ados-wfb            (after ados-video)
  ├── ados-cloud          (after ados-control)
  ├── ados-plugin-host    (after ados-control)
  └── ados-peripherals    (after ados-control)
```

Hardware-dependent services (video, wfb, modem) only start when the relevant hardware is detected.

## Resource limits

Each service runs with cgroup resource limits to prevent any single service from consuming all system resources:

```ini theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Example from ados-mavlink.service
[Service]
MemoryMax=50M
CPUQuota=25%
```

Memory caps configured on a representative drone profile:

| Service      | MemoryMax | Typical usage |
| ------------ | --------- | ------------- |
| ados-mavlink | 50M       | \~20 MB       |
| ados-control | 128M      | \~25 MB       |
| ados-api     | 128M      | \~40 MB       |
| ados-video   | 512M      | \~100-200 MB  |
| ados-cloud   | 256M      | \~30 MB       |
| ados-health  | 20M       | \~15 MB       |

This leaves headroom for the OS, ffmpeg, and MediaMTX.

## Circuit breaker

The supervisor implements a circuit breaker: if a service fails 5 times within 60 seconds, the circuit breaker opens and stops restarting it. This prevents crash loops from consuming all system resources.

The circuit breaker state is visible through the REST API:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl http://localhost:8080/api/services \
  -H "X-ADOS-Key: $KEY"
```

## Managing services

### Check service status

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# All services
sudo systemctl list-units "ados-*" --all

# Specific service
sudo systemctl status ados-video
```

### Restart a service

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sudo systemctl restart ados-video
```

### View service logs

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Last 50 lines from a specific service
sudo journalctl -u ados-video -n 50

# Follow live
sudo journalctl -u ados-video -f

# All agent logs combined
sudo journalctl -u "ados-*" -f

# Since a specific time
sudo journalctl -u ados-mavlink --since "10 minutes ago"
```

### Stop everything

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sudo systemctl stop ados-supervisor
```

This stops the supervisor and all child services.

### Start everything

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sudo systemctl start ados-supervisor
```

## Service dependencies

Ground station services have explicit dependency ordering:

```
ados-kiosk
  └── After=ados-mediamtx-gs.service
       └── After=ados-wfb-rx.service

ados-uplink-router
  └── After=ados-hostapd.service
       └── After=ados-dnsmasq-gs.service
```

This ensures the WiFi hotspot is up before the uplink router starts, and the video relay is ready before the kiosk opens the HUD page.

## Automatic restart

All services are configured with automatic restart on failure:

```ini theme={"theme":{"light":"github-light","dark":"github-dark"}}
[Service]
Restart=on-failure
RestartSec=3
```

The supervisor detects when a child service restarts and logs the event. Combined with the circuit breaker, this means transient failures self-heal while persistent failures are contained.

## Viewing all service files

The systemd unit files are installed at:

```
/etc/systemd/system/ados-*.service
```

To see the contents of a specific service file:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat /etc/systemd/system/ados-video.service
```

<Note>
  Do not edit the service files directly. They are overwritten on each upgrade. If you need to customize a service (e.g., add environment variables or change resource limits), use a systemd override:

  ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
  sudo systemctl edit ados-video
  ```

  This creates a drop-in override file that persists across upgrades.
</Note>