Canary

A SMART disk health monitor that remembers what it saw last time.

Beszel and Uptime Kuma watch hosts, containers, and services — but neither one looks at the physical drives underneath them. Canary closes that gap: it scans every drive on every configured host via smartctl, classifies each one, and tracks changes across runs so a slowly-growing reallocated-sector count gets flagged as the active problem it is — long before it becomes a degraded ZFS pool at 2 AM.

Note

The whole point is the trend, not the snapshot. A drive with two reallocated sectors that's had two reallocated sectors for three years is normal wear. A drive that had zero last month and has three today is the actual early-warning signal — and that's the one Canary escalates to Critical.

Why this instead of a Scrutiny container

Both work from the same smartctl data. Canary is a few hundred lines of Python that writes a Markdown report into the vault (same convention as Crandle's HardwareSurvey.md) and fires a desktop notification — no extra container, database, or web UI to maintain for a check that runs once a month. smartmontools was already on the package list waiting for exactly this.

Setup

cd ~/Documents/Projects/canary
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

cp hosts.example.yml hosts.yml
# Edit hosts.yml — add the hosts you want scanned

hosts.yml is gitignored — it never leaves your machine.

Configuration

Edit hosts.yml:

hosts:
  - name: Trevor-PC
    sudo: true              # raw block devices need root on Linux

  - name: Proxmox
    host: 192.168.0.70
    user: root
    sudo: false             # already root — no sudo needed

  - name: Plex
    host: 192.168.0.53
    user: admin
    sudo: true
    devices:                # skip auto-discovery, scan exactly these
      - path: /dev/sda
        type: scsi

Field	Required	Description
name	✓	Display name — also the trend-matching key alongside the drive's serial number
host		IP/hostname for SSH; omit to scan the local machine directly
user		SSH username
port		SSH port (default 22)
sudo		Set true if the connecting user can't read raw block devices directly (the Debian/Arch default — /dev/sdX is root:disk 0660). You'll be prompted once per host per run; the password is piped straight into sudo -S over the existing channel, never written to disk — same approach plex-dupe-scanner uses.
devices		Explicit [{path, type}, ...] list — bypasses smartctl --scan auto-discovery. Useful when a HBA/RAID controller hides drives from the scan, or you only want to track a subset

SSH authenticates via key first, falling back to a cached password prompt (shared across hosts using the same user, like Crandle).

Tip

For the monthly systemd timer to run unattended, every host needs either key-based SSH + no sudo requirement (e.g. root on Proxmox), or a narrow passwordless-sudo rule scoped to smartctl:

# /etc/sudoers.d/canary — visudo -f to create
admin ALL=(root) NOPASSWD: /usr/bin/smartctl

Hosts that still need an interactive sudo password will simply hang the timer waiting on getpass — run those manually instead, or add the rule.

Usage

.venv/bin/python canary.py [options]

Flag	Description
(none)	Scan all hosts, print results, save a timestamped archive
--master	Overwrite DiskHealth.md, archive, and update the trend baseline
--master --diff	Show what changed vs. the last master before overwriting
--diff	Show what changed vs. the last master, without overwriting it
--no-report	Terminal output only — write nothing, don't touch the baseline
--notify	Send a desktop notification summary (used by the systemd timer)
--host NAME	Scan only hosts whose name contains NAME (substring match)
--dry-run	Check connectivity to every configured host, no scan
--config FILE	Use a config file other than hosts.yml

Important

The trend baseline only moves on --master runs. Run plain canary.py as often as you like to poke at things — it never shifts what "since last time" means. That's reserved for the runs you trust enough to publish.

Example output

                              Canary — Disk Health
┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Host    ┃ Device  ┃ Model        ┃Capacity ┃Temp ┃Power-On┃Life Lf ┃ Status   ┃ Notes                            ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Proxmox │ /dev/sdb│ WD Red 4TB   │  4.0 TB │ 34°C│ 28,400h│    96% │ Healthy  │ all monitored attributes nominal │
│ Proxmox │ /dev/sda│ Samsung 870  │  2.0 TB │ 41°C│ 19,200h│    91% │ Critical │ reallocated sectors grew by 3    │
│         │         │              │         │     │        │        │          │ since last check (2 → 5)         │
│ Trevor-PC│/dev/nvme0│Samsung 960 │  1.0 TB │ 38°C│  5,000h│    96% │ Healthy  │ all monitored attributes nominal │
└─────────┴─────────┴──────────────┴─────────┴─────┴────────┴────────┴──────────┴──────────────────────────────────┘

Healthy: 2  ·  Critical: 1

What it checks

Signal	Drive types	Escalates to
Overall SMART self-assessment (PASSED/FAILED)	All	Critical (immediately, short-circuits everything else)
NVMe critical warning flag	NVMe	Critical
Reallocated / pending / uncorrectable sectors, CRC errors, media errors	ATA/SCSI + NVMe	Watch if stable and nonzero, Critical if grown since the last master run
Rated life remaining (SSD_Life_Left / Percent_Lifetime_Remain / Media_Wearout_Indicator / NVMe percentage_used)	SSD + NVMe	Warning under 20%, Critical under 10%
NVMe available spare vs. its threshold	NVMe	Warning within 10 points of threshold, Critical at or below it
Temperature	All	Warning at 50°C, Critical at 60°C

Thresholds live in health.py (DEFAULT_THRESHOLDS) if your drives' normal operating ranges differ.

Reports

Timestamped archives are written to ~/Documents/Notes/Ventoz/Reference/Disk Health/ (last 6 kept, older ones pruned automatically). The master DiskHealth.md stays at the top of Reference/, mirroring HardwareSurvey.md.

Automation

Systemd user units live in systemd-units/ here — copy them into the dotfiles repo as a canary/ stow package (same layout as crandle/) and install:

# From ~/dotfiles, after adding canary/.config/systemd/user/{canary.service,canary.timer}
stow canary
systemctl --user daemon-reload
systemctl --user enable --now canary.timer

# Verify
systemctl --user list-timers canary.timer

Runs canary.py --master --notify at 03:00 on the 1st of each month — firing a notify-send summary through dunst either way, so "all clear" and "go check this" both show up without opening a terminal.

Tests

.venv/bin/python -m unittest test_canary -v

33 tests covering SMART JSON parsing (ATA, SCSI, NVMe, malformed payloads), health classification and threshold bands, and — the part that actually matters — trend detection: stable vs. growing counters, first-sight handling, and the history snapshot/diff round trip. All built on hand-written fixtures, because the entire premise ("flag the counter that grew since last month") is something you can't wait for a real failing drive to demonstrate.

Project structure

canary.py             # CLI entry point — arg parsing, orchestration, reports, notifications
collector.py          # Host connections — local + SSH (paramiko), smartctl invocation, sudo handling
smart_parser.py       # smartctl JSON → normalized DriveHealth records (ATA/SCSI + NVMe)
health.py             # Threshold-based classification — Status, Verdict, trend evaluation
history.py            # Snapshot persistence — the trend baseline that survives between runs
report.py             # Rich terminal table, Markdown report, diff rendering
hosts.example.yml     # Config template (hosts.yml is gitignored)
systemd-units/        # canary.service + canary.timer for monthly automation
test_canary.py        # 33 tests on synthetic SMART fixtures