The phone tunnel established fine (TLS peer-fingerprint OK) but no traffic
reached the LAN: ru-vps has policy FORWARD DROP and only per-service game
ACCEPTs. The site tunnel never needed a FORWARD rule because it carries
ru-vps's own traffic, not forwarded packets; the road-warrior instance is
the first forwarded path.
Add a `ufw route allow` (tun2 -> tun0, 10.80.0.0/30 -> 192.168.1.0/24). ufw
route rules survive `ufw reload`, unlike a raw `iptables -I FORWARD` which
would sit before the ufw chains and be flushed on reload. Return traffic is
covered by the global RELATED,ESTABLISHED accept in ufw-before-forward.
Verified on ru-vps: `ufw status` shows "192.168.1.0/24 on tun0 ALLOW FWD
10.80.0.0/30 on tun2"; make openvpn-phone idempotent.
Same gap exists in openvpn-laptop.yml (tun1) — still not deployed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbuZrUoevfBgCpf5DCF4DG
The Android client ships OpenVPN 2.7, which refuses a --secret config
("Options error: No tls-client or tls-server option"; 2.8 drops it
entirely). Rework the phone instance to a CA-less TLS point-to-point:
two self-signed EC keypairs generated on ru-vps (server.* / client.*),
each side pins the other by SHA256 with peer-fingerprint, data-ciphers
AES-256-GCM. Everything else unchanged (tcp/9444, tun2, 10.80.0.0/30,
NAT via tun0, systemd unit, shared homelab_vpn_client_routes).
Verified on ru-vps 2026-09-03: openvpn 2.6.19 starts clean ("Using
certificate fingerprint to verify peer"), listens on 9444, tun2 up,
make openvpn-phone idempotent (changed=0 on rerun), make lint green.
The regenerated ansible/generated/ada-phone.ovpn (self-contained
cert+key, gitignored) was handed to the operator.
openvpn-laptop.yml is left on static key — not deployed; needs the same
TLS treatment when someone actually uses it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbuZrUoevfBgCpf5DCF4DG
playbooks/openvpn-phone.yml (make openvpn-phone) stands up a separate
point-to-point static-key instance on ru-vps: tcp/9444, tun2, 10.80.0.0/30,
homelab-openvpn-phone unit, NAT 10.80.0.0/30 -> LAN via tun0. The client
profile (with the secret) lands in ansible/generated/ada-phone.ovpn
(gitignored). Android client: "OpenVPN for Android" (Arne Schwabe) — the
official OpenVPN Connect does not support static-key configs.
- homelab_vpn_client_routes in group_vars/all/main.yml: shared surgical
route list for both road-warrior profiles; not the whole /24, since the
phone's home network is almost certainly 192.168.1.0/24 too
- openvpn-laptop.yml reuses that list instead of its own literal copy
- both playbooks: local profile write moved from `become: false` to
`vars: {ansible_connection: local, ansible_become: false}` — the keyword
did not suppress the inherited ansible_become on delegate_to: localhost
Deployed and verified on ru-vps 2026-09-03: service active, tun2 up, ufw
9444/tcp, NAT rule present, make openvpn-phone idempotent (changed=0 on
rerun), 192.168.1.30:8082 reachable from ru-vps. openvpn-laptop.yml is
still not applied on the live host.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbuZrUoevfBgCpf5DCF4DG
playbooks/dashboard.yml deploys Homepage as a second compose stack on the
monitoring LXC (CT 155) next to Uptime Kuma and renders its config from
homelab_services: one tile per service, link to its UI, grouped by Proxmox
node. Adding a service to the registry is enough — no second service list.
- new registry consumer: playbooks/dashboard.yml + playbooks/templates/homepage-*.j2
- homelab_dashboard_* vars in group_vars/all/services.yml (top-level, like
homelab_reverse_proxy_*); image pinned by digest, floating tag needs an
explicit -e dashboard_allow_floating_tag=true
- bootstrap-dashboard-pve-token.yml: read-only homepage@pve!dashboard token
(PVEAuditor) for the Proxmox widget, secret in the root .env as DASHBOARD_PVE_*
- Makefile: dashboard, dry-dashboard, bootstrap-dashboard-token
- container binds the LAN address only (192.168.1.30:8082), not published via Caddy
- docs: architecture.md Monitoring section, plan.md active task, consumer lists
Deployed to CT 155 on 2026-09-03: container healthy, http://192.168.1.30:8082/
returns 200, `make dashboard` idempotent, `make validate` and `make lint` green.
Pending operator steps: `make bootstrap-dashboard-token` (blocked in the agent
session as credential creation) and an Uptime Kuma status page with slug homelab.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbuZrUoevfBgCpf5DCF4DG
pve-emergency-bot.yml, pve-monitoring.yml and pve-docker-test.yml still ran
their creation plays unconditionally. Invoking make deploy-<svc> would pct
start / pct reboot the stopped OLD VMID (148/146/145) on its production IP,
colliding with the live tofu-managed container.
Add the same pre_tasks `meta: end_play` guard the other seven migrated services
already carry: skip while homelab_services['<svc>'].provisioner == 'tofu',
override with -e pve_<svc>_legacy_provisioning_enabled=true for an intentional
legacy rollback. Verified with --check: both plays in each file end immediately,
no pct calls.
Config for these three lives elsewhere (emergency-access.yml, uptime-kuma.yml,
pve-docker-test.yml play 2), so gating the creation plays makes the files
full no-ops under tofu, like pve-gyro.yml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
- docs/ai/: stable, repo-verified context - README, architecture, tech-stack,
edge-cases, plan (confirmed active work only), migration-tofu (the blue-green
OpenTofu migration runbook and per-service findings), legacy-warning, links.
- AGENTS.md: slimmed to a working contract that points at docs/ai instead of
restating it; CLAUDE.md is an adapter that @-includes it.
- README.md, ansible/README.md, ansible/roles/README.md,
roles/lxc_docker_host/README.md: bring wording in line with the current
control plane (Makefile entry point, registry, tofu, memoir-bot gone).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
The *-update.yml playbooks hard-coded the pre-migration VMIDs, so after cutover
the "safety backup" ran against the stopped OLD container. Now the VMID comes
from homelab_services['<svc>'].vmid.
Also removes `--prune-backups keep-all=1` from the vzdump calls: retention is
PBS's job (prune-pbs), and the client-side flag needed Datastore.Modify/Prune
the ansible@pve token does not have, which made vzdump print "Backup ... failed"
and exit non-zero after a successful upload.
gitea-update.yml additionally splits the offsite backup (hosts: gitea) from its
audit (hosts: cloud-pc), matching the profile move into the container.
pve-*.yml imports pass pve_provisioning_enabled: false so the runtime update
path never re-enters container creation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
Blue-green: a new container is created beside the old one, data is copied, the
IP is moved onto it, and the old container is kept stopped as rollback for at
least a week. Keeping the IP means only the VMID changes, and its consumers
(backup jobs, backup audit) already derive it from the registry.
Batch 1 (2026-09-02): emergency-bot 148->151, docker-test 145->152,
gitea 141->153, vaultwarden 140->154, monitoring 146->155, gyro 150->156,
grimmory 149->157.
Batch 2 (2026-09-03): adguard 144->158, mihomo 143->159, ovpn-mini 132->160.
All migratable LXC are now provisioner: tofu. hermes-ai (frozen) and pbs stay.
- tofu/services.tf + tofu/svc-*.tf: one resource per service, reproducing the
pct-config etalon. /dev/fuse -> features.fuse; /dev/net/tun ->
device_passthrough (first live use on mihomo and ovpn-mini); gitea bind mount
-> datastore volume (data finally reaches PBS); console { type = "shell" }
declared explicitly (provider tracks cmode there).
- services.yml: vmid + provisioner: tofu for every migrated service; features
strings and device notes updated to the tofu representation; also drops the
memoir-bot entry and adds homelab_reverse_proxy_image/_unit.
- pve-*.yml: configuration play target is `{{ pve_config_target | default(...) }}`
so it can run against <name>-new on a temp address (a bare --limit zeroes the
play instead of retargeting it). Container-creation plays are gated behind
`provisioner != 'tofu'` / `pve_provisioning_enabled` (meta: end_play), so a
stray run cannot pct start a stopped OLD VMID on a live IP. Override for
intentional legacy rollback: -e pve_<svc>_legacy_provisioning_enabled=true.
- ssh_config: drop memoir-bot; ovpn-mini gets ProxyJump none (a jump via ru-vps
would route through the very tunnel ovpn-mini terminates).
- gyro.yml / uptime-kuma.yml: same pve_config_target override.
- roles/uptime_kuma: only freeze homelab-monitoring when the unit actually
exists (a fresh blue-green container never had it).
- offsite-restic-yadisk.yml: the gitea restic profile now runs inside the LXC
(hosts: gitea), since the bind-mount host path is gone after the volume move;
lost+found excluded (unreadable in an unprivileged LXC, restic exit 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
playbooks/pve-storage-pbs.yml: drop the prune-backups policy from the PVE
storage entry `pbs` so retention authority lives only in the PBS prune job
(prune-pbs). The weekly local PBS-container backup on storage `backup`
(keep-last=2) is an intentional exception and left alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
One ru-vps housekeeping sweep (2026-09-02/03):
- playbooks/ru-vps-base.yml (new): adopt the Caddy compose stack into Ansible
(pinned image by digest, homelab-caddy.service), and manage the corosync-qnetd
UFW rule - allow 5403/tcp from homelab_pve_egress_ip, drop the stale rule for
the retired ZeroTier 10.122.62.0/24. The qdevice had gone silent because its
only allowed path was the decommissioned ZeroTier network.
- group_vars/all/main.yml: homelab_pve_egress_ip (the NATed home egress the PVE
nodes reach corosync-qnetd from - a direct path that does not depend on the
OpenVPN tunnel). Marked dynamic: a change silently re-breaks the qdevice.
- playbooks/status.yml: CLUSTER QUORUM section (pvecm status per PVE node) so a
repeat failure is visible. Also drops the memoir-bot unit list and moves the
gitea offsite-restic unit to the gitea host (see the OpenTofu-migration commit).
- playbooks/ru-vps-zerotier-decommission.yml (new): stop the zerotier container,
disable ssh-zt22.service, remove the interface/9993/9001/10.122.62.0/24 UFW
rules. Node identity and data are kept; removal is a separate step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
Service was unused. Live steps (revoke SecondBrain deploy key, drop the Uptime
Kuma monitor, pct stop 142) were done 2026-09-02; pct destroy is deliberately
deferred a week.
- delete playbooks/pve-memoir-bot.yml.
- hosts.yml: drop the memoir-bot host and its monitoring_exporters entry.
- roles/monitoring_server/templates/prometheus.yml.j2: drop 192.168.1.26 target.
- roles/uptime_kuma/defaults: drop 192.168.1.26 from no_proxy.
- roles/lxc_docker_host/defaults: drop the memoir-bot extra-packages comment.
The registry entry, the ssh_config Host block and the status.yml unit list are
removed in the commits that also carry OpenTofu-migration / quorum changes to
those files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
- pve-backup-jobs.yml: each job's vmid list is now computed from
homelab_services by backup.job instead of a hand-maintained CSV. Adding a
service no longer needs a separate edit here (the forgotten-edit failure
mode that left CT 148 emergency-bot without a backup).
- roles/backup_audit/defaults: backup_audit_pbs_vmids derived from
homelab_services by the monitoring.backup_audit_vmid flag; single shared
freshness threshold backup_audit_pbs_max_age_hours (48).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
playbooks/validate.yml (make validate): reads pct config for every service in
homelab_services and fails if hostname, IP, cores, memory or swap disagree
with the registry. Read-only; meant as a pre/post gate around any inventory or
provisioning change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
.env is now consumed by both Ansible and OpenTofu, so keep a single copy at
the repo root instead of ansible/.env:
- rename ansible/.env.example -> .env.example.
- Makefile: ENV_FILE ?= $(REPO_ROOT)/.env (absolute, works from any cwd);
REQUIRE_ENV/LOAD_ENV updated; help text.
- bootstrap-pve-api-token.yml / bootstrap-monitoring-pve-token.yml write and
read ../../.env; bootstrap now keeps backup: true (it rewrites the whole
file, clobbering MONITORING_*/EMERGENCY_*/PROXMOX_ROOT_PASSWORD).
- roles/pve_lxc, roles/monitoring_server: fail_msg points at the repo-root .env.
The Makefile also picks up the new targets added by later commits
(tofu-*, validate, pbs-storage, ru-vps-base, zerotier-decommission); they are
grouped here so all recipes land together.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
Knowing how to run something required reading ansible/README.md and
remembering to source .env first. Provide `make help` instead, with
targets grouped by purpose and pattern rules for the repetitive families:
deploy-%, dry-%, update-% and play-%.
.env is sourced automatically; targets that need Proxmox credentials fail
with an actionable message when it is missing. Destructive targets --
mihomo-harden, which rotates live credentials, the frozen monitoring
stack, and update-all -- require CONFIRM=1.
The interpreter is resolved at runtime rather than hardcoded to .venv:
the repository's venv is currently broken, so the Makefile falls back to
whatever is on PATH, which is what the Nix devshell provides.
gen-inventory-docs.py prints the host and group tables from
ansible-inventory, so documentation can be regenerated instead of being
maintained by hand and drifting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
One command to see the state of everything: reachability, uptime, disk
usage, service unit states, failed units, pct list on the Proxmox nodes,
OpenVPN transport health, and the last run of each backup job.
An unreachable host is reported as data, not as a run failure, so a
single host being down still produces a full summary. Every command is
changed_when: false with check_mode: false, so the playbook is read-only
and works under --check. Backup freshness is read from what systemd
already recorded rather than by invoking the audit scripts, which would
hit PBS and Yandex Disk and take locks.
Service units are derived from inventory groups where possible; only
app-specific units need the per-host map, and each was taken from the
playbook or role that installs it.
Verified against live infrastructure: 15 hosts, changed=0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
Collect the facts about all 14 services -- VMID, node, address, ports,
domain, pinned images, resources, backup and monitoring participation --
into group_vars/all/services.yml. Values are taken from the existing
playbooks; gaps are marked null rather than invented.
Replace reverse-proxy-{gitea,vaultwarden,grimmory}.yml with a single
playbook iterating over registry entries that declare a domain. It keeps
every check the three had, preserves grimmory's richer Caddy block
byte-for-byte, and restarts Caddy once when any site changed instead of
up to three times. Verified with --check --diff against ru-vps: ok=6
changed=0, so it reproduces the current Caddyfile exactly.
Add two roles factoring out the skeleton duplicated across the pve-*
playbooks: lxc_docker_host (packages, /dev/fuse assertion, fuse-overlayfs
storage driver, UFW baseline) and compose_service (compose file, systemd
unit, config validation, health check). They are not wired into any
playbook yet -- migrating a live service is a separate, per-service step;
compose_service/README.md shows the Gitea example and spells out what
actually changes on the host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
hosts.yml repeated the same authentication block for 13 LXC hosts and
carried 13 byte-identical copies of the ru-vps ProxyCommand. Describe the
transport once in ansible/ssh_config instead: jump host, per-host users,
keys, and the fact that pbs and ovpn-mini are reached directly rather
than through ru-vps.
Ansible loads that file through ansible_ssh_common_args in
group_vars/all/main.yml, where the path is derived from inventory_dir so
it depends on neither the current directory nor the clone location.
The same file makes `ssh gitea` work from a plain terminal once
~/.ssh/config includes it.
hosts.yml drops from 209 to 137 lines and now holds only addresses and
per-host facts. Verified equivalent: ansible-inventory --list before and
after differ only by the removed ansible_ssh_common_args, with group
membership and ordering byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
Commit the accumulated infrastructure work that was living only in the
working tree: monitoring stack, emergency access/bot, gyro allocator,
grimmory, adguard, backup audit and the OpenCode agent definitions.
Also ignore Python bytecode, local archives and Nix/direnv artifacts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
