The *-update.yml playbooks hard-coded the pre-migration VMIDs, so after cutover
the "safety backup" ran against the stopped OLD container. Now the VMID comes
from homelab_services['<svc>'].vmid.
Also removes `--prune-backups keep-all=1` from the vzdump calls: retention is
PBS's job (prune-pbs), and the client-side flag needed Datastore.Modify/Prune
the ansible@pve token does not have, which made vzdump print "Backup ... failed"
and exit non-zero after a successful upload.
gitea-update.yml additionally splits the offsite backup (hosts: gitea) from its
audit (hosts: cloud-pc), matching the profile move into the container.
pve-*.yml imports pass pve_provisioning_enabled: false so the runtime update
path never re-enters container creation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
Blue-green: a new container is created beside the old one, data is copied, the
IP is moved onto it, and the old container is kept stopped as rollback for at
least a week. Keeping the IP means only the VMID changes, and its consumers
(backup jobs, backup audit) already derive it from the registry.
Batch 1 (2026-09-02): emergency-bot 148->151, docker-test 145->152,
gitea 141->153, vaultwarden 140->154, monitoring 146->155, gyro 150->156,
grimmory 149->157.
Batch 2 (2026-09-03): adguard 144->158, mihomo 143->159, ovpn-mini 132->160.
All migratable LXC are now provisioner: tofu. hermes-ai (frozen) and pbs stay.
- tofu/services.tf + tofu/svc-*.tf: one resource per service, reproducing the
pct-config etalon. /dev/fuse -> features.fuse; /dev/net/tun ->
device_passthrough (first live use on mihomo and ovpn-mini); gitea bind mount
-> datastore volume (data finally reaches PBS); console { type = "shell" }
declared explicitly (provider tracks cmode there).
- services.yml: vmid + provisioner: tofu for every migrated service; features
strings and device notes updated to the tofu representation; also drops the
memoir-bot entry and adds homelab_reverse_proxy_image/_unit.
- pve-*.yml: configuration play target is `{{ pve_config_target | default(...) }}`
so it can run against <name>-new on a temp address (a bare --limit zeroes the
play instead of retargeting it). Container-creation plays are gated behind
`provisioner != 'tofu'` / `pve_provisioning_enabled` (meta: end_play), so a
stray run cannot pct start a stopped OLD VMID on a live IP. Override for
intentional legacy rollback: -e pve_<svc>_legacy_provisioning_enabled=true.
- ssh_config: drop memoir-bot; ovpn-mini gets ProxyJump none (a jump via ru-vps
would route through the very tunnel ovpn-mini terminates).
- gyro.yml / uptime-kuma.yml: same pve_config_target override.
- roles/uptime_kuma: only freeze homelab-monitoring when the unit actually
exists (a fresh blue-green container never had it).
- offsite-restic-yadisk.yml: the gitea restic profile now runs inside the LXC
(hosts: gitea), since the bind-mount host path is gone after the volume move;
lost+found excluded (unreadable in an unprivileged LXC, restic exit 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
playbooks/pve-storage-pbs.yml: drop the prune-backups policy from the PVE
storage entry `pbs` so retention authority lives only in the PBS prune job
(prune-pbs). The weekly local PBS-container backup on storage `backup`
(keep-last=2) is an intentional exception and left alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
One ru-vps housekeeping sweep (2026-09-02/03):
- playbooks/ru-vps-base.yml (new): adopt the Caddy compose stack into Ansible
(pinned image by digest, homelab-caddy.service), and manage the corosync-qnetd
UFW rule - allow 5403/tcp from homelab_pve_egress_ip, drop the stale rule for
the retired ZeroTier 10.122.62.0/24. The qdevice had gone silent because its
only allowed path was the decommissioned ZeroTier network.
- group_vars/all/main.yml: homelab_pve_egress_ip (the NATed home egress the PVE
nodes reach corosync-qnetd from - a direct path that does not depend on the
OpenVPN tunnel). Marked dynamic: a change silently re-breaks the qdevice.
- playbooks/status.yml: CLUSTER QUORUM section (pvecm status per PVE node) so a
repeat failure is visible. Also drops the memoir-bot unit list and moves the
gitea offsite-restic unit to the gitea host (see the OpenTofu-migration commit).
- playbooks/ru-vps-zerotier-decommission.yml (new): stop the zerotier container,
disable ssh-zt22.service, remove the interface/9993/9001/10.122.62.0/24 UFW
rules. Node identity and data are kept; removal is a separate step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
Service was unused. Live steps (revoke SecondBrain deploy key, drop the Uptime
Kuma monitor, pct stop 142) were done 2026-09-02; pct destroy is deliberately
deferred a week.
- delete playbooks/pve-memoir-bot.yml.
- hosts.yml: drop the memoir-bot host and its monitoring_exporters entry.
- roles/monitoring_server/templates/prometheus.yml.j2: drop 192.168.1.26 target.
- roles/uptime_kuma/defaults: drop 192.168.1.26 from no_proxy.
- roles/lxc_docker_host/defaults: drop the memoir-bot extra-packages comment.
The registry entry, the ssh_config Host block and the status.yml unit list are
removed in the commits that also carry OpenTofu-migration / quorum changes to
those files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
- pve-backup-jobs.yml: each job's vmid list is now computed from
homelab_services by backup.job instead of a hand-maintained CSV. Adding a
service no longer needs a separate edit here (the forgotten-edit failure
mode that left CT 148 emergency-bot without a backup).
- roles/backup_audit/defaults: backup_audit_pbs_vmids derived from
homelab_services by the monitoring.backup_audit_vmid flag; single shared
freshness threshold backup_audit_pbs_max_age_hours (48).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
playbooks/validate.yml (make validate): reads pct config for every service in
homelab_services and fails if hostname, IP, cores, memory or swap disagree
with the registry. Read-only; meant as a pre/post gate around any inventory or
provisioning change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
- providers.tf / variables.tf / versions.tf: bpg/proxmox ~> 0.84, endpoint and
credentials from TF_VAR_* (set by the Makefile tofu-* targets from the
repo-root .env). Two auth modes: root@pam by password (privileged: features
beyond nesting, device passthrough, datastore mount points) or the
ansible@pve token.
- README.md: pilot results on VMID 199 - what the token can and cannot do,
why a root token still fails the literal `$authuser eq 'root@pam'` check,
the cmode/console drift finding, and the chosen root@pam-by-password mode.
- pilot.tf.example: reference resource shape (features, device_passthrough,
mount_point), not loaded (.example).
- .terraform.lock.hcl: pin the provider.
State has no backend yet; tofu/*.tfstate stays local and git-ignored.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
.env is now consumed by both Ansible and OpenTofu, so keep a single copy at
the repo root instead of ansible/.env:
- rename ansible/.env.example -> .env.example.
- Makefile: ENV_FILE ?= $(REPO_ROOT)/.env (absolute, works from any cwd);
REQUIRE_ENV/LOAD_ENV updated; help text.
- bootstrap-pve-api-token.yml / bootstrap-monitoring-pve-token.yml write and
read ../../.env; bootstrap now keeps backup: true (it rewrites the whole
file, clobbering MONITORING_*/EMERGENCY_*/PROXMOX_ROOT_PASSWORD).
- roles/pve_lxc, roles/monitoring_server: fail_msg points at the repo-root .env.
The Makefile also picks up the new targets added by later commits
(tofu-*, validate, pbs-storage, ru-vps-base, zerotier-decommission); they are
grouped here so all recipes land together.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
- flake.nix: add pkgs.opentofu to the devshell (LXC provisioning pilot).
- .gitignore: ignore tofu/.terraform/, *.tfstate*, *.tfplan and tofu/.env.
The provider lock file (tofu/.terraform.lock.hcl) stays tracked on purpose:
it pins the provider version.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uoq5AVK8mkBgg83Mq6o5V
The host table listed 9 hosts against 16 in the inventory, the role tree
did not match roles/, and the documented setup path used a venv that no
longer works.
Describe the current entry points instead: nix develop, make, and the
ssh_config include that makes `ssh gitea` work by hand. Point at
`make docs` as the way to regenerate the host table rather than editing
it, since that is what drifted.
Also record what is deliberately incomplete: lxc_docker_host and
compose_service exist but are not wired into any playbook, and LXC
creation is still split between direct pct create over SSH and the
pve_lxc API role.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
Knowing how to run something required reading ansible/README.md and
remembering to source .env first. Provide `make help` instead, with
targets grouped by purpose and pattern rules for the repetitive families:
deploy-%, dry-%, update-% and play-%.
.env is sourced automatically; targets that need Proxmox credentials fail
with an actionable message when it is missing. Destructive targets --
mihomo-harden, which rotates live credentials, the frozen monitoring
stack, and update-all -- require CONFIRM=1.
The interpreter is resolved at runtime rather than hardcoded to .venv:
the repository's venv is currently broken, so the Makefile falls back to
whatever is on PATH, which is what the Nix devshell provides.
gen-inventory-docs.py prints the host and group tables from
ansible-inventory, so documentation can be regenerated instead of being
maintained by hand and drifting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
One command to see the state of everything: reachability, uptime, disk
usage, service unit states, failed units, pct list on the Proxmox nodes,
OpenVPN transport health, and the last run of each backup job.
An unreachable host is reported as data, not as a run failure, so a
single host being down still produces a full summary. Every command is
changed_when: false with check_mode: false, so the playbook is read-only
and works under --check. Backup freshness is read from what systemd
already recorded rather than by invoking the audit scripts, which would
hit PBS and Yandex Disk and take locks.
Service units are derived from inventory groups where possible; only
app-specific units need the per-host map, and each was taken from the
playbook or role that installs it.
Verified against live infrastructure: 15 hosts, changed=0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
Collect the facts about all 14 services -- VMID, node, address, ports,
domain, pinned images, resources, backup and monitoring participation --
into group_vars/all/services.yml. Values are taken from the existing
playbooks; gaps are marked null rather than invented.
Replace reverse-proxy-{gitea,vaultwarden,grimmory}.yml with a single
playbook iterating over registry entries that declare a domain. It keeps
every check the three had, preserves grimmory's richer Caddy block
byte-for-byte, and restarts Caddy once when any site changed instead of
up to three times. Verified with --check --diff against ru-vps: ok=6
changed=0, so it reproduces the current Caddyfile exactly.
Add two roles factoring out the skeleton duplicated across the pve-*
playbooks: lxc_docker_host (packages, /dev/fuse assertion, fuse-overlayfs
storage driver, UFW baseline) and compose_service (compose file, systemd
unit, config validation, health check). They are not wired into any
playbook yet -- migrating a live service is a separate, per-service step;
compose_service/README.md shows the Gitea example and spells out what
actually changes on the host.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
hosts.yml repeated the same authentication block for 13 LXC hosts and
carried 13 byte-identical copies of the ru-vps ProxyCommand. Describe the
transport once in ansible/ssh_config instead: jump host, per-host users,
keys, and the fact that pbs and ovpn-mini are reached directly rather
than through ru-vps.
Ansible loads that file through ansible_ssh_common_args in
group_vars/all/main.yml, where the path is derived from inventory_dir so
it depends on neither the current directory nor the clone location.
The same file makes `ssh gitea` work from a plain terminal once
~/.ssh/config includes it.
hosts.yml drops from 209 to 137 lines and now holds only addresses and
per-host facts. Verified equivalent: ansible-inventory --list before and
after differ only by the removed ansible_ssh_common_args, with group
membership and ordering byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
Configure yamllint and ansible-lint, plus a workflow running yamllint,
ansible-lint and ansible-playbook --syntax-check over every playbook.
ansible-lint uses the moderate profile: on the current code it reports
exactly the same violations as basic, so it costs nothing today while
holding a higher bar for new code. skip_list is empty; noisy legacy
rules go to warn_list with a comment on why and when to restore them.
Correctness and safety rules stay fatal.
Two constraints are encoded in the workflow: syntax-check must run from
ansible/ because roles_path is relative, and ansible-lint needs absolute
ANSIBLE_ROLES_PATH/ANSIBLE_COLLECTIONS_PATH when run from the root.
The runner is not registered yet; registration notes are in the workflow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
Replace the Python venv with a Nix devshell pinning ansible-core 2.21.3,
ansible-lint, yamllint and a Python with proxmoxer/requests. The Python
dependencies share the interpreter that runs ansible, so pve-*.yml plays
on implicit localhost can import proxmoxer without inventory changes.
The shellHook exports absolute ANSIBLE_CONFIG, ANSIBLE_INVENTORY,
ANSIBLE_ROLES_PATH and ANSIBLE_COLLECTIONS_PATH, so commands work from
the repository root as well as from ansible/.
Also un-ignore .envrc, which the global gitignore hides, and ignore the
stray .ansible/ runtime directory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
Commit the accumulated infrastructure work that was living only in the
working tree: monitoring stack, emergency access/bot, gyro allocator,
grimmory, adguard, backup audit and the OpenCode agent definitions.
Also ignore Python bytecode, local archives and Nix/direnv artifacts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71
