Capture current Ansible control plane state

Commit the accumulated infrastructure work that was living only in the
working tree: monitoring stack, emergency access/bot, gyro allocator,
grimmory, adguard, backup audit and the OpenCode agent definitions.

Also ignore Python bytecode, local archives and Nix/direnv artifacts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71

ansible/.env.example

@@ -3,6 +3,23 @@ export PROXMOX_USER='ansible@pve'
 export PROXMOX_TOKEN_ID='homelab'
 export PROXMOX_TOKEN_SECRET='replace-me'
 export PROXMOX_VALIDATE_CERTS=false
+# Optional local SSH-forward port when the control node reaches PVE through ru-vps.
+# export PROXMOX_PORT=18006
 
 # Override if the downloaded template name differs.
 export PVE_LXC_OSTEMPLATE='local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst'
+
+# Monitoring secrets. Keep actual values in ignored ansible/.env or Ansible Vault.
+export MONITORING_TELEGRAM_BOT_TOKEN='replace-me'
+export MONITORING_TELEGRAM_CHAT_ID='replace-me'
+export MONITORING_GRAFANA_ADMIN_PASSWORD='replace-me'
+export MONITORING_PVE_API_USER='monitoring@pve'
+export MONITORING_PVE_API_TOKEN_ID='prometheus'
+export MONITORING_PVE_API_TOKEN_SECRET='replace-me'
+
+# Emergency reverse SSH Telegram bot. Keep actual values in ignored ansible/.env
+# or Ansible Vault. Host keys are pinned public keys in the form "ssh-ed25519 AAAA...".
+export EMERGENCY_BOT_TOKEN='replace-me'
+export EMERGENCY_ALLOWED_USER_IDS='replace-me'
+export EMERGENCY_VPS_HOST_KEY='ssh-ed25519 replace-me'
+export EMERGENCY_MINI_PC_HOST_KEY='ssh-ed25519 replace-me'

ansible/README.md

@@ -17,10 +17,13 @@ Ansible is the control plane for HomeLab infrastructure changes.
 
 ## Current Groups
 
-- `ru-vps` — public VPS, JumpHost, qdevice, ZeroTier member.
+- `ru-vps` — public VPS, JumpHost, qdevice, OpenVPN server.
 - `pve_nodes` — Proxmox hosts: `cloud-pc`, `mini-pc`.
-- `lxc_infra` — infrastructure LXC containers: `pbs`, `zt-cloud`, `zt-mini`.
-- `vpn_openvpn` — OpenVPN transport hosts: `ru-vps`, `wg-mini`.
+- `lxc_infra` — infrastructure LXC containers, including the outbound-only `gyro` investment allocator host.
+- `monitoring_server` — monitoring LXC; Prometheus stack is frozen and Uptime Kuma is active.
+- `monitoring_exporters` — hosts exposing Node Exporter metrics.
+- `monitoring_smart_exporters` — Proxmox nodes exposing SMART metrics.
+- `vpn_openvpn` — OpenVPN transport hosts: `ru-vps`, `ovpn-mini`.
 - `shell_hosts` — hosts with unified bash config: `ru-vps`, `cloud-pc`, `mini-pc`.
 - `servers` — all managed hosts.
 
@@ -41,12 +44,44 @@ Run from `ansible/`:
 ansible-playbook playbooks/check.yml
 ```
 
+## Controlled Updates
+
+Service updates are manual and use pinned `tag@sha256:digest` image references only; floating tags and auto-update agents are not used.
+
+Run the dedicated playbook for the target service from `ansible/`:
+
+```bash
+.venv/bin/ansible-playbook playbooks/vaultwarden-update.yml
+.venv/bin/ansible-playbook playbooks/gitea-update.yml
+.venv/bin/ansible-playbook playbooks/adguard-update.yml
+.venv/bin/ansible-playbook playbooks/mihomo-update.yml
+.venv/bin/ansible-playbook playbooks/grimmory-update.yml
+```
+
+Update flow is always: fresh backup/audit first, then the update playbook, then health verification.
+
+- Gitea, Vaultwarden and Grimmory use app-aware offsite restic backups/audits before update.
+- AdGuard and Mihomo use fresh PBS LXC backups before update.
+- `grimmory-update.yml` currently validates the existing pinned release; any future actual app image upgrade must be preceded by release-note and migration review.
+
+## Mihomo Hardening
+
+Use the dedicated hardening playbook only when explicitly approved:
+
+```bash
+.venv/bin/ansible-playbook playbooks/ru-vps-mihomo-harden.yml -e ru_vps_mihomo_harden_confirm=true
+```
+
+It rotates the live Mihomo SOCKS credentials on `ru-vps`, locks the proxy to loopback, and removes the public UFW exposure for ports `7890` and `7891`.
+
+The rotated credentials are not recoverable for clients unless you roll back the saved config backup.
+
 For Proxmox API playbooks, create ignored `.env` from `.env.example` and load it:
 
 ```bash
 cp .env.example .env
 . ./.env
-.venv/bin/ansible-playbook playbooks/pve-wg-mini.yml
+.venv/bin/ansible-playbook playbooks/pve-ovpn-mini.yml
 ```
 
 Or bootstrap the token from `mini-pc` with sudo:
@@ -55,6 +90,12 @@ Or bootstrap the token from `mini-pc` with sudo:
 .venv/bin/ansible-playbook playbooks/bootstrap-pve-api-token.yml -K
 ```
 
+Create the separate read-only PVE token used by the monitoring exporter:
+
+```bash
+.venv/bin/ansible-playbook playbooks/bootstrap-monitoring-pve-token.yml
+```
+
 OpenVPN transport:
 
 ```bash
@@ -62,6 +103,67 @@ OpenVPN transport:
 .venv/bin/ansible-playbook playbooks/openvpn-check.yml
 ```
 
+Monitoring is provisioned in two steps after loading the monitoring secrets from ignored `.env` or Ansible Vault:
+
+```bash
+. ./.env
+.venv/bin/ansible-playbook playbooks/pve-monitoring.yml
+.venv/bin/ansible-playbook playbooks/monitoring.yml
+```
+
+`pve-monitoring.yml` creates CT `146` (`monitoring`, `192.168.1.30`) on `cloud-pc`. `monitoring.yml` configures exporters, the `ru-vps` probe vantage point, and the central Prometheus stack. It is frozen while Uptime Kuma is in use; do not run it unless restoring Prometheus monitoring.
+
+## Gyro Investment Allocator
+
+`pve-gyro.yml` creates unprivileged CT `150` (`gyro`, `192.168.1.35`) on `mini-pc`. `gyro.yml` installs Python 3.13+, pinned `uv`, the `gyro` service user, a container-local GitHub deploy key, restrictive firewall rules, and a weekday systemd timer.
+
+UFW is the currently enforced isolation layer: inbound is denied except SSH from LAN/OpenVPN, and east-west outbound is denied except the Mihomo HTTP proxy. The equivalent CT `150` Proxmox firewall is staged, but the cluster-wide PVE firewall remains disabled; do not enable it without auditing every node and guest with `firewall=1`.
+
+The role keeps deployment and the timer disabled by default. The active host vars deploy `git@github.com:ada-dmitry/t_tech-gyro.git` with GitHub's verified ED25519 host key; the timer still requires the ignored Vault file:
+
+```bash
+cp inventory/host_vars/gyro/vault.yml.example inventory/host_vars/gyro/vault.yml
+ansible-vault encrypt inventory/host_vars/gyro/vault.yml
+```
+
+After encrypting the secrets, set `gyro_timer_enabled: true` in `main.yml` and apply with `--ask-vault-pass`. `DRY_RUN_OVERRIDE` remains `true` until real trading is explicitly approved.
+
+```bash
+. ./.env
+.venv/bin/ansible-playbook playbooks/pve-gyro.yml
+.venv/bin/ansible-playbook playbooks/gyro.yml --ask-vault-pass
+```
+
+The timer runs at 11:00 Europe/Moscow from Monday through Friday and uses `OnFailure=` for a best-effort Telegram alert. CT `150` is included in the daily mini-pc PBS job and backup freshness audit.
+
+Uptime Kuma uses the existing monitoring LXC and stops/disables `homelab-monitoring` without deleting its configuration or data. Its UI is available only from the LAN at `http://192.168.1.30:3001`; create monitors and notification settings in the UI.
+
+```bash
+.venv/bin/ansible-playbook playbooks/uptime-kuma.yml
+```
+
+## Emergency Reverse SSH
+
+`pve-emergency-bot.yml` creates CT `148` (`emergency-bot`, `192.168.1.32`) on `mini-pc`. `emergency-access.yml` configures the bot, a TTL-limited reverse tunnel from `mini-pc` to `ru-vps`, and the restricted SSH identities used between them.
+
+Before applying, set `EMERGENCY_BOT_TOKEN`, `EMERGENCY_ALLOWED_USER_IDS`, `EMERGENCY_VPS_HOST_KEY`, and `EMERGENCY_MINI_PC_HOST_KEY` in ignored `.env` or Ansible Vault. The host-key variables must be verified public host keys, not values obtained during deployment.
+
+```bash
+. ./.env
+.venv/bin/ansible-playbook playbooks/pve-emergency-bot.yml
+.venv/bin/ansible-playbook playbooks/emergency-access.yml
+```
+
+From an authorized private Telegram chat, use the `Enable SSH`, `Status`, and `Stop` buttons or `/emergency ssh`, `/emergency status`, and `/emergency stop`. `/emergency ssh` enables a 60-minute tunnel only; it does not expose a public port. Connect while it is active with:
+
+The bot uses the LAN Mihomo HTTP proxy at `192.168.1.27:7890` because direct Telegram TCP access is unavailable from HomeLab.
+
+```bash
+ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -J vps -p 22010 ansible@127.0.0.1
+```
+
+The target account is `ansible`; it has no password login. Use the existing private key `~/.ssh/id_ed25519_homelab_ansible`.
+
 Bootstrap the Ansible service account on shell hosts:
 
 ```bash

ansible/files/ru-vps-mihomo-harden.py

@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import os
+import secrets
+import stat
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+
+
+DEFAULT_PROXY_HOST = "127.0.0.1"
+DEFAULT_PROXY_PORT = 7891
+
+
+def _atomic_write_bytes(path: Path, data: bytes, mode: int) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, tmp_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=str(path.parent))
+    try:
+        with os.fdopen(fd, "wb") as handle:
+            handle.write(data)
+            handle.flush()
+            os.fsync(handle.fileno())
+        os.chmod(tmp_name, mode)
+        os.replace(tmp_name, path)
+    finally:
+        if os.path.exists(tmp_name):
+            os.unlink(tmp_name)
+
+
+def _atomic_write_json(path: Path, payload: dict[str, str]) -> None:
+    _atomic_write_bytes(path, json.dumps(payload, indent=2, sort_keys=True).encode("utf-8") + b"\n", 0o600)
+
+
+def _load_json(path: Path) -> dict[str, str]:
+    with path.open("r", encoding="utf-8") as handle:
+        payload = json.load(handle)
+    if not isinstance(payload, dict):
+        raise ValueError(f"{path} must contain a JSON object")
+    return payload
+
+
+def _generate_username() -> str:
+    return f"mihomo-{secrets.token_hex(8)}"
+
+
+def _generate_password() -> str:
+    return secrets.token_urlsafe(48)
+
+
+def ensure_credentials(state_path: Path) -> tuple[dict[str, str], bool]:
+    if state_path.exists():
+        payload = _load_json(state_path)
+        username = payload.get("username")
+        password = payload.get("password")
+        if not username or not password:
+            raise ValueError(f"{state_path} is missing username/password")
+        return {"username": username, "password": password}, False
+
+    state_path.parent.mkdir(parents=True, exist_ok=True)
+    os.chmod(state_path.parent, 0o700)
+    credentials = {"username": _generate_username(), "password": _generate_password()}
+    _atomic_write_json(state_path, credentials)
+    return credentials, True
+
+
+def load_config(config_path: Path) -> dict:
+    with config_path.open("r", encoding="utf-8") as handle:
+        payload = yaml.safe_load(handle) or {}
+    if not isinstance(payload, dict):
+        raise ValueError(f"{config_path} must contain a YAML mapping")
+    return payload
+
+
+def write_config(config_path: Path, payload: dict) -> None:
+    serialized = yaml.safe_dump(payload, sort_keys=False, allow_unicode=True)
+    mode = 0o640
+    if config_path.exists():
+        mode = stat.S_IMODE(config_path.stat().st_mode)
+    _atomic_write_bytes(config_path, serialized.encode("utf-8"), mode)
+
+
+def cmd_apply(args: argparse.Namespace) -> int:
+    credentials, state_created = ensure_credentials(args.state)
+    config = load_config(args.config)
+    updated = copy.deepcopy(config)
+    updated["authentication"] = [f"{credentials['username']}:{credentials['password']}"]
+    updated["allow-lan"] = False
+    updated["bind-address"] = "127.0.0.1"
+    changed = updated != config
+    if changed:
+        write_config(args.config, updated)
+    print(json.dumps({"changed": changed, "state_created": state_created}, sort_keys=True))
+    return 0
+
+
+def cmd_probe(args: argparse.Namespace) -> int:
+    credentials = ensure_credentials(args.state)[0]
+    curl_config = "\n".join(
+        [
+            'silent',
+            'show-error',
+            f'proxy = "socks5h://{DEFAULT_PROXY_HOST}:{DEFAULT_PROXY_PORT}"',
+            f'proxy-user = "{credentials["username"]}:{credentials["password"]}"',
+            'connect-timeout = 5',
+            'max-time = 20',
+            'url = "https://api.telegram.org"',
+            'output = "/dev/null"',
+            '',
+        ]
+    )
+    result = subprocess.run(
+        ["curl", "--config", "-"],
+        input=curl_config,
+        text=True,
+        capture_output=True,
+        timeout=25,
+        check=False,
+    )
+    if result.returncode != 0:
+        sys.stderr.write(result.stderr)
+        return result.returncode
+    print(json.dumps({"ok": True}, sort_keys=True))
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    for name, func in (("apply", cmd_apply), ("probe", cmd_probe)):
+        sub = subparsers.add_parser(name)
+        sub.add_argument("--config", required=True, type=Path)
+        sub.add_argument("--state", required=True, type=Path)
+        sub.set_defaults(func=func)
+
+    return parser.parse_args()
+
+
+def main() -> int:
+    args = parse_args()
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ansible/inventory/host_vars/gyro/main.yml

@@ -0,0 +1,9 @@
+---
+gyro_repo_url: git@github.com:ada-dmitry/t_tech-gyro.git
+gyro_repo_version: main
+gyro_git_known_hosts_name: github.com
+gyro_git_host_key: "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"
+gyro_deploy_enabled: true
+gyro_secrets_configured: false
+gyro_timer_enabled: true
+gyro_timer_on_calendar: "Mon..Fri *-*-* 11:00:00 Europe/Moscow"

ansible/inventory/host_vars/gyro/vault.yml.example

@@ -0,0 +1,10 @@
+---
+# Copy to vault.yml, replace the placeholders, then run:
+# ansible-vault encrypt inventory/host_vars/gyro/vault.yml
+gyro_tinvest_token: t.replace-me
+gyro_tinvest_account_id: replace-me
+gyro_telegram_bot_token: replace-me
+gyro_telegram_user_id: "replace-me"
+gyro_telegram_proxy: http://192.168.1.27:7890
+gyro_dry_run_override: "true"
+gyro_secrets_configured: true

ansible/inventory/hosts.yml

@@ -6,15 +6,21 @@ all:
     ansible_become: true
     homelab_lan_cidr: 192.168.1.0/24
     homelab_service_range: 192.168.1.5-192.168.1.40
-    homelab_zerotier_network_id: 743993800fa5a34c
     openvpn_network_cidr: 10.78.0.0/30
     openvpn_listen_port: 8443
+    openvpn_forwarded_services:
+      - name: satisfactory
+        protocol: udp
+        public_port: 7777
+        target_host: 192.168.1.100
+        target_port: 7777
 
   children:
     homelab:
       hosts:
         ru-vps:
           ansible_host: vps
+          monitoring_exporter_node_listen_address: 127.0.0.1:9100
           openvpn_local_ip: 10.78.0.1
           openvpn_peer_ip: 10.78.0.2
           openvpn_role: server
@@ -44,29 +50,39 @@ all:
       hosts:
         pbs:
           ansible_host: 192.168.1.20
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
           expected_lan_ip: 192.168.1.20
 
-        zt-cloud:
-          ansible_host: 192.168.1.21
-          expected_lan_ip: 192.168.1.21
-          zerotier_ip: 10.122.62.206
-          zerotier_node_id: c606e6d181
-
-        zt-mini:
-          ansible_host: 192.168.1.22
-          expected_lan_ip: 192.168.1.22
-          zerotier_ip: 10.122.62.213
-          zerotier_node_id: 022ac284e1
-
-        wg-mini:
+        ovpn-mini:
           ansible_host: 192.168.1.23
           ansible_user: root
+          ansible_become: false
           ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
           expected_lan_ip: 192.168.1.23
           openvpn_local_ip: 10.78.0.2
           openvpn_peer_ip: 10.78.0.1
           openvpn_role: gateway
 
+        adguard:
+          ansible_host: 192.168.1.28
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.28
+
+        docker-test:
+          ansible_host: 192.168.1.29
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.29
+
         vaultwarden:
           ansible_host: 192.168.1.24
           ansible_user: root
@@ -85,16 +101,106 @@ all:
             -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
           expected_lan_ip: 192.168.1.25
 
+        memoir-bot:
+          ansible_host: 192.168.1.26
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.26
+
+        mihomo:
+          ansible_host: 192.168.1.27
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.27
+
+        monitoring:
+          ansible_host: 192.168.1.30
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.30
+
+        hermes-ai:
+          ansible_host: 192.168.1.31
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.31
+          bash_config_proxy_http_url: http://192.168.1.27:7890
+          bash_config_proxy_socks_url: socks5h://192.168.1.27:7890
+
+        emergency-bot:
+          ansible_host: 192.168.1.32
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.32
+
+        grimmory:
+          ansible_host: 192.168.1.34
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.34
+
+        gyro:
+          ansible_host: 192.168.1.35
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.35
+
+    monitoring_exporters:
+      hosts:
+        ru-vps:
+        cloud-pc:
+        mini-pc:
+        pbs:
+        ovpn-mini:
+        vaultwarden:
+        gitea:
+        memoir-bot:
+        mihomo:
+        adguard:
+        monitoring:
+        grimmory:
+
+    monitoring_smart_exporters:
+      hosts:
+        cloud-pc:
+        mini-pc:
+
+    monitoring_server:
+      hosts:
+        monitoring:
+
     vpn_openvpn:
       hosts:
         ru-vps:
-        wg-mini:
+        ovpn-mini:
 
     shell_hosts:
       hosts:
         ru-vps:
         cloud-pc:
         mini-pc:
+        hermes-ai:
 
     servers:
       children:

ansible/playbooks/adguard-update.yml

@@ -0,0 +1,110 @@
+---
+- name: Create and verify current PBS audit before AdGuard update
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Verify AdGuard VMID ownership before backup
+      ansible.builtin.command: pct config 144
+      register: adguard_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to back up a foreign VMID 144
+      ansible.builtin.assert:
+        that:
+          - adguard_pct_config.rc == 0
+          - adguard_existing_hostname == 'adguard'
+        fail_msg: VMID 144 is not the AdGuard container.
+      vars:
+        adguard_existing_hostname: >-
+          {{ adguard_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Check for active Proxmox backup before AdGuard PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: adguard_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before AdGuard PBS backup
+      ansible.builtin.assert:
+        that:
+          - adguard_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on mini-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh AdGuard PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "144"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+    - name: Run current PBS audit service
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-adguard.yml
+
+- name: Verify AdGuard public services after update
+  hosts: adguard
+  gather_facts: false
+  tasks:
+    - name: Check AdGuard HTTP root
+      ansible.builtin.uri:
+        url: http://127.0.0.1/
+        status_code: [200, 302]
+        return_content: false
+      register: adguard_http_root_check
+      retries: 24
+      delay: 5
+      until: adguard_http_root_check.status in [200, 302]
+
+    - name: Check AdGuard DNS over UDP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_udp_check
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_udp_check.rc == 0 and '127.0.0.1' in adguard_dns_udp_check.stdout
+
+    - name: Check AdGuard DNS over TCP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +tcp
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_tcp_check
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_tcp_check.rc == 0 and '127.0.0.1' in adguard_dns_tcp_check.stdout

ansible/playbooks/backup-audit.yml

@@ -0,0 +1,45 @@
+- name: Configure PBS backup audit on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: pbs
+
+- name: Configure restic offsite audit on cloud-pc (Gitea)
+  hosts: cloud-pc
+  gather_facts: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: restic
+        backup_audit_restic_profiles:
+          - name: gitea
+            sqlite_name: gitea.db
+            max_age_hours: 36
+
+- name: Configure restic offsite audit on vaultwarden
+  hosts: vaultwarden
+  gather_facts: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: restic
+        backup_audit_restic_profiles:
+          - name: vaultwarden
+            sqlite_name: db.sqlite3
+            max_age_hours: 36
+
+- name: Configure restic offsite audit on Grimmory
+  hosts: grimmory
+  gather_facts: false
+  vars:
+    ansible_become: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: restic
+        backup_audit_restic_profiles:
+          - name: grimmory
+            expected_name: grimmory.sql
+            max_age_hours: 36

ansible/playbooks/bootstrap-monitoring-pve-token.yml

@@ -0,0 +1,76 @@
+---
+- name: Create read-only Proxmox token for monitoring
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    monitoring_pve_user: monitoring@pve
+    monitoring_pve_token_id: prometheus
+    monitoring_pve_env_file: "{{ playbook_dir }}/../.env"
+    monitoring_pve_rotate_existing_token: false
+  tasks:
+    - name: Read existing Proxmox users
+      ansible.builtin.command: pveum user list --output-format json
+      register: monitoring_pve_users_raw
+      changed_when: false
+
+    - name: Create monitoring Proxmox user
+      ansible.builtin.command: >-
+        pveum user add {{ monitoring_pve_user }}
+        --comment 'Read-only Prometheus monitoring user'
+      when: monitoring_pve_user not in (monitoring_pve_users_raw.stdout | from_json | map(attribute='userid') | list)
+
+    - name: Grant PVEAuditor role to monitoring user
+      ansible.builtin.command: >-
+        pveum acl modify / -user {{ monitoring_pve_user }} -role PVEAuditor
+      changed_when: false
+
+    - name: Read monitoring user tokens
+      ansible.builtin.command: >-
+        pveum user token list {{ monitoring_pve_user }} --output-format json
+      register: monitoring_pve_tokens_raw
+      changed_when: false
+
+    - name: Refuse to overwrite an existing monitoring token
+      ansible.builtin.assert:
+        that:
+          - monitoring_pve_token_id not in (monitoring_pve_tokens_raw.stdout | from_json | map(attribute='tokenid') | list)
+        fail_msg: Existing monitoring token secret cannot be recovered safely. Rotate it explicitly before rerunning this playbook.
+      when: not monitoring_pve_rotate_existing_token | bool
+
+    - name: Rotate existing monitoring token explicitly
+      ansible.builtin.command: >-
+        pveum user token remove {{ monitoring_pve_user }} {{ monitoring_pve_token_id }}
+      when:
+        - monitoring_pve_rotate_existing_token | bool
+        - monitoring_pve_token_id in (monitoring_pve_tokens_raw.stdout | from_json | map(attribute='tokenid') | list)
+
+    - name: Create separated monitoring token
+      ansible.builtin.command: >-
+        pveum user token add {{ monitoring_pve_user }} {{ monitoring_pve_token_id }}
+        --privsep 1 --comment 'Prometheus PVE exporter' --output-format json
+      register: monitoring_pve_token_created
+      no_log: true
+
+    - name: Grant PVEAuditor role to separated monitoring token
+      ansible.builtin.command: >-
+        pveum acl modify / -token {{ monitoring_pve_user }}!{{ monitoring_pve_token_id }} -role PVEAuditor
+      changed_when: false
+
+    - name: Store monitoring token variables locally
+      ansible.builtin.lineinfile:
+        path: "{{ monitoring_pve_env_file }}"
+        regexp: "^export {{ item.name }}="
+        line: "export {{ item.name }}='{{ item.value }}'"
+        create: false
+      loop:
+        - name: MONITORING_PVE_API_USER
+          value: "{{ monitoring_pve_user }}"
+        - name: MONITORING_PVE_API_TOKEN_ID
+          value: "{{ monitoring_pve_token_id }}"
+        - name: MONITORING_PVE_API_TOKEN_SECRET
+          value: "{{ (monitoring_pve_token_created.stdout | from_json).value }}"
+      delegate_to: localhost
+      vars:
+        ansible_connection: local
+        ansible_become: false
+      no_log: true

ansible/playbooks/emergency-access.yml

@@ -0,0 +1,42 @@
+---
+- name: Bootstrap emergency-bot control identity
+  hosts: emergency-bot
+  gather_facts: false
+  tasks:
+    - name: Create emergency-bot control identity
+      ansible.builtin.include_role:
+        name: emergency_bot
+        tasks_from: bootstrap
+
+- name: Configure mini-pc reverse SSH client and restricted control path
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    emergency_vps_host_key: "{{ lookup('env', 'EMERGENCY_VPS_HOST_KEY') }}"
+    emergency_bot_control_public_key: "{{ hostvars['emergency-bot'].emergency_bot_control_public_key }}"
+  tasks:
+    - name: Configure mini-pc emergency access client
+      ansible.builtin.include_role:
+        name: emergency_access
+        tasks_from: client
+
+- name: Configure ru-vps reverse SSH endpoint
+  hosts: ru-vps
+  gather_facts: false
+  vars:
+    emergency_reverse_public_key: "{{ hostvars['mini-pc'].emergency_reverse_public_key }}"
+  tasks:
+    - name: Configure ru-vps emergency access endpoint
+      ansible.builtin.include_role:
+        name: emergency_access
+        tasks_from: endpoint
+
+- name: Configure and enable emergency Telegram bot
+  hosts: emergency-bot
+  gather_facts: false
+  vars:
+    emergency_bot_token: "{{ lookup('env', 'EMERGENCY_BOT_TOKEN') }}"
+    emergency_bot_allowed_user_ids: "{{ lookup('env', 'EMERGENCY_ALLOWED_USER_IDS') }}"
+    emergency_mini_pc_host_key: "{{ lookup('env', 'EMERGENCY_MINI_PC_HOST_KEY') }}"
+  roles:
+    - role: emergency_bot

ansible/playbooks/gitea-update.yml

@@ -0,0 +1,45 @@
+---
+- name: Create and verify Gitea backup before update
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Create a fresh Gitea offsite backup
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-restic-offsite-gitea.service
+      changed_when: true
+
+    - name: Run Gitea offsite backup audit
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-gitea.service
+      changed_when: true
+
+- import_playbook: pve-gitea.yml
+
+- name: Verify Gitea public endpoints after update
+  hosts: ru-vps
+  gather_facts: false
+  tasks:
+    - name: Check Gitea HTTPS endpoint
+      ansible.builtin.uri:
+        url: https://git.ada-dev.ru/
+        status_code: 200
+        return_content: false
+      register: gitea_https_check
+      retries: 24
+      delay: 5
+      until: gitea_https_check.status == 200
+
+    - name: Check Gitea SSH port from ru-vps
+      ansible.builtin.wait_for:
+        host: 192.168.1.25
+        port: 2222
+        state: started
+        timeout: 30

ansible/playbooks/grimmory-update.yml

@@ -0,0 +1,106 @@
+---
+- name: Create and verify Grimmory backup before update
+  hosts: grimmory
+  gather_facts: false
+  tasks:
+    - name: Create a fresh Grimmory offsite backup
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-restic-offsite-grimmory.service
+      changed_when: true
+
+    - name: Run Grimmory offsite backup audit
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-grimmory.service
+      changed_when: true
+
+- name: Create a fresh Grimmory PBS backup
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Read Grimmory LXC config
+      ansible.builtin.command:
+        argv:
+          - pct
+          - config
+          - "149"
+      register: grimmory_pct_config
+      changed_when: false
+
+    - name: Assert VMID 149 belongs to Grimmory
+      ansible.builtin.assert:
+        that:
+          - grimmory_pct_hostname_line != ""
+          - grimmory_pct_hostname == "grimmory"
+        fail_msg: >-
+          Refusing to run vzdump 149 because pct config hostname is not grimmory:
+          {{ grimmory_pct_hostname_line | default('missing hostname line') }}
+      vars:
+        grimmory_pct_hostname_line: >-
+          {{ (grimmory_pct_config.stdout_lines | select('match', '^hostname:\\s+') | list | first | default('')) }}
+        grimmory_pct_hostname: >-
+          {{ grimmory_pct_hostname_line | regex_replace('^hostname:\\s*', '') }}
+
+    - name: Check for active Proxmox backup before Grimmory PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: grimmory_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before Grimmory PBS backup
+      ansible.builtin.assert:
+        that:
+          - grimmory_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on cloud-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh Grimmory PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "149"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+- name: Run PBS backup audit on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Run current PBS backup audit on mini-pc
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-grimmory.yml
+
+- name: Verify Grimmory public endpoint after update
+  hosts: ru-vps
+  gather_facts: false
+  tasks:
+    - name: Check Grimmory public health endpoint
+      ansible.builtin.uri:
+        url: https://books.ada-dev.ru/api/v1/healthcheck
+        status_code: 200
+        return_content: false
+      register: grimmory_public_health
+      retries: 24
+      delay: 5
+      until: grimmory_public_health.status == 200

ansible/playbooks/gyro.yml

@@ -0,0 +1,6 @@
+---
+- name: Configure Gyro investment allocator host
+  hosts: gyro
+  gather_facts: true
+  roles:
+    - role: gyro

ansible/playbooks/mihomo-update.yml

@@ -0,0 +1,96 @@
+---
+- name: Verify Mihomo PBS audit before update
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read VMID 143 configuration
+      ansible.builtin.command: "pct config 143"
+      register: mihomo_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to run backup unless VMID 143 is Mihomo
+      ansible.builtin.assert:
+        that:
+          - mihomo_pct_config.rc == 0
+          - mihomo_update_hostname == 'mihomo'
+        fail_msg: VMID 143 must be the Mihomo container before backup.
+      vars:
+        mihomo_update_hostname: >-
+          {{ mihomo_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Check for active Proxmox backup before Mihomo PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: mihomo_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before Mihomo PBS backup
+      ansible.builtin.assert:
+        that:
+          - mihomo_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on mini-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh Mihomo PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "143"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+    - name: Run current PBS backup audit on mini-pc
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-mihomo.yml
+
+- name: Verify Mihomo after update
+  hosts: mihomo
+  gather_facts: false
+  tasks:
+    - name: Wait for Mihomo proxy TCP ports
+      ansible.builtin.wait_for:
+        host: 127.0.0.1
+        port: "{{ item }}"
+        state: started
+        timeout: 120
+      loop:
+        - 7890
+        - 7891
+        - 9090
+
+    - name: Check Mihomo UI endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:8080/
+        status_code: 200
+      register: mihomo_ui_health
+      retries: 24
+      delay: 5
+      until: mihomo_ui_health.status == 200
+
+    - name: Check Mihomo controller /version endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:9090/version
+        status_code: 200
+      register: mihomo_controller_health
+      retries: 24
+      delay: 5
+      until: mihomo_controller_health.status == 200

ansible/playbooks/monitoring.yml

@@ -0,0 +1,26 @@
+---
+- name: Configure Prometheus exporters
+  hosts: monitoring_exporters
+  gather_facts: false
+  roles:
+    - role: monitoring_exporter
+
+- name: Configure SMART exporters on Proxmox nodes
+  hosts: monitoring_smart_exporters
+  gather_facts: false
+  roles:
+    - role: monitoring_exporter
+      monitoring_exporter_install_node: false
+      monitoring_exporter_install_smart: true
+
+- name: Configure ru-vps external probes
+  hosts: ru-vps
+  gather_facts: false
+  roles:
+    - role: monitoring_blackbox
+
+- name: Configure central monitoring stack
+  hosts: monitoring_server
+  gather_facts: false
+  roles:
+    - role: monitoring_server

ansible/playbooks/offsite-restic-yadisk.yml

@@ -0,0 +1,74 @@
+- name: Configure Gitea offsite backup to Yandex Disk
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    offsite_profile: gitea
+    offsite_repository: rclone:yadisk:System/Backups/HomeLab/restic/gitea
+    offsite_source_path: /opt/data/gitea
+    offsite_sqlite_db: /opt/data/gitea/gitea/gitea.db
+    offsite_backup_tag: gitea,cloud-pc,yadisk
+    offsite_timer_oncalendar: "*-*-* 04:15:00"
+    offsite_rclone_config_local: ~/.config/rclone/rclone.conf
+    offsite_restic_password_local: "{{ playbook_dir }}/../generated/restic-offsite-password"
+    offsite_excludes:
+      - /opt/data/gitea/gitea/gitea.db
+      - /opt/data/gitea/gitea/gitea.db-shm
+      - /opt/data/gitea/gitea/gitea.db-wal
+      - /opt/data/gitea/gitea/log/**
+      - /opt/data/gitea/gitea/sessions/**
+      - /opt/data/gitea/gitea/queues/**
+      - /opt/data/gitea/gitea/tmp/**
+  tasks:
+    - name: Configure restic offsite profile
+      ansible.builtin.include_tasks: ../tasks/offsite-restic-profile.yml
+
+- name: Configure Vaultwarden offsite backup to Yandex Disk
+  hosts: vaultwarden
+  gather_facts: false
+  vars:
+    ansible_become: false
+    offsite_profile: vaultwarden
+    offsite_repository: rclone:yadisk:System/Backups/HomeLab/restic/vaultwarden
+    offsite_source_path: /opt/vaultwarden/data
+    offsite_sqlite_db: /opt/vaultwarden/data/db.sqlite3
+    offsite_backup_tag: vaultwarden,mini-pc,yadisk
+    offsite_timer_oncalendar: "*-*-* 04:45:00"
+    offsite_rclone_config_local: ~/.config/rclone/rclone.conf
+    offsite_restic_password_local: "{{ playbook_dir }}/../generated/restic-offsite-password"
+    offsite_excludes:
+      - /opt/vaultwarden/data/db.sqlite3
+      - /opt/vaultwarden/data/db.sqlite3-shm
+      - /opt/vaultwarden/data/db.sqlite3-wal
+      - /opt/vaultwarden/data/tmp/**
+  tasks:
+    - name: Configure restic offsite profile
+      ansible.builtin.include_tasks: ../tasks/offsite-restic-profile.yml
+
+- name: Configure Grimmory offsite backup to Yandex Disk
+  hosts: grimmory
+  gather_facts: false
+  vars:
+    ansible_become: false
+    offsite_profile: grimmory
+    offsite_repository: rclone:yadisk:System/Backups/HomeLab/restic/grimmory
+    offsite_source_path: /opt/grimmory
+    offsite_sqlite_db: ""
+    offsite_mariadb_container: grimmory-mariadb
+    offsite_mariadb_database: grimmory
+    offsite_mariadb_user: grimmory
+    offsite_mariadb_env: /opt/grimmory/.env
+    offsite_mariadb_dump_name: grimmory.sql
+    offsite_backup_tag: grimmory,cloud-pc,yadisk
+    offsite_timer_oncalendar: "*-*-* 05:15:00"
+    offsite_rclone_config_local: ~/.config/rclone/rclone.conf
+    offsite_restic_password_local: "{{ playbook_dir }}/../generated/restic-offsite-password"
+    offsite_excludes:
+      - /opt/grimmory/books/**
+      - /opt/grimmory/bookdrop/**
+      - /opt/grimmory/mariadb/**
+      - /opt/grimmory/backup-staging/**
+      - /opt/grimmory/**/heapdump*.hprof
+      - /opt/grimmory/**/cache/**
+  tasks:
+    - name: Configure restic offsite profile
+      ansible.builtin.include_tasks: ../tasks/offsite-restic-profile.yml

ansible/playbooks/openvpn-check.yml

@@ -10,10 +10,22 @@
       ansible.builtin.command: nc -vz -w 5 192.168.1.10 8006
       changed_when: false
 
+    - name: Check mini-pc SSH through OpenVPN gateway
+      ansible.builtin.command: nc -vz -w 5 192.168.1.10 22
+      changed_when: false
+
     - name: Check cloud-pc PVE port through OpenVPN gateway
       ansible.builtin.command: nc -vz -w 5 192.168.1.5 8006
       changed_when: false
 
+    - name: Check cloud-pc SSH through OpenVPN gateway
+      ansible.builtin.command: nc -vz -w 5 192.168.1.5 22
+      changed_when: false
+
     - name: Check PBS port through OpenVPN gateway
       ansible.builtin.command: nc -vz -w 5 192.168.1.20 8007
       changed_when: false
+
+    - name: Check PBS SSH through OpenVPN gateway
+      ansible.builtin.command: nc -vz -w 5 192.168.1.20 22
+      changed_when: false

ansible/playbooks/pve-adguard.yml

@@ -0,0 +1,247 @@
+---
+- name: Create AdGuard Home LXC on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    adguard_vmid: 144
+    adguard_hostname: adguard
+    adguard_ip: 192.168.1.28/24
+    adguard_gateway: 192.168.1.1
+    adguard_rootfs: local-lvm:8
+    adguard_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    adguard_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+    adguard_image: adguard/adguardhome:v0.107.78@sha256:2c127294fa5f96151d9d3a433fb9d66c17e4d18cf698c2b04372a80e26fdd26f
+  handlers:
+    - name: restart adguard lxc
+      ansible.builtin.shell: "pct stop {{ adguard_vmid }} || true; pct start {{ adguard_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if AdGuard LXC exists
+      ansible.builtin.command: "pct config {{ adguard_vmid }}"
+      register: adguard_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID {{ adguard_vmid }}
+      ansible.builtin.assert:
+        that:
+          - adguard_pct_config.rc != 0 or adguard_existing_hostname == adguard_hostname
+        fail_msg: VMID {{ adguard_vmid }} already exists and is not the AdGuard container.
+      vars:
+        adguard_existing_hostname: >-
+          {{ adguard_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Install AdGuard LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/adguard-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', adguard_pubkey_file) }}\n"
+      when: adguard_pct_config.rc != 0
+
+    - name: Create AdGuard LXC
+      ansible.builtin.command: >-
+        pct create {{ adguard_vmid }} {{ adguard_ostemplate }}
+        --hostname {{ adguard_hostname }}
+        --rootfs {{ adguard_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge=vmbr0,gw={{ adguard_gateway }},ip={{ adguard_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=40
+        --cmode shell
+        --ssh-public-keys /tmp/adguard-lxc.pub
+      when: adguard_pct_config.rc != 0
+
+    - name: Start AdGuard LXC
+      ansible.builtin.command: "pct start {{ adguard_vmid }}"
+      register: adguard_pct_start
+      changed_when: adguard_pct_start.rc == 0
+      failed_when: adguard_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in AdGuard LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ adguard_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart adguard lxc
+
+    - name: Bind mount FUSE device in AdGuard LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ adguard_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart adguard lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for AdGuard SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: adguard
+      vars:
+        ansible_become: false
+
+- name: Configure AdGuard Home inside LXC
+  hosts: adguard
+  gather_facts: true
+  vars:
+    ansible_become: false
+    adguard_image: adguard/adguardhome:v0.107.78@sha256:2c127294fa5f96151d9d3a433fb9d66c17e4d18cf698c2b04372a80e26fdd26f
+  tasks:
+    - name: Install Docker packages
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - ca-certificates
+          - curl
+          - dnsutils
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: adguard_docker_daemon_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if adguard_docker_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Ensure AdGuard data directories exist
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - /opt/adguard/work
+        - /opt/adguard/conf
+
+    - name: Check if the configured AdGuard image is present
+      ansible.builtin.command: "docker image inspect {{ adguard_image }}"
+      register: adguard_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured AdGuard image
+      ansible.builtin.command: "docker pull {{ adguard_image }}"
+      when: adguard_image_inspect.rc != 0
+      register: adguard_image_pull
+      changed_when: true
+
+    - name: Install AdGuard Home systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/adguard.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=AdGuard Home container
+          After=docker.service
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f adguard
+          ExecStart=/usr/bin/docker run --rm --name adguard --pull never \
+            -p 53:53/tcp -p 53:53/udp -p 80:80/tcp -p 3000:3000/tcp \
+            -v /opt/adguard/work:/opt/adguardhome/work \
+            -v /opt/adguard/conf:/opt/adguardhome/conf \
+            {{ adguard_image }}
+          ExecStop=/usr/bin/docker stop adguard
+
+          [Install]
+          WantedBy=multi-user.target
+      register: adguard_unit
+
+    - name: Remove obsolete AdGuard post-start validation unit
+      ansible.builtin.file:
+        path: /etc/systemd/system/adguard-post-start.service
+        state: absent
+      register: adguard_post_start_unit_removed
+
+    - name: Reload systemd when AdGuard units change
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: adguard_unit.changed or adguard_post_start_unit_removed.changed
+
+    - name: Enable and start AdGuard Home
+      ansible.builtin.systemd:
+        name: adguard
+        state: "{{ 'restarted' if adguard_unit.changed or adguard_image_pull.changed else 'started' }}"
+        enabled: true
+
+    - name: Wait for AdGuard HTTP root
+      ansible.builtin.uri:
+        url: http://127.0.0.1/
+        status_code: [200, 302]
+        return_content: false
+      register: adguard_http_root
+      retries: 24
+      delay: 5
+      until: adguard_http_root.status in [200, 302]
+
+    - name: Verify AdGuard DNS over UDP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_udp
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_udp.rc == 0 and '127.0.0.1' in adguard_dns_udp.stdout
+
+    - name: Verify AdGuard DNS over TCP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +tcp
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_tcp
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_tcp.rc == 0 and '127.0.0.1' in adguard_dns_tcp.stdout

ansible/playbooks/pve-backup-jobs.yml

@@ -0,0 +1,114 @@
+- name: Configure Proxmox backup jobs
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    pve_backup_jobs:
+      - id: homelab-pbs-daily-cloud
+        comment: Daily PBS backup for cloud-pc service containers
+        node: cloud-pc
+        vmid: 141,145,146,147,149
+        storage: pbs
+        schedule: "02:10"
+        mode: snapshot
+        exclude_path: /var/lib/docker/fuse-overlayfs/*/merged
+      - id: homelab-pbs-daily-mini
+        comment: Daily PBS backup for mini-pc service containers
+        node: mini-pc
+        vmid: 132,140,142,143,144,150
+        storage: pbs
+        schedule: "02:40"
+        mode: snapshot
+        exclude_path: /var/lib/docker/fuse-overlayfs/*/merged
+      - id: homelab-local-weekly-pbs
+        comment: Weekly local backup for PBS container rootfs/config
+        node: cloud-pc
+        vmid: 120
+        storage: backup
+        schedule: "Sun 03:30"
+        mode: snapshot
+        prune_backups: keep-last=2
+
+  tasks:
+    - name: Get existing Proxmox backup jobs
+      ansible.builtin.command: pvesh get /cluster/backup --output-format json
+      register: pve_backup_jobs_existing_raw
+      changed_when: false
+
+    - name: Parse existing Proxmox backup jobs
+      ansible.builtin.set_fact:
+        pve_backup_jobs_existing: "{{ pve_backup_jobs_existing_raw.stdout | from_json }}"
+
+    - name: Create missing Proxmox backup jobs
+      ansible.builtin.command: >-
+        pvesh create /cluster/backup
+        --id {{ item.id }}
+        --enabled 1
+        --node {{ item.node }}
+        --vmid {{ item.vmid }}
+        --storage {{ item.storage }}
+        --schedule {{ item.schedule | quote }}
+        --mode {{ item.mode }}
+        {% if item.prune_backups is defined %}
+        --prune-backups {{ item.prune_backups }} --remove 1
+        {% else %}
+        --remove 0
+        {% endif %}
+        {% if item.exclude_path is defined %}
+        --exclude-path {{ item.exclude_path | quote }}
+        {% endif %}
+        --notes-template {{ '{{' }}guestname{{ '}}' }}
+        --comment {{ item.comment | quote }}
+      loop: "{{ pve_backup_jobs }}"
+      when: item.id not in (pve_backup_jobs_existing | map(attribute='id') | list)
+
+    - name: Update existing Proxmox backup jobs
+      ansible.builtin.command: >-
+        pvesh set /cluster/backup/{{ item.id }}
+        --enabled 1
+        --node {{ item.node }}
+        --vmid {{ item.vmid }}
+        --storage {{ item.storage }}
+        --schedule {{ item.schedule | quote }}
+        --mode {{ item.mode }}
+        {% if item.prune_backups is defined %}
+        --prune-backups {{ item.prune_backups }} --remove 1
+        {% else %}
+        --delete prune-backups --remove 0
+        {% endif %}
+        {% if item.exclude_path is defined %}
+        --exclude-path {{ item.exclude_path | quote }}
+        {% else %}
+        --delete exclude-path
+        {% endif %}
+        --notes-template {{ '{{' }}guestname{{ '}}' }}
+        --comment {{ item.comment | quote }}
+      loop: "{{ pve_backup_jobs }}"
+      vars:
+        current_job: >-
+          {{ pve_backup_jobs_existing | selectattr('id', 'equalto', item.id) | first }}
+      when:
+        - item.id in (pve_backup_jobs_existing | map(attribute='id') | list)
+        - >-
+          current_job.enabled | int != 1 or
+          current_job.node != item.node or
+          current_job.vmid | string != item.vmid | string or
+          current_job.storage != item.storage or
+          current_job.schedule != item.schedule or
+          current_job.mode != item.mode or
+          current_job.comment != item.comment or
+          current_job.remove | int != (1 if item.prune_backups is defined else 0) or
+          current_job['exclude-path'] | default([]) != ([item.exclude_path] if item.exclude_path is defined else []) or
+          (item.prune_backups is not defined and current_job['prune-backups'] is defined) or
+          (item.prune_backups is defined and
+           (current_job['prune-backups'] | default({})).get('keep-last', 0) | int !=
+           item.prune_backups | regex_replace('^keep-last=', '') | int)
+      changed_when: true
+
+    - name: Show configured Proxmox backup jobs
+      ansible.builtin.command: pvesh get /cluster/backup --output-format yaml
+      register: pve_backup_jobs_configured
+      changed_when: false
+
+    - name: Print configured Proxmox backup jobs
+      ansible.builtin.debug:
+        var: pve_backup_jobs_configured.stdout_lines

ansible/playbooks/pve-docker-test.yml

@@ -0,0 +1,125 @@
+---
+- name: Create Docker test LXC on cloud-pc
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    docker_test_vmid: 145
+    docker_test_hostname: docker-test
+    docker_test_ip: 192.168.1.29/24
+    docker_test_gateway: 192.168.1.1
+    docker_test_rootfs: data:8
+    docker_test_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    docker_test_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  handlers:
+    - name: restart docker test lxc
+      ansible.builtin.shell: "pct stop {{ docker_test_vmid }} || true; pct start {{ docker_test_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if Docker test LXC exists
+      ansible.builtin.command: "pct config {{ docker_test_vmid }}"
+      register: docker_test_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Install Docker test LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/docker-test-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', docker_test_pubkey_file) }}\n"
+      when: docker_test_pct_config.rc != 0
+
+    - name: Create Docker test LXC
+      ansible.builtin.command: >-
+        pct create {{ docker_test_vmid }} {{ docker_test_ostemplate }}
+        --hostname {{ docker_test_hostname }}
+        --rootfs {{ docker_test_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge=vmbr0,gw={{ docker_test_gateway }},ip={{ docker_test_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=50
+        --cmode shell
+        --ssh-public-keys /tmp/docker-test-lxc.pub
+      when: docker_test_pct_config.rc != 0
+
+    - name: Start Docker test LXC
+      ansible.builtin.command: "pct start {{ docker_test_vmid }}"
+      register: docker_test_pct_start
+      changed_when: docker_test_pct_start.rc == 0
+      failed_when: docker_test_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in Docker test LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ docker_test_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart docker test lxc
+
+    - name: Bind mount FUSE device in Docker test LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ docker_test_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart docker test lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for Docker test SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: docker-test
+      vars:
+        ansible_become: false
+
+- name: Configure Docker test host
+  hosts: docker-test
+  gather_facts: true
+  vars:
+    ansible_become: false
+  tasks:
+    - name: Install Docker packages
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - ca-certificates
+          - curl
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: docker_test_daemon_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if docker_test_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Verify Docker can run a container
+      ansible.builtin.command: docker run --rm hello-world
+      changed_when: false

ansible/playbooks/pve-emergency-bot.yml

@@ -0,0 +1,30 @@
+---
+- name: Create emergency-bot LXC on mini-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 148
+    pve_lxc_node: mini-pc
+    pve_lxc_hostname: emergency-bot
+    pve_lxc_ip: 192.168.1.32/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: local-lvm:4
+    pve_lxc_cores: 1
+    pve_lxc_memory: 512
+    pve_lxc_swap: 256
+    pve_lxc_startup: order=70
+    pve_lxc_ostemplate: "{{ lookup('env', 'PVE_LXC_OSTEMPLATE') | default('local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst', true) }}"
+  roles:
+    - role: pve_lxc
+
+- name: Wait for emergency-bot SSH
+  hosts: emergency-bot
+  gather_facts: false
+  tasks:
+    - name: Wait for emergency-bot to accept SSH connections
+      ansible.builtin.wait_for_connection:
+        timeout: 120

ansible/playbooks/pve-gitea.yml

@@ -25,6 +25,19 @@
       changed_when: false
       failed_when: false
 
+    - name: Refuse to modify a foreign Gitea VMID
+      ansible.builtin.assert:
+        that:
+          - gitea_pct_config.rc != 0 or gitea_existing_hostname == 'gitea'
+        fail_msg: VMID {{ gitea_vmid }} already exists and is not the Gitea container.
+      vars:
+        gitea_existing_hostname: >-
+          {{ gitea_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
     - name: Install Gitea LXC SSH public key on PVE host
       ansible.builtin.copy:
         dest: /tmp/gitea-lxc.pub
@@ -103,7 +116,7 @@
   vars:
     ansible_become: false
     gitea_data_dir: /opt/gitea/data
-    gitea_image: gitea/gitea:latest
+    gitea_image: gitea/gitea:1.27.1@sha256:b64126cf5c3f4e5f0f231b510bb13715f6cb8e508188b44de90bdb9a04f3055d
     gitea_container_name: gitea
     gitea_http_port: 3000
     gitea_ssh_port: 2222
@@ -154,6 +167,18 @@
         group: "1000"
         mode: "0750"
 
+    - name: Check if the configured Gitea image is present
+      ansible.builtin.command: "docker image inspect {{ gitea_image }}"
+      register: gitea_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Gitea image
+      ansible.builtin.command: "docker pull {{ gitea_image }}"
+      when: gitea_image_inspect.rc != 0
+      register: gitea_image_pull
+      changed_when: true
+
     - name: Install Gitea systemd unit
       ansible.builtin.copy:
         dest: /etc/systemd/system/gitea.service
@@ -172,7 +197,7 @@
           ExecStartPre=-/usr/bin/docker rm -f {{ gitea_container_name }}
           ExecStart=/usr/bin/docker run --rm \
             --name {{ gitea_container_name }} \
-            --pull always \
+            --pull never \
             -p {{ gitea_http_port }}:3000 \
             -p {{ gitea_ssh_port }}:22 \
             -v {{ gitea_data_dir }}:/data \
@@ -193,5 +218,14 @@
     - name: Enable and start Gitea
       ansible.builtin.systemd:
         name: gitea
-        state: started
+        state: "{{ 'restarted' if gitea_unit.changed or gitea_image_pull.changed else 'started' }}"
         enabled: true
+
+    - name: Wait for Gitea HTTP health endpoint
+      ansible.builtin.uri:
+        url: "http://127.0.0.1:{{ gitea_http_port }}/api/healthz"
+        status_code: 200
+      register: gitea_health
+      retries: 24
+      delay: 5
+      until: gitea_health.status == 200

ansible/playbooks/pve-grimmory.yml

@@ -0,0 +1,431 @@
+---
+- name: Guard Grimmory VMID before API updates
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 149 configuration
+      ansible.builtin.command: pct config 149
+      register: grimmory_existing_vmid
+      check_mode: false
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 149
+      ansible.builtin.assert:
+        that:
+          - grimmory_existing_vmid.rc != 0 or grimmory_existing_hostname == 'grimmory'
+        fail_msg: VMID 149 already exists and is not the Grimmory container.
+      vars:
+        grimmory_existing_hostname: >-
+          {{ grimmory_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+- name: Create Grimmory LXC on cloud-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 149
+    pve_lxc_node: cloud-pc
+    pve_lxc_hostname: grimmory
+    pve_lxc_ip: 192.168.1.34/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: data:64
+    pve_lxc_cores: 2
+    pve_lxc_memory: 4096
+    pve_lxc_swap: 1024
+    pve_lxc_startup: order=100
+    pve_lxc_unprivileged: true
+    pve_lxc_update: false
+    pve_lxc_features:
+      - nesting=1
+    pve_lxc_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+  roles:
+    - role: pve_lxc
+
+- name: Configure Grimmory LXC devices
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    grimmory_vmid: 149
+  handlers:
+    - name: restart Grimmory LXC
+      ansible.builtin.command: "pct reboot {{ grimmory_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Read Grimmory LXC configuration
+      ansible.builtin.command: "pct config {{ grimmory_vmid }}"
+      register: grimmory_lxc_config
+      changed_when: false
+
+    - name: Require expected Grimmory LXC properties
+      ansible.builtin.assert:
+        that:
+          - "'unprivileged: 1' in grimmory_lxc_config.stdout"
+          - "'rootfs: data:' in grimmory_lxc_config.stdout"
+          - "'onboot: 1' in grimmory_lxc_config.stdout"
+
+    - name: Configure Grimmory LXC network
+      ansible.builtin.command: >-
+        pct set {{ grimmory_vmid }}
+        --net0 name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,ip=192.168.1.34/24
+      vars:
+        grimmory_net0: >-
+          {{ grimmory_lxc_config.stdout_lines
+             | select('match', '^net0: ')
+             | first
+             | default('') }}
+      when: >-
+        'name=eth0' not in grimmory_net0 or
+        'bridge=vmbr0' not in grimmory_net0 or
+        'firewall=1' not in grimmory_net0 or
+        'gw=192.168.1.1' not in grimmory_net0 or
+        'ip=192.168.1.34/24' not in grimmory_net0
+      notify: restart Grimmory LXC
+
+    - name: Enable keyctl for Docker in Grimmory LXC
+      ansible.builtin.command: "pct set {{ grimmory_vmid }} --features nesting=1,keyctl=1"
+      when: "'nesting=1' not in grimmory_lxc_config.stdout or 'keyctl=1' not in grimmory_lxc_config.stdout"
+      notify: restart Grimmory LXC
+
+    - name: Allow FUSE device in Grimmory LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ grimmory_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart Grimmory LXC
+
+    - name: Bind mount FUSE device in Grimmory LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ grimmory_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart Grimmory LXC
+
+    - name: Apply pending Grimmory LXC configuration
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for Grimmory SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 180
+      delegate_to: grimmory
+      vars:
+        ansible_become: false
+
+- name: Configure Grimmory runtime
+  hosts: grimmory
+  gather_facts: true
+  vars:
+    ansible_become: false
+    grimmory_root: /opt/grimmory
+    grimmory_image: grimmory/grimmory:v3.2.4@sha256:dfa7afdfcf25d649fd664497a62385dd00cd9678c37546e182c172e41c8e80cb
+    grimmory_mariadb_image: lscr.io/linuxserver/mariadb:11.4.8@sha256:91de7f701bc7fc3a424b81beafca7a7c6c4c5b7c8be6afd2ae148698695c0b0c
+  tasks:
+    - name: Install Grimmory runtime packages
+      ansible.builtin.apt:
+        name:
+          - ca-certificates
+          - curl
+          - docker-compose
+          - docker.io
+          - fuse-overlayfs
+          - mariadb-client
+          - openssl
+          - prometheus-node-exporter
+          - ufw
+        state: present
+        update_cache: true
+
+    - name: Verify FUSE device exists
+      ansible.builtin.stat:
+        path: /dev/fuse
+      register: grimmory_fuse
+
+    - name: Require FUSE device
+      ansible.builtin.assert:
+        that:
+          - grimmory_fuse.stat.exists
+          - grimmory_fuse.stat.ischr
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: grimmory_docker_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        enabled: true
+        state: "{{ 'restarted' if grimmory_docker_config.changed else 'started' }}"
+
+    - name: Allow SSH from the HomeLab LAN
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ homelab_lan_cidr }}"
+
+    - name: Allow SSH from the OpenVPN network
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ openvpn_network_cidr }}"
+
+    - name: Allow Grimmory from the HomeLab LAN
+      community.general.ufw:
+        rule: allow
+        port: "6060"
+        proto: tcp
+        src: "{{ homelab_lan_cidr }}"
+
+    - name: Allow Grimmory from the OpenVPN network
+      community.general.ufw:
+        rule: allow
+        port: "6060"
+        proto: tcp
+        src: "{{ openvpn_network_cidr }}"
+
+    - name: Allow Node Exporter from the monitoring LXC
+      community.general.ufw:
+        rule: allow
+        port: "9100"
+        proto: tcp
+        src: 192.168.1.30
+
+    - name: Enable restrictive Grimmory firewall
+      community.general.ufw:
+        state: enabled
+        policy: deny
+        direction: incoming
+
+    - name: Create Grimmory directories
+      ansible.builtin.file:
+        path: "{{ item.path }}"
+        state: directory
+        owner: "{{ item.owner }}"
+        group: "{{ item.group }}"
+        mode: "{{ item.mode }}"
+      loop:
+        - { path: "{{ grimmory_root }}", owner: root, group: root, mode: "0750" }
+        - { path: "{{ grimmory_root }}/data", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/books", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/bookdrop", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/mariadb", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/backup-staging", owner: root, group: root, mode: "0700" }
+
+    - name: Generate Grimmory secrets once
+      ansible.builtin.shell: |
+        set -eu
+        umask 077
+        if [ -e {{ grimmory_root }}/.env ]; then
+          exit 0
+        fi
+        db_password=$(openssl rand -hex 32)
+        root_password=$(openssl rand -hex 32)
+        cat > {{ grimmory_root }}/.env <<EOF
+        TZ=Europe/Moscow
+        APP_USER_ID=1000
+        APP_GROUP_ID=1000
+        DB_USER_ID=1000
+        DB_GROUP_ID=1000
+        DB_PASSWORD=$db_password
+        MYSQL_ROOT_PASSWORD=$root_password
+        EOF
+        printf created
+      args:
+        executable: /bin/sh
+      register: grimmory_secrets
+      changed_when: grimmory_secrets.stdout == 'created'
+      no_log: true
+
+    - name: Enforce Grimmory secret file permissions
+      ansible.builtin.file:
+        path: "{{ grimmory_root }}/.env"
+        owner: root
+        group: root
+        mode: "0600"
+
+    - name: Install Grimmory environment example
+      ansible.builtin.copy:
+        dest: "{{ grimmory_root }}/.env.example"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          TZ=Europe/Moscow
+          APP_USER_ID=1000
+          APP_GROUP_ID=1000
+          DB_USER_ID=1000
+          DB_GROUP_ID=1000
+          DB_PASSWORD=replace-with-random-password
+          MYSQL_ROOT_PASSWORD=replace-with-separate-random-password
+
+    - name: Install Grimmory Compose configuration
+      ansible.builtin.copy:
+        dest: "{{ grimmory_root }}/compose.yml"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          services:
+            grimmory:
+              image: {{ grimmory_image }}
+              container_name: grimmory
+              environment:
+                USER_ID: "${APP_USER_ID}"
+                GROUP_ID: "${APP_GROUP_ID}"
+                TZ: "${TZ}"
+                DATABASE_URL: jdbc:mariadb://mariadb:3306/grimmory
+                DATABASE_USERNAME: grimmory
+                DATABASE_PASSWORD: "${DB_PASSWORD}"
+                DISK_TYPE: LOCAL
+                ALLOWED_ORIGINS: https://books.ada-dev.ru
+              depends_on:
+                mariadb:
+                  condition: service_healthy
+              ports:
+                - "192.168.1.34:6060:6060"
+              volumes:
+                - ./data:/app/data
+                - ./books:/books
+                - ./bookdrop:/bookdrop
+              restart: unless-stopped
+
+            mariadb:
+              image: {{ grimmory_mariadb_image }}
+              container_name: grimmory-mariadb
+              environment:
+                PUID: "${DB_USER_ID}"
+                PGID: "${DB_GROUP_ID}"
+                TZ: "${TZ}"
+                MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
+                MYSQL_DATABASE: grimmory
+                MYSQL_USER: grimmory
+                MYSQL_PASSWORD: "${DB_PASSWORD}"
+              volumes:
+                - ./mariadb:/config
+              healthcheck:
+                test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"]
+                interval: 5s
+                timeout: 5s
+                retries: 20
+              restart: unless-stopped
+      register: grimmory_compose
+
+    - name: Install Docker firewall script
+      ansible.builtin.copy:
+        dest: /usr/local/sbin/grimmory-docker-firewall
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          set -eu
+          iptables -N GRIMMORY-FILTER 2>/dev/null || true
+          iptables -F GRIMMORY-FILTER
+          iptables -A GRIMMORY-FILTER -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+          iptables -A GRIMMORY-FILTER -s {{ homelab_lan_cidr }} -p tcp -m conntrack --ctorigdst 192.168.1.34 --ctorigdstport 6060 -j ACCEPT
+          iptables -A GRIMMORY-FILTER -s {{ openvpn_network_cidr }} -p tcp -m conntrack --ctorigdst 192.168.1.34 --ctorigdstport 6060 -j ACCEPT
+          iptables -A GRIMMORY-FILTER -p tcp -m conntrack --ctorigdst 192.168.1.34 --ctorigdstport 6060 -j DROP
+          iptables -A GRIMMORY-FILTER -j RETURN
+          iptables -C DOCKER-USER -j GRIMMORY-FILTER 2>/dev/null || iptables -I DOCKER-USER 1 -j GRIMMORY-FILTER
+      register: grimmory_firewall_script
+
+    - name: Install Docker firewall service
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/grimmory-docker-firewall.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Restrict Grimmory Docker published port
+          After=docker.service
+          Requires=docker.service
+          Before=grimmory.service
+
+          [Service]
+          Type=oneshot
+          ExecStart=/usr/local/sbin/grimmory-docker-firewall
+          RemainAfterExit=yes
+
+          [Install]
+          WantedBy=multi-user.target
+      register: grimmory_firewall_unit
+
+    - name: Install Grimmory systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/grimmory.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Grimmory Compose stack
+          Wants=network-online.target
+          After=network-online.target docker.service grimmory-docker-firewall.service
+          Requires=docker.service grimmory-docker-firewall.service
+
+          [Service]
+          Type=oneshot
+          RemainAfterExit=yes
+          WorkingDirectory={{ grimmory_root }}
+          ExecStart=/usr/bin/docker compose -f {{ grimmory_root }}/compose.yml up -d --remove-orphans
+          ExecStop=/usr/bin/docker compose -f {{ grimmory_root }}/compose.yml down
+
+          [Install]
+          WantedBy=multi-user.target
+      register: grimmory_unit
+
+    - name: Reload systemd for Grimmory units
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: grimmory_firewall_unit.changed or grimmory_unit.changed
+
+    - name: Enable and apply Docker firewall
+      ansible.builtin.systemd:
+        name: grimmory-docker-firewall
+        enabled: true
+        state: "{{ 'restarted' if grimmory_firewall_script.changed or grimmory_firewall_unit.changed else 'started' }}"
+
+    - name: Validate Grimmory Compose configuration
+      ansible.builtin.command: docker compose -f {{ grimmory_root }}/compose.yml config --quiet
+      args:
+        chdir: "{{ grimmory_root }}"
+      changed_when: false
+      no_log: true
+
+    - name: Enable and start Grimmory
+      ansible.builtin.systemd:
+        name: grimmory
+        enabled: true
+        state: "{{ 'restarted' if grimmory_compose.changed or grimmory_unit.changed else 'started' }}"
+
+    - name: Wait for Grimmory health endpoint
+      ansible.builtin.uri:
+        url: http://192.168.1.34:6060/api/v1/healthcheck
+        status_code: 200
+      register: grimmory_health
+      retries: 120
+      delay: 5
+      until: grimmory_health.status == 200
+
+    - name: Verify Docker storage driver
+      ansible.builtin.command: docker info --format '{{ "{{" }}.Driver{{ "}}" }}'
+      register: grimmory_docker_driver
+      changed_when: false
+      failed_when: grimmory_docker_driver.stdout != 'fuse-overlayfs'

ansible/playbooks/pve-gyro.yml

@@ -0,0 +1,189 @@
+---
+- name: Guard Gyro VMID before API updates
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 150 configuration
+      ansible.builtin.command: pct config 150
+      register: gyro_existing_vmid
+      check_mode: false
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 150
+      ansible.builtin.assert:
+        that:
+          - gyro_existing_vmid.rc != 0 or gyro_existing_hostname == 'gyro'
+        fail_msg: VMID 150 already exists and is not the Gyro container.
+      vars:
+        gyro_existing_hostname: >-
+          {{ gyro_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+- name: Create Gyro LXC on mini-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 150
+    pve_lxc_node: mini-pc
+    pve_lxc_hostname: gyro
+    pve_lxc_ip: 192.168.1.35/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: local-lvm:2
+    pve_lxc_cores: 1
+    pve_lxc_memory: 512
+    pve_lxc_swap: 256
+    pve_lxc_startup: order=80
+    pve_lxc_unprivileged: true
+    pve_lxc_update: false
+    pve_lxc_features: []
+    pve_lxc_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+  roles:
+    - role: pve_lxc
+
+- name: Configure Gyro LXC isolation
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    gyro_vmid: 150
+  tasks:
+    - name: Read Gyro LXC configuration
+      ansible.builtin.command: "pct config {{ gyro_vmid }}"
+      register: gyro_lxc_config
+      changed_when: false
+
+    - name: Require expected Gyro LXC properties
+      ansible.builtin.assert:
+        that:
+          - "'hostname: gyro' in gyro_lxc_config.stdout"
+          - "'unprivileged: 1' in gyro_lxc_config.stdout"
+          - "'rootfs: local-lvm:' in gyro_lxc_config.stdout"
+          - "'onboot: 1' in gyro_lxc_config.stdout"
+
+    - name: Check for an existing Gyro firewall file
+      ansible.builtin.stat:
+        path: "/etc/pve/firewall/{{ gyro_vmid }}.fw"
+      register: gyro_firewall_file
+
+    - name: Preserve the existing Gyro firewall file
+      ansible.builtin.slurp:
+        src: "/etc/pve/firewall/{{ gyro_vmid }}.fw"
+      register: gyro_previous_firewall
+      when: gyro_firewall_file.stat.exists
+
+    - name: Apply and verify Gyro firewall
+      block:
+        - name: Render Proxmox firewall for Gyro
+          ansible.builtin.copy:
+            dest: "/tmp/gyro-{{ gyro_vmid }}.fw"
+            owner: root
+            group: root
+            mode: "0640"
+            content: |
+              [OPTIONS]
+              enable: 1
+              policy_in: DROP
+              policy_out: ACCEPT
+
+              [RULES]
+              IN ACCEPT -source {{ homelab_lan_cidr }} -p tcp -dport 22 -log nolog
+              IN ACCEPT -source {{ openvpn_network_cidr }} -p tcp -dport 22 -log nolog
+              OUT ACCEPT -dest 192.168.1.27 -p tcp -dport 7890 -log nolog
+              OUT DROP -dest {{ homelab_lan_cidr }} -log nolog
+          register: gyro_rendered_firewall
+          changed_when: false
+
+        - name: Compare rendered and active Gyro firewall
+          ansible.builtin.command: >-
+            cmp -s /tmp/gyro-{{ gyro_vmid }}.fw /etc/pve/firewall/{{ gyro_vmid }}.fw
+          register: gyro_firewall_comparison
+          changed_when: false
+          failed_when: gyro_firewall_comparison.rc not in [0, 1]
+          when: gyro_firewall_file.stat.exists
+
+        - name: Install Proxmox firewall for Gyro
+          ansible.builtin.command: >-
+            cp /tmp/gyro-{{ gyro_vmid }}.fw /etc/pve/firewall/{{ gyro_vmid }}.fw
+          when: not gyro_firewall_file.stat.exists or gyro_firewall_comparison.rc != 0
+          changed_when: true
+
+        - name: Validate Proxmox firewall configuration
+          ansible.builtin.command: pve-firewall compile
+          changed_when: false
+
+        - name: Read cluster firewall options
+          ansible.builtin.command: pvesh get /cluster/firewall/options --output-format json
+          register: gyro_cluster_firewall_options
+          changed_when: false
+
+        - name: Report staged Proxmox firewall state
+          ansible.builtin.debug:
+            msg: >-
+              {{ 'Gyro Proxmox firewall is active.'
+                 if (gyro_cluster_firewall_options.stdout | from_json).enable | default(0) | int == 1
+                 else 'Gyro Proxmox firewall is staged but inactive because the cluster firewall is disabled; UFW remains the enforced isolation layer.' }}
+
+        - name: Verify SSH remains reachable through the firewall
+          ansible.builtin.wait_for_connection:
+            timeout: 180
+          delegate_to: gyro
+          vars:
+            ansible_become: false
+      rescue:
+        - name: Restore the previous Gyro firewall file
+          ansible.builtin.copy:
+            dest: "/tmp/gyro-{{ gyro_vmid }}-previous.fw"
+            content: "{{ gyro_previous_firewall.content | b64decode }}"
+            owner: root
+            group: root
+            mode: "0640"
+          when: gyro_firewall_file.stat.exists
+
+        - name: Reinstall the previous Gyro firewall file
+          ansible.builtin.command: >-
+            cp /tmp/gyro-{{ gyro_vmid }}-previous.fw /etc/pve/firewall/{{ gyro_vmid }}.fw
+          when: gyro_firewall_file.stat.exists
+          changed_when: true
+
+        - name: Remove the failed new Gyro firewall file
+          ansible.builtin.file:
+            path: "/etc/pve/firewall/{{ gyro_vmid }}.fw"
+            state: absent
+          when: not gyro_firewall_file.stat.exists
+
+        - name: Recompile restored Proxmox firewall configuration
+          ansible.builtin.command: pve-firewall compile
+          changed_when: false
+
+        - name: Remove temporary Gyro firewall files after rollback
+          ansible.builtin.file:
+            path: "{{ item }}"
+            state: absent
+          loop:
+            - "/tmp/gyro-{{ gyro_vmid }}.fw"
+            - "/tmp/gyro-{{ gyro_vmid }}-previous.fw"
+
+        - name: Stop after rolling back the Gyro firewall
+          ansible.builtin.fail:
+            msg: Gyro firewall validation or SSH reachability failed; the previous firewall state was restored.
+
+    - name: Remove temporary Gyro firewall file
+      ansible.builtin.file:
+        path: "/tmp/gyro-{{ gyro_vmid }}.fw"
+        state: absent
+      changed_when: false
+
+- name: Wait for Gyro SSH
+  hosts: gyro
+  gather_facts: false
+  tasks:
+    - name: Wait for Gyro to accept SSH connections
+      ansible.builtin.wait_for_connection:
+        timeout: 180

ansible/playbooks/pve-hermes-ai.yml

@@ -0,0 +1,334 @@
+---
+- name: Create Hermes AI LXC on cloud-pc
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    hermes_ai_vmid: 147
+    hermes_ai_hostname: hermes-ai
+    hermes_ai_ip: 192.168.1.31/24
+    hermes_ai_gateway: 192.168.1.1
+    hermes_ai_rootfs: data:24
+    hermes_ai_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    hermes_ai_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  tasks:
+    - name: Check if Hermes AI LXC exists
+      ansible.builtin.command: "pct config {{ hermes_ai_vmid }}"
+      register: hermes_ai_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Install Hermes AI LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/hermes-ai-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', hermes_ai_pubkey_file) }}\n"
+      when: hermes_ai_pct_config.rc != 0
+
+    - name: Create Hermes AI LXC
+      ansible.builtin.command: >-
+        pct create {{ hermes_ai_vmid }} {{ hermes_ai_ostemplate }}
+        --hostname {{ hermes_ai_hostname }}
+        --rootfs {{ hermes_ai_rootfs }}
+        --cores 2
+        --memory 4096
+        --swap 512
+        --net0 name=eth0,bridge=vmbr0,gw={{ hermes_ai_gateway }},ip={{ hermes_ai_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=90
+        --cmode shell
+        --ssh-public-keys /tmp/hermes-ai-lxc.pub
+      when: hermes_ai_pct_config.rc != 0
+
+    - name: Start Hermes AI LXC
+      ansible.builtin.command: "pct start {{ hermes_ai_vmid }}"
+      register: hermes_ai_pct_start
+      changed_when: hermes_ai_pct_start.rc == 0
+      failed_when: hermes_ai_pct_start.rc not in [0, 255]
+
+- name: Configure Hermes AI runtime
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    hermes_ai_vmid: 147
+  handlers:
+    - name: restart Hermes AI LXC
+      ansible.builtin.command: "pct reboot {{ hermes_ai_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Read Hermes AI LXC configuration
+      ansible.builtin.command: "pct config {{ hermes_ai_vmid }}"
+      register: hermes_ai_lxc_config
+      changed_when: false
+
+    - name: Enable keyctl for Docker in Hermes AI LXC
+      ansible.builtin.command: "pct set {{ hermes_ai_vmid }} --features nesting=1,keyctl=1"
+      when: "'keyctl=1' not in hermes_ai_lxc_config.stdout"
+      notify: restart Hermes AI LXC
+
+    - name: Allow FUSE device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Bind mount FUSE device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Check TUN device on PVE host
+      ansible.builtin.stat:
+        path: /dev/net/tun
+      register: hermes_ai_tun_device
+
+    - name: Require TUN device on PVE host
+      ansible.builtin.assert:
+        that:
+          - hermes_ai_tun_device.stat.exists
+          - hermes_ai_tun_device.stat.ischr
+        fail_msg: /dev/net/tun must exist on cloud-pc before enabling the Hermes AI transparent proxy.
+
+    - name: Allow TUN device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:200 rwm"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Bind mount TUN device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Apply pending Hermes AI LXC configuration
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for Hermes AI SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: hermes-ai
+      vars:
+        ansible_become: false
+
+- name: Install Hermes AI base runtime
+  hosts: hermes-ai
+  gather_facts: false
+  vars:
+    ansible_become: false
+    hermes_ai_root: /opt/hermes-ai
+    hermes_ai_state: /srv/hermes-ai
+    hermes_ai_proxy_dir: /opt/hermes-ai/tun-proxy
+    hermes_ai_proxy_image: metacubex/mihomo@sha256:e6acd921addecfd59a8e2d38203f88356d635b54de6c0673db0e015139989312
+    hermes_ai_proxy_name: hermes-ai-tun-proxy
+  tasks:
+    - name: Install Hermes AI runtime packages
+      ansible.builtin.apt:
+        name:
+          - ca-certificates
+          - curl
+          - docker-compose
+          - docker.io
+          - fuse-overlayfs
+          - git
+          - ufw
+        state: present
+        update_cache: true
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: hermes_ai_docker_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        enabled: true
+        state: "{{ 'restarted' if hermes_ai_docker_config.changed else 'started' }}"
+
+    - name: Allow SSH only from the HomeLab LAN
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ homelab_lan_cidr }}"
+
+    - name: Allow SSH from the OpenVPN network
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ openvpn_network_cidr }}"
+
+    - name: Enable restrictive Hermes AI firewall
+      community.general.ufw:
+        state: enabled
+        policy: deny
+        direction: incoming
+
+    - name: Create Hermes AI application directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - "{{ hermes_ai_root }}"
+        - "{{ hermes_ai_state }}"
+        - "{{ hermes_ai_proxy_dir }}"
+
+    - name: Configure Hermes AI transparent proxy
+      ansible.builtin.copy:
+        dest: "{{ hermes_ai_proxy_dir }}/config.yaml"
+        owner: root
+        group: root
+        mode: "0640"
+        content: |
+          ipv6: false
+
+          tun:
+            enable: true
+            stack: system
+            device: hermes-tun
+            auto-route: true
+            auto-redirect: true
+            auto-detect-interface: true
+            route-exclude-address:
+              - 192.168.1.27/32
+            dns-hijack:
+              - any:53
+
+          dns:
+            enable: true
+            enhanced-mode: redir-host
+            nameserver:
+              - https://cloudflare-dns.com/dns-query
+
+          sniffer:
+            enable: true
+            force-dns-mapping: true
+            parse-pure-ip: true
+
+          proxies:
+            - name: mihomo-lan
+              type: socks5
+              server: 192.168.1.27
+              port: 7891
+
+          proxy-groups:
+            - name: PROXY
+              type: select
+              proxies:
+                - mihomo-lan
+
+          rules:
+            - IP-CIDR,127.0.0.0/8,DIRECT,no-resolve
+            - IP-CIDR,10.0.0.0/8,DIRECT,no-resolve
+            - IP-CIDR,172.16.0.0/12,DIRECT,no-resolve
+            - IP-CIDR,192.168.0.0/16,DIRECT,no-resolve
+            - IP-CIDR,169.254.0.0/16,DIRECT,no-resolve
+            - MATCH,PROXY
+      register: hermes_ai_proxy_config
+
+    - name: Wait for Mihomo SOCKS5 upstream
+      ansible.builtin.wait_for:
+        host: 192.168.1.27
+        port: 7891
+        timeout: 15
+
+    - name: Validate Hermes AI transparent proxy config
+      ansible.builtin.command: >-
+        docker run --rm --network none
+        -v {{ hermes_ai_proxy_dir }}:/root/.config/mihomo:ro
+        {{ hermes_ai_proxy_image }} -t -d /root/.config/mihomo
+      changed_when: false
+
+    - name: Install Hermes AI transparent proxy service
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/hermes-ai-tun-proxy.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Hermes AI transparent Mihomo proxy
+          After=docker.service network-online.target
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ hermes_ai_proxy_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ hermes_ai_proxy_name }} \
+            --network host \
+            --cap-drop ALL \
+            --cap-add NET_ADMIN \
+            --security-opt no-new-privileges \
+            --device /dev/net/tun \
+            -v {{ hermes_ai_proxy_dir }}:/root/.config/mihomo:ro \
+            {{ hermes_ai_proxy_image }}
+          ExecStop=/usr/bin/docker stop {{ hermes_ai_proxy_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: hermes_ai_proxy_unit
+
+    - name: Reload systemd when transparent proxy service changes
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: hermes_ai_proxy_unit.changed
+
+    - name: Enable Hermes AI transparent proxy
+      ansible.builtin.systemd:
+        name: hermes-ai-tun-proxy
+        enabled: true
+        state: "{{ 'restarted' if hermes_ai_proxy_config.changed or hermes_ai_proxy_unit.changed else 'started' }}"
+
+    - name: Check Hermes AI transparent proxy service
+      ansible.builtin.command: systemctl is-active hermes-ai-tun-proxy
+      register: hermes_ai_proxy_status
+      changed_when: false
+      failed_when: hermes_ai_proxy_status.stdout != 'active'
+
+    - name: Check Hermes AI external HTTPS access
+      ansible.builtin.command: curl --fail --silent --show-error --max-time 20 https://api.ipify.org
+      changed_when: false
+      no_log: true
+
+    - name: Document Hermes AI runtime layout
+      ansible.builtin.copy:
+        dest: "{{ hermes_ai_root }}/README.md"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Hermes AI runtime
+
+          Deploy the Hermes application here. Keep runtime state and the real
+          `.env` file in /srv/hermes-ai. Do not commit Telegram or LLM tokens.
+
+          Docker is installed with fuse-overlayfs for this unprivileged LXC.
+          Hermes is a Telegram bot and needs no published Docker ports. Do not
+          use `ports:` or `-p` without adding an explicit firewall policy:
+          Docker port publishing can bypass UFW.
+
+          The hermes-ai-tun-proxy service routes external traffic through
+          Mihomo at 192.168.1.27:7890. LAN traffic remains direct.

ansible/playbooks/pve-memoir-bot.yml

@@ -0,0 +1,305 @@
+---
+- name: Create memoir-bot LXC on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    memoir_bot_vmid: 142
+    memoir_bot_hostname: memoir-bot
+    memoir_bot_ip: 192.168.1.26/24
+    memoir_bot_gateway: 192.168.1.1
+    memoir_bot_bridge: vmbr0
+    memoir_bot_rootfs: local-lvm:16
+    memoir_bot_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    memoir_bot_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  handlers:
+    - name: restart memoir-bot lxc
+      ansible.builtin.shell: "pct stop {{ memoir_bot_vmid }} || true; pct start {{ memoir_bot_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if memoir-bot LXC exists
+      ansible.builtin.command: "pct config {{ memoir_bot_vmid }}"
+      register: memoir_bot_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Install memoir-bot LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/memoir-bot-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', memoir_bot_pubkey_file) }}\n"
+      when: memoir_bot_pct_config.rc != 0
+
+    - name: Create memoir-bot LXC
+      ansible.builtin.command: >-
+        pct create {{ memoir_bot_vmid }} {{ memoir_bot_ostemplate }}
+        --hostname {{ memoir_bot_hostname }}
+        --rootfs {{ memoir_bot_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge={{ memoir_bot_bridge }},gw={{ memoir_bot_gateway }},ip={{ memoir_bot_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=60
+        --cmode shell
+        --ssh-public-keys /tmp/memoir-bot-lxc.pub
+      when: memoir_bot_pct_config.rc != 0
+
+    - name: Start memoir-bot LXC
+      ansible.builtin.command: "pct start {{ memoir_bot_vmid }}"
+      register: memoir_bot_pct_start
+      changed_when: memoir_bot_pct_start.rc == 0
+      failed_when: memoir_bot_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in memoir-bot LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ memoir_bot_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart memoir-bot lxc
+
+    - name: Bind mount FUSE device in memoir-bot LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ memoir_bot_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart memoir-bot lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for memoir-bot SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: memoir-bot
+      vars:
+        ansible_become: false
+
+- name: Configure Docker and memoir-bot inside LXC
+  hosts: memoir-bot
+  gather_facts: true
+  vars:
+    ansible_become: false
+    memoir_bot_source_dir: /home/ada/Documents/Projects/Other/memoir_bot/
+    memoir_bot_app_dir: /opt/memoir-bot/app
+    memoir_bot_state_dir: /srv/memoir-bot
+    memoir_bot_env_file: /srv/memoir-bot/.env
+    memoir_bot_env_source: /home/ada/Documents/Projects/Other/memoir_bot/.env
+    memoir_bot_ssh_private_key_file: ~/.ssh/id_ed25519
+    memoir_bot_ssh_public_key_file: ~/.ssh/id_ed25519.pub
+    memoir_bot_vault_repo: git@github.com:ada-dmitry/SecondBrain.git
+    memoir_bot_vault_dir: /srv/memoir-bot/vault
+    memoir_bot_image: memoir-bot:local
+    memoir_bot_container_name: memoir-bot
+  tasks:
+    - name: Install Docker and deploy dependencies
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - git
+          - openssh-client
+          - rsync
+          - ca-certificates
+          - curl
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: docker_daemon_config
+
+    - name: Ensure Docker service is enabled and running
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if docker_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Ensure memoir-bot directories exist
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - "{{ memoir_bot_app_dir }}"
+        - "{{ memoir_bot_state_dir }}"
+        - "{{ memoir_bot_state_dir }}/ssh"
+
+    - name: Copy memoir-bot env example
+      ansible.builtin.copy:
+        dest: "{{ memoir_bot_state_dir }}/.env.example"
+        owner: root
+        group: root
+        mode: "0640"
+        content: |
+          BOT_TOKEN=123456:telegram-token
+          ALLOWED_USER_IDS=123456789
+          OBSIDIAN_VAULT_PATH=/srv/obsidian/vault
+          DAILY_NOTES_DIR=03 Journal
+          NOTE_PATH_FORMAT=%Y/%m/%d.%m.%y.md
+          MESSAGE_TIME_FORMAT=%H:%M
+          TELEGRAM_PROXY=
+          REMINDER_MIN_HOURS=2
+          REMINDER_MAX_HOURS=3
+          REMINDER_START_HOUR=7
+          REMINDER_END_HOUR=23
+          REMINDER_TIMEZONE=Europe/Moscow
+
+    - name: Copy memoir-bot env file when present locally
+      ansible.builtin.copy:
+        src: "{{ memoir_bot_env_source }}"
+        dest: "{{ memoir_bot_env_file }}"
+        owner: root
+        group: root
+        mode: "0600"
+      when: lookup('ansible.builtin.fileglob', memoir_bot_env_source) | length > 0
+      no_log: true
+
+    - name: Install GitHub SSH private key for memoir-bot
+      ansible.builtin.copy:
+        src: "{{ memoir_bot_ssh_private_key_file }}"
+        dest: "{{ memoir_bot_state_dir }}/ssh/id_ed25519"
+        owner: root
+        group: root
+        mode: "0600"
+      no_log: true
+
+    - name: Install GitHub SSH public key for memoir-bot when present locally
+      ansible.builtin.copy:
+        src: "{{ memoir_bot_ssh_public_key_file }}"
+        dest: "{{ memoir_bot_state_dir }}/ssh/id_ed25519.pub"
+        owner: root
+        group: root
+        mode: "0644"
+      when: lookup('ansible.builtin.fileglob', memoir_bot_ssh_public_key_file) | length > 0
+
+    - name: Scan GitHub SSH host key
+      ansible.builtin.command: ssh-keyscan github.com
+      register: memoir_bot_github_host_key
+      changed_when: false
+
+    - name: Trust GitHub SSH host key for memoir-bot
+      ansible.builtin.known_hosts:
+        path: "{{ memoir_bot_state_dir }}/ssh/known_hosts"
+        name: github.com
+        key: "{{ memoir_bot_github_host_key.stdout }}"
+        state: present
+
+    - name: Clone SecondBrain vault
+      ansible.builtin.git:
+        repo: "{{ memoir_bot_vault_repo }}"
+        dest: "{{ memoir_bot_vault_dir }}"
+        key_file: "{{ memoir_bot_state_dir }}/ssh/id_ed25519"
+        accept_hostkey: true
+        update: true
+        version: main
+      register: memoir_bot_vault_checkout
+
+    - name: Configure memoir-bot vault Git author name
+      ansible.builtin.command: git config user.name memoir-bot
+      args:
+        chdir: "{{ memoir_bot_vault_dir }}"
+      changed_when: false
+
+    - name: Configure memoir-bot vault Git author email
+      ansible.builtin.command: git config user.email memoir-bot@homelab.local
+      args:
+        chdir: "{{ memoir_bot_vault_dir }}"
+      changed_when: false
+
+    - name: Sync memoir-bot source code
+      ansible.posix.synchronize:
+        src: "{{ memoir_bot_source_dir }}"
+        dest: "{{ memoir_bot_app_dir }}/"
+        delete: true
+        rsync_opts:
+          - "--exclude=.env"
+          - "--exclude=.git"
+          - "--exclude=.venv"
+          - "--exclude=__pycache__"
+          - "--exclude=*.pyc"
+      register: memoir_bot_source_sync
+
+    - name: Check if memoir-bot image exists
+      ansible.builtin.command: "docker image inspect {{ memoir_bot_image }}"
+      register: memoir_bot_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Build memoir-bot image
+      ansible.builtin.command: "docker build -t {{ memoir_bot_image }} {{ memoir_bot_app_dir }}"
+      when: memoir_bot_source_sync.changed or memoir_bot_image_inspect.rc != 0
+      register: memoir_bot_image_build
+      changed_when: memoir_bot_image_build.rc == 0
+
+    - name: Install memoir-bot systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/memoir-bot.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Memoir Telegram bot container
+          After=docker.service
+          Requires=docker.service
+          ConditionPathExists={{ memoir_bot_env_file }}
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ memoir_bot_container_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ memoir_bot_container_name }} \
+            --env-file {{ memoir_bot_env_file }} \
+            -v {{ memoir_bot_state_dir }}/vault:/srv/obsidian/vault \
+            -v {{ memoir_bot_state_dir }}/ssh:/root/.ssh:ro \
+            {{ memoir_bot_image }}
+          ExecStop=/usr/bin/docker stop {{ memoir_bot_container_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: memoir_bot_unit
+
+    - name: Reload systemd when memoir-bot unit changes
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: memoir_bot_unit.changed
+
+    - name: Check memoir-bot env file
+      ansible.builtin.stat:
+        path: "{{ memoir_bot_env_file }}"
+      register: memoir_bot_env
+
+    - name: Enable memoir-bot service
+      ansible.builtin.systemd:
+        name: memoir-bot
+        enabled: true
+
+    - name: Start memoir-bot when env file exists
+      ansible.builtin.systemd:
+        name: memoir-bot
+        state: "{{ 'restarted' if memoir_bot_source_sync.changed or memoir_bot_image_build.changed or memoir_bot_unit.changed or memoir_bot_vault_checkout.changed else 'started' }}"
+      when: memoir_bot_env.stat.exists

ansible/playbooks/pve-mihomo.yml

@@ -0,0 +1,332 @@
+---
+- name: Create mihomo LXC on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    mihomo_vmid: 143
+    mihomo_hostname: mihomo
+    mihomo_ip: 192.168.1.27/24
+    mihomo_gateway: 192.168.1.1
+    mihomo_bridge: vmbr0
+    mihomo_rootfs: local-lvm:8
+    mihomo_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    mihomo_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  handlers:
+    - name: restart mihomo lxc
+      ansible.builtin.shell: "pct stop {{ mihomo_vmid }} || true; pct start {{ mihomo_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if mihomo LXC exists
+      ansible.builtin.command: "pct config {{ mihomo_vmid }}"
+      register: mihomo_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 143
+      ansible.builtin.assert:
+        that:
+          - mihomo_pct_config.rc != 0 or mihomo_existing_hostname == mihomo_hostname
+        fail_msg: VMID 143 already exists and is not the Mihomo container.
+      vars:
+        mihomo_existing_hostname: >-
+          {{ mihomo_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Install mihomo LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/mihomo-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', mihomo_pubkey_file) }}\n"
+      when: mihomo_pct_config.rc != 0
+
+    - name: Create mihomo LXC
+      ansible.builtin.command: >-
+        pct create {{ mihomo_vmid }} {{ mihomo_ostemplate }}
+        --hostname {{ mihomo_hostname }}
+        --rootfs {{ mihomo_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge={{ mihomo_bridge }},gw={{ mihomo_gateway }},ip={{ mihomo_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=70
+        --cmode shell
+        --ssh-public-keys /tmp/mihomo-lxc.pub
+      when: mihomo_pct_config.rc != 0
+
+    - name: Start mihomo LXC
+      ansible.builtin.command: "pct start {{ mihomo_vmid }}"
+      register: mihomo_pct_start
+      changed_when: mihomo_pct_start.rc == 0
+      failed_when: mihomo_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Bind mount FUSE device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Allow TUN device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:200 rwm"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Bind mount TUN device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for mihomo SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: mihomo
+      vars:
+        ansible_become: false
+
+- name: Prepare mihomo runtime host
+  hosts: mihomo
+  gather_facts: true
+  vars:
+    ansible_become: false
+    mihomo_config_dir: /opt/mihomo/config
+    mihomo_ui_dir: /opt/mihomo/ui
+    mihomo_image: metacubex/mihomo:v1.19.29@sha256:5e7bcc5e7a866afcc8b007ef827c9ba773f2f34b6d7311b6d39ed1751f37cfd5
+    mihomo_ui_image: ghcr.io/metacubex/metacubexd:v1.270.6@sha256:156d55be885d4ba6254d840bd781b715c20c00afee6e6c24c76be4cfe5eb89d4
+    mihomo_container_name: mihomo
+    mihomo_ui_container_name: mihomo-ui
+  tasks:
+    - name: Install Docker and mihomo runtime packages
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - ca-certificates
+          - curl
+          - git
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: docker_daemon_config
+
+    - name: Ensure Docker service is enabled and running
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if docker_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Ensure mihomo directories exist
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - /opt/mihomo
+        - "{{ mihomo_config_dir }}"
+        - "{{ mihomo_ui_dir }}"
+
+    - name: Check if the configured Mihomo image is present
+      ansible.builtin.command: "docker image inspect {{ mihomo_image }}"
+      register: mihomo_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Mihomo image
+      ansible.builtin.command: "docker pull {{ mihomo_image }}"
+      when: mihomo_image_inspect.rc != 0
+      register: mihomo_image_pull
+      changed_when: true
+
+    - name: Check if the configured Mihomo UI image is present
+      ansible.builtin.command: "docker image inspect {{ mihomo_ui_image }}"
+      register: mihomo_ui_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Mihomo UI image
+      ansible.builtin.command: "docker pull {{ mihomo_ui_image }}"
+      when: mihomo_ui_image_inspect.rc != 0
+      register: mihomo_ui_image_pull
+      changed_when: true
+
+    - name: Install default mihomo config if missing
+      ansible.builtin.copy:
+        dest: "{{ mihomo_config_dir }}/config.yaml"
+        owner: root
+        group: root
+        mode: "0640"
+        force: false
+        content: |
+          mixed-port: 7890
+          socks-port: 7891
+          allow-lan: true
+          bind-address: 0.0.0.0
+          mode: rule
+          log-level: info
+          external-controller: 0.0.0.0:9090
+
+          dns:
+            enable: true
+            listen: 0.0.0.0:1053
+            enhanced-mode: fake-ip
+            nameserver:
+              - 1.1.1.1
+              - 8.8.8.8
+
+          proxies: []
+          proxy-groups:
+            - name: PROXY
+              type: select
+              proxies:
+                - DIRECT
+
+          rules:
+            - MATCH,DIRECT
+      register: mihomo_config
+
+    - name: Install mihomo systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/mihomo.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Mihomo proxy container
+          After=docker.service
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ mihomo_container_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ mihomo_container_name }} \
+            --pull never \
+            --cap-add NET_ADMIN \
+            --device /dev/net/tun \
+            -p 7890:7890 \
+            -p 7891:7891 \
+            -p 9090:9090 \
+            -v {{ mihomo_config_dir }}:/root/.config/mihomo \
+            {{ mihomo_image }}
+          ExecStop=/usr/bin/docker stop {{ mihomo_container_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: mihomo_unit
+
+    - name: Install mihomo UI systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/mihomo-ui.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Mihomo MetaCubeXD web UI container
+          After=docker.service mihomo.service
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ mihomo_ui_container_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ mihomo_ui_container_name }} \
+            --pull never \
+            -p 8080:80 \
+            {{ mihomo_ui_image }}
+          ExecStop=/usr/bin/docker stop {{ mihomo_ui_container_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: mihomo_ui_unit
+
+    - name: Reload systemd when mihomo units change
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: mihomo_unit.changed or mihomo_ui_unit.changed
+
+    - name: Enable and start mihomo
+      ansible.builtin.systemd:
+        name: mihomo
+        enabled: true
+        state: "{{ 'restarted' if mihomo_config.changed or mihomo_unit.changed or mihomo_image_pull is changed else 'started' }}"
+
+    - name: Enable and start mihomo UI
+      ansible.builtin.systemd:
+        name: mihomo-ui
+        enabled: true
+        state: "{{ 'restarted' if mihomo_ui_unit.changed or mihomo_ui_image_pull is changed else 'started' }}"
+
+    - name: Wait for Mihomo proxy TCP ports
+      ansible.builtin.wait_for:
+        host: 127.0.0.1
+        port: "{{ item }}"
+        state: started
+        timeout: 120
+      loop:
+        - 7890
+        - 7891
+        - 9090
+
+    - name: Check Mihomo UI endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:8080/
+        status_code: 200
+      register: mihomo_ui_health
+      retries: 24
+      delay: 5
+      until: mihomo_ui_health.status == 200
+
+    - name: Check Mihomo controller /version endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:9090/version
+        status_code: 200
+      register: mihomo_controller_health
+      retries: 24
+      delay: 5
+      until: mihomo_controller_health.status == 200

ansible/playbooks/pve-monitoring.yml

@@ -0,0 +1,42 @@
+---
+- name: Create monitoring LXC on cloud-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 146
+    pve_lxc_node: cloud-pc
+    pve_lxc_hostname: monitoring
+    pve_lxc_ip: 192.168.1.30/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: data:24
+    pve_lxc_cores: 2
+    pve_lxc_memory: 4096
+    pve_lxc_swap: 512
+    pve_lxc_startup: order=80
+    pve_lxc_features:
+      - nesting=1
+    pve_lxc_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+  roles:
+    - role: pve_lxc
+
+- name: Enable Docker keyctl feature for monitoring LXC
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Read monitoring LXC configuration
+      ansible.builtin.command: pct config 146
+      register: monitoring_lxc_config
+      changed_when: false
+
+    - name: Enable keyctl for Docker in monitoring LXC
+      ansible.builtin.command: pct set 146 --features nesting=1,keyctl=1
+      when: "'keyctl=1' not in monitoring_lxc_config.stdout"
+      register: monitoring_lxc_keyctl
+
+    - name: Restart monitoring LXC after feature change
+      ansible.builtin.command: pct reboot 146
+      when: monitoring_lxc_keyctl.changed

ansible/playbooks/pve-ovpn-mini.yml

@@ -1,4 +1,4 @@
-- name: Create wg-mini LXC on mini-pc
+- name: Create ovpn-mini LXC on mini-pc
   hosts: localhost
   connection: local
   gather_facts: false
@@ -7,19 +7,19 @@
       vars:
         pve_lxc_vmid: 132
         pve_lxc_node: mini-pc
-        pve_lxc_hostname: wg-mini
+        pve_lxc_hostname: ovpn-mini
         pve_lxc_ip: 192.168.1.23/24
         pve_lxc_gateway: 192.168.1.1
         pve_lxc_storage: local-lvm
         pve_lxc_disk: local-lvm:8
         pve_lxc_ostemplate: "{{ lookup('env', 'PVE_LXC_OSTEMPLATE') | default('local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst', true) }}"
 
-- name: Allow TUN device in wg-mini LXC config
+- name: Allow TUN device in ovpn-mini LXC config
   hosts: mini-pc
   gather_facts: false
   become: true
   handlers:
-    - name: restart wg-mini lxc
+    - name: restart ovpn-mini lxc
       ansible.builtin.shell: pct stop 132 || true; pct start 132
       changed_when: true
   tasks:
@@ -28,11 +28,11 @@
         path: /etc/pve/lxc/132.conf
         line: "lxc.cgroup2.devices.allow: c 10:200 rwm"
         state: present
-      notify: restart wg-mini lxc
+      notify: restart ovpn-mini lxc
 
     - name: Bind mount /dev/net/tun
       ansible.builtin.lineinfile:
         path: /etc/pve/lxc/132.conf
         line: "lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file"
         state: present
-      notify: restart wg-mini lxc
+      notify: restart ovpn-mini lxc

ansible/playbooks/pve-vaultwarden.yml

@@ -1,3 +1,27 @@
+---
+- name: Guard Vaultwarden VMID before API updates
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 140 configuration
+      ansible.builtin.command: pct config 140
+      register: vaultwarden_existing_vmid
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 140
+      ansible.builtin.assert:
+        that:
+          - vaultwarden_existing_vmid.rc != 0 or vaultwarden_existing_hostname == 'vaultwarden'
+        fail_msg: VMID 140 already exists and is not the Vaultwarden container.
+      vars:
+        vaultwarden_existing_hostname: >-
+          {{ vaultwarden_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
 - name: Create Vaultwarden LXC on mini-pc
   hosts: mini-pc
   gather_facts: false
@@ -84,7 +108,7 @@
   vars:
     ansible_become: false
     vaultwarden_data_dir: /opt/vaultwarden/data
-    vaultwarden_image: vaultwarden/server:latest
+    vaultwarden_image: vaultwarden/server:1.37.1@sha256:e9efdf001bf0d68c21f2cbfb8e1d9b5961a7ca9c85e0a7e58bf51a13b997d744
     vaultwarden_container_name: vaultwarden
     vaultwarden_http_port: 80
   tasks:
@@ -133,6 +157,18 @@
         group: root
         mode: "0750"
 
+    - name: Check if the configured Vaultwarden image is present
+      ansible.builtin.command: "docker image inspect {{ vaultwarden_image }}"
+      register: vaultwarden_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Vaultwarden image
+      ansible.builtin.command: "docker pull {{ vaultwarden_image }}"
+      when: vaultwarden_image_inspect.rc != 0
+      register: vaultwarden_image_pull
+      changed_when: true
+
     - name: Install Vaultwarden systemd unit
       ansible.builtin.copy:
         dest: /etc/systemd/system/vaultwarden.service
@@ -151,7 +187,7 @@
           ExecStartPre=-/usr/bin/docker rm -f {{ vaultwarden_container_name }}
           ExecStart=/usr/bin/docker run --rm \
             --name {{ vaultwarden_container_name }} \
-            --pull always \
+            --pull never \
             -p {{ vaultwarden_http_port }}:80 \
             -v {{ vaultwarden_data_dir }}:/data \
             -e WEBSOCKET_ENABLED=true \
@@ -170,5 +206,20 @@
     - name: Enable and start Vaultwarden
       ansible.builtin.systemd:
         name: vaultwarden
-        state: started
+        state: "{{ 'restarted' if vaultwarden_unit.changed or vaultwarden_image_pull.changed else 'started' }}"
         enabled: true
+
+    - name: Wait for Vaultwarden HTTP endpoint
+      ansible.builtin.uri:
+        url: "http://127.0.0.1:{{ vaultwarden_http_port }}/"
+        status_code: 200
+      register: vaultwarden_health
+      retries: 24
+      delay: 5
+      until: vaultwarden_health.status == 200
+
+    - name: Verify Docker storage driver
+      ansible.builtin.command: docker info --format '{{ "{{" }}.Driver{{ "}}" }}'
+      register: vaultwarden_docker_driver
+      changed_when: false
+      failed_when: vaultwarden_docker_driver.stdout != 'fuse-overlayfs'

ansible/playbooks/reverse-proxy-grimmory.yml

@@ -0,0 +1,88 @@
+---
+- name: Configure Grimmory reverse proxy on ru-vps
+  hosts: ru-vps
+  gather_facts: false
+  vars:
+    caddy_dir: /opt/services/ru-vps/caddy
+    caddyfile_path: /opt/services/ru-vps/caddy/Caddyfile
+    grimmory_domain: books.ada-dev.ru
+    grimmory_upstream: 192.168.1.34:6060
+  tasks:
+    - name: Ensure Caddy service directory exists
+      ansible.builtin.file:
+        path: "{{ caddy_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Grimmory Caddy site
+      ansible.builtin.blockinfile:
+        path: "{{ caddyfile_path }}"
+        create: true
+        owner: root
+        group: root
+        mode: "0644"
+        marker: "# {mark} ANSIBLE MANAGED GRIMMORY SITE"
+        block: |
+          {{ grimmory_domain }} {
+            @grimmory_opds_atom path /api/v1/opds /api/v1/opds/libraries /api/v1/opds/shelves /api/v1/opds/magic-shelves /api/v1/opds/authors /api/v1/opds/series /api/v1/opds/catalog /api/v1/opds/recent /api/v1/opds/surprise
+            handle @grimmory_opds_atom {
+              reverse_proxy {{ grimmory_upstream }} {
+                header_up Accept "application/atom+xml"
+                header_up Accept-Encoding identity
+                transport http {
+                  compression off
+                }
+              }
+            }
+
+            @grimmory_opds_search path /api/v1/opds/search.opds
+            handle @grimmory_opds_search {
+              reverse_proxy {{ grimmory_upstream }} {
+                header_up Accept "application/opensearchdescription+xml"
+                header_up Accept-Encoding identity
+                transport http {
+                  compression off
+                }
+              }
+            }
+
+            @grimmory_device_api path /api/koreader /api/koreader/* /api/v1/opds /api/v1/opds/*
+            handle @grimmory_device_api {
+              reverse_proxy {{ grimmory_upstream }} {
+                header_up Accept-Encoding identity
+                transport http {
+                  compression off
+                }
+              }
+            }
+
+            handle {
+              reverse_proxy {{ grimmory_upstream }}
+            }
+          }
+      register: grimmory_caddy_site
+
+    - name: Validate host Caddy config
+      ansible.builtin.command: caddy validate --config {{ caddyfile_path }}
+      changed_when: false
+
+    - name: Check Caddy container mounted config
+      ansible.builtin.command: docker exec caddy grep -F '@grimmory_device_api path /api/koreader' /etc/caddy/Caddyfile
+      register: grimmory_container_caddyfile
+      changed_when: false
+      failed_when: false
+
+    - name: Restart Caddy when config changed or bind mount is stale
+      ansible.builtin.command: docker restart caddy
+      when: grimmory_caddy_site.changed or grimmory_container_caddyfile.rc != 0
+
+    - name: Validate Caddy container config after restart
+      ansible.builtin.command: docker exec caddy caddy validate --config /etc/caddy/Caddyfile
+      changed_when: false
+
+    - name: Check Grimmory upstream from ru-vps
+      ansible.builtin.uri:
+        url: http://{{ grimmory_upstream }}/api/v1/healthcheck
+        status_code: 200

ansible/playbooks/ru-vps-mihomo-harden.yml

@@ -0,0 +1,280 @@
+---
+- name: Harden live Mihomo on ru-vps
+  hosts: ru-vps
+  gather_facts: false
+  vars:
+    ru_vps_mihomo_harden_confirm: false
+    mihomo_root: /opt/services/ru-vps/mihomo
+    mihomo_config_path: "{{ mihomo_root }}/config/config.yaml"
+    mihomo_compose_path: "{{ mihomo_root }}/docker-compose.yml"
+    mihomo_backup_dir: /var/backups/ru-vps-mihomo
+    mihomo_state_dir: /var/lib/ru-vps-mihomo
+    mihomo_state_path: "{{ mihomo_state_dir }}/auth.json"
+    mihomo_harden_script_path: /usr/local/sbin/ru-vps-mihomo-harden
+    mihomo_harden_script_src: "{{ playbook_dir }}/../files/ru-vps-mihomo-harden.py"
+    mihomo_backup_path: "{{ mihomo_backup_dir }}/config.yaml.{{ lookup('pipe', 'date -u +%Y%m%d%H%M%S') }}"
+  pre_tasks:
+    - name: Require explicit Mihomo hardening confirmation
+      ansible.builtin.assert:
+        that:
+          - ru_vps_mihomo_harden_confirm | bool
+        fail_msg: Run with -e ru_vps_mihomo_harden_confirm=true.
+
+    - name: Preflight Mihomo config file exists
+      ansible.builtin.stat:
+        path: "{{ mihomo_config_path }}"
+      register: mihomo_config_stat
+
+    - name: Preflight Mihomo compose file exists
+      ansible.builtin.stat:
+        path: "{{ mihomo_compose_path }}"
+      register: mihomo_compose_stat
+
+    - name: Preflight Mihomo state file exists
+      ansible.builtin.stat:
+        path: "{{ mihomo_state_path }}"
+      register: mihomo_state_stat
+
+    - name: Refuse to run without live Mihomo config and compose files
+      ansible.builtin.assert:
+        that:
+          - mihomo_config_stat.stat.exists
+          - mihomo_compose_stat.stat.exists
+        fail_msg: Live Mihomo config or compose file is missing.
+
+    - name: Preflight Mihomo container is running
+      ansible.builtin.command:
+        argv:
+          - docker
+          - inspect
+          - -f
+          - '{{ "{{" }}.State.Running{{ "}}" }}'
+          - mihomo
+      register: mihomo_container_state
+      changed_when: false
+      failed_when: mihomo_container_state.stdout.strip() != 'true'
+
+    - name: Preflight PyYAML is available on the target
+      ansible.builtin.command:
+        argv:
+          - python3
+          - -c
+          - import yaml
+      changed_when: false
+
+    - name: Preflight current Mihomo config validates
+      ansible.builtin.command:
+        argv:
+          - docker
+          - compose
+          - -f
+          - "{{ mihomo_compose_path }}"
+          - exec
+          - -T
+          - mihomo
+          - /mihomo
+          - -t
+          - -d
+          - /root/.config/mihomo
+      changed_when: false
+
+    - name: Ensure Mihomo backup directory exists
+      ansible.builtin.file:
+        path: "{{ mihomo_backup_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0700"
+
+    - name: Install root-only Mihomo hardening helper
+      ansible.builtin.copy:
+        src: "{{ mihomo_harden_script_src }}"
+        dest: "{{ mihomo_harden_script_path }}"
+        owner: root
+        group: root
+        mode: "0700"
+
+    - name: Create fresh backup of the live Mihomo config after validation
+      ansible.builtin.copy:
+        src: "{{ mihomo_config_path }}"
+        dest: "{{ mihomo_backup_path }}"
+        remote_src: true
+        owner: root
+        group: root
+        mode: "0600"
+
+  tasks:
+    - block:
+        - name: Apply Mihomo hardening in place
+          ansible.builtin.command:
+            argv:
+              - "{{ mihomo_harden_script_path }}"
+              - apply
+              - --config
+              - "{{ mihomo_config_path }}"
+              - --state
+              - "{{ mihomo_state_path }}"
+          register: mihomo_harden_apply
+          changed_when: (mihomo_harden_apply.stdout | from_json).changed
+          no_log: true
+
+        - name: Validate hardened Mihomo config in the running container
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - exec
+              - -T
+              - mihomo
+              - /mihomo
+              - -t
+              - -d
+              - /root/.config/mihomo
+          changed_when: false
+
+        - name: Recreate Mihomo container after config hardening
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - up
+              - -d
+              - --force-recreate
+              - mihomo
+          changed_when: true
+
+        - name: Wait for hardened Mihomo listeners on loopback
+          ansible.builtin.wait_for:
+            host: 127.0.0.1
+            port: "{{ item }}"
+            state: started
+            timeout: 30
+          loop:
+            - 7890
+            - 7891
+
+        - name: Run authenticated Mihomo SOCKS probe to Telegram
+          ansible.builtin.command:
+            argv:
+              - "{{ mihomo_harden_script_path }}"
+              - probe
+              - --config
+              - "{{ mihomo_config_path }}"
+              - --state
+              - "{{ mihomo_state_path }}"
+          register: mihomo_harden_probe
+          changed_when: false
+          no_log: true
+
+        - name: Check anonymous SOCKS access fails
+          ansible.builtin.command:
+            argv:
+              - curl
+              - --silent
+              - --show-error
+              - --connect-timeout
+              - "5"
+              - --max-time
+              - "15"
+              - --proxy
+              - socks5h://127.0.0.1:7891
+              - https://api.telegram.org
+              - --output
+              - /dev/null
+          register: mihomo_anon_probe
+          changed_when: false
+          failed_when: false
+
+        - name: Remove public Mihomo UFW rules after successful hardening
+          community.general.ufw:
+            rule: allow
+            port: "{{ item.port }}"
+            proto: "{{ item.proto }}"
+            delete: true
+          loop:
+            - { port: 7890, proto: tcp }
+            - { port: 7890, proto: udp }
+            - { port: 7891, proto: tcp }
+            - { port: 7891, proto: udp }
+
+        - name: Verify Mihomo listeners are loopback only
+          ansible.builtin.command:
+            argv:
+              - ss
+              - -H
+              - -ltnp
+              - '( sport = :7890 or sport = :7891 )'
+          register: mihomo_ss
+          changed_when: false
+
+        - name: Confirm hardened Mihomo is bound to loopback only and anonymous access fails
+          ansible.builtin.assert:
+            that:
+              - mihomo_ss.stdout is search('127\\.0\\.0\\.1:7890')
+              - mihomo_ss.stdout is search('127\\.0\\.0\\.1:7891')
+              - mihomo_ss.stdout is not search('0\\.0\\.0\\.0:7890|:::7890|0\\.0\\.0\\.0:7891|:::7891')
+              - mihomo_anon_probe.rc != 0
+            fail_msg: Hardened Mihomo must listen on loopback only and reject anonymous SOCKS access.
+
+      rescue:
+        - name: Restore the live Mihomo config from backup
+          ansible.builtin.copy:
+            src: "{{ mihomo_backup_path }}"
+            dest: "{{ mihomo_config_path }}"
+            remote_src: true
+            owner: root
+            group: root
+            mode: "0640"
+
+        - name: Recreate Mihomo container after rollback
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - up
+              - -d
+              - --force-recreate
+              - mihomo
+          changed_when: true
+
+        - name: Remove newly created Mihomo credentials state after rollback
+          ansible.builtin.file:
+            path: "{{ mihomo_state_path }}"
+            state: absent
+          when: not mihomo_state_stat.stat.exists
+
+        - name: Wait for restored Mihomo listeners on loopback
+          ansible.builtin.wait_for:
+            host: 127.0.0.1
+            port: "{{ item }}"
+            state: started
+            timeout: 30
+          loop:
+            - 7890
+            - 7891
+
+        - name: Validate restored Mihomo config in the running container
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - exec
+              - -T
+              - mihomo
+              - /mihomo
+              - -t
+              - -d
+              - /root/.config/mihomo
+          changed_when: false
+
+        - name: Fail Mihomo hardening after restoring the backup
+          ansible.builtin.fail:
+            msg: Mihomo hardening failed and the live config was restored from backup.

ansible/playbooks/uptime-kuma.yml

@@ -0,0 +1,6 @@
+---
+- name: Freeze Prometheus monitoring and configure Uptime Kuma
+  hosts: monitoring_server
+  gather_facts: false
+  roles:
+    - role: uptime_kuma

ansible/playbooks/user-ssh-key.yml

@@ -0,0 +1,52 @@
+---
+- name: Install user SSH public key on managed hosts
+  hosts: servers
+  gather_facts: false
+  vars:
+    homelab_user_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+    homelab_user_pubkey: "{{ lookup('file', homelab_user_pubkey_file) }}"
+  tasks:
+    - name: Install user SSH key for connection user
+      ansible.posix.authorized_key:
+        user: "{{ ansible_user }}"
+        key: "{{ homelab_user_pubkey }}"
+        state: present
+
+    - name: Install user SSH key for root
+      ansible.posix.authorized_key:
+        user: root
+        key: "{{ homelab_user_pubkey }}"
+        state: present
+      become: true
+      when: ansible_user != 'root'
+
+- name: Bootstrap user SSH public key into legacy LXC containers via Proxmox
+  hosts: pve_nodes
+  gather_facts: false
+  vars:
+    homelab_user_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+    homelab_user_pubkey: "{{ lookup('file', homelab_user_pubkey_file) }}"
+    legacy_lxc_key_targets:
+      - node: cloud-pc
+        vmid: 120
+        name: pbs
+  tasks:
+    - name: Install user SSH key for root inside legacy LXC
+      ansible.builtin.shell: |
+        pct exec {{ item.vmid }} -- sh -c '
+          key=$(printf "%s" {{ homelab_user_pubkey | b64encode | quote }} | base64 -d)
+          mkdir -p /root/.ssh
+          chmod 700 /root/.ssh
+          touch /root/.ssh/authorized_keys
+          if grep -qxF "$key" /root/.ssh/authorized_keys; then
+            echo present
+          else
+            printf "%s\n" "$key" >> /root/.ssh/authorized_keys
+            echo added
+          fi
+          chmod 600 /root/.ssh/authorized_keys
+        '
+      loop: "{{ legacy_lxc_key_targets }}"
+      when: item.node == inventory_hostname
+      register: legacy_lxc_key_install
+      changed_when: "'added' in legacy_lxc_key_install.stdout"

ansible/playbooks/vaultwarden-update.yml

@@ -0,0 +1,98 @@
+---
+- name: Create and verify Vaultwarden backup before update
+  hosts: vaultwarden
+  gather_facts: false
+  tasks:
+    - name: Create a fresh Vaultwarden offsite backup
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-restic-offsite-vaultwarden.service
+      changed_when: true
+
+    - name: Run Vaultwarden offsite backup audit
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-vaultwarden.service
+      changed_when: true
+
+- name: Create and verify Vaultwarden PBS backup before update
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 140 configuration
+      ansible.builtin.command: pct config 140
+      register: vaultwarden_existing_vmid
+      changed_when: false
+
+    - name: Refuse to modify a foreign VMID 140
+      ansible.builtin.assert:
+        that:
+          - vaultwarden_existing_vmid.rc == 0
+          - vaultwarden_existing_hostname == 'vaultwarden'
+        fail_msg: VMID 140 already exists and is not the Vaultwarden container.
+      vars:
+        vaultwarden_existing_hostname: >-
+          {{ vaultwarden_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Check for active Proxmox backup before Vaultwarden PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: vaultwarden_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before Vaultwarden PBS backup
+      ansible.builtin.assert:
+        that:
+          - vaultwarden_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on mini-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh Vaultwarden PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "140"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+    - name: Run current PBS backup audit on mini-pc
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-vaultwarden.yml
+
+- name: Verify Vaultwarden public endpoint after update
+  hosts: ru-vps
+  gather_facts: false
+  tasks:
+    - name: Check Vaultwarden HTTPS endpoint
+      ansible.builtin.uri:
+        url: https://pass.ada-dev.ru/
+        status_code: 200
+        return_content: false
+      register: vaultwarden_public_health
+      retries: 24
+      delay: 5
+      until: vaultwarden_public_health.status == 200

ansible/requirements.yml

@@ -4,3 +4,5 @@ collections:
     version: ">=2.0.0"
   - name: community.proxmox
     version: ">=2.0.0"
+  - name: community.general
+    version: ">=10.0.0"

ansible/roles/backup_audit/defaults/main.yml

@@ -0,0 +1,31 @@
+---
+backup_audit_log_file: /var/log/homelab-backup-audit.log
+backup_audit_timer_oncalendar: "*-*-* 06:00:00"
+backup_audit_timer_randomized_delay: 10m
+
+backup_audit_pbs_vmids:
+  - vmid: 132
+    max_age_hours: 48
+  - vmid: 140
+    max_age_hours: 48
+  - vmid: 141
+    max_age_hours: 48
+  - vmid: 142
+    max_age_hours: 48
+  - vmid: 143
+    max_age_hours: 48
+  - vmid: 144
+    max_age_hours: 48
+  - vmid: 145
+    max_age_hours: 48
+  - vmid: 146
+    max_age_hours: 48
+  - vmid: 147
+    max_age_hours: 48
+  - vmid: 149
+    max_age_hours: 48
+  - vmid: 150
+    max_age_hours: 48
+
+backup_audit_restic_profiles: []
+backup_audit_metrics_dir: /var/lib/node_exporter/textfile_collector

ansible/roles/backup_audit/tasks/main.yml

@@ -0,0 +1,146 @@
+---
+- name: Install audit dependencies
+  ansible.builtin.apt:
+    name:
+      - jq
+      - sqlite3
+    state: present
+    update_cache: true
+
+- name: Ensure audit directories exist
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  loop:
+    - /etc/homelab-backup-audit
+    - /var/lib/homelab-backup-audit
+    - "{{ backup_audit_metrics_dir }}"
+
+- name: Install PBS audit script
+  ansible.builtin.template:
+    src: audit-pbs.sh.j2
+    dest: /usr/local/sbin/homelab-backup-audit-pbs
+    owner: root
+    group: root
+    mode: "0755"
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Install restic audit script
+  ansible.builtin.template:
+    src: audit-restic.sh.j2
+    dest: /usr/local/sbin/homelab-backup-audit-restic
+    owner: root
+    group: root
+    mode: "0755"
+  when: backup_audit_type | default('') == 'restic'
+
+- name: Install restic audit env files
+  ansible.builtin.copy:
+    dest: "/etc/homelab-backup-audit/{{ item.name }}.env"
+    owner: root
+    group: root
+    mode: "0600"
+    content: |
+      HOMELAB_AUDIT_PROFILE={{ item.name }}
+      HOMELAB_AUDIT_MAX_AGE_HOURS={{ item.max_age_hours | default(36) }}
+      HOMELAB_AUDIT_SQLITE_NAME={{ item.sqlite_name | default('') }}
+      HOMELAB_AUDIT_EXPECTED_NAME={{ item.expected_name | default('') }}
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'
+  no_log: true
+
+- name: Install PBS audit systemd service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-backup-audit-pbs.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=HomeLab PBS backup audit
+      Wants=network-online.target
+      After=network-online.target
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/local/sbin/homelab-backup-audit-pbs
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Install restic audit systemd services
+  ansible.builtin.copy:
+    dest: "/etc/systemd/system/homelab-backup-audit-{{ item.name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=HomeLab restic offsite backup audit ({{ item.name }})
+      Wants=network-online.target
+      After=network-online.target
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/bin/flock -w 1800 /var/lock/homelab-restic-{{ item.name }}.lock /usr/local/sbin/homelab-backup-audit-restic {{ item.name }}
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'
+
+- name: Install PBS audit systemd timer
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-backup-audit-pbs.timer
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Run HomeLab PBS backup audit
+
+      [Timer]
+      OnCalendar={{ backup_audit_timer_oncalendar }}
+      Persistent=true
+      RandomizedDelaySec={{ backup_audit_timer_randomized_delay }}
+
+      [Install]
+      WantedBy=timers.target
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Install restic audit systemd timers
+  ansible.builtin.copy:
+    dest: "/etc/systemd/system/homelab-backup-audit-{{ item.name }}.timer"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Run HomeLab restic backup audit ({{ item.name }})
+
+      [Timer]
+      OnCalendar={{ backup_audit_timer_oncalendar }}
+      Persistent=true
+      RandomizedDelaySec={{ backup_audit_timer_randomized_delay }}
+
+      [Install]
+      WantedBy=timers.target
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: Enable PBS audit timer
+  ansible.builtin.systemd:
+    name: homelab-backup-audit-pbs.timer
+    enabled: true
+    state: started
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Enable restic audit timers
+  ansible.builtin.systemd:
+    name: "homelab-backup-audit-{{ item.name }}.timer"
+    enabled: true
+    state: started
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'

ansible/roles/backup_audit/templates/audit-pbs.sh.j2

@@ -0,0 +1,70 @@
+#!/bin/sh
+# Managed by Ansible: HomeLab PBS backup audit (L0 freshness)
+set -eu
+
+LOG_FILE="{{ backup_audit_log_file }}"
+METRICS_DIR="{{ backup_audit_metrics_dir }}"
+METRICS_FILE="$METRICS_DIR/homelab_backup_audit_pbs.prom"
+METRICS_TMP=$(mktemp "$METRICS_FILE.XXXXXX")
+STATUS=OK
+NOW_EPOCH=$(date +%s)
+
+trap 'rm -f "$METRICS_TMP"' EXIT
+
+publish_metrics() {
+  chmod 0644 "$METRICS_TMP"
+  mv "$METRICS_TMP" "$METRICS_FILE"
+}
+
+ts() { date '+%Y-%m-%dT%H:%M:%S%z'; }
+log() { echo "[$(ts)] [pbs] $*" | tee -a "$LOG_FILE"; }
+
+get_latest_snapshot() {
+  pvesm list pbs --vmid "$1" 2>/dev/null \
+    | tail -n +2 \
+    | awk '{print $1}' \
+    | sort -t/ -k4 \
+    | tail -1
+}
+
+{% for entry in backup_audit_pbs_vmids %}
+audit_vmid_{{ entry.vmid }}() {
+  vmid={{ entry.vmid }}
+  max_age={{ entry.max_age_hours }}
+  latest=$(get_latest_snapshot "$vmid")
+  if [ -z "$latest" ]; then
+    log "FAIL L0: vmid $vmid — no snapshots in PBS"
+    printf 'homelab_backup_audit_snapshot_age_hours{profile="pbs",vmid="%s"} -1\n' "$vmid" >> "$METRICS_TMP"
+    printf 'homelab_backup_audit_snapshot_success{profile="pbs",vmid="%s"} 0\n' "$vmid" >> "$METRICS_TMP"
+    STATUS=FAIL
+    return
+  fi
+  ts_str=$(printf '%s' "$latest" | sed -n 's#.*/\([0-9T:Z-]*\)$#\1#p')
+  snap_epoch=$(date -d "$ts_str" +%s 2>/dev/null || echo 0)
+  age_hours=$(( (NOW_EPOCH - snap_epoch) / 3600 ))
+  if [ "$age_hours" -gt "$max_age" ]; then
+    log "FAIL L0: vmid $vmid — latest snapshot age ${age_hours}h > ${max_age}h (snapshot: $ts_str)"
+    printf 'homelab_backup_audit_snapshot_success{profile="pbs",vmid="%s"} 0\n' "$vmid" >> "$METRICS_TMP"
+    STATUS=FAIL
+  else
+    log "OK L0: vmid $vmid — latest snapshot age ${age_hours}h"
+    printf 'homelab_backup_audit_snapshot_success{profile="pbs",vmid="%s"} 1\n' "$vmid" >> "$METRICS_TMP"
+  fi
+  printf 'homelab_backup_audit_snapshot_age_hours{profile="pbs",vmid="%s"} %s\n' "$vmid" "$age_hours" >> "$METRICS_TMP"
+}
+audit_vmid_{{ entry.vmid }}
+
+{% endfor %}
+if [ "$STATUS" = "OK" ]; then
+  log "AUDIT PASSED"
+  printf 'homelab_backup_audit_success{profile="pbs"} 1\n' >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="pbs"} %s\n' "$NOW_EPOCH" >> "$METRICS_TMP"
+  publish_metrics
+  exit 0
+else
+  log "AUDIT FAILED"
+  printf 'homelab_backup_audit_success{profile="pbs"} 0\n' >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="pbs"} %s\n' "$NOW_EPOCH" >> "$METRICS_TMP"
+  publish_metrics
+  exit 1
+fi

ansible/roles/backup_audit/templates/audit-restic.sh.j2

@@ -0,0 +1,167 @@
+#!/bin/sh
+# Managed by Ansible: HomeLab restic offsite backup audit (L0+L1+L2)
+set -eu
+
+if [ "$#" -ne 1 ]; then
+  echo "usage: $0 <profile>" >&2
+  exit 64
+fi
+
+PROFILE="$1"
+case "$PROFILE" in
+  ''|*[!A-Za-z0-9_-]*)
+    echo "invalid profile name" >&2
+    exit 64
+    ;;
+esac
+RESTIC_ENV="/etc/homelab-restic/${PROFILE}.env"
+AUDIT_ENV="/etc/homelab-backup-audit/${PROFILE}.env"
+LOG_FILE="{{ backup_audit_log_file }}"
+TMP="/var/lib/homelab-backup-audit/${PROFILE}"
+METRICS_DIR="{{ backup_audit_metrics_dir }}"
+METRICS_FILE="$METRICS_DIR/homelab_backup_audit_${PROFILE}.prom"
+METRICS_TMP=$(mktemp "$METRICS_FILE.XXXXXX")
+STATUS=OK
+
+trap 'rm -f "$METRICS_TMP"' EXIT
+
+publish_metrics() {
+  chmod 0644 "$METRICS_TMP"
+  mv "$METRICS_TMP" "$METRICS_FILE"
+}
+
+for f in "$RESTIC_ENV" "$AUDIT_ENV"; do
+  if [ ! -f "$f" ]; then
+    echo "[$(date '+%Y-%m-%dT%H:%M:%S%z')] [${PROFILE}] FAIL: missing env file $f" | tee -a "$LOG_FILE"
+    printf 'homelab_backup_audit_success{profile="%s"} 0\n' "$PROFILE" > "$METRICS_TMP"
+    printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+    publish_metrics
+    exit 66
+  fi
+done
+
+set -a
+. "$RESTIC_ENV"
+. "$AUDIT_ENV"
+set +a
+
+export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE RCLONE_CONFIG
+
+ts() { date '+%Y-%m-%dT%H:%M:%S%z'; }
+log() { echo "[$(ts)] [${PROFILE}] $*" | tee -a "$LOG_FILE"; }
+
+# ── L0: freshness ──────────────────────────────────────────────────────────
+latest_time=$(restic snapshots --latest 1 --json 2>/dev/null | jq -r '.[0].time // empty')
+if [ -z "$latest_time" ]; then
+  log "FAIL L0: no snapshots found in repository"
+  printf 'homelab_backup_audit_success{profile="%s"} 0\n' "$PROFILE" > "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+  publish_metrics
+  exit 1
+fi
+
+now_epoch=$(date +%s)
+snap_epoch=$(date -d "$latest_time" +%s 2>/dev/null || echo 0)
+age_hours=$(( (now_epoch - snap_epoch) / 3600 ))
+max_age="${HOMELAB_AUDIT_MAX_AGE_HOURS:-36}"
+
+if [ "$age_hours" -gt "$max_age" ]; then
+  log "FAIL L0: latest snapshot age ${age_hours}h > ${max_age}h (snapshot: $latest_time)"
+  STATUS=FAIL
+else
+  log "OK L0: latest snapshot age ${age_hours}h"
+fi
+printf 'homelab_backup_audit_snapshot_age_hours{profile="%s"} %s\n' "$PROFILE" "$age_hours" >> "$METRICS_TMP"
+
+# ── L1: repository integrity ───────────────────────────────────────────────
+if restic check 2>&1 | tee -a "$LOG_FILE"; then
+  log "OK L1: restic check passed"
+  printf 'homelab_backup_audit_level_success{profile="%s",level="l1"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+else
+  log "FAIL L1: restic check failed"
+  printf 'homelab_backup_audit_level_success{profile="%s",level="l1"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+  STATUS=FAIL
+fi
+
+# ── L2: SQLite restore + integrity ─────────────────────────────────────────
+if [ -n "${HOMELAB_AUDIT_SQLITE_NAME:-}" ]; then
+  case "$TMP" in
+    /var/lib/homelab-backup-audit/[A-Za-z0-9_-]*) ;;
+    *)
+      log "FAIL L2: unsafe temporary path"
+      exit 64
+      ;;
+  esac
+  rm -rf "$TMP"
+  mkdir -p "$TMP"
+
+  if restic restore latest --target "$TMP" --include "**/${HOMELAB_AUDIT_SQLITE_NAME}" 2>&1 | tee -a "$LOG_FILE"; then
+    db=$(find "$TMP" -name "$HOMELAB_AUDIT_SQLITE_NAME" -type f | head -1)
+    if [ -z "$db" ] || [ ! -f "$db" ]; then
+      log "FAIL L2: $HOMELAB_AUDIT_SQLITE_NAME not found in restored data"
+      printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+      STATUS=FAIL
+    else
+      check=$(sqlite3 "$db" "PRAGMA integrity_check;" 2>&1)
+      if [ "$check" = "ok" ]; then
+        log "OK L2: SQLite integrity_check ok ($db)"
+        printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+      else
+        log "FAIL L2: SQLite integrity_check: $check"
+        printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+        STATUS=FAIL
+      fi
+    fi
+  else
+    log "FAIL L2: restic restore failed for $HOMELAB_AUDIT_SQLITE_NAME"
+    printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+    STATUS=FAIL
+  fi
+
+  rm -rf "$TMP"
+fi
+
+# ── L2: expected file restore ───────────────────────────────────────────────
+if [ -n "${HOMELAB_AUDIT_EXPECTED_NAME:-}" ]; then
+  case "$TMP" in
+    /var/lib/homelab-backup-audit/[A-Za-z0-9_-]*) ;;
+    *)
+      log "FAIL L2: unsafe temporary path"
+      exit 64
+      ;;
+  esac
+  rm -rf "$TMP"
+  mkdir -p "$TMP"
+
+  if restic restore latest --target "$TMP" --include "**/${HOMELAB_AUDIT_EXPECTED_NAME}" 2>&1 | tee -a "$LOG_FILE"; then
+    expected=$(find "$TMP" -name "$HOMELAB_AUDIT_EXPECTED_NAME" -type f | head -1)
+    if [ -n "$expected" ] && [ -s "$expected" ]; then
+      log "OK L2: restored non-empty expected file ($expected)"
+      printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+    else
+      log "FAIL L2: $HOMELAB_AUDIT_EXPECTED_NAME not found or empty"
+      printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+      STATUS=FAIL
+    fi
+  else
+    log "FAIL L2: restic restore failed for $HOMELAB_AUDIT_EXPECTED_NAME"
+    printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+    STATUS=FAIL
+  fi
+
+  rm -rf "$TMP"
+fi
+
+if [ "$STATUS" = "OK" ]; then
+  log "AUDIT PASSED"
+  printf 'homelab_backup_audit_success{profile="%s"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+  publish_metrics
+  exit 0
+else
+  log "AUDIT FAILED"
+  printf 'homelab_backup_audit_success{profile="%s"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+  publish_metrics
+  exit 1
+fi

ansible/roles/bash_config/templates/homelab.sh.j2

@@ -30,6 +30,23 @@ alias gs='git status --short --branch'
 alias gd='git diff'
 alias gl='git log --oneline --decorate -10'
 
+{% if bash_config_proxy_http_url | default('') | length > 0 %}
+proxy_on() {
+  export http_proxy="{{ bash_config_proxy_http_url }}"
+  export https_proxy="{{ bash_config_proxy_http_url }}"
+  export all_proxy="{{ bash_config_proxy_socks_url }}"
+  export HTTP_PROXY="$http_proxy"
+  export HTTPS_PROXY="$https_proxy"
+  export ALL_PROXY="$all_proxy"
+  export no_proxy="localhost,127.0.0.1,::1,192.168.1.0/24"
+  export NO_PROXY="$no_proxy"
+}
+
+proxy_off() {
+  unset http_proxy https_proxy all_proxy HTTP_PROXY HTTPS_PROXY ALL_PROXY no_proxy NO_PROXY
+}
+{% endif %}
+
 if command -v systemctl >/dev/null 2>&1; then
   alias sctl='systemctl'
   alias jctl='journalctl'

ansible/roles/emergency_access/defaults/main.yml

@@ -0,0 +1,11 @@
+---
+emergency_reverse_user: homelab-rescue
+emergency_reverse_port: 22010
+emergency_reverse_bind_address: 127.0.0.1
+emergency_reverse_tunnel_user: emergency-tunnel
+emergency_control_user: emergency-control
+emergency_control_key_path: /etc/emergency-access/control_ed25519
+emergency_tunnel_state_dir: /var/lib/emergency-reverse-ssh
+emergency_reverse_key_path: /var/lib/emergency-reverse-ssh/reverse_ed25519
+emergency_access_dir: /etc/emergency-access
+emergency_tunnel_ttl: 60m

ansible/roles/emergency_access/handlers/main.yml

@@ -0,0 +1,8 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
+
+- name: Validate and reload SSH
+  ansible.builtin.shell: sshd -t && systemctl reload ssh
+  changed_when: true

ansible/roles/emergency_access/tasks/client.yml

@@ -0,0 +1,134 @@
+---
+- name: Validate emergency access client inputs
+  ansible.builtin.assert:
+    that:
+      - emergency_vps_host_key is match('^ssh-(ed25519|rsa|ecdsa-[^ ]+) [A-Za-z0-9+/=]+( [A-Za-z0-9@._:-]+)?$')
+      - emergency_bot_control_public_key | length > 0
+    fail_msg: Set EMERGENCY_VPS_HOST_KEY and provision the emergency-bot control key first.
+  no_log: true
+
+- name: Install reverse SSH client
+  ansible.builtin.apt:
+    name:
+      - openssh-client
+      - sudo
+    state: present
+    update_cache: true
+
+- name: Create emergency access service accounts
+  ansible.builtin.user:
+    name: "{{ item }}"
+    system: true
+    # SSHD must start the forced command under a valid shell; authorized_keys
+    # still prevents this account from receiving an arbitrary command or PTY.
+    shell: "{{ '/bin/sh' if item == emergency_control_user else '/usr/sbin/nologin' }}"
+    create_home: true
+  loop:
+    - "{{ emergency_reverse_tunnel_user }}"
+    - "{{ emergency_control_user }}"
+
+- name: Create emergency access configuration directory
+  ansible.builtin.file:
+    path: "{{ emergency_access_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Create reverse SSH state directory
+  ansible.builtin.file:
+    path: "{{ emergency_tunnel_state_dir }}"
+    state: directory
+    owner: "{{ emergency_reverse_tunnel_user }}"
+    group: "{{ emergency_reverse_tunnel_user }}"
+    mode: "0700"
+
+- name: Generate dedicated reverse SSH key
+  ansible.builtin.command:
+    cmd: "ssh-keygen -q -t ed25519 -N '' -f {{ emergency_reverse_key_path }}"
+    creates: "{{ emergency_reverse_key_path }}"
+  become_user: "{{ emergency_reverse_tunnel_user }}"
+  no_log: true
+
+- name: Read reverse SSH public key
+  ansible.builtin.slurp:
+    src: "{{ emergency_reverse_key_path }}.pub"
+  register: emergency_reverse_key
+  no_log: true
+
+- name: Store reverse SSH public key for VPS configuration
+  ansible.builtin.set_fact:
+    emergency_reverse_public_key: "{{ emergency_reverse_key.content | b64decode | trim }}"
+  no_log: true
+
+- name: Pin ru-vps SSH host key for the tunnel user
+  ansible.builtin.copy:
+    dest: "{{ emergency_tunnel_state_dir }}/known_hosts"
+    content: "ru-vps-emergency {{ emergency_vps_host_key }}\n"
+    owner: "{{ emergency_reverse_tunnel_user }}"
+    group: "{{ emergency_reverse_tunnel_user }}"
+    mode: "0600"
+  no_log: true
+
+- name: Install reverse SSH systemd unit
+  ansible.builtin.template:
+    src: emergency-reverse-ssh.service.j2
+    dest: /etc/systemd/system/emergency-reverse-ssh.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install emergency tunnel expiry unit
+  ansible.builtin.template:
+    src: emergency-reverse-ssh-expire.service.j2
+    dest: /etc/systemd/system/emergency-reverse-ssh-expire.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install emergency tunnel expiry timer
+  ansible.builtin.template:
+    src: emergency-reverse-ssh.timer.j2
+    dest: /etc/systemd/system/emergency-reverse-ssh.timer
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install restricted emergency control helper
+  ansible.builtin.template:
+    src: emergency-control.j2
+    dest: /usr/local/libexec/emergency-control
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install emergency control sudo policy
+  ansible.builtin.template:
+    src: emergency-control.sudoers.j2
+    dest: /etc/sudoers.d/emergency-control
+    owner: root
+    group: root
+    mode: "0440"
+    validate: visudo -cf %s
+
+- name: Authorize emergency-bot control key with forced command
+  ansible.posix.authorized_key:
+    user: "{{ emergency_control_user }}"
+    key: "command=\"/usr/local/libexec/emergency-control\",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty {{ emergency_bot_control_public_key }}"
+    state: present
+    exclusive: true
+  no_log: true
+
+- name: Reload systemd before managing emergency units
+  ansible.builtin.meta: flush_handlers
+
+- name: Keep emergency tunnel disabled at boot
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    enabled: false
+  loop:
+    - emergency-reverse-ssh.service
+    - emergency-reverse-ssh.timer

ansible/roles/emergency_access/tasks/endpoint.yml

@@ -0,0 +1,24 @@
+---
+- name: Validate emergency access endpoint input
+  ansible.builtin.assert:
+    that:
+      - emergency_reverse_public_key | length > 0
+    fail_msg: Configure the mini-pc reverse SSH client before ru-vps.
+  no_log: true
+
+- name: Create restricted reverse SSH endpoint user
+  ansible.builtin.user:
+    name: "{{ emergency_reverse_user }}"
+    system: true
+    # Remote forwarding is accepted before a session exists. A valid shell is
+    # required by SSHD; the forced command below rejects every session.
+    shell: /bin/sh
+    create_home: true
+
+- name: Authorize only the dedicated reverse SSH key
+  ansible.posix.authorized_key:
+    user: "{{ emergency_reverse_user }}"
+    key: "command=\"/usr/bin/false\",restrict,port-forwarding,permitlisten=\"{{ emergency_reverse_bind_address }}:{{ emergency_reverse_port }}\" {{ emergency_reverse_public_key }}"
+    state: present
+    exclusive: true
+  no_log: true

ansible/roles/emergency_access/templates/emergency-control.j2

@@ -0,0 +1,27 @@
+#!/bin/sh
+set -eu
+
+case "${SSH_ORIGINAL_COMMAND:-}" in
+  start)
+    /usr/bin/sudo /usr/bin/systemctl start emergency-reverse-ssh.service
+    /usr/bin/sudo /usr/bin/systemctl restart emergency-reverse-ssh.timer
+    /usr/bin/systemctl is-active --quiet emergency-reverse-ssh.service
+    printf 'started; expires in {{ emergency_tunnel_ttl }}\nConnect:\nssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -J vps -p 22010 ansible@127.0.0.1\n'
+    ;;
+  stop)
+    /usr/bin/sudo /usr/bin/systemctl stop emergency-reverse-ssh.timer
+    /usr/bin/sudo /usr/bin/systemctl stop emergency-reverse-ssh.service
+    printf 'stopped\n'
+    ;;
+  status)
+    if /usr/bin/systemctl is-active --quiet emergency-reverse-ssh.service; then
+      /usr/bin/systemctl show --property=ActiveState --property=ActiveEnterTimestamp --value emergency-reverse-ssh.service
+    else
+      printf 'stopped\n'
+    fi
+    ;;
+  *)
+    printf 'unsupported command\n' >&2
+    exit 64
+    ;;
+esac

ansible/roles/emergency_access/templates/emergency-control.sudoers.j2

@@ -0,0 +1 @@
+{{ emergency_control_user }} ALL=(root) NOPASSWD: /usr/bin/systemctl start emergency-reverse-ssh.service, /usr/bin/systemctl restart emergency-reverse-ssh.timer, /usr/bin/systemctl stop emergency-reverse-ssh.timer, /usr/bin/systemctl stop emergency-reverse-ssh.service

ansible/roles/emergency_access/templates/emergency-reverse-ssh-expire.service.j2

@@ -0,0 +1,6 @@
+[Unit]
+Description=Close expired HomeLab reverse SSH rescue tunnel
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/systemctl stop emergency-reverse-ssh.service

ansible/roles/emergency_access/templates/emergency-reverse-ssh.service.j2

@@ -0,0 +1,19 @@
+[Unit]
+Description=Temporary HomeLab reverse SSH rescue tunnel
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User={{ emergency_reverse_tunnel_user }}
+ExecStart=/usr/bin/ssh -N -i {{ emergency_reverse_key_path }} -o BatchMode=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o StrictHostKeyChecking=yes -o UserKnownHostsFile={{ emergency_tunnel_state_dir }}/known_hosts -o HostKeyAlias=ru-vps-emergency -p 3422 -R {{ emergency_reverse_bind_address }}:{{ emergency_reverse_port }}:127.0.0.1:22 {{ emergency_reverse_user }}@157.22.231.198
+Restart=on-failure
+RestartSec=10
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=strict
+ProtectHome=yes
+ReadWritePaths={{ emergency_tunnel_state_dir }}
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/emergency_access/templates/emergency-reverse-ssh.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Expire HomeLab reverse SSH rescue tunnel after {{ emergency_tunnel_ttl }}
+
+[Timer]
+OnActiveSec={{ emergency_tunnel_ttl }}
+AccuracySec=1s
+Unit=emergency-reverse-ssh-expire.service
+
+[Install]
+WantedBy=timers.target

ansible/roles/emergency_bot/defaults/main.yml

@@ -0,0 +1,8 @@
+---
+emergency_bot_user: emergency-bot
+emergency_bot_state_dir: /var/lib/emergency-bot
+emergency_bot_config_dir: /etc/emergency-bot
+emergency_bot_control_key_path: /var/lib/emergency-bot/control_ed25519
+emergency_bot_mini_pc_host: 192.168.1.10
+emergency_bot_mini_pc_user: emergency-control
+emergency_telegram_proxy: http://192.168.1.27:7890

ansible/roles/emergency_bot/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
+
+- name: Restart emergency bot
+  ansible.builtin.systemd_service:
+    name: emergency-bot.service
+    state: restarted

ansible/roles/emergency_bot/tasks/bootstrap.yml

@@ -0,0 +1,48 @@
+---
+- name: Create emergency bot user
+  ansible.builtin.user:
+    name: "{{ emergency_bot_user }}"
+    system: true
+    shell: /usr/sbin/nologin
+    create_home: false
+
+- name: Create emergency bot state directory
+  ansible.builtin.file:
+    path: "{{ emergency_bot_state_dir }}"
+    state: directory
+    owner: "{{ emergency_bot_user }}"
+    group: "{{ emergency_bot_user }}"
+    mode: "0700"
+
+- name: Generate emergency bot control key
+  ansible.builtin.command:
+    cmd: "ssh-keygen -q -t ed25519 -N '' -f {{ emergency_bot_control_key_path }}"
+    creates: "{{ emergency_bot_control_key_path }}"
+  become: true
+  become_user: "{{ emergency_bot_user }}"
+  no_log: true
+
+- name: Set emergency bot control key ownership
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: file
+    owner: "{{ emergency_bot_user }}"
+    group: "{{ emergency_bot_user }}"
+    mode: "{{ item.mode }}"
+  loop:
+    - path: "{{ emergency_bot_control_key_path }}"
+      mode: "0600"
+    - path: "{{ emergency_bot_control_key_path }}.pub"
+      mode: "0644"
+  no_log: true
+
+- name: Read emergency bot control public key
+  ansible.builtin.slurp:
+    src: "{{ emergency_bot_control_key_path }}.pub"
+  register: emergency_bot_control_key
+  no_log: true
+
+- name: Store emergency bot control public key for mini-pc configuration
+  ansible.builtin.set_fact:
+    emergency_bot_control_public_key: "{{ emergency_bot_control_key.content | b64decode | trim }}"
+  no_log: true

ansible/roles/emergency_bot/tasks/main.yml

@@ -0,0 +1,77 @@
+---
+- name: Bootstrap emergency bot control identity
+  ansible.builtin.import_tasks: bootstrap.yml
+
+- name: Validate emergency bot inputs
+  ansible.builtin.assert:
+    that:
+      - emergency_bot_token is match('^[0-9]+:[A-Za-z0-9_-]+$')
+      - emergency_bot_allowed_user_ids is match('^[0-9]+(,[0-9]+)*$')
+      - emergency_mini_pc_host_key is match('^ssh-(ed25519|rsa|ecdsa-[^ ]+) [A-Za-z0-9+/=]+( [A-Za-z0-9@._:-]+)?$')
+    fail_msg: Set EMERGENCY_BOT_TOKEN, EMERGENCY_ALLOWED_USER_IDS and EMERGENCY_MINI_PC_HOST_KEY.
+  no_log: true
+
+- name: Install emergency bot dependencies
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - openssh-client
+      - python3
+    state: present
+    update_cache: true
+
+- name: Create emergency bot configuration directory
+  ansible.builtin.file:
+    path: "{{ emergency_bot_config_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Pin mini-pc SSH host key
+  ansible.builtin.copy:
+    dest: "{{ emergency_bot_state_dir }}/known_hosts"
+    content: "mini-pc-emergency {{ emergency_mini_pc_host_key }}\n"
+    owner: "{{ emergency_bot_user }}"
+    group: "{{ emergency_bot_user }}"
+    mode: "0600"
+  no_log: true
+
+- name: Install emergency bot runtime
+  ansible.builtin.template:
+    src: emergency_bot.py.j2
+    dest: /usr/local/libexec/emergency-bot
+    owner: root
+    group: root
+    mode: "0755"
+  notify: Restart emergency bot
+
+- name: Install emergency bot environment
+  ansible.builtin.template:
+    src: emergency-bot.env.j2
+    dest: "{{ emergency_bot_config_dir }}/bot.env"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: Restart emergency bot
+
+- name: Install emergency bot systemd unit
+  ansible.builtin.template:
+    src: emergency-bot.service.j2
+    dest: /etc/systemd/system/emergency-bot.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify:
+    - Reload systemd
+    - Restart emergency bot
+
+- name: Enable and start emergency bot
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start emergency bot
+  ansible.builtin.systemd_service:
+    name: emergency-bot.service
+    enabled: true
+    state: started

ansible/roles/emergency_bot/templates/emergency-bot.env.j2

@@ -0,0 +1,7 @@
+EMERGENCY_BOT_TOKEN={{ emergency_bot_token }}
+EMERGENCY_ALLOWED_USER_IDS={{ emergency_bot_allowed_user_ids }}
+EMERGENCY_MINI_PC_HOST={{ emergency_bot_mini_pc_host }}
+EMERGENCY_MINI_PC_USER={{ emergency_bot_mini_pc_user }}
+EMERGENCY_CONTROL_KEY={{ emergency_bot_control_key_path }}
+EMERGENCY_KNOWN_HOSTS={{ emergency_bot_state_dir }}/known_hosts
+EMERGENCY_TELEGRAM_PROXY={{ emergency_telegram_proxy }}

ansible/roles/emergency_bot/templates/emergency-bot.service.j2

@@ -0,0 +1,20 @@
+[Unit]
+Description=HomeLab emergency access Telegram bot
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User={{ emergency_bot_user }}
+EnvironmentFile={{ emergency_bot_config_dir }}/bot.env
+ExecStart=/usr/local/libexec/emergency-bot
+Restart=always
+RestartSec=10
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=strict
+ProtectHome=yes
+ReadWritePaths={{ emergency_bot_state_dir }}
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/emergency_bot/templates/emergency_bot.py.j2

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+import json
+import os
+import subprocess
+import sys
+import time
+import urllib.parse
+import urllib.request
+
+API = "https://api.telegram.org/bot" + os.environ["EMERGENCY_BOT_TOKEN"]
+ALLOWED_USERS = {item.strip() for item in os.environ["EMERGENCY_ALLOWED_USER_IDS"].split(",") if item.strip()}
+STATE_FILE = "{{ emergency_bot_state_dir }}/updates.json"
+BUTTONS = {
+    "inline_keyboard": [[
+        {"text": "Enable SSH", "callback_data": "start"},
+        {"text": "Status", "callback_data": "status"},
+        {"text": "Stop", "callback_data": "stop"},
+    ]]
+}
+
+
+HTTP = urllib.request.build_opener(
+    urllib.request.ProxyHandler({"https": os.environ["EMERGENCY_TELEGRAM_PROXY"]})
+)
+
+
+def api(method, payload=None, timeout=35):
+    if payload is not None:
+        payload = {
+            key: json.dumps(value) if isinstance(value, (dict, list)) else value
+            for key, value in payload.items()
+        }
+    data = None if payload is None else urllib.parse.urlencode(payload).encode()
+    with HTTP.open(API + "/" + method, data=data, timeout=timeout) as response:
+        result = json.load(response)
+    if not result.get("ok"):
+        raise RuntimeError(result.get("description", "Telegram API request failed"))
+    return result["result"]
+
+
+def control(command):
+    args = [
+        "/usr/bin/ssh", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=yes",
+        "-o", "HostKeyAlias=mini-pc-emergency", "-o", "UserKnownHostsFile=" + os.environ["EMERGENCY_KNOWN_HOSTS"],
+        "-i", os.environ["EMERGENCY_CONTROL_KEY"],
+        os.environ["EMERGENCY_MINI_PC_USER"] + "@" + os.environ["EMERGENCY_MINI_PC_HOST"], command,
+    ]
+    result = subprocess.run(args, capture_output=True, text=True, timeout=25, check=False)
+    output = (result.stdout or result.stderr).strip()
+    if result.returncode:
+        return "Control command failed: " + (output or "unknown error")[:1000]
+    return output or "ok"
+
+
+def reply(chat_id, text):
+    payload = {"chat_id": chat_id, "text": text[:3500], "reply_markup": BUTTONS}
+    if text.startswith("started; expires in") and "\nConnect:\n" in text:
+        status, command = text.split("\nConnect:\n", 1)
+        payload["text"] = status + "\nConnect:\n`" + command + "`"
+        payload["parse_mode"] = "Markdown"
+    api("sendMessage", payload)
+
+
+def allowed(chat, sender):
+    return chat.get("type") == "private" and str(sender.get("id")) in ALLOWED_USERS
+
+
+def handle_command(chat_id, command):
+    if command == "help":
+        reply(chat_id, "Choose an action or use /emergency ssh, /emergency status, /emergency stop")
+    elif command:
+        print("accepted emergency command: " + command, file=sys.stderr, flush=True)
+        reply(chat_id, control(command))
+
+
+def load_offset():
+    try:
+        with open(STATE_FILE, encoding="utf-8") as state_file:
+            return int(json.load(state_file).get("offset", 0))
+    except (FileNotFoundError, ValueError, json.JSONDecodeError):
+        return 0
+
+
+def save_offset(offset):
+    temporary = STATE_FILE + ".tmp"
+    with open(temporary, "w", encoding="utf-8") as state_file:
+        json.dump({"offset": offset}, state_file)
+    os.replace(temporary, STATE_FILE)
+
+
+def command_for(text):
+    parts = text.strip().split()
+    if not parts:
+        return None
+    command_name = parts[0].split("@", 1)[0]
+    if command_name == "/start":
+        return "help"
+    if command_name != "/emergency":
+        return None
+    if len(parts) != 2:
+        return "help"
+    return {"ssh": "start", "stop": "stop", "status": "status"}.get(parts[1], "help")
+
+
+def run():
+    offset = load_offset()
+    while True:
+        try:
+            for update in api("getUpdates", {"offset": offset, "timeout": 30}, timeout=40):
+                offset = update["update_id"] + 1
+                save_offset(offset)
+                callback = update.get("callback_query")
+                if callback:
+                    callback_message = callback.get("message", {})
+                    chat = callback_message.get("chat", {})
+                    sender = callback.get("from", {})
+                    command = callback.get("data")
+                    if not allowed(chat, sender) or command not in {"start", "status", "stop"}:
+                        print("ignored callback: chat_type=" + str(chat.get("type")) + " sender=" + str(sender.get("id")), file=sys.stderr, flush=True)
+                        api("answerCallbackQuery", {"callback_query_id": callback["id"], "text": "Not authorized", "show_alert": True})
+                    else:
+                        api("answerCallbackQuery", {"callback_query_id": callback["id"]})
+                        handle_command(chat["id"], command)
+                    continue
+                message = update.get("message", {})
+                chat = message.get("chat", {})
+                sender = message.get("from", {})
+                text = message.get("text", "")
+                if not allowed(chat, sender):
+                    print("ignored update: chat_type=" + str(chat.get("type")) + " sender=" + str(sender.get("id")), file=sys.stderr, flush=True)
+                    continue
+                command = command_for(text)
+                handle_command(chat["id"], command)
+        except Exception as error:
+            print("emergency-bot error: " + str(error), file=sys.stderr, flush=True)
+            time.sleep(10)
+
+
+if __name__ == "__main__":
+    run()

ansible/roles/gyro/defaults/main.yml

@@ -0,0 +1,28 @@
+---
+gyro_user: gyro
+gyro_group: gyro
+gyro_home: /home/gyro
+gyro_app_dir: /opt/gyro/app
+gyro_config_dir: /etc/gyro
+gyro_cache_dir: /var/cache/gyro
+gyro_uv_venv: /opt/uv
+gyro_uv_version: 0.12.5
+gyro_python_min_version: "3.13"
+gyro_timezone: Europe/Moscow
+
+gyro_repo_url: ""
+gyro_repo_version: main
+gyro_git_known_hosts_name: ""
+gyro_git_host_key: ""
+gyro_deploy_enabled: false
+
+gyro_tinvest_token: ""
+gyro_tinvest_account_id: ""
+gyro_telegram_bot_token: ""
+gyro_telegram_user_id: ""
+gyro_telegram_proxy: ""
+gyro_dry_run_override: "true"
+gyro_secrets_configured: false
+
+gyro_timer_enabled: false
+gyro_timer_on_calendar: "Mon..Fri *-*-* 11:00:00 Europe/Moscow"

ansible/roles/gyro/files/gyro-failure-notify.py

@@ -0,0 +1,37 @@
+#!/usr/bin/python3
+import os
+import socket
+import sys
+import urllib.parse
+import urllib.request
+
+
+def main() -> int:
+    token = os.environ.get("TELEGRAM_BOT_TOKEN", "").strip()
+    user_id = os.environ.get("TELEGRAM_USER_ID", "").strip()
+    if not token or not user_id:
+        print("Gyro failure notification skipped: Telegram credentials are absent")
+        return 1
+
+    proxy = os.environ.get("TELEGRAM_PROXY", "").strip()
+    handlers = [urllib.request.ProxyHandler({"http": proxy, "https": proxy})] if proxy else []
+    opener = urllib.request.build_opener(*handlers)
+    failed_unit = sys.argv[1] if len(sys.argv) > 1 else "gyro.service"
+    message = f"Gyro job failed on {socket.gethostname()}: {failed_unit}. Check journalctl -u gyro.service."
+    body = urllib.parse.urlencode({"chat_id": user_id, "text": message}).encode()
+    request = urllib.request.Request(
+        f"https://api.telegram.org/bot{token}/sendMessage",
+        data=body,
+        method="POST",
+    )
+
+    try:
+        with opener.open(request, timeout=20) as response:
+            return 0 if response.status == 200 else 1
+    except Exception as exc:
+        print(f"Gyro failure notification failed: {type(exc).__name__}")
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ansible/roles/gyro/handlers/main.yml

@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true

ansible/roles/gyro/tasks/main.yml

@@ -0,0 +1,365 @@
+---
+- name: Install Gyro runtime packages
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - git
+      - openssh-client
+      - python3
+      - python3-packaging
+      - python3-venv
+      - sudo
+      - ufw
+    state: present
+    update_cache: true
+
+- name: Configure Gyro timezone
+  community.general.timezone:
+    name: "{{ gyro_timezone }}"
+
+- name: Create Gyro service group
+  ansible.builtin.group:
+    name: "{{ gyro_group }}"
+    system: true
+
+- name: Create Gyro service user
+  ansible.builtin.user:
+    name: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    home: "{{ gyro_home }}"
+    shell: /bin/bash
+    system: true
+    create_home: true
+
+- name: Create Gyro directories
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    owner: "{{ item.owner }}"
+    group: "{{ item.group }}"
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ gyro_app_dir }}", owner: "{{ gyro_user }}", group: "{{ gyro_group }}", mode: "0750" }
+    - { path: "{{ gyro_config_dir }}", owner: root, group: root, mode: "0700" }
+    - { path: "{{ gyro_cache_dir }}", owner: "{{ gyro_user }}", group: "{{ gyro_group }}", mode: "0700" }
+    - { path: "{{ gyro_home }}/.ssh", owner: "{{ gyro_user }}", group: "{{ gyro_group }}", mode: "0700" }
+
+- name: Read system Python version
+  ansible.builtin.command: python3 -c "import platform; print(platform.python_version())"
+  register: gyro_python_version
+  changed_when: false
+
+- name: Require Python 3.13 or newer
+  ansible.builtin.assert:
+    that:
+      - gyro_python_version.stdout is version(gyro_python_min_version, '>=')
+    fail_msg: "Gyro requires Python {{ gyro_python_min_version }} or newer; found {{ gyro_python_version.stdout }}."
+
+- name: Create isolated uv installation environment
+  ansible.builtin.command:
+    argv:
+      - python3
+      - -m
+      - venv
+      - "{{ gyro_uv_venv }}"
+    creates: "{{ gyro_uv_venv }}/bin/pip"
+
+- name: Install pinned uv version
+  ansible.builtin.pip:
+    name: "uv=={{ gyro_uv_version }}"
+    executable: "{{ gyro_uv_venv }}/bin/pip"
+
+- name: Link uv into the system path
+  ansible.builtin.file:
+    src: "{{ gyro_uv_venv }}/bin/uv"
+    dest: /usr/local/bin/uv
+    state: link
+
+- name: Generate Git deploy key on the container
+  ansible.builtin.command:
+    argv:
+      - ssh-keygen
+      - -q
+      - -t
+      - ed25519
+      - -N
+      - ""
+      - -C
+      - gyro@homelab
+      - -f
+      - "{{ gyro_home }}/.ssh/id_ed25519_gitea"
+    creates: "{{ gyro_home }}/.ssh/id_ed25519_gitea"
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+
+- name: Secure Git deploy key ownership
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    owner: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ gyro_home }}/.ssh/id_ed25519_gitea", mode: "0600" }
+    - { path: "{{ gyro_home }}/.ssh/id_ed25519_gitea.pub", mode: "0644" }
+
+- name: Read Git deploy public key
+  ansible.builtin.slurp:
+    src: "{{ gyro_home }}/.ssh/id_ed25519_gitea.pub"
+  register: gyro_deploy_public_key
+
+- name: Show Git deploy public key
+  ansible.builtin.debug:
+    msg: "{{ gyro_deploy_public_key.content | b64decode | trim }}"
+
+- name: Allow SSH from the HomeLab LAN
+  community.general.ufw:
+    rule: allow
+    port: "22"
+    proto: tcp
+    src: "{{ homelab_lan_cidr }}"
+
+- name: Allow SSH from the OpenVPN network
+  community.general.ufw:
+    rule: allow
+    port: "22"
+    proto: tcp
+    src: "{{ openvpn_network_cidr }}"
+
+- name: Remove obsolete outbound Gitea SSH allowance
+  community.general.ufw:
+    rule: allow
+    delete: true
+    direction: out
+    dest: 192.168.1.25
+    port: "2222"
+    proto: tcp
+
+- name: Allow outbound Telegram proxy access
+  community.general.ufw:
+    rule: allow
+    direction: out
+    dest: 192.168.1.27
+    port: "7890"
+    proto: tcp
+
+- name: Deny other outbound HomeLab LAN access
+  community.general.ufw:
+    rule: deny
+    direction: out
+    dest: "{{ homelab_lan_cidr }}"
+
+- name: Enable restrictive Gyro firewall
+  community.general.ufw:
+    state: enabled
+    policy: deny
+    direction: incoming
+
+- name: Mask mount units already provided by unprivileged LXC
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    enabled: false
+    masked: true
+    state: stopped
+  loop:
+    - dev-mqueue.mount
+    - run-lock.mount
+    - tmp.mount
+
+- name: Clear stale failures from masked LXC mount units
+  ansible.builtin.command: >-
+    systemctl reset-failed dev-mqueue.mount run-lock.mount tmp.mount
+  changed_when: false
+
+- name: Create locked placeholder environment file
+  ansible.builtin.copy:
+    dest: "{{ gyro_config_dir }}/gyro.env"
+    owner: root
+    group: root
+    mode: "0600"
+    force: false
+    content: |
+      # Managed by Ansible after gyro_secrets_configured is enabled.
+      DRY_RUN_OVERRIDE=true
+
+- name: Validate deployment inputs
+  ansible.builtin.assert:
+    that:
+      - gyro_repo_url | length > 0
+      - gyro_git_known_hosts_name | length > 0
+      - gyro_git_host_key | length > 0
+    fail_msg: Set the repository URL and verified SSH host key before enabling deployment.
+  when: gyro_deploy_enabled | bool
+
+- name: Remove obsolete Git SSH alias
+  ansible.builtin.file:
+    path: "{{ gyro_home }}/.ssh/config"
+    state: absent
+  when: gyro_deploy_enabled | bool
+
+- name: Remove obsolete Gitea known host
+  ansible.builtin.known_hosts:
+    path: "{{ gyro_home }}/.ssh/known_hosts"
+    name: "[192.168.1.25]:2222"
+    state: absent
+  when: gyro_deploy_enabled | bool
+
+- name: Pin verified Git host key
+  ansible.builtin.known_hosts:
+    path: "{{ gyro_home }}/.ssh/known_hosts"
+    name: "{{ gyro_git_known_hosts_name }}"
+    key: "{{ gyro_git_host_key }}"
+  when: gyro_deploy_enabled | bool
+
+- name: Secure Git known hosts file
+  ansible.builtin.file:
+    path: "{{ gyro_home }}/.ssh/known_hosts"
+    owner: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    mode: "0600"
+  when: gyro_deploy_enabled | bool
+
+- name: Ensure Gyro checkout belongs to the service user
+  ansible.builtin.file:
+    path: "{{ gyro_app_dir }}"
+    owner: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    recurse: true
+  when: gyro_deploy_enabled | bool
+
+- name: Clone Gyro from Git remote
+  ansible.builtin.git:
+    repo: "{{ gyro_repo_url }}"
+    dest: "{{ gyro_app_dir }}"
+    version: "{{ gyro_repo_version }}"
+    key_file: "{{ gyro_home }}/.ssh/id_ed25519_gitea"
+    accept_hostkey: false
+    ssh_opts: >-
+      -o UserKnownHostsFile={{ gyro_home }}/.ssh/known_hosts
+      -o StrictHostKeyChecking=yes
+      -o IdentitiesOnly=yes
+    update: true
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+  when: gyro_deploy_enabled | bool
+
+- name: Synchronize locked Gyro dependencies
+  ansible.builtin.command:
+    argv:
+      - /usr/local/bin/uv
+      - sync
+      - --frozen
+  args:
+    chdir: "{{ gyro_app_dir }}"
+  environment:
+    UV_CACHE_DIR: "{{ gyro_cache_dir }}/uv"
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+  register: gyro_uv_sync
+  changed_when: "'Installed' in gyro_uv_sync.stderr or 'Uninstalled' in gyro_uv_sync.stderr"
+  when: gyro_deploy_enabled | bool
+
+- name: Verify bundled T-Invest CA file
+  ansible.builtin.stat:
+    path: "{{ gyro_app_dir }}/config/certs/russian_ca.pem"
+  register: gyro_ca_bundle
+  when: gyro_deploy_enabled | bool
+
+- name: Require complete Gyro checkout
+  ansible.builtin.assert:
+    that:
+      - gyro_ca_bundle.stat.exists
+      - gyro_ca_bundle.stat.isreg | default(false)
+    fail_msg: The Git checkout does not contain config/certs/russian_ca.pem.
+  when: gyro_deploy_enabled | bool
+
+- name: Run Gyro unit tests
+  ansible.builtin.command:
+    argv:
+      - /usr/local/bin/uv
+      - run
+      - --frozen
+      - --no-sync
+      - python
+      - -m
+      - unittest
+      - discover
+      - -s
+      - tests
+  args:
+    chdir: "{{ gyro_app_dir }}"
+  environment:
+    UV_CACHE_DIR: "{{ gyro_cache_dir }}/uv"
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+  changed_when: false
+  when: gyro_deploy_enabled | bool
+
+- name: Validate Gyro Vault secrets
+  ansible.builtin.assert:
+    that:
+      - gyro_tinvest_token | length > 0
+      - gyro_tinvest_account_id | length > 0
+      - gyro_telegram_bot_token | length > 0
+      - (gyro_telegram_user_id | string | length) > 0
+      - gyro_dry_run_override in ['true', 'false']
+    fail_msg: Populate and encrypt inventory/host_vars/gyro/vault.yml before enabling secrets.
+  no_log: true
+  when: gyro_secrets_configured | bool
+
+- name: Install Gyro environment file from Vault
+  ansible.builtin.template:
+    src: gyro.env.j2
+    dest: "{{ gyro_config_dir }}/gyro.env"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  when: gyro_secrets_configured | bool
+
+- name: Install Gyro failure notifier
+  ansible.builtin.copy:
+    src: gyro-failure-notify.py
+    dest: /usr/local/libexec/gyro-failure-notify
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install Gyro systemd units
+  ansible.builtin.template:
+    src: "{{ item.src }}"
+    dest: "/etc/systemd/system/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - { src: gyro.service.j2, dest: gyro.service }
+    - { src: gyro.timer.j2, dest: gyro.timer }
+    - { src: gyro-failure@.service.j2, dest: "gyro-failure@.service" }
+  notify: Reload systemd
+
+- name: Apply systemd unit changes
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable Gyro timer only after deployment and secret setup
+  ansible.builtin.systemd:
+    name: gyro.timer
+    enabled: "{{ gyro_timer_ready }}"
+    state: "{{ 'started' if gyro_timer_ready else 'stopped' }}"
+  vars:
+    gyro_timer_ready: "{{ gyro_timer_enabled | bool and gyro_deploy_enabled | bool and gyro_secrets_configured | bool }}"
+
+- name: Verify Gyro unit definitions
+  ansible.builtin.command: >-
+    systemd-analyze verify
+    /etc/systemd/system/gyro.service
+    /etc/systemd/system/gyro.timer
+    /etc/systemd/system/gyro-failure@.service
+  changed_when: false

ansible/roles/gyro/templates/gyro-failure@.service.j2

@@ -0,0 +1,15 @@
+[Unit]
+Description=Notify Telegram about failed Gyro unit %i
+
+[Service]
+Type=oneshot
+User={{ gyro_user }}
+Group={{ gyro_group }}
+EnvironmentFile={{ gyro_config_dir }}/gyro.env
+ExecStart=/usr/local/libexec/gyro-failure-notify %i
+UMask=0077
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=strict
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

ansible/roles/gyro/templates/gyro.env.j2

@@ -0,0 +1,6 @@
+TINVEST_TOKEN={{ gyro_tinvest_token | string | to_json }}
+TINVEST_ACCOUNT_ID={{ gyro_tinvest_account_id | string | to_json }}
+TELEGRAM_BOT_TOKEN={{ gyro_telegram_bot_token | string | to_json }}
+TELEGRAM_USER_ID={{ gyro_telegram_user_id | string | to_json }}
+TELEGRAM_PROXY={{ gyro_telegram_proxy | string | to_json }}
+DRY_RUN_OVERRIDE={{ gyro_dry_run_override | string | to_json }}

ansible/roles/gyro/templates/gyro.service.j2

@@ -0,0 +1,31 @@
+[Unit]
+Description=Gyro investment allocator
+Wants=network-online.target
+After=network-online.target
+OnFailure=gyro-failure@%n.service
+
+[Service]
+Type=oneshot
+User={{ gyro_user }}
+Group={{ gyro_group }}
+WorkingDirectory={{ gyro_app_dir }}
+EnvironmentFile={{ gyro_config_dir }}/gyro.env
+Environment=UV_CACHE_DIR={{ gyro_cache_dir }}/uv
+Environment=PYTHONDONTWRITEBYTECODE=1
+ExecStart=/usr/local/bin/uv run --frozen --no-sync python main.py
+UMask=0077
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectClock=true
+ProtectControlGroups=true
+ProtectHome=true
+ProtectHostname=true
+ProtectKernelLogs=true
+ProtectKernelModules=true
+ProtectKernelTunables=true
+ProtectSystem=strict
+ReadWritePaths={{ gyro_cache_dir }}
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+RestrictNamespaces=true
+LockPersonality=true
+MemoryDenyWriteExecute=true

ansible/roles/gyro/templates/gyro.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run Gyro investment allocator on weekdays
+
+[Timer]
+OnCalendar={{ gyro_timer_on_calendar }}
+Persistent=true
+Unit=gyro.service
+
+[Install]
+WantedBy=timers.target

ansible/roles/monitoring_blackbox/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+monitoring_blackbox_listen_address: 127.0.0.1:9115
+monitoring_pushgateway_url: http://192.168.1.30:9091

ansible/roles/monitoring_blackbox/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: restart blackbox exporter
+  ansible.builtin.systemd:
+    name: prometheus-blackbox-exporter
+    state: restarted

ansible/roles/monitoring_blackbox/tasks/main.yml

@@ -0,0 +1,96 @@
+---
+- name: Install Blackbox Exporter
+  ansible.builtin.apt:
+    name:
+      - curl
+      - prometheus-blackbox-exporter
+    state: present
+    update_cache: true
+
+- name: Install external probe configuration
+  ansible.builtin.template:
+    src: blackbox.yml.j2
+    dest: /etc/prometheus/blackbox.yml
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart blackbox exporter
+
+- name: Create Blackbox Exporter systemd override directory
+  ansible.builtin.file:
+    path: /etc/systemd/system/prometheus-blackbox-exporter.service.d
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Configure Blackbox Exporter listen address
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/prometheus-blackbox-exporter.service.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Service]
+      ExecStart=
+      ExecStart=/usr/bin/prometheus-blackbox-exporter --config.file=/etc/prometheus/blackbox.yml --web.listen-address={{ monitoring_blackbox_listen_address }}
+  notify:
+    - reload systemd
+    - restart blackbox exporter
+
+- name: Install ru-vps metrics push script
+  ansible.builtin.template:
+    src: push-metrics.sh.j2
+    dest: /usr/local/sbin/homelab-monitoring-push
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install ru-vps metrics push service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-monitoring-push.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Push ru-vps monitoring metrics through OpenVPN
+      Wants=network-online.target
+      After=network-online.target
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/local/sbin/homelab-monitoring-push
+  notify: reload systemd
+
+- name: Install ru-vps metrics push timer
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-monitoring-push.timer
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Run ru-vps monitoring metric push every minute
+
+      [Timer]
+      OnBootSec=2m
+      OnUnitActiveSec=1m
+      Persistent=true
+
+      [Install]
+      WantedBy=timers.target
+  notify: reload systemd
+
+- name: Ensure Blackbox Exporter is enabled and running
+  ansible.builtin.systemd:
+    name: prometheus-blackbox-exporter
+    enabled: true
+    state: started
+
+- name: Enable ru-vps metrics push timer
+  ansible.builtin.systemd:
+    name: homelab-monitoring-push.timer
+    daemon_reload: true
+    enabled: true
+    state: started

ansible/roles/monitoring_blackbox/templates/blackbox.yml.j2

@@ -0,0 +1,13 @@
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [200, 301, 302]
+  tcp_connect:
+    prober: tcp
+    timeout: 10s
+  icmp:
+    prober: icmp
+    timeout: 10s

ansible/roles/monitoring_blackbox/templates/push-metrics.sh.j2

@@ -0,0 +1,37 @@
+#!/bin/sh
+# Managed by Ansible. Push local ru-vps metrics through the OpenVPN route.
+set -eu
+
+PUSHGATEWAY_URL="{{ monitoring_pushgateway_url }}"
+NOW=$(date +%s)
+NODE_METRICS=$(mktemp)
+
+trap 'rm -f "$NODE_METRICS"' EXIT
+
+push_stdin() {
+  path="$1"
+  curl --fail --silent --show-error --data-binary @- "$PUSHGATEWAY_URL/metrics/$path"
+}
+
+curl --fail --silent --show-error http://127.0.0.1:9100/metrics > "$NODE_METRICS"
+push_stdin "job/node/instance/ru-vps" < "$NODE_METRICS"
+
+probe() {
+  name="$1"
+  module="$2"
+  target="$3"
+  payload=$(curl --fail --silent --show-error --get \
+    --data-urlencode "module=$module" \
+    --data-urlencode "target=$target" \
+    http://127.0.0.1:9115/probe || printf 'probe_success 0\n')
+  payload=$(printf '%s\nhomelab_vps_probe_timestamp_seconds %s\n' "$payload" "$NOW")
+  printf '%s\n' "$payload" | push_stdin "job/blackbox-vps/probe_name/$name"
+}
+
+probe vaultwarden http_2xx https://pass.ada-dev.ru
+probe gitea http_2xx https://git.ada-dev.ru
+probe grimmory http_2xx https://books.ada-dev.ru/api/v1/healthcheck
+probe ovpn_gateway tcp_connect 10.78.0.2:22
+probe cloud_pve tcp_connect 192.168.1.5:8006
+probe mini_pve tcp_connect 192.168.1.10:8006
+probe pbs tcp_connect 192.168.1.20:8007

ansible/roles/monitoring_exporter/defaults/main.yml

@@ -0,0 +1,6 @@
+---
+monitoring_exporter_install_node: true
+monitoring_exporter_install_smart: false
+monitoring_exporter_textfile_dir: /var/lib/node_exporter/textfile_collector
+monitoring_exporter_node_listen_address: 0.0.0.0:9100
+monitoring_exporter_smartctl_timer: "*:0/15"

ansible/roles/monitoring_exporter/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: restart node exporter
+  ansible.builtin.systemd:
+    name: prometheus-node-exporter
+    state: restarted

ansible/roles/monitoring_exporter/tasks/main.yml

@@ -0,0 +1,121 @@
+---
+- name: Install Node Exporter
+  ansible.builtin.apt:
+    name: prometheus-node-exporter
+    state: present
+  when: monitoring_exporter_install_node
+
+- name: Create Node Exporter textfile directory
+  ansible.builtin.file:
+    path: "{{ monitoring_exporter_textfile_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  when: monitoring_exporter_install_node
+
+- name: Create Node Exporter systemd override directory
+  ansible.builtin.file:
+    path: /etc/systemd/system/prometheus-node-exporter.service.d
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  when: monitoring_exporter_install_node
+
+- name: Configure Node Exporter collectors
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/prometheus-node-exporter.service.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Service]
+      ExecStart=
+      ExecStart=/usr/bin/prometheus-node-exporter --web.listen-address={{ monitoring_exporter_node_listen_address }} --collector.systemd --collector.textfile.directory={{ monitoring_exporter_textfile_dir }}
+  when: monitoring_exporter_install_node
+  notify:
+    - reload systemd
+    - restart node exporter
+
+- name: Ensure Node Exporter is enabled and running
+  ansible.builtin.systemd:
+    name: prometheus-node-exporter
+    enabled: true
+    state: started
+  when: monitoring_exporter_install_node
+
+- name: Install SMART Exporter dependencies
+  ansible.builtin.apt:
+    name:
+      - jq
+      - smartmontools
+    state: present
+  when: monitoring_exporter_install_smart
+
+- name: Remove obsolete SMART Exporter container unit
+  ansible.builtin.systemd:
+    name: homelab-smartctl-exporter
+    enabled: false
+    state: stopped
+  failed_when: false
+  when: monitoring_exporter_install_smart
+
+- name: Install SMART metric collection script
+  ansible.builtin.template:
+    src: smartctl-metrics.sh.j2
+    dest: /usr/local/sbin/homelab-smartctl-metrics
+    owner: root
+    group: root
+    mode: "0755"
+  when: monitoring_exporter_install_smart
+
+- name: Install SMART metric collection service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-smartctl-metrics.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Publish SMART metrics for Node Exporter
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/local/sbin/homelab-smartctl-metrics
+  when: monitoring_exporter_install_smart
+  notify: reload systemd
+
+- name: Install SMART metric collection timer
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-smartctl-metrics.timer
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Collect SMART metrics every 15 minutes
+
+      [Timer]
+      OnBootSec=2m
+      OnUnitActiveSec=15m
+      Persistent=true
+
+      [Install]
+      WantedBy=multi-user.target
+  when: monitoring_exporter_install_smart
+  notify: reload systemd
+
+- name: Enable SMART metric collection timer
+  ansible.builtin.systemd:
+    name: homelab-smartctl-metrics.timer
+    daemon_reload: true
+    enabled: true
+    state: started
+  when: monitoring_exporter_install_smart
+
+- name: Run initial SMART metric collection
+  ansible.builtin.systemd:
+    name: homelab-smartctl-metrics.service
+    state: started
+  when: monitoring_exporter_install_smart

ansible/roles/monitoring_exporter/templates/smartctl-metrics.sh.j2

@@ -0,0 +1,30 @@
+#!/bin/sh
+# Managed by Ansible. Export SMART health to Node Exporter's textfile collector.
+set -eu
+
+METRICS_DIR="{{ monitoring_exporter_textfile_dir }}"
+METRICS_FILE="$METRICS_DIR/homelab_smartctl.prom"
+METRICS_TMP=$(mktemp "$METRICS_FILE.XXXXXX")
+
+trap 'rm -f "$METRICS_TMP"' EXIT
+
+printf '# HELP homelab_smart_device_healthy SMART overall health (1 healthy, 0 failed)\n' >> "$METRICS_TMP"
+printf '# TYPE homelab_smart_device_healthy gauge\n' >> "$METRICS_TMP"
+
+lsblk -dn -o NAME,TYPE | while read -r name type; do
+  [ "$type" = "disk" ] || continue
+  device="/dev/$name"
+  json=$(smartctl -a -j "$device" 2>/dev/null || true)
+  [ -n "$json" ] || continue
+
+  healthy=$(printf '%s' "$json" | jq -r 'if .smart_status.passed == true then 1 else 0 end')
+  temperature=$(printf '%s' "$json" | jq -r '.temperature.current // .nvme_smart_health_information_log.temperature // empty')
+  nvme_warning=$(printf '%s' "$json" | jq -r '.nvme_smart_health_information_log.critical_warning // empty')
+
+  printf 'homelab_smart_device_healthy{device="%s"} %s\n' "$name" "$healthy" >> "$METRICS_TMP"
+  [ -z "$temperature" ] || printf 'homelab_smart_temperature_celsius{device="%s"} %s\n' "$name" "$temperature" >> "$METRICS_TMP"
+  [ -z "$nvme_warning" ] || printf 'homelab_smart_nvme_critical_warning{device="%s"} %s\n' "$name" "$nvme_warning" >> "$METRICS_TMP"
+done
+
+chmod 0644 "$METRICS_TMP"
+mv "$METRICS_TMP" "$METRICS_FILE"

ansible/roles/monitoring_server/defaults/main.yml

@@ -0,0 +1,14 @@
+---
+monitoring_root: /opt/monitoring
+monitoring_prometheus_retention: 30d
+monitoring_prometheus_retention_size: 8GB
+monitoring_grafana_admin_user: admin
+monitoring_grafana_admin_password: "{{ lookup('env', 'MONITORING_GRAFANA_ADMIN_PASSWORD') }}"
+monitoring_telegram_bot_token: "{{ lookup('env', 'MONITORING_TELEGRAM_BOT_TOKEN') }}"
+monitoring_telegram_chat_id: "{{ lookup('env', 'MONITORING_TELEGRAM_CHAT_ID') }}"
+monitoring_pve_api_user: "{{ lookup('env', 'MONITORING_PVE_API_USER') }}"
+monitoring_pve_api_token_id: "{{ lookup('env', 'MONITORING_PVE_API_TOKEN_ID') }}"
+monitoring_pve_api_token_secret: "{{ lookup('env', 'MONITORING_PVE_API_TOKEN_SECRET') }}"
+monitoring_lan_cidr: 192.168.1.0/24
+monitoring_openvpn_gateway_ip: 192.168.1.23
+monitoring_docker_cidr: 172.16.0.0/12

ansible/roles/monitoring_server/files/dashboards/backups.json

@@ -0,0 +1,26 @@
+{
+  "title": "Backups",
+  "uid": "homelab-backups",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "table",
+      "title": "Audit Result",
+      "gridPos": {"h": 10, "w": 12, "x": 0, "y": 0},
+      "targets": [{"expr": "homelab_backup_audit_success", "format": "table", "instant": true, "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Backup Age (Hours)",
+      "gridPos": {"h": 10, "w": 12, "x": 12, "y": 0},
+      "targets": [{"expr": "homelab_backup_audit_snapshot_age_hours", "legendFormat": "{{profile}} {{vmid}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Audit Levels",
+      "gridPos": {"h": 10, "w": 24, "x": 0, "y": 10},
+      "targets": [{"expr": "homelab_backup_audit_level_success", "legendFormat": "{{profile}} {{level}}", "refId": "A"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/dashboards/homelab-overview.json

@@ -0,0 +1,34 @@
+{
+  "title": "HomeLab Overview",
+  "uid": "homelab-overview",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "stat",
+      "title": "Active Alerts",
+      "gridPos": {"h": 8, "w": 6, "x": 0, "y": 0},
+      "targets": [{"expr": "count(ALERTS{alertstate=\"firing\"})", "refId": "A"}],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "type": "stat",
+      "title": "Successful Probes",
+      "gridPos": {"h": 8, "w": 6, "x": 6, "y": 0},
+      "targets": [{"expr": "sum(probe_success)", "refId": "A"}],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "type": "timeseries",
+      "title": "Filesystem Free Space",
+      "gridPos": {"h": 10, "w": 12, "x": 0, "y": 8},
+      "targets": [{"expr": "node_filesystem_avail_bytes{fstype!~\"tmpfs|overlay\"}", "legendFormat": "{{instance}} {{mountpoint}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Backup Audit Success",
+      "gridPos": {"h": 10, "w": 12, "x": 12, "y": 8},
+      "targets": [{"expr": "homelab_backup_audit_success", "legendFormat": "{{profile}} {{instance}}", "refId": "A"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/dashboards/proxmox-storage.json

@@ -0,0 +1,26 @@
+{
+  "title": "Proxmox and Storage",
+  "uid": "proxmox-storage",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "timeseries",
+      "title": "Node CPU Usage",
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 0},
+      "targets": [{"expr": "1 - avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))", "legendFormat": "{{instance}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Node Memory Available",
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 0},
+      "targets": [{"expr": "node_memory_MemAvailable_bytes", "legendFormat": "{{instance}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Disk Read and Write",
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 9},
+      "targets": [{"expr": "rate(node_disk_read_bytes_total[5m])", "legendFormat": "read {{instance}} {{device}}", "refId": "A"}, {"expr": "rate(node_disk_written_bytes_total[5m])", "legendFormat": "write {{instance}} {{device}}", "refId": "B"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/dashboards/services-network.json

@@ -0,0 +1,26 @@
+{
+  "title": "Services and Network",
+  "uid": "services-network",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "table",
+      "title": "Probe State",
+      "gridPos": {"h": 10, "w": 12, "x": 0, "y": 0},
+      "targets": [{"expr": "probe_success", "format": "table", "instant": true, "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Probe Duration",
+      "gridPos": {"h": 10, "w": 12, "x": 12, "y": 0},
+      "targets": [{"expr": "probe_duration_seconds", "legendFormat": "{{job}} {{instance}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "TLS Certificate Remaining",
+      "gridPos": {"h": 10, "w": 24, "x": 0, "y": 10},
+      "targets": [{"expr": "(probe_ssl_earliest_cert_expiry - time()) / 86400", "legendFormat": "{{instance}}", "refId": "A"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/grafana-dashboard-provider.yml

@@ -0,0 +1,11 @@
+apiVersion: 1
+
+providers:
+  - name: HomeLab
+    orgId: 1
+    folder: HomeLab
+    type: file
+    disableDeletion: true
+    editable: false
+    options:
+      path: /etc/grafana/dashboards

ansible/roles/monitoring_server/files/grafana-datasource.yml

@@ -0,0 +1,10 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    uid: prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: false

ansible/roles/monitoring_server/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: restart monitoring stack
+  ansible.builtin.systemd:
+    name: homelab-monitoring
+    state: restarted

ansible/roles/monitoring_server/tasks/main.yml

@@ -0,0 +1,233 @@
+---
+- name: Validate monitoring secrets
+  ansible.builtin.assert:
+    that:
+      - monitoring_telegram_bot_token | length > 0
+      - monitoring_telegram_bot_token != 'replace-me'
+      - monitoring_telegram_chat_id | length > 0
+      - monitoring_telegram_chat_id != 'replace-me'
+      - monitoring_pve_api_user | length > 0
+      - monitoring_pve_api_token_id | length > 0
+      - monitoring_pve_api_token_secret | length > 0
+      - monitoring_pve_api_token_secret != 'replace-me'
+      - monitoring_grafana_admin_password | length > 0
+      - monitoring_grafana_admin_password != 'change-me'
+    fail_msg: Load monitoring secrets from ignored ansible/.env or Ansible Vault before running this playbook.
+  no_log: true
+
+- name: Install monitoring runtime packages
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - docker.io
+      - docker-compose
+      - ufw
+    state: present
+    update_cache: true
+
+- name: Ensure Docker is enabled and running
+  ansible.builtin.systemd:
+    name: docker
+    enabled: true
+    state: started
+
+- name: Allow SSH and Grafana access from the LAN
+  community.general.ufw:
+    rule: allow
+    port: "{{ item }}"
+    proto: tcp
+    src: "{{ monitoring_lan_cidr }}"
+  loop:
+    - "22"
+    - "3000"
+
+- name: Allow Pushgateway only from the OpenVPN gateway
+  community.general.ufw:
+    rule: allow
+    port: "9091"
+    proto: tcp
+    src: "{{ monitoring_openvpn_gateway_ip }}"
+
+- name: Allow local Prometheus container to scrape Node Exporter
+  community.general.ufw:
+    rule: allow
+    port: "9100"
+    proto: tcp
+    src: "{{ monitoring_docker_cidr }}"
+
+- name: Enable monitoring LXC firewall
+  community.general.ufw:
+    state: enabled
+    policy: deny
+    direction: incoming
+
+- name: Create monitoring directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  loop:
+    - "{{ monitoring_root }}"
+    - "{{ monitoring_root }}/prometheus"
+    - "{{ monitoring_root }}/alertmanager"
+    - "{{ monitoring_root }}/grafana"
+    - "{{ monitoring_root }}/grafana/provisioning"
+    - "{{ monitoring_root }}/grafana/provisioning/dashboards"
+    - "{{ monitoring_root }}/grafana/provisioning/datasources"
+    - "{{ monitoring_root }}/grafana/dashboards"
+    - "{{ monitoring_root }}/pve-exporter"
+
+- name: Grant Prometheus access to its data directory
+  ansible.builtin.file:
+    path: "{{ monitoring_root }}/prometheus/data"
+    state: directory
+    owner: "65534"
+    group: "65534"
+    mode: "0755"
+
+- name: Install monitoring compose definition
+  ansible.builtin.template:
+    src: compose.yml.j2
+    dest: "{{ monitoring_root }}/compose.yml"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: restart monitoring stack
+
+- name: Install Prometheus configuration
+  ansible.builtin.template:
+    src: prometheus.yml.j2
+    dest: "{{ monitoring_root }}/prometheus/prometheus.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install Prometheus alert rules
+  ansible.builtin.template:
+    src: alerts.yml.j2
+    dest: "{{ monitoring_root }}/prometheus/alerts.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install central Blackbox configuration
+  ansible.builtin.template:
+    src: blackbox.yml.j2
+    dest: "{{ monitoring_root }}/blackbox.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install Alertmanager configuration
+  ansible.builtin.template:
+    src: alertmanager.yml.j2
+    dest: "{{ monitoring_root }}/alertmanager/alertmanager.yml"
+    owner: root
+    group: root
+    mode: "0640"
+  no_log: true
+  notify: restart monitoring stack
+
+- name: Install PVE exporter configuration
+  ansible.builtin.template:
+    src: pve.yml.j2
+    dest: "{{ monitoring_root }}/pve-exporter/pve.yml"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: restart monitoring stack
+
+- name: Provision Grafana datasource
+  ansible.builtin.copy:
+    src: grafana-datasource.yml
+    dest: "{{ monitoring_root }}/grafana/provisioning/datasources/prometheus.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Provision Grafana dashboard provider
+  ansible.builtin.copy:
+    src: grafana-dashboard-provider.yml
+    dest: "{{ monitoring_root }}/grafana/provisioning/dashboards/homelab.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install Grafana dashboards
+  ansible.builtin.copy:
+    src: "dashboards/{{ item }}"
+    dest: "{{ monitoring_root }}/grafana/dashboards/{{ item }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - homelab-overview.json
+    - proxmox-storage.json
+    - services-network.json
+    - backups.json
+  notify: restart monitoring stack
+
+- name: Validate Docker Compose configuration
+  ansible.builtin.command: >-
+    docker-compose -f {{ monitoring_root }}/compose.yml config --quiet
+  changed_when: false
+  no_log: true
+
+- name: Validate Prometheus configuration
+  ansible.builtin.command: >-
+    docker run --rm --network none
+    -v {{ monitoring_root }}/prometheus:/etc/prometheus:ro
+    --entrypoint promtool prom/prometheus:v3.2.1
+    check config /etc/prometheus/prometheus.yml
+  changed_when: false
+
+- name: Validate Alertmanager configuration
+  ansible.builtin.command: >-
+    docker run --rm --network none
+    -v {{ monitoring_root }}/alertmanager:/etc/alertmanager:ro
+    --user 0:0
+    --entrypoint amtool prom/alertmanager:v0.28.0
+    check-config /etc/alertmanager/alertmanager.yml
+  changed_when: false
+  no_log: true
+
+- name: Install monitoring systemd unit
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-monitoring.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=HomeLab monitoring stack
+      Requires=docker.service
+      After=docker.service
+
+      [Service]
+      Type=oneshot
+      RemainAfterExit=yes
+      WorkingDirectory={{ monitoring_root }}
+      ExecStart=/usr/bin/docker-compose -f {{ monitoring_root }}/compose.yml up -d --remove-orphans
+      ExecStop=/usr/bin/docker-compose -f {{ monitoring_root }}/compose.yml down
+
+      [Install]
+      WantedBy=multi-user.target
+  notify:
+    - reload systemd
+    - restart monitoring stack
+
+- name: Enable monitoring stack
+  ansible.builtin.systemd:
+    name: homelab-monitoring
+    daemon_reload: true
+    enabled: true
+    state: started

ansible/roles/monitoring_server/templates/alertmanager.yml.j2

@@ -0,0 +1,40 @@
+global:
+  resolve_timeout: 5m
+
+route:
+  receiver: telegram-warning
+  group_by: [alertname, instance]
+  group_wait: 30s
+  group_interval: 10m
+  repeat_interval: 6h
+  routes:
+    - matchers:
+        - severity="critical"
+      receiver: telegram-critical
+      repeat_interval: 2h
+
+inhibit_rules:
+  - source_matchers:
+      - alertname="ExporterDown"
+    target_matchers:
+      - alertname="LanServiceUnavailable"
+    equal: [instance]
+
+receivers:
+  - name: telegram-warning
+    telegram_configs:
+      - bot_token: "{{ monitoring_telegram_bot_token }}"
+        chat_id: {{ monitoring_telegram_chat_id }}
+        send_resolved: true
+        message: |-
+          [{{ "{{" }} .Status | toUpper {{ "}}" }}] {{ "{{" }} .CommonLabels.alertname {{ "}}" }}
+          {{ "{{" }} .CommonAnnotations.summary {{ "}}" }}
+
+  - name: telegram-critical
+    telegram_configs:
+      - bot_token: "{{ monitoring_telegram_bot_token }}"
+        chat_id: {{ monitoring_telegram_chat_id }}
+        send_resolved: true
+        message: |-
+          [{{ "{{" }} .Status | toUpper {{ "}}" }}] CRITICAL: {{ "{{" }} .CommonLabels.alertname {{ "}}" }}
+          {{ "{{" }} .CommonAnnotations.summary {{ "}}" }}

ansible/roles/monitoring_server/templates/alerts.yml.j2

@@ -0,0 +1,134 @@
+groups:
+  - name: homelab-availability
+    rules:
+      - alert: ExporterDown
+        expr: up{job=~"node|smartctl|pve"} == 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Exporter unavailable: {{ '{{' }} $labels.job {{ '}}' }} {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: CriticalServiceUnavailable
+        expr: probe_success{job="blackbox-vps"} == 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "External or VPN check failed: {{ '{{' }} $labels.probe_name {{ '}}' }}"
+
+      - alert: RuVpsMonitoringStale
+        expr: time() - homelab_vps_probe_timestamp_seconds > 5 * 60
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "ru-vps has not pushed monitoring metrics through OpenVPN"
+
+      - alert: RuVpsMonitoringMissing
+        expr: absent(homelab_vps_probe_timestamp_seconds{job="blackbox-vps"})
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          summary: "ru-vps has no monitoring metrics in Pushgateway"
+
+      - alert: LanServiceUnavailable
+        expr: probe_success{job=~"blackbox-lan-http|blackbox-lan-tcp"} == 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "LAN service unavailable: {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: TLSCertificateExpiringSoon
+        expr: (probe_ssl_earliest_cert_expiry - time()) < 14 * 24 * 3600
+        for: 1h
+        labels:
+          severity: warning
+        annotations:
+          summary: "TLS certificate expires within 14 days: {{ '{{' }} $labels.instance {{ '}}' }}"
+
+  - name: homelab-capacity
+    rules:
+      - alert: FilesystemAlmostFull
+        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) < 0.15
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Filesystem is more than 85% full: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.mountpoint {{ '}}' }}"
+
+      - alert: FilesystemWillFillSoon
+        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 7 * 24 * 3600) < 0
+        for: 30m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Filesystem may fill within seven days: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.mountpoint {{ '}}' }}"
+
+      - alert: HostMemoryPressure
+        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.1
+        for: 20m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Memory pressure on {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: SMARTDeviceFailed
+        expr: homelab_smart_device_healthy == 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "SMART health check failed: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.device {{ '}}' }}"
+
+      - alert: NVMECriticalWarning
+        expr: homelab_smart_nvme_critical_warning > 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "NVMe critical warning: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.device {{ '}}' }}"
+
+  - name: homelab-backups
+    rules:
+      - alert: BackupAuditFailed
+        expr: homelab_backup_audit_success == 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Backup audit failed: {{ '{{' }} $labels.profile {{ '}}' }} on {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: BackupAuditStale
+        expr: time() - homelab_backup_audit_timestamp_seconds > 48 * 3600
+        for: 15m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Backup audit result is stale: {{ '{{' }} $labels.profile {{ '}}' }} on {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: BackupAuditMissing
+        expr: absent(homelab_backup_audit_timestamp_seconds{profile="pbs"}) or absent(homelab_backup_audit_timestamp_seconds{profile="gitea"}) or absent(homelab_backup_audit_timestamp_seconds{profile="vaultwarden"}) or absent(homelab_backup_audit_timestamp_seconds{profile="grimmory"})
+        for: 12h
+        labels:
+          severity: critical
+        annotations:
+          summary: "Expected backup audit metric has not been published"
+
+      - alert: ResticBackupTooOld
+        expr: homelab_backup_audit_snapshot_age_hours{profile=~"gitea|vaultwarden|grimmory"} > 36
+        for: 15m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Restic backup is older than 36 hours: {{ '{{' }} $labels.profile {{ '}}' }}"
+
+      - alert: PBSBackupTooOld
+        expr: homelab_backup_audit_snapshot_age_hours{profile="pbs"} > 48
+        for: 15m
+        labels:
+          severity: critical
+        annotations:
+          summary: "PBS backup is older than 48 hours: VMID {{ '{{' }} $labels.vmid {{ '}}' }}"

ansible/roles/monitoring_server/templates/blackbox.yml.j2

@@ -0,0 +1,10 @@
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [200, 301, 302]
+  tcp_connect:
+    prober: tcp
+    timeout: 10s

ansible/roles/monitoring_server/templates/compose.yml.j2

@@ -0,0 +1,63 @@
+services:
+  prometheus:
+    image: prom/prometheus:v3.2.1
+    command:
+      - --config.file=/etc/prometheus/prometheus.yml
+      - --storage.tsdb.path=/prometheus
+      - --storage.tsdb.retention.time={{ monitoring_prometheus_retention }}
+      - --storage.tsdb.retention.size={{ monitoring_prometheus_retention_size }}
+      - --web.enable-lifecycle
+    volumes:
+      - ./prometheus:/etc/prometheus:ro
+      - ./prometheus/data:/prometheus
+    restart: unless-stopped
+
+  alertmanager:
+    image: prom/alertmanager:v0.28.0
+    user: "0:0"
+    command:
+      - --config.file=/etc/alertmanager/alertmanager.yml
+      - --storage.path=/alertmanager
+    volumes:
+      - ./alertmanager:/etc/alertmanager:ro
+    restart: unless-stopped
+
+  grafana:
+    image: grafana/grafana:11.5.1
+    environment:
+      GF_SECURITY_ADMIN_USER: {{ monitoring_grafana_admin_user | to_json }}
+      GF_SECURITY_ADMIN_PASSWORD: {{ monitoring_grafana_admin_password | to_json }}
+      GF_USERS_ALLOW_SIGN_UP: "false"
+    ports:
+      - "192.168.1.30:3000:3000"
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/etc/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    restart: unless-stopped
+
+  blackbox-exporter:
+    image: prom/blackbox-exporter:v0.25.0
+    command:
+      - --config.file=/config/blackbox.yml
+    volumes:
+      - ./blackbox.yml:/config/blackbox.yml:ro
+    restart: unless-stopped
+
+  pve-exporter:
+    image: prompve/prometheus-pve-exporter:3.5.5
+    user: "0:0"
+    command:
+      - --config.file=/etc/pve-exporter/pve.yml
+    volumes:
+      - ./pve-exporter/pve.yml:/etc/pve-exporter/pve.yml:ro
+    restart: unless-stopped
+
+  pushgateway:
+    image: prom/pushgateway:v1.11.0
+    ports:
+      - "192.168.1.30:9091:9091"
+    restart: unless-stopped
+
+volumes:
+  grafana-data:

ansible/roles/monitoring_server/templates/prometheus.yml.j2

@@ -0,0 +1,94 @@
+global:
+  scrape_interval: 30s
+  evaluation_interval: 30s
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ["alertmanager:9093"]
+
+rule_files:
+  - /etc/prometheus/alerts.yml
+
+scrape_configs:
+  - job_name: prometheus
+    static_configs:
+      - targets: ["localhost:9090"]
+
+  - job_name: node
+    static_configs:
+      - targets:
+          - "192.168.1.5:9100"
+          - "192.168.1.10:9100"
+          - "192.168.1.20:9100"
+          - "192.168.1.23:9100"
+          - "192.168.1.24:9100"
+          - "192.168.1.25:9100"
+          - "192.168.1.26:9100"
+          - "192.168.1.27:9100"
+          - "192.168.1.28:9100"
+          - "192.168.1.30:9100"
+          - "192.168.1.34:9100"
+
+  - job_name: pve
+    metrics_path: /pve
+    params:
+      module: [default]
+    static_configs:
+      - targets:
+          - "192.168.1.5"
+          - "192.168.1.10"
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: pve-exporter:9221
+
+  - job_name: blackbox-lan-http
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
+    static_configs:
+      - targets:
+          - "http://192.168.1.24"
+          - "http://192.168.1.25:3000"
+          - "http://192.168.1.27:8080"
+          - "http://192.168.1.28"
+          - "http://192.168.1.34:6060/api/v1/healthcheck"
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: blackbox-exporter:9115
+      - target_label: probe_location
+        replacement: lan
+
+  - job_name: blackbox-lan-tcp
+    metrics_path: /probe
+    params:
+      module: [tcp_connect]
+    static_configs:
+      - targets:
+          - "192.168.1.5:8006"
+          - "192.168.1.10:8006"
+          - "192.168.1.20:8007"
+          - "192.168.1.25:2222"
+          - "192.168.1.27:9090"
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: blackbox-exporter:9115
+      - target_label: probe_location
+        replacement: lan
+
+  - job_name: pushgateway
+    honor_labels: true
+    static_configs:
+      - targets: ["pushgateway:9091"]

ansible/roles/monitoring_server/templates/pve.yml.j2

@@ -0,0 +1,5 @@
+default:
+  user: "{{ monitoring_pve_api_user }}"
+  token_name: "{{ monitoring_pve_api_token_id }}"
+  token_value: "{{ monitoring_pve_api_token_secret }}"
+  verify_ssl: false

ansible/roles/openvpn_gateway/defaults/main.yml

@@ -7,6 +7,9 @@ openvpn_config_dir: /etc/openvpn/homelab
 openvpn_config_path: /etc/openvpn/homelab/homelab.conf
 openvpn_static_key_path: /etc/openvpn/homelab/static.key
 openvpn_service_name: homelab-openvpn
+openvpn_public_interface: "{{ ansible_facts['default_ipv4']['interface'] }}"
+openvpn_lan_interface: eth0
+openvpn_forwarded_services: []
 openvpn_home_routes:
   - network: 192.168.1.5
     netmask: 255.255.255.255

ansible/roles/openvpn_gateway/tasks/main.yml

@@ -41,13 +41,13 @@
   no_log: true
   notify: restart openvpn
 
-- name: Enable IPv4 forwarding on gateway
+- name: Enable IPv4 forwarding for routed traffic
   ansible.posix.sysctl:
     name: net.ipv4.ip_forward
     value: "1"
     state: present
     reload: true
-  when: openvpn_role == 'gateway'
+  when: openvpn_role == 'gateway' or openvpn_forwarded_services | length > 0
 
 - name: Allow OpenVPN TCP in UFW on server
   ansible.builtin.command: "ufw allow {{ openvpn_listen_port }}/tcp"
@@ -65,7 +65,7 @@
     mode: "0600"
   notify: restart openvpn
 
-- name: Render OpenVPN gateway up script
+- name: Render OpenVPN up script
   ansible.builtin.template:
     src: up.sh.j2
     dest: "{{ openvpn_config_dir }}/up.sh"
@@ -73,9 +73,9 @@
     group: root
     mode: "0700"
   notify: restart openvpn
-  when: openvpn_role == 'gateway'
+  when: openvpn_role == 'gateway' or openvpn_forwarded_services | length > 0
 
-- name: Render OpenVPN gateway down script
+- name: Render OpenVPN down script
   ansible.builtin.template:
     src: down.sh.j2
     dest: "{{ openvpn_config_dir }}/down.sh"
@@ -83,7 +83,7 @@
     group: root
     mode: "0700"
   notify: restart openvpn
-  when: openvpn_role == 'gateway'
+  when: openvpn_role == 'gateway' or openvpn_forwarded_services | length > 0
 
 - name: Render OpenVPN systemd service
   ansible.builtin.template:

ansible/roles/openvpn_gateway/templates/down.sh.j2

@@ -1,2 +1,15 @@
 #!/bin/sh
-iptables -t nat -D POSTROUTING -s {{ openvpn_network_cidr }} -d {{ homelab_lan_cidr }} -o eth0 -j MASQUERADE 2>/dev/null || true
+{% if openvpn_role == 'gateway' %}
+iptables -t nat -D POSTROUTING -s {{ openvpn_network_cidr }} -d {{ homelab_lan_cidr }} -o {{ openvpn_lan_interface }} -j MASQUERADE 2>/dev/null || true
+{% for service in openvpn_forwarded_services %}
+iptables -D FORWARD -i {{ openvpn_interface }} -o {{ openvpn_lan_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT 2>/dev/null || true
+iptables -D FORWARD -i {{ openvpn_lan_interface }} -o {{ openvpn_interface }} -p {{ service.protocol }} -s {{ service.target_host }} --sport {{ service.target_port }} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT 2>/dev/null || true
+{% endfor %}
+{% else %}
+{% for service in openvpn_forwarded_services %}
+iptables -t nat -D PREROUTING -i {{ openvpn_public_interface }} -p {{ service.protocol }} --dport {{ service.public_port }} -j DNAT --to-destination {{ service.target_host }}:{{ service.target_port }} 2>/dev/null || true
+iptables -t nat -D POSTROUTING -o {{ openvpn_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -j SNAT --to-source {{ openvpn_local_ip }} 2>/dev/null || true
+iptables -D FORWARD -i {{ openvpn_public_interface }} -o {{ openvpn_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT 2>/dev/null || true
+iptables -D FORWARD -i {{ openvpn_interface }} -o {{ openvpn_public_interface }} -p {{ service.protocol }} -s {{ service.target_host }} --sport {{ service.target_port }} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT 2>/dev/null || true
+{% endfor %}
+{% endif %}

ansible/roles/openvpn_gateway/templates/homelab.conf.j2

@@ -13,6 +13,14 @@ port {{ openvpn_listen_port }}
 {% for route in openvpn_home_routes %}
 route {{ route.network }} {{ route.netmask }}
 {% endfor %}
+{% for service in openvpn_forwarded_services %}
+route {{ service.target_host }} 255.255.255.255
+{% endfor %}
+{% if openvpn_forwarded_services | length > 0 %}
+script-security 2
+up /etc/openvpn/homelab/up.sh
+down /etc/openvpn/homelab/down.sh
+{% endif %}
 {% endif %}
 {% if openvpn_role == 'gateway' %}
 proto {{ openvpn_proto_client }}

ansible/roles/openvpn_gateway/templates/up.sh.j2

@@ -1,3 +1,23 @@
 #!/bin/sh
-iptables -t nat -C POSTROUTING -s {{ openvpn_network_cidr }} -d {{ homelab_lan_cidr }} -o eth0 -j MASQUERADE 2>/dev/null || \
-  iptables -t nat -A POSTROUTING -s {{ openvpn_network_cidr }} -d {{ homelab_lan_cidr }} -o eth0 -j MASQUERADE
+sysctl -w net.ipv4.ip_forward=1 >/dev/null
+{% if openvpn_role == 'gateway' %}
+iptables -t nat -C POSTROUTING -s {{ openvpn_network_cidr }} -d {{ homelab_lan_cidr }} -o {{ openvpn_lan_interface }} -j MASQUERADE 2>/dev/null || \
+  iptables -t nat -A POSTROUTING -s {{ openvpn_network_cidr }} -d {{ homelab_lan_cidr }} -o {{ openvpn_lan_interface }} -j MASQUERADE
+{% for service in openvpn_forwarded_services %}
+iptables -C FORWARD -i {{ openvpn_interface }} -o {{ openvpn_lan_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT 2>/dev/null || \
+  iptables -I FORWARD 1 -i {{ openvpn_interface }} -o {{ openvpn_lan_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
+iptables -C FORWARD -i {{ openvpn_lan_interface }} -o {{ openvpn_interface }} -p {{ service.protocol }} -s {{ service.target_host }} --sport {{ service.target_port }} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT 2>/dev/null || \
+  iptables -I FORWARD 1 -i {{ openvpn_lan_interface }} -o {{ openvpn_interface }} -p {{ service.protocol }} -s {{ service.target_host }} --sport {{ service.target_port }} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+{% endfor %}
+{% else %}
+{% for service in openvpn_forwarded_services %}
+iptables -t nat -C PREROUTING -i {{ openvpn_public_interface }} -p {{ service.protocol }} --dport {{ service.public_port }} -j DNAT --to-destination {{ service.target_host }}:{{ service.target_port }} 2>/dev/null || \
+  iptables -t nat -A PREROUTING -i {{ openvpn_public_interface }} -p {{ service.protocol }} --dport {{ service.public_port }} -j DNAT --to-destination {{ service.target_host }}:{{ service.target_port }}
+iptables -t nat -C POSTROUTING -o {{ openvpn_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -j SNAT --to-source {{ openvpn_local_ip }} 2>/dev/null || \
+  iptables -t nat -A POSTROUTING -o {{ openvpn_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -j SNAT --to-source {{ openvpn_local_ip }}
+iptables -C FORWARD -i {{ openvpn_public_interface }} -o {{ openvpn_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT 2>/dev/null || \
+  iptables -I FORWARD 1 -i {{ openvpn_public_interface }} -o {{ openvpn_interface }} -p {{ service.protocol }} -d {{ service.target_host }} --dport {{ service.target_port }} -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
+iptables -C FORWARD -i {{ openvpn_interface }} -o {{ openvpn_public_interface }} -p {{ service.protocol }} -s {{ service.target_host }} --sport {{ service.target_port }} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT 2>/dev/null || \
+  iptables -I FORWARD 1 -i {{ openvpn_interface }} -o {{ openvpn_public_interface }} -p {{ service.protocol }} -s {{ service.target_host }} --sport {{ service.target_port }} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+{% endfor %}
+{% endif %}

ansible/roles/pve_lxc/defaults/main.yml

@@ -1,7 +1,7 @@
 ---
 pve_lxc_vmid: 132
 pve_lxc_node: mini-pc
-pve_lxc_hostname: wg-mini
+pve_lxc_hostname: ovpn-mini
 pve_lxc_ip: 192.168.1.23/24
 pve_lxc_gateway: 192.168.1.1
 pve_lxc_bridge: vmbr0
@@ -13,8 +13,10 @@ pve_lxc_swap: 128
 pve_lxc_onboot: true
 pve_lxc_startup: order=30
 pve_lxc_unprivileged: true
+pve_lxc_update: true
 pve_lxc_features:
   - nesting=1
 pve_lxc_nameserver: 1.1.1.1
 pve_lxc_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
 pve_lxc_cmode: shell
+pve_lxc_api_port: "{{ lookup('env', 'PROXMOX_PORT') | default('8006', true) | int }}"

ansible/roles/pve_lxc/tasks/main.yml

@@ -11,6 +11,7 @@
 - name: Create or update LXC container
   community.proxmox.proxmox:
     api_host: "{{ lookup('env', 'PROXMOX_HOST') }}"
+    api_port: "{{ pve_lxc_api_port }}"
     api_user: "{{ lookup('env', 'PROXMOX_USER') }}"
     api_token_id: "{{ lookup('env', 'PROXMOX_TOKEN_ID') }}"
     api_token_secret: "{{ lookup('env', 'PROXMOX_TOKEN_SECRET') }}"
@@ -32,15 +33,18 @@
     unprivileged: "{{ pve_lxc_unprivileged }}"
     onboot: "{{ pve_lxc_onboot }}"
     startup: "{{ pve_lxc_startup }}"
-    update: true
+    update: "{{ pve_lxc_update }}"
     state: present
+  no_log: true
 
 - name: Start LXC container
   community.proxmox.proxmox:
     api_host: "{{ lookup('env', 'PROXMOX_HOST') }}"
+    api_port: "{{ pve_lxc_api_port }}"
     api_user: "{{ lookup('env', 'PROXMOX_USER') }}"
     api_token_id: "{{ lookup('env', 'PROXMOX_TOKEN_ID') }}"
     api_token_secret: "{{ lookup('env', 'PROXMOX_TOKEN_SECRET') }}"
     validate_certs: "{{ lookup('env', 'PROXMOX_VALIDATE_CERTS') | default('false', true) | bool }}"
     vmid: "{{ pve_lxc_vmid }}"
     state: started
+  no_log: true

ansible/roles/uptime_kuma/defaults/main.yml

@@ -0,0 +1,9 @@
+---
+uptime_kuma_root: /opt/uptime-kuma
+uptime_kuma_image: louislam/uptime-kuma:1.23.16@sha256:431fee3be822b04861cf0e35daf4beef6b7cb37391c5f26c3ad6e12ce280fe18
+uptime_kuma_port: 3001
+uptime_kuma_lan_cidr: 192.168.1.0/24
+uptime_kuma_listen_address: "{{ expected_lan_ip }}"
+uptime_kuma_openvpn_gateway_ip: 192.168.1.23
+uptime_kuma_http_proxy: http://192.168.1.27:7890
+uptime_kuma_no_proxy: "localhost,127.0.0.1,::1,ada-dev.ru,.ada-dev.ru,192.168.1.5,192.168.1.10,192.168.1.20,192.168.1.23,192.168.1.24,192.168.1.25,192.168.1.26,192.168.1.27,192.168.1.28,192.168.1.29,192.168.1.30,192.168.1.31,192.168.1.32,192.168.1.34,192.168.1.100"

ansible/roles/uptime_kuma/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+- name: restart uptime kuma
+  ansible.builtin.systemd:
+    name: uptime-kuma
+    state: restarted

ansible/roles/uptime_kuma/tasks/main.yml

@@ -0,0 +1,159 @@
+---
+- name: Install Uptime Kuma runtime packages
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - docker.io
+      - docker-compose
+      - ufw
+    state: present
+    update_cache: true
+
+- name: Ensure Docker is enabled and running
+  ansible.builtin.systemd:
+    name: docker
+    enabled: true
+    state: started
+
+- name: Allow SSH and Uptime Kuma access from the LAN
+  community.general.ufw:
+    rule: allow
+    port: "{{ item }}"
+    proto: tcp
+    src: "{{ uptime_kuma_lan_cidr }}"
+  loop:
+    - "22"
+    - "{{ uptime_kuma_port }}"
+
+- name: Allow SSH from the OpenVPN gateway
+  community.general.ufw:
+    rule: allow
+    port: "22"
+    proto: tcp
+    src: "{{ uptime_kuma_openvpn_gateway_ip }}"
+
+- name: Enable Uptime Kuma LXC firewall
+  community.general.ufw:
+    state: enabled
+    policy: deny
+    direction: incoming
+
+- name: Create Uptime Kuma directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - "{{ uptime_kuma_root }}"
+    - "{{ uptime_kuma_root }}/data"
+
+- name: Check if the configured Uptime Kuma image is present
+  ansible.builtin.command: "docker image inspect {{ uptime_kuma_image }}"
+  register: uptime_kuma_image_inspect
+  changed_when: false
+  failed_when: false
+
+- name: Pull the configured Uptime Kuma image
+  ansible.builtin.command: "docker pull {{ uptime_kuma_image }}"
+  when: uptime_kuma_image_inspect.rc != 0
+  register: uptime_kuma_image_pull
+  changed_when: true
+
+- name: Install Uptime Kuma compose definition
+  ansible.builtin.copy:
+    dest: "{{ uptime_kuma_root }}/compose.yml"
+    owner: root
+    group: root
+    mode: "0640"
+    content: |
+      services:
+        uptime-kuma:
+          image: {{ uptime_kuma_image }}
+          container_name: uptime-kuma
+          restart: unless-stopped
+          network_mode: host
+          environment:
+            HTTP_PROXY: {{ uptime_kuma_http_proxy }}
+            HTTPS_PROXY: {{ uptime_kuma_http_proxy }}
+            http_proxy: {{ uptime_kuma_http_proxy }}
+            https_proxy: {{ uptime_kuma_http_proxy }}
+            NO_PROXY: {{ uptime_kuma_no_proxy }}
+            no_proxy: {{ uptime_kuma_no_proxy }}
+          volumes:
+            - {{ uptime_kuma_root }}/data:/app/data
+  notify: restart uptime kuma
+
+- name: Validate Uptime Kuma Compose configuration
+  ansible.builtin.command: >-
+    docker-compose -f {{ uptime_kuma_root }}/compose.yml config --quiet
+  changed_when: false
+
+- name: Install Uptime Kuma systemd unit
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/uptime-kuma.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Uptime Kuma
+      Requires=docker.service
+      After=docker.service
+
+      [Service]
+      Type=oneshot
+      RemainAfterExit=yes
+      WorkingDirectory={{ uptime_kuma_root }}
+      ExecStart=/usr/bin/docker-compose -f {{ uptime_kuma_root }}/compose.yml up -d --remove-orphans
+      ExecStop=/usr/bin/docker-compose -f {{ uptime_kuma_root }}/compose.yml down
+
+      [Install]
+      WantedBy=multi-user.target
+  register: uptime_kuma_unit
+  notify: restart uptime kuma
+
+- name: Reload systemd when the Uptime Kuma unit changes
+  ansible.builtin.systemd:
+    daemon_reload: true
+  when: uptime_kuma_unit.changed
+
+- name: Enable and start Uptime Kuma
+  ansible.builtin.systemd:
+    name: uptime-kuma
+    enabled: true
+    state: "{{ 'restarted' if uptime_kuma_unit.changed or uptime_kuma_image_pull is changed else 'started' }}"
+
+- name: Apply pending Uptime Kuma configuration changes
+  ansible.builtin.meta: flush_handlers
+
+- name: Check Uptime Kuma web interface
+  ansible.builtin.uri:
+    url: "http://{{ uptime_kuma_listen_address }}:{{ uptime_kuma_port }}/"
+    status_code: 200
+  register: uptime_kuma_health
+  retries: 24
+  delay: 5
+  until: uptime_kuma_health.status == 200
+
+- name: Freeze the legacy Prometheus monitoring stack after Uptime Kuma is healthy
+  ansible.builtin.systemd:
+    name: homelab-monitoring
+    enabled: false
+    state: stopped
+
+- name: Remove legacy monitoring firewall rules
+  community.general.ufw:
+    rule: allow
+    port: "{{ item.port }}"
+    proto: tcp
+    src: "{{ item.src }}"
+    delete: true
+  loop:
+    - port: "3000"
+      src: "{{ uptime_kuma_lan_cidr }}"
+    - port: "9091"
+      src: "{{ uptime_kuma_openvpn_gateway_ip }}"
+    - port: "9100"
+      src: "172.16.0.0/12"

ansible/tasks/offsite-restic-profile.yml

@@ -0,0 +1,219 @@
+---
+- name: Install offsite backup packages
+  ansible.builtin.apt:
+    name:
+      - restic
+      - rclone
+      - sqlite3
+      - ca-certificates
+    state: present
+    update_cache: true
+
+- name: Ensure rclone config directory exists
+  ansible.builtin.file:
+    path: /root/.config/rclone
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Install rclone config
+  ansible.builtin.copy:
+    src: "{{ offsite_rclone_config_local }}"
+    dest: /root/.config/rclone/rclone.conf
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+
+- name: Ensure restic config directory exists
+  ansible.builtin.file:
+    path: /etc/restic
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Install restic repository password
+  ansible.builtin.copy:
+    src: "{{ offsite_restic_password_local }}"
+    dest: "/etc/restic/{{ offsite_profile }}.password"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+
+- name: Ensure offsite backup directories exist
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+  loop:
+    - /etc/homelab-restic
+    - /var/lib/homelab-restic
+    - /var/log/homelab-restic
+
+- name: Install restic exclude file
+  ansible.builtin.copy:
+    dest: "/etc/homelab-restic/{{ offsite_profile }}.exclude"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      {% for exclude in offsite_excludes %}
+      {{ exclude }}
+      {% endfor %}
+
+- name: Install restic environment file
+  ansible.builtin.copy:
+    dest: "/etc/homelab-restic/{{ offsite_profile }}.env"
+    owner: root
+    group: root
+    mode: "0600"
+    content: |
+      RESTIC_REPOSITORY={{ offsite_repository }}
+      RESTIC_PASSWORD_FILE=/etc/restic/{{ offsite_profile }}.password
+      RCLONE_CONFIG=/root/.config/rclone/rclone.conf
+      HOMELAB_RESTIC_PROFILE={{ offsite_profile }}
+      HOMELAB_RESTIC_SOURCE={{ offsite_source_path }}
+      HOMELAB_RESTIC_SQLITE_DB={{ offsite_sqlite_db | default('') }}
+      HOMELAB_RESTIC_MARIADB_CONTAINER={{ offsite_mariadb_container | default('') }}
+      HOMELAB_RESTIC_MARIADB_DATABASE={{ offsite_mariadb_database | default('') }}
+      HOMELAB_RESTIC_MARIADB_USER={{ offsite_mariadb_user | default('') }}
+      HOMELAB_RESTIC_MARIADB_ENV={{ offsite_mariadb_env | default('') }}
+      HOMELAB_RESTIC_MARIADB_DUMP_NAME={{ offsite_mariadb_dump_name | default('') }}
+      HOMELAB_RESTIC_EXCLUDE_FILE=/etc/homelab-restic/{{ offsite_profile }}.exclude
+      HOMELAB_RESTIC_STAGING=/var/lib/homelab-restic/{{ offsite_profile }}
+      HOMELAB_RESTIC_TAGS={{ offsite_backup_tag }}
+      HOMELAB_RESTIC_KEEP_DAILY=14
+      HOMELAB_RESTIC_KEEP_WEEKLY=8
+      HOMELAB_RESTIC_KEEP_MONTHLY=6
+
+- name: Install restic offsite backup script
+  ansible.builtin.copy:
+    dest: /usr/local/sbin/homelab-restic-offsite-backup
+    owner: root
+    group: root
+    mode: "0755"
+    content: |
+      #!/bin/sh
+      set -eu
+
+      if [ "$#" -ne 1 ]; then
+        echo "usage: $0 <profile>" >&2
+        exit 64
+      fi
+
+      env_file="/etc/homelab-restic/$1.env"
+      if [ ! -f "$env_file" ]; then
+        echo "missing env file: $env_file" >&2
+        exit 66
+      fi
+
+      set -a
+      . "$env_file"
+      set +a
+
+      export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE RCLONE_CONFIG
+
+      mkdir -p "$HOMELAB_RESTIC_STAGING"
+      rm -rf "$HOMELAB_RESTIC_STAGING"/*
+
+      if [ -n "${HOMELAB_RESTIC_SQLITE_DB:-}" ] && [ -f "$HOMELAB_RESTIC_SQLITE_DB" ]; then
+        sqlite_backup="$HOMELAB_RESTIC_STAGING/$(basename "$HOMELAB_RESTIC_SQLITE_DB")"
+        sqlite3 -cmd 'PRAGMA busy_timeout=30000;' "$HOMELAB_RESTIC_SQLITE_DB" ".backup '$sqlite_backup'"
+        sqlite3 -cmd 'PRAGMA busy_timeout=30000;' "$sqlite_backup" "PRAGMA integrity_check;" | grep -qx ok
+      fi
+
+      if [ -n "${HOMELAB_RESTIC_MARIADB_CONTAINER:-}" ]; then
+        if [ ! -f "$HOMELAB_RESTIC_MARIADB_ENV" ]; then
+          echo "missing MariaDB env file: $HOMELAB_RESTIC_MARIADB_ENV" >&2
+          exit 66
+        fi
+        set -a
+        . "$HOMELAB_RESTIC_MARIADB_ENV"
+        set +a
+        dump_tmp="$HOMELAB_RESTIC_STAGING/${HOMELAB_RESTIC_MARIADB_DUMP_NAME}.tmp"
+        dump_final="$HOMELAB_RESTIC_STAGING/$HOMELAB_RESTIC_MARIADB_DUMP_NAME"
+        docker exec -e MYSQL_PWD="$DB_PASSWORD" "$HOMELAB_RESTIC_MARIADB_CONTAINER" \
+          mariadb-dump --user="$HOMELAB_RESTIC_MARIADB_USER" --single-transaction --routines --events \
+          --databases "$HOMELAB_RESTIC_MARIADB_DATABASE" > "$dump_tmp"
+        test -s "$dump_tmp"
+        mv "$dump_tmp" "$dump_final"
+        {
+          printf '%s\n' 'grimmory_image=grimmory/grimmory:v3.2.4@sha256:dfa7afdfcf25d649fd664497a62385dd00cd9678c37546e182c172e41c8e80cb'
+          printf '%s\n' 'mariadb_image=lscr.io/linuxserver/mariadb:11.4.8@sha256:91de7f701bc7fc3a424b81beafca7a7c6c4c5b7c8be6afd2ae148698695c0b0c'
+          printf 'created_at=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+        } > "$HOMELAB_RESTIC_STAGING/manifest.txt"
+      fi
+
+      restic snapshots >/dev/null 2>&1 || restic init
+
+      tag_args=""
+      old_ifs="$IFS"
+      IFS=,
+      for tag in $HOMELAB_RESTIC_TAGS; do
+        tag_args="$tag_args --tag $tag"
+      done
+      IFS="$old_ifs"
+
+      # shellcheck disable=SC2086
+      restic backup "$HOMELAB_RESTIC_SOURCE" "$HOMELAB_RESTIC_STAGING" \
+        --exclude-file "$HOMELAB_RESTIC_EXCLUDE_FILE" \
+        --one-file-system \
+        $tag_args
+
+      restic forget --prune \
+        --keep-daily "$HOMELAB_RESTIC_KEEP_DAILY" \
+        --keep-weekly "$HOMELAB_RESTIC_KEEP_WEEKLY" \
+        --keep-monthly "$HOMELAB_RESTIC_KEEP_MONTHLY"
+
+      if [ -n "${HOMELAB_RESTIC_MARIADB_CONTAINER:-}" ]; then
+        rm -rf "$HOMELAB_RESTIC_STAGING"/*
+      fi
+
+- name: Install restic offsite systemd service
+  ansible.builtin.copy:
+    dest: "/etc/systemd/system/homelab-restic-offsite-{{ offsite_profile }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=HomeLab restic offsite backup ({{ offsite_profile }})
+      Wants=network-online.target
+      After=network-online.target
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/bin/flock -w 1800 /var/lock/homelab-restic-{{ offsite_profile }}.lock /usr/local/sbin/homelab-restic-offsite-backup {{ offsite_profile }}
+
+- name: Install restic offsite systemd timer
+  ansible.builtin.copy:
+    dest: "/etc/systemd/system/homelab-restic-offsite-{{ offsite_profile }}.timer"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Run HomeLab restic offsite backup ({{ offsite_profile }})
+
+      [Timer]
+      OnCalendar={{ offsite_timer_oncalendar }}
+      Persistent=true
+      RandomizedDelaySec=15m
+
+      [Install]
+      WantedBy=timers.target
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: Enable restic offsite timer
+  ansible.builtin.systemd:
+    name: "homelab-restic-offsite-{{ offsite_profile }}.timer"
+    enabled: true
+    state: started