Capture current Ansible control plane state

Commit the accumulated infrastructure work that was living only in the
working tree: monitoring stack, emergency access/bot, gyro allocator,
grimmory, adguard, backup audit and the OpenCode agent definitions.

Also ignore Python bytecode, local archives and Nix/direnv artifacts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTocXkGUUazHdKKd3r9k71

.gitignore

@@ -6,3 +6,17 @@ passwd
 ansible/.venv/
 ansible/collections/
 ansible/generated/
+ansible/inventory/host_vars/gyro/vault.yml
+node_modules/
+
+# Python bytecode
+__pycache__/
+*.py[cod]
+
+# Local build artifacts
+*.tar.xz
+
+# Nix / direnv
+.direnv/
+result
+result-*

.opencode/agents/ansible-safety-reviewer.md

@@ -0,0 +1,19 @@
+---
+description: Reviews Ansible playbooks and roles for unsafe, non-idempotent, or secret-exposing changes before infrastructure execution.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash: deny
+  task: deny
+  todowrite: deny
+  skill: deny
+---
+You are a read-only Ansible safety reviewer for HomeLab infrastructure.
+
+Review the requested files or git diff for destructive operations, unsafe shell commands, missing idempotency, incorrect `changed_when` or `failed_when`, missing `no_log` around secrets, excessive privilege, broad host targeting, risky handlers, firewall/network exposure, and absent validation. Respect repository instructions and distinguish definite defects from risks.
+
+Return findings first, ordered by severity, with `path:line`, impact, and a minimal remediation. State explicitly when no findings are identified. Do not edit files or run commands.

.opencode/agents/ansible-verifier.md

@@ -0,0 +1,24 @@
+---
+description: Validates Ansible inventory, YAML, and playbook syntax without changing infrastructure. Use after Ansible configuration changes.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash:
+    "*": deny
+    "ansible-inventory *": allow
+    "ansible-playbook * --syntax-check": allow
+    "ansible-lint *": allow
+    "yamllint *": allow
+  task: deny
+  todowrite: deny
+  skill: deny
+---
+You validate Ansible changes using the smallest safe local checks.
+
+Never run a playbook against remote hosts, use `-K`, load `.env`, or run commands that can change infrastructure. Inspect the changed files and project guidance first, then select only relevant validation commands.
+
+Report commands run, pass/fail status, exact failures with paths, checks intentionally not run, and the smallest next action. Do not edit files.

.opencode/agents/backup-audit-reader.md

@@ -0,0 +1,24 @@
+---
+description: Analyzes PBS, restic, and systemd backup-audit output to assess backup freshness and identify failures without changing backup configuration.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash:
+    "*": deny
+    "systemctl status *": ask
+    "journalctl *": ask
+    "proxmox-backup-client *": ask
+    "restic *": ask
+  task: deny
+  todowrite: deny
+  skill: deny
+---
+You diagnose backup freshness and failures from provided output or explicitly approved read-only backup commands.
+
+Understand the active `backup_audit` role before interpreting results. Do not change schedules, repositories, retention, credentials, timers, services, or backup state. Never reveal secrets.
+
+Return backup status by target, the newest verified backup timestamp when available, root cause evidence, uncertainty, and one safe next check. Keep raw-log quotations minimal.

.opencode/agents/command-output-summarizer.md

@@ -0,0 +1,19 @@
+---
+description: Compresses large Ansible, systemd, Docker, and command outputs into structured failures, changed hosts, and safe next steps.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash: deny
+  task: deny
+  todowrite: deny
+  skill: deny
+---
+You summarize supplied command output without executing commands or editing files.
+
+Extract successful and failed hosts, changed tasks, warnings, error signatures, probable root cause, relevant output fragments, and the smallest safe next check. Do not repeat routine output. Redact or omit tokens, keys, passwords, URLs with credentials, and `.env` values.
+
+Use this format: Summary, Failures, Changed State, Evidence, Safe Next Step, Unknowns.

.opencode/agents/diff-reviewer.md

@@ -0,0 +1,23 @@
+---
+description: Reviews the current Git diff for HomeLab infrastructure regressions, missing documentation, and overly broad changes before execution or commit.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash:
+    "*": deny
+    "git status *": allow
+    "git diff *": allow
+    "git log *": allow
+  task: deny
+  todowrite: deny
+  skill: deny
+---
+You perform a read-only review of the working-tree or staged Git diff in this HomeLab repository.
+
+Follow AGENTS.md and compare changes to relevant inventory, roles, playbooks, and Obsidian documentation when needed. Identify behavioral regressions, overly broad targeting, secret exposure, missing documentation, and missing safe checks. Do not modify files, stage changes, or commit.
+
+Return findings first by severity with `path:line`, then residual risks and a short change summary. Say explicitly if no findings are found.

.opencode/agents/homelab-recon.md

@@ -0,0 +1,201 @@
+---
+description: >-
+  Use this agent when you need read-only reconnaissance of a HomeLab repository
+  before making infrastructure changes. Use it to navigate the codebase, locate
+  relevant Ansible inventory variables, host_vars/group_vars, playbooks, roles,
+  tasks, templates, handlers, dependencies, and Obsidian documentation; identify
+  existing patterns and constraints; determine the minimal set of files that
+  should be changed by a later implementation agent; or verify configuration
+  consistency without modifying files or infrastructure.
+
+
+  <example>
+
+  Context: The user wants to add a new backup setting for a HomeLab service, but
+  the relevant Ansible role and variables are unknown.
+
+  user: "Нужно добавить настройку retention для backups в HomeLab. Сначала
+  разберись, где это правильно делать."
+
+  assistant: "Сначала запущу homelab-recon agent для read-only анализа
+  репозитория и поиска существующих паттернов конфигурации."
+
+  <commentary>
+
+  Since the user needs to find the correct place for a new infrastructure
+  setting before changes, use the Task/Agent tool to launch the homelab-recon
+  agent. The agent should inspect inventory, vars, roles, templates, handlers,
+  dependencies, and docs without editing anything.
+
+  </commentary>
+
+  </example>
+
+
+  <example>
+
+  Context: The assistant has just completed a logical chunk of planning for an
+  Ansible change and needs to validate that it matches the current repository
+  structure.
+
+  user: "Проверь, не пропустили ли мы какие-то group_vars или шаблоны для
+  nginx."
+
+  assistant: "Запущу homelab-recon agent, чтобы read-only проверить связанные
+  vars, templates, роли, handlers и документацию по nginx."
+
+  <commentary>
+
+  Since the user asks for consistency/navigation across existing HomeLab
+  configuration, use the Task/Agent tool to launch the homelab-recon agent
+  rather than making assumptions from memory.
+
+  </commentary>
+
+  </example>
+
+
+  <example>
+
+  Context: The user asks where to implement a new monitoring exporter in the
+  HomeLab repo.
+
+  user: "Куда лучше добавить node exporter для нового хоста?"
+
+  assistant: "Использую homelab-recon agent для исследования inventory,
+  host_vars/group_vars, ролей мониторинга и Obsidian-документации перед
+  предложением места изменения."
+
+  <commentary>
+
+  Since the task is to locate the appropriate implementation point and
+  understand existing patterns, use the Task/Agent tool to launch the
+  homelab-recon agent in read-only mode.
+
+  </commentary>
+
+  </example>
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  bash: deny
+  edit: deny
+  webfetch: deny
+  task: deny
+  todowrite: deny
+  websearch: deny
+  lsp: deny
+  skill: deny
+  external_directory:
+    "*": deny
+    "/home/ada/Documents/Vaults/SecondBrain/02 Projects/HomeLab/**": allow
+---
+You are a senior HomeLab infrastructure reconnaissance specialist focused on read-only analysis of Ansible-based repositories and adjacent Obsidian documentation. Your mission is to investigate the current implementation before any change is made, identify the existing patterns and constraints, and provide a precise map of the minimal files that a later implementation step would need to modify.
+
+You operate strictly in read-only mode.
+
+Core responsibilities:
+1. Locate relevant Ansible configuration:
+   - inventories and inventory variables
+   - host_vars and group_vars
+   - playbooks and included playbooks
+   - roles, defaults, vars, tasks, templates, files, handlers, meta dependencies
+   - collections, requirements files, plugins, filters, and lookup usage when relevant
+   - service-specific configuration files and generated artifacts referenced by Ansible
+2. Locate and use supporting documentation:
+   - Obsidian notes, markdown documentation, runbooks, architecture notes, decision records, operational checklists, and service documentation
+   - README files and inline comments that explain conventions or constraints
+3. Reconstruct the current behavior:
+   - determine how the relevant service, host, group, or infrastructure component is configured today
+   - trace variable precedence where possible
+   - identify conditionals, tags, includes, dependencies, handlers, templates, and restart/reload behavior
+   - identify deployment ordering and cross-role relationships
+4. Identify repository patterns:
+   - naming conventions for hosts, groups, variables, roles, templates, tasks, and tags
+   - common ways new services/settings are added
+   - secrets handling conventions, vault usage, and boundaries around sensitive values
+   - idempotency and handler patterns
+   - documentation conventions
+5. Define the minimal change surface:
+   - list the smallest set of files that likely need modification for the requested future change
+   - list files that are relevant for context but should probably not be modified
+   - call out unknowns or decisions requiring user confirmation
+6. Verify consistency:
+   - check whether related vars/templates/tasks/docs agree with each other
+   - identify duplicate, stale, conflicting, or shadowed variables
+   - identify missing documentation or mismatches between docs and implementation
+
+Strict read-only constraints:
+- You must not edit, create, delete, rename, format, or write any file.
+- You must not run commands that change repository state, infrastructure state, secrets, generated files, caches, lockfiles, or external systems.
+- You must not run Ansible playbooks against infrastructure, apply Terraform/OpenTofu, restart services, install dependencies, or execute scripts that may mutate state.
+- You may inspect files, search text, list directories, and run clearly read-only commands such as grep/rg/find/ls/cat/sed for viewing, git status/log/diff/show, ansible-inventory --list when safe and local, and syntax-like inspection only if it is clearly non-mutating.
+- If a command may be mutating or ambiguous, do not run it. Explain the risk and suggest a safe alternative.
+- If repository instructions from CLAUDE.md or similar files define stricter rules, follow those rules.
+
+Investigation workflow:
+1. Clarify scope if needed:
+   - If the target service, host, group, environment, or desired setting is ambiguous, ask a concise clarification question.
+   - If enough context exists to begin, proceed and state your assumptions.
+2. Read project guidance first:
+   - Look for CLAUDE.md, README files, docs indexes, inventory layout notes, or repository conventions.
+   - Incorporate those rules into your analysis.
+3. Map the repository structure:
+   - Identify the inventory root, playbook entry points, role directories, documentation directories, and Obsidian vault locations.
+   - Note nonstandard layout choices.
+4. Search broadly, then narrow:
+   - Search for the service/component name, hostnames, group names, variable prefixes, role names, template names, ports, domains, package names, container names, systemd units, and documentation aliases.
+   - Follow references from playbooks to roles, from roles to tasks/templates/handlers, and from variables to template usage.
+5. Trace configuration flow:
+   - Determine where defaults are defined, where they are overridden, and where they are consumed.
+   - Pay special attention to group_vars/host_vars precedence, role defaults versus role vars, include_vars, set_fact, vars_files, extra vars references, and inventory group hierarchy.
+6. Analyze dependencies and side effects:
+   - Identify role dependencies, handlers triggered by template/task changes, service reload/restart behavior, firewall/DNS/reverse-proxy/monitoring/backup interactions, and documentation requirements.
+7. Produce a concise but actionable report.
+
+Output format:
+Provide your findings in a structured report with these sections:
+
+1. Scope and assumptions
+- State what you investigated and any assumptions made.
+
+2. Relevant files and why they matter
+- List paths grouped by category: inventory, host/group vars, playbooks, roles/tasks, templates/files, handlers, dependencies, docs.
+- For each path, include a short reason it is relevant.
+
+3. Current implementation summary
+- Explain how the current configuration works, including variable flow and execution flow.
+
+4. Existing patterns and constraints
+- Summarize naming, structure, variable, secrets, handler, template, documentation, and deployment conventions.
+
+5. Consistency findings
+- Note conflicts, stale docs, duplicate variables, unclear precedence, missing references, or mismatches.
+- If no issues were found, say so explicitly while noting the limits of the inspection.
+
+6. Minimal files for a future change
+- Provide a prioritized list of files likely requiring modification.
+- Separate "must change", "may need change", and "context only / probably do not change".
+
+7. Open questions and risks
+- List decisions that require user confirmation, unresolved ambiguity, and operational risks.
+
+8. Suggested next step
+- Recommend what the implementation agent or user should do next, without making changes yourself.
+
+Quality standards:
+- Be evidence-driven. Reference concrete file paths and, when useful, specific variable names, role names, task names, or documentation headings.
+- Do not overstate certainty. Distinguish confirmed facts from inferred patterns.
+- Prefer the minimal viable change surface over broad rewrites.
+- Preserve HomeLab safety: avoid recommendations that could accidentally affect unrelated hosts or services.
+- Treat secrets carefully: identify where secret values are referenced, but do not print secret contents. If secrets appear in plain text, mention the exposure without repeating the value.
+- If documentation conflicts with code, clearly identify both sources and which appears authoritative.
+- If you cannot find relevant files, report the searches performed and propose likely next search terms or clarification questions.
+
+Behavioral boundaries:
+- You are not an implementation agent. Do not patch files.
+- You are not an operations executor. Do not deploy or validate against live infrastructure.
+- You are a repository reconnaissance and consistency-analysis agent. Your deliverable is a map, diagnosis, and minimal-change recommendation for subsequent work.

.opencode/agents/inventory-doc-auditor.md

@@ -0,0 +1,22 @@
+---
+description: Audits HomeLab inventory against Ansible documentation and Obsidian notes to find stale hosts, groups, services, and topology drift.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash: deny
+  task: deny
+  todowrite: deny
+  skill: deny
+  external_directory:
+    "*": deny
+    "/home/ada/Documents/Vaults/SecondBrain/02 Projects/HomeLab/**": allow
+---
+You are a read-only consistency auditor for HomeLab.
+
+Compare the canonical Ansible inventory with active playbooks, repository README files, and the HomeLab Obsidian vault. Check host names, groups, IPs, service assignments, topology, and operational instructions. Treat active Ansible configuration as authoritative unless the repository says otherwise; do not use the archive as an active source of truth.
+
+Return confirmed mismatches with both sources and paths, suspected drift separately, the minimal files to update, and any ambiguity. Do not edit files or infrastructure.

.opencode/agents/log-diagnostician.md

@@ -0,0 +1,23 @@
+---
+description: Diagnoses bounded Ansible, systemd, Docker, and service logs to identify root causes and safe next checks without changing systems.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash:
+    "*": deny
+    "journalctl *": ask
+    "systemctl status *": ask
+    "docker logs *": ask
+  task: deny
+  todowrite: deny
+  skill: deny
+---
+You are a read-only operations diagnostician. Analyze provided logs or run only explicitly approved, bounded diagnostic commands.
+
+For logs, use time and line limits such as `--since`, `-n`, and `--tail`. Do not restart, install, reconfigure, or contact services in a way that changes state. Never expose secrets.
+
+Return Summary, most likely root cause, evidence with timestamps, alternative hypotheses, and one smallest safe next check. Avoid listing every log line.

.opencode/agents/network-diagnostician.md

@@ -0,0 +1,27 @@
+---
+description: Diagnoses HomeLab OpenVPN, ProxyJump, DNS, routing, and service reachability from configuration and approved read-only diagnostics.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash:
+    "*": deny
+    "ip route *": ask
+    "ip addr *": ask
+    "ss *": ask
+    "ping *": ask
+    "nc *": ask
+    "getent hosts *": ask
+    "journalctl *": ask
+  task: deny
+  todowrite: deny
+  skill: deny
+---
+You diagnose HomeLab network issues using active Ansible configuration, documentation, and explicitly approved read-only diagnostics.
+
+Trace the expected path through OpenVPN, the JumpHost, routes, DNS, firewall ports, and target services. Do not modify interfaces, firewall rules, VPN configuration, routes, DNS, or remote hosts. Do not expose credentials.
+
+Return expected path, observed break point, evidence, likely root cause, and the smallest safe next check.

.opencode/agents/obsidian-context-reader.md

@@ -0,0 +1,22 @@
+---
+description: Reads relevant HomeLab Obsidian notes before infrastructure work and extracts decisions, constraints, topology, and recent operational context.
+mode: all
+model: openai/gpt-5.5
+permission:
+  read: allow
+  glob: allow
+  grep: allow
+  edit: deny
+  bash: deny
+  task: deny
+  todowrite: deny
+  skill: deny
+  external_directory:
+    "*": deny
+    "/home/ada/Documents/Vaults/SecondBrain/02 Projects/HomeLab/**": allow
+---
+You are a read-only documentation researcher for HomeLab infrastructure.
+
+Find only the notes relevant to the requested service or change. Extract confirmed decisions, current topology, operational constraints, recent changes, and documentation that should be updated afterward. Cross-check any important claim with active Ansible configuration when it is available. Do not edit notes, repository files, or infrastructure.
+
+Return a concise report with sources, confirmed constraints, potential documentation drift, and open questions.

AGENTS.md

@@ -0,0 +1,295 @@
+# AGENTS.md
+
+## Project Summary
+
+**HomeLab infras** is a GitOps-like repository for managing home infrastructure through Ansible.
+
+- Management: `ansible/` (playbooks, roles, inventory)
+- Documentation: Obsidian vault `/home/ada/Documents/Vaults/SecondBrain/02 Projects/HomeLab/`
+- Active infrastructure: Proxmox VE cluster (`cloud-pc`, `mini-pc`), PBS, OpenVPN, ZeroTier
+- Archive: `archive/2026-07-proxmox-migration/` (historical NixOS/Docker configs, reference only)
+
+## Design Goal
+
+**Ansible-first infrastructure**: HomeLab configuration is managed through Ansible inventory, roles and playbooks. Direct server changes are allowed only for break-glass recovery or read-only diagnostics; afterwards the intended state must be captured in Ansible.
+
+## Obsidian Integration
+
+Project documentation lives in `/home/ada/Documents/Vaults/SecondBrain/02 Projects/HomeLab/`. Use it as wiki:
+
+- **Read** before making infra changes — context, decisions, constraints.
+- **Update** after making changes — document non-obvious decisions, new patterns, lessons learned.
+- Key files: `HomeLab.md`, `Notes/Текущее состояние HomeLab после миграции на Proxmox.md`, `Log.md`.
+
+When introducing infra changes, update the relevant Obsidian note to keep documentation in sync.
+
+## Operating Model
+
+- User: passwords, SSH access, base network reachability.
+- Agent: infrastructure changes via `ansible/` files.
+- Prefer adding/updating a role or playbook over ad-hoc commands.
+- For every new VM, LXC container, or managed host, provision the user's SSH public key by default unless explicitly told otherwise.
+- Run smallest safe Ansible check before broader changes.
+- Do not commit secrets (`.env`, tokens, keys, real passwords).
+
+## Quick Start
+
+```bash
+cd ansible
+
+# Setup (once)
+python3 -m venv .venv
+. .venv/bin/activate
+pip install -r requirements.txt
+ansible-galaxy collection install -r requirements.yml -p collections
+
+# Basic connectivity check
+ansible-playbook playbooks/check.yml
+
+# Privileged tasks
+ansible-playbook playbooks/<name>.yml -K
+
+# Proxmox API tasks (from .env)
+cp .env.example .env
+# Edit .env with real values
+. ./.env
+ansible-playbook playbooks/pve-*.yml
+```
+
+## Active Infrastructure
+
+### Nodes
+
+| Name | Role | LAN IP | Notes |
+|---|---|---:|---|
+| `ru-vps` | Public VPS, JumpHost, qdevice, OpenVPN server | `157.22.231.198:3422` | OpenVPN `10.78.0.1` |
+| `cloud-pc` | Proxmox VE node, PBS LXC, storage | `192.168.1.5` | Main node |
+| `mini-pc` | Proxmox VE node | `192.168.1.10` | Secondary node |
+| `pbs` | Proxmox Backup Server LXC | `192.168.1.20` | On cloud-pc |
+| `ovpn-mini` | OpenVPN gateway LXC | `192.168.1.23` | OpenVPN `10.78.0.2` |
+| `vaultwarden` | Vaultwarden LXC | `192.168.1.24` | |
+| `gitea` | Gitea LXC | `192.168.1.25` | |
+| `memoir-bot` | Telegram memoir bot LXC | `192.168.1.26` | |
+| `mihomo` | Local proxy/UI LXC | `192.168.1.27` | UI `:8080`, API `:9090` |
+
+### Inventory Groups
+
+- `homelab` — ru-vps
+- `pve_nodes` — cloud-pc, mini-pc
+- `lxc_infra` — pbs, ovpn-mini, vaultwarden, gitea, memoir-bot, mihomo
+- `vpn_openvpn` — ru-vps, ovpn-mini
+- `shell_hosts` — ru-vps, cloud-pc, mini-pc
+- `servers` — all managed hosts
+
+### Transport
+
+- **OpenVPN**: ru-vps (10.78.0.1) ↔ ovpn-mini (10.78.0.2), port 8443/tcp
+- **JumpHost**: SSH to cloud-pc/mini-pc via ProxyJump ru-vps
+
+## Directory Structure
+
+```
+ansible/
+├── ansible.cfg              # Ansible configuration
+├── inventory/
+│   └── hosts.yml            # Canonical host list and variables
+├── playbooks/               # Entry points
+│   ├── check.yml            # Connectivity check
+│   ├── pve-*.yml            # Proxmox API tasks (need .env)
+│   ├── openvpn-*.yml        # OpenVPN transport setup
+│   └── *.yml                # Other tasks
+├── roles/                   # Reusable units
+│   ├── backup_audit/        # PBS/restic backup audit
+│   ├── bash_config/         # Unified bash config
+│   ├── base/                # Base packages/config
+│   ├── docker/              # Docker setup
+│   ├── openvpn_gateway/     # OpenVPN client gateway
+│   ├── pve_lxc/             # Proxmox LXC creation
+│   └── ufw/                 # Firewall rules
+├── tasks/                   # Task snippets
+└── requirements.txt/yml    # Dependencies
+
+archive/2026-07-proxmox-migration/  # Historical configs (reference only)
+```
+
+## Common Playbooks
+
+```bash
+# Check connectivity and expected IPs
+ansible-playbook playbooks/check.yml
+
+# Bootstrap Proxmox API token from mini-pc (requires sudo)
+ansible-playbook playbooks/bootstrap-pve-api-token.yml -K
+
+# OpenVPN transport setup (requires privilege)
+ansible-playbook playbooks/openvpn-vps-mini.yml -K
+ansible-playbook playbooks/openvpn-check.yml
+
+# Unified bash config for shell hosts
+ansible-playbook playbooks/bash-config.yml
+
+# Install user's SSH public key on all managed hosts
+ansible-playbook playbooks/user-ssh-key.yml
+
+# Backup audit (PBS + restic offsite)
+ansible-playbook playbooks/backup-audit.yml
+```
+
+## Proxmox API Tasks
+
+Proxmox playbooks require environment variables:
+
+```bash
+# From .env file
+PROXMOX_HOST=<host>
+PROXMOX_USER=<user@realm>
+PROXMOX_TOKEN_ID=<token_id>
+PROXMOX_TOKEN_SECRET=<secret>
+PROXMOX_VALIDATE_CERTS=false
+```
+
+Usage:
+
+```bash
+. ./.env
+ansible-playbook playbooks/pve-ovpn-mini.yml
+ansible-playbook playbooks/pve-vaultwarden.yml
+ansible-playbook playbooks/pve-gitea.yml
+```
+
+Or bootstrap token directly from node:
+
+```bash
+ansible-playbook playbooks/bootstrap-pve-api-token.yml -K
+```
+
+## Secrets Policy
+
+- Never commit real secrets to git.
+- Use `.env` files (in `.gitignore`), Ansible vault, or runtime prompt.
+- Store `.env.example` templates in repo.
+- Keep: passwords, tokens, private keys, PBS secrets, TLS keys outside repo.
+
+## Workflow
+
+1. **Read Obsidian** — understand context, constraints, prior decisions.
+2. **Make change** — edit/add role/playbook in `ansible/`.
+3. **Test safe** — run smallest applicable check/playbook.
+4. **Update Obsidian** — document non-obvious decisions and changes.
+
+## Archive Policy
+
+- `archive/2026-07-proxmox-migration/` contains historical NixOS, Docker Compose, GitOps configs.
+- Use archived files only as reference when porting behavior to Ansible.
+- Do not edit archived files for active infrastructure.
+- Restore services via new Ansible roles/playbooks, not by moving old files back.
+
+## Token Economy
+
+Принципы минимизации токенов при управлении инфраструктурой.
+
+### Delegate Heavy Reading
+
+Когда нужно анализировать большие логи или выводы — делегируй субагенту:
+
+```
+agent("Read this log and summarize key errors")
+```
+
+**Субагент использует для:**
+- Анализа логов (journalctl, docker logs, PBS logs)
+- Summarization больших файлов
+- Поиска паттернов в выводах
+- Диагностики статусов
+
+**Основной агент для:**
+- Принятия решений на основе конспекта
+- Сложных рассуждений над findings
+- Изменения кода и архитектуры
+
+### Prefer Scripts Over LLM
+
+Для повторяющихся задач — скрипты, а не объяснения:
+
+```bash
+# Вместо: "проверь статус всех LXC"
+ansible/scripts/check-lxc-status.sh
+
+# Вместо: "верифицируй PBS backup jobs"
+ansible/scripts/check-pbs-backups.sh
+```
+
+Паттерн: написать один раз → переиспользовать. LLM только пишет или вызывает.
+
+### Focused Context — читай только нужное
+
+Вместо чтения всего файла — только релевантные части:
+
+```python
+# Вместо всего inventory.yml
+Read(file_path, offset=1, limit=50)  # только hosts vars
+
+# Или grep для конкретного хоста
+grep("mini-pc", "inventory/hosts.yml")
+```
+
+### Tool Limits — ограничивай вывод команд
+
+```bash
+# Вместо: journalctl -u openvpn (тысячи строк)
+journalctl -u openvpn --since "1 hour ago" -n 100
+
+# Вместо: docker logs (весь лог)
+docker logs --tail 50 gitea
+```
+
+Всегда ограничивай вывод: `--tail`, `--since`, `-n`, `head`.
+
+### Batch Operations — группируй задачи
+
+Вместо нескольких запусков — один с несколькими role:
+
+```python
+# Плохо
+ansible-playbook playbooks/bash-config.yml
+ansible-playbook playbooks/docker.yml
+ansible-playbook playbooks/ufw.yml
+
+# Хорошо — один плейбук
+ansible-playbook playbooks/base-setup.yml  # включает bash, docker, ufw
+```
+
+### State Caching — не перечитывай статичное
+
+Структура инфры меняется редко. Не перечитывай `hosts.yml`, если структура не изменилась. Кэшируй в memory стабильные данные: network topology, storage layout.
+
+### Structured Output — избегай повторного парсинга
+
+Когда субагент анализирует логи — проси сразу JSON с findings:
+
+```json
+{
+  "summary": "OpenVPN handshake fails due to certificate expired",
+  "findings": ["cert expired 2026-07-01", "client retries every 5s"],
+  "relevant_lines": ["Jul 08 10:23:01 TLS auth error"]
+}
+```
+
+Работа со структурированным результатом вместо повторного чтения логов.
+
+### Declarative Over Imperative
+
+Описывай желаемое состояние, а не шаги:
+
+```yaml
+# Плохо: каждый раз перечислять шаги
+"Создай LXC, установи Docker, добавь пользователя..."
+
+# Хорошо: декларативно
+lxc_container:
+  name: "vaultwarden"
+  features: ["docker", "autostart"]
+  user: "ansible"
+```
+
+Ansible/Terraform сами разберут шаги.

CLAUDE.md

@@ -0,0 +1,7 @@
+# CLAUDE.md
+
+Этот файл содержит адаптер инструкций для Claude Code (claude.ai/code) при работе с этим репозиторием.
+
+Канонические проектные инструкции находятся в [AGENTS.md](./AGENTS.md). Не дублируй их здесь, чтобы `CLAUDE.md` и `AGENTS.md` не расходились.
+
+@AGENTS.md

README.md

@@ -24,3 +24,18 @@ It contains old NixOS configs, Docker Compose service definitions, Gitea workflo
 cd ansible
 ansible-playbook playbooks/check.yml
 ```
+
+## Grimmory MCP
+
+`tools/grimmory-mcp/` contains the read-only Grimmory API integration for
+OpenCode and the explicitly invoked Obsidian synchronization tools.
+
+```bash
+npm install --prefix tools/grimmory-mcp
+npm run configure --prefix tools/grimmory-mcp
+npm test --prefix tools/grimmory-mcp
+```
+
+OpenCode registers the server globally. After restarting OpenCode, use
+`/grimmory-sync` to update individual book notes under `90 Library/Books` and
+covers under `99 System/Export/Grimmory/Covers`.

ansible/.env.example

@@ -3,6 +3,23 @@ export PROXMOX_USER='ansible@pve'
 export PROXMOX_TOKEN_ID='homelab'
 export PROXMOX_TOKEN_SECRET='replace-me'
 export PROXMOX_VALIDATE_CERTS=false
+# Optional local SSH-forward port when the control node reaches PVE through ru-vps.
+# export PROXMOX_PORT=18006
 
 # Override if the downloaded template name differs.
 export PVE_LXC_OSTEMPLATE='local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst'
+
+# Monitoring secrets. Keep actual values in ignored ansible/.env or Ansible Vault.
+export MONITORING_TELEGRAM_BOT_TOKEN='replace-me'
+export MONITORING_TELEGRAM_CHAT_ID='replace-me'
+export MONITORING_GRAFANA_ADMIN_PASSWORD='replace-me'
+export MONITORING_PVE_API_USER='monitoring@pve'
+export MONITORING_PVE_API_TOKEN_ID='prometheus'
+export MONITORING_PVE_API_TOKEN_SECRET='replace-me'
+
+# Emergency reverse SSH Telegram bot. Keep actual values in ignored ansible/.env
+# or Ansible Vault. Host keys are pinned public keys in the form "ssh-ed25519 AAAA...".
+export EMERGENCY_BOT_TOKEN='replace-me'
+export EMERGENCY_ALLOWED_USER_IDS='replace-me'
+export EMERGENCY_VPS_HOST_KEY='ssh-ed25519 replace-me'
+export EMERGENCY_MINI_PC_HOST_KEY='ssh-ed25519 replace-me'

ansible/README.md

@@ -17,10 +17,13 @@ Ansible is the control plane for HomeLab infrastructure changes.
 
 ## Current Groups
 
-- `ru-vps` — public VPS, JumpHost, qdevice, ZeroTier member.
+- `ru-vps` — public VPS, JumpHost, qdevice, OpenVPN server.
 - `pve_nodes` — Proxmox hosts: `cloud-pc`, `mini-pc`.
-- `lxc_infra` — infrastructure LXC containers: `pbs`, `zt-cloud`, `zt-mini`.
+- `lxc_infra` — infrastructure LXC containers, including the outbound-only `gyro` investment allocator host.
-- `vpn_openvpn` — OpenVPN transport hosts: `ru-vps`, `wg-mini`.
+- `monitoring_server` — monitoring LXC; Prometheus stack is frozen and Uptime Kuma is active.
+- `monitoring_exporters` — hosts exposing Node Exporter metrics.
+- `monitoring_smart_exporters` — Proxmox nodes exposing SMART metrics.
+- `vpn_openvpn` — OpenVPN transport hosts: `ru-vps`, `ovpn-mini`.
 - `shell_hosts` — hosts with unified bash config: `ru-vps`, `cloud-pc`, `mini-pc`.
 - `servers` — all managed hosts.
 
@@ -41,12 +44,44 @@ Run from `ansible/`:
 ansible-playbook playbooks/check.yml
 ```
 
+## Controlled Updates
+
+Service updates are manual and use pinned `tag@sha256:digest` image references only; floating tags and auto-update agents are not used.
+
+Run the dedicated playbook for the target service from `ansible/`:
+
+```bash
+.venv/bin/ansible-playbook playbooks/vaultwarden-update.yml
+.venv/bin/ansible-playbook playbooks/gitea-update.yml
+.venv/bin/ansible-playbook playbooks/adguard-update.yml
+.venv/bin/ansible-playbook playbooks/mihomo-update.yml
+.venv/bin/ansible-playbook playbooks/grimmory-update.yml
+```
+
+Update flow is always: fresh backup/audit first, then the update playbook, then health verification.
+
+- Gitea, Vaultwarden and Grimmory use app-aware offsite restic backups/audits before update.
+- AdGuard and Mihomo use fresh PBS LXC backups before update.
+- `grimmory-update.yml` currently validates the existing pinned release; any future actual app image upgrade must be preceded by release-note and migration review.
+
+## Mihomo Hardening
+
+Use the dedicated hardening playbook only when explicitly approved:
+
+```bash
+.venv/bin/ansible-playbook playbooks/ru-vps-mihomo-harden.yml -e ru_vps_mihomo_harden_confirm=true
+```
+
+It rotates the live Mihomo SOCKS credentials on `ru-vps`, locks the proxy to loopback, and removes the public UFW exposure for ports `7890` and `7891`.
+
+The rotated credentials are not recoverable for clients unless you roll back the saved config backup.
+
 For Proxmox API playbooks, create ignored `.env` from `.env.example` and load it:
 
 ```bash
 cp .env.example .env
 . ./.env
-.venv/bin/ansible-playbook playbooks/pve-wg-mini.yml
+.venv/bin/ansible-playbook playbooks/pve-ovpn-mini.yml
 ```
 
 Or bootstrap the token from `mini-pc` with sudo:
@@ -55,6 +90,12 @@ Or bootstrap the token from `mini-pc` with sudo:
 .venv/bin/ansible-playbook playbooks/bootstrap-pve-api-token.yml -K
 ```
 
+Create the separate read-only PVE token used by the monitoring exporter:
+
+```bash
+.venv/bin/ansible-playbook playbooks/bootstrap-monitoring-pve-token.yml
+```
+
 OpenVPN transport:
 
 ```bash
@@ -62,6 +103,67 @@ OpenVPN transport:
 .venv/bin/ansible-playbook playbooks/openvpn-check.yml
 ```
 
+Monitoring is provisioned in two steps after loading the monitoring secrets from ignored `.env` or Ansible Vault:
+
+```bash
+. ./.env
+.venv/bin/ansible-playbook playbooks/pve-monitoring.yml
+.venv/bin/ansible-playbook playbooks/monitoring.yml
+```
+
+`pve-monitoring.yml` creates CT `146` (`monitoring`, `192.168.1.30`) on `cloud-pc`. `monitoring.yml` configures exporters, the `ru-vps` probe vantage point, and the central Prometheus stack. It is frozen while Uptime Kuma is in use; do not run it unless restoring Prometheus monitoring.
+
+## Gyro Investment Allocator
+
+`pve-gyro.yml` creates unprivileged CT `150` (`gyro`, `192.168.1.35`) on `mini-pc`. `gyro.yml` installs Python 3.13+, pinned `uv`, the `gyro` service user, a container-local GitHub deploy key, restrictive firewall rules, and a weekday systemd timer.
+
+UFW is the currently enforced isolation layer: inbound is denied except SSH from LAN/OpenVPN, and east-west outbound is denied except the Mihomo HTTP proxy. The equivalent CT `150` Proxmox firewall is staged, but the cluster-wide PVE firewall remains disabled; do not enable it without auditing every node and guest with `firewall=1`.
+
+The role keeps deployment and the timer disabled by default. The active host vars deploy `git@github.com:ada-dmitry/t_tech-gyro.git` with GitHub's verified ED25519 host key; the timer still requires the ignored Vault file:
+
+```bash
+cp inventory/host_vars/gyro/vault.yml.example inventory/host_vars/gyro/vault.yml
+ansible-vault encrypt inventory/host_vars/gyro/vault.yml
+```
+
+After encrypting the secrets, set `gyro_timer_enabled: true` in `main.yml` and apply with `--ask-vault-pass`. `DRY_RUN_OVERRIDE` remains `true` until real trading is explicitly approved.
+
+```bash
+. ./.env
+.venv/bin/ansible-playbook playbooks/pve-gyro.yml
+.venv/bin/ansible-playbook playbooks/gyro.yml --ask-vault-pass
+```
+
+The timer runs at 11:00 Europe/Moscow from Monday through Friday and uses `OnFailure=` for a best-effort Telegram alert. CT `150` is included in the daily mini-pc PBS job and backup freshness audit.
+
+Uptime Kuma uses the existing monitoring LXC and stops/disables `homelab-monitoring` without deleting its configuration or data. Its UI is available only from the LAN at `http://192.168.1.30:3001`; create monitors and notification settings in the UI.
+
+```bash
+.venv/bin/ansible-playbook playbooks/uptime-kuma.yml
+```
+
+## Emergency Reverse SSH
+
+`pve-emergency-bot.yml` creates CT `148` (`emergency-bot`, `192.168.1.32`) on `mini-pc`. `emergency-access.yml` configures the bot, a TTL-limited reverse tunnel from `mini-pc` to `ru-vps`, and the restricted SSH identities used between them.
+
+Before applying, set `EMERGENCY_BOT_TOKEN`, `EMERGENCY_ALLOWED_USER_IDS`, `EMERGENCY_VPS_HOST_KEY`, and `EMERGENCY_MINI_PC_HOST_KEY` in ignored `.env` or Ansible Vault. The host-key variables must be verified public host keys, not values obtained during deployment.
+
+```bash
+. ./.env
+.venv/bin/ansible-playbook playbooks/pve-emergency-bot.yml
+.venv/bin/ansible-playbook playbooks/emergency-access.yml
+```
+
+From an authorized private Telegram chat, use the `Enable SSH`, `Status`, and `Stop` buttons or `/emergency ssh`, `/emergency status`, and `/emergency stop`. `/emergency ssh` enables a 60-minute tunnel only; it does not expose a public port. Connect while it is active with:
+
+The bot uses the LAN Mihomo HTTP proxy at `192.168.1.27:7890` because direct Telegram TCP access is unavailable from HomeLab.
+
+```bash
+ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -J vps -p 22010 ansible@127.0.0.1
+```
+
+The target account is `ansible`; it has no password login. Use the existing private key `~/.ssh/id_ed25519_homelab_ansible`.
+
 Bootstrap the Ansible service account on shell hosts:
 
 ```bash

ansible/files/ru-vps-mihomo-harden.py

@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import os
+import secrets
+import stat
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+
+
+DEFAULT_PROXY_HOST = "127.0.0.1"
+DEFAULT_PROXY_PORT = 7891
+
+
+def _atomic_write_bytes(path: Path, data: bytes, mode: int) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, tmp_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=str(path.parent))
+    try:
+        with os.fdopen(fd, "wb") as handle:
+            handle.write(data)
+            handle.flush()
+            os.fsync(handle.fileno())
+        os.chmod(tmp_name, mode)
+        os.replace(tmp_name, path)
+    finally:
+        if os.path.exists(tmp_name):
+            os.unlink(tmp_name)
+
+
+def _atomic_write_json(path: Path, payload: dict[str, str]) -> None:
+    _atomic_write_bytes(path, json.dumps(payload, indent=2, sort_keys=True).encode("utf-8") + b"\n", 0o600)
+
+
+def _load_json(path: Path) -> dict[str, str]:
+    with path.open("r", encoding="utf-8") as handle:
+        payload = json.load(handle)
+    if not isinstance(payload, dict):
+        raise ValueError(f"{path} must contain a JSON object")
+    return payload
+
+
+def _generate_username() -> str:
+    return f"mihomo-{secrets.token_hex(8)}"
+
+
+def _generate_password() -> str:
+    return secrets.token_urlsafe(48)
+
+
+def ensure_credentials(state_path: Path) -> tuple[dict[str, str], bool]:
+    if state_path.exists():
+        payload = _load_json(state_path)
+        username = payload.get("username")
+        password = payload.get("password")
+        if not username or not password:
+            raise ValueError(f"{state_path} is missing username/password")
+        return {"username": username, "password": password}, False
+
+    state_path.parent.mkdir(parents=True, exist_ok=True)
+    os.chmod(state_path.parent, 0o700)
+    credentials = {"username": _generate_username(), "password": _generate_password()}
+    _atomic_write_json(state_path, credentials)
+    return credentials, True
+
+
+def load_config(config_path: Path) -> dict:
+    with config_path.open("r", encoding="utf-8") as handle:
+        payload = yaml.safe_load(handle) or {}
+    if not isinstance(payload, dict):
+        raise ValueError(f"{config_path} must contain a YAML mapping")
+    return payload
+
+
+def write_config(config_path: Path, payload: dict) -> None:
+    serialized = yaml.safe_dump(payload, sort_keys=False, allow_unicode=True)
+    mode = 0o640
+    if config_path.exists():
+        mode = stat.S_IMODE(config_path.stat().st_mode)
+    _atomic_write_bytes(config_path, serialized.encode("utf-8"), mode)
+
+
+def cmd_apply(args: argparse.Namespace) -> int:
+    credentials, state_created = ensure_credentials(args.state)
+    config = load_config(args.config)
+    updated = copy.deepcopy(config)
+    updated["authentication"] = [f"{credentials['username']}:{credentials['password']}"]
+    updated["allow-lan"] = False
+    updated["bind-address"] = "127.0.0.1"
+    changed = updated != config
+    if changed:
+        write_config(args.config, updated)
+    print(json.dumps({"changed": changed, "state_created": state_created}, sort_keys=True))
+    return 0
+
+
+def cmd_probe(args: argparse.Namespace) -> int:
+    credentials = ensure_credentials(args.state)[0]
+    curl_config = "\n".join(
+        [
+            'silent',
+            'show-error',
+            f'proxy = "socks5h://{DEFAULT_PROXY_HOST}:{DEFAULT_PROXY_PORT}"',
+            f'proxy-user = "{credentials["username"]}:{credentials["password"]}"',
+            'connect-timeout = 5',
+            'max-time = 20',
+            'url = "https://api.telegram.org"',
+            'output = "/dev/null"',
+            '',
+        ]
+    )
+    result = subprocess.run(
+        ["curl", "--config", "-"],
+        input=curl_config,
+        text=True,
+        capture_output=True,
+        timeout=25,
+        check=False,
+    )
+    if result.returncode != 0:
+        sys.stderr.write(result.stderr)
+        return result.returncode
+    print(json.dumps({"ok": True}, sort_keys=True))
+    return 0
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    for name, func in (("apply", cmd_apply), ("probe", cmd_probe)):
+        sub = subparsers.add_parser(name)
+        sub.add_argument("--config", required=True, type=Path)
+        sub.add_argument("--state", required=True, type=Path)
+        sub.set_defaults(func=func)
+
+    return parser.parse_args()
+
+
+def main() -> int:
+    args = parse_args()
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ansible/inventory/host_vars/gyro/main.yml

@@ -0,0 +1,9 @@
+---
+gyro_repo_url: git@github.com:ada-dmitry/t_tech-gyro.git
+gyro_repo_version: main
+gyro_git_known_hosts_name: github.com
+gyro_git_host_key: "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"
+gyro_deploy_enabled: true
+gyro_secrets_configured: false
+gyro_timer_enabled: true
+gyro_timer_on_calendar: "Mon..Fri *-*-* 11:00:00 Europe/Moscow"

ansible/inventory/host_vars/gyro/vault.yml.example

@@ -0,0 +1,10 @@
+---
+# Copy to vault.yml, replace the placeholders, then run:
+# ansible-vault encrypt inventory/host_vars/gyro/vault.yml
+gyro_tinvest_token: t.replace-me
+gyro_tinvest_account_id: replace-me
+gyro_telegram_bot_token: replace-me
+gyro_telegram_user_id: "replace-me"
+gyro_telegram_proxy: http://192.168.1.27:7890
+gyro_dry_run_override: "true"
+gyro_secrets_configured: true

ansible/inventory/hosts.yml

@@ -6,15 +6,21 @@ all:
     ansible_become: true
     homelab_lan_cidr: 192.168.1.0/24
     homelab_service_range: 192.168.1.5-192.168.1.40
-    homelab_zerotier_network_id: 743993800fa5a34c
     openvpn_network_cidr: 10.78.0.0/30
     openvpn_listen_port: 8443
+    openvpn_forwarded_services:
+      - name: satisfactory
+        protocol: udp
+        public_port: 7777
+        target_host: 192.168.1.100
+        target_port: 7777
 
   children:
     homelab:
       hosts:
         ru-vps:
           ansible_host: vps
+          monitoring_exporter_node_listen_address: 127.0.0.1:9100
           openvpn_local_ip: 10.78.0.1
           openvpn_peer_ip: 10.78.0.2
           openvpn_role: server
@@ -44,29 +50,39 @@ all:
       hosts:
         pbs:
           ansible_host: 192.168.1.20
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
           expected_lan_ip: 192.168.1.20
 
-        zt-cloud:
+        ovpn-mini:
-          ansible_host: 192.168.1.21
-          expected_lan_ip: 192.168.1.21
-          zerotier_ip: 10.122.62.206
-          zerotier_node_id: c606e6d181
-
-        zt-mini:
-          ansible_host: 192.168.1.22
-          expected_lan_ip: 192.168.1.22
-          zerotier_ip: 10.122.62.213
-          zerotier_node_id: 022ac284e1
-
-        wg-mini:
           ansible_host: 192.168.1.23
           ansible_user: root
+          ansible_become: false
           ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
           expected_lan_ip: 192.168.1.23
           openvpn_local_ip: 10.78.0.2
           openvpn_peer_ip: 10.78.0.1
           openvpn_role: gateway
 
+        adguard:
+          ansible_host: 192.168.1.28
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.28
+
+        docker-test:
+          ansible_host: 192.168.1.29
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.29
+
         vaultwarden:
           ansible_host: 192.168.1.24
           ansible_user: root
@@ -85,16 +101,106 @@ all:
             -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
           expected_lan_ip: 192.168.1.25
 
+        memoir-bot:
+          ansible_host: 192.168.1.26
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.26
+
+        mihomo:
+          ansible_host: 192.168.1.27
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.27
+
+        monitoring:
+          ansible_host: 192.168.1.30
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.30
+
+        hermes-ai:
+          ansible_host: 192.168.1.31
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.31
+          bash_config_proxy_http_url: http://192.168.1.27:7890
+          bash_config_proxy_socks_url: socks5h://192.168.1.27:7890
+
+        emergency-bot:
+          ansible_host: 192.168.1.32
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.32
+
+        grimmory:
+          ansible_host: 192.168.1.34
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.34
+
+        gyro:
+          ansible_host: 192.168.1.35
+          ansible_user: root
+          ansible_become: false
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519_homelab
+          ansible_ssh_common_args: >-
+            -o ProxyCommand="ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -p 3422 -W %h:%p ansible@157.22.231.198"
+          expected_lan_ip: 192.168.1.35
+
+    monitoring_exporters:
+      hosts:
+        ru-vps:
+        cloud-pc:
+        mini-pc:
+        pbs:
+        ovpn-mini:
+        vaultwarden:
+        gitea:
+        memoir-bot:
+        mihomo:
+        adguard:
+        monitoring:
+        grimmory:
+
+    monitoring_smart_exporters:
+      hosts:
+        cloud-pc:
+        mini-pc:
+
+    monitoring_server:
+      hosts:
+        monitoring:
+
     vpn_openvpn:
       hosts:
         ru-vps:
-        wg-mini:
+        ovpn-mini:
 
     shell_hosts:
       hosts:
         ru-vps:
         cloud-pc:
         mini-pc:
+        hermes-ai:
 
     servers:
       children:

ansible/playbooks/adguard-update.yml

@@ -0,0 +1,110 @@
+---
+- name: Create and verify current PBS audit before AdGuard update
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Verify AdGuard VMID ownership before backup
+      ansible.builtin.command: pct config 144
+      register: adguard_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to back up a foreign VMID 144
+      ansible.builtin.assert:
+        that:
+          - adguard_pct_config.rc == 0
+          - adguard_existing_hostname == 'adguard'
+        fail_msg: VMID 144 is not the AdGuard container.
+      vars:
+        adguard_existing_hostname: >-
+          {{ adguard_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Check for active Proxmox backup before AdGuard PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: adguard_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before AdGuard PBS backup
+      ansible.builtin.assert:
+        that:
+          - adguard_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on mini-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh AdGuard PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "144"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+    - name: Run current PBS audit service
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-adguard.yml
+
+- name: Verify AdGuard public services after update
+  hosts: adguard
+  gather_facts: false
+  tasks:
+    - name: Check AdGuard HTTP root
+      ansible.builtin.uri:
+        url: http://127.0.0.1/
+        status_code: [200, 302]
+        return_content: false
+      register: adguard_http_root_check
+      retries: 24
+      delay: 5
+      until: adguard_http_root_check.status in [200, 302]
+
+    - name: Check AdGuard DNS over UDP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_udp_check
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_udp_check.rc == 0 and '127.0.0.1' in adguard_dns_udp_check.stdout
+
+    - name: Check AdGuard DNS over TCP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +tcp
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_tcp_check
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_tcp_check.rc == 0 and '127.0.0.1' in adguard_dns_tcp_check.stdout

ansible/playbooks/backup-audit.yml

@@ -0,0 +1,45 @@
+- name: Configure PBS backup audit on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: pbs
+
+- name: Configure restic offsite audit on cloud-pc (Gitea)
+  hosts: cloud-pc
+  gather_facts: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: restic
+        backup_audit_restic_profiles:
+          - name: gitea
+            sqlite_name: gitea.db
+            max_age_hours: 36
+
+- name: Configure restic offsite audit on vaultwarden
+  hosts: vaultwarden
+  gather_facts: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: restic
+        backup_audit_restic_profiles:
+          - name: vaultwarden
+            sqlite_name: db.sqlite3
+            max_age_hours: 36
+
+- name: Configure restic offsite audit on Grimmory
+  hosts: grimmory
+  gather_facts: false
+  vars:
+    ansible_become: false
+  roles:
+    - role: backup_audit
+      vars:
+        backup_audit_type: restic
+        backup_audit_restic_profiles:
+          - name: grimmory
+            expected_name: grimmory.sql
+            max_age_hours: 36

ansible/playbooks/bootstrap-monitoring-pve-token.yml

@@ -0,0 +1,76 @@
+---
+- name: Create read-only Proxmox token for monitoring
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    monitoring_pve_user: monitoring@pve
+    monitoring_pve_token_id: prometheus
+    monitoring_pve_env_file: "{{ playbook_dir }}/../.env"
+    monitoring_pve_rotate_existing_token: false
+  tasks:
+    - name: Read existing Proxmox users
+      ansible.builtin.command: pveum user list --output-format json
+      register: monitoring_pve_users_raw
+      changed_when: false
+
+    - name: Create monitoring Proxmox user
+      ansible.builtin.command: >-
+        pveum user add {{ monitoring_pve_user }}
+        --comment 'Read-only Prometheus monitoring user'
+      when: monitoring_pve_user not in (monitoring_pve_users_raw.stdout | from_json | map(attribute='userid') | list)
+
+    - name: Grant PVEAuditor role to monitoring user
+      ansible.builtin.command: >-
+        pveum acl modify / -user {{ monitoring_pve_user }} -role PVEAuditor
+      changed_when: false
+
+    - name: Read monitoring user tokens
+      ansible.builtin.command: >-
+        pveum user token list {{ monitoring_pve_user }} --output-format json
+      register: monitoring_pve_tokens_raw
+      changed_when: false
+
+    - name: Refuse to overwrite an existing monitoring token
+      ansible.builtin.assert:
+        that:
+          - monitoring_pve_token_id not in (monitoring_pve_tokens_raw.stdout | from_json | map(attribute='tokenid') | list)
+        fail_msg: Existing monitoring token secret cannot be recovered safely. Rotate it explicitly before rerunning this playbook.
+      when: not monitoring_pve_rotate_existing_token | bool
+
+    - name: Rotate existing monitoring token explicitly
+      ansible.builtin.command: >-
+        pveum user token remove {{ monitoring_pve_user }} {{ monitoring_pve_token_id }}
+      when:
+        - monitoring_pve_rotate_existing_token | bool
+        - monitoring_pve_token_id in (monitoring_pve_tokens_raw.stdout | from_json | map(attribute='tokenid') | list)
+
+    - name: Create separated monitoring token
+      ansible.builtin.command: >-
+        pveum user token add {{ monitoring_pve_user }} {{ monitoring_pve_token_id }}
+        --privsep 1 --comment 'Prometheus PVE exporter' --output-format json
+      register: monitoring_pve_token_created
+      no_log: true
+
+    - name: Grant PVEAuditor role to separated monitoring token
+      ansible.builtin.command: >-
+        pveum acl modify / -token {{ monitoring_pve_user }}!{{ monitoring_pve_token_id }} -role PVEAuditor
+      changed_when: false
+
+    - name: Store monitoring token variables locally
+      ansible.builtin.lineinfile:
+        path: "{{ monitoring_pve_env_file }}"
+        regexp: "^export {{ item.name }}="
+        line: "export {{ item.name }}='{{ item.value }}'"
+        create: false
+      loop:
+        - name: MONITORING_PVE_API_USER
+          value: "{{ monitoring_pve_user }}"
+        - name: MONITORING_PVE_API_TOKEN_ID
+          value: "{{ monitoring_pve_token_id }}"
+        - name: MONITORING_PVE_API_TOKEN_SECRET
+          value: "{{ (monitoring_pve_token_created.stdout | from_json).value }}"
+      delegate_to: localhost
+      vars:
+        ansible_connection: local
+        ansible_become: false
+      no_log: true

ansible/playbooks/emergency-access.yml

@@ -0,0 +1,42 @@
+---
+- name: Bootstrap emergency-bot control identity
+  hosts: emergency-bot
+  gather_facts: false
+  tasks:
+    - name: Create emergency-bot control identity
+      ansible.builtin.include_role:
+        name: emergency_bot
+        tasks_from: bootstrap
+
+- name: Configure mini-pc reverse SSH client and restricted control path
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    emergency_vps_host_key: "{{ lookup('env', 'EMERGENCY_VPS_HOST_KEY') }}"
+    emergency_bot_control_public_key: "{{ hostvars['emergency-bot'].emergency_bot_control_public_key }}"
+  tasks:
+    - name: Configure mini-pc emergency access client
+      ansible.builtin.include_role:
+        name: emergency_access
+        tasks_from: client
+
+- name: Configure ru-vps reverse SSH endpoint
+  hosts: ru-vps
+  gather_facts: false
+  vars:
+    emergency_reverse_public_key: "{{ hostvars['mini-pc'].emergency_reverse_public_key }}"
+  tasks:
+    - name: Configure ru-vps emergency access endpoint
+      ansible.builtin.include_role:
+        name: emergency_access
+        tasks_from: endpoint
+
+- name: Configure and enable emergency Telegram bot
+  hosts: emergency-bot
+  gather_facts: false
+  vars:
+    emergency_bot_token: "{{ lookup('env', 'EMERGENCY_BOT_TOKEN') }}"
+    emergency_bot_allowed_user_ids: "{{ lookup('env', 'EMERGENCY_ALLOWED_USER_IDS') }}"
+    emergency_mini_pc_host_key: "{{ lookup('env', 'EMERGENCY_MINI_PC_HOST_KEY') }}"
+  roles:
+    - role: emergency_bot

ansible/playbooks/gitea-update.yml

@@ -0,0 +1,45 @@
+---
+- name: Create and verify Gitea backup before update
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Create a fresh Gitea offsite backup
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-restic-offsite-gitea.service
+      changed_when: true
+
+    - name: Run Gitea offsite backup audit
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-gitea.service
+      changed_when: true
+
+- import_playbook: pve-gitea.yml
+
+- name: Verify Gitea public endpoints after update
+  hosts: ru-vps
+  gather_facts: false
+  tasks:
+    - name: Check Gitea HTTPS endpoint
+      ansible.builtin.uri:
+        url: https://git.ada-dev.ru/
+        status_code: 200
+        return_content: false
+      register: gitea_https_check
+      retries: 24
+      delay: 5
+      until: gitea_https_check.status == 200
+
+    - name: Check Gitea SSH port from ru-vps
+      ansible.builtin.wait_for:
+        host: 192.168.1.25
+        port: 2222
+        state: started
+        timeout: 30

ansible/playbooks/grimmory-update.yml

@@ -0,0 +1,106 @@
+---
+- name: Create and verify Grimmory backup before update
+  hosts: grimmory
+  gather_facts: false
+  tasks:
+    - name: Create a fresh Grimmory offsite backup
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-restic-offsite-grimmory.service
+      changed_when: true
+
+    - name: Run Grimmory offsite backup audit
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-grimmory.service
+      changed_when: true
+
+- name: Create a fresh Grimmory PBS backup
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Read Grimmory LXC config
+      ansible.builtin.command:
+        argv:
+          - pct
+          - config
+          - "149"
+      register: grimmory_pct_config
+      changed_when: false
+
+    - name: Assert VMID 149 belongs to Grimmory
+      ansible.builtin.assert:
+        that:
+          - grimmory_pct_hostname_line != ""
+          - grimmory_pct_hostname == "grimmory"
+        fail_msg: >-
+          Refusing to run vzdump 149 because pct config hostname is not grimmory:
+          {{ grimmory_pct_hostname_line | default('missing hostname line') }}
+      vars:
+        grimmory_pct_hostname_line: >-
+          {{ (grimmory_pct_config.stdout_lines | select('match', '^hostname:\\s+') | list | first | default('')) }}
+        grimmory_pct_hostname: >-
+          {{ grimmory_pct_hostname_line | regex_replace('^hostname:\\s*', '') }}
+
+    - name: Check for active Proxmox backup before Grimmory PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: grimmory_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before Grimmory PBS backup
+      ansible.builtin.assert:
+        that:
+          - grimmory_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on cloud-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh Grimmory PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "149"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+- name: Run PBS backup audit on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Run current PBS backup audit on mini-pc
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-grimmory.yml
+
+- name: Verify Grimmory public endpoint after update
+  hosts: ru-vps
+  gather_facts: false
+  tasks:
+    - name: Check Grimmory public health endpoint
+      ansible.builtin.uri:
+        url: https://books.ada-dev.ru/api/v1/healthcheck
+        status_code: 200
+        return_content: false
+      register: grimmory_public_health
+      retries: 24
+      delay: 5
+      until: grimmory_public_health.status == 200

ansible/playbooks/gyro.yml

@@ -0,0 +1,6 @@
+---
+- name: Configure Gyro investment allocator host
+  hosts: gyro
+  gather_facts: true
+  roles:
+    - role: gyro

ansible/playbooks/mihomo-update.yml

@@ -0,0 +1,96 @@
+---
+- name: Verify Mihomo PBS audit before update
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read VMID 143 configuration
+      ansible.builtin.command: "pct config 143"
+      register: mihomo_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to run backup unless VMID 143 is Mihomo
+      ansible.builtin.assert:
+        that:
+          - mihomo_pct_config.rc == 0
+          - mihomo_update_hostname == 'mihomo'
+        fail_msg: VMID 143 must be the Mihomo container before backup.
+      vars:
+        mihomo_update_hostname: >-
+          {{ mihomo_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Check for active Proxmox backup before Mihomo PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: mihomo_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before Mihomo PBS backup
+      ansible.builtin.assert:
+        that:
+          - mihomo_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on mini-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh Mihomo PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "143"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+    - name: Run current PBS backup audit on mini-pc
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-mihomo.yml
+
+- name: Verify Mihomo after update
+  hosts: mihomo
+  gather_facts: false
+  tasks:
+    - name: Wait for Mihomo proxy TCP ports
+      ansible.builtin.wait_for:
+        host: 127.0.0.1
+        port: "{{ item }}"
+        state: started
+        timeout: 120
+      loop:
+        - 7890
+        - 7891
+        - 9090
+
+    - name: Check Mihomo UI endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:8080/
+        status_code: 200
+      register: mihomo_ui_health
+      retries: 24
+      delay: 5
+      until: mihomo_ui_health.status == 200
+
+    - name: Check Mihomo controller /version endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:9090/version
+        status_code: 200
+      register: mihomo_controller_health
+      retries: 24
+      delay: 5
+      until: mihomo_controller_health.status == 200

ansible/playbooks/monitoring.yml

@@ -0,0 +1,26 @@
+---
+- name: Configure Prometheus exporters
+  hosts: monitoring_exporters
+  gather_facts: false
+  roles:
+    - role: monitoring_exporter
+
+- name: Configure SMART exporters on Proxmox nodes
+  hosts: monitoring_smart_exporters
+  gather_facts: false
+  roles:
+    - role: monitoring_exporter
+      monitoring_exporter_install_node: false
+      monitoring_exporter_install_smart: true
+
+- name: Configure ru-vps external probes
+  hosts: ru-vps
+  gather_facts: false
+  roles:
+    - role: monitoring_blackbox
+
+- name: Configure central monitoring stack
+  hosts: monitoring_server
+  gather_facts: false
+  roles:
+    - role: monitoring_server

ansible/playbooks/offsite-restic-yadisk.yml

@@ -0,0 +1,74 @@
+- name: Configure Gitea offsite backup to Yandex Disk
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    offsite_profile: gitea
+    offsite_repository: rclone:yadisk:System/Backups/HomeLab/restic/gitea
+    offsite_source_path: /opt/data/gitea
+    offsite_sqlite_db: /opt/data/gitea/gitea/gitea.db
+    offsite_backup_tag: gitea,cloud-pc,yadisk
+    offsite_timer_oncalendar: "*-*-* 04:15:00"
+    offsite_rclone_config_local: ~/.config/rclone/rclone.conf
+    offsite_restic_password_local: "{{ playbook_dir }}/../generated/restic-offsite-password"
+    offsite_excludes:
+      - /opt/data/gitea/gitea/gitea.db
+      - /opt/data/gitea/gitea/gitea.db-shm
+      - /opt/data/gitea/gitea/gitea.db-wal
+      - /opt/data/gitea/gitea/log/**
+      - /opt/data/gitea/gitea/sessions/**
+      - /opt/data/gitea/gitea/queues/**
+      - /opt/data/gitea/gitea/tmp/**
+  tasks:
+    - name: Configure restic offsite profile
+      ansible.builtin.include_tasks: ../tasks/offsite-restic-profile.yml
+
+- name: Configure Vaultwarden offsite backup to Yandex Disk
+  hosts: vaultwarden
+  gather_facts: false
+  vars:
+    ansible_become: false
+    offsite_profile: vaultwarden
+    offsite_repository: rclone:yadisk:System/Backups/HomeLab/restic/vaultwarden
+    offsite_source_path: /opt/vaultwarden/data
+    offsite_sqlite_db: /opt/vaultwarden/data/db.sqlite3
+    offsite_backup_tag: vaultwarden,mini-pc,yadisk
+    offsite_timer_oncalendar: "*-*-* 04:45:00"
+    offsite_rclone_config_local: ~/.config/rclone/rclone.conf
+    offsite_restic_password_local: "{{ playbook_dir }}/../generated/restic-offsite-password"
+    offsite_excludes:
+      - /opt/vaultwarden/data/db.sqlite3
+      - /opt/vaultwarden/data/db.sqlite3-shm
+      - /opt/vaultwarden/data/db.sqlite3-wal
+      - /opt/vaultwarden/data/tmp/**
+  tasks:
+    - name: Configure restic offsite profile
+      ansible.builtin.include_tasks: ../tasks/offsite-restic-profile.yml
+
+- name: Configure Grimmory offsite backup to Yandex Disk
+  hosts: grimmory
+  gather_facts: false
+  vars:
+    ansible_become: false
+    offsite_profile: grimmory
+    offsite_repository: rclone:yadisk:System/Backups/HomeLab/restic/grimmory
+    offsite_source_path: /opt/grimmory
+    offsite_sqlite_db: ""
+    offsite_mariadb_container: grimmory-mariadb
+    offsite_mariadb_database: grimmory
+    offsite_mariadb_user: grimmory
+    offsite_mariadb_env: /opt/grimmory/.env
+    offsite_mariadb_dump_name: grimmory.sql
+    offsite_backup_tag: grimmory,cloud-pc,yadisk
+    offsite_timer_oncalendar: "*-*-* 05:15:00"
+    offsite_rclone_config_local: ~/.config/rclone/rclone.conf
+    offsite_restic_password_local: "{{ playbook_dir }}/../generated/restic-offsite-password"
+    offsite_excludes:
+      - /opt/grimmory/books/**
+      - /opt/grimmory/bookdrop/**
+      - /opt/grimmory/mariadb/**
+      - /opt/grimmory/backup-staging/**
+      - /opt/grimmory/**/heapdump*.hprof
+      - /opt/grimmory/**/cache/**
+  tasks:
+    - name: Configure restic offsite profile
+      ansible.builtin.include_tasks: ../tasks/offsite-restic-profile.yml

ansible/playbooks/openvpn-check.yml

@@ -10,10 +10,22 @@
       ansible.builtin.command: nc -vz -w 5 192.168.1.10 8006
       changed_when: false
 
+    - name: Check mini-pc SSH through OpenVPN gateway
+      ansible.builtin.command: nc -vz -w 5 192.168.1.10 22
+      changed_when: false
+
     - name: Check cloud-pc PVE port through OpenVPN gateway
       ansible.builtin.command: nc -vz -w 5 192.168.1.5 8006
       changed_when: false
 
+    - name: Check cloud-pc SSH through OpenVPN gateway
+      ansible.builtin.command: nc -vz -w 5 192.168.1.5 22
+      changed_when: false
+
     - name: Check PBS port through OpenVPN gateway
       ansible.builtin.command: nc -vz -w 5 192.168.1.20 8007
       changed_when: false
+
+    - name: Check PBS SSH through OpenVPN gateway
+      ansible.builtin.command: nc -vz -w 5 192.168.1.20 22
+      changed_when: false

ansible/playbooks/pve-adguard.yml

@@ -0,0 +1,247 @@
+---
+- name: Create AdGuard Home LXC on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    adguard_vmid: 144
+    adguard_hostname: adguard
+    adguard_ip: 192.168.1.28/24
+    adguard_gateway: 192.168.1.1
+    adguard_rootfs: local-lvm:8
+    adguard_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    adguard_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+    adguard_image: adguard/adguardhome:v0.107.78@sha256:2c127294fa5f96151d9d3a433fb9d66c17e4d18cf698c2b04372a80e26fdd26f
+  handlers:
+    - name: restart adguard lxc
+      ansible.builtin.shell: "pct stop {{ adguard_vmid }} || true; pct start {{ adguard_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if AdGuard LXC exists
+      ansible.builtin.command: "pct config {{ adguard_vmid }}"
+      register: adguard_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID {{ adguard_vmid }}
+      ansible.builtin.assert:
+        that:
+          - adguard_pct_config.rc != 0 or adguard_existing_hostname == adguard_hostname
+        fail_msg: VMID {{ adguard_vmid }} already exists and is not the AdGuard container.
+      vars:
+        adguard_existing_hostname: >-
+          {{ adguard_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Install AdGuard LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/adguard-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', adguard_pubkey_file) }}\n"
+      when: adguard_pct_config.rc != 0
+
+    - name: Create AdGuard LXC
+      ansible.builtin.command: >-
+        pct create {{ adguard_vmid }} {{ adguard_ostemplate }}
+        --hostname {{ adguard_hostname }}
+        --rootfs {{ adguard_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge=vmbr0,gw={{ adguard_gateway }},ip={{ adguard_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=40
+        --cmode shell
+        --ssh-public-keys /tmp/adguard-lxc.pub
+      when: adguard_pct_config.rc != 0
+
+    - name: Start AdGuard LXC
+      ansible.builtin.command: "pct start {{ adguard_vmid }}"
+      register: adguard_pct_start
+      changed_when: adguard_pct_start.rc == 0
+      failed_when: adguard_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in AdGuard LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ adguard_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart adguard lxc
+
+    - name: Bind mount FUSE device in AdGuard LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ adguard_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart adguard lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for AdGuard SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: adguard
+      vars:
+        ansible_become: false
+
+- name: Configure AdGuard Home inside LXC
+  hosts: adguard
+  gather_facts: true
+  vars:
+    ansible_become: false
+    adguard_image: adguard/adguardhome:v0.107.78@sha256:2c127294fa5f96151d9d3a433fb9d66c17e4d18cf698c2b04372a80e26fdd26f
+  tasks:
+    - name: Install Docker packages
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - ca-certificates
+          - curl
+          - dnsutils
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: adguard_docker_daemon_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if adguard_docker_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Ensure AdGuard data directories exist
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - /opt/adguard/work
+        - /opt/adguard/conf
+
+    - name: Check if the configured AdGuard image is present
+      ansible.builtin.command: "docker image inspect {{ adguard_image }}"
+      register: adguard_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured AdGuard image
+      ansible.builtin.command: "docker pull {{ adguard_image }}"
+      when: adguard_image_inspect.rc != 0
+      register: adguard_image_pull
+      changed_when: true
+
+    - name: Install AdGuard Home systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/adguard.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=AdGuard Home container
+          After=docker.service
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f adguard
+          ExecStart=/usr/bin/docker run --rm --name adguard --pull never \
+            -p 53:53/tcp -p 53:53/udp -p 80:80/tcp -p 3000:3000/tcp \
+            -v /opt/adguard/work:/opt/adguardhome/work \
+            -v /opt/adguard/conf:/opt/adguardhome/conf \
+            {{ adguard_image }}
+          ExecStop=/usr/bin/docker stop adguard
+
+          [Install]
+          WantedBy=multi-user.target
+      register: adguard_unit
+
+    - name: Remove obsolete AdGuard post-start validation unit
+      ansible.builtin.file:
+        path: /etc/systemd/system/adguard-post-start.service
+        state: absent
+      register: adguard_post_start_unit_removed
+
+    - name: Reload systemd when AdGuard units change
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: adguard_unit.changed or adguard_post_start_unit_removed.changed
+
+    - name: Enable and start AdGuard Home
+      ansible.builtin.systemd:
+        name: adguard
+        state: "{{ 'restarted' if adguard_unit.changed or adguard_image_pull.changed else 'started' }}"
+        enabled: true
+
+    - name: Wait for AdGuard HTTP root
+      ansible.builtin.uri:
+        url: http://127.0.0.1/
+        status_code: [200, 302]
+        return_content: false
+      register: adguard_http_root
+      retries: 24
+      delay: 5
+      until: adguard_http_root.status in [200, 302]
+
+    - name: Verify AdGuard DNS over UDP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_udp
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_udp.rc == 0 and '127.0.0.1' in adguard_dns_udp.stdout
+
+    - name: Verify AdGuard DNS over TCP
+      ansible.builtin.command:
+        argv:
+          - dig
+          - "@127.0.0.1"
+          - localhost
+          - A
+          - +tcp
+          - +time=2
+          - +tries=1
+          - +short
+      register: adguard_dns_tcp
+      changed_when: false
+      retries: 12
+      delay: 5
+      until: adguard_dns_tcp.rc == 0 and '127.0.0.1' in adguard_dns_tcp.stdout

ansible/playbooks/pve-backup-jobs.yml

@@ -0,0 +1,114 @@
+- name: Configure Proxmox backup jobs
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    pve_backup_jobs:
+      - id: homelab-pbs-daily-cloud
+        comment: Daily PBS backup for cloud-pc service containers
+        node: cloud-pc
+        vmid: 141,145,146,147,149
+        storage: pbs
+        schedule: "02:10"
+        mode: snapshot
+        exclude_path: /var/lib/docker/fuse-overlayfs/*/merged
+      - id: homelab-pbs-daily-mini
+        comment: Daily PBS backup for mini-pc service containers
+        node: mini-pc
+        vmid: 132,140,142,143,144,150
+        storage: pbs
+        schedule: "02:40"
+        mode: snapshot
+        exclude_path: /var/lib/docker/fuse-overlayfs/*/merged
+      - id: homelab-local-weekly-pbs
+        comment: Weekly local backup for PBS container rootfs/config
+        node: cloud-pc
+        vmid: 120
+        storage: backup
+        schedule: "Sun 03:30"
+        mode: snapshot
+        prune_backups: keep-last=2
+
+  tasks:
+    - name: Get existing Proxmox backup jobs
+      ansible.builtin.command: pvesh get /cluster/backup --output-format json
+      register: pve_backup_jobs_existing_raw
+      changed_when: false
+
+    - name: Parse existing Proxmox backup jobs
+      ansible.builtin.set_fact:
+        pve_backup_jobs_existing: "{{ pve_backup_jobs_existing_raw.stdout | from_json }}"
+
+    - name: Create missing Proxmox backup jobs
+      ansible.builtin.command: >-
+        pvesh create /cluster/backup
+        --id {{ item.id }}
+        --enabled 1
+        --node {{ item.node }}
+        --vmid {{ item.vmid }}
+        --storage {{ item.storage }}
+        --schedule {{ item.schedule | quote }}
+        --mode {{ item.mode }}
+        {% if item.prune_backups is defined %}
+        --prune-backups {{ item.prune_backups }} --remove 1
+        {% else %}
+        --remove 0
+        {% endif %}
+        {% if item.exclude_path is defined %}
+        --exclude-path {{ item.exclude_path | quote }}
+        {% endif %}
+        --notes-template {{ '{{' }}guestname{{ '}}' }}
+        --comment {{ item.comment | quote }}
+      loop: "{{ pve_backup_jobs }}"
+      when: item.id not in (pve_backup_jobs_existing | map(attribute='id') | list)
+
+    - name: Update existing Proxmox backup jobs
+      ansible.builtin.command: >-
+        pvesh set /cluster/backup/{{ item.id }}
+        --enabled 1
+        --node {{ item.node }}
+        --vmid {{ item.vmid }}
+        --storage {{ item.storage }}
+        --schedule {{ item.schedule | quote }}
+        --mode {{ item.mode }}
+        {% if item.prune_backups is defined %}
+        --prune-backups {{ item.prune_backups }} --remove 1
+        {% else %}
+        --delete prune-backups --remove 0
+        {% endif %}
+        {% if item.exclude_path is defined %}
+        --exclude-path {{ item.exclude_path | quote }}
+        {% else %}
+        --delete exclude-path
+        {% endif %}
+        --notes-template {{ '{{' }}guestname{{ '}}' }}
+        --comment {{ item.comment | quote }}
+      loop: "{{ pve_backup_jobs }}"
+      vars:
+        current_job: >-
+          {{ pve_backup_jobs_existing | selectattr('id', 'equalto', item.id) | first }}
+      when:
+        - item.id in (pve_backup_jobs_existing | map(attribute='id') | list)
+        - >-
+          current_job.enabled | int != 1 or
+          current_job.node != item.node or
+          current_job.vmid | string != item.vmid | string or
+          current_job.storage != item.storage or
+          current_job.schedule != item.schedule or
+          current_job.mode != item.mode or
+          current_job.comment != item.comment or
+          current_job.remove | int != (1 if item.prune_backups is defined else 0) or
+          current_job['exclude-path'] | default([]) != ([item.exclude_path] if item.exclude_path is defined else []) or
+          (item.prune_backups is not defined and current_job['prune-backups'] is defined) or
+          (item.prune_backups is defined and
+           (current_job['prune-backups'] | default({})).get('keep-last', 0) | int !=
+           item.prune_backups | regex_replace('^keep-last=', '') | int)
+      changed_when: true
+
+    - name: Show configured Proxmox backup jobs
+      ansible.builtin.command: pvesh get /cluster/backup --output-format yaml
+      register: pve_backup_jobs_configured
+      changed_when: false
+
+    - name: Print configured Proxmox backup jobs
+      ansible.builtin.debug:
+        var: pve_backup_jobs_configured.stdout_lines

ansible/playbooks/pve-docker-test.yml

@@ -0,0 +1,125 @@
+---
+- name: Create Docker test LXC on cloud-pc
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    docker_test_vmid: 145
+    docker_test_hostname: docker-test
+    docker_test_ip: 192.168.1.29/24
+    docker_test_gateway: 192.168.1.1
+    docker_test_rootfs: data:8
+    docker_test_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    docker_test_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  handlers:
+    - name: restart docker test lxc
+      ansible.builtin.shell: "pct stop {{ docker_test_vmid }} || true; pct start {{ docker_test_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if Docker test LXC exists
+      ansible.builtin.command: "pct config {{ docker_test_vmid }}"
+      register: docker_test_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Install Docker test LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/docker-test-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', docker_test_pubkey_file) }}\n"
+      when: docker_test_pct_config.rc != 0
+
+    - name: Create Docker test LXC
+      ansible.builtin.command: >-
+        pct create {{ docker_test_vmid }} {{ docker_test_ostemplate }}
+        --hostname {{ docker_test_hostname }}
+        --rootfs {{ docker_test_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge=vmbr0,gw={{ docker_test_gateway }},ip={{ docker_test_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=50
+        --cmode shell
+        --ssh-public-keys /tmp/docker-test-lxc.pub
+      when: docker_test_pct_config.rc != 0
+
+    - name: Start Docker test LXC
+      ansible.builtin.command: "pct start {{ docker_test_vmid }}"
+      register: docker_test_pct_start
+      changed_when: docker_test_pct_start.rc == 0
+      failed_when: docker_test_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in Docker test LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ docker_test_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart docker test lxc
+
+    - name: Bind mount FUSE device in Docker test LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ docker_test_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart docker test lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for Docker test SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: docker-test
+      vars:
+        ansible_become: false
+
+- name: Configure Docker test host
+  hosts: docker-test
+  gather_facts: true
+  vars:
+    ansible_become: false
+  tasks:
+    - name: Install Docker packages
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - ca-certificates
+          - curl
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: docker_test_daemon_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if docker_test_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Verify Docker can run a container
+      ansible.builtin.command: docker run --rm hello-world
+      changed_when: false

ansible/playbooks/pve-emergency-bot.yml

@@ -0,0 +1,30 @@
+---
+- name: Create emergency-bot LXC on mini-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 148
+    pve_lxc_node: mini-pc
+    pve_lxc_hostname: emergency-bot
+    pve_lxc_ip: 192.168.1.32/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: local-lvm:4
+    pve_lxc_cores: 1
+    pve_lxc_memory: 512
+    pve_lxc_swap: 256
+    pve_lxc_startup: order=70
+    pve_lxc_ostemplate: "{{ lookup('env', 'PVE_LXC_OSTEMPLATE') | default('local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst', true) }}"
+  roles:
+    - role: pve_lxc
+
+- name: Wait for emergency-bot SSH
+  hosts: emergency-bot
+  gather_facts: false
+  tasks:
+    - name: Wait for emergency-bot to accept SSH connections
+      ansible.builtin.wait_for_connection:
+        timeout: 120

ansible/playbooks/pve-gitea.yml

@@ -25,6 +25,19 @@
       changed_when: false
       failed_when: false
 
+    - name: Refuse to modify a foreign Gitea VMID
+      ansible.builtin.assert:
+        that:
+          - gitea_pct_config.rc != 0 or gitea_existing_hostname == 'gitea'
+        fail_msg: VMID {{ gitea_vmid }} already exists and is not the Gitea container.
+      vars:
+        gitea_existing_hostname: >-
+          {{ gitea_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
     - name: Install Gitea LXC SSH public key on PVE host
       ansible.builtin.copy:
         dest: /tmp/gitea-lxc.pub
@@ -103,7 +116,7 @@
   vars:
     ansible_become: false
     gitea_data_dir: /opt/gitea/data
-    gitea_image: gitea/gitea:latest
+    gitea_image: gitea/gitea:1.27.1@sha256:b64126cf5c3f4e5f0f231b510bb13715f6cb8e508188b44de90bdb9a04f3055d
     gitea_container_name: gitea
     gitea_http_port: 3000
     gitea_ssh_port: 2222
@@ -154,6 +167,18 @@
         group: "1000"
         mode: "0750"
 
+    - name: Check if the configured Gitea image is present
+      ansible.builtin.command: "docker image inspect {{ gitea_image }}"
+      register: gitea_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Gitea image
+      ansible.builtin.command: "docker pull {{ gitea_image }}"
+      when: gitea_image_inspect.rc != 0
+      register: gitea_image_pull
+      changed_when: true
+
     - name: Install Gitea systemd unit
       ansible.builtin.copy:
         dest: /etc/systemd/system/gitea.service
@@ -172,7 +197,7 @@
           ExecStartPre=-/usr/bin/docker rm -f {{ gitea_container_name }}
           ExecStart=/usr/bin/docker run --rm \
             --name {{ gitea_container_name }} \
-            --pull always \
+            --pull never \
             -p {{ gitea_http_port }}:3000 \
             -p {{ gitea_ssh_port }}:22 \
             -v {{ gitea_data_dir }}:/data \
@@ -193,5 +218,14 @@
     - name: Enable and start Gitea
       ansible.builtin.systemd:
         name: gitea
-        state: started
+        state: "{{ 'restarted' if gitea_unit.changed or gitea_image_pull.changed else 'started' }}"
         enabled: true
+
+    - name: Wait for Gitea HTTP health endpoint
+      ansible.builtin.uri:
+        url: "http://127.0.0.1:{{ gitea_http_port }}/api/healthz"
+        status_code: 200
+      register: gitea_health
+      retries: 24
+      delay: 5
+      until: gitea_health.status == 200

ansible/playbooks/pve-grimmory.yml

@@ -0,0 +1,431 @@
+---
+- name: Guard Grimmory VMID before API updates
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 149 configuration
+      ansible.builtin.command: pct config 149
+      register: grimmory_existing_vmid
+      check_mode: false
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 149
+      ansible.builtin.assert:
+        that:
+          - grimmory_existing_vmid.rc != 0 or grimmory_existing_hostname == 'grimmory'
+        fail_msg: VMID 149 already exists and is not the Grimmory container.
+      vars:
+        grimmory_existing_hostname: >-
+          {{ grimmory_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+- name: Create Grimmory LXC on cloud-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 149
+    pve_lxc_node: cloud-pc
+    pve_lxc_hostname: grimmory
+    pve_lxc_ip: 192.168.1.34/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: data:64
+    pve_lxc_cores: 2
+    pve_lxc_memory: 4096
+    pve_lxc_swap: 1024
+    pve_lxc_startup: order=100
+    pve_lxc_unprivileged: true
+    pve_lxc_update: false
+    pve_lxc_features:
+      - nesting=1
+    pve_lxc_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+  roles:
+    - role: pve_lxc
+
+- name: Configure Grimmory LXC devices
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    grimmory_vmid: 149
+  handlers:
+    - name: restart Grimmory LXC
+      ansible.builtin.command: "pct reboot {{ grimmory_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Read Grimmory LXC configuration
+      ansible.builtin.command: "pct config {{ grimmory_vmid }}"
+      register: grimmory_lxc_config
+      changed_when: false
+
+    - name: Require expected Grimmory LXC properties
+      ansible.builtin.assert:
+        that:
+          - "'unprivileged: 1' in grimmory_lxc_config.stdout"
+          - "'rootfs: data:' in grimmory_lxc_config.stdout"
+          - "'onboot: 1' in grimmory_lxc_config.stdout"
+
+    - name: Configure Grimmory LXC network
+      ansible.builtin.command: >-
+        pct set {{ grimmory_vmid }}
+        --net0 name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,ip=192.168.1.34/24
+      vars:
+        grimmory_net0: >-
+          {{ grimmory_lxc_config.stdout_lines
+             | select('match', '^net0: ')
+             | first
+             | default('') }}
+      when: >-
+        'name=eth0' not in grimmory_net0 or
+        'bridge=vmbr0' not in grimmory_net0 or
+        'firewall=1' not in grimmory_net0 or
+        'gw=192.168.1.1' not in grimmory_net0 or
+        'ip=192.168.1.34/24' not in grimmory_net0
+      notify: restart Grimmory LXC
+
+    - name: Enable keyctl for Docker in Grimmory LXC
+      ansible.builtin.command: "pct set {{ grimmory_vmid }} --features nesting=1,keyctl=1"
+      when: "'nesting=1' not in grimmory_lxc_config.stdout or 'keyctl=1' not in grimmory_lxc_config.stdout"
+      notify: restart Grimmory LXC
+
+    - name: Allow FUSE device in Grimmory LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ grimmory_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart Grimmory LXC
+
+    - name: Bind mount FUSE device in Grimmory LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ grimmory_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart Grimmory LXC
+
+    - name: Apply pending Grimmory LXC configuration
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for Grimmory SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 180
+      delegate_to: grimmory
+      vars:
+        ansible_become: false
+
+- name: Configure Grimmory runtime
+  hosts: grimmory
+  gather_facts: true
+  vars:
+    ansible_become: false
+    grimmory_root: /opt/grimmory
+    grimmory_image: grimmory/grimmory:v3.2.4@sha256:dfa7afdfcf25d649fd664497a62385dd00cd9678c37546e182c172e41c8e80cb
+    grimmory_mariadb_image: lscr.io/linuxserver/mariadb:11.4.8@sha256:91de7f701bc7fc3a424b81beafca7a7c6c4c5b7c8be6afd2ae148698695c0b0c
+  tasks:
+    - name: Install Grimmory runtime packages
+      ansible.builtin.apt:
+        name:
+          - ca-certificates
+          - curl
+          - docker-compose
+          - docker.io
+          - fuse-overlayfs
+          - mariadb-client
+          - openssl
+          - prometheus-node-exporter
+          - ufw
+        state: present
+        update_cache: true
+
+    - name: Verify FUSE device exists
+      ansible.builtin.stat:
+        path: /dev/fuse
+      register: grimmory_fuse
+
+    - name: Require FUSE device
+      ansible.builtin.assert:
+        that:
+          - grimmory_fuse.stat.exists
+          - grimmory_fuse.stat.ischr
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: grimmory_docker_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        enabled: true
+        state: "{{ 'restarted' if grimmory_docker_config.changed else 'started' }}"
+
+    - name: Allow SSH from the HomeLab LAN
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ homelab_lan_cidr }}"
+
+    - name: Allow SSH from the OpenVPN network
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ openvpn_network_cidr }}"
+
+    - name: Allow Grimmory from the HomeLab LAN
+      community.general.ufw:
+        rule: allow
+        port: "6060"
+        proto: tcp
+        src: "{{ homelab_lan_cidr }}"
+
+    - name: Allow Grimmory from the OpenVPN network
+      community.general.ufw:
+        rule: allow
+        port: "6060"
+        proto: tcp
+        src: "{{ openvpn_network_cidr }}"
+
+    - name: Allow Node Exporter from the monitoring LXC
+      community.general.ufw:
+        rule: allow
+        port: "9100"
+        proto: tcp
+        src: 192.168.1.30
+
+    - name: Enable restrictive Grimmory firewall
+      community.general.ufw:
+        state: enabled
+        policy: deny
+        direction: incoming
+
+    - name: Create Grimmory directories
+      ansible.builtin.file:
+        path: "{{ item.path }}"
+        state: directory
+        owner: "{{ item.owner }}"
+        group: "{{ item.group }}"
+        mode: "{{ item.mode }}"
+      loop:
+        - { path: "{{ grimmory_root }}", owner: root, group: root, mode: "0750" }
+        - { path: "{{ grimmory_root }}/data", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/books", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/bookdrop", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/mariadb", owner: "1000", group: "1000", mode: "0750" }
+        - { path: "{{ grimmory_root }}/backup-staging", owner: root, group: root, mode: "0700" }
+
+    - name: Generate Grimmory secrets once
+      ansible.builtin.shell: |
+        set -eu
+        umask 077
+        if [ -e {{ grimmory_root }}/.env ]; then
+          exit 0
+        fi
+        db_password=$(openssl rand -hex 32)
+        root_password=$(openssl rand -hex 32)
+        cat > {{ grimmory_root }}/.env <<EOF
+        TZ=Europe/Moscow
+        APP_USER_ID=1000
+        APP_GROUP_ID=1000
+        DB_USER_ID=1000
+        DB_GROUP_ID=1000
+        DB_PASSWORD=$db_password
+        MYSQL_ROOT_PASSWORD=$root_password
+        EOF
+        printf created
+      args:
+        executable: /bin/sh
+      register: grimmory_secrets
+      changed_when: grimmory_secrets.stdout == 'created'
+      no_log: true
+
+    - name: Enforce Grimmory secret file permissions
+      ansible.builtin.file:
+        path: "{{ grimmory_root }}/.env"
+        owner: root
+        group: root
+        mode: "0600"
+
+    - name: Install Grimmory environment example
+      ansible.builtin.copy:
+        dest: "{{ grimmory_root }}/.env.example"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          TZ=Europe/Moscow
+          APP_USER_ID=1000
+          APP_GROUP_ID=1000
+          DB_USER_ID=1000
+          DB_GROUP_ID=1000
+          DB_PASSWORD=replace-with-random-password
+          MYSQL_ROOT_PASSWORD=replace-with-separate-random-password
+
+    - name: Install Grimmory Compose configuration
+      ansible.builtin.copy:
+        dest: "{{ grimmory_root }}/compose.yml"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          services:
+            grimmory:
+              image: {{ grimmory_image }}
+              container_name: grimmory
+              environment:
+                USER_ID: "${APP_USER_ID}"
+                GROUP_ID: "${APP_GROUP_ID}"
+                TZ: "${TZ}"
+                DATABASE_URL: jdbc:mariadb://mariadb:3306/grimmory
+                DATABASE_USERNAME: grimmory
+                DATABASE_PASSWORD: "${DB_PASSWORD}"
+                DISK_TYPE: LOCAL
+                ALLOWED_ORIGINS: https://books.ada-dev.ru
+              depends_on:
+                mariadb:
+                  condition: service_healthy
+              ports:
+                - "192.168.1.34:6060:6060"
+              volumes:
+                - ./data:/app/data
+                - ./books:/books
+                - ./bookdrop:/bookdrop
+              restart: unless-stopped
+
+            mariadb:
+              image: {{ grimmory_mariadb_image }}
+              container_name: grimmory-mariadb
+              environment:
+                PUID: "${DB_USER_ID}"
+                PGID: "${DB_GROUP_ID}"
+                TZ: "${TZ}"
+                MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
+                MYSQL_DATABASE: grimmory
+                MYSQL_USER: grimmory
+                MYSQL_PASSWORD: "${DB_PASSWORD}"
+              volumes:
+                - ./mariadb:/config
+              healthcheck:
+                test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"]
+                interval: 5s
+                timeout: 5s
+                retries: 20
+              restart: unless-stopped
+      register: grimmory_compose
+
+    - name: Install Docker firewall script
+      ansible.builtin.copy:
+        dest: /usr/local/sbin/grimmory-docker-firewall
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          set -eu
+          iptables -N GRIMMORY-FILTER 2>/dev/null || true
+          iptables -F GRIMMORY-FILTER
+          iptables -A GRIMMORY-FILTER -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
+          iptables -A GRIMMORY-FILTER -s {{ homelab_lan_cidr }} -p tcp -m conntrack --ctorigdst 192.168.1.34 --ctorigdstport 6060 -j ACCEPT
+          iptables -A GRIMMORY-FILTER -s {{ openvpn_network_cidr }} -p tcp -m conntrack --ctorigdst 192.168.1.34 --ctorigdstport 6060 -j ACCEPT
+          iptables -A GRIMMORY-FILTER -p tcp -m conntrack --ctorigdst 192.168.1.34 --ctorigdstport 6060 -j DROP
+          iptables -A GRIMMORY-FILTER -j RETURN
+          iptables -C DOCKER-USER -j GRIMMORY-FILTER 2>/dev/null || iptables -I DOCKER-USER 1 -j GRIMMORY-FILTER
+      register: grimmory_firewall_script
+
+    - name: Install Docker firewall service
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/grimmory-docker-firewall.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Restrict Grimmory Docker published port
+          After=docker.service
+          Requires=docker.service
+          Before=grimmory.service
+
+          [Service]
+          Type=oneshot
+          ExecStart=/usr/local/sbin/grimmory-docker-firewall
+          RemainAfterExit=yes
+
+          [Install]
+          WantedBy=multi-user.target
+      register: grimmory_firewall_unit
+
+    - name: Install Grimmory systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/grimmory.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Grimmory Compose stack
+          Wants=network-online.target
+          After=network-online.target docker.service grimmory-docker-firewall.service
+          Requires=docker.service grimmory-docker-firewall.service
+
+          [Service]
+          Type=oneshot
+          RemainAfterExit=yes
+          WorkingDirectory={{ grimmory_root }}
+          ExecStart=/usr/bin/docker compose -f {{ grimmory_root }}/compose.yml up -d --remove-orphans
+          ExecStop=/usr/bin/docker compose -f {{ grimmory_root }}/compose.yml down
+
+          [Install]
+          WantedBy=multi-user.target
+      register: grimmory_unit
+
+    - name: Reload systemd for Grimmory units
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: grimmory_firewall_unit.changed or grimmory_unit.changed
+
+    - name: Enable and apply Docker firewall
+      ansible.builtin.systemd:
+        name: grimmory-docker-firewall
+        enabled: true
+        state: "{{ 'restarted' if grimmory_firewall_script.changed or grimmory_firewall_unit.changed else 'started' }}"
+
+    - name: Validate Grimmory Compose configuration
+      ansible.builtin.command: docker compose -f {{ grimmory_root }}/compose.yml config --quiet
+      args:
+        chdir: "{{ grimmory_root }}"
+      changed_when: false
+      no_log: true
+
+    - name: Enable and start Grimmory
+      ansible.builtin.systemd:
+        name: grimmory
+        enabled: true
+        state: "{{ 'restarted' if grimmory_compose.changed or grimmory_unit.changed else 'started' }}"
+
+    - name: Wait for Grimmory health endpoint
+      ansible.builtin.uri:
+        url: http://192.168.1.34:6060/api/v1/healthcheck
+        status_code: 200
+      register: grimmory_health
+      retries: 120
+      delay: 5
+      until: grimmory_health.status == 200
+
+    - name: Verify Docker storage driver
+      ansible.builtin.command: docker info --format '{{ "{{" }}.Driver{{ "}}" }}'
+      register: grimmory_docker_driver
+      changed_when: false
+      failed_when: grimmory_docker_driver.stdout != 'fuse-overlayfs'

ansible/playbooks/pve-gyro.yml

@@ -0,0 +1,189 @@
+---
+- name: Guard Gyro VMID before API updates
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 150 configuration
+      ansible.builtin.command: pct config 150
+      register: gyro_existing_vmid
+      check_mode: false
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 150
+      ansible.builtin.assert:
+        that:
+          - gyro_existing_vmid.rc != 0 or gyro_existing_hostname == 'gyro'
+        fail_msg: VMID 150 already exists and is not the Gyro container.
+      vars:
+        gyro_existing_hostname: >-
+          {{ gyro_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+- name: Create Gyro LXC on mini-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 150
+    pve_lxc_node: mini-pc
+    pve_lxc_hostname: gyro
+    pve_lxc_ip: 192.168.1.35/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: local-lvm:2
+    pve_lxc_cores: 1
+    pve_lxc_memory: 512
+    pve_lxc_swap: 256
+    pve_lxc_startup: order=80
+    pve_lxc_unprivileged: true
+    pve_lxc_update: false
+    pve_lxc_features: []
+    pve_lxc_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+  roles:
+    - role: pve_lxc
+
+- name: Configure Gyro LXC isolation
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    gyro_vmid: 150
+  tasks:
+    - name: Read Gyro LXC configuration
+      ansible.builtin.command: "pct config {{ gyro_vmid }}"
+      register: gyro_lxc_config
+      changed_when: false
+
+    - name: Require expected Gyro LXC properties
+      ansible.builtin.assert:
+        that:
+          - "'hostname: gyro' in gyro_lxc_config.stdout"
+          - "'unprivileged: 1' in gyro_lxc_config.stdout"
+          - "'rootfs: local-lvm:' in gyro_lxc_config.stdout"
+          - "'onboot: 1' in gyro_lxc_config.stdout"
+
+    - name: Check for an existing Gyro firewall file
+      ansible.builtin.stat:
+        path: "/etc/pve/firewall/{{ gyro_vmid }}.fw"
+      register: gyro_firewall_file
+
+    - name: Preserve the existing Gyro firewall file
+      ansible.builtin.slurp:
+        src: "/etc/pve/firewall/{{ gyro_vmid }}.fw"
+      register: gyro_previous_firewall
+      when: gyro_firewall_file.stat.exists
+
+    - name: Apply and verify Gyro firewall
+      block:
+        - name: Render Proxmox firewall for Gyro
+          ansible.builtin.copy:
+            dest: "/tmp/gyro-{{ gyro_vmid }}.fw"
+            owner: root
+            group: root
+            mode: "0640"
+            content: |
+              [OPTIONS]
+              enable: 1
+              policy_in: DROP
+              policy_out: ACCEPT
+
+              [RULES]
+              IN ACCEPT -source {{ homelab_lan_cidr }} -p tcp -dport 22 -log nolog
+              IN ACCEPT -source {{ openvpn_network_cidr }} -p tcp -dport 22 -log nolog
+              OUT ACCEPT -dest 192.168.1.27 -p tcp -dport 7890 -log nolog
+              OUT DROP -dest {{ homelab_lan_cidr }} -log nolog
+          register: gyro_rendered_firewall
+          changed_when: false
+
+        - name: Compare rendered and active Gyro firewall
+          ansible.builtin.command: >-
+            cmp -s /tmp/gyro-{{ gyro_vmid }}.fw /etc/pve/firewall/{{ gyro_vmid }}.fw
+          register: gyro_firewall_comparison
+          changed_when: false
+          failed_when: gyro_firewall_comparison.rc not in [0, 1]
+          when: gyro_firewall_file.stat.exists
+
+        - name: Install Proxmox firewall for Gyro
+          ansible.builtin.command: >-
+            cp /tmp/gyro-{{ gyro_vmid }}.fw /etc/pve/firewall/{{ gyro_vmid }}.fw
+          when: not gyro_firewall_file.stat.exists or gyro_firewall_comparison.rc != 0
+          changed_when: true
+
+        - name: Validate Proxmox firewall configuration
+          ansible.builtin.command: pve-firewall compile
+          changed_when: false
+
+        - name: Read cluster firewall options
+          ansible.builtin.command: pvesh get /cluster/firewall/options --output-format json
+          register: gyro_cluster_firewall_options
+          changed_when: false
+
+        - name: Report staged Proxmox firewall state
+          ansible.builtin.debug:
+            msg: >-
+              {{ 'Gyro Proxmox firewall is active.'
+                 if (gyro_cluster_firewall_options.stdout | from_json).enable | default(0) | int == 1
+                 else 'Gyro Proxmox firewall is staged but inactive because the cluster firewall is disabled; UFW remains the enforced isolation layer.' }}
+
+        - name: Verify SSH remains reachable through the firewall
+          ansible.builtin.wait_for_connection:
+            timeout: 180
+          delegate_to: gyro
+          vars:
+            ansible_become: false
+      rescue:
+        - name: Restore the previous Gyro firewall file
+          ansible.builtin.copy:
+            dest: "/tmp/gyro-{{ gyro_vmid }}-previous.fw"
+            content: "{{ gyro_previous_firewall.content | b64decode }}"
+            owner: root
+            group: root
+            mode: "0640"
+          when: gyro_firewall_file.stat.exists
+
+        - name: Reinstall the previous Gyro firewall file
+          ansible.builtin.command: >-
+            cp /tmp/gyro-{{ gyro_vmid }}-previous.fw /etc/pve/firewall/{{ gyro_vmid }}.fw
+          when: gyro_firewall_file.stat.exists
+          changed_when: true
+
+        - name: Remove the failed new Gyro firewall file
+          ansible.builtin.file:
+            path: "/etc/pve/firewall/{{ gyro_vmid }}.fw"
+            state: absent
+          when: not gyro_firewall_file.stat.exists
+
+        - name: Recompile restored Proxmox firewall configuration
+          ansible.builtin.command: pve-firewall compile
+          changed_when: false
+
+        - name: Remove temporary Gyro firewall files after rollback
+          ansible.builtin.file:
+            path: "{{ item }}"
+            state: absent
+          loop:
+            - "/tmp/gyro-{{ gyro_vmid }}.fw"
+            - "/tmp/gyro-{{ gyro_vmid }}-previous.fw"
+
+        - name: Stop after rolling back the Gyro firewall
+          ansible.builtin.fail:
+            msg: Gyro firewall validation or SSH reachability failed; the previous firewall state was restored.
+
+    - name: Remove temporary Gyro firewall file
+      ansible.builtin.file:
+        path: "/tmp/gyro-{{ gyro_vmid }}.fw"
+        state: absent
+      changed_when: false
+
+- name: Wait for Gyro SSH
+  hosts: gyro
+  gather_facts: false
+  tasks:
+    - name: Wait for Gyro to accept SSH connections
+      ansible.builtin.wait_for_connection:
+        timeout: 180

ansible/playbooks/pve-hermes-ai.yml

@@ -0,0 +1,334 @@
+---
+- name: Create Hermes AI LXC on cloud-pc
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    hermes_ai_vmid: 147
+    hermes_ai_hostname: hermes-ai
+    hermes_ai_ip: 192.168.1.31/24
+    hermes_ai_gateway: 192.168.1.1
+    hermes_ai_rootfs: data:24
+    hermes_ai_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    hermes_ai_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  tasks:
+    - name: Check if Hermes AI LXC exists
+      ansible.builtin.command: "pct config {{ hermes_ai_vmid }}"
+      register: hermes_ai_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Install Hermes AI LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/hermes-ai-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', hermes_ai_pubkey_file) }}\n"
+      when: hermes_ai_pct_config.rc != 0
+
+    - name: Create Hermes AI LXC
+      ansible.builtin.command: >-
+        pct create {{ hermes_ai_vmid }} {{ hermes_ai_ostemplate }}
+        --hostname {{ hermes_ai_hostname }}
+        --rootfs {{ hermes_ai_rootfs }}
+        --cores 2
+        --memory 4096
+        --swap 512
+        --net0 name=eth0,bridge=vmbr0,gw={{ hermes_ai_gateway }},ip={{ hermes_ai_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=90
+        --cmode shell
+        --ssh-public-keys /tmp/hermes-ai-lxc.pub
+      when: hermes_ai_pct_config.rc != 0
+
+    - name: Start Hermes AI LXC
+      ansible.builtin.command: "pct start {{ hermes_ai_vmid }}"
+      register: hermes_ai_pct_start
+      changed_when: hermes_ai_pct_start.rc == 0
+      failed_when: hermes_ai_pct_start.rc not in [0, 255]
+
+- name: Configure Hermes AI runtime
+  hosts: cloud-pc
+  gather_facts: false
+  vars:
+    hermes_ai_vmid: 147
+  handlers:
+    - name: restart Hermes AI LXC
+      ansible.builtin.command: "pct reboot {{ hermes_ai_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Read Hermes AI LXC configuration
+      ansible.builtin.command: "pct config {{ hermes_ai_vmid }}"
+      register: hermes_ai_lxc_config
+      changed_when: false
+
+    - name: Enable keyctl for Docker in Hermes AI LXC
+      ansible.builtin.command: "pct set {{ hermes_ai_vmid }} --features nesting=1,keyctl=1"
+      when: "'keyctl=1' not in hermes_ai_lxc_config.stdout"
+      notify: restart Hermes AI LXC
+
+    - name: Allow FUSE device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Bind mount FUSE device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Check TUN device on PVE host
+      ansible.builtin.stat:
+        path: /dev/net/tun
+      register: hermes_ai_tun_device
+
+    - name: Require TUN device on PVE host
+      ansible.builtin.assert:
+        that:
+          - hermes_ai_tun_device.stat.exists
+          - hermes_ai_tun_device.stat.ischr
+        fail_msg: /dev/net/tun must exist on cloud-pc before enabling the Hermes AI transparent proxy.
+
+    - name: Allow TUN device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:200 rwm"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Bind mount TUN device in Hermes AI LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ hermes_ai_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file"
+        state: present
+      notify: restart Hermes AI LXC
+
+    - name: Apply pending Hermes AI LXC configuration
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for Hermes AI SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: hermes-ai
+      vars:
+        ansible_become: false
+
+- name: Install Hermes AI base runtime
+  hosts: hermes-ai
+  gather_facts: false
+  vars:
+    ansible_become: false
+    hermes_ai_root: /opt/hermes-ai
+    hermes_ai_state: /srv/hermes-ai
+    hermes_ai_proxy_dir: /opt/hermes-ai/tun-proxy
+    hermes_ai_proxy_image: metacubex/mihomo@sha256:e6acd921addecfd59a8e2d38203f88356d635b54de6c0673db0e015139989312
+    hermes_ai_proxy_name: hermes-ai-tun-proxy
+  tasks:
+    - name: Install Hermes AI runtime packages
+      ansible.builtin.apt:
+        name:
+          - ca-certificates
+          - curl
+          - docker-compose
+          - docker.io
+          - fuse-overlayfs
+          - git
+          - ufw
+        state: present
+        update_cache: true
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: hermes_ai_docker_config
+
+    - name: Enable Docker service
+      ansible.builtin.systemd:
+        name: docker
+        enabled: true
+        state: "{{ 'restarted' if hermes_ai_docker_config.changed else 'started' }}"
+
+    - name: Allow SSH only from the HomeLab LAN
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ homelab_lan_cidr }}"
+
+    - name: Allow SSH from the OpenVPN network
+      community.general.ufw:
+        rule: allow
+        port: "22"
+        proto: tcp
+        src: "{{ openvpn_network_cidr }}"
+
+    - name: Enable restrictive Hermes AI firewall
+      community.general.ufw:
+        state: enabled
+        policy: deny
+        direction: incoming
+
+    - name: Create Hermes AI application directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - "{{ hermes_ai_root }}"
+        - "{{ hermes_ai_state }}"
+        - "{{ hermes_ai_proxy_dir }}"
+
+    - name: Configure Hermes AI transparent proxy
+      ansible.builtin.copy:
+        dest: "{{ hermes_ai_proxy_dir }}/config.yaml"
+        owner: root
+        group: root
+        mode: "0640"
+        content: |
+          ipv6: false
+
+          tun:
+            enable: true
+            stack: system
+            device: hermes-tun
+            auto-route: true
+            auto-redirect: true
+            auto-detect-interface: true
+            route-exclude-address:
+              - 192.168.1.27/32
+            dns-hijack:
+              - any:53
+
+          dns:
+            enable: true
+            enhanced-mode: redir-host
+            nameserver:
+              - https://cloudflare-dns.com/dns-query
+
+          sniffer:
+            enable: true
+            force-dns-mapping: true
+            parse-pure-ip: true
+
+          proxies:
+            - name: mihomo-lan
+              type: socks5
+              server: 192.168.1.27
+              port: 7891
+
+          proxy-groups:
+            - name: PROXY
+              type: select
+              proxies:
+                - mihomo-lan
+
+          rules:
+            - IP-CIDR,127.0.0.0/8,DIRECT,no-resolve
+            - IP-CIDR,10.0.0.0/8,DIRECT,no-resolve
+            - IP-CIDR,172.16.0.0/12,DIRECT,no-resolve
+            - IP-CIDR,192.168.0.0/16,DIRECT,no-resolve
+            - IP-CIDR,169.254.0.0/16,DIRECT,no-resolve
+            - MATCH,PROXY
+      register: hermes_ai_proxy_config
+
+    - name: Wait for Mihomo SOCKS5 upstream
+      ansible.builtin.wait_for:
+        host: 192.168.1.27
+        port: 7891
+        timeout: 15
+
+    - name: Validate Hermes AI transparent proxy config
+      ansible.builtin.command: >-
+        docker run --rm --network none
+        -v {{ hermes_ai_proxy_dir }}:/root/.config/mihomo:ro
+        {{ hermes_ai_proxy_image }} -t -d /root/.config/mihomo
+      changed_when: false
+
+    - name: Install Hermes AI transparent proxy service
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/hermes-ai-tun-proxy.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Hermes AI transparent Mihomo proxy
+          After=docker.service network-online.target
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ hermes_ai_proxy_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ hermes_ai_proxy_name }} \
+            --network host \
+            --cap-drop ALL \
+            --cap-add NET_ADMIN \
+            --security-opt no-new-privileges \
+            --device /dev/net/tun \
+            -v {{ hermes_ai_proxy_dir }}:/root/.config/mihomo:ro \
+            {{ hermes_ai_proxy_image }}
+          ExecStop=/usr/bin/docker stop {{ hermes_ai_proxy_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: hermes_ai_proxy_unit
+
+    - name: Reload systemd when transparent proxy service changes
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: hermes_ai_proxy_unit.changed
+
+    - name: Enable Hermes AI transparent proxy
+      ansible.builtin.systemd:
+        name: hermes-ai-tun-proxy
+        enabled: true
+        state: "{{ 'restarted' if hermes_ai_proxy_config.changed or hermes_ai_proxy_unit.changed else 'started' }}"
+
+    - name: Check Hermes AI transparent proxy service
+      ansible.builtin.command: systemctl is-active hermes-ai-tun-proxy
+      register: hermes_ai_proxy_status
+      changed_when: false
+      failed_when: hermes_ai_proxy_status.stdout != 'active'
+
+    - name: Check Hermes AI external HTTPS access
+      ansible.builtin.command: curl --fail --silent --show-error --max-time 20 https://api.ipify.org
+      changed_when: false
+      no_log: true
+
+    - name: Document Hermes AI runtime layout
+      ansible.builtin.copy:
+        dest: "{{ hermes_ai_root }}/README.md"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Hermes AI runtime
+
+          Deploy the Hermes application here. Keep runtime state and the real
+          `.env` file in /srv/hermes-ai. Do not commit Telegram or LLM tokens.
+
+          Docker is installed with fuse-overlayfs for this unprivileged LXC.
+          Hermes is a Telegram bot and needs no published Docker ports. Do not
+          use `ports:` or `-p` without adding an explicit firewall policy:
+          Docker port publishing can bypass UFW.
+
+          The hermes-ai-tun-proxy service routes external traffic through
+          Mihomo at 192.168.1.27:7890. LAN traffic remains direct.

ansible/playbooks/pve-memoir-bot.yml

@@ -0,0 +1,305 @@
+---
+- name: Create memoir-bot LXC on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    memoir_bot_vmid: 142
+    memoir_bot_hostname: memoir-bot
+    memoir_bot_ip: 192.168.1.26/24
+    memoir_bot_gateway: 192.168.1.1
+    memoir_bot_bridge: vmbr0
+    memoir_bot_rootfs: local-lvm:16
+    memoir_bot_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    memoir_bot_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  handlers:
+    - name: restart memoir-bot lxc
+      ansible.builtin.shell: "pct stop {{ memoir_bot_vmid }} || true; pct start {{ memoir_bot_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if memoir-bot LXC exists
+      ansible.builtin.command: "pct config {{ memoir_bot_vmid }}"
+      register: memoir_bot_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Install memoir-bot LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/memoir-bot-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', memoir_bot_pubkey_file) }}\n"
+      when: memoir_bot_pct_config.rc != 0
+
+    - name: Create memoir-bot LXC
+      ansible.builtin.command: >-
+        pct create {{ memoir_bot_vmid }} {{ memoir_bot_ostemplate }}
+        --hostname {{ memoir_bot_hostname }}
+        --rootfs {{ memoir_bot_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge={{ memoir_bot_bridge }},gw={{ memoir_bot_gateway }},ip={{ memoir_bot_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=60
+        --cmode shell
+        --ssh-public-keys /tmp/memoir-bot-lxc.pub
+      when: memoir_bot_pct_config.rc != 0
+
+    - name: Start memoir-bot LXC
+      ansible.builtin.command: "pct start {{ memoir_bot_vmid }}"
+      register: memoir_bot_pct_start
+      changed_when: memoir_bot_pct_start.rc == 0
+      failed_when: memoir_bot_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in memoir-bot LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ memoir_bot_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart memoir-bot lxc
+
+    - name: Bind mount FUSE device in memoir-bot LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ memoir_bot_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart memoir-bot lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for memoir-bot SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: memoir-bot
+      vars:
+        ansible_become: false
+
+- name: Configure Docker and memoir-bot inside LXC
+  hosts: memoir-bot
+  gather_facts: true
+  vars:
+    ansible_become: false
+    memoir_bot_source_dir: /home/ada/Documents/Projects/Other/memoir_bot/
+    memoir_bot_app_dir: /opt/memoir-bot/app
+    memoir_bot_state_dir: /srv/memoir-bot
+    memoir_bot_env_file: /srv/memoir-bot/.env
+    memoir_bot_env_source: /home/ada/Documents/Projects/Other/memoir_bot/.env
+    memoir_bot_ssh_private_key_file: ~/.ssh/id_ed25519
+    memoir_bot_ssh_public_key_file: ~/.ssh/id_ed25519.pub
+    memoir_bot_vault_repo: git@github.com:ada-dmitry/SecondBrain.git
+    memoir_bot_vault_dir: /srv/memoir-bot/vault
+    memoir_bot_image: memoir-bot:local
+    memoir_bot_container_name: memoir-bot
+  tasks:
+    - name: Install Docker and deploy dependencies
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - git
+          - openssh-client
+          - rsync
+          - ca-certificates
+          - curl
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: docker_daemon_config
+
+    - name: Ensure Docker service is enabled and running
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if docker_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Ensure memoir-bot directories exist
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - "{{ memoir_bot_app_dir }}"
+        - "{{ memoir_bot_state_dir }}"
+        - "{{ memoir_bot_state_dir }}/ssh"
+
+    - name: Copy memoir-bot env example
+      ansible.builtin.copy:
+        dest: "{{ memoir_bot_state_dir }}/.env.example"
+        owner: root
+        group: root
+        mode: "0640"
+        content: |
+          BOT_TOKEN=123456:telegram-token
+          ALLOWED_USER_IDS=123456789
+          OBSIDIAN_VAULT_PATH=/srv/obsidian/vault
+          DAILY_NOTES_DIR=03 Journal
+          NOTE_PATH_FORMAT=%Y/%m/%d.%m.%y.md
+          MESSAGE_TIME_FORMAT=%H:%M
+          TELEGRAM_PROXY=
+          REMINDER_MIN_HOURS=2
+          REMINDER_MAX_HOURS=3
+          REMINDER_START_HOUR=7
+          REMINDER_END_HOUR=23
+          REMINDER_TIMEZONE=Europe/Moscow
+
+    - name: Copy memoir-bot env file when present locally
+      ansible.builtin.copy:
+        src: "{{ memoir_bot_env_source }}"
+        dest: "{{ memoir_bot_env_file }}"
+        owner: root
+        group: root
+        mode: "0600"
+      when: lookup('ansible.builtin.fileglob', memoir_bot_env_source) | length > 0
+      no_log: true
+
+    - name: Install GitHub SSH private key for memoir-bot
+      ansible.builtin.copy:
+        src: "{{ memoir_bot_ssh_private_key_file }}"
+        dest: "{{ memoir_bot_state_dir }}/ssh/id_ed25519"
+        owner: root
+        group: root
+        mode: "0600"
+      no_log: true
+
+    - name: Install GitHub SSH public key for memoir-bot when present locally
+      ansible.builtin.copy:
+        src: "{{ memoir_bot_ssh_public_key_file }}"
+        dest: "{{ memoir_bot_state_dir }}/ssh/id_ed25519.pub"
+        owner: root
+        group: root
+        mode: "0644"
+      when: lookup('ansible.builtin.fileglob', memoir_bot_ssh_public_key_file) | length > 0
+
+    - name: Scan GitHub SSH host key
+      ansible.builtin.command: ssh-keyscan github.com
+      register: memoir_bot_github_host_key
+      changed_when: false
+
+    - name: Trust GitHub SSH host key for memoir-bot
+      ansible.builtin.known_hosts:
+        path: "{{ memoir_bot_state_dir }}/ssh/known_hosts"
+        name: github.com
+        key: "{{ memoir_bot_github_host_key.stdout }}"
+        state: present
+
+    - name: Clone SecondBrain vault
+      ansible.builtin.git:
+        repo: "{{ memoir_bot_vault_repo }}"
+        dest: "{{ memoir_bot_vault_dir }}"
+        key_file: "{{ memoir_bot_state_dir }}/ssh/id_ed25519"
+        accept_hostkey: true
+        update: true
+        version: main
+      register: memoir_bot_vault_checkout
+
+    - name: Configure memoir-bot vault Git author name
+      ansible.builtin.command: git config user.name memoir-bot
+      args:
+        chdir: "{{ memoir_bot_vault_dir }}"
+      changed_when: false
+
+    - name: Configure memoir-bot vault Git author email
+      ansible.builtin.command: git config user.email memoir-bot@homelab.local
+      args:
+        chdir: "{{ memoir_bot_vault_dir }}"
+      changed_when: false
+
+    - name: Sync memoir-bot source code
+      ansible.posix.synchronize:
+        src: "{{ memoir_bot_source_dir }}"
+        dest: "{{ memoir_bot_app_dir }}/"
+        delete: true
+        rsync_opts:
+          - "--exclude=.env"
+          - "--exclude=.git"
+          - "--exclude=.venv"
+          - "--exclude=__pycache__"
+          - "--exclude=*.pyc"
+      register: memoir_bot_source_sync
+
+    - name: Check if memoir-bot image exists
+      ansible.builtin.command: "docker image inspect {{ memoir_bot_image }}"
+      register: memoir_bot_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Build memoir-bot image
+      ansible.builtin.command: "docker build -t {{ memoir_bot_image }} {{ memoir_bot_app_dir }}"
+      when: memoir_bot_source_sync.changed or memoir_bot_image_inspect.rc != 0
+      register: memoir_bot_image_build
+      changed_when: memoir_bot_image_build.rc == 0
+
+    - name: Install memoir-bot systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/memoir-bot.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Memoir Telegram bot container
+          After=docker.service
+          Requires=docker.service
+          ConditionPathExists={{ memoir_bot_env_file }}
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ memoir_bot_container_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ memoir_bot_container_name }} \
+            --env-file {{ memoir_bot_env_file }} \
+            -v {{ memoir_bot_state_dir }}/vault:/srv/obsidian/vault \
+            -v {{ memoir_bot_state_dir }}/ssh:/root/.ssh:ro \
+            {{ memoir_bot_image }}
+          ExecStop=/usr/bin/docker stop {{ memoir_bot_container_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: memoir_bot_unit
+
+    - name: Reload systemd when memoir-bot unit changes
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: memoir_bot_unit.changed
+
+    - name: Check memoir-bot env file
+      ansible.builtin.stat:
+        path: "{{ memoir_bot_env_file }}"
+      register: memoir_bot_env
+
+    - name: Enable memoir-bot service
+      ansible.builtin.systemd:
+        name: memoir-bot
+        enabled: true
+
+    - name: Start memoir-bot when env file exists
+      ansible.builtin.systemd:
+        name: memoir-bot
+        state: "{{ 'restarted' if memoir_bot_source_sync.changed or memoir_bot_image_build.changed or memoir_bot_unit.changed or memoir_bot_vault_checkout.changed else 'started' }}"
+      when: memoir_bot_env.stat.exists

ansible/playbooks/pve-mihomo.yml

@@ -0,0 +1,332 @@
+---
+- name: Create mihomo LXC on mini-pc
+  hosts: mini-pc
+  gather_facts: false
+  vars:
+    mihomo_vmid: 143
+    mihomo_hostname: mihomo
+    mihomo_ip: 192.168.1.27/24
+    mihomo_gateway: 192.168.1.1
+    mihomo_bridge: vmbr0
+    mihomo_rootfs: local-lvm:8
+    mihomo_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+    mihomo_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+  handlers:
+    - name: restart mihomo lxc
+      ansible.builtin.shell: "pct stop {{ mihomo_vmid }} || true; pct start {{ mihomo_vmid }}"
+      changed_when: true
+  tasks:
+    - name: Check if mihomo LXC exists
+      ansible.builtin.command: "pct config {{ mihomo_vmid }}"
+      register: mihomo_pct_config
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 143
+      ansible.builtin.assert:
+        that:
+          - mihomo_pct_config.rc != 0 or mihomo_existing_hostname == mihomo_hostname
+        fail_msg: VMID 143 already exists and is not the Mihomo container.
+      vars:
+        mihomo_existing_hostname: >-
+          {{ mihomo_pct_config.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Install mihomo LXC SSH public key on PVE host
+      ansible.builtin.copy:
+        dest: /tmp/mihomo-lxc.pub
+        owner: root
+        group: root
+        mode: "0600"
+        content: "{{ lookup('file', mihomo_pubkey_file) }}\n"
+      when: mihomo_pct_config.rc != 0
+
+    - name: Create mihomo LXC
+      ansible.builtin.command: >-
+        pct create {{ mihomo_vmid }} {{ mihomo_ostemplate }}
+        --hostname {{ mihomo_hostname }}
+        --rootfs {{ mihomo_rootfs }}
+        --cores 1
+        --memory 512
+        --swap 512
+        --net0 name=eth0,bridge={{ mihomo_bridge }},gw={{ mihomo_gateway }},ip={{ mihomo_ip }},firewall=1
+        --nameserver 1.1.1.1
+        --unprivileged 1
+        --features nesting=1,keyctl=1
+        --onboot 1
+        --startup order=70
+        --cmode shell
+        --ssh-public-keys /tmp/mihomo-lxc.pub
+      when: mihomo_pct_config.rc != 0
+
+    - name: Start mihomo LXC
+      ansible.builtin.command: "pct start {{ mihomo_vmid }}"
+      register: mihomo_pct_start
+      changed_when: mihomo_pct_start.rc == 0
+      failed_when: mihomo_pct_start.rc not in [0, 255]
+
+    - name: Allow FUSE device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:229 rwm"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Bind mount FUSE device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Allow TUN device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.cgroup2.devices.allow: c 10:200 rwm"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Bind mount TUN device in mihomo LXC config
+      ansible.builtin.lineinfile:
+        path: "/etc/pve/lxc/{{ mihomo_vmid }}.conf"
+        line: "lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file"
+        state: present
+      notify: restart mihomo lxc
+
+    - name: Apply pending LXC config changes
+      ansible.builtin.meta: flush_handlers
+
+    - name: Wait for mihomo SSH through ru-vps
+      ansible.builtin.wait_for_connection:
+        timeout: 120
+      delegate_to: mihomo
+      vars:
+        ansible_become: false
+
+- name: Prepare mihomo runtime host
+  hosts: mihomo
+  gather_facts: true
+  vars:
+    ansible_become: false
+    mihomo_config_dir: /opt/mihomo/config
+    mihomo_ui_dir: /opt/mihomo/ui
+    mihomo_image: metacubex/mihomo:v1.19.29@sha256:5e7bcc5e7a866afcc8b007ef827c9ba773f2f34b6d7311b6d39ed1751f37cfd5
+    mihomo_ui_image: ghcr.io/metacubex/metacubexd:v1.270.6@sha256:156d55be885d4ba6254d840bd781b715c20c00afee6e6c24c76be4cfe5eb89d4
+    mihomo_container_name: mihomo
+    mihomo_ui_container_name: mihomo-ui
+  tasks:
+    - name: Install Docker and mihomo runtime packages
+      ansible.builtin.apt:
+        name:
+          - docker.io
+          - fuse-overlayfs
+          - ca-certificates
+          - curl
+          - git
+        state: present
+        update_cache: true
+
+    - name: Ensure Docker config directory exists
+      ansible.builtin.file:
+        path: /etc/docker
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Docker storage driver for unprivileged LXC
+      ansible.builtin.copy:
+        dest: /etc/docker/daemon.json
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          {
+            "storage-driver": "fuse-overlayfs"
+          }
+      register: docker_daemon_config
+
+    - name: Ensure Docker service is enabled and running
+      ansible.builtin.systemd:
+        name: docker
+        state: "{{ 'restarted' if docker_daemon_config.changed else 'started' }}"
+        enabled: true
+
+    - name: Ensure mihomo directories exist
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+      loop:
+        - /opt/mihomo
+        - "{{ mihomo_config_dir }}"
+        - "{{ mihomo_ui_dir }}"
+
+    - name: Check if the configured Mihomo image is present
+      ansible.builtin.command: "docker image inspect {{ mihomo_image }}"
+      register: mihomo_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Mihomo image
+      ansible.builtin.command: "docker pull {{ mihomo_image }}"
+      when: mihomo_image_inspect.rc != 0
+      register: mihomo_image_pull
+      changed_when: true
+
+    - name: Check if the configured Mihomo UI image is present
+      ansible.builtin.command: "docker image inspect {{ mihomo_ui_image }}"
+      register: mihomo_ui_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Mihomo UI image
+      ansible.builtin.command: "docker pull {{ mihomo_ui_image }}"
+      when: mihomo_ui_image_inspect.rc != 0
+      register: mihomo_ui_image_pull
+      changed_when: true
+
+    - name: Install default mihomo config if missing
+      ansible.builtin.copy:
+        dest: "{{ mihomo_config_dir }}/config.yaml"
+        owner: root
+        group: root
+        mode: "0640"
+        force: false
+        content: |
+          mixed-port: 7890
+          socks-port: 7891
+          allow-lan: true
+          bind-address: 0.0.0.0
+          mode: rule
+          log-level: info
+          external-controller: 0.0.0.0:9090
+
+          dns:
+            enable: true
+            listen: 0.0.0.0:1053
+            enhanced-mode: fake-ip
+            nameserver:
+              - 1.1.1.1
+              - 8.8.8.8
+
+          proxies: []
+          proxy-groups:
+            - name: PROXY
+              type: select
+              proxies:
+                - DIRECT
+
+          rules:
+            - MATCH,DIRECT
+      register: mihomo_config
+
+    - name: Install mihomo systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/mihomo.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Mihomo proxy container
+          After=docker.service
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ mihomo_container_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ mihomo_container_name }} \
+            --pull never \
+            --cap-add NET_ADMIN \
+            --device /dev/net/tun \
+            -p 7890:7890 \
+            -p 7891:7891 \
+            -p 9090:9090 \
+            -v {{ mihomo_config_dir }}:/root/.config/mihomo \
+            {{ mihomo_image }}
+          ExecStop=/usr/bin/docker stop {{ mihomo_container_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: mihomo_unit
+
+    - name: Install mihomo UI systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/mihomo-ui.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Mihomo MetaCubeXD web UI container
+          After=docker.service mihomo.service
+          Requires=docker.service
+
+          [Service]
+          Restart=always
+          RestartSec=10
+          ExecStartPre=-/usr/bin/docker rm -f {{ mihomo_ui_container_name }}
+          ExecStart=/usr/bin/docker run --rm \
+            --name {{ mihomo_ui_container_name }} \
+            --pull never \
+            -p 8080:80 \
+            {{ mihomo_ui_image }}
+          ExecStop=/usr/bin/docker stop {{ mihomo_ui_container_name }}
+
+          [Install]
+          WantedBy=multi-user.target
+      register: mihomo_ui_unit
+
+    - name: Reload systemd when mihomo units change
+      ansible.builtin.systemd:
+        daemon_reload: true
+      when: mihomo_unit.changed or mihomo_ui_unit.changed
+
+    - name: Enable and start mihomo
+      ansible.builtin.systemd:
+        name: mihomo
+        enabled: true
+        state: "{{ 'restarted' if mihomo_config.changed or mihomo_unit.changed or mihomo_image_pull is changed else 'started' }}"
+
+    - name: Enable and start mihomo UI
+      ansible.builtin.systemd:
+        name: mihomo-ui
+        enabled: true
+        state: "{{ 'restarted' if mihomo_ui_unit.changed or mihomo_ui_image_pull is changed else 'started' }}"
+
+    - name: Wait for Mihomo proxy TCP ports
+      ansible.builtin.wait_for:
+        host: 127.0.0.1
+        port: "{{ item }}"
+        state: started
+        timeout: 120
+      loop:
+        - 7890
+        - 7891
+        - 9090
+
+    - name: Check Mihomo UI endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:8080/
+        status_code: 200
+      register: mihomo_ui_health
+      retries: 24
+      delay: 5
+      until: mihomo_ui_health.status == 200
+
+    - name: Check Mihomo controller /version endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:9090/version
+        status_code: 200
+      register: mihomo_controller_health
+      retries: 24
+      delay: 5
+      until: mihomo_controller_health.status == 200

ansible/playbooks/pve-monitoring.yml

@@ -0,0 +1,42 @@
+---
+- name: Create monitoring LXC on cloud-pc
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: false
+  vars:
+    ansible_become: false
+    ansible_python_interpreter: "{{ ansible_playbook_python }}"
+    pve_lxc_vmid: 146
+    pve_lxc_node: cloud-pc
+    pve_lxc_hostname: monitoring
+    pve_lxc_ip: 192.168.1.30/24
+    pve_lxc_gateway: 192.168.1.1
+    pve_lxc_disk: data:24
+    pve_lxc_cores: 2
+    pve_lxc_memory: 4096
+    pve_lxc_swap: 512
+    pve_lxc_startup: order=80
+    pve_lxc_features:
+      - nesting=1
+    pve_lxc_ostemplate: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
+  roles:
+    - role: pve_lxc
+
+- name: Enable Docker keyctl feature for monitoring LXC
+  hosts: cloud-pc
+  gather_facts: false
+  tasks:
+    - name: Read monitoring LXC configuration
+      ansible.builtin.command: pct config 146
+      register: monitoring_lxc_config
+      changed_when: false
+
+    - name: Enable keyctl for Docker in monitoring LXC
+      ansible.builtin.command: pct set 146 --features nesting=1,keyctl=1
+      when: "'keyctl=1' not in monitoring_lxc_config.stdout"
+      register: monitoring_lxc_keyctl
+
+    - name: Restart monitoring LXC after feature change
+      ansible.builtin.command: pct reboot 146
+      when: monitoring_lxc_keyctl.changed

ansible/playbooks/pve-ovpn-mini.yml

@@ -1,4 +1,4 @@
-- name: Create wg-mini LXC on mini-pc
+- name: Create ovpn-mini LXC on mini-pc
   hosts: localhost
   connection: local
   gather_facts: false
@@ -7,19 +7,19 @@
       vars:
         pve_lxc_vmid: 132
         pve_lxc_node: mini-pc
-        pve_lxc_hostname: wg-mini
+        pve_lxc_hostname: ovpn-mini
         pve_lxc_ip: 192.168.1.23/24
         pve_lxc_gateway: 192.168.1.1
         pve_lxc_storage: local-lvm
         pve_lxc_disk: local-lvm:8
         pve_lxc_ostemplate: "{{ lookup('env', 'PVE_LXC_OSTEMPLATE') | default('local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst', true) }}"
 
-- name: Allow TUN device in wg-mini LXC config
+- name: Allow TUN device in ovpn-mini LXC config
   hosts: mini-pc
   gather_facts: false
   become: true
   handlers:
-    - name: restart wg-mini lxc
+    - name: restart ovpn-mini lxc
       ansible.builtin.shell: pct stop 132 || true; pct start 132
       changed_when: true
   tasks:
@@ -28,11 +28,11 @@
         path: /etc/pve/lxc/132.conf
         line: "lxc.cgroup2.devices.allow: c 10:200 rwm"
         state: present
-      notify: restart wg-mini lxc
+      notify: restart ovpn-mini lxc
 
     - name: Bind mount /dev/net/tun
       ansible.builtin.lineinfile:
         path: /etc/pve/lxc/132.conf
         line: "lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file"
         state: present
-      notify: restart wg-mini lxc
+      notify: restart ovpn-mini lxc

ansible/playbooks/pve-vaultwarden.yml

@@ -1,3 +1,27 @@
+---
+- name: Guard Vaultwarden VMID before API updates
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 140 configuration
+      ansible.builtin.command: pct config 140
+      register: vaultwarden_existing_vmid
+      changed_when: false
+      failed_when: false
+
+    - name: Refuse to modify a foreign VMID 140
+      ansible.builtin.assert:
+        that:
+          - vaultwarden_existing_vmid.rc != 0 or vaultwarden_existing_hostname == 'vaultwarden'
+        fail_msg: VMID 140 already exists and is not the Vaultwarden container.
+      vars:
+        vaultwarden_existing_hostname: >-
+          {{ vaultwarden_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
 - name: Create Vaultwarden LXC on mini-pc
   hosts: mini-pc
   gather_facts: false
@@ -84,7 +108,7 @@
   vars:
     ansible_become: false
     vaultwarden_data_dir: /opt/vaultwarden/data
-    vaultwarden_image: vaultwarden/server:latest
+    vaultwarden_image: vaultwarden/server:1.37.1@sha256:e9efdf001bf0d68c21f2cbfb8e1d9b5961a7ca9c85e0a7e58bf51a13b997d744
     vaultwarden_container_name: vaultwarden
     vaultwarden_http_port: 80
   tasks:
@@ -133,6 +157,18 @@
         group: root
         mode: "0750"
 
+    - name: Check if the configured Vaultwarden image is present
+      ansible.builtin.command: "docker image inspect {{ vaultwarden_image }}"
+      register: vaultwarden_image_inspect
+      changed_when: false
+      failed_when: false
+
+    - name: Pull the configured Vaultwarden image
+      ansible.builtin.command: "docker pull {{ vaultwarden_image }}"
+      when: vaultwarden_image_inspect.rc != 0
+      register: vaultwarden_image_pull
+      changed_when: true
+
     - name: Install Vaultwarden systemd unit
       ansible.builtin.copy:
         dest: /etc/systemd/system/vaultwarden.service
@@ -151,7 +187,7 @@
           ExecStartPre=-/usr/bin/docker rm -f {{ vaultwarden_container_name }}
           ExecStart=/usr/bin/docker run --rm \
             --name {{ vaultwarden_container_name }} \
-            --pull always \
+            --pull never \
             -p {{ vaultwarden_http_port }}:80 \
             -v {{ vaultwarden_data_dir }}:/data \
             -e WEBSOCKET_ENABLED=true \
@@ -170,5 +206,20 @@
     - name: Enable and start Vaultwarden
       ansible.builtin.systemd:
         name: vaultwarden
-        state: started
+        state: "{{ 'restarted' if vaultwarden_unit.changed or vaultwarden_image_pull.changed else 'started' }}"
         enabled: true
+
+    - name: Wait for Vaultwarden HTTP endpoint
+      ansible.builtin.uri:
+        url: "http://127.0.0.1:{{ vaultwarden_http_port }}/"
+        status_code: 200
+      register: vaultwarden_health
+      retries: 24
+      delay: 5
+      until: vaultwarden_health.status == 200
+
+    - name: Verify Docker storage driver
+      ansible.builtin.command: docker info --format '{{ "{{" }}.Driver{{ "}}" }}'
+      register: vaultwarden_docker_driver
+      changed_when: false
+      failed_when: vaultwarden_docker_driver.stdout != 'fuse-overlayfs'

ansible/playbooks/reverse-proxy-grimmory.yml

@@ -0,0 +1,88 @@
+---
+- name: Configure Grimmory reverse proxy on ru-vps
+  hosts: ru-vps
+  gather_facts: false
+  vars:
+    caddy_dir: /opt/services/ru-vps/caddy
+    caddyfile_path: /opt/services/ru-vps/caddy/Caddyfile
+    grimmory_domain: books.ada-dev.ru
+    grimmory_upstream: 192.168.1.34:6060
+  tasks:
+    - name: Ensure Caddy service directory exists
+      ansible.builtin.file:
+        path: "{{ caddy_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Configure Grimmory Caddy site
+      ansible.builtin.blockinfile:
+        path: "{{ caddyfile_path }}"
+        create: true
+        owner: root
+        group: root
+        mode: "0644"
+        marker: "# {mark} ANSIBLE MANAGED GRIMMORY SITE"
+        block: |
+          {{ grimmory_domain }} {
+            @grimmory_opds_atom path /api/v1/opds /api/v1/opds/libraries /api/v1/opds/shelves /api/v1/opds/magic-shelves /api/v1/opds/authors /api/v1/opds/series /api/v1/opds/catalog /api/v1/opds/recent /api/v1/opds/surprise
+            handle @grimmory_opds_atom {
+              reverse_proxy {{ grimmory_upstream }} {
+                header_up Accept "application/atom+xml"
+                header_up Accept-Encoding identity
+                transport http {
+                  compression off
+                }
+              }
+            }
+
+            @grimmory_opds_search path /api/v1/opds/search.opds
+            handle @grimmory_opds_search {
+              reverse_proxy {{ grimmory_upstream }} {
+                header_up Accept "application/opensearchdescription+xml"
+                header_up Accept-Encoding identity
+                transport http {
+                  compression off
+                }
+              }
+            }
+
+            @grimmory_device_api path /api/koreader /api/koreader/* /api/v1/opds /api/v1/opds/*
+            handle @grimmory_device_api {
+              reverse_proxy {{ grimmory_upstream }} {
+                header_up Accept-Encoding identity
+                transport http {
+                  compression off
+                }
+              }
+            }
+
+            handle {
+              reverse_proxy {{ grimmory_upstream }}
+            }
+          }
+      register: grimmory_caddy_site
+
+    - name: Validate host Caddy config
+      ansible.builtin.command: caddy validate --config {{ caddyfile_path }}
+      changed_when: false
+
+    - name: Check Caddy container mounted config
+      ansible.builtin.command: docker exec caddy grep -F '@grimmory_device_api path /api/koreader' /etc/caddy/Caddyfile
+      register: grimmory_container_caddyfile
+      changed_when: false
+      failed_when: false
+
+    - name: Restart Caddy when config changed or bind mount is stale
+      ansible.builtin.command: docker restart caddy
+      when: grimmory_caddy_site.changed or grimmory_container_caddyfile.rc != 0
+
+    - name: Validate Caddy container config after restart
+      ansible.builtin.command: docker exec caddy caddy validate --config /etc/caddy/Caddyfile
+      changed_when: false
+
+    - name: Check Grimmory upstream from ru-vps
+      ansible.builtin.uri:
+        url: http://{{ grimmory_upstream }}/api/v1/healthcheck
+        status_code: 200

ansible/playbooks/ru-vps-mihomo-harden.yml

@@ -0,0 +1,280 @@
+---
+- name: Harden live Mihomo on ru-vps
+  hosts: ru-vps
+  gather_facts: false
+  vars:
+    ru_vps_mihomo_harden_confirm: false
+    mihomo_root: /opt/services/ru-vps/mihomo
+    mihomo_config_path: "{{ mihomo_root }}/config/config.yaml"
+    mihomo_compose_path: "{{ mihomo_root }}/docker-compose.yml"
+    mihomo_backup_dir: /var/backups/ru-vps-mihomo
+    mihomo_state_dir: /var/lib/ru-vps-mihomo
+    mihomo_state_path: "{{ mihomo_state_dir }}/auth.json"
+    mihomo_harden_script_path: /usr/local/sbin/ru-vps-mihomo-harden
+    mihomo_harden_script_src: "{{ playbook_dir }}/../files/ru-vps-mihomo-harden.py"
+    mihomo_backup_path: "{{ mihomo_backup_dir }}/config.yaml.{{ lookup('pipe', 'date -u +%Y%m%d%H%M%S') }}"
+  pre_tasks:
+    - name: Require explicit Mihomo hardening confirmation
+      ansible.builtin.assert:
+        that:
+          - ru_vps_mihomo_harden_confirm | bool
+        fail_msg: Run with -e ru_vps_mihomo_harden_confirm=true.
+
+    - name: Preflight Mihomo config file exists
+      ansible.builtin.stat:
+        path: "{{ mihomo_config_path }}"
+      register: mihomo_config_stat
+
+    - name: Preflight Mihomo compose file exists
+      ansible.builtin.stat:
+        path: "{{ mihomo_compose_path }}"
+      register: mihomo_compose_stat
+
+    - name: Preflight Mihomo state file exists
+      ansible.builtin.stat:
+        path: "{{ mihomo_state_path }}"
+      register: mihomo_state_stat
+
+    - name: Refuse to run without live Mihomo config and compose files
+      ansible.builtin.assert:
+        that:
+          - mihomo_config_stat.stat.exists
+          - mihomo_compose_stat.stat.exists
+        fail_msg: Live Mihomo config or compose file is missing.
+
+    - name: Preflight Mihomo container is running
+      ansible.builtin.command:
+        argv:
+          - docker
+          - inspect
+          - -f
+          - '{{ "{{" }}.State.Running{{ "}}" }}'
+          - mihomo
+      register: mihomo_container_state
+      changed_when: false
+      failed_when: mihomo_container_state.stdout.strip() != 'true'
+
+    - name: Preflight PyYAML is available on the target
+      ansible.builtin.command:
+        argv:
+          - python3
+          - -c
+          - import yaml
+      changed_when: false
+
+    - name: Preflight current Mihomo config validates
+      ansible.builtin.command:
+        argv:
+          - docker
+          - compose
+          - -f
+          - "{{ mihomo_compose_path }}"
+          - exec
+          - -T
+          - mihomo
+          - /mihomo
+          - -t
+          - -d
+          - /root/.config/mihomo
+      changed_when: false
+
+    - name: Ensure Mihomo backup directory exists
+      ansible.builtin.file:
+        path: "{{ mihomo_backup_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0700"
+
+    - name: Install root-only Mihomo hardening helper
+      ansible.builtin.copy:
+        src: "{{ mihomo_harden_script_src }}"
+        dest: "{{ mihomo_harden_script_path }}"
+        owner: root
+        group: root
+        mode: "0700"
+
+    - name: Create fresh backup of the live Mihomo config after validation
+      ansible.builtin.copy:
+        src: "{{ mihomo_config_path }}"
+        dest: "{{ mihomo_backup_path }}"
+        remote_src: true
+        owner: root
+        group: root
+        mode: "0600"
+
+  tasks:
+    - block:
+        - name: Apply Mihomo hardening in place
+          ansible.builtin.command:
+            argv:
+              - "{{ mihomo_harden_script_path }}"
+              - apply
+              - --config
+              - "{{ mihomo_config_path }}"
+              - --state
+              - "{{ mihomo_state_path }}"
+          register: mihomo_harden_apply
+          changed_when: (mihomo_harden_apply.stdout | from_json).changed
+          no_log: true
+
+        - name: Validate hardened Mihomo config in the running container
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - exec
+              - -T
+              - mihomo
+              - /mihomo
+              - -t
+              - -d
+              - /root/.config/mihomo
+          changed_when: false
+
+        - name: Recreate Mihomo container after config hardening
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - up
+              - -d
+              - --force-recreate
+              - mihomo
+          changed_when: true
+
+        - name: Wait for hardened Mihomo listeners on loopback
+          ansible.builtin.wait_for:
+            host: 127.0.0.1
+            port: "{{ item }}"
+            state: started
+            timeout: 30
+          loop:
+            - 7890
+            - 7891
+
+        - name: Run authenticated Mihomo SOCKS probe to Telegram
+          ansible.builtin.command:
+            argv:
+              - "{{ mihomo_harden_script_path }}"
+              - probe
+              - --config
+              - "{{ mihomo_config_path }}"
+              - --state
+              - "{{ mihomo_state_path }}"
+          register: mihomo_harden_probe
+          changed_when: false
+          no_log: true
+
+        - name: Check anonymous SOCKS access fails
+          ansible.builtin.command:
+            argv:
+              - curl
+              - --silent
+              - --show-error
+              - --connect-timeout
+              - "5"
+              - --max-time
+              - "15"
+              - --proxy
+              - socks5h://127.0.0.1:7891
+              - https://api.telegram.org
+              - --output
+              - /dev/null
+          register: mihomo_anon_probe
+          changed_when: false
+          failed_when: false
+
+        - name: Remove public Mihomo UFW rules after successful hardening
+          community.general.ufw:
+            rule: allow
+            port: "{{ item.port }}"
+            proto: "{{ item.proto }}"
+            delete: true
+          loop:
+            - { port: 7890, proto: tcp }
+            - { port: 7890, proto: udp }
+            - { port: 7891, proto: tcp }
+            - { port: 7891, proto: udp }
+
+        - name: Verify Mihomo listeners are loopback only
+          ansible.builtin.command:
+            argv:
+              - ss
+              - -H
+              - -ltnp
+              - '( sport = :7890 or sport = :7891 )'
+          register: mihomo_ss
+          changed_when: false
+
+        - name: Confirm hardened Mihomo is bound to loopback only and anonymous access fails
+          ansible.builtin.assert:
+            that:
+              - mihomo_ss.stdout is search('127\\.0\\.0\\.1:7890')
+              - mihomo_ss.stdout is search('127\\.0\\.0\\.1:7891')
+              - mihomo_ss.stdout is not search('0\\.0\\.0\\.0:7890|:::7890|0\\.0\\.0\\.0:7891|:::7891')
+              - mihomo_anon_probe.rc != 0
+            fail_msg: Hardened Mihomo must listen on loopback only and reject anonymous SOCKS access.
+
+      rescue:
+        - name: Restore the live Mihomo config from backup
+          ansible.builtin.copy:
+            src: "{{ mihomo_backup_path }}"
+            dest: "{{ mihomo_config_path }}"
+            remote_src: true
+            owner: root
+            group: root
+            mode: "0640"
+
+        - name: Recreate Mihomo container after rollback
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - up
+              - -d
+              - --force-recreate
+              - mihomo
+          changed_when: true
+
+        - name: Remove newly created Mihomo credentials state after rollback
+          ansible.builtin.file:
+            path: "{{ mihomo_state_path }}"
+            state: absent
+          when: not mihomo_state_stat.stat.exists
+
+        - name: Wait for restored Mihomo listeners on loopback
+          ansible.builtin.wait_for:
+            host: 127.0.0.1
+            port: "{{ item }}"
+            state: started
+            timeout: 30
+          loop:
+            - 7890
+            - 7891
+
+        - name: Validate restored Mihomo config in the running container
+          ansible.builtin.command:
+            argv:
+              - docker
+              - compose
+              - -f
+              - "{{ mihomo_compose_path }}"
+              - exec
+              - -T
+              - mihomo
+              - /mihomo
+              - -t
+              - -d
+              - /root/.config/mihomo
+          changed_when: false
+
+        - name: Fail Mihomo hardening after restoring the backup
+          ansible.builtin.fail:
+            msg: Mihomo hardening failed and the live config was restored from backup.

ansible/playbooks/uptime-kuma.yml

@@ -0,0 +1,6 @@
+---
+- name: Freeze Prometheus monitoring and configure Uptime Kuma
+  hosts: monitoring_server
+  gather_facts: false
+  roles:
+    - role: uptime_kuma

ansible/playbooks/user-ssh-key.yml

@@ -0,0 +1,52 @@
+---
+- name: Install user SSH public key on managed hosts
+  hosts: servers
+  gather_facts: false
+  vars:
+    homelab_user_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+    homelab_user_pubkey: "{{ lookup('file', homelab_user_pubkey_file) }}"
+  tasks:
+    - name: Install user SSH key for connection user
+      ansible.posix.authorized_key:
+        user: "{{ ansible_user }}"
+        key: "{{ homelab_user_pubkey }}"
+        state: present
+
+    - name: Install user SSH key for root
+      ansible.posix.authorized_key:
+        user: root
+        key: "{{ homelab_user_pubkey }}"
+        state: present
+      become: true
+      when: ansible_user != 'root'
+
+- name: Bootstrap user SSH public key into legacy LXC containers via Proxmox
+  hosts: pve_nodes
+  gather_facts: false
+  vars:
+    homelab_user_pubkey_file: ~/.ssh/id_ed25519_homelab.pub
+    homelab_user_pubkey: "{{ lookup('file', homelab_user_pubkey_file) }}"
+    legacy_lxc_key_targets:
+      - node: cloud-pc
+        vmid: 120
+        name: pbs
+  tasks:
+    - name: Install user SSH key for root inside legacy LXC
+      ansible.builtin.shell: |
+        pct exec {{ item.vmid }} -- sh -c '
+          key=$(printf "%s" {{ homelab_user_pubkey | b64encode | quote }} | base64 -d)
+          mkdir -p /root/.ssh
+          chmod 700 /root/.ssh
+          touch /root/.ssh/authorized_keys
+          if grep -qxF "$key" /root/.ssh/authorized_keys; then
+            echo present
+          else
+            printf "%s\n" "$key" >> /root/.ssh/authorized_keys
+            echo added
+          fi
+          chmod 600 /root/.ssh/authorized_keys
+        '
+      loop: "{{ legacy_lxc_key_targets }}"
+      when: item.node == inventory_hostname
+      register: legacy_lxc_key_install
+      changed_when: "'added' in legacy_lxc_key_install.stdout"

ansible/playbooks/vaultwarden-update.yml

@@ -0,0 +1,98 @@
+---
+- name: Create and verify Vaultwarden backup before update
+  hosts: vaultwarden
+  gather_facts: false
+  tasks:
+    - name: Create a fresh Vaultwarden offsite backup
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-restic-offsite-vaultwarden.service
+      changed_when: true
+
+    - name: Run Vaultwarden offsite backup audit
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-vaultwarden.service
+      changed_when: true
+
+- name: Create and verify Vaultwarden PBS backup before update
+  hosts: mini-pc
+  gather_facts: false
+  tasks:
+    - name: Read existing VMID 140 configuration
+      ansible.builtin.command: pct config 140
+      register: vaultwarden_existing_vmid
+      changed_when: false
+
+    - name: Refuse to modify a foreign VMID 140
+      ansible.builtin.assert:
+        that:
+          - vaultwarden_existing_vmid.rc == 0
+          - vaultwarden_existing_hostname == 'vaultwarden'
+        fail_msg: VMID 140 already exists and is not the Vaultwarden container.
+      vars:
+        vaultwarden_existing_hostname: >-
+          {{ vaultwarden_existing_vmid.stdout_lines
+             | select('match', '^hostname: ')
+             | map('regex_replace', '^hostname: ', '')
+             | first
+             | default('') }}
+
+    - name: Check for active Proxmox backup before Vaultwarden PBS backup
+      ansible.builtin.command: pgrep -x vzdump
+      register: vaultwarden_vzdump_preflight
+      changed_when: false
+      failed_when: false
+
+    - name: Require no active Proxmox backup before Vaultwarden PBS backup
+      ansible.builtin.assert:
+        that:
+          - vaultwarden_vzdump_preflight.rc != 0
+        fail_msg: >-
+          A Proxmox backup is already running on mini-pc.
+          Retry after the existing backup completes.
+
+    - name: Create a fresh Vaultwarden PBS backup
+      ansible.builtin.command:
+        argv:
+          - vzdump
+          - "140"
+          - --storage
+          - pbs
+          - --mode
+          - snapshot
+          - --prune-backups
+          - keep-all=1
+          - --exclude-path
+          - /var/lib/docker/fuse-overlayfs/*/merged
+
+    - name: Run current PBS backup audit on mini-pc
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - start
+          - --wait
+          - homelab-backup-audit-pbs.service
+      changed_when: true
+
+- import_playbook: pve-vaultwarden.yml
+
+- name: Verify Vaultwarden public endpoint after update
+  hosts: ru-vps
+  gather_facts: false
+  tasks:
+    - name: Check Vaultwarden HTTPS endpoint
+      ansible.builtin.uri:
+        url: https://pass.ada-dev.ru/
+        status_code: 200
+        return_content: false
+      register: vaultwarden_public_health
+      retries: 24
+      delay: 5
+      until: vaultwarden_public_health.status == 200

ansible/requirements.yml

@@ -4,3 +4,5 @@ collections:
     version: ">=2.0.0"
   - name: community.proxmox
     version: ">=2.0.0"
+  - name: community.general
+    version: ">=10.0.0"

ansible/roles/backup_audit/defaults/main.yml

@@ -0,0 +1,31 @@
+---
+backup_audit_log_file: /var/log/homelab-backup-audit.log
+backup_audit_timer_oncalendar: "*-*-* 06:00:00"
+backup_audit_timer_randomized_delay: 10m
+
+backup_audit_pbs_vmids:
+  - vmid: 132
+    max_age_hours: 48
+  - vmid: 140
+    max_age_hours: 48
+  - vmid: 141
+    max_age_hours: 48
+  - vmid: 142
+    max_age_hours: 48
+  - vmid: 143
+    max_age_hours: 48
+  - vmid: 144
+    max_age_hours: 48
+  - vmid: 145
+    max_age_hours: 48
+  - vmid: 146
+    max_age_hours: 48
+  - vmid: 147
+    max_age_hours: 48
+  - vmid: 149
+    max_age_hours: 48
+  - vmid: 150
+    max_age_hours: 48
+
+backup_audit_restic_profiles: []
+backup_audit_metrics_dir: /var/lib/node_exporter/textfile_collector

ansible/roles/backup_audit/tasks/main.yml

@@ -0,0 +1,146 @@
+---
+- name: Install audit dependencies
+  ansible.builtin.apt:
+    name:
+      - jq
+      - sqlite3
+    state: present
+    update_cache: true
+
+- name: Ensure audit directories exist
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  loop:
+    - /etc/homelab-backup-audit
+    - /var/lib/homelab-backup-audit
+    - "{{ backup_audit_metrics_dir }}"
+
+- name: Install PBS audit script
+  ansible.builtin.template:
+    src: audit-pbs.sh.j2
+    dest: /usr/local/sbin/homelab-backup-audit-pbs
+    owner: root
+    group: root
+    mode: "0755"
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Install restic audit script
+  ansible.builtin.template:
+    src: audit-restic.sh.j2
+    dest: /usr/local/sbin/homelab-backup-audit-restic
+    owner: root
+    group: root
+    mode: "0755"
+  when: backup_audit_type | default('') == 'restic'
+
+- name: Install restic audit env files
+  ansible.builtin.copy:
+    dest: "/etc/homelab-backup-audit/{{ item.name }}.env"
+    owner: root
+    group: root
+    mode: "0600"
+    content: |
+      HOMELAB_AUDIT_PROFILE={{ item.name }}
+      HOMELAB_AUDIT_MAX_AGE_HOURS={{ item.max_age_hours | default(36) }}
+      HOMELAB_AUDIT_SQLITE_NAME={{ item.sqlite_name | default('') }}
+      HOMELAB_AUDIT_EXPECTED_NAME={{ item.expected_name | default('') }}
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'
+  no_log: true
+
+- name: Install PBS audit systemd service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-backup-audit-pbs.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=HomeLab PBS backup audit
+      Wants=network-online.target
+      After=network-online.target
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/local/sbin/homelab-backup-audit-pbs
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Install restic audit systemd services
+  ansible.builtin.copy:
+    dest: "/etc/systemd/system/homelab-backup-audit-{{ item.name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=HomeLab restic offsite backup audit ({{ item.name }})
+      Wants=network-online.target
+      After=network-online.target
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/bin/flock -w 1800 /var/lock/homelab-restic-{{ item.name }}.lock /usr/local/sbin/homelab-backup-audit-restic {{ item.name }}
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'
+
+- name: Install PBS audit systemd timer
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-backup-audit-pbs.timer
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Run HomeLab PBS backup audit
+
+      [Timer]
+      OnCalendar={{ backup_audit_timer_oncalendar }}
+      Persistent=true
+      RandomizedDelaySec={{ backup_audit_timer_randomized_delay }}
+
+      [Install]
+      WantedBy=timers.target
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Install restic audit systemd timers
+  ansible.builtin.copy:
+    dest: "/etc/systemd/system/homelab-backup-audit-{{ item.name }}.timer"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Run HomeLab restic backup audit ({{ item.name }})
+
+      [Timer]
+      OnCalendar={{ backup_audit_timer_oncalendar }}
+      Persistent=true
+      RandomizedDelaySec={{ backup_audit_timer_randomized_delay }}
+
+      [Install]
+      WantedBy=timers.target
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: Enable PBS audit timer
+  ansible.builtin.systemd:
+    name: homelab-backup-audit-pbs.timer
+    enabled: true
+    state: started
+  when: backup_audit_type | default('') == 'pbs'
+
+- name: Enable restic audit timers
+  ansible.builtin.systemd:
+    name: "homelab-backup-audit-{{ item.name }}.timer"
+    enabled: true
+    state: started
+  loop: "{{ backup_audit_restic_profiles }}"
+  when: backup_audit_type | default('') == 'restic'

ansible/roles/backup_audit/templates/audit-pbs.sh.j2

@@ -0,0 +1,70 @@
+#!/bin/sh
+# Managed by Ansible: HomeLab PBS backup audit (L0 freshness)
+set -eu
+
+LOG_FILE="{{ backup_audit_log_file }}"
+METRICS_DIR="{{ backup_audit_metrics_dir }}"
+METRICS_FILE="$METRICS_DIR/homelab_backup_audit_pbs.prom"
+METRICS_TMP=$(mktemp "$METRICS_FILE.XXXXXX")
+STATUS=OK
+NOW_EPOCH=$(date +%s)
+
+trap 'rm -f "$METRICS_TMP"' EXIT
+
+publish_metrics() {
+  chmod 0644 "$METRICS_TMP"
+  mv "$METRICS_TMP" "$METRICS_FILE"
+}
+
+ts() { date '+%Y-%m-%dT%H:%M:%S%z'; }
+log() { echo "[$(ts)] [pbs] $*" | tee -a "$LOG_FILE"; }
+
+get_latest_snapshot() {
+  pvesm list pbs --vmid "$1" 2>/dev/null \
+    | tail -n +2 \
+    | awk '{print $1}' \
+    | sort -t/ -k4 \
+    | tail -1
+}
+
+{% for entry in backup_audit_pbs_vmids %}
+audit_vmid_{{ entry.vmid }}() {
+  vmid={{ entry.vmid }}
+  max_age={{ entry.max_age_hours }}
+  latest=$(get_latest_snapshot "$vmid")
+  if [ -z "$latest" ]; then
+    log "FAIL L0: vmid $vmid — no snapshots in PBS"
+    printf 'homelab_backup_audit_snapshot_age_hours{profile="pbs",vmid="%s"} -1\n' "$vmid" >> "$METRICS_TMP"
+    printf 'homelab_backup_audit_snapshot_success{profile="pbs",vmid="%s"} 0\n' "$vmid" >> "$METRICS_TMP"
+    STATUS=FAIL
+    return
+  fi
+  ts_str=$(printf '%s' "$latest" | sed -n 's#.*/\([0-9T:Z-]*\)$#\1#p')
+  snap_epoch=$(date -d "$ts_str" +%s 2>/dev/null || echo 0)
+  age_hours=$(( (NOW_EPOCH - snap_epoch) / 3600 ))
+  if [ "$age_hours" -gt "$max_age" ]; then
+    log "FAIL L0: vmid $vmid — latest snapshot age ${age_hours}h > ${max_age}h (snapshot: $ts_str)"
+    printf 'homelab_backup_audit_snapshot_success{profile="pbs",vmid="%s"} 0\n' "$vmid" >> "$METRICS_TMP"
+    STATUS=FAIL
+  else
+    log "OK L0: vmid $vmid — latest snapshot age ${age_hours}h"
+    printf 'homelab_backup_audit_snapshot_success{profile="pbs",vmid="%s"} 1\n' "$vmid" >> "$METRICS_TMP"
+  fi
+  printf 'homelab_backup_audit_snapshot_age_hours{profile="pbs",vmid="%s"} %s\n' "$vmid" "$age_hours" >> "$METRICS_TMP"
+}
+audit_vmid_{{ entry.vmid }}
+
+{% endfor %}
+if [ "$STATUS" = "OK" ]; then
+  log "AUDIT PASSED"
+  printf 'homelab_backup_audit_success{profile="pbs"} 1\n' >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="pbs"} %s\n' "$NOW_EPOCH" >> "$METRICS_TMP"
+  publish_metrics
+  exit 0
+else
+  log "AUDIT FAILED"
+  printf 'homelab_backup_audit_success{profile="pbs"} 0\n' >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="pbs"} %s\n' "$NOW_EPOCH" >> "$METRICS_TMP"
+  publish_metrics
+  exit 1
+fi

ansible/roles/backup_audit/templates/audit-restic.sh.j2

@@ -0,0 +1,167 @@
+#!/bin/sh
+# Managed by Ansible: HomeLab restic offsite backup audit (L0+L1+L2)
+set -eu
+
+if [ "$#" -ne 1 ]; then
+  echo "usage: $0 <profile>" >&2
+  exit 64
+fi
+
+PROFILE="$1"
+case "$PROFILE" in
+  ''|*[!A-Za-z0-9_-]*)
+    echo "invalid profile name" >&2
+    exit 64
+    ;;
+esac
+RESTIC_ENV="/etc/homelab-restic/${PROFILE}.env"
+AUDIT_ENV="/etc/homelab-backup-audit/${PROFILE}.env"
+LOG_FILE="{{ backup_audit_log_file }}"
+TMP="/var/lib/homelab-backup-audit/${PROFILE}"
+METRICS_DIR="{{ backup_audit_metrics_dir }}"
+METRICS_FILE="$METRICS_DIR/homelab_backup_audit_${PROFILE}.prom"
+METRICS_TMP=$(mktemp "$METRICS_FILE.XXXXXX")
+STATUS=OK
+
+trap 'rm -f "$METRICS_TMP"' EXIT
+
+publish_metrics() {
+  chmod 0644 "$METRICS_TMP"
+  mv "$METRICS_TMP" "$METRICS_FILE"
+}
+
+for f in "$RESTIC_ENV" "$AUDIT_ENV"; do
+  if [ ! -f "$f" ]; then
+    echo "[$(date '+%Y-%m-%dT%H:%M:%S%z')] [${PROFILE}] FAIL: missing env file $f" | tee -a "$LOG_FILE"
+    printf 'homelab_backup_audit_success{profile="%s"} 0\n' "$PROFILE" > "$METRICS_TMP"
+    printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+    publish_metrics
+    exit 66
+  fi
+done
+
+set -a
+. "$RESTIC_ENV"
+. "$AUDIT_ENV"
+set +a
+
+export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE RCLONE_CONFIG
+
+ts() { date '+%Y-%m-%dT%H:%M:%S%z'; }
+log() { echo "[$(ts)] [${PROFILE}] $*" | tee -a "$LOG_FILE"; }
+
+# ── L0: freshness ──────────────────────────────────────────────────────────
+latest_time=$(restic snapshots --latest 1 --json 2>/dev/null | jq -r '.[0].time // empty')
+if [ -z "$latest_time" ]; then
+  log "FAIL L0: no snapshots found in repository"
+  printf 'homelab_backup_audit_success{profile="%s"} 0\n' "$PROFILE" > "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+  publish_metrics
+  exit 1
+fi
+
+now_epoch=$(date +%s)
+snap_epoch=$(date -d "$latest_time" +%s 2>/dev/null || echo 0)
+age_hours=$(( (now_epoch - snap_epoch) / 3600 ))
+max_age="${HOMELAB_AUDIT_MAX_AGE_HOURS:-36}"
+
+if [ "$age_hours" -gt "$max_age" ]; then
+  log "FAIL L0: latest snapshot age ${age_hours}h > ${max_age}h (snapshot: $latest_time)"
+  STATUS=FAIL
+else
+  log "OK L0: latest snapshot age ${age_hours}h"
+fi
+printf 'homelab_backup_audit_snapshot_age_hours{profile="%s"} %s\n' "$PROFILE" "$age_hours" >> "$METRICS_TMP"
+
+# ── L1: repository integrity ───────────────────────────────────────────────
+if restic check 2>&1 | tee -a "$LOG_FILE"; then
+  log "OK L1: restic check passed"
+  printf 'homelab_backup_audit_level_success{profile="%s",level="l1"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+else
+  log "FAIL L1: restic check failed"
+  printf 'homelab_backup_audit_level_success{profile="%s",level="l1"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+  STATUS=FAIL
+fi
+
+# ── L2: SQLite restore + integrity ─────────────────────────────────────────
+if [ -n "${HOMELAB_AUDIT_SQLITE_NAME:-}" ]; then
+  case "$TMP" in
+    /var/lib/homelab-backup-audit/[A-Za-z0-9_-]*) ;;
+    *)
+      log "FAIL L2: unsafe temporary path"
+      exit 64
+      ;;
+  esac
+  rm -rf "$TMP"
+  mkdir -p "$TMP"
+
+  if restic restore latest --target "$TMP" --include "**/${HOMELAB_AUDIT_SQLITE_NAME}" 2>&1 | tee -a "$LOG_FILE"; then
+    db=$(find "$TMP" -name "$HOMELAB_AUDIT_SQLITE_NAME" -type f | head -1)
+    if [ -z "$db" ] || [ ! -f "$db" ]; then
+      log "FAIL L2: $HOMELAB_AUDIT_SQLITE_NAME not found in restored data"
+      printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+      STATUS=FAIL
+    else
+      check=$(sqlite3 "$db" "PRAGMA integrity_check;" 2>&1)
+      if [ "$check" = "ok" ]; then
+        log "OK L2: SQLite integrity_check ok ($db)"
+        printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+      else
+        log "FAIL L2: SQLite integrity_check: $check"
+        printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+        STATUS=FAIL
+      fi
+    fi
+  else
+    log "FAIL L2: restic restore failed for $HOMELAB_AUDIT_SQLITE_NAME"
+    printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+    STATUS=FAIL
+  fi
+
+  rm -rf "$TMP"
+fi
+
+# ── L2: expected file restore ───────────────────────────────────────────────
+if [ -n "${HOMELAB_AUDIT_EXPECTED_NAME:-}" ]; then
+  case "$TMP" in
+    /var/lib/homelab-backup-audit/[A-Za-z0-9_-]*) ;;
+    *)
+      log "FAIL L2: unsafe temporary path"
+      exit 64
+      ;;
+  esac
+  rm -rf "$TMP"
+  mkdir -p "$TMP"
+
+  if restic restore latest --target "$TMP" --include "**/${HOMELAB_AUDIT_EXPECTED_NAME}" 2>&1 | tee -a "$LOG_FILE"; then
+    expected=$(find "$TMP" -name "$HOMELAB_AUDIT_EXPECTED_NAME" -type f | head -1)
+    if [ -n "$expected" ] && [ -s "$expected" ]; then
+      log "OK L2: restored non-empty expected file ($expected)"
+      printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+    else
+      log "FAIL L2: $HOMELAB_AUDIT_EXPECTED_NAME not found or empty"
+      printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+      STATUS=FAIL
+    fi
+  else
+    log "FAIL L2: restic restore failed for $HOMELAB_AUDIT_EXPECTED_NAME"
+    printf 'homelab_backup_audit_level_success{profile="%s",level="l2"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+    STATUS=FAIL
+  fi
+
+  rm -rf "$TMP"
+fi
+
+if [ "$STATUS" = "OK" ]; then
+  log "AUDIT PASSED"
+  printf 'homelab_backup_audit_success{profile="%s"} 1\n' "$PROFILE" >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+  publish_metrics
+  exit 0
+else
+  log "AUDIT FAILED"
+  printf 'homelab_backup_audit_success{profile="%s"} 0\n' "$PROFILE" >> "$METRICS_TMP"
+  printf 'homelab_backup_audit_timestamp_seconds{profile="%s"} %s\n' "$PROFILE" "$(date +%s)" >> "$METRICS_TMP"
+  publish_metrics
+  exit 1
+fi

ansible/roles/bash_config/templates/homelab.sh.j2

@@ -30,6 +30,23 @@ alias gs='git status --short --branch'
 alias gd='git diff'
 alias gl='git log --oneline --decorate -10'
 
+{% if bash_config_proxy_http_url | default('') | length > 0 %}
+proxy_on() {
+  export http_proxy="{{ bash_config_proxy_http_url }}"
+  export https_proxy="{{ bash_config_proxy_http_url }}"
+  export all_proxy="{{ bash_config_proxy_socks_url }}"
+  export HTTP_PROXY="$http_proxy"
+  export HTTPS_PROXY="$https_proxy"
+  export ALL_PROXY="$all_proxy"
+  export no_proxy="localhost,127.0.0.1,::1,192.168.1.0/24"
+  export NO_PROXY="$no_proxy"
+}
+
+proxy_off() {
+  unset http_proxy https_proxy all_proxy HTTP_PROXY HTTPS_PROXY ALL_PROXY no_proxy NO_PROXY
+}
+{% endif %}
+
 if command -v systemctl >/dev/null 2>&1; then
   alias sctl='systemctl'
   alias jctl='journalctl'

ansible/roles/emergency_access/defaults/main.yml

@@ -0,0 +1,11 @@
+---
+emergency_reverse_user: homelab-rescue
+emergency_reverse_port: 22010
+emergency_reverse_bind_address: 127.0.0.1
+emergency_reverse_tunnel_user: emergency-tunnel
+emergency_control_user: emergency-control
+emergency_control_key_path: /etc/emergency-access/control_ed25519
+emergency_tunnel_state_dir: /var/lib/emergency-reverse-ssh
+emergency_reverse_key_path: /var/lib/emergency-reverse-ssh/reverse_ed25519
+emergency_access_dir: /etc/emergency-access
+emergency_tunnel_ttl: 60m

ansible/roles/emergency_access/handlers/main.yml

@@ -0,0 +1,8 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
+
+- name: Validate and reload SSH
+  ansible.builtin.shell: sshd -t && systemctl reload ssh
+  changed_when: true

ansible/roles/emergency_access/tasks/client.yml

@@ -0,0 +1,134 @@
+---
+- name: Validate emergency access client inputs
+  ansible.builtin.assert:
+    that:
+      - emergency_vps_host_key is match('^ssh-(ed25519|rsa|ecdsa-[^ ]+) [A-Za-z0-9+/=]+( [A-Za-z0-9@._:-]+)?$')
+      - emergency_bot_control_public_key | length > 0
+    fail_msg: Set EMERGENCY_VPS_HOST_KEY and provision the emergency-bot control key first.
+  no_log: true
+
+- name: Install reverse SSH client
+  ansible.builtin.apt:
+    name:
+      - openssh-client
+      - sudo
+    state: present
+    update_cache: true
+
+- name: Create emergency access service accounts
+  ansible.builtin.user:
+    name: "{{ item }}"
+    system: true
+    # SSHD must start the forced command under a valid shell; authorized_keys
+    # still prevents this account from receiving an arbitrary command or PTY.
+    shell: "{{ '/bin/sh' if item == emergency_control_user else '/usr/sbin/nologin' }}"
+    create_home: true
+  loop:
+    - "{{ emergency_reverse_tunnel_user }}"
+    - "{{ emergency_control_user }}"
+
+- name: Create emergency access configuration directory
+  ansible.builtin.file:
+    path: "{{ emergency_access_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Create reverse SSH state directory
+  ansible.builtin.file:
+    path: "{{ emergency_tunnel_state_dir }}"
+    state: directory
+    owner: "{{ emergency_reverse_tunnel_user }}"
+    group: "{{ emergency_reverse_tunnel_user }}"
+    mode: "0700"
+
+- name: Generate dedicated reverse SSH key
+  ansible.builtin.command:
+    cmd: "ssh-keygen -q -t ed25519 -N '' -f {{ emergency_reverse_key_path }}"
+    creates: "{{ emergency_reverse_key_path }}"
+  become_user: "{{ emergency_reverse_tunnel_user }}"
+  no_log: true
+
+- name: Read reverse SSH public key
+  ansible.builtin.slurp:
+    src: "{{ emergency_reverse_key_path }}.pub"
+  register: emergency_reverse_key
+  no_log: true
+
+- name: Store reverse SSH public key for VPS configuration
+  ansible.builtin.set_fact:
+    emergency_reverse_public_key: "{{ emergency_reverse_key.content | b64decode | trim }}"
+  no_log: true
+
+- name: Pin ru-vps SSH host key for the tunnel user
+  ansible.builtin.copy:
+    dest: "{{ emergency_tunnel_state_dir }}/known_hosts"
+    content: "ru-vps-emergency {{ emergency_vps_host_key }}\n"
+    owner: "{{ emergency_reverse_tunnel_user }}"
+    group: "{{ emergency_reverse_tunnel_user }}"
+    mode: "0600"
+  no_log: true
+
+- name: Install reverse SSH systemd unit
+  ansible.builtin.template:
+    src: emergency-reverse-ssh.service.j2
+    dest: /etc/systemd/system/emergency-reverse-ssh.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install emergency tunnel expiry unit
+  ansible.builtin.template:
+    src: emergency-reverse-ssh-expire.service.j2
+    dest: /etc/systemd/system/emergency-reverse-ssh-expire.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install emergency tunnel expiry timer
+  ansible.builtin.template:
+    src: emergency-reverse-ssh.timer.j2
+    dest: /etc/systemd/system/emergency-reverse-ssh.timer
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install restricted emergency control helper
+  ansible.builtin.template:
+    src: emergency-control.j2
+    dest: /usr/local/libexec/emergency-control
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install emergency control sudo policy
+  ansible.builtin.template:
+    src: emergency-control.sudoers.j2
+    dest: /etc/sudoers.d/emergency-control
+    owner: root
+    group: root
+    mode: "0440"
+    validate: visudo -cf %s
+
+- name: Authorize emergency-bot control key with forced command
+  ansible.posix.authorized_key:
+    user: "{{ emergency_control_user }}"
+    key: "command=\"/usr/local/libexec/emergency-control\",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty {{ emergency_bot_control_public_key }}"
+    state: present
+    exclusive: true
+  no_log: true
+
+- name: Reload systemd before managing emergency units
+  ansible.builtin.meta: flush_handlers
+
+- name: Keep emergency tunnel disabled at boot
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    enabled: false
+  loop:
+    - emergency-reverse-ssh.service
+    - emergency-reverse-ssh.timer

ansible/roles/emergency_access/tasks/endpoint.yml

@@ -0,0 +1,24 @@
+---
+- name: Validate emergency access endpoint input
+  ansible.builtin.assert:
+    that:
+      - emergency_reverse_public_key | length > 0
+    fail_msg: Configure the mini-pc reverse SSH client before ru-vps.
+  no_log: true
+
+- name: Create restricted reverse SSH endpoint user
+  ansible.builtin.user:
+    name: "{{ emergency_reverse_user }}"
+    system: true
+    # Remote forwarding is accepted before a session exists. A valid shell is
+    # required by SSHD; the forced command below rejects every session.
+    shell: /bin/sh
+    create_home: true
+
+- name: Authorize only the dedicated reverse SSH key
+  ansible.posix.authorized_key:
+    user: "{{ emergency_reverse_user }}"
+    key: "command=\"/usr/bin/false\",restrict,port-forwarding,permitlisten=\"{{ emergency_reverse_bind_address }}:{{ emergency_reverse_port }}\" {{ emergency_reverse_public_key }}"
+    state: present
+    exclusive: true
+  no_log: true

ansible/roles/emergency_access/templates/emergency-control.j2

@@ -0,0 +1,27 @@
+#!/bin/sh
+set -eu
+
+case "${SSH_ORIGINAL_COMMAND:-}" in
+  start)
+    /usr/bin/sudo /usr/bin/systemctl start emergency-reverse-ssh.service
+    /usr/bin/sudo /usr/bin/systemctl restart emergency-reverse-ssh.timer
+    /usr/bin/systemctl is-active --quiet emergency-reverse-ssh.service
+    printf 'started; expires in {{ emergency_tunnel_ttl }}\nConnect:\nssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -J vps -p 22010 ansible@127.0.0.1\n'
+    ;;
+  stop)
+    /usr/bin/sudo /usr/bin/systemctl stop emergency-reverse-ssh.timer
+    /usr/bin/sudo /usr/bin/systemctl stop emergency-reverse-ssh.service
+    printf 'stopped\n'
+    ;;
+  status)
+    if /usr/bin/systemctl is-active --quiet emergency-reverse-ssh.service; then
+      /usr/bin/systemctl show --property=ActiveState --property=ActiveEnterTimestamp --value emergency-reverse-ssh.service
+    else
+      printf 'stopped\n'
+    fi
+    ;;
+  *)
+    printf 'unsupported command\n' >&2
+    exit 64
+    ;;
+esac

ansible/roles/emergency_access/templates/emergency-control.sudoers.j2

@@ -0,0 +1 @@
+{{ emergency_control_user }} ALL=(root) NOPASSWD: /usr/bin/systemctl start emergency-reverse-ssh.service, /usr/bin/systemctl restart emergency-reverse-ssh.timer, /usr/bin/systemctl stop emergency-reverse-ssh.timer, /usr/bin/systemctl stop emergency-reverse-ssh.service

ansible/roles/emergency_access/templates/emergency-reverse-ssh-expire.service.j2

@@ -0,0 +1,6 @@
+[Unit]
+Description=Close expired HomeLab reverse SSH rescue tunnel
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/systemctl stop emergency-reverse-ssh.service

ansible/roles/emergency_access/templates/emergency-reverse-ssh.service.j2

@@ -0,0 +1,19 @@
+[Unit]
+Description=Temporary HomeLab reverse SSH rescue tunnel
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User={{ emergency_reverse_tunnel_user }}
+ExecStart=/usr/bin/ssh -N -i {{ emergency_reverse_key_path }} -o BatchMode=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o StrictHostKeyChecking=yes -o UserKnownHostsFile={{ emergency_tunnel_state_dir }}/known_hosts -o HostKeyAlias=ru-vps-emergency -p 3422 -R {{ emergency_reverse_bind_address }}:{{ emergency_reverse_port }}:127.0.0.1:22 {{ emergency_reverse_user }}@157.22.231.198
+Restart=on-failure
+RestartSec=10
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=strict
+ProtectHome=yes
+ReadWritePaths={{ emergency_tunnel_state_dir }}
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/emergency_access/templates/emergency-reverse-ssh.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Expire HomeLab reverse SSH rescue tunnel after {{ emergency_tunnel_ttl }}
+
+[Timer]
+OnActiveSec={{ emergency_tunnel_ttl }}
+AccuracySec=1s
+Unit=emergency-reverse-ssh-expire.service
+
+[Install]
+WantedBy=timers.target

ansible/roles/emergency_bot/defaults/main.yml

@@ -0,0 +1,8 @@
+---
+emergency_bot_user: emergency-bot
+emergency_bot_state_dir: /var/lib/emergency-bot
+emergency_bot_config_dir: /etc/emergency-bot
+emergency_bot_control_key_path: /var/lib/emergency-bot/control_ed25519
+emergency_bot_mini_pc_host: 192.168.1.10
+emergency_bot_mini_pc_user: emergency-control
+emergency_telegram_proxy: http://192.168.1.27:7890

ansible/roles/emergency_bot/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
+
+- name: Restart emergency bot
+  ansible.builtin.systemd_service:
+    name: emergency-bot.service
+    state: restarted

ansible/roles/emergency_bot/tasks/bootstrap.yml

@@ -0,0 +1,48 @@
+---
+- name: Create emergency bot user
+  ansible.builtin.user:
+    name: "{{ emergency_bot_user }}"
+    system: true
+    shell: /usr/sbin/nologin
+    create_home: false
+
+- name: Create emergency bot state directory
+  ansible.builtin.file:
+    path: "{{ emergency_bot_state_dir }}"
+    state: directory
+    owner: "{{ emergency_bot_user }}"
+    group: "{{ emergency_bot_user }}"
+    mode: "0700"
+
+- name: Generate emergency bot control key
+  ansible.builtin.command:
+    cmd: "ssh-keygen -q -t ed25519 -N '' -f {{ emergency_bot_control_key_path }}"
+    creates: "{{ emergency_bot_control_key_path }}"
+  become: true
+  become_user: "{{ emergency_bot_user }}"
+  no_log: true
+
+- name: Set emergency bot control key ownership
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: file
+    owner: "{{ emergency_bot_user }}"
+    group: "{{ emergency_bot_user }}"
+    mode: "{{ item.mode }}"
+  loop:
+    - path: "{{ emergency_bot_control_key_path }}"
+      mode: "0600"
+    - path: "{{ emergency_bot_control_key_path }}.pub"
+      mode: "0644"
+  no_log: true
+
+- name: Read emergency bot control public key
+  ansible.builtin.slurp:
+    src: "{{ emergency_bot_control_key_path }}.pub"
+  register: emergency_bot_control_key
+  no_log: true
+
+- name: Store emergency bot control public key for mini-pc configuration
+  ansible.builtin.set_fact:
+    emergency_bot_control_public_key: "{{ emergency_bot_control_key.content | b64decode | trim }}"
+  no_log: true

ansible/roles/emergency_bot/tasks/main.yml

@@ -0,0 +1,77 @@
+---
+- name: Bootstrap emergency bot control identity
+  ansible.builtin.import_tasks: bootstrap.yml
+
+- name: Validate emergency bot inputs
+  ansible.builtin.assert:
+    that:
+      - emergency_bot_token is match('^[0-9]+:[A-Za-z0-9_-]+$')
+      - emergency_bot_allowed_user_ids is match('^[0-9]+(,[0-9]+)*$')
+      - emergency_mini_pc_host_key is match('^ssh-(ed25519|rsa|ecdsa-[^ ]+) [A-Za-z0-9+/=]+( [A-Za-z0-9@._:-]+)?$')
+    fail_msg: Set EMERGENCY_BOT_TOKEN, EMERGENCY_ALLOWED_USER_IDS and EMERGENCY_MINI_PC_HOST_KEY.
+  no_log: true
+
+- name: Install emergency bot dependencies
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - openssh-client
+      - python3
+    state: present
+    update_cache: true
+
+- name: Create emergency bot configuration directory
+  ansible.builtin.file:
+    path: "{{ emergency_bot_config_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Pin mini-pc SSH host key
+  ansible.builtin.copy:
+    dest: "{{ emergency_bot_state_dir }}/known_hosts"
+    content: "mini-pc-emergency {{ emergency_mini_pc_host_key }}\n"
+    owner: "{{ emergency_bot_user }}"
+    group: "{{ emergency_bot_user }}"
+    mode: "0600"
+  no_log: true
+
+- name: Install emergency bot runtime
+  ansible.builtin.template:
+    src: emergency_bot.py.j2
+    dest: /usr/local/libexec/emergency-bot
+    owner: root
+    group: root
+    mode: "0755"
+  notify: Restart emergency bot
+
+- name: Install emergency bot environment
+  ansible.builtin.template:
+    src: emergency-bot.env.j2
+    dest: "{{ emergency_bot_config_dir }}/bot.env"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: Restart emergency bot
+
+- name: Install emergency bot systemd unit
+  ansible.builtin.template:
+    src: emergency-bot.service.j2
+    dest: /etc/systemd/system/emergency-bot.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify:
+    - Reload systemd
+    - Restart emergency bot
+
+- name: Enable and start emergency bot
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start emergency bot
+  ansible.builtin.systemd_service:
+    name: emergency-bot.service
+    enabled: true
+    state: started

ansible/roles/emergency_bot/templates/emergency-bot.env.j2

@@ -0,0 +1,7 @@
+EMERGENCY_BOT_TOKEN={{ emergency_bot_token }}
+EMERGENCY_ALLOWED_USER_IDS={{ emergency_bot_allowed_user_ids }}
+EMERGENCY_MINI_PC_HOST={{ emergency_bot_mini_pc_host }}
+EMERGENCY_MINI_PC_USER={{ emergency_bot_mini_pc_user }}
+EMERGENCY_CONTROL_KEY={{ emergency_bot_control_key_path }}
+EMERGENCY_KNOWN_HOSTS={{ emergency_bot_state_dir }}/known_hosts
+EMERGENCY_TELEGRAM_PROXY={{ emergency_telegram_proxy }}

ansible/roles/emergency_bot/templates/emergency-bot.service.j2

@@ -0,0 +1,20 @@
+[Unit]
+Description=HomeLab emergency access Telegram bot
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User={{ emergency_bot_user }}
+EnvironmentFile={{ emergency_bot_config_dir }}/bot.env
+ExecStart=/usr/local/libexec/emergency-bot
+Restart=always
+RestartSec=10
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=strict
+ProtectHome=yes
+ReadWritePaths={{ emergency_bot_state_dir }}
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/emergency_bot/templates/emergency_bot.py.j2

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+import json
+import os
+import subprocess
+import sys
+import time
+import urllib.parse
+import urllib.request
+
+API = "https://api.telegram.org/bot" + os.environ["EMERGENCY_BOT_TOKEN"]
+ALLOWED_USERS = {item.strip() for item in os.environ["EMERGENCY_ALLOWED_USER_IDS"].split(",") if item.strip()}
+STATE_FILE = "{{ emergency_bot_state_dir }}/updates.json"
+BUTTONS = {
+    "inline_keyboard": [[
+        {"text": "Enable SSH", "callback_data": "start"},
+        {"text": "Status", "callback_data": "status"},
+        {"text": "Stop", "callback_data": "stop"},
+    ]]
+}
+
+
+HTTP = urllib.request.build_opener(
+    urllib.request.ProxyHandler({"https": os.environ["EMERGENCY_TELEGRAM_PROXY"]})
+)
+
+
+def api(method, payload=None, timeout=35):
+    if payload is not None:
+        payload = {
+            key: json.dumps(value) if isinstance(value, (dict, list)) else value
+            for key, value in payload.items()
+        }
+    data = None if payload is None else urllib.parse.urlencode(payload).encode()
+    with HTTP.open(API + "/" + method, data=data, timeout=timeout) as response:
+        result = json.load(response)
+    if not result.get("ok"):
+        raise RuntimeError(result.get("description", "Telegram API request failed"))
+    return result["result"]
+
+
+def control(command):
+    args = [
+        "/usr/bin/ssh", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=yes",
+        "-o", "HostKeyAlias=mini-pc-emergency", "-o", "UserKnownHostsFile=" + os.environ["EMERGENCY_KNOWN_HOSTS"],
+        "-i", os.environ["EMERGENCY_CONTROL_KEY"],
+        os.environ["EMERGENCY_MINI_PC_USER"] + "@" + os.environ["EMERGENCY_MINI_PC_HOST"], command,
+    ]
+    result = subprocess.run(args, capture_output=True, text=True, timeout=25, check=False)
+    output = (result.stdout or result.stderr).strip()
+    if result.returncode:
+        return "Control command failed: " + (output or "unknown error")[:1000]
+    return output or "ok"
+
+
+def reply(chat_id, text):
+    payload = {"chat_id": chat_id, "text": text[:3500], "reply_markup": BUTTONS}
+    if text.startswith("started; expires in") and "\nConnect:\n" in text:
+        status, command = text.split("\nConnect:\n", 1)
+        payload["text"] = status + "\nConnect:\n`" + command + "`"
+        payload["parse_mode"] = "Markdown"
+    api("sendMessage", payload)
+
+
+def allowed(chat, sender):
+    return chat.get("type") == "private" and str(sender.get("id")) in ALLOWED_USERS
+
+
+def handle_command(chat_id, command):
+    if command == "help":
+        reply(chat_id, "Choose an action or use /emergency ssh, /emergency status, /emergency stop")
+    elif command:
+        print("accepted emergency command: " + command, file=sys.stderr, flush=True)
+        reply(chat_id, control(command))
+
+
+def load_offset():
+    try:
+        with open(STATE_FILE, encoding="utf-8") as state_file:
+            return int(json.load(state_file).get("offset", 0))
+    except (FileNotFoundError, ValueError, json.JSONDecodeError):
+        return 0
+
+
+def save_offset(offset):
+    temporary = STATE_FILE + ".tmp"
+    with open(temporary, "w", encoding="utf-8") as state_file:
+        json.dump({"offset": offset}, state_file)
+    os.replace(temporary, STATE_FILE)
+
+
+def command_for(text):
+    parts = text.strip().split()
+    if not parts:
+        return None
+    command_name = parts[0].split("@", 1)[0]
+    if command_name == "/start":
+        return "help"
+    if command_name != "/emergency":
+        return None
+    if len(parts) != 2:
+        return "help"
+    return {"ssh": "start", "stop": "stop", "status": "status"}.get(parts[1], "help")
+
+
+def run():
+    offset = load_offset()
+    while True:
+        try:
+            for update in api("getUpdates", {"offset": offset, "timeout": 30}, timeout=40):
+                offset = update["update_id"] + 1
+                save_offset(offset)
+                callback = update.get("callback_query")
+                if callback:
+                    callback_message = callback.get("message", {})
+                    chat = callback_message.get("chat", {})
+                    sender = callback.get("from", {})
+                    command = callback.get("data")
+                    if not allowed(chat, sender) or command not in {"start", "status", "stop"}:
+                        print("ignored callback: chat_type=" + str(chat.get("type")) + " sender=" + str(sender.get("id")), file=sys.stderr, flush=True)
+                        api("answerCallbackQuery", {"callback_query_id": callback["id"], "text": "Not authorized", "show_alert": True})
+                    else:
+                        api("answerCallbackQuery", {"callback_query_id": callback["id"]})
+                        handle_command(chat["id"], command)
+                    continue
+                message = update.get("message", {})
+                chat = message.get("chat", {})
+                sender = message.get("from", {})
+                text = message.get("text", "")
+                if not allowed(chat, sender):
+                    print("ignored update: chat_type=" + str(chat.get("type")) + " sender=" + str(sender.get("id")), file=sys.stderr, flush=True)
+                    continue
+                command = command_for(text)
+                handle_command(chat["id"], command)
+        except Exception as error:
+            print("emergency-bot error: " + str(error), file=sys.stderr, flush=True)
+            time.sleep(10)
+
+
+if __name__ == "__main__":
+    run()

ansible/roles/gyro/defaults/main.yml

@@ -0,0 +1,28 @@
+---
+gyro_user: gyro
+gyro_group: gyro
+gyro_home: /home/gyro
+gyro_app_dir: /opt/gyro/app
+gyro_config_dir: /etc/gyro
+gyro_cache_dir: /var/cache/gyro
+gyro_uv_venv: /opt/uv
+gyro_uv_version: 0.12.5
+gyro_python_min_version: "3.13"
+gyro_timezone: Europe/Moscow
+
+gyro_repo_url: ""
+gyro_repo_version: main
+gyro_git_known_hosts_name: ""
+gyro_git_host_key: ""
+gyro_deploy_enabled: false
+
+gyro_tinvest_token: ""
+gyro_tinvest_account_id: ""
+gyro_telegram_bot_token: ""
+gyro_telegram_user_id: ""
+gyro_telegram_proxy: ""
+gyro_dry_run_override: "true"
+gyro_secrets_configured: false
+
+gyro_timer_enabled: false
+gyro_timer_on_calendar: "Mon..Fri *-*-* 11:00:00 Europe/Moscow"

ansible/roles/gyro/files/gyro-failure-notify.py

@@ -0,0 +1,37 @@
+#!/usr/bin/python3
+import os
+import socket
+import sys
+import urllib.parse
+import urllib.request
+
+
+def main() -> int:
+    token = os.environ.get("TELEGRAM_BOT_TOKEN", "").strip()
+    user_id = os.environ.get("TELEGRAM_USER_ID", "").strip()
+    if not token or not user_id:
+        print("Gyro failure notification skipped: Telegram credentials are absent")
+        return 1
+
+    proxy = os.environ.get("TELEGRAM_PROXY", "").strip()
+    handlers = [urllib.request.ProxyHandler({"http": proxy, "https": proxy})] if proxy else []
+    opener = urllib.request.build_opener(*handlers)
+    failed_unit = sys.argv[1] if len(sys.argv) > 1 else "gyro.service"
+    message = f"Gyro job failed on {socket.gethostname()}: {failed_unit}. Check journalctl -u gyro.service."
+    body = urllib.parse.urlencode({"chat_id": user_id, "text": message}).encode()
+    request = urllib.request.Request(
+        f"https://api.telegram.org/bot{token}/sendMessage",
+        data=body,
+        method="POST",
+    )
+
+    try:
+        with opener.open(request, timeout=20) as response:
+            return 0 if response.status == 200 else 1
+    except Exception as exc:
+        print(f"Gyro failure notification failed: {type(exc).__name__}")
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ansible/roles/gyro/handlers/main.yml

@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true

ansible/roles/gyro/tasks/main.yml

@@ -0,0 +1,365 @@
+---
+- name: Install Gyro runtime packages
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - git
+      - openssh-client
+      - python3
+      - python3-packaging
+      - python3-venv
+      - sudo
+      - ufw
+    state: present
+    update_cache: true
+
+- name: Configure Gyro timezone
+  community.general.timezone:
+    name: "{{ gyro_timezone }}"
+
+- name: Create Gyro service group
+  ansible.builtin.group:
+    name: "{{ gyro_group }}"
+    system: true
+
+- name: Create Gyro service user
+  ansible.builtin.user:
+    name: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    home: "{{ gyro_home }}"
+    shell: /bin/bash
+    system: true
+    create_home: true
+
+- name: Create Gyro directories
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    owner: "{{ item.owner }}"
+    group: "{{ item.group }}"
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ gyro_app_dir }}", owner: "{{ gyro_user }}", group: "{{ gyro_group }}", mode: "0750" }
+    - { path: "{{ gyro_config_dir }}", owner: root, group: root, mode: "0700" }
+    - { path: "{{ gyro_cache_dir }}", owner: "{{ gyro_user }}", group: "{{ gyro_group }}", mode: "0700" }
+    - { path: "{{ gyro_home }}/.ssh", owner: "{{ gyro_user }}", group: "{{ gyro_group }}", mode: "0700" }
+
+- name: Read system Python version
+  ansible.builtin.command: python3 -c "import platform; print(platform.python_version())"
+  register: gyro_python_version
+  changed_when: false
+
+- name: Require Python 3.13 or newer
+  ansible.builtin.assert:
+    that:
+      - gyro_python_version.stdout is version(gyro_python_min_version, '>=')
+    fail_msg: "Gyro requires Python {{ gyro_python_min_version }} or newer; found {{ gyro_python_version.stdout }}."
+
+- name: Create isolated uv installation environment
+  ansible.builtin.command:
+    argv:
+      - python3
+      - -m
+      - venv
+      - "{{ gyro_uv_venv }}"
+    creates: "{{ gyro_uv_venv }}/bin/pip"
+
+- name: Install pinned uv version
+  ansible.builtin.pip:
+    name: "uv=={{ gyro_uv_version }}"
+    executable: "{{ gyro_uv_venv }}/bin/pip"
+
+- name: Link uv into the system path
+  ansible.builtin.file:
+    src: "{{ gyro_uv_venv }}/bin/uv"
+    dest: /usr/local/bin/uv
+    state: link
+
+- name: Generate Git deploy key on the container
+  ansible.builtin.command:
+    argv:
+      - ssh-keygen
+      - -q
+      - -t
+      - ed25519
+      - -N
+      - ""
+      - -C
+      - gyro@homelab
+      - -f
+      - "{{ gyro_home }}/.ssh/id_ed25519_gitea"
+    creates: "{{ gyro_home }}/.ssh/id_ed25519_gitea"
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+
+- name: Secure Git deploy key ownership
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    owner: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ gyro_home }}/.ssh/id_ed25519_gitea", mode: "0600" }
+    - { path: "{{ gyro_home }}/.ssh/id_ed25519_gitea.pub", mode: "0644" }
+
+- name: Read Git deploy public key
+  ansible.builtin.slurp:
+    src: "{{ gyro_home }}/.ssh/id_ed25519_gitea.pub"
+  register: gyro_deploy_public_key
+
+- name: Show Git deploy public key
+  ansible.builtin.debug:
+    msg: "{{ gyro_deploy_public_key.content | b64decode | trim }}"
+
+- name: Allow SSH from the HomeLab LAN
+  community.general.ufw:
+    rule: allow
+    port: "22"
+    proto: tcp
+    src: "{{ homelab_lan_cidr }}"
+
+- name: Allow SSH from the OpenVPN network
+  community.general.ufw:
+    rule: allow
+    port: "22"
+    proto: tcp
+    src: "{{ openvpn_network_cidr }}"
+
+- name: Remove obsolete outbound Gitea SSH allowance
+  community.general.ufw:
+    rule: allow
+    delete: true
+    direction: out
+    dest: 192.168.1.25
+    port: "2222"
+    proto: tcp
+
+- name: Allow outbound Telegram proxy access
+  community.general.ufw:
+    rule: allow
+    direction: out
+    dest: 192.168.1.27
+    port: "7890"
+    proto: tcp
+
+- name: Deny other outbound HomeLab LAN access
+  community.general.ufw:
+    rule: deny
+    direction: out
+    dest: "{{ homelab_lan_cidr }}"
+
+- name: Enable restrictive Gyro firewall
+  community.general.ufw:
+    state: enabled
+    policy: deny
+    direction: incoming
+
+- name: Mask mount units already provided by unprivileged LXC
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    enabled: false
+    masked: true
+    state: stopped
+  loop:
+    - dev-mqueue.mount
+    - run-lock.mount
+    - tmp.mount
+
+- name: Clear stale failures from masked LXC mount units
+  ansible.builtin.command: >-
+    systemctl reset-failed dev-mqueue.mount run-lock.mount tmp.mount
+  changed_when: false
+
+- name: Create locked placeholder environment file
+  ansible.builtin.copy:
+    dest: "{{ gyro_config_dir }}/gyro.env"
+    owner: root
+    group: root
+    mode: "0600"
+    force: false
+    content: |
+      # Managed by Ansible after gyro_secrets_configured is enabled.
+      DRY_RUN_OVERRIDE=true
+
+- name: Validate deployment inputs
+  ansible.builtin.assert:
+    that:
+      - gyro_repo_url | length > 0
+      - gyro_git_known_hosts_name | length > 0
+      - gyro_git_host_key | length > 0
+    fail_msg: Set the repository URL and verified SSH host key before enabling deployment.
+  when: gyro_deploy_enabled | bool
+
+- name: Remove obsolete Git SSH alias
+  ansible.builtin.file:
+    path: "{{ gyro_home }}/.ssh/config"
+    state: absent
+  when: gyro_deploy_enabled | bool
+
+- name: Remove obsolete Gitea known host
+  ansible.builtin.known_hosts:
+    path: "{{ gyro_home }}/.ssh/known_hosts"
+    name: "[192.168.1.25]:2222"
+    state: absent
+  when: gyro_deploy_enabled | bool
+
+- name: Pin verified Git host key
+  ansible.builtin.known_hosts:
+    path: "{{ gyro_home }}/.ssh/known_hosts"
+    name: "{{ gyro_git_known_hosts_name }}"
+    key: "{{ gyro_git_host_key }}"
+  when: gyro_deploy_enabled | bool
+
+- name: Secure Git known hosts file
+  ansible.builtin.file:
+    path: "{{ gyro_home }}/.ssh/known_hosts"
+    owner: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    mode: "0600"
+  when: gyro_deploy_enabled | bool
+
+- name: Ensure Gyro checkout belongs to the service user
+  ansible.builtin.file:
+    path: "{{ gyro_app_dir }}"
+    owner: "{{ gyro_user }}"
+    group: "{{ gyro_group }}"
+    recurse: true
+  when: gyro_deploy_enabled | bool
+
+- name: Clone Gyro from Git remote
+  ansible.builtin.git:
+    repo: "{{ gyro_repo_url }}"
+    dest: "{{ gyro_app_dir }}"
+    version: "{{ gyro_repo_version }}"
+    key_file: "{{ gyro_home }}/.ssh/id_ed25519_gitea"
+    accept_hostkey: false
+    ssh_opts: >-
+      -o UserKnownHostsFile={{ gyro_home }}/.ssh/known_hosts
+      -o StrictHostKeyChecking=yes
+      -o IdentitiesOnly=yes
+    update: true
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+  when: gyro_deploy_enabled | bool
+
+- name: Synchronize locked Gyro dependencies
+  ansible.builtin.command:
+    argv:
+      - /usr/local/bin/uv
+      - sync
+      - --frozen
+  args:
+    chdir: "{{ gyro_app_dir }}"
+  environment:
+    UV_CACHE_DIR: "{{ gyro_cache_dir }}/uv"
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+  register: gyro_uv_sync
+  changed_when: "'Installed' in gyro_uv_sync.stderr or 'Uninstalled' in gyro_uv_sync.stderr"
+  when: gyro_deploy_enabled | bool
+
+- name: Verify bundled T-Invest CA file
+  ansible.builtin.stat:
+    path: "{{ gyro_app_dir }}/config/certs/russian_ca.pem"
+  register: gyro_ca_bundle
+  when: gyro_deploy_enabled | bool
+
+- name: Require complete Gyro checkout
+  ansible.builtin.assert:
+    that:
+      - gyro_ca_bundle.stat.exists
+      - gyro_ca_bundle.stat.isreg | default(false)
+    fail_msg: The Git checkout does not contain config/certs/russian_ca.pem.
+  when: gyro_deploy_enabled | bool
+
+- name: Run Gyro unit tests
+  ansible.builtin.command:
+    argv:
+      - /usr/local/bin/uv
+      - run
+      - --frozen
+      - --no-sync
+      - python
+      - -m
+      - unittest
+      - discover
+      - -s
+      - tests
+  args:
+    chdir: "{{ gyro_app_dir }}"
+  environment:
+    UV_CACHE_DIR: "{{ gyro_cache_dir }}/uv"
+  become: true
+  become_user: "{{ gyro_user }}"
+  vars:
+    ansible_become: true
+  changed_when: false
+  when: gyro_deploy_enabled | bool
+
+- name: Validate Gyro Vault secrets
+  ansible.builtin.assert:
+    that:
+      - gyro_tinvest_token | length > 0
+      - gyro_tinvest_account_id | length > 0
+      - gyro_telegram_bot_token | length > 0
+      - (gyro_telegram_user_id | string | length) > 0
+      - gyro_dry_run_override in ['true', 'false']
+    fail_msg: Populate and encrypt inventory/host_vars/gyro/vault.yml before enabling secrets.
+  no_log: true
+  when: gyro_secrets_configured | bool
+
+- name: Install Gyro environment file from Vault
+  ansible.builtin.template:
+    src: gyro.env.j2
+    dest: "{{ gyro_config_dir }}/gyro.env"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  when: gyro_secrets_configured | bool
+
+- name: Install Gyro failure notifier
+  ansible.builtin.copy:
+    src: gyro-failure-notify.py
+    dest: /usr/local/libexec/gyro-failure-notify
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install Gyro systemd units
+  ansible.builtin.template:
+    src: "{{ item.src }}"
+    dest: "/etc/systemd/system/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - { src: gyro.service.j2, dest: gyro.service }
+    - { src: gyro.timer.j2, dest: gyro.timer }
+    - { src: gyro-failure@.service.j2, dest: "gyro-failure@.service" }
+  notify: Reload systemd
+
+- name: Apply systemd unit changes
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable Gyro timer only after deployment and secret setup
+  ansible.builtin.systemd:
+    name: gyro.timer
+    enabled: "{{ gyro_timer_ready }}"
+    state: "{{ 'started' if gyro_timer_ready else 'stopped' }}"
+  vars:
+    gyro_timer_ready: "{{ gyro_timer_enabled | bool and gyro_deploy_enabled | bool and gyro_secrets_configured | bool }}"
+
+- name: Verify Gyro unit definitions
+  ansible.builtin.command: >-
+    systemd-analyze verify
+    /etc/systemd/system/gyro.service
+    /etc/systemd/system/gyro.timer
+    /etc/systemd/system/gyro-failure@.service
+  changed_when: false

ansible/roles/gyro/templates/gyro-failure@.service.j2

@@ -0,0 +1,15 @@
+[Unit]
+Description=Notify Telegram about failed Gyro unit %i
+
+[Service]
+Type=oneshot
+User={{ gyro_user }}
+Group={{ gyro_group }}
+EnvironmentFile={{ gyro_config_dir }}/gyro.env
+ExecStart=/usr/local/libexec/gyro-failure-notify %i
+UMask=0077
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=strict
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

ansible/roles/gyro/templates/gyro.env.j2

@@ -0,0 +1,6 @@
+TINVEST_TOKEN={{ gyro_tinvest_token | string | to_json }}
+TINVEST_ACCOUNT_ID={{ gyro_tinvest_account_id | string | to_json }}
+TELEGRAM_BOT_TOKEN={{ gyro_telegram_bot_token | string | to_json }}
+TELEGRAM_USER_ID={{ gyro_telegram_user_id | string | to_json }}
+TELEGRAM_PROXY={{ gyro_telegram_proxy | string | to_json }}
+DRY_RUN_OVERRIDE={{ gyro_dry_run_override | string | to_json }}

ansible/roles/gyro/templates/gyro.service.j2

@@ -0,0 +1,31 @@
+[Unit]
+Description=Gyro investment allocator
+Wants=network-online.target
+After=network-online.target
+OnFailure=gyro-failure@%n.service
+
+[Service]
+Type=oneshot
+User={{ gyro_user }}
+Group={{ gyro_group }}
+WorkingDirectory={{ gyro_app_dir }}
+EnvironmentFile={{ gyro_config_dir }}/gyro.env
+Environment=UV_CACHE_DIR={{ gyro_cache_dir }}/uv
+Environment=PYTHONDONTWRITEBYTECODE=1
+ExecStart=/usr/local/bin/uv run --frozen --no-sync python main.py
+UMask=0077
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectClock=true
+ProtectControlGroups=true
+ProtectHome=true
+ProtectHostname=true
+ProtectKernelLogs=true
+ProtectKernelModules=true
+ProtectKernelTunables=true
+ProtectSystem=strict
+ReadWritePaths={{ gyro_cache_dir }}
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+RestrictNamespaces=true
+LockPersonality=true
+MemoryDenyWriteExecute=true

ansible/roles/gyro/templates/gyro.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run Gyro investment allocator on weekdays
+
+[Timer]
+OnCalendar={{ gyro_timer_on_calendar }}
+Persistent=true
+Unit=gyro.service
+
+[Install]
+WantedBy=timers.target

ansible/roles/monitoring_blackbox/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+monitoring_blackbox_listen_address: 127.0.0.1:9115
+monitoring_pushgateway_url: http://192.168.1.30:9091

ansible/roles/monitoring_blackbox/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: restart blackbox exporter
+  ansible.builtin.systemd:
+    name: prometheus-blackbox-exporter
+    state: restarted

ansible/roles/monitoring_blackbox/tasks/main.yml

@@ -0,0 +1,96 @@
+---
+- name: Install Blackbox Exporter
+  ansible.builtin.apt:
+    name:
+      - curl
+      - prometheus-blackbox-exporter
+    state: present
+    update_cache: true
+
+- name: Install external probe configuration
+  ansible.builtin.template:
+    src: blackbox.yml.j2
+    dest: /etc/prometheus/blackbox.yml
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart blackbox exporter
+
+- name: Create Blackbox Exporter systemd override directory
+  ansible.builtin.file:
+    path: /etc/systemd/system/prometheus-blackbox-exporter.service.d
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Configure Blackbox Exporter listen address
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/prometheus-blackbox-exporter.service.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Service]
+      ExecStart=
+      ExecStart=/usr/bin/prometheus-blackbox-exporter --config.file=/etc/prometheus/blackbox.yml --web.listen-address={{ monitoring_blackbox_listen_address }}
+  notify:
+    - reload systemd
+    - restart blackbox exporter
+
+- name: Install ru-vps metrics push script
+  ansible.builtin.template:
+    src: push-metrics.sh.j2
+    dest: /usr/local/sbin/homelab-monitoring-push
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install ru-vps metrics push service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-monitoring-push.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Push ru-vps monitoring metrics through OpenVPN
+      Wants=network-online.target
+      After=network-online.target
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/local/sbin/homelab-monitoring-push
+  notify: reload systemd
+
+- name: Install ru-vps metrics push timer
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-monitoring-push.timer
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Run ru-vps monitoring metric push every minute
+
+      [Timer]
+      OnBootSec=2m
+      OnUnitActiveSec=1m
+      Persistent=true
+
+      [Install]
+      WantedBy=timers.target
+  notify: reload systemd
+
+- name: Ensure Blackbox Exporter is enabled and running
+  ansible.builtin.systemd:
+    name: prometheus-blackbox-exporter
+    enabled: true
+    state: started
+
+- name: Enable ru-vps metrics push timer
+  ansible.builtin.systemd:
+    name: homelab-monitoring-push.timer
+    daemon_reload: true
+    enabled: true
+    state: started

ansible/roles/monitoring_blackbox/templates/blackbox.yml.j2

@@ -0,0 +1,13 @@
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [200, 301, 302]
+  tcp_connect:
+    prober: tcp
+    timeout: 10s
+  icmp:
+    prober: icmp
+    timeout: 10s

ansible/roles/monitoring_blackbox/templates/push-metrics.sh.j2

@@ -0,0 +1,37 @@
+#!/bin/sh
+# Managed by Ansible. Push local ru-vps metrics through the OpenVPN route.
+set -eu
+
+PUSHGATEWAY_URL="{{ monitoring_pushgateway_url }}"
+NOW=$(date +%s)
+NODE_METRICS=$(mktemp)
+
+trap 'rm -f "$NODE_METRICS"' EXIT
+
+push_stdin() {
+  path="$1"
+  curl --fail --silent --show-error --data-binary @- "$PUSHGATEWAY_URL/metrics/$path"
+}
+
+curl --fail --silent --show-error http://127.0.0.1:9100/metrics > "$NODE_METRICS"
+push_stdin "job/node/instance/ru-vps" < "$NODE_METRICS"
+
+probe() {
+  name="$1"
+  module="$2"
+  target="$3"
+  payload=$(curl --fail --silent --show-error --get \
+    --data-urlencode "module=$module" \
+    --data-urlencode "target=$target" \
+    http://127.0.0.1:9115/probe || printf 'probe_success 0\n')
+  payload=$(printf '%s\nhomelab_vps_probe_timestamp_seconds %s\n' "$payload" "$NOW")
+  printf '%s\n' "$payload" | push_stdin "job/blackbox-vps/probe_name/$name"
+}
+
+probe vaultwarden http_2xx https://pass.ada-dev.ru
+probe gitea http_2xx https://git.ada-dev.ru
+probe grimmory http_2xx https://books.ada-dev.ru/api/v1/healthcheck
+probe ovpn_gateway tcp_connect 10.78.0.2:22
+probe cloud_pve tcp_connect 192.168.1.5:8006
+probe mini_pve tcp_connect 192.168.1.10:8006
+probe pbs tcp_connect 192.168.1.20:8007

ansible/roles/monitoring_exporter/defaults/main.yml

@@ -0,0 +1,6 @@
+---
+monitoring_exporter_install_node: true
+monitoring_exporter_install_smart: false
+monitoring_exporter_textfile_dir: /var/lib/node_exporter/textfile_collector
+monitoring_exporter_node_listen_address: 0.0.0.0:9100
+monitoring_exporter_smartctl_timer: "*:0/15"

ansible/roles/monitoring_exporter/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: restart node exporter
+  ansible.builtin.systemd:
+    name: prometheus-node-exporter
+    state: restarted

ansible/roles/monitoring_exporter/tasks/main.yml

@@ -0,0 +1,121 @@
+---
+- name: Install Node Exporter
+  ansible.builtin.apt:
+    name: prometheus-node-exporter
+    state: present
+  when: monitoring_exporter_install_node
+
+- name: Create Node Exporter textfile directory
+  ansible.builtin.file:
+    path: "{{ monitoring_exporter_textfile_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  when: monitoring_exporter_install_node
+
+- name: Create Node Exporter systemd override directory
+  ansible.builtin.file:
+    path: /etc/systemd/system/prometheus-node-exporter.service.d
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  when: monitoring_exporter_install_node
+
+- name: Configure Node Exporter collectors
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/prometheus-node-exporter.service.d/override.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Service]
+      ExecStart=
+      ExecStart=/usr/bin/prometheus-node-exporter --web.listen-address={{ monitoring_exporter_node_listen_address }} --collector.systemd --collector.textfile.directory={{ monitoring_exporter_textfile_dir }}
+  when: monitoring_exporter_install_node
+  notify:
+    - reload systemd
+    - restart node exporter
+
+- name: Ensure Node Exporter is enabled and running
+  ansible.builtin.systemd:
+    name: prometheus-node-exporter
+    enabled: true
+    state: started
+  when: monitoring_exporter_install_node
+
+- name: Install SMART Exporter dependencies
+  ansible.builtin.apt:
+    name:
+      - jq
+      - smartmontools
+    state: present
+  when: monitoring_exporter_install_smart
+
+- name: Remove obsolete SMART Exporter container unit
+  ansible.builtin.systemd:
+    name: homelab-smartctl-exporter
+    enabled: false
+    state: stopped
+  failed_when: false
+  when: monitoring_exporter_install_smart
+
+- name: Install SMART metric collection script
+  ansible.builtin.template:
+    src: smartctl-metrics.sh.j2
+    dest: /usr/local/sbin/homelab-smartctl-metrics
+    owner: root
+    group: root
+    mode: "0755"
+  when: monitoring_exporter_install_smart
+
+- name: Install SMART metric collection service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-smartctl-metrics.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Publish SMART metrics for Node Exporter
+
+      [Service]
+      Type=oneshot
+      ExecStart=/usr/local/sbin/homelab-smartctl-metrics
+  when: monitoring_exporter_install_smart
+  notify: reload systemd
+
+- name: Install SMART metric collection timer
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-smartctl-metrics.timer
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=Collect SMART metrics every 15 minutes
+
+      [Timer]
+      OnBootSec=2m
+      OnUnitActiveSec=15m
+      Persistent=true
+
+      [Install]
+      WantedBy=multi-user.target
+  when: monitoring_exporter_install_smart
+  notify: reload systemd
+
+- name: Enable SMART metric collection timer
+  ansible.builtin.systemd:
+    name: homelab-smartctl-metrics.timer
+    daemon_reload: true
+    enabled: true
+    state: started
+  when: monitoring_exporter_install_smart
+
+- name: Run initial SMART metric collection
+  ansible.builtin.systemd:
+    name: homelab-smartctl-metrics.service
+    state: started
+  when: monitoring_exporter_install_smart

ansible/roles/monitoring_exporter/templates/smartctl-metrics.sh.j2

@@ -0,0 +1,30 @@
+#!/bin/sh
+# Managed by Ansible. Export SMART health to Node Exporter's textfile collector.
+set -eu
+
+METRICS_DIR="{{ monitoring_exporter_textfile_dir }}"
+METRICS_FILE="$METRICS_DIR/homelab_smartctl.prom"
+METRICS_TMP=$(mktemp "$METRICS_FILE.XXXXXX")
+
+trap 'rm -f "$METRICS_TMP"' EXIT
+
+printf '# HELP homelab_smart_device_healthy SMART overall health (1 healthy, 0 failed)\n' >> "$METRICS_TMP"
+printf '# TYPE homelab_smart_device_healthy gauge\n' >> "$METRICS_TMP"
+
+lsblk -dn -o NAME,TYPE | while read -r name type; do
+  [ "$type" = "disk" ] || continue
+  device="/dev/$name"
+  json=$(smartctl -a -j "$device" 2>/dev/null || true)
+  [ -n "$json" ] || continue
+
+  healthy=$(printf '%s' "$json" | jq -r 'if .smart_status.passed == true then 1 else 0 end')
+  temperature=$(printf '%s' "$json" | jq -r '.temperature.current // .nvme_smart_health_information_log.temperature // empty')
+  nvme_warning=$(printf '%s' "$json" | jq -r '.nvme_smart_health_information_log.critical_warning // empty')
+
+  printf 'homelab_smart_device_healthy{device="%s"} %s\n' "$name" "$healthy" >> "$METRICS_TMP"
+  [ -z "$temperature" ] || printf 'homelab_smart_temperature_celsius{device="%s"} %s\n' "$name" "$temperature" >> "$METRICS_TMP"
+  [ -z "$nvme_warning" ] || printf 'homelab_smart_nvme_critical_warning{device="%s"} %s\n' "$name" "$nvme_warning" >> "$METRICS_TMP"
+done
+
+chmod 0644 "$METRICS_TMP"
+mv "$METRICS_TMP" "$METRICS_FILE"

ansible/roles/monitoring_server/defaults/main.yml

@@ -0,0 +1,14 @@
+---
+monitoring_root: /opt/monitoring
+monitoring_prometheus_retention: 30d
+monitoring_prometheus_retention_size: 8GB
+monitoring_grafana_admin_user: admin
+monitoring_grafana_admin_password: "{{ lookup('env', 'MONITORING_GRAFANA_ADMIN_PASSWORD') }}"
+monitoring_telegram_bot_token: "{{ lookup('env', 'MONITORING_TELEGRAM_BOT_TOKEN') }}"
+monitoring_telegram_chat_id: "{{ lookup('env', 'MONITORING_TELEGRAM_CHAT_ID') }}"
+monitoring_pve_api_user: "{{ lookup('env', 'MONITORING_PVE_API_USER') }}"
+monitoring_pve_api_token_id: "{{ lookup('env', 'MONITORING_PVE_API_TOKEN_ID') }}"
+monitoring_pve_api_token_secret: "{{ lookup('env', 'MONITORING_PVE_API_TOKEN_SECRET') }}"
+monitoring_lan_cidr: 192.168.1.0/24
+monitoring_openvpn_gateway_ip: 192.168.1.23
+monitoring_docker_cidr: 172.16.0.0/12

ansible/roles/monitoring_server/files/dashboards/backups.json

@@ -0,0 +1,26 @@
+{
+  "title": "Backups",
+  "uid": "homelab-backups",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "table",
+      "title": "Audit Result",
+      "gridPos": {"h": 10, "w": 12, "x": 0, "y": 0},
+      "targets": [{"expr": "homelab_backup_audit_success", "format": "table", "instant": true, "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Backup Age (Hours)",
+      "gridPos": {"h": 10, "w": 12, "x": 12, "y": 0},
+      "targets": [{"expr": "homelab_backup_audit_snapshot_age_hours", "legendFormat": "{{profile}} {{vmid}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Audit Levels",
+      "gridPos": {"h": 10, "w": 24, "x": 0, "y": 10},
+      "targets": [{"expr": "homelab_backup_audit_level_success", "legendFormat": "{{profile}} {{level}}", "refId": "A"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/dashboards/homelab-overview.json

@@ -0,0 +1,34 @@
+{
+  "title": "HomeLab Overview",
+  "uid": "homelab-overview",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "stat",
+      "title": "Active Alerts",
+      "gridPos": {"h": 8, "w": 6, "x": 0, "y": 0},
+      "targets": [{"expr": "count(ALERTS{alertstate=\"firing\"})", "refId": "A"}],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "type": "stat",
+      "title": "Successful Probes",
+      "gridPos": {"h": 8, "w": 6, "x": 6, "y": 0},
+      "targets": [{"expr": "sum(probe_success)", "refId": "A"}],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "type": "timeseries",
+      "title": "Filesystem Free Space",
+      "gridPos": {"h": 10, "w": 12, "x": 0, "y": 8},
+      "targets": [{"expr": "node_filesystem_avail_bytes{fstype!~\"tmpfs|overlay\"}", "legendFormat": "{{instance}} {{mountpoint}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Backup Audit Success",
+      "gridPos": {"h": 10, "w": 12, "x": 12, "y": 8},
+      "targets": [{"expr": "homelab_backup_audit_success", "legendFormat": "{{profile}} {{instance}}", "refId": "A"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/dashboards/proxmox-storage.json

@@ -0,0 +1,26 @@
+{
+  "title": "Proxmox and Storage",
+  "uid": "proxmox-storage",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "timeseries",
+      "title": "Node CPU Usage",
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 0},
+      "targets": [{"expr": "1 - avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))", "legendFormat": "{{instance}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Node Memory Available",
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 0},
+      "targets": [{"expr": "node_memory_MemAvailable_bytes", "legendFormat": "{{instance}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Disk Read and Write",
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 9},
+      "targets": [{"expr": "rate(node_disk_read_bytes_total[5m])", "legendFormat": "read {{instance}} {{device}}", "refId": "A"}, {"expr": "rate(node_disk_written_bytes_total[5m])", "legendFormat": "write {{instance}} {{device}}", "refId": "B"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/dashboards/services-network.json

@@ -0,0 +1,26 @@
+{
+  "title": "Services and Network",
+  "uid": "services-network",
+  "schemaVersion": 39,
+  "version": 1,
+  "panels": [
+    {
+      "type": "table",
+      "title": "Probe State",
+      "gridPos": {"h": 10, "w": 12, "x": 0, "y": 0},
+      "targets": [{"expr": "probe_success", "format": "table", "instant": true, "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "Probe Duration",
+      "gridPos": {"h": 10, "w": 12, "x": 12, "y": 0},
+      "targets": [{"expr": "probe_duration_seconds", "legendFormat": "{{job}} {{instance}}", "refId": "A"}]
+    },
+    {
+      "type": "timeseries",
+      "title": "TLS Certificate Remaining",
+      "gridPos": {"h": 10, "w": 24, "x": 0, "y": 10},
+      "targets": [{"expr": "(probe_ssl_earliest_cert_expiry - time()) / 86400", "legendFormat": "{{instance}}", "refId": "A"}]
+    }
+  ]
+}

ansible/roles/monitoring_server/files/grafana-dashboard-provider.yml

@@ -0,0 +1,11 @@
+apiVersion: 1
+
+providers:
+  - name: HomeLab
+    orgId: 1
+    folder: HomeLab
+    type: file
+    disableDeletion: true
+    editable: false
+    options:
+      path: /etc/grafana/dashboards

ansible/roles/monitoring_server/files/grafana-datasource.yml

@@ -0,0 +1,10 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    uid: prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: false

ansible/roles/monitoring_server/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: restart monitoring stack
+  ansible.builtin.systemd:
+    name: homelab-monitoring
+    state: restarted

ansible/roles/monitoring_server/tasks/main.yml

@@ -0,0 +1,233 @@
+---
+- name: Validate monitoring secrets
+  ansible.builtin.assert:
+    that:
+      - monitoring_telegram_bot_token | length > 0
+      - monitoring_telegram_bot_token != 'replace-me'
+      - monitoring_telegram_chat_id | length > 0
+      - monitoring_telegram_chat_id != 'replace-me'
+      - monitoring_pve_api_user | length > 0
+      - monitoring_pve_api_token_id | length > 0
+      - monitoring_pve_api_token_secret | length > 0
+      - monitoring_pve_api_token_secret != 'replace-me'
+      - monitoring_grafana_admin_password | length > 0
+      - monitoring_grafana_admin_password != 'change-me'
+    fail_msg: Load monitoring secrets from ignored ansible/.env or Ansible Vault before running this playbook.
+  no_log: true
+
+- name: Install monitoring runtime packages
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - docker.io
+      - docker-compose
+      - ufw
+    state: present
+    update_cache: true
+
+- name: Ensure Docker is enabled and running
+  ansible.builtin.systemd:
+    name: docker
+    enabled: true
+    state: started
+
+- name: Allow SSH and Grafana access from the LAN
+  community.general.ufw:
+    rule: allow
+    port: "{{ item }}"
+    proto: tcp
+    src: "{{ monitoring_lan_cidr }}"
+  loop:
+    - "22"
+    - "3000"
+
+- name: Allow Pushgateway only from the OpenVPN gateway
+  community.general.ufw:
+    rule: allow
+    port: "9091"
+    proto: tcp
+    src: "{{ monitoring_openvpn_gateway_ip }}"
+
+- name: Allow local Prometheus container to scrape Node Exporter
+  community.general.ufw:
+    rule: allow
+    port: "9100"
+    proto: tcp
+    src: "{{ monitoring_docker_cidr }}"
+
+- name: Enable monitoring LXC firewall
+  community.general.ufw:
+    state: enabled
+    policy: deny
+    direction: incoming
+
+- name: Create monitoring directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  loop:
+    - "{{ monitoring_root }}"
+    - "{{ monitoring_root }}/prometheus"
+    - "{{ monitoring_root }}/alertmanager"
+    - "{{ monitoring_root }}/grafana"
+    - "{{ monitoring_root }}/grafana/provisioning"
+    - "{{ monitoring_root }}/grafana/provisioning/dashboards"
+    - "{{ monitoring_root }}/grafana/provisioning/datasources"
+    - "{{ monitoring_root }}/grafana/dashboards"
+    - "{{ monitoring_root }}/pve-exporter"
+
+- name: Grant Prometheus access to its data directory
+  ansible.builtin.file:
+    path: "{{ monitoring_root }}/prometheus/data"
+    state: directory
+    owner: "65534"
+    group: "65534"
+    mode: "0755"
+
+- name: Install monitoring compose definition
+  ansible.builtin.template:
+    src: compose.yml.j2
+    dest: "{{ monitoring_root }}/compose.yml"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: restart monitoring stack
+
+- name: Install Prometheus configuration
+  ansible.builtin.template:
+    src: prometheus.yml.j2
+    dest: "{{ monitoring_root }}/prometheus/prometheus.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install Prometheus alert rules
+  ansible.builtin.template:
+    src: alerts.yml.j2
+    dest: "{{ monitoring_root }}/prometheus/alerts.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install central Blackbox configuration
+  ansible.builtin.template:
+    src: blackbox.yml.j2
+    dest: "{{ monitoring_root }}/blackbox.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install Alertmanager configuration
+  ansible.builtin.template:
+    src: alertmanager.yml.j2
+    dest: "{{ monitoring_root }}/alertmanager/alertmanager.yml"
+    owner: root
+    group: root
+    mode: "0640"
+  no_log: true
+  notify: restart monitoring stack
+
+- name: Install PVE exporter configuration
+  ansible.builtin.template:
+    src: pve.yml.j2
+    dest: "{{ monitoring_root }}/pve-exporter/pve.yml"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: restart monitoring stack
+
+- name: Provision Grafana datasource
+  ansible.builtin.copy:
+    src: grafana-datasource.yml
+    dest: "{{ monitoring_root }}/grafana/provisioning/datasources/prometheus.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Provision Grafana dashboard provider
+  ansible.builtin.copy:
+    src: grafana-dashboard-provider.yml
+    dest: "{{ monitoring_root }}/grafana/provisioning/dashboards/homelab.yml"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: restart monitoring stack
+
+- name: Install Grafana dashboards
+  ansible.builtin.copy:
+    src: "dashboards/{{ item }}"
+    dest: "{{ monitoring_root }}/grafana/dashboards/{{ item }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - homelab-overview.json
+    - proxmox-storage.json
+    - services-network.json
+    - backups.json
+  notify: restart monitoring stack
+
+- name: Validate Docker Compose configuration
+  ansible.builtin.command: >-
+    docker-compose -f {{ monitoring_root }}/compose.yml config --quiet
+  changed_when: false
+  no_log: true
+
+- name: Validate Prometheus configuration
+  ansible.builtin.command: >-
+    docker run --rm --network none
+    -v {{ monitoring_root }}/prometheus:/etc/prometheus:ro
+    --entrypoint promtool prom/prometheus:v3.2.1
+    check config /etc/prometheus/prometheus.yml
+  changed_when: false
+
+- name: Validate Alertmanager configuration
+  ansible.builtin.command: >-
+    docker run --rm --network none
+    -v {{ monitoring_root }}/alertmanager:/etc/alertmanager:ro
+    --user 0:0
+    --entrypoint amtool prom/alertmanager:v0.28.0
+    check-config /etc/alertmanager/alertmanager.yml
+  changed_when: false
+  no_log: true
+
+- name: Install monitoring systemd unit
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/homelab-monitoring.service
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=HomeLab monitoring stack
+      Requires=docker.service
+      After=docker.service
+
+      [Service]
+      Type=oneshot
+      RemainAfterExit=yes
+      WorkingDirectory={{ monitoring_root }}
+      ExecStart=/usr/bin/docker-compose -f {{ monitoring_root }}/compose.yml up -d --remove-orphans
+      ExecStop=/usr/bin/docker-compose -f {{ monitoring_root }}/compose.yml down
+
+      [Install]
+      WantedBy=multi-user.target
+  notify:
+    - reload systemd
+    - restart monitoring stack
+
+- name: Enable monitoring stack
+  ansible.builtin.systemd:
+    name: homelab-monitoring
+    daemon_reload: true
+    enabled: true
+    state: started

ansible/roles/monitoring_server/templates/alertmanager.yml.j2

@@ -0,0 +1,40 @@
+global:
+  resolve_timeout: 5m
+
+route:
+  receiver: telegram-warning
+  group_by: [alertname, instance]
+  group_wait: 30s
+  group_interval: 10m
+  repeat_interval: 6h
+  routes:
+    - matchers:
+        - severity="critical"
+      receiver: telegram-critical
+      repeat_interval: 2h
+
+inhibit_rules:
+  - source_matchers:
+      - alertname="ExporterDown"
+    target_matchers:
+      - alertname="LanServiceUnavailable"
+    equal: [instance]
+
+receivers:
+  - name: telegram-warning
+    telegram_configs:
+      - bot_token: "{{ monitoring_telegram_bot_token }}"
+        chat_id: {{ monitoring_telegram_chat_id }}
+        send_resolved: true
+        message: |-
+          [{{ "{{" }} .Status | toUpper {{ "}}" }}] {{ "{{" }} .CommonLabels.alertname {{ "}}" }}
+          {{ "{{" }} .CommonAnnotations.summary {{ "}}" }}
+
+  - name: telegram-critical
+    telegram_configs:
+      - bot_token: "{{ monitoring_telegram_bot_token }}"
+        chat_id: {{ monitoring_telegram_chat_id }}
+        send_resolved: true
+        message: |-
+          [{{ "{{" }} .Status | toUpper {{ "}}" }}] CRITICAL: {{ "{{" }} .CommonLabels.alertname {{ "}}" }}
+          {{ "{{" }} .CommonAnnotations.summary {{ "}}" }}

ansible/roles/monitoring_server/templates/alerts.yml.j2

@@ -0,0 +1,134 @@
+groups:
+  - name: homelab-availability
+    rules:
+      - alert: ExporterDown
+        expr: up{job=~"node|smartctl|pve"} == 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Exporter unavailable: {{ '{{' }} $labels.job {{ '}}' }} {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: CriticalServiceUnavailable
+        expr: probe_success{job="blackbox-vps"} == 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "External or VPN check failed: {{ '{{' }} $labels.probe_name {{ '}}' }}"
+
+      - alert: RuVpsMonitoringStale
+        expr: time() - homelab_vps_probe_timestamp_seconds > 5 * 60
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "ru-vps has not pushed monitoring metrics through OpenVPN"
+
+      - alert: RuVpsMonitoringMissing
+        expr: absent(homelab_vps_probe_timestamp_seconds{job="blackbox-vps"})
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          summary: "ru-vps has no monitoring metrics in Pushgateway"
+
+      - alert: LanServiceUnavailable
+        expr: probe_success{job=~"blackbox-lan-http|blackbox-lan-tcp"} == 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "LAN service unavailable: {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: TLSCertificateExpiringSoon
+        expr: (probe_ssl_earliest_cert_expiry - time()) < 14 * 24 * 3600
+        for: 1h
+        labels:
+          severity: warning
+        annotations:
+          summary: "TLS certificate expires within 14 days: {{ '{{' }} $labels.instance {{ '}}' }}"
+
+  - name: homelab-capacity
+    rules:
+      - alert: FilesystemAlmostFull
+        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) < 0.15
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Filesystem is more than 85% full: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.mountpoint {{ '}}' }}"
+
+      - alert: FilesystemWillFillSoon
+        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 7 * 24 * 3600) < 0
+        for: 30m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Filesystem may fill within seven days: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.mountpoint {{ '}}' }}"
+
+      - alert: HostMemoryPressure
+        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.1
+        for: 20m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Memory pressure on {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: SMARTDeviceFailed
+        expr: homelab_smart_device_healthy == 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "SMART health check failed: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.device {{ '}}' }}"
+
+      - alert: NVMECriticalWarning
+        expr: homelab_smart_nvme_critical_warning > 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "NVMe critical warning: {{ '{{' }} $labels.instance {{ '}}' }} {{ '{{' }} $labels.device {{ '}}' }}"
+
+  - name: homelab-backups
+    rules:
+      - alert: BackupAuditFailed
+        expr: homelab_backup_audit_success == 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Backup audit failed: {{ '{{' }} $labels.profile {{ '}}' }} on {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: BackupAuditStale
+        expr: time() - homelab_backup_audit_timestamp_seconds > 48 * 3600
+        for: 15m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Backup audit result is stale: {{ '{{' }} $labels.profile {{ '}}' }} on {{ '{{' }} $labels.instance {{ '}}' }}"
+
+      - alert: BackupAuditMissing
+        expr: absent(homelab_backup_audit_timestamp_seconds{profile="pbs"}) or absent(homelab_backup_audit_timestamp_seconds{profile="gitea"}) or absent(homelab_backup_audit_timestamp_seconds{profile="vaultwarden"}) or absent(homelab_backup_audit_timestamp_seconds{profile="grimmory"})
+        for: 12h
+        labels:
+          severity: critical
+        annotations:
+          summary: "Expected backup audit metric has not been published"
+
+      - alert: ResticBackupTooOld
+        expr: homelab_backup_audit_snapshot_age_hours{profile=~"gitea|vaultwarden|grimmory"} > 36
+        for: 15m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Restic backup is older than 36 hours: {{ '{{' }} $labels.profile {{ '}}' }}"
+
+      - alert: PBSBackupTooOld
+        expr: homelab_backup_audit_snapshot_age_hours{profile="pbs"} > 48
+        for: 15m
+        labels:
+          severity: critical
+        annotations:
+          summary: "PBS backup is older than 48 hours: VMID {{ '{{' }} $labels.vmid {{ '}}' }}"

ansible/roles/monitoring_server/templates/blackbox.yml.j2

@@ -0,0 +1,10 @@
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [200, 301, 302]
+  tcp_connect:
+    prober: tcp
+    timeout: 10s