infra/ansible/README.md

# HomeLab Ansible

Ansible is the control plane for HomeLab infrastructure changes.

## Contract

- The operator keeps passwords, SSH access and network reachability working.
- The agent changes infrastructure only through inventory, roles and playbooks in this directory.
- Manual server changes are allowed only for break-glass recovery; afterwards they must be captured in Ansible.
- Secrets stay outside git. Use `.env`, vault files or local prompt input, not committed variables.

## Layout

- `inventory/hosts.yml` — canonical host list and host-specific facts.
- `inventory/group_vars/all/services.yml` — service registry and reverse-proxy input; deployment playbooks still duplicate these facts.
- `playbooks/` — entry points for tasks.
- `roles/` — reusable configuration units.
- `Makefile` — canonical manual entry point and safety gates.

## Current Groups

- `ru-vps` — public VPS, JumpHost, qdevice, OpenVPN server.
- `pve_nodes` — Proxmox hosts: `cloud-pc`, `mini-pc`.
- `lxc_infra` — infrastructure LXC containers, including the outbound-only `gyro` investment allocator host.
- `monitoring_server` — monitoring LXC; Prometheus stack is frozen and Uptime Kuma is active.
- `monitoring_exporters` — hosts exposing Node Exporter metrics.
- `monitoring_smart_exporters` — Proxmox nodes exposing SMART metrics.
- `vpn_openvpn` — OpenVPN transport hosts: `ru-vps`, `ovpn-mini`.
- `shell_hosts` — hosts with unified bash config: `ru-vps`, `cloud-pc`, `mini-pc`, `hermes-ai`.
- `servers` — all managed hosts.

## First Checks

From the repository root, use the Nix environment and install Galaxy collections once per clone:

```bash
nix develop
ansible-galaxy collection install -r ansible/requirements.yml -p ansible/collections
```

Run manual operations through Make from `ansible/`:

```bash
make help
make check
make status EXTRA="--limit '!gyro'"
make lint
```

`make setup` remains a local venv fallback when Nix is unavailable.

## Controlled Updates

Service updates are manual. Active update-managed remote images are pinned as
`tag@sha256:digest`; the frozen Prometheus stack is tag-only. Floating
auto-update agents are not used.

Run the dedicated Make target from `ansible/`:

```bash
make update-vaultwarden
make update-gitea
make update-adguard
make update-mihomo
make update-grimmory
```

Update flow is always: fresh backup/audit first, then the update playbook, then health verification.

- Gitea, Vaultwarden and Grimmory use app-aware offsite restic backups/audits before update.
- AdGuard and Mihomo use fresh PBS LXC backups before update.
- `grimmory-update.yml` currently validates the existing pinned release; any future actual app image upgrade must be preceded by release-note and migration review.

## Mihomo Hardening

Use the dedicated hardening playbook only when explicitly approved:

```bash
make mihomo-harden CONFIRM=1
```

It rotates the live Mihomo SOCKS credentials on `ru-vps`, locks the proxy to loopback, and removes the public UFW exposure for ports `7890` and `7891`.

The rotated credentials are not recoverable for clients unless you roll back the saved config backup.

For Proxmox API playbooks, create ignored `.env` in the **repository root** from
`.env.example`. It is shared with OpenTofu (`tofu/`). Make loads it
automatically:

```bash
cp ../.env.example ../.env    # секреты живут в корне репозитория
make env-check
make dry-ovpn-mini
make deploy-ovpn-mini
```

Or bootstrap the token from `mini-pc` with sudo:

```bash
make bootstrap-pve-token
```

Create the separate read-only PVE token used by the monitoring exporter:

```bash
make bootstrap-monitoring-token
```

OpenVPN transport:

```bash
make openvpn
make openvpn-check
```

CT 146 can be provisioned separately. Uptime Kuma is the active monitoring service:

```bash
make deploy-monitoring
make uptime-kuma
```

`pve-monitoring.yml` creates CT `146` (`monitoring`, `192.168.1.30`) on
`cloud-pc`. The older `monitoring.yml` configures Prometheus, Alertmanager,
Grafana and exporters; it is frozen and `make monitoring CONFIRM=1` is reserved
for an explicitly approved restoration decision.

## Gyro Investment Allocator

Production `gyro` now runs in Tofu-provisioned CT `156` (`gyro`, `192.168.1.35`) on `mini-pc`. CT `150` is stopped and kept only as rollback for at least a week; it is not removed.

`make gyro` configures Python 3.13+, pinned `uv`, the `gyro` service user, the container-local GitHub deploy key, restrictive firewall rules, and the weekday systemd timer. Do not use `make deploy-gyro` for the cutover path.

UFW is the currently enforced isolation layer: inbound is denied except SSH from LAN/OpenVPN, and east-west outbound is denied except the Mihomo HTTP proxy. The equivalent CT `156` Proxmox firewall is already carried over; the cluster-wide PVE firewall remains disabled, so do not enable it without auditing every node and guest with `firewall=1`.

The role keeps deployment and the timer disabled by default. The active host vars deploy `git@github.com:ada-dmitry/t_tech-gyro.git` with GitHub's verified ED25519 host key; the timer still requires the ignored Vault file:

```bash
cp inventory/host_vars/gyro/vault.yml.example inventory/host_vars/gyro/vault.yml
ansible-vault encrypt inventory/host_vars/gyro/vault.yml
```

After encrypting the secrets, set `gyro_timer_enabled: true` in `main.yml` and apply with `--ask-vault-pass`. `DRY_RUN_OVERRIDE` remains `true` until real trading is explicitly approved.

```bash
make gyro
```

The timer runs at 11:00 Europe/Moscow from Monday through Friday and uses `OnFailure=` for a best-effort Telegram alert. PBS backup jobs and the backup freshness audit derive the current Gyro VMID from the registry once regenerated, so CT `156` is picked up automatically.

The legacy `pve-gyro.yml` remains available for rollback recovery only and is blocked by default after cutover unless an explicit override is passed.

Uptime Kuma uses the existing monitoring LXC and stops/disables `homelab-monitoring` without deleting its configuration or data. Its UI is available only from the LAN at `http://192.168.1.30:3001`; create monitors and notification settings in the UI.

```bash
make uptime-kuma
```

## Emergency Reverse SSH

`pve-emergency-bot.yml` creates CT `148` (`emergency-bot`, `192.168.1.32`) on `mini-pc`. `emergency-access.yml` configures the bot, a TTL-limited reverse tunnel from `mini-pc` to `ru-vps`, and the restricted SSH identities used between them.

Before applying, set `EMERGENCY_BOT_TOKEN`, `EMERGENCY_ALLOWED_USER_IDS`, `EMERGENCY_VPS_HOST_KEY`, and `EMERGENCY_MINI_PC_HOST_KEY` in ignored `.env` or Ansible Vault. The host-key variables must be verified public host keys, not values obtained during deployment.

```bash
make deploy-emergency-bot
make emergency-access
```

From an authorized private Telegram chat, use the `Enable SSH`, `Status`, and `Stop` buttons or `/emergency ssh`, `/emergency status`, and `/emergency stop`. `/emergency ssh` enables a 60-minute tunnel only; it does not expose a public port. Connect while it is active with:

The bot uses the LAN Mihomo HTTP proxy at `192.168.1.27:7890` because direct Telegram TCP access is unavailable from HomeLab.

```bash
ssh -i ~/.ssh/id_ed25519_homelab_ansible -o IdentitiesOnly=yes -J vps -p 22010 ansible@127.0.0.1
```

The target account is `ansible`; it has no password login. Use the existing private key `~/.ssh/id_ed25519_homelab_ansible`.

Bootstrap the Ansible service account on shell hosts:

```bash
make bootstrap-ansible-user
```

When a task needs privilege escalation:

```bash
make play-<name> EXTRA="-K"
```

## Workflow

1. Describe the desired infrastructure change.
2. Add or update a role/playbook.
3. Run the smallest safe check or playbook.
4. Document non-obvious decisions in the project notes.