# AC Cluster Monitoring — Deploy Guide

**Stack:** Prometheus 3.5 · Grafana 12.4 · Alertmanager 0.32 · blackbox_exporter · node_exporter · mysqld_exporter · postgres_exporter  
**SPEC:** [ac-docs/specs/20260626_cluster_monitoring.md](../../ac-docs/specs/20260626_cluster_monitoring.md)  
**DR:** [ac-docs/DRs/20260626_cluster_monitoring.md](../../ac-docs/DRs/20260626_cluster_monitoring.md)  
**Status:** R0 live on cast04 as of 2026-06-26

---

## Topology

| Node | IP | Role | Exporters |
|------|----|------|-----------|
| cast04 | 10.7.16.239 | Monitoring host | Prometheus + Grafana + Alertmanager + blackbox + node |
| cast01 | 10.7.16.236 | App + Gitea + PostgreSQL | node + postgres_exporter |
| cast02 | 10.7.16.237 | App | node_exporter |
| cast03 | 10.7.16.238 | App | node_exporter |
| artc0 | 10.7.16.128 | BE + MariaDB + Docker | node + mysqld_exporter (Docker) |

## Access

- **Grafana (public):** https://apps.f0xx.org/app/androidcast_project/monitor/  
  Login via existing PHP session (auto-SSO). Redirects to `/login` if not authenticated.
- **Grafana (direct internal):** http://10.7.16.239:3000/ (admin/acMonitor2026!)
- **Prometheus UI:** http://10.7.16.239:9090/ (no auth; internal only)
- **Alertmanager UI:** http://10.7.16.239:9093/ (no auth; internal only)

## Config files (on cast04)

| File | Purpose |
|------|---------|
| `/etc/prometheus/prometheus.yml` | Scrape targets |
| `/etc/prometheus/rules/node_alerts.yml` | HW/OS alert rules (10 rules) |
| `/etc/prometheus/rules/service_alerts.yml` | Service/HTTP/DB alert rules (10 rules) |
| `/etc/alertmanager/alertmanager.yml` | Alert routing (email + Telegram stub) |
| `/etc/grafana/grafana.ini` | Grafana config (subpath, auth proxy) |
| `/var/lib/grafana/provisioning/` | Auto-provisioned datasource + dashboards dir |
| `/etc/conf.d/grafana` | Grafana runtime overrides (bind address, provisioning path) |

## Config files (on artc0/BE)

| File | Purpose |
|------|---------|
| `/etc/nginx/conf.d/apps.conf` | BE nginx — includes monitoring proxy location |
| `/var/www/.../api/grafana-auth-check.php` | PHP session → X-WEBAUTH-USER auth bridge |
| `/etc/mysqld-exporter.cnf` | MariaDB exporter credentials |

## Services management (cast04)

```sh
# Status
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
  echo "$svc: $(rc-service $svc status 2>&1 | grep -o 'started\|stopped')"
done

# Restart all
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
  sudo rc-service $svc restart
done
```

## Grafana dashboards (imported)

| Dashboard | URL |
|-----------|-----|
| Node Exporter Full | `/d/rYdddlPWk/node-exporter-full` |
| MySQL Exporter Quickstart | `/d/...` |
| Prometheus Blackbox Exporter | `/d/xtkCtBkiz/prometheus-blackbox-exporter` |
| Node Exporter EN Stats | `/d/xfpJB9FGz/...` |

## TODO before production

1. **Email alerts:** Fill in Gmail app password in `/etc/alertmanager/alertmanager.yml`
2. **Telegram alerts:** Add `bot_token` + `chat_id` to `multi-critical` receiver
3. **postgres_exporter:** Verify cast01 `pg_up 1` after next restart
4. **UptimeRobot:** Create account + add HTTP monitor for `apps.f0xx.org/app/androidcast_project/`
5. **Grafana admin password:** Change from default `acMonitor2026!` after setting up auth proxy

## Alert thresholds

| Metric | Warning | Critical |
|--------|---------|----------|
| CPU | >85% for 10m | >95% for 5m |
| RAM free | <15% for 5m | <5% for 2m |
| Disk free | <20% for 15m | <10% for 10m |
| Load (per vCPU) | >1.5× for 5m | >3.0× for 5m |
| HTTP probe | >5s for 5m | down >3m |
| SSL cert | <14 days | <7 days |
| MariaDB | high connections | down >2m |