ac-deploy/monitoring

AC Cluster Monitoring — Deploy Guide

Stack: Prometheus 3.5 · Grafana 12.4 · Alertmanager 0.32 · blackbox_exporter · node_exporter · mysqld_exporter · postgres_exporter
SPEC: ac-docs/specs/20260626_cluster_monitoring.md
DR: ac-docs/DRs/20260626_cluster_monitoring.md
Status: R0 live on cast04 as of 2026-06-26

---

Topology

Node	IP	Role	Exporters
cast04	10.7.16.239	Monitoring host	Prometheus + Grafana + Alertmanager + blackbox + node
cast01	10.7.16.236	App + Gitea + PostgreSQL	node + postgres_exporter
cast02	10.7.16.237	App	node_exporter
cast03	10.7.16.238	App	node_exporter
artc0	10.7.16.128	BE + MariaDB + Docker	node + mysqld_exporter (Docker)

Access

• Grafana (public): https://apps.f0xx.org/app/androidcast_project/monitor/
  Login via existing PHP session (auto-SSO). Redirects to /login if not authenticated.
• Grafana (direct internal): http://10.7.16.239:3000/ (admin/acMonitor2026!)
• Prometheus UI: http://10.7.16.239:9090/ (no auth; internal only)
• Alertmanager UI: http://10.7.16.239:9093/ (no auth; internal only)

Config files (on cast04)

File	Purpose
/etc/prometheus/prometheus.yml	Scrape targets
/etc/prometheus/rules/node_alerts.yml	HW/OS alert rules (10 rules)
/etc/prometheus/rules/service_alerts.yml	Service/HTTP/DB alert rules (10 rules)
/etc/alertmanager/alertmanager.yml	Alert routing (email + Telegram stub)
/etc/grafana/grafana.ini	Grafana config (subpath, auth proxy)
/var/lib/grafana/provisioning/	Auto-provisioned datasource + dashboards dir
/etc/conf.d/grafana	Grafana runtime overrides (bind address, provisioning path)

Config files (on artc0/BE)

File	Purpose
/etc/nginx/conf.d/apps.conf	BE nginx — includes monitoring proxy location
/var/www/.../api/grafana-auth-check.php	PHP session → X-WEBAUTH-USER auth bridge
/etc/mysqld-exporter.cnf	MariaDB exporter credentials

Services management (cast04)

# Status
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
  echo "$svc: $(rc-service $svc status 2>&1 | grep -o 'started\|stopped')"
done

# Restart all
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
  sudo rc-service $svc restart
done

Grafana dashboards (imported)

Dashboard	URL
Node Exporter Full	/d/rYdddlPWk/node-exporter-full
MySQL Exporter Quickstart	/d/...
Prometheus Blackbox Exporter	/d/xtkCtBkiz/prometheus-blackbox-exporter
Node Exporter EN Stats	/d/xfpJB9FGz/...

TODO before production

1. Email alerts: Fill in Gmail app password in /etc/alertmanager/alertmanager.yml
2. Telegram alerts: Add bot_token + chat_id to multi-critical receiver
3. postgres_exporter: Verify cast01 pg_up 1 after next restart
4. UptimeRobot: Create account + add HTTP monitor for apps.f0xx.org/app/androidcast_project/
5. Grafana admin password: Change from default acMonitor2026! after setting up auth proxy

Alert thresholds

Metric	Warning	Critical
CPU	>85% for 10m	>95% for 5m
RAM free	<15% for 5m	<5% for 2m
Disk free	<20% for 15m	<10% for 10m
Load (per vCPU)	>1.5× for 5m	>3.0× for 5m
HTTP probe	>5s for 5m	down >3m
SSL cert	<14 days	<7 days
MariaDB	high connections	down >2m