mirror of
git://f0xx.org/ac/ac-deploy
synced 2026-07-29 00:58:20 +03:00
d045653a7b
artc0 VM is off; remove node_exporter target and relabel rules from Prometheus. Point Grafana auth-proxy whitelist and alert-notifier shortener API at cast01–03 cluster instead of 10.7.16.128. Co-authored-by: Cursor <cursoragent@cursor.com>
AC Cluster Monitoring — Deploy Guide
Stack: Prometheus 3.5 · Grafana 12.4 · Alertmanager 0.32 · blackbox_exporter · node_exporter · mysqld_exporter · postgres_exporter
SPEC: ac-docs/specs/20260626_cluster_monitoring.md
DR: ac-docs/DRs/20260626_cluster_monitoring.md
Status: R0 live on cast04 as of 2026-06-26
Topology
| Node | IP | Role | Exporters |
|---|---|---|---|
| cast04 | 10.7.16.239 | Monitoring host | Prometheus + Grafana + Alertmanager + blackbox + node |
| cast01 | 10.7.16.236 | App + Gitea + PostgreSQL | node + postgres_exporter |
| cast02 | 10.7.16.237 | App | node_exporter |
| cast03 | 10.7.16.238 | App + MariaDB replica | node_exporter |
Access
- Grafana (public): https://apps.f0xx.org/app/androidcast_project/monitor/
Login via existing PHP session (auto-SSO). Redirects to/loginif not authenticated. - Grafana (direct internal): http://10.7.16.239:3000/ (admin/acMonitor2026!)
- Prometheus UI: http://10.7.16.239:9090/ (no auth; internal only)
- Alertmanager UI: http://10.7.16.239:9093/ (no auth; internal only)
Config files (on cast04)
| File | Purpose |
|---|---|
/etc/prometheus/prometheus.yml |
Scrape targets |
/etc/prometheus/rules/node_alerts.yml |
HW/OS alert rules (10 rules) |
/etc/prometheus/rules/service_alerts.yml |
Service/HTTP/DB alert rules (10 rules) |
/etc/alertmanager/alertmanager.yml |
Alert routing (email + Telegram stub) |
/etc/grafana/grafana.ini |
Grafana config (subpath, auth proxy) |
/var/lib/grafana/provisioning/ |
Auto-provisioned datasource + dashboards dir |
/etc/conf.d/grafana |
Grafana runtime overrides (bind address, provisioning path) |
Config files (on cast01–03 BE)
| File | Purpose |
|---|---|
/etc/nginx/conf.d/androidcast.conf |
BE nginx — Grafana/Alertmanager proxy + auth_request |
/var/www/ac/composed/backend/public/api/grafana-auth-check.php |
PHP session → X-WEBAUTH-USER auth bridge |
Services management (cast04)
# Status
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
echo "$svc: $(rc-service $svc status 2>&1 | grep -o 'started\|stopped')"
done
# Restart all
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
sudo rc-service $svc restart
done
Grafana dashboards (imported)
| Dashboard | URL |
|---|---|
| Node Exporter Full | /d/rYdddlPWk/node-exporter-full |
| MySQL Exporter Quickstart | /d/... |
| Prometheus Blackbox Exporter | /d/xtkCtBkiz/prometheus-blackbox-exporter |
| Node Exporter EN Stats | /d/xfpJB9FGz/... |
TODO before production
- Email alerts: Fill in Gmail app password in
/etc/alertmanager/alertmanager.yml - Telegram alerts: Add
bot_token+chat_idtomulti-criticalreceiver - postgres_exporter: Verify cast01
pg_up 1after next restart - UptimeRobot: Create account + add HTTP monitor for
apps.f0xx.org/app/androidcast_project/ - Grafana admin password: Change from default
acMonitor2026!after setting up auth proxy
Alert thresholds
| Metric | Warning | Critical |
|---|---|---|
| CPU | >85% for 10m | >95% for 5m |
| RAM free | <15% for 5m | <5% for 2m |
| Disk free | <20% for 15m | <10% for 10m |
| Load (per vCPU) | >1.5× for 5m | >3.0× for 5m |
| HTTP probe | >5s for 5m | down >3m |
| SSL cert | <14 days | <7 days |
| MariaDB | high connections | down >2m |