mirror of
git://f0xx.org/ac/ac-deploy
synced 2026-07-29 04:38:48 +03:00
3.6 KiB
3.6 KiB
AC Cluster Monitoring — Deploy Guide
Stack: Prometheus 3.5 · Grafana 12.4 · Alertmanager 0.32 · blackbox_exporter · node_exporter · mysqld_exporter · postgres_exporter
SPEC: ac-docs/specs/20260626_cluster_monitoring.md
DR: ac-docs/DRs/20260626_cluster_monitoring.md
Status: R0 live on cast04 as of 2026-06-26
Topology
| Node | IP | Role | Exporters |
|---|---|---|---|
| cast04 | 10.7.16.239 | Monitoring host | Prometheus + Grafana + Alertmanager + blackbox + node |
| cast01 | 10.7.16.236 | App + Gitea + PostgreSQL | node + postgres_exporter |
| cast02 | 10.7.16.237 | App | node_exporter |
| cast03 | 10.7.16.238 | App | node_exporter |
| artc0 | 10.7.16.128 | BE + MariaDB + Docker | node + mysqld_exporter (Docker) |
Access
- Grafana (public): https://apps.f0xx.org/app/androidcast_project/monitor/
Login via existing PHP session (auto-SSO). Redirects to/loginif not authenticated. - Grafana (direct internal): http://10.7.16.239:3000/ (admin/acMonitor2026!)
- Prometheus UI: http://10.7.16.239:9090/ (no auth; internal only)
- Alertmanager UI: http://10.7.16.239:9093/ (no auth; internal only)
Config files (on cast04)
| File | Purpose |
|---|---|
/etc/prometheus/prometheus.yml |
Scrape targets |
/etc/prometheus/rules/node_alerts.yml |
HW/OS alert rules (10 rules) |
/etc/prometheus/rules/service_alerts.yml |
Service/HTTP/DB alert rules (10 rules) |
/etc/alertmanager/alertmanager.yml |
Alert routing (email + Telegram stub) |
/etc/grafana/grafana.ini |
Grafana config (subpath, auth proxy) |
/var/lib/grafana/provisioning/ |
Auto-provisioned datasource + dashboards dir |
/etc/conf.d/grafana |
Grafana runtime overrides (bind address, provisioning path) |
Config files (on artc0/BE)
| File | Purpose |
|---|---|
/etc/nginx/conf.d/apps.conf |
BE nginx — includes monitoring proxy location |
/var/www/.../api/grafana-auth-check.php |
PHP session → X-WEBAUTH-USER auth bridge |
/etc/mysqld-exporter.cnf |
MariaDB exporter credentials |
Services management (cast04)
# Status
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
echo "$svc: $(rc-service $svc status 2>&1 | grep -o 'started\|stopped')"
done
# Restart all
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
sudo rc-service $svc restart
done
Grafana dashboards (imported)
| Dashboard | URL |
|---|---|
| Node Exporter Full | /d/rYdddlPWk/node-exporter-full |
| MySQL Exporter Quickstart | /d/... |
| Prometheus Blackbox Exporter | /d/xtkCtBkiz/prometheus-blackbox-exporter |
| Node Exporter EN Stats | /d/xfpJB9FGz/... |
TODO before production
- Email alerts: Fill in Gmail app password in
/etc/alertmanager/alertmanager.yml - Telegram alerts: Add
bot_token+chat_idtomulti-criticalreceiver - postgres_exporter: Verify cast01
pg_up 1after next restart - UptimeRobot: Create account + add HTTP monitor for
apps.f0xx.org/app/androidcast_project/ - Grafana admin password: Change from default
acMonitor2026!after setting up auth proxy
Alert thresholds
| Metric | Warning | Critical |
|---|---|---|
| CPU | >85% for 10m | >95% for 5m |
| RAM free | <15% for 5m | <5% for 2m |
| Disk free | <20% for 15m | <10% for 10m |
| Load (per vCPU) | >1.5× for 5m | >3.0× for 5m |
| HTTP probe | >5s for 5m | down >3m |
| SSL cert | <14 days | <7 days |
| MariaDB | high connections | down >2m |