mirror of
git://f0xx.org/ac/ac-deploy
synced 2026-07-29 05:59:05 +03:00
d045653a7b
artc0 VM is off; remove node_exporter target and relabel rules from Prometheus. Point Grafana auth-proxy whitelist and alert-notifier shortener API at cast01–03 cluster instead of 10.7.16.128. Co-authored-by: Cursor <cursoragent@cursor.com>
88 lines
3.5 KiB
Markdown
88 lines
3.5 KiB
Markdown
# AC Cluster Monitoring — Deploy Guide
|
||
|
||
**Stack:** Prometheus 3.5 · Grafana 12.4 · Alertmanager 0.32 · blackbox_exporter · node_exporter · mysqld_exporter · postgres_exporter
|
||
**SPEC:** [ac-docs/specs/20260626_cluster_monitoring.md](../../ac-docs/specs/20260626_cluster_monitoring.md)
|
||
**DR:** [ac-docs/DRs/20260626_cluster_monitoring.md](../../ac-docs/DRs/20260626_cluster_monitoring.md)
|
||
**Status:** R0 live on cast04 as of 2026-06-26
|
||
|
||
---
|
||
|
||
## Topology
|
||
|
||
| Node | IP | Role | Exporters |
|
||
|------|----|------|-----------|
|
||
| cast04 | 10.7.16.239 | Monitoring host | Prometheus + Grafana + Alertmanager + blackbox + node |
|
||
| cast01 | 10.7.16.236 | App + Gitea + PostgreSQL | node + postgres_exporter |
|
||
| cast02 | 10.7.16.237 | App | node_exporter |
|
||
| cast03 | 10.7.16.238 | App + MariaDB replica | node_exporter |
|
||
|
||
## Access
|
||
|
||
- **Grafana (public):** https://apps.f0xx.org/app/androidcast_project/monitor/
|
||
Login via existing PHP session (auto-SSO). Redirects to `/login` if not authenticated.
|
||
- **Grafana (direct internal):** http://10.7.16.239:3000/ (admin/acMonitor2026!)
|
||
- **Prometheus UI:** http://10.7.16.239:9090/ (no auth; internal only)
|
||
- **Alertmanager UI:** http://10.7.16.239:9093/ (no auth; internal only)
|
||
|
||
## Config files (on cast04)
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `/etc/prometheus/prometheus.yml` | Scrape targets |
|
||
| `/etc/prometheus/rules/node_alerts.yml` | HW/OS alert rules (10 rules) |
|
||
| `/etc/prometheus/rules/service_alerts.yml` | Service/HTTP/DB alert rules (10 rules) |
|
||
| `/etc/alertmanager/alertmanager.yml` | Alert routing (email + Telegram stub) |
|
||
| `/etc/grafana/grafana.ini` | Grafana config (subpath, auth proxy) |
|
||
| `/var/lib/grafana/provisioning/` | Auto-provisioned datasource + dashboards dir |
|
||
| `/etc/conf.d/grafana` | Grafana runtime overrides (bind address, provisioning path) |
|
||
|
||
## Config files (on cast01–03 BE)
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `/etc/nginx/conf.d/androidcast.conf` | BE nginx — Grafana/Alertmanager proxy + auth_request |
|
||
| `/var/www/ac/composed/backend/public/api/grafana-auth-check.php` | PHP session → X-WEBAUTH-USER auth bridge |
|
||
|
||
## Services management (cast04)
|
||
|
||
```sh
|
||
# Status
|
||
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
|
||
echo "$svc: $(rc-service $svc status 2>&1 | grep -o 'started\|stopped')"
|
||
done
|
||
|
||
# Restart all
|
||
for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
|
||
sudo rc-service $svc restart
|
||
done
|
||
```
|
||
|
||
## Grafana dashboards (imported)
|
||
|
||
| Dashboard | URL |
|
||
|-----------|-----|
|
||
| Node Exporter Full | `/d/rYdddlPWk/node-exporter-full` |
|
||
| MySQL Exporter Quickstart | `/d/...` |
|
||
| Prometheus Blackbox Exporter | `/d/xtkCtBkiz/prometheus-blackbox-exporter` |
|
||
| Node Exporter EN Stats | `/d/xfpJB9FGz/...` |
|
||
|
||
## TODO before production
|
||
|
||
1. **Email alerts:** Fill in Gmail app password in `/etc/alertmanager/alertmanager.yml`
|
||
2. **Telegram alerts:** Add `bot_token` + `chat_id` to `multi-critical` receiver
|
||
3. **postgres_exporter:** Verify cast01 `pg_up 1` after next restart
|
||
4. **UptimeRobot:** Create account + add HTTP monitor for `apps.f0xx.org/app/androidcast_project/`
|
||
5. **Grafana admin password:** Change from default `acMonitor2026!` after setting up auth proxy
|
||
|
||
## Alert thresholds
|
||
|
||
| Metric | Warning | Critical |
|
||
|--------|---------|----------|
|
||
| CPU | >85% for 10m | >95% for 5m |
|
||
| RAM free | <15% for 5m | <5% for 2m |
|
||
| Disk free | <20% for 15m | <10% for 10m |
|
||
| Load (per vCPU) | >1.5× for 5m | >3.0× for 5m |
|
||
| HTTP probe | >5s for 5m | down >3m |
|
||
| SSL cert | <14 days | <7 days |
|
||
| MariaDB | high connections | down >2m |
|