added 'monitoring' path - ask devs why it's there

monitoring/README.md

@@ -0,0 +1,89 @@
+# AC Cluster Monitoring — Deploy Guide
+
+**Stack:** Prometheus 3.5 · Grafana 12.4 · Alertmanager 0.32 · blackbox_exporter · node_exporter · mysqld_exporter · postgres_exporter  
+**SPEC:** [ac-docs/specs/20260626_cluster_monitoring.md](../../ac-docs/specs/20260626_cluster_monitoring.md)  
+**DR:** [ac-docs/DRs/20260626_cluster_monitoring.md](../../ac-docs/DRs/20260626_cluster_monitoring.md)  
+**Status:** R0 live on cast04 as of 2026-06-26
+
+---
+
+## Topology
+
+| Node | IP | Role | Exporters |
+|------|----|------|-----------|
+| cast04 | 10.7.16.239 | Monitoring host | Prometheus + Grafana + Alertmanager + blackbox + node |
+| cast01 | 10.7.16.236 | App + Gitea + PostgreSQL | node + postgres_exporter |
+| cast02 | 10.7.16.237 | App | node_exporter |
+| cast03 | 10.7.16.238 | App | node_exporter |
+| artc0 | 10.7.16.128 | BE + MariaDB + Docker | node + mysqld_exporter (Docker) |
+
+## Access
+
+- **Grafana (public):** https://apps.f0xx.org/app/androidcast_project/monitor/  
+  Login via existing PHP session (auto-SSO). Redirects to `/login` if not authenticated.
+- **Grafana (direct internal):** http://10.7.16.239:3000/ (admin/acMonitor2026!)
+- **Prometheus UI:** http://10.7.16.239:9090/ (no auth; internal only)
+- **Alertmanager UI:** http://10.7.16.239:9093/ (no auth; internal only)
+
+## Config files (on cast04)
+
+| File | Purpose |
+|------|---------|
+| `/etc/prometheus/prometheus.yml` | Scrape targets |
+| `/etc/prometheus/rules/node_alerts.yml` | HW/OS alert rules (10 rules) |
+| `/etc/prometheus/rules/service_alerts.yml` | Service/HTTP/DB alert rules (10 rules) |
+| `/etc/alertmanager/alertmanager.yml` | Alert routing (email + Telegram stub) |
+| `/etc/grafana/grafana.ini` | Grafana config (subpath, auth proxy) |
+| `/var/lib/grafana/provisioning/` | Auto-provisioned datasource + dashboards dir |
+| `/etc/conf.d/grafana` | Grafana runtime overrides (bind address, provisioning path) |
+
+## Config files (on artc0/BE)
+
+| File | Purpose |
+|------|---------|
+| `/etc/nginx/conf.d/apps.conf` | BE nginx — includes monitoring proxy location |
+| `/var/www/.../api/grafana-auth-check.php` | PHP session → X-WEBAUTH-USER auth bridge |
+| `/etc/mysqld-exporter.cnf` | MariaDB exporter credentials |
+
+## Services management (cast04)
+
+```sh
+# Status
+for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
+  echo "$svc: $(rc-service $svc status 2>&1 | grep -o 'started\|stopped')"
+done
+
+# Restart all
+for svc in prometheus node-exporter blackbox-exporter alertmanager grafana; do
+  sudo rc-service $svc restart
+done
+```
+
+## Grafana dashboards (imported)
+
+| Dashboard | URL |
+|-----------|-----|
+| Node Exporter Full | `/d/rYdddlPWk/node-exporter-full` |
+| MySQL Exporter Quickstart | `/d/...` |
+| Prometheus Blackbox Exporter | `/d/xtkCtBkiz/prometheus-blackbox-exporter` |
+| Node Exporter EN Stats | `/d/xfpJB9FGz/...` |
+
+## TODO before production
+
+1. **Email alerts:** Fill in Gmail app password in `/etc/alertmanager/alertmanager.yml`
+2. **Telegram alerts:** Add `bot_token` + `chat_id` to `multi-critical` receiver
+3. **postgres_exporter:** Verify cast01 `pg_up 1` after next restart
+4. **UptimeRobot:** Create account + add HTTP monitor for `apps.f0xx.org/app/androidcast_project/`
+5. **Grafana admin password:** Change from default `acMonitor2026!` after setting up auth proxy
+
+## Alert thresholds
+
+| Metric | Warning | Critical |
+|--------|---------|----------|
+| CPU | >85% for 10m | >95% for 5m |
+| RAM free | <15% for 5m | <5% for 2m |
+| Disk free | <20% for 15m | <10% for 10m |
+| Load (per vCPU) | >1.5× for 5m | >3.0× for 5m |
+| HTTP probe | >5s for 5m | down >3m |
+| SSL cert | <14 days | <7 days |
+| MariaDB | high connections | down >2m |