cluster monitoring Q/A sesion 01, TBD

drafts/20260626_cluster_monitoring.txt

@@ -1,7 +1,7 @@
 Infrastructure health monitoring draft
 milestone: R0-pre
 severity: high
-state: draft
+state: draft (annotated — see [AGENT NOTE] blocks; please review before DR/SPEC conversion)
 author: Anton Afanasyeu
 date: 2026-06-26
 related to: 20260618_repos_reorganizing*, microservice split, clustered CI/CD
@@ -35,11 +35,36 @@ with such a floating architecture, especially if we gonna use on-demand cluster
 - the data storage Y flied away because of HW damage - allocate the replica of storage Y and put it instead of broken one to the cluster
 etc.
 
+[AGENT NOTE — auto-remediation scope:
+  Everything above (auto-scale, auto-DDoS-protect, auto-failover storage) is NOT monitoring — it is
+  automated orchestration / cluster self-healing. These are valuable long-term goals but they are an
+  entirely separate and much larger engineering track (think: Kubernetes operators, custom controllers,
+  Ansible event-driven, or a home-grown orchestration plane). Mixing them into the monitoring spec
+  risks scope-creep and may block the R0 delivery.
+  SUGGESTION: Split into two tracks explicitly in the SPEC:
+    Track A (this SPEC, R0): passive monitoring + alerting — observe, record, notify
+    Track B (future, R1+): active remediation — orchestration reacts to monitoring events
+  The DR should decide whether "smart hypervisor" is a Track B placeholder or in-scope for R0.
+  My recommendation: Track B placeholder — list the scenarios as "desired future automation" but
+  do not design them in the monitoring SPEC.]
+[Anton's answer: agree with suggestion. please help me to organize Track B as a separate draft document as you see it; make sure it would be referenced in Track A like a next-gen step of supervising over the infra, clean up everything from exactly this draft that refers to Track B, so we'll expect Track B will not be forgotten]
+
 Basically, the most important set of monitored events may be:
 - hardware health monitoring
 -- node / microservice availability
 -- ... TBD
 
+[AGENT NOTE — hardware monitoring gaps worth filling in TBD:
+  Standard items for Xen/KVM VM nodes (project already runs XEN HVM/paravirt):
+  -- CPU: utilization %, steal time (critical for Xen — if hypervisor steals CPU, steal > 10% is a red flag)
+  -- RAM: used / available / swap usage
+  -- Disk: utilization %, IOPS, write latency (key for MariaDB nodes), disk SMART via smartmontools
+  -- NIC: ingress/egress bytes/sec, packet drops, errors
+  -- Load average (1m/5m/15m)
+  -- Xen-specific: dom0 balloon driver state, vcpu credit scheduler stats
+  Suggest confirming which of these you want in SPEC §3 (requirements table).]
+[Anton's answer: you have clearer vision on that rather than me, i'm not a devops, please help to reconstruct/organize the doc that way, i can just confirm that again your suggestion is more structured; i would say it's better to keep an eye over all of the metrics, but up to you there]
+
 - software health monitoring
 -- node / microservice availability
 -- the reason
@@ -49,12 +74,48 @@ Basically, the most important set of monitored events may be:
 --- out of resources
 -- ... TBD
 
+[AGENT NOTE — software monitoring gaps worth filling:
+  Standard per-service items:
+  -- HTTP health endpoint response time + status code (blackbox probe, e.g. /health or /ping)
+  -- Process presence check (is nginx/php-fpm/mariadb PID alive?)
+  -- Log error rate (ERROR/CRITICAL lines per minute from syslog/journald)
+  -- MariaDB: replication lag (seconds_behind_master), connection count, slow query count
+  -- nginx: 5xx rate, upstream connection errors
+  -- php-fpm: queue depth, worker saturation
+  These map naturally to exporters (see tool section note below).]
+[Anton's answer: agreed]
+
 - provider's fault
 -- hardware or software
 -- service absense
 -- payment expired
 -- ... TBD
 
+[AGENT NOTE — provider fault detection:
+  This category requires EXTERNAL monitoring — probes from outside your network to verify
+  that apps.f0xx.org is reachable from the public internet. Internal monitoring is blind to:
+    - Anton's router/uplink going down
+    - XEN host hardware failure (the VM's own monitoring dies with it)
+    - BGP/routing issues upstream
+  Options for external check:
+    a) UptimeRobot / Freshping / BetterUptime free tier (simple HTTP check from cloud)
+    b) Self-hosted Uptime Kuma on a VPS outside this network (cheapest independent probe)
+    c) Blackbox Exporter + Prometheus alerting IF the Prometheus instance is outside the cluster
+  "Payment expired" can't really be auto-detected; it's an ops reminder / calendar alert.
+  Suggest clarifying in SPEC: external uptime check is REQUIRED for provider-fault detection,
+  and it must live OUTSIDE the monitored cluster.]
+[Anton's answer: unfortunately there are some sw limits that may probably being a soft-blockers, for example (using current architecture):
+  - we cannot deeply change the set of software on the router
+  - we cannot change the software on FE side as well
+  - we can rely and fully operate only with cluster/nodes
+future, renting the true cloud infrastructure, AWS, GClouds, Azure, etc.:
+  - also some risks/limitations on changing top-level software because owned by cloud provider
+  - but probably it's a kind of 'hackable', don't know - newer rented foreign cloud infra before
+
+think about it
+
+]
+
 - ... TBD
 
 a simple example, using the current lab infrastructure:
@@ -75,6 +136,31 @@ the VM delegated cluster owner must set its own monitoring software where it nee
 3) slow egress/ingress -> delegate to Anton
 4) one of project services or its relatives aren't responding well - there should be per-service self monitoring and should raise an alert if so
 
+[AGENT NOTE — monitoring placement (unanswered design question):
+  The draft does not address where the monitoring stack itself lives. This is a key SPEC decision:
+  Option A: Dedicated monitoring VM (4th node, outside the 3-node cluster)
+    + monitoring survives a cluster node failure
+    - costs an extra VM
+  Option B: On one of the 3 existing cluster nodes (e.g. the "ops" node)
+    - if that node fails, monitoring goes dark — worst case: you're blind exactly when you need it
+    + no extra cost
+  Option C: On Anton's infra (outside delegated cluster entirely)
+    + most resilient for node-level failure detection
+    - requires coordination with Anton; may not be acceptable for project-owner autonomy
+  Option D: Lightweight agents on each node, reporting to a cloud-hosted aggregator (e.g. Grafana Cloud
+    free tier, or any external SaaS)
+    + zero infra overhead; survives full cluster outage
+    - data leaves your network
+  SUGGESTION: Decide and lock in SPEC §2 (topology). My recommendation: Option A or D depending
+  on budget constraints. Option B is acceptable for lab/pre-alpha only.]
+[Anton's answer:
+  Option A: currently doable, having full control over hypervised VMs; dedicated machine isn't something that can block us
+  Option B: likely the chosed one for now, once we on Anton's provided resources but a bit limited on their capacity; thinking about A and B what to choose
+  Option C: good, but no. prefer to keep everything together until going to real clouds
+  Option D: good about lightweight agents, but don't want to track the data somewhere outside; prefer to keep everything single-hosted
+
+Now i think option A is the best from robust and future knowledge, although, probably, the hardest in implementation. if another Alpine instance (cast04 as 4th node) will take care of monitoring, I'm fine with option A
+]
 
 The how:
 the actor (any of) may be triggered on faulty node, cluster, service, ..., state, first off, by email. any other services are also likely to have (messengers, SMS, etc.), but anyway we should integrate some monitoring visualizer(s):
@@ -84,19 +170,135 @@ the actor (any of) may be triggered on faulty node, cluster, service, ..., state
 4) Visualization per service of project's software inside the each Node
 based on the requirements, a few almost-ready solutions touch the souls:
 1. Grafana (Kibana)
+
+[AGENT NOTE — Grafana ≠ Kibana — FACTUAL CORRECTION:
+  These are two completely different products from different companies and stacks:
+  - Grafana (grafana.com): standalone visualization layer; plugs into Prometheus, InfluxDB,
+    Loki, TimescaleDB/PostgreSQL, etc. Self-hosted or Grafana Cloud.
+  - Kibana: the "K" in the ELK stack (Elasticsearch + Logstash + Kibana). Primarily a log
+    analysis/search UI over Elasticsearch indices. NOT typically used for infra metrics.
+  The project already has a Grafana pivot analysis doc: GRAFANA_vs_others_graphvis_pivot.md.
+  They should be listed and evaluated separately. My suggestion: drop Kibana from this section
+  (ELK is heavy and log-centric, not the primary choice for VM metrics), or add it as a separate
+  option specifically for centralized log aggregation (not metrics dashboards).]
+[Anton's answer: ok, let's exclude Kibana at all from our goals/perspectives]
+
 1.1 Pros: the most beautiful and user-friendly; pretty easy to play with pre-stored parameters, building customized graphics; may be self-hosted; may reflect anything you want, but the administrator's eye is required on it, at least on the first steps
 1.2 Cons: requires additional software, like Prometheus (the most commonly used) or maybe other time-series based engine; so flexible that it is sometimes unbelievably hard to create pre-stored parameters set to visualize it later; requires also additional data converters/plugins, Prometheus-friendly
 
+[AGENT NOTE — Prometheus deserves its own section:
+  Prometheus is the core of the recommended stack for VM cluster monitoring and should be called
+  out explicitly, not just as a "Grafana requirement". The stack is:
+    Prometheus (time-series DB + scraper) + node_exporter (per VM hardware metrics)
+    + blackbox_exporter (HTTP/TCP service probes) + Alertmanager (routing alerts to email/Slack/etc.)
+    + Grafana (dashboards on top)
+  Each component is independently useful and has a clear role. Ready-made "community dashboards"
+  exist in Grafana for node_exporter (dashboard ID 1860 — "Node Exporter Full") and MariaDB.
+  Also worth mentioning: VictoriaMetrics as a drop-in Prometheus-compatible TSDB — uses 5-10x
+  less RAM/disk, easier to run on a small cluster, retains data longer by default. If the cluster
+  nodes are small (8GB RAM per node), VictoriaMetrics may be preferable to Prometheus itself.
+  Relationship to existing project TimescaleDB platform: Prometheus/VictoriaMetrics is for INFRA
+  metrics (CPU/RAM/disk/latency). TimescaleDB is proposed for APPLICATION/BUSINESS metrics
+  (crash rates, session counts, graph analytics). These can coexist — different stores, different
+  retention, different queries. Grafana queries both. Clarify this split in the SPEC.]
+[Anton's answer: good to know,i just want you again to re-read the soft limitations i've described above; pressure is that we cannot somehow install new software neither on router nor on FE unfortunately; if there's no need, then just OK; if need is not intended as mandatory - also ok, we just lose some of metrics, which is currently fine)
+
 2. Zabbix
 2.1 Pros: good UI/UX, pretty easy to deploy; good support, good abilities;
 2.2 Cons: not as good as Grafana+... from the UI/UX perspective
 2.3 Need to check the licensing
 
+[AGENT NOTE — Zabbix licensing:
+  Zabbix Community Edition is GPL v2 licensed — free for any use. No commercial license needed
+  for self-hosted monitoring of your own infrastructure. The Zabbix Agent on monitored nodes is
+  also LGPL. This concern is resolved; not a blocker.
+  Zabbix is a "batteries included" solution — agent + server + DB + UI in one package, which is
+  its main practical advantage over Prometheus+Grafana. Easier initial setup, built-in alerting,
+  no separate Alertmanager. Downside: more opinionated, harder to integrate with custom app metrics
+  vs Prometheus's pull model.]
+
 3. Nagios
 3.1 Pros: the most probably known and loved by many administrators, can capture any data, lightweight;
 3.2 Cons: i don't know nothing about data visualization, may be Nagios is just data collector and visualization being made by for example Grafana? disadvantage also in that this is very complex solution, hard to tune, hard to maintain
 
+[AGENT NOTE — Nagios clarification:
+  Nagios Core does have its own built-in web UI (CGI-based, classic green/red status board), but it
+  is very basic — good for status/alerting, poor for time-series graphs.
+  Nagios is NOT a time-series data collector — it is a status/alert engine (checks return OK/WARNING/
+  CRITICAL states). For historical graphs, Nagios is typically paired with PNP4Nagios or Graphite, NOT
+  Prometheus/Grafana directly (though adapters exist).
+  Modern consensus: Nagios has been largely superseded by Prometheus+Alertmanager for metrics-based
+  monitoring. Nagios Core is still used for pure availability checks (is the host/port up?) but the
+  ecosystem has moved on. Suggest downgrading this option to "legacy/not recommended for new setups".
+  Nagios XI (commercial) is polished but not free.
+  If lightweight availability-only checks are needed, Uptime Kuma is a modern self-hosted alternative
+  to Nagios with a much better UI and zero configuration overhead.]
+
 4. ... TBD, any suggestions
 
+[AGENT NOTE — additional options worth considering:
+  4. Prometheus + Grafana (recommended for this cluster, see notes above)
+     - node_exporter: CPU/RAM/disk/NIC per VM
+     - blackbox_exporter: HTTP/TCP service health probes
+     - mysqld_exporter: MariaDB replication lag, slow queries, connections
+     - Alertmanager: route alerts to email + Telegram/Slack
+     - Grafana: dashboards (community templates available for all the above)
+     - VictoriaMetrics (optional): replace Prometheus TSDB for lower resource use on 8GB nodes
+  5. Netdata (https://netdata.cloud)
+     - Installs as a single agent per node (apt/apk/bash one-liner)
+     - Gives real-time per-node dashboard out of the box — zero configuration for hardware metrics
+     - Can stream to a central Netdata parent node
+     - Netdata Cloud (free tier) aggregates cross-node view without self-hosting a central stack
+     - Useful for quick bootstrapping before a full Prometheus stack is deployed
+     - License: GPL v3
+  6. Uptime Kuma (https://github.com/louislam/uptime-kuma)
+     - Lightweight self-hosted service uptime monitor (HTTP/TCP/DNS/ping checks)
+     - Very easy to deploy (Docker single container)
+     - Covers "is service X responding?" with beautiful status page
+     - Good complement to Prometheus (Prometheus = deep metrics, Uptime Kuma = simple uptime)
+     - License: MIT
+  7. ELK / OpenSearch (for logs, not metrics)
+     - Elasticsearch + Logstash/Fluent Bit + Kibana/OpenSearch Dashboards
+     - Best for centralized log aggregation and search (nginx access logs, php-fpm error logs, syslog)
+     - Heavy (needs 4+ GB RAM for Elasticsearch alone) — probably overkill for current lab
+     - Grafana Loki is a lighter alternative for log aggregation that integrates natively with Grafana]
+[Anton's answer: cool, I like everything proposed there]
+
 5. Mixed (hybrid), like Grafana + Zabbix
 ... TBD
+
+[AGENT NOTE — hybrid recommendation:
+  Based on the project's existing infra (Alpine VMs, MariaDB, nginx, PHP-FPM, Xen, 8GB RAM/node)
+  and what's already discussed in GRAFANA_vs_others_graphvis_pivot.md, my suggestion for SPEC:
+
+  RECOMMENDED STACK (R0):
+    Infra layer: Prometheus (or VictoriaMetrics) + node_exporter on each VM
+                 + blackbox_exporter for HTTP/TCP probes
+                 + mysqld_exporter for MariaDB
+                 + Alertmanager → email (required) + Telegram bot (optional)
+    Dashboard:   Grafana (community dashboards for node_exporter + MariaDB)
+    Uptime:      Uptime Kuma on a separate lightweight VM or the FE (external to cluster)
+    Deployment:  ac-deploy (new monitoring/ role in cluster0 scripts)
+
+  FUTURE (R1):
+    Log aggregation: Grafana Loki + Promtail (lighter than ELK)
+    Auto-remediation: custom event handler scripts triggered by Alertmanager webhooks
+    Business metrics: Grafana queries TimescaleDB (already in project plan)
+
+  OPEN QUESTION FOR PO: Should the monitoring stack get its own VM node (4th node), or run
+  on one of the 3 cluster nodes (and which one)? This affects DR decision §2.]
+[Anton's answer: good, so we agreed on most of the points there, i also concluded this combination as the better one for myself. I think I've addressed almost all the questions. the Q from my side:
+imagine I've allocated yet another VM and put it inside cluster as an empty node (only os-basic software is installed)
+  - do I need something from DNS hosting provider, like CNAME records, additional TXT records maybe?
+  - do we need a kind of tuning of the existing FE configs (nginx which is a forwarder there, for example)
+  - ImprovMX, addy.io, Telegram bot where? internally deployed inside the cluster or externally used by sort of API? need to create accounts for the document target feature? on which services, just to be ready before we go?
+]
+
+
+[Anton's nodes:
+just to remind, we're limited on this on the server's side:
+  - machine type: next type will be likely paravirtualized (no HVM)
+  - OS type: Alpine latest LTS (or Edge, but manually configured changing the repos)
+  - software: likely from the Alpine community packages/ package manager; meaning we'll try to avoid any source-based software, preferring apk's built for the particular Alpine
+  - FE/router: unlikely to install the software there, almost untouchable; we can play just with the configuration, like adding routes, playing with iptables, etc. with the existing software, but no new installations there
+]