ac-docs/drafts/20260626_cluster_monitoring.txt

Infrastructure health monitoring draft
milestone: R0-pre
severity: high
state: draft (annotated — see [AGENT NOTE] blocks; please review before DR/SPEC conversion)
author: Anton Afanasyeu
date: 2026-06-26
related to: 20260618_repos_reorganizing*, microservice split, clustered CI/CD

The problem:
Actually there's no problem while everything is working. But as many know, sometimes there's a time where the software fails, hardware tires, and vice versa.
The goal is to do constant infrastructure health monitoring, probably even with prediction warnings, if possible


The why:
1) now we do have a lots of virtual machines (will be more if we move forward to real clouds) and lots of microservices splitted around the virtual machines. Sometimes the relation between machine and microservice is pretty straight forward:
| microservice A | -> | VM A | -> ... networking
sometimes more robust, but with additional cost of redundancy, like:
| microservice A | -> | VM B database  frontend |
                                |------> [ VM A master db node  ]----------> [aggregator + frontent]-----> ... networking
                                |------> [ VM C replica db node ]-----^
sometimes it may be very complex, for example
| microservice A (master) |
      |------------- [delegation A.A, slave ]
      |                   |-------------------[ VM A master db node ]-----|--------->[aggregator + frontend]------> ... next pipeline ...
      |                             |---------[ VM C replica db node ]----X----^
      |                                                                 | |
      |------------- [delegation A.B, slave ]                           | |
                          |-------------------[ VM A master db node ]----X----
                                    |---------[ VM B replica db node ]---|

with such a floating architecture, especially if we gonna use on-demand cluster scaling, there's very hard to perform manual watch of each service or node. even though, good to have a kind of smart hypervisor over all our microservices and cluster nodes that will make a decision what to do next in case, for example:
- the microservice X experiences high CPU load, needs to ask CPU load balancer to allocate quickly another node Y and redirect a part of microservice's X requests to new node Y
- the microservice X stays untouched for a last Y hours and the cluster's overall load is less than 75% during this time - hypervisor makes a decision to stop redundant instance, simply to save money
- the network pressure is superbly high - investigate why, if there's hackers DDoS attack - perform set of actions for DDoS protection, if there's simply spike because a lots of users(customers) - do actions to increase cluster size and bandwidth, etc.
- the data storage Y flied away because of HW damage - allocate the replica of storage Y and put it instead of broken one to the cluster
etc.

[AGENT NOTE — auto-remediation scope:
  Everything above (auto-scale, auto-DDoS-protect, auto-failover storage) is NOT monitoring — it is
  automated orchestration / cluster self-healing. These are valuable long-term goals but they are an
  entirely separate and much larger engineering track (think: Kubernetes operators, custom controllers,
  Ansible event-driven, or a home-grown orchestration plane). Mixing them into the monitoring spec
  risks scope-creep and may block the R0 delivery.
  SUGGESTION: Split into two tracks explicitly in the SPEC:
    Track A (this SPEC, R0): passive monitoring + alerting — observe, record, notify
    Track B (future, R1+): active remediation — orchestration reacts to monitoring events
  The DR should decide whether "smart hypervisor" is a Track B placeholder or in-scope for R0.
  My recommendation: Track B placeholder — list the scenarios as "desired future automation" but
  do not design them in the monitoring SPEC.]
[Anton's answer: agree with suggestion. please help me to organize Track B as a separate draft document as you see it; make sure it would be referenced in Track A like a next-gen step of supervising over the infra, clean up everything from exactly this draft that refers to Track B, so we'll expect Track B will not be forgotten]

Basically, the most important set of monitored events may be:
- hardware health monitoring
-- node / microservice availability
-- ... TBD

[AGENT NOTE — hardware monitoring gaps worth filling in TBD:
  Standard items for Xen/KVM VM nodes (project already runs XEN HVM/paravirt):
  -- CPU: utilization %, steal time (critical for Xen — if hypervisor steals CPU, steal > 10% is a red flag)
  -- RAM: used / available / swap usage
  -- Disk: utilization %, IOPS, write latency (key for MariaDB nodes), disk SMART via smartmontools
  -- NIC: ingress/egress bytes/sec, packet drops, errors
  -- Load average (1m/5m/15m)
  -- Xen-specific: dom0 balloon driver state, vcpu credit scheduler stats
  Suggest confirming which of these you want in SPEC §3 (requirements table).]
[Anton's answer: you have clearer vision on that rather than me, i'm not a devops, please help to reconstruct/organize the doc that way, i can just confirm that again your suggestion is more structured; i would say it's better to keep an eye over all of the metrics, but up to you there]

- software health monitoring
-- node / microservice availability
-- the reason
--- scale needed
--- unsuccessful configuration
--- software not getting up
--- out of resources
-- ... TBD

[AGENT NOTE — software monitoring gaps worth filling:
  Standard per-service items:
  -- HTTP health endpoint response time + status code (blackbox probe, e.g. /health or /ping)
  -- Process presence check (is nginx/php-fpm/mariadb PID alive?)
  -- Log error rate (ERROR/CRITICAL lines per minute from syslog/journald)
  -- MariaDB: replication lag (seconds_behind_master), connection count, slow query count
  -- nginx: 5xx rate, upstream connection errors
  -- php-fpm: queue depth, worker saturation
  These map naturally to exporters (see tool section note below).]
[Anton's answer: agreed]

- provider's fault
-- hardware or software
-- service absense
-- payment expired
-- ... TBD

[AGENT NOTE — provider fault detection:
  This category requires EXTERNAL monitoring — probes from outside your network to verify
  that apps.f0xx.org is reachable from the public internet. Internal monitoring is blind to:
    - Anton's router/uplink going down
    - XEN host hardware failure (the VM's own monitoring dies with it)
    - BGP/routing issues upstream
  Options for external check:
    a) UptimeRobot / Freshping / BetterUptime free tier (simple HTTP check from cloud)
    b) Self-hosted Uptime Kuma on a VPS outside this network (cheapest independent probe)
    c) Blackbox Exporter + Prometheus alerting IF the Prometheus instance is outside the cluster
  "Payment expired" can't really be auto-detected; it's an ops reminder / calendar alert.
  Suggest clarifying in SPEC: external uptime check is REQUIRED for provider-fault detection,
  and it must live OUTSIDE the monitored cluster.]
[Anton's answer: unfortunately there are some sw limits that may probably being a soft-blockers, for example (using current architecture):
  - we cannot deeply change the set of software on the router
  - we cannot change the software on FE side as well
  - we can rely and fully operate only with cluster/nodes
future, renting the true cloud infrastructure, AWS, GClouds, Azure, etc.:
  - also some risks/limitations on changing top-level software because owned by cloud provider
  - but probably it's a kind of 'hackable', don't know - newer rented foreign cloud infra before

think about it

]

- ... TBD

a simple example, using the current lab infrastructure:
[global net x.x.x.x] <-> [(134.17.26.161 external IP) router (debian-based] (internal IP 10.7.6.228) ] <-> [(10.7.0.10 intra network 1) FE (intra network 2: 10.7.0.0/8)] [intra network 3: ...]
the owner of top-level services is Anton Afanasyeu. top level services include:
- internet provider
- router maintenance
- intra network delegation, configuration, maintenance, VMs (empty, just a (re)source providing), git as a service, disk storage
- basic on-demand OS installation, on-demand VM snapshots
----------- all up is SaaS, for all the questions contact owner ---------

-- cluster of VM's, XEN-based (hvm or paravirt, depends)
-- VM's control, VM's internal software installation, access, aggregation, fine tunings, software development, ssh, internal setups, etc.
----------- the last few lines are totally controllable by developer, developed product owner and projects workers as decided by product owner ---------
the VM delegated cluster owner must set its own monitoring software where it needs to be placed and monitor all the parameters he decided to keep an eye on, like:
1) hardware resources load, per cluster node and overall in-cluster, comparing the max. values delegated by Anton Afanasyeu according to payment fee for the cluster; like Anton guarantees that CPU will be real (not emulated) and it will be Intel 8 cores of Gold 5218 CPU @ 2.30GHz series, 8Gb of RAM per node, 32Gb of non-shared disk space per node, 3 nodes in cluster, 200Mbit in/out internet, etc; the project owner should have monitoring based of the extremum of each of parameters, compare with proposed plan, alert of some of values differ than what Anton guaranteed
2) project owner took 3 nodes / cluster; after month of heavy load one of the nodes show instabilities and slow data writes using distributed among the nodes MariaDb; MariaDb monitoring show exactly that Node B contains the replica of master db node and it's possible that there's a hardware fault (disk writes), so he should notify Anton to take it into account and provide spare node
3) slow egress/ingress -> delegate to Anton
4) one of project services or its relatives aren't responding well - there should be per-service self monitoring and should raise an alert if so

[AGENT NOTE — monitoring placement (unanswered design question):
  The draft does not address where the monitoring stack itself lives. This is a key SPEC decision:
  Option A: Dedicated monitoring VM (4th node, outside the 3-node cluster)
    + monitoring survives a cluster node failure
    - costs an extra VM
  Option B: On one of the 3 existing cluster nodes (e.g. the "ops" node)
    - if that node fails, monitoring goes dark — worst case: you're blind exactly when you need it
    + no extra cost
  Option C: On Anton's infra (outside delegated cluster entirely)
    + most resilient for node-level failure detection
    - requires coordination with Anton; may not be acceptable for project-owner autonomy
  Option D: Lightweight agents on each node, reporting to a cloud-hosted aggregator (e.g. Grafana Cloud
    free tier, or any external SaaS)
    + zero infra overhead; survives full cluster outage
    - data leaves your network
  SUGGESTION: Decide and lock in SPEC §2 (topology). My recommendation: Option A or D depending
  on budget constraints. Option B is acceptable for lab/pre-alpha only.]
[Anton's answer:
  Option A: currently doable, having full control over hypervised VMs; dedicated machine isn't something that can block us
  Option B: likely the chosed one for now, once we on Anton's provided resources but a bit limited on their capacity; thinking about A and B what to choose
  Option C: good, but no. prefer to keep everything together until going to real clouds
  Option D: good about lightweight agents, but don't want to track the data somewhere outside; prefer to keep everything single-hosted

Now i think option A is the best from robust and future knowledge, although, probably, the hardest in implementation. if another Alpine instance (cast04 as 4th node) will take care of monitoring, I'm fine with option A
]

The how:
the actor (any of) may be triggered on faulty node, cluster, service, ..., state, first off, by email. any other services are also likely to have (messengers, SMS, etc.), but anyway we should integrate some monitoring visualizer(s):
1) Visualization of the whole infrastructure
2) Visualization per cluster and per Node health
3) Visualization per service provided by distro used inside the each Node
4) Visualization per service of project's software inside the each Node
based on the requirements, a few almost-ready solutions touch the souls:
1. Grafana (Kibana)

[AGENT NOTE — Grafana ≠ Kibana — FACTUAL CORRECTION:
  These are two completely different products from different companies and stacks:
  - Grafana (grafana.com): standalone visualization layer; plugs into Prometheus, InfluxDB,
    Loki, TimescaleDB/PostgreSQL, etc. Self-hosted or Grafana Cloud.
  - Kibana: the "K" in the ELK stack (Elasticsearch + Logstash + Kibana). Primarily a log
    analysis/search UI over Elasticsearch indices. NOT typically used for infra metrics.
  The project already has a Grafana pivot analysis doc: GRAFANA_vs_others_graphvis_pivot.md.
  They should be listed and evaluated separately. My suggestion: drop Kibana from this section
  (ELK is heavy and log-centric, not the primary choice for VM metrics), or add it as a separate
  option specifically for centralized log aggregation (not metrics dashboards).]
[Anton's answer: ok, let's exclude Kibana at all from our goals/perspectives]

1.1 Pros: the most beautiful and user-friendly; pretty easy to play with pre-stored parameters, building customized graphics; may be self-hosted; may reflect anything you want, but the administrator's eye is required on it, at least on the first steps
1.2 Cons: requires additional software, like Prometheus (the most commonly used) or maybe other time-series based engine; so flexible that it is sometimes unbelievably hard to create pre-stored parameters set to visualize it later; requires also additional data converters/plugins, Prometheus-friendly

[AGENT NOTE — Prometheus deserves its own section:
  Prometheus is the core of the recommended stack for VM cluster monitoring and should be called
  out explicitly, not just as a "Grafana requirement". The stack is:
    Prometheus (time-series DB + scraper) + node_exporter (per VM hardware metrics)
    + blackbox_exporter (HTTP/TCP service probes) + Alertmanager (routing alerts to email/Slack/etc.)
    + Grafana (dashboards on top)
  Each component is independently useful and has a clear role. Ready-made "community dashboards"
  exist in Grafana for node_exporter (dashboard ID 1860 — "Node Exporter Full") and MariaDB.
  Also worth mentioning: VictoriaMetrics as a drop-in Prometheus-compatible TSDB — uses 5-10x
  less RAM/disk, easier to run on a small cluster, retains data longer by default. If the cluster
  nodes are small (8GB RAM per node), VictoriaMetrics may be preferable to Prometheus itself.
  Relationship to existing project TimescaleDB platform: Prometheus/VictoriaMetrics is for INFRA
  metrics (CPU/RAM/disk/latency). TimescaleDB is proposed for APPLICATION/BUSINESS metrics
  (crash rates, session counts, graph analytics). These can coexist — different stores, different
  retention, different queries. Grafana queries both. Clarify this split in the SPEC.]
[Anton's answer: good to know,i just want you again to re-read the soft limitations i've described above; pressure is that we cannot somehow install new software neither on router nor on FE unfortunately; if there's no need, then just OK; if need is not intended as mandatory - also ok, we just lose some of metrics, which is currently fine)

2. Zabbix
2.1 Pros: good UI/UX, pretty easy to deploy; good support, good abilities;
2.2 Cons: not as good as Grafana+... from the UI/UX perspective
2.3 Need to check the licensing

[AGENT NOTE — Zabbix licensing:
  Zabbix Community Edition is GPL v2 licensed — free for any use. No commercial license needed
  for self-hosted monitoring of your own infrastructure. The Zabbix Agent on monitored nodes is
  also LGPL. This concern is resolved; not a blocker.
  Zabbix is a "batteries included" solution — agent + server + DB + UI in one package, which is
  its main practical advantage over Prometheus+Grafana. Easier initial setup, built-in alerting,
  no separate Alertmanager. Downside: more opinionated, harder to integrate with custom app metrics
  vs Prometheus's pull model.]

3. Nagios
3.1 Pros: the most probably known and loved by many administrators, can capture any data, lightweight;
3.2 Cons: i don't know nothing about data visualization, may be Nagios is just data collector and visualization being made by for example Grafana? disadvantage also in that this is very complex solution, hard to tune, hard to maintain

[AGENT NOTE — Nagios clarification:
  Nagios Core does have its own built-in web UI (CGI-based, classic green/red status board), but it
  is very basic — good for status/alerting, poor for time-series graphs.
  Nagios is NOT a time-series data collector — it is a status/alert engine (checks return OK/WARNING/
  CRITICAL states). For historical graphs, Nagios is typically paired with PNP4Nagios or Graphite, NOT
  Prometheus/Grafana directly (though adapters exist).
  Modern consensus: Nagios has been largely superseded by Prometheus+Alertmanager for metrics-based
  monitoring. Nagios Core is still used for pure availability checks (is the host/port up?) but the
  ecosystem has moved on. Suggest downgrading this option to "legacy/not recommended for new setups".
  Nagios XI (commercial) is polished but not free.
  If lightweight availability-only checks are needed, Uptime Kuma is a modern self-hosted alternative
  to Nagios with a much better UI and zero configuration overhead.]

4. ... TBD, any suggestions

[AGENT NOTE — additional options worth considering:
  4. Prometheus + Grafana (recommended for this cluster, see notes above)
     - node_exporter: CPU/RAM/disk/NIC per VM
     - blackbox_exporter: HTTP/TCP service health probes
     - mysqld_exporter: MariaDB replication lag, slow queries, connections
     - Alertmanager: route alerts to email + Telegram/Slack
     - Grafana: dashboards (community templates available for all the above)
     - VictoriaMetrics (optional): replace Prometheus TSDB for lower resource use on 8GB nodes
  5. Netdata (https://netdata.cloud)
     - Installs as a single agent per node (apt/apk/bash one-liner)
     - Gives real-time per-node dashboard out of the box — zero configuration for hardware metrics
     - Can stream to a central Netdata parent node
     - Netdata Cloud (free tier) aggregates cross-node view without self-hosting a central stack
     - Useful for quick bootstrapping before a full Prometheus stack is deployed
     - License: GPL v3
  6. Uptime Kuma (https://github.com/louislam/uptime-kuma)
     - Lightweight self-hosted service uptime monitor (HTTP/TCP/DNS/ping checks)
     - Very easy to deploy (Docker single container)
     - Covers "is service X responding?" with beautiful status page
     - Good complement to Prometheus (Prometheus = deep metrics, Uptime Kuma = simple uptime)
     - License: MIT
  7. ELK / OpenSearch (for logs, not metrics)
     - Elasticsearch + Logstash/Fluent Bit + Kibana/OpenSearch Dashboards
     - Best for centralized log aggregation and search (nginx access logs, php-fpm error logs, syslog)
     - Heavy (needs 4+ GB RAM for Elasticsearch alone) — probably overkill for current lab
     - Grafana Loki is a lighter alternative for log aggregation that integrates natively with Grafana]
[Anton's answer: cool, I like everything proposed there]

5. Mixed (hybrid), like Grafana + Zabbix
... TBD

[AGENT NOTE — hybrid recommendation:
  Based on the project's existing infra (Alpine VMs, MariaDB, nginx, PHP-FPM, Xen, 8GB RAM/node)
  and what's already discussed in GRAFANA_vs_others_graphvis_pivot.md, my suggestion for SPEC:

  RECOMMENDED STACK (R0):
    Infra layer: Prometheus (or VictoriaMetrics) + node_exporter on each VM
                 + blackbox_exporter for HTTP/TCP probes
                 + mysqld_exporter for MariaDB
                 + Alertmanager → email (required) + Telegram bot (optional)
    Dashboard:   Grafana (community dashboards for node_exporter + MariaDB)
    Uptime:      Uptime Kuma on a separate lightweight VM or the FE (external to cluster)
    Deployment:  ac-deploy (new monitoring/ role in cluster0 scripts)

  FUTURE (R1):
    Log aggregation: Grafana Loki + Promtail (lighter than ELK)
    Auto-remediation: custom event handler scripts triggered by Alertmanager webhooks
    Business metrics: Grafana queries TimescaleDB (already in project plan)

  OPEN QUESTION FOR PO: Should the monitoring stack get its own VM node (4th node), or run
  on one of the 3 cluster nodes (and which one)? This affects DR decision §2.]
[Anton's answer: good, so we agreed on most of the points there, i also concluded this combination as the better one for myself. I think I've addressed almost all the questions. the Q from my side:
imagine I've allocated yet another VM and put it inside cluster as an empty node (only os-basic software is installed)
  - do I need something from DNS hosting provider, like CNAME records, additional TXT records maybe?
  - do we need a kind of tuning of the existing FE configs (nginx which is a forwarder there, for example)
  - ImprovMX, addy.io, Telegram bot where? internally deployed inside the cluster or externally used by sort of API? need to create accounts for the document target feature? on which services, just to be ready before we go?
]


[Anton's nodes:
just to remind, we're limited on this on the server's side:
  - machine type: next type will be likely paravirtualized (no HVM)
  - OS type: Alpine latest LTS (or Edge, but manually configured changing the repos)
  - software: likely from the Alpine community packages/ package manager; meaning we'll try to avoid any source-based software, preferring apk's built for the particular Alpine
  - FE/router: unlikely to install the software there, almost untouchable; we can play just with the configuration, like adding routes, playing with iptables, etc. with the existing software, but no new installations there
]