ac/ac-docs
1
0
Fork 0
mirror of git://f0xx.org/ac/ac-docs synced 2026-07-29 05:18:46 +03:00
Code Issues Packages Projects Releases Wiki Activity

new DR and spec for cluster monitoring, docs updated

Browse Source
This commit is contained in:
Anton Afanasyeu
2026-06-26 19:47:17 +02:00
parent df726f5993
commit aab2b8512d
53 changed files with 5505 additions and 737 deletions
Download Patch File Download Diff File Expand all files Collapse all files

321
drafts/20260626_cluster_monitoring.txt
View File

@@ -1,7 +1,8 @@
Infrastructure health monitoring draft
milestone: R0-pre
severity: high
state: draft (annotated — see [AGENT NOTE] blocks; please review before DR/SPEC conversion)
state: closed — DR/SPEC generated; see DRs/20260626_cluster_monitoring.md + specs/20260626_cluster_monitoring.md
Track B (orchestration) → drafts/20260626_cluster_orchestration.txt
author: Anton Afanasyeu
date: 2026-06-26
related to: 20260618_repos_reorganizing*, microservice split, clustered CI/CD
@@ -28,95 +29,29 @@ sometimes it may be very complex, for example
|-------------------[ VM A master db node ]----X----
|---------[ VM B replica db node ]---|
with such a floating architecture, especially if we gonna use on-demand cluster scaling, there's very hard to perform manual watch of each service or node. even though, good to have a kind of smart hypervisor over all our microservices and cluster nodes that will make a decision what to do next in case, for example:
- the microservice X experiences high CPU load, needs to ask CPU load balancer to allocate quickly another node Y and redirect a part of microservice's X requests to new node Y
- the microservice X stays untouched for a last Y hours and the cluster's overall load is less than 75% during this time - hypervisor makes a decision to stop redundant instance, simply to save money
- the network pressure is superbly high - investigate why, if there's hackers DDoS attack - perform set of actions for DDoS protection, if there's simply spike because a lots of users(customers) - do actions to increase cluster size and bandwidth, etc.
- the data storage Y flied away because of HW damage - allocate the replica of storage Y and put it instead of broken one to the cluster
etc.
[AGENT NOTE — auto-remediation scope:
Everything above (auto-scale, auto-DDoS-protect, auto-failover storage) is NOT monitoring — it is
automated orchestration / cluster self-healing. These are valuable long-term goals but they are an
entirely separate and much larger engineering track (think: Kubernetes operators, custom controllers,
Ansible event-driven, or a home-grown orchestration plane). Mixing them into the monitoring spec
risks scope-creep and may block the R0 delivery.
SUGGESTION: Split into two tracks explicitly in the SPEC:
Track A (this SPEC, R0): passive monitoring + alerting — observe, record, notify
Track B (future, R1+): active remediation — orchestration reacts to monitoring events
The DR should decide whether "smart hypervisor" is a Track B placeholder or in-scope for R0.
My recommendation: Track B placeholder — list the scenarios as "desired future automation" but
do not design them in the monitoring SPEC.]
[Anton's answer: agree with suggestion. please help me to organize Track B as a separate draft document as you see it; make sure it would be referenced in Track A like a next-gen step of supervising over the infra, clean up everything from exactly this draft that refers to Track B, so we'll expect Track B will not be forgotten]
[TRACK B — auto-remediation and auto-scaling scenarios moved to drafts/20260626_cluster_orchestration.txt]
Basically, the most important set of monitored events may be:
- hardware health monitoring
-- node / microservice availability
-- ... TBD
[AGENT NOTE — hardware monitoring gaps worth filling in TBD:
Standard items for Xen/KVM VM nodes (project already runs XEN HVM/paravirt):
-- CPU: utilization %, steal time (critical for Xen — if hypervisor steals CPU, steal > 10% is a red flag)
-- RAM: used / available / swap usage
-- Disk: utilization %, IOPS, write latency (key for MariaDB nodes), disk SMART via smartmontools
-- NIC: ingress/egress bytes/sec, packet drops, errors
-- Load average (1m/5m/15m)
-- Xen-specific: dom0 balloon driver state, vcpu credit scheduler stats
Suggest confirming which of these you want in SPEC §3 (requirements table).]
[Anton's answer: you have clearer vision on that rather than me, i'm not a devops, please help to reconstruct/organize the doc that way, i can just confirm that again your suggestion is more structured; i would say it's better to keep an eye over all of the metrics, but up to you there]
-- CPU: utilization %, steal time (Xen-critical), load average 1m/5m/15m
-- RAM: used / available / swap usage
-- Disk: utilization %, IOPS, write latency, SMART health
-- NIC: ingress/egress bytes/sec, packet drops, errors
-- Xen guest: vcpu steal, balloon driver state
- software health monitoring
-- node / microservice availability
-- the reason
--- scale needed
--- unsuccessful configuration
--- software not getting up
--- out of resources
-- ... TBD
[AGENT NOTE — software monitoring gaps worth filling:
Standard per-service items:
-- HTTP health endpoint response time + status code (blackbox probe, e.g. /health or /ping)
-- Process presence check (is nginx/php-fpm/mariadb PID alive?)
-- Log error rate (ERROR/CRITICAL lines per minute from syslog/journald)
-- MariaDB: replication lag (seconds_behind_master), connection count, slow query count
-- nginx: 5xx rate, upstream connection errors
-- php-fpm: queue depth, worker saturation
These map naturally to exporters (see tool section note below).]
[Anton's answer: agreed]
-- node / microservice availability (process presence + HTTP health probe)
-- HTTP response time and status code per service endpoint
-- log error rate (ERROR/CRITICAL lines per minute, syslog/journald)
-- MariaDB: replication lag (seconds_behind_master), connection count, slow query count
-- nginx: 5xx rate, upstream connection errors
-- php-fpm: queue depth, worker saturation
-- cause classification: scale needed / bad config / process crash / OOM
- provider's fault
-- hardware or software
-- service absense
-- payment expired
-- ... TBD
[AGENT NOTE — provider fault detection:
This category requires EXTERNAL monitoring — probes from outside your network to verify
that apps.f0xx.org is reachable from the public internet. Internal monitoring is blind to:
- Anton's router/uplink going down
- XEN host hardware failure (the VM's own monitoring dies with it)
- BGP/routing issues upstream
Options for external check:
a) UptimeRobot / Freshping / BetterUptime free tier (simple HTTP check from cloud)
b) Self-hosted Uptime Kuma on a VPS outside this network (cheapest independent probe)
c) Blackbox Exporter + Prometheus alerting IF the Prometheus instance is outside the cluster
"Payment expired" can't really be auto-detected; it's an ops reminder / calendar alert.
Suggest clarifying in SPEC: external uptime check is REQUIRED for provider-fault detection,
and it must live OUTSIDE the monitored cluster.]
[Anton's answer: unfortunately there are some sw limits that may probably being a soft-blockers, for example (using current architecture):
- we cannot deeply change the set of software on the router
- we cannot change the software on FE side as well
- we can rely and fully operate only with cluster/nodes
future, renting the true cloud infrastructure, AWS, GClouds, Azure, etc.:
- also some risks/limitations on changing top-level software because owned by cloud provider
- but probably it's a kind of 'hackable', don't know - newer rented foreign cloud infra before
think about it
]
- ... TBD
-- Xen host / uplink availability detected via external probe (UptimeRobot, no install)
-- internal: cast04 blackbox_exporter checks FE endpoint from inside cluster
-- software/service absence, payment expiry: ops calendar + manual review (not auto-detectable)
a simple example, using the current lab infrastructure:
[global net x.x.x.x] <-> [(134.17.26.161 external IP) router (debian-based] (internal IP 10.7.6.228) ] <-> [(10.7.0.10 intra network 1) FE (intra network 2: 10.7.0.0/8)] [intra network 3: ...]
@@ -127,178 +62,60 @@ the owner of top-level services is Anton Afanasyeu. top level services include:
- basic on-demand OS installation, on-demand VM snapshots
----------- all up is SaaS, for all the questions contact owner ---------
-- cluster of VM's, XEN-based (hvm or paravirt, depends)
-- cluster of VM's, XEN-based (paravirt, Alpine latest LTS)
-- VM's control, VM's internal software installation, access, aggregation, fine tunings, software development, ssh, internal setups, etc.
----------- the last few lines are totally controllable by developer, developed product owner and projects workers as decided by product owner ---------
the VM delegated cluster owner must set its own monitoring software where it needs to be placed and monitor all the parameters he decided to keep an eye on, like:
1) hardware resources load, per cluster node and overall in-cluster, comparing the max. values delegated by Anton Afanasyeu according to payment fee for the cluster; like Anton guarantees that CPU will be real (not emulated) and it will be Intel 8 cores of Gold 5218 CPU @ 2.30GHz series, 8Gb of RAM per node, 32Gb of non-shared disk space per node, 3 nodes in cluster, 200Mbit in/out internet, etc; the project owner should have monitoring based of the extremum of each of parameters, compare with proposed plan, alert of some of values differ than what Anton guaranteed
2) project owner took 3 nodes / cluster; after month of heavy load one of the nodes show instabilities and slow data writes using distributed among the nodes MariaDb; MariaDb monitoring show exactly that Node B contains the replica of master db node and it's possible that there's a hardware fault (disk writes), so he should notify Anton to take it into account and provide spare node
3) slow egress/ingress -> delegate to Anton
4) one of project services or its relatives aren't responding well - there should be per-service self monitoring and should raise an alert if so
[AGENT NOTE — monitoring placement (unanswered design question):
The draft does not address where the monitoring stack itself lives. This is a key SPEC decision:
Option A: Dedicated monitoring VM (4th node, outside the 3-node cluster)
+ monitoring survives a cluster node failure
- costs an extra VM
Option B: On one of the 3 existing cluster nodes (e.g. the "ops" node)
- if that node fails, monitoring goes dark — worst case: you're blind exactly when you need it
+ no extra cost
Option C: On Anton's infra (outside delegated cluster entirely)
+ most resilient for node-level failure detection
- requires coordination with Anton; may not be acceptable for project-owner autonomy
Option D: Lightweight agents on each node, reporting to a cloud-hosted aggregator (e.g. Grafana Cloud
free tier, or any external SaaS)
+ zero infra overhead; survives full cluster outage
- data leaves your network
SUGGESTION: Decide and lock in SPEC §2 (topology). My recommendation: Option A or D depending
on budget constraints. Option B is acceptable for lab/pre-alpha only.]
[Anton's answer:
Option A: currently doable, having full control over hypervised VMs; dedicated machine isn't something that can block us
Option B: likely the chosed one for now, once we on Anton's provided resources but a bit limited on their capacity; thinking about A and B what to choose
Option C: good, but no. prefer to keep everything together until going to real clouds
Option D: good about lightweight agents, but don't want to track the data somewhere outside; prefer to keep everything single-hosted
MONITORING PLACEMENT DECISION: Option A — cast04 (dedicated 4th Alpine VM, monitoring-only node).
Monitoring stack lives on cast04, isolated from the 3 workload nodes cast01/cast02/cast03.
Router and FE are config-only (no new software); monitoring uses cluster-internal network only.
Now i think option A is the best from robust and future knowledge, although, probably, the hardest in implementation. if another Alpine instance (cast04 as 4th node) will take care of monitoring, I'm fine with option A
]
The how:
the actor (any of) may be triggered on faulty node, cluster, service, ..., state, first off, by email. any other services are also likely to have (messengers, SMS, etc.), but anyway we should integrate some monitoring visualizer(s):
1) Visualization of the whole infrastructure
2) Visualization per cluster and per Node health
3) Visualization per service provided by distro used inside the each Node
4) Visualization per service of project's software inside the each Node
based on the requirements, a few almost-ready solutions touch the souls:
1. Grafana (Kibana)
[AGENT NOTE — Grafana ≠ Kibana — FACTUAL CORRECTION:
These are two completely different products from different companies and stacks:
- Grafana (grafana.com): standalone visualization layer; plugs into Prometheus, InfluxDB,
Loki, TimescaleDB/PostgreSQL, etc. Self-hosted or Grafana Cloud.
- Kibana: the "K" in the ELK stack (Elasticsearch + Logstash + Kibana). Primarily a log
analysis/search UI over Elasticsearch indices. NOT typically used for infra metrics.
The project already has a Grafana pivot analysis doc: GRAFANA_vs_others_graphvis_pivot.md.
They should be listed and evaluated separately. My suggestion: drop Kibana from this section
(ELK is heavy and log-centric, not the primary choice for VM metrics), or add it as a separate
option specifically for centralized log aggregation (not metrics dashboards).]
[Anton's answer: ok, let's exclude Kibana at all from our goals/perspectives]
1.1 Pros: the most beautiful and user-friendly; pretty easy to play with pre-stored parameters, building customized graphics; may be self-hosted; may reflect anything you want, but the administrator's eye is required on it, at least on the first steps
1.2 Cons: requires additional software, like Prometheus (the most commonly used) or maybe other time-series based engine; so flexible that it is sometimes unbelievably hard to create pre-stored parameters set to visualize it later; requires also additional data converters/plugins, Prometheus-friendly
[AGENT NOTE — Prometheus deserves its own section:
Prometheus is the core of the recommended stack for VM cluster monitoring and should be called
out explicitly, not just as a "Grafana requirement". The stack is:
Prometheus (time-series DB + scraper) + node_exporter (per VM hardware metrics)
+ blackbox_exporter (HTTP/TCP service probes) + Alertmanager (routing alerts to email/Slack/etc.)
+ Grafana (dashboards on top)
Each component is independently useful and has a clear role. Ready-made "community dashboards"
exist in Grafana for node_exporter (dashboard ID 1860 — "Node Exporter Full") and MariaDB.
Also worth mentioning: VictoriaMetrics as a drop-in Prometheus-compatible TSDB — uses 5-10x
less RAM/disk, easier to run on a small cluster, retains data longer by default. If the cluster
nodes are small (8GB RAM per node), VictoriaMetrics may be preferable to Prometheus itself.
Relationship to existing project TimescaleDB platform: Prometheus/VictoriaMetrics is for INFRA
metrics (CPU/RAM/disk/latency). TimescaleDB is proposed for APPLICATION/BUSINESS metrics
(crash rates, session counts, graph analytics). These can coexist — different stores, different
retention, different queries. Grafana queries both. Clarify this split in the SPEC.]
[Anton's answer: good to know,i just want you again to re-read the soft limitations i've described above; pressure is that we cannot somehow install new software neither on router nor on FE unfortunately; if there's no need, then just OK; if need is not intended as mandatory - also ok, we just lose some of metrics, which is currently fine)
2. Zabbix
2.1 Pros: good UI/UX, pretty easy to deploy; good support, good abilities;
2.2 Cons: not as good as Grafana+... from the UI/UX perspective
2.3 Need to check the licensing
[AGENT NOTE — Zabbix licensing:
Zabbix Community Edition is GPL v2 licensed — free for any use. No commercial license needed
for self-hosted monitoring of your own infrastructure. The Zabbix Agent on monitored nodes is
also LGPL. This concern is resolved; not a blocker.
Zabbix is a "batteries included" solution — agent + server + DB + UI in one package, which is
its main practical advantage over Prometheus+Grafana. Easier initial setup, built-in alerting,
no separate Alertmanager. Downside: more opinionated, harder to integrate with custom app metrics
vs Prometheus's pull model.]
3. Nagios
3.1 Pros: the most probably known and loved by many administrators, can capture any data, lightweight;
3.2 Cons: i don't know nothing about data visualization, may be Nagios is just data collector and visualization being made by for example Grafana? disadvantage also in that this is very complex solution, hard to tune, hard to maintain
[AGENT NOTE — Nagios clarification:
Nagios Core does have its own built-in web UI (CGI-based, classic green/red status board), but it
is very basic — good for status/alerting, poor for time-series graphs.
Nagios is NOT a time-series data collector — it is a status/alert engine (checks return OK/WARNING/
CRITICAL states). For historical graphs, Nagios is typically paired with PNP4Nagios or Graphite, NOT
Prometheus/Grafana directly (though adapters exist).
Modern consensus: Nagios has been largely superseded by Prometheus+Alertmanager for metrics-based
monitoring. Nagios Core is still used for pure availability checks (is the host/port up?) but the
ecosystem has moved on. Suggest downgrading this option to "legacy/not recommended for new setups".
Nagios XI (commercial) is polished but not free.
If lightweight availability-only checks are needed, Uptime Kuma is a modern self-hosted alternative
to Nagios with a much better UI and zero configuration overhead.]
4. ... TBD, any suggestions
[AGENT NOTE — additional options worth considering:
4. Prometheus + Grafana (recommended for this cluster, see notes above)
- node_exporter: CPU/RAM/disk/NIC per VM
- blackbox_exporter: HTTP/TCP service health probes
- mysqld_exporter: MariaDB replication lag, slow queries, connections
- Alertmanager: route alerts to email + Telegram/Slack
- Grafana: dashboards (community templates available for all the above)
- VictoriaMetrics (optional): replace Prometheus TSDB for lower resource use on 8GB nodes
5. Netdata (https://netdata.cloud)
- Installs as a single agent per node (apt/apk/bash one-liner)
- Gives real-time per-node dashboard out of the box — zero configuration for hardware metrics
- Can stream to a central Netdata parent node
- Netdata Cloud (free tier) aggregates cross-node view without self-hosting a central stack
- Useful for quick bootstrapping before a full Prometheus stack is deployed
- License: GPL v3
6. Uptime Kuma (https://github.com/louislam/uptime-kuma)
- Lightweight self-hosted service uptime monitor (HTTP/TCP/DNS/ping checks)
- Very easy to deploy (Docker single container)
- Covers "is service X responding?" with beautiful status page
- Good complement to Prometheus (Prometheus = deep metrics, Uptime Kuma = simple uptime)
- License: MIT
7. ELK / OpenSearch (for logs, not metrics)
- Elasticsearch + Logstash/Fluent Bit + Kibana/OpenSearch Dashboards
- Best for centralized log aggregation and search (nginx access logs, php-fpm error logs, syslog)
- Heavy (needs 4+ GB RAM for Elasticsearch alone) — probably overkill for current lab
- Grafana Loki is a lighter alternative for log aggregation that integrates natively with Grafana]
[Anton's answer: cool, I like everything proposed there]
5. Mixed (hybrid), like Grafana + Zabbix
... TBD
[AGENT NOTE — hybrid recommendation:
Based on the project's existing infra (Alpine VMs, MariaDB, nginx, PHP-FPM, Xen, 8GB RAM/node)
and what's already discussed in GRAFANA_vs_others_graphvis_pivot.md, my suggestion for SPEC:
RECOMMENDED STACK (R0):
Infra layer: Prometheus (or VictoriaMetrics) + node_exporter on each VM
+ blackbox_exporter for HTTP/TCP probes
+ mysqld_exporter for MariaDB
+ Alertmanager → email (required) + Telegram bot (optional)
Dashboard: Grafana (community dashboards for node_exporter + MariaDB)
Uptime: Uptime Kuma on a separate lightweight VM or the FE (external to cluster)
Deployment: ac-deploy (new monitoring/ role in cluster0 scripts)
FUTURE (R1):
Log aggregation: Grafana Loki + Promtail (lighter than ELK)
Auto-remediation: custom event handler scripts triggered by Alertmanager webhooks
Business metrics: Grafana queries TimescaleDB (already in project plan)
OPEN QUESTION FOR PO: Should the monitoring stack get its own VM node (4th node), or run
on one of the 3 cluster nodes (and which one)? This affects DR decision §2.]
[Anton's answer: good, so we agreed on most of the points there, i also concluded this combination as the better one for myself. I think I've addressed almost all the questions. the Q from my side:
imagine I've allocated yet another VM and put it inside cluster as an empty node (only os-basic software is installed)
- do I need something from DNS hosting provider, like CNAME records, additional TXT records maybe?
- do we need a kind of tuning of the existing FE configs (nginx which is a forwarder there, for example)
- ImprovMX, addy.io, Telegram bot where? internally deployed inside the cluster or externally used by sort of API? need to create accounts for the document target feature? on which services, just to be ready before we go?
]
External check for provider-fault: UptimeRobot free tier (web registration only, no install anywhere).
[Anton's nodes:
just to remind, we're limited on this on the server's side:
- machine type: next type will be likely paravirtualized (no HVM)
- OS type: Alpine latest LTS (or Edge, but manually configured changing the repos)
- software: likely from the Alpine community packages/ package manager; meaning we'll try to avoid any source-based software, preferring apk's built for the particular Alpine
- FE/router: unlikely to install the software there, almost untouchable; we can play just with the configuration, like adding routes, playing with iptables, etc. with the existing software, but no new installations there
]
The how — DECIDED STACK:
INFRA METRICS (cast04):
- prometheus + node_exporter on cast01/cast02/cast03/cast04 (apk add)
- prometheus-blackbox-exporter on cast04 for HTTP/TCP service probes (apk add)
- prometheus-mysqld-exporter on MariaDB node (apk add)
- alertmanager on cast04 → email (SMTP relay) + Telegram bot (API only)
- grafana on cast04 (apk add); community dashboards for node_exporter + MariaDB
UPTIME STATUS PAGE (cast04, Docker):
- Uptime Kuma (Docker) on cast04 for a clean per-service status page
- Docker itself available via apk
EXTERNAL AVAILABILITY:
- UptimeRobot free tier — no install, web registration, HTTP probe to apps.f0xx.org
- Alerting via email from UptimeRobot
NOT CHOSEN (and why):
- Kibana/ELK: excluded (log-centric, heavy, wrong tool for infra metrics)
- Zabbix: licensing OK but less flexible for future business metrics integration vs Grafana
- Nagios: legacy, superseded by Prometheus+Alertmanager for this use case
- VictoriaMetrics: not in Alpine apk repos; use standard Prometheus at this scale
- Grafana Cloud / external SaaS aggregator: data must stay self-hosted (Anton's decision)
FUTURE (Track B, R1+):
- Grafana Loki + Promtail for centralized log aggregation
- Alertmanager webhook handlers for auto-remediation triggers
- Grafana queries TimescaleDB for business/application metrics (complement, not replace Prometheus)
CONSTRAINTS (Alpine notes):
- Machine: paravirtualized XEN guest (no HVM)
- OS: Alpine latest LTS (community + edge repos)
- Software: apk packages only; Docker available via apk for Uptime Kuma
- FE/router: no software installation; config-only (iptables, routes, existing software)
PRE-REQUISITES (accounts to create before deploy):
- Email alert relay: Gmail account + app password, OR Mailgun free tier (outbound SMTP only)
NOTE: ImprovMX and addy.io are inbound email routing services — NOT usable as SMTP relay for Alertmanager
- Telegram alerts: @BotFather → create bot → get TOKEN + CHAT_ID (add bot to ops group)
No software install; Alertmanager calls Telegram API from cast04
- UptimeRobot: free account at uptimerobot.com → add HTTP monitor for apps.f0xx.org
DNS / FE NGINX:
- R0: no DNS changes, no FE nginx changes required
- cast04 monitoring stack is internal-only (access via SSH tunnel or WireGuard from dev machine)
- Future optional: FE nginx proxy rule (config-only) to expose Grafana at e.g. /monitoring/ externally

72
drafts/20260626_cluster_orchestration.txt Normal file
View File

@@ -0,0 +1,72 @@
Cluster self-healing and automated orchestration draft (Track B)
milestone: R1+
severity: high
state: placeholder — not for immediate implementation; referenced by monitoring SPEC (Track A)
author: Anton Afanasyeu
date: 2026-06-26
related to: 20260626_cluster_monitoring*, 20260618_repos_reorganizing*, microservice split
Track A (prerequisite, must be complete first):
- Full passive monitoring + alerting operational (see specs/20260626_cluster_monitoring.md)
- Alert routing proven stable (email + Telegram)
- At least 4 weeks of baseline metrics collected (needed to calibrate thresholds)
The idea:
Once the monitoring layer (Track A) is stable and producing reliable alert signals, the next step is
to make the cluster respond to those signals automatically — rather than requiring a human to act on
every alert. This is commonly called cluster self-healing, auto-remediation, or a "smart hypervisor"
layer. It is NOT part of the monitoring SPEC; it is an orchestration concern.
Target scenarios (desired future automation):
Scenario 1 — CPU overload → horizontal scale-out
Trigger: microservice X CPU utilization > threshold for N minutes (Alertmanager signal)
Action: provision new node Y (via Xen/cloud API), deploy microservice X replica on it,
update load balancer / nginx upstream, re-route portion of traffic
Complexity: HIGH — requires VM provisioning API + dynamic nginx config reload
Scenario 2 — Idle scale-in (cost saving)
Trigger: microservice X CPU < threshold for Y hours AND overall cluster load < 75%
Action: gracefully drain microservice X instance, deprovision node, update upstream config
Complexity: HIGH — requires graceful drain protocol per microservice type
Scenario 3 — DDoS / network spike response
Trigger: NIC ingress/egress > threshold OR nginx 5xx rate spikes (Alertmanager signal)
Sub-action A (DDoS suspected): block suspect IPs, rate-limit at iptables, notify Anton
Sub-action B (organic spike): scale out cluster capacity, increase bandwidth (notify Anton for
bandwidth tier change — cannot be auto-done from cluster)
Complexity: VERY HIGH — requires traffic analysis, IP reputation, coordinated with upstream
Scenario 4 — Storage fault recovery
Trigger: disk SMART pre-failure / disk write latency > threshold on node Y
Action: elect replica to master (if MariaDB GTID replication is active), notify Anton to
provision spare node, re-sync replica on new node
Complexity: HIGH — MariaDB GTID failover + new node provisioning
Common prerequisites for all scenarios:
1. Track A monitoring + alerting fully operational (Alertmanager webhook available)
2. VM provisioning API (Xen xl toolstack or cloud provider API with automation credentials)
3. Configuration management (Ansible / ac-deploy role) for dynamic service placement
4. Graceful drain protocol defined per microservice (health endpoint + connection draining)
5. Load balancer / nginx upstream reconfiguration automation (e.g. consul-template or custom script)
6. Decision logic (thresholds, hysteresis, human-in-the-loop gates vs fully-automatic)
Approach options (TBD — to be designed in R1 DR):
a) Kubernetes operators (standard industry path; requires migrating to K8s)
b) Ansible event-driven (Ansible EDA) — react to Alertmanager webhooks with playbooks
c) Home-grown orchestration scripts called by Alertmanager webhook receiver on cast04
d) HashiCorp Nomad + Consul (lighter than K8s; service placement + dynamic upstream config)
Open questions (for R1 DR):
- Does Anton's infrastructure support Xen VM provisioning via API / xl scripting?
- What is the minimum cluster node count for safe scale-in (must keep >= N nodes)?
- Is Kubernetes acceptable given paravirt constraint (k3s on Alpine paravirt is feasible)?
- What is the human-in-the-loop policy: always require approval for scale-out/in, or fully
automatic within defined bounds?
- Cost model for scale-out: how are additional nodes billed / what is the budget ceiling?
Related documents:
- Track A: docs/DRs/20260626_cluster_monitoring.md + docs/specs/20260626_cluster_monitoring.md
- Scaling SPEC: docs/specs/20100612_1_scaling.md
- Repo reorganization SPEC: docs/specs/20260618_repos_reorganizing.md
- ac-deploy cluster scripts: orchestration/sim/cluster0/