mirror of
git://f0xx.org/ac/ac-docs
synced 2026-07-29 02:58:15 +03:00
73 lines
4.3 KiB
Plaintext
73 lines
4.3 KiB
Plaintext
Cluster self-healing and automated orchestration draft (Track B)
|
|
milestone: R1+
|
|
severity: high
|
|
state: placeholder — not for immediate implementation; referenced by monitoring SPEC (Track A)
|
|
author: Anton Afanasyeu
|
|
date: 2026-06-26
|
|
related to: 20260626_cluster_monitoring*, 20260618_repos_reorganizing*, microservice split
|
|
|
|
Track A (prerequisite, must be complete first):
|
|
- Full passive monitoring + alerting operational (see specs/20260626_cluster_monitoring.md)
|
|
- Alert routing proven stable (email + Telegram)
|
|
- At least 4 weeks of baseline metrics collected (needed to calibrate thresholds)
|
|
|
|
The idea:
|
|
Once the monitoring layer (Track A) is stable and producing reliable alert signals, the next step is
|
|
to make the cluster respond to those signals automatically — rather than requiring a human to act on
|
|
every alert. This is commonly called cluster self-healing, auto-remediation, or a "smart hypervisor"
|
|
layer. It is NOT part of the monitoring SPEC; it is an orchestration concern.
|
|
|
|
Target scenarios (desired future automation):
|
|
|
|
Scenario 1 — CPU overload → horizontal scale-out
|
|
Trigger: microservice X CPU utilization > threshold for N minutes (Alertmanager signal)
|
|
Action: provision new node Y (via Xen/cloud API), deploy microservice X replica on it,
|
|
update load balancer / nginx upstream, re-route portion of traffic
|
|
Complexity: HIGH — requires VM provisioning API + dynamic nginx config reload
|
|
|
|
Scenario 2 — Idle scale-in (cost saving)
|
|
Trigger: microservice X CPU < threshold for Y hours AND overall cluster load < 75%
|
|
Action: gracefully drain microservice X instance, deprovision node, update upstream config
|
|
Complexity: HIGH — requires graceful drain protocol per microservice type
|
|
|
|
Scenario 3 — DDoS / network spike response
|
|
Trigger: NIC ingress/egress > threshold OR nginx 5xx rate spikes (Alertmanager signal)
|
|
Sub-action A (DDoS suspected): block suspect IPs, rate-limit at iptables, notify Anton
|
|
Sub-action B (organic spike): scale out cluster capacity, increase bandwidth (notify Anton for
|
|
bandwidth tier change — cannot be auto-done from cluster)
|
|
Complexity: VERY HIGH — requires traffic analysis, IP reputation, coordinated with upstream
|
|
|
|
Scenario 4 — Storage fault recovery
|
|
Trigger: disk SMART pre-failure / disk write latency > threshold on node Y
|
|
Action: elect replica to master (if MariaDB GTID replication is active), notify Anton to
|
|
provision spare node, re-sync replica on new node
|
|
Complexity: HIGH — MariaDB GTID failover + new node provisioning
|
|
|
|
Common prerequisites for all scenarios:
|
|
1. Track A monitoring + alerting fully operational (Alertmanager webhook available)
|
|
2. VM provisioning API (Xen xl toolstack or cloud provider API with automation credentials)
|
|
3. Configuration management (Ansible / ac-deploy role) for dynamic service placement
|
|
4. Graceful drain protocol defined per microservice (health endpoint + connection draining)
|
|
5. Load balancer / nginx upstream reconfiguration automation (e.g. consul-template or custom script)
|
|
6. Decision logic (thresholds, hysteresis, human-in-the-loop gates vs fully-automatic)
|
|
|
|
Approach options (TBD — to be designed in R1 DR):
|
|
a) Kubernetes operators (standard industry path; requires migrating to K8s)
|
|
b) Ansible event-driven (Ansible EDA) — react to Alertmanager webhooks with playbooks
|
|
c) Home-grown orchestration scripts called by Alertmanager webhook receiver on cast04
|
|
d) HashiCorp Nomad + Consul (lighter than K8s; service placement + dynamic upstream config)
|
|
|
|
Open questions (for R1 DR):
|
|
- Does Anton's infrastructure support Xen VM provisioning via API / xl scripting?
|
|
- What is the minimum cluster node count for safe scale-in (must keep >= N nodes)?
|
|
- Is Kubernetes acceptable given paravirt constraint (k3s on Alpine paravirt is feasible)?
|
|
- What is the human-in-the-loop policy: always require approval for scale-out/in, or fully
|
|
automatic within defined bounds?
|
|
- Cost model for scale-out: how are additional nodes billed / what is the budget ceiling?
|
|
|
|
Related documents:
|
|
- Track A: docs/DRs/20260626_cluster_monitoring.md + docs/specs/20260626_cluster_monitoring.md
|
|
- Scaling SPEC: docs/specs/20100612_1_scaling.md
|
|
- Repo reorganization SPEC: docs/specs/20260618_repos_reorganizing.md
|
|
- ac-deploy cluster scripts: orchestration/sim/cluster0/
|