diff --git a/drafts/20260626_cluster_monitoring.txt b/drafts/20260626_cluster_monitoring.txt new file mode 100644 index 0000000..4842697 --- /dev/null +++ b/drafts/20260626_cluster_monitoring.txt @@ -0,0 +1,102 @@ +Infrastructure health monitoring draft +milestone: R0-pre +severity: high +state: draft +author: Anton Afanasyeu +date: 2026-06-26 +related to: 20260618_repos_reorganizing*, microservice split, clustered CI/CD + +The problem: +Actually there's no problem while everything is working. But as many know, sometimes there's a time where the software fails, hardware tires, and vice versa. +The goal is to do constant infrastructure health monitoring, probably even with prediction warnings, if possible + + +The why: +1) now we do have a lots of virtual machines (will be more if we move forward to real clouds) and lots of microservices splitted around the virtual machines. Sometimes the relation between machine and microservice is pretty straight forward: +| microservice A | -> | VM A | -> ... networking +sometimes more robust, but with additional cost of redundancy, like: +| microservice A | -> | VM B database frontend | + |------> [ VM A master db node ]----------> [aggregator + frontent]-----> ... networking + |------> [ VM C replica db node ]-----^ +sometimes it may be very complex, for example +| microservice A (master) | + |------------- [delegation A.A, slave ] + | |-------------------[ VM A master db node ]-----|--------->[aggregator + frontend]------> ... next pipeline ... + | |---------[ VM C replica db node ]----X----^ + | | | + |------------- [delegation A.B, slave ] | | + |-------------------[ VM A master db node ]----X---- + |---------[ VM B replica db node ]---| + +with such a floating architecture, especially if we gonna use on-demand cluster scaling, there's very hard to perform manual watch of each service or node. even though, good to have a kind of smart hypervisor over all our microservices and cluster nodes that will make a decision what to do next in case, for example: +- the microservice X experiences high CPU load, needs to ask CPU load balancer to allocate quickly another node Y and redirect a part of microservice's X requests to new node Y +- the microservice X stays untouched for a last Y hours and the cluster's overall load is less than 75% during this time - hypervisor makes a decision to stop redundant instance, simply to save money +- the network pressure is superbly high - investigate why, if there's hackers DDoS attack - perform set of actions for DDoS protection, if there's simply spike because a lots of users(customers) - do actions to increase cluster size and bandwidth, etc. +- the data storage Y flied away because of HW damage - allocate the replica of storage Y and put it instead of broken one to the cluster +etc. + +Basically, the most important set of monitored events may be: +- hardware health monitoring +-- node / microservice availability +-- ... TBD + +- software health monitoring +-- node / microservice availability +-- the reason +--- scale needed +--- unsuccessful configuration +--- software not getting up +--- out of resources +-- ... TBD + +- provider's fault +-- hardware or software +-- service absense +-- payment expired +-- ... TBD + +- ... TBD + +a simple example, using the current lab infrastructure: +[global net x.x.x.x] <-> [(134.17.26.161 external IP) router (debian-based] (internal IP 10.7.6.228) ] <-> [(10.7.0.10 intra network 1) FE (intra network 2: 10.7.0.0/8)] [intra network 3: ...] +the owner of top-level services is Anton Afanasyeu. top level services include: +- internet provider +- router maintenance +- intra network delegation, configuration, maintenance, VMs (empty, just a (re)source providing), git as a service, disk storage +- basic on-demand OS installation, on-demand VM snapshots +----------- all up is SaaS, for all the questions contact owner --------- + +-- cluster of VM's, XEN-based (hvm or paravirt, depends) +-- VM's control, VM's internal software installation, access, aggregation, fine tunings, software development, ssh, internal setups, etc. +----------- the last few lines are totally controllable by developer, developed product owner and projects workers as decided by product owner --------- +the VM delegated cluster owner must set its own monitoring software where it needs to be placed and monitor all the parameters he decided to keep an eye on, like: +1) hardware resources load, per cluster node and overall in-cluster, comparing the max. values delegated by Anton Afanasyeu according to payment fee for the cluster; like Anton guarantees that CPU will be real (not emulated) and it will be Intel 8 cores of Gold 5218 CPU @ 2.30GHz series, 8Gb of RAM per node, 32Gb of non-shared disk space per node, 3 nodes in cluster, 200Mbit in/out internet, etc; the project owner should have monitoring based of the extremum of each of parameters, compare with proposed plan, alert of some of values differ than what Anton guaranteed +2) project owner took 3 nodes / cluster; after month of heavy load one of the nodes show instabilities and slow data writes using distributed among the nodes MariaDb; MariaDb monitoring show exactly that Node B contains the replica of master db node and it's possible that there's a hardware fault (disk writes), so he should notify Anton to take it into account and provide spare node +3) slow egress/ingress -> delegate to Anton +4) one of project services or its relatives aren't responding well - there should be per-service self monitoring and should raise an alert if so + + +The how: +the actor (any of) may be triggered on faulty node, cluster, service, ..., state, first off, by email. any other services are also likely to have (messengers, SMS, etc.), but anyway we should integrate some monitoring visualizer(s): +1) Visualization of the whole infrastructure +2) Visualization per cluster and per Node health +3) Visualization per service provided by distro used inside the each Node +4) Visualization per service of project's software inside the each Node +based on the requirements, a few almost-ready solutions touch the souls: +1. Grafana (Kibana) +1.1 Pros: the most beautiful and user-friendly; pretty easy to play with pre-stored parameters, building customized graphics; may be self-hosted; may reflect anything you want, but the administrator's eye is required on it, at least on the first steps +1.2 Cons: requires additional software, like Prometheus (the most commonly used) or maybe other time-series based engine; so flexible that it is sometimes unbelievably hard to create pre-stored parameters set to visualize it later; requires also additional data converters/plugins, Prometheus-friendly + +2. Zabbix +2.1 Pros: good UI/UX, pretty easy to deploy; good support, good abilities; +2.2 Cons: not as good as Grafana+... from the UI/UX perspective +2.3 Need to check the licensing + +3. Nagios +3.1 Pros: the most probably known and loved by many administrators, can capture any data, lightweight; +3.2 Cons: i don't know nothing about data visualization, may be Nagios is just data collector and visualization being made by for example Grafana? disadvantage also in that this is very complex solution, hard to tune, hard to maintain + +4. ... TBD, any suggestions + +5. Mixed (hybrid), like Grafana + Zabbix +... TBD