ac/android_cast
1
0
Fork 0
mirror of git://f0xx.org/android_cast synced 2026-07-29 06:58:51 +03:00
Code Issues Packages Projects Releases Wiki Activity

sync on be, vpn

Browse Source
This commit is contained in:
Anton Afanasyeu
2026-06-13 15:31:18 +02:00
parent f9b73b48da
commit 86055bf509
42 changed files with 2472 additions and 917 deletions
Download Patch File Download Diff File Expand all files Collapse all files

268
examples/network/README.md
View File

@@ -1,204 +1,92 @@
# Edge network probes
# Edge network watchdog
Scripts for **router**, **FE**, and **external vantage** (e.g. Raspberry Pi) — not BE app hosts.
**One script:** `wan-watch.sh` — deploy everywhere, read one log, get a verdict every 5 minutes.
## Auto watchdog (v2.1)
When a cron snapshot exceeds thresholds, `wan-snapshot.sh` appends **alert** + **deep** lines to the same log (`WAN_WATCH_LOG`).
| `kind=` | Meaning |
|---------|---------|
| `snapshot` | Normal 15-min ping row |
| `alert` | Threshold breached; lists `triggers=` |
| `deep` | One parseable fact (`section=host`, `burst`, `iface`, `iptables`, …) |
**Deploy both files:**
## Deploy
```bash
sudo cp examples/network/wan-snapshot.sh examples/network/wan-deep-diag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/wan-snapshot.sh /usr/local/bin/wan-deep-diag.sh
sudo cp examples/network/wan-watch.sh /usr/local/bin/wan-watch.sh
sudo chmod +x /usr/local/bin/wan-watch.sh
sudo cp examples/network/wan-watch.env.example /etc/conf.d/wan-watch
# edit WAN_WATCH_ROLE=pi|router|fe
```
**Env** (`/etc/conf.d/wan-snapshot`):
Remove old cron entries for `wan-snapshot.sh` / `monstro-netdiag.sh`. **One cron only:**
```cron
*/5 * * * * root . /etc/conf.d/wan-watch 2>/dev/null; /usr/local/bin/wan-watch.sh >>/var/log/wan-watch.cron.log 2>&1
```
Log: `/var/log/wan-watch.log`
## Every run (5 min) captures
| Layer | What |
|-------|------|
| **Ping matrix** | wan, fe, be, gw, router — loss/min/avg/max/mdev |
| **Verdict** | `kind=verdict` — immediate conclusion + action hint |
| **Tunnels** | GRE links/modules, PPTP (1723/pppd/pptpd), **tinc** (tincctl status), **WireGuard**, **iodine/dns0/tun0** |
| **Edge services** | named, squid, nginx, iodined — proc count + CPU |
| **Host** | load, cpu, conntrack, softnet drops, ss summary |
| **Network** | routes, neigh, iface drops, qdisc backlog, iptables hot rules (router) |
| **On anomaly** | `kind=alert` + `kind=deep` burst pings + full probe repeat (5 min cooldown) |
## Commands
```bash
wan-watch.sh # cron default
wan-watch.sh --full # instant human dump (when slow NOW)
wan-watch.sh --analyze # final conclusion from log (last 24h) — no waiting
wan-watch.sh --analyze 6 # last 6 hours
wan-watch.sh --timeline # hourly good/bad bars
wan-watch.sh --correlate /var/log/wan-watch.log path/to/other/log
wan-watch.sh --force-deep # full capture even if OK
wan-watch.sh --tail 30
```
## Verdicts (automatic)
| verdict | Meaning |
|---------|---------|
| `OK` | Within thresholds |
| `MONSTRO_WAN_BLOAT` | Router WAN slow, FE local OK — **your main issue** |
| `UPSTREAM_GW` | WAN + gw 10.7.16.228 both slow |
| `PATH_TO_FE` | Pi ISP OK, path to 10.7.0.10 slow |
| `CONNTRACK_PRESSURE` | nf_conntrack > 70% |
| `TUNNEL_DNS0_LOAD` | iodine dns0 drops elevated |
| `WAN_JITTER` | High mdev, bufferbloat signal |
Cron stdout example:
```text
WAN-WATCH [router/monstro] verdict=MONSTRO_WAN_BLOAT confidence=high summary="WAN leg slow..." action="SQM/fq_codel on xenbr0..."
```
## `--analyze` example output
```text
CONCLUSION: MONSTRO WAN EDGE QUEUEING (confidence=HIGH)
Fix order: 1) fq_codel/SQM on xenbr0 2) gw 10.7.16.228 3) squid/named 4) iodine dns0
```
Run this **any time** — no need to wait 24h after deploy.
## Host config (`/etc/conf.d/wan-watch`)
```sh
WAN_WATCH_LOG=/var/log/wan-watch.log
WAN_SNAPSHOT_LOG=/var/log/wan-watch.log
WAN_SNAPSHOT_WATCH=1
WAN_WATCH_ROLE=pi # or router on monstro
ROUTER_PING_TARGET=10.7.6.228 # Pi only
WAN_WATCH_DEEP=auto
WAN_WATCH_COOLDOWN_S=300
```
**Triggers (defaults):** `*_avg_ms >= 50`, `*_loss >= 1%`, `mdev >= 80`, `conntrack >= 80%`, `neigh_fe` not REACHABLE/DELAY, plus role hints `pi_fe_path` / `router_wan_path`.
## Legacy
**Cooldown:** one deep capture per 14 min (`WAN_WATCH_COOLDOWN_S=840`) to avoid cron pile-up.
- `wan-snapshot.sh` → wrapper to `wan-watch.sh`
- `wan-deep-diag.sh` → merged into `wan-watch.sh`
- `monstro-netdiag.sh` → use `wan-watch.sh --full` instead
**Review after a bad spell:**
## Known conclusion (from your logs)
```bash
/usr/local/bin/wan-snapshot.sh --alerts 80
/usr/local/bin/wan-snapshot.sh --correlate /var/log/wan-watch.log …
grep 'kind=deep.*section=burst' /var/log/wan-watch.log | tail
grep 'kind=deep.*section=iptables' /var/log/wan-watch.log | tail
```
Example deep lines:
```text
kind=alert ts=2026-06-12T21:45:01+03:00 role=router host=monstro parent_ts=... triggers="wan_avg=535.365,router_wan_path"
kind=deep ts=2026-06-12T21:45:01+03:00 role=router host=monstro parent_ts=... section=burst label=wan target=1.1.1.1 loss=0% avg_ms=520 max_ms=790 mdev_ms=110
kind=deep ts=2026-06-12T21:45:01+03:00 role=router host=monstro parent_ts=... section=iptables tbl=filter chain=FORWARD line=3 pkts=188000 target=ACCEPT
```
## `wan-snapshot.sh` (v2)
Periodic ping comparison plus lightweight host/path context.
| Target | Default | Meaning |
|--------|---------|---------|
| **WAN** | `1.1.1.1` | ISP/upstream leg |
| **FE** | `10.7.0.10` | Path to Gentoo FE |
| **BE** | `10.7.16.128` | Deeper LAN (disable: `BE_PING_TARGET=none`) |
| **router** | `none` | Optional modem ping (`ROUTER_PING_TARGET` on Pi) |
| **gw** | auto | Default gateway ping |
Each target logs: `loss`, `min`, `avg`, `max`, `mdev` (jitter).
Extras (`WAN_SNAPSHOT_EXTRAS=1`): `load1`, `cpu_busy`, `conntrack`, `route_wan`/`route_fe`, `via_wan`, `neigh_fe`, non-zero `drops`.
Log format: `ver=2 role=pi|fe|router host=…` — older lines without `ver=` still work in `--report` / `--correlate`.
### Install (any host)
```bash
sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
sudo cp examples/network/wan-snapshot.env.example /etc/conf.d/wan-snapshot
# edit /etc/conf.d/wan-snapshot for role-specific targets
```
### Cron (every 15 minutes — **one** entry per host)
```cron
*/15 * * * * root . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
```
The script uses `flock` on `/var/run/wan-snapshot.lock` so duplicate crontab lines do not double-log.
Log: `/var/log/wan-quality.log` (override with `WAN_SNAPSHOT_LOG`).
### Suggested `/etc/conf.d/wan-snapshot`
**Raspberry Pi** (path into infra matters most):
```sh
WAN_SNAPSHOT_ROLE=pi
ROUTER_PING_TARGET=10.7.6.228
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
**Gentoo FE**:
```sh
WAN_SNAPSHOT_ROLE=fe
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
ROUTER_PING_TARGET=10.7.6.228
```
**Debian router**:
```sh
WAN_SNAPSHOT_ROLE=router
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
### Usage
```bash
# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh
# today vs yesterday (mean, p50, p90, bad>=50ms counts)
/usr/local/bin/wan-snapshot.sh --report
# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48
# merge Pi + FE (+ router) logs by 15-min slot
/usr/local/bin/wan-snapshot.sh --correlate \
tmp/raspberrypi/var/log/wan-quality.log \
tmp/FE_gentoo/var/log/wan-quality.log
```
Example v2 line:
```text
ts=2026-06-12T15:30:01+03:00 ver=2 role=fe host=wg0 wan=1.1.1.1 wan_loss=0% wan_min_ms=9.1 wan_avg_ms=10.6 wan_max_ms=12.4 wan_mdev_ms=0.8 fe=10.7.0.10 fe_loss=0% fe_min_ms=0.05 fe_avg_ms=0.07 fe_max_ms=0.10 fe_mdev_ms=0.01 be=10.7.16.128 be_loss=0% be_avg_ms=0.20 load1=0.12 cpu_busy=4% route_wan=eno0 route_fe=lo neigh_fe=REACHABLE
```
### Reading results
| Pattern | Likely cause |
|---------|----------------|
| `wan` bad, `fe` good (on router) | ISP/upstream |
| `wan` + `fe` bad on router | modem CPU, conntrack, LAN |
| `fe` bad on **pi**, `fe` good on **fe** | path/tunnel to FE, not FE VM |
| `wan_max_ms` >> `wan_avg_ms` or high `mdev` | jitter / bufferbloat |
| `conntrack>80%` | edge table full |
| `neigh_fe` not `REACHABLE` | ARP/L2 toward FE |
| `route_fe=tun0` / `ppp0` on Pi | metric is tunnel quality |
| ping good, HTTPS slow | DNAT/nginx (not measured here) |
From **outside** LAN, still run occasionally:
```bash
mtr -rwzc 50 134.17.26.161
```
## `monstro-netdiag.sh` (bottleneck + iptables + processes)
Run on the **Debian edge router** (`10.7.6.228`) when infra feels stuck. Complements `wan-snapshot` with on-demand depth.
```bash
sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh
```
**Cron** (optional; pair with wan-snapshot on router only):
```cron
*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1
```
**When sluggish**:
```bash
sudo monstro-netdiag.sh --full
sudo monstro-netdiag.sh --watch 30
monstro-netdiag.sh --tail 48
```
### Why intermittent (excluding ISP)?
| Cause | What to look for |
|-------|------------------|
| **conntrack table full** | `conntrack>80%` in wan-snapshot or monstro |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` in `--full` |
| **iptables rule cost** | FORWARD/NAT rules with huge packet counts |
| **FE/BE VM host busy** | FE ping high from router, low `load1` on router |
| **ARP/route flap** | `neigh_fe` not REACHABLE |
| **Bufferbloat on LAN** | high `mdev` / `max`, 0% loss |
Good vs bad: save one `--full` output on a rocketship evening and diff when stuck.
### Safe checks on monstro (read-only)
```bash
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
iptables -L FORWARD -n -v --line-numbers | head
ip neigh show 10.7.0.10
```
Do **not** change iptables/NAT on monstro without a rollback plan (per infra policy).
**MONSTRO WAN EDGE QUEUEING** on `xenbr0` → upstream `10.7.16.228`. Not FE/BE, not Pi ISP. Contributing load: **named**, **squid**, **dns0** (iodine). Fix: **SQM/fq_codel** on WAN iface first.