ac/android_cast
1
0
Fork 0
mirror of git://f0xx.org/android_cast synced 2026-07-29 06:18:42 +03:00
Code Issues Packages Projects Releases Wiki Activity

small sync

Browse Source
This commit is contained in:
Anton Afanasyeu
2026-06-12 18:48:03 +02:00
parent 170211ef1f
commit 539341d8f1
8 changed files with 653 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
examples/network/README.md
View File

@@ -1,56 +1,108 @@
# Edge network probes
Scripts for the **Debian router** (`134.17.26.161` / `10.7.6.228`) — not FE/BE app hosts.
Scripts for **router**, **FE**, and **external vantage** (e.g. Raspberry Pi) — not BE app hosts.
## `wan-snapshot.sh`
## `wan-snapshot.sh` (v2)
Periodic ping comparison:
Periodic ping comparison plus lightweight host/path context.
| Target | Default | Meaning |
|--------|---------|---------|
| **WAN** | `1.1.1.1` | ISP/upstream leg (outbound from router) |
| **FE** | `10.7.0.10` | LAN leg router → Gentoo FE |
| **BE** | `10.7.16.128` | Optional deeper LAN (disable: `BE_PING_TARGET=none`) |
| **WAN** | `1.1.1.1` | ISP/upstream leg |
| **FE** | `10.7.0.10` | Path to Gentoo FE |
| **BE** | `10.7.16.128` | Deeper LAN (disable: `BE_PING_TARGET=none`) |
| **router** | `none` | Optional modem ping (`ROUTER_PING_TARGET` on Pi) |
| **gw** | auto | Default gateway ping |
### Install (router)
Each target logs: `loss`, `min`, `avg`, `max`, `mdev` (jitter).
Extras (`WAN_SNAPSHOT_EXTRAS=1`): `load1`, `cpu_busy`, `conntrack`, `route_wan`/`route_fe`, `via_wan`, `neigh_fe`, non-zero `drops`.
Log format: `ver=2 role=pi|fe|router host=…` — older lines without `ver=` still work in `--report` / `--correlate`.
### Install (any host)
```bash
sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
sudo cp examples/network/wan-snapshot.env.example /etc/conf.d/wan-snapshot
# edit /etc/conf.d/wan-snapshot for role-specific targets
```
### Cron (every 15 minutes)
### Cron (every 15 minutes — **one** entry per host)
```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
*/15 * * * * root . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
```
The script uses `flock` on `/var/run/wan-snapshot.lock` so duplicate crontab lines do not double-log.
Log: `/var/log/wan-quality.log` (override with `WAN_SNAPSHOT_LOG`).
### Suggested `/etc/conf.d/wan-snapshot`
**Raspberry Pi** (path into infra matters most):
```sh
WAN_SNAPSHOT_ROLE=pi
ROUTER_PING_TARGET=10.7.6.228
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
**Gentoo FE**:
```sh
WAN_SNAPSHOT_ROLE=fe
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
ROUTER_PING_TARGET=10.7.6.228
```
**Debian router**:
```sh
WAN_SNAPSHOT_ROLE=router
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
### Usage
```bash
# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh
# compare tonight vs yesterday
# today vs yesterday (mean, p50, p90, bad>=50ms counts)
/usr/local/bin/wan-snapshot.sh --report
# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48
# merge Pi + FE (+ router) logs by 15-min slot
/usr/local/bin/wan-snapshot.sh --correlate \
tmp/raspberrypi/var/log/wan-quality.log \
tmp/FE_gentoo/var/log/wan-quality.log
```
Example log line:
Example v2 line:
```text
ts=2026-06-04T21:15:01+02:00 wan=1.1.1.1 wan_loss=0% wan_avg_ms=11.2 fe=10.7.0.10 fe_loss=0% fe_avg_ms=0.3 be=10.7.16.128 be_loss=0% be_avg_ms=0.5
ts=2026-06-12T15:30:01+03:00 ver=2 role=fe host=wg0 wan=1.1.1.1 wan_loss=0% wan_min_ms=9.1 wan_avg_ms=10.6 wan_max_ms=12.4 wan_mdev_ms=0.8 fe=10.7.0.10 fe_loss=0% fe_min_ms=0.05 fe_avg_ms=0.07 fe_max_ms=0.10 fe_mdev_ms=0.01 be=10.7.16.128 be_loss=0% be_avg_ms=0.20 load1=0.12 cpu_busy=4% route_wan=eno0 route_fe=lo neigh_fe=REACHABLE
```
### Reading results
- **`wan_avg_ms` high, `fe_avg_ms` low** → ISP/upstream (your “rocketship vs gap” bet).
- **Both high** → router CPU, conntrack, or local link.
- **Ping good, HTTPS/git slow** → DNAT or service layer (not measured here).
| Pattern | Likely cause |
|---------|----------------|
| `wan` bad, `fe` good (on router) | ISP/upstream |
| `wan` + `fe` bad on router | modem CPU, conntrack, LAN |
| `fe` bad on **pi**, `fe` good on **fe** | path/tunnel to FE, not FE VM |
| `wan_max_ms` >> `wan_avg_ms` or high `mdev` | jitter / bufferbloat |
| `conntrack>80%` | edge table full |
| `neigh_fe` not `REACHABLE` | ARP/L2 toward FE |
| `route_fe=tun0` / `ppp0` on Pi | metric is tunnel quality |
| ping good, HTTPS slow | DNAT/nginx (not measured here) |
From **outside** LAN, still run occasionally:
@@ -58,62 +110,49 @@ From **outside** LAN, still run occasionally:
mtr -rwzc 50 134.17.26.161
```
That catches path issues ping from the router cannot see.
## `monstro-netdiag.sh` (bottleneck + iptables + processes)
Run on the **Debian edge router** (`10.7.6.228`, same host as `wan-snapshot.sh`) when infra feels stuck.
Run on the **Debian edge router** (`10.7.6.228`) when infra feels stuck. Complements `wan-snapshot` with on-demand depth.
```bash
sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh
```
**Cron** (pair with wan-snapshot):
**Cron** (optional; pair with wan-snapshot on router only):
```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1
```
**When sluggish** (run immediately, compare to a “good” capture):
**When sluggish**:
```bash
sudo monstro-netdiag.sh --full # ping + conntrack + iptables -v + top CPU
sudo monstro-netdiag.sh --watch 30 # stream one-liners every 30s
sudo monstro-netdiag.sh --full
sudo monstro-netdiag.sh --watch 30
monstro-netdiag.sh --tail 48
```
One-line log (`/var/log/monstro-netdiag.log`) includes: `load1`, `cpu_busy`, `conntrack=cur/max (%)`, `top=process`, `fe_avg_ms`, `wan_avg_ms`, iface drops.
### Why intermittent (excluding ISP)?
Your data showed **WAN ~9 ms** but **FE ~230 ms** — classic **intra-LAN** flapping:
| Cause | What to look for in `--full` |
|-------|------------------------------|
| **conntrack table full** | `conntrack` >80%, new flows stall |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` |
| Cause | What to look for |
|-------|------------------|
| **conntrack table full** | `conntrack>80%` in wan-snapshot or monstro |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` in `--full` |
| **iptables rule cost** | FORWARD/NAT rules with huge packet counts |
| **FE/BE VM host busy** | FE ping high from router, low load on router |
| **ARP/route flap** | `ip neigh` INCOMPLETE/FAILED for 10.7.0.10 |
| **Bufferbloat on LAN** | rising `fe_avg_ms` under load, 0% loss |
| **FE/BE VM host busy** | FE ping high from router, low `load1` on router |
| **ARP/route flap** | `neigh_fe` not REACHABLE |
| **Bufferbloat on LAN** | high `mdev` / `max`, 0% loss |
Good vs bad: save one `--full` output on a rocketship evening and diff when stuck.
### Safe checks on monstro (read-only)
```bash
# conntrack pressure
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# hottest iptables rules (no changes)
iptables -L FORWARD -n -v --line-numbers | sort -k1 -rn | head
iptables -t nat -L PREROUTING -n -v --line-numbers | head -20
# who eats CPU during slowness
ps -eo pcpu,pmem,comm --sort=-pcpu | head
iptables -L FORWARD -n -v --line-numbers | head
ip neigh show 10.7.0.10
```
Do **not** change iptables/NAT on monstro without a rollback plan (per infra policy).