android_cast/examples/network/README.md

Edge network probes

Scripts for router, FE, and external vantage (e.g. Raspberry Pi) — not BE app hosts.

Auto watchdog (v2.1)

When a cron snapshot exceeds thresholds, wan-snapshot.sh appends alert + deep lines to the same log (WAN_WATCH_LOG).

kind=	Meaning
snapshot	Normal 15-min ping row
alert	Threshold breached; lists triggers=
deep	One parseable fact (section=host, burst, iface, iptables, …)

Deploy both files:

sudo cp examples/network/wan-snapshot.sh examples/network/wan-deep-diag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/wan-snapshot.sh /usr/local/bin/wan-deep-diag.sh

Env (/etc/conf.d/wan-snapshot):

WAN_WATCH_LOG=/var/log/wan-watch.log
WAN_SNAPSHOT_LOG=/var/log/wan-watch.log
WAN_SNAPSHOT_WATCH=1

Triggers (defaults): *_avg_ms >= 50, *_loss >= 1%, mdev >= 80, conntrack >= 80%, neigh_fe not REACHABLE/DELAY, plus role hints pi_fe_path / router_wan_path.

Cooldown: one deep capture per 14 min (WAN_WATCH_COOLDOWN_S=840) to avoid cron pile-up.

Review after a bad spell:

/usr/local/bin/wan-snapshot.sh --alerts 80
/usr/local/bin/wan-snapshot.sh --correlate /var/log/wan-watch.log …
grep 'kind=deep.*section=burst' /var/log/wan-watch.log | tail
grep 'kind=deep.*section=iptables' /var/log/wan-watch.log | tail

Example deep lines:

kind=alert ts=2026-06-12T21:45:01+03:00 role=router host=monstro parent_ts=... triggers="wan_avg=535.365,router_wan_path"
kind=deep ts=2026-06-12T21:45:01+03:00 role=router host=monstro parent_ts=... section=burst label=wan target=1.1.1.1 loss=0% avg_ms=520 max_ms=790 mdev_ms=110
kind=deep ts=2026-06-12T21:45:01+03:00 role=router host=monstro parent_ts=... section=iptables tbl=filter chain=FORWARD line=3 pkts=188000 target=ACCEPT

wan-snapshot.sh (v2)

Periodic ping comparison plus lightweight host/path context.

Target	Default	Meaning
WAN	1.1.1.1	ISP/upstream leg
FE	10.7.0.10	Path to Gentoo FE
BE	10.7.16.128	Deeper LAN (disable: BE_PING_TARGET=none)
router	none	Optional modem ping (ROUTER_PING_TARGET on Pi)
gw	auto	Default gateway ping

Each target logs: loss, min, avg, max, mdev (jitter).

Extras (WAN_SNAPSHOT_EXTRAS=1): load1, cpu_busy, conntrack, route_wan/route_fe, via_wan, neigh_fe, non-zero drops.

Log format: ver=2 role=pi|fe|router host=… — older lines without ver= still work in --report / --correlate.

Install (any host)

sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
sudo cp examples/network/wan-snapshot.env.example /etc/conf.d/wan-snapshot
# edit /etc/conf.d/wan-snapshot for role-specific targets

Cron (every 15 minutes — one entry per host)

*/15 * * * * root . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1

The script uses flock on /var/run/wan-snapshot.lock so duplicate crontab lines do not double-log.

Log: /var/log/wan-quality.log (override with WAN_SNAPSHOT_LOG).

Suggested /etc/conf.d/wan-snapshot

Raspberry Pi (path into infra matters most):

WAN_SNAPSHOT_ROLE=pi
ROUTER_PING_TARGET=10.7.6.228
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128

Gentoo FE:

WAN_SNAPSHOT_ROLE=fe
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
ROUTER_PING_TARGET=10.7.6.228

Debian router:

WAN_SNAPSHOT_ROLE=router
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128

Usage

# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh

# today vs yesterday (mean, p50, p90, bad>=50ms counts)
/usr/local/bin/wan-snapshot.sh --report

# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48

# merge Pi + FE (+ router) logs by 15-min slot
/usr/local/bin/wan-snapshot.sh --correlate \
  tmp/raspberrypi/var/log/wan-quality.log \
  tmp/FE_gentoo/var/log/wan-quality.log

Example v2 line:

ts=2026-06-12T15:30:01+03:00 ver=2 role=fe host=wg0 wan=1.1.1.1 wan_loss=0% wan_min_ms=9.1 wan_avg_ms=10.6 wan_max_ms=12.4 wan_mdev_ms=0.8 fe=10.7.0.10 fe_loss=0% fe_min_ms=0.05 fe_avg_ms=0.07 fe_max_ms=0.10 fe_mdev_ms=0.01 be=10.7.16.128 be_loss=0% be_avg_ms=0.20 load1=0.12 cpu_busy=4% route_wan=eno0 route_fe=lo neigh_fe=REACHABLE

Reading results

Pattern	Likely cause
wan bad, fe good (on router)	ISP/upstream
wan + fe bad on router	modem CPU, conntrack, LAN
fe bad on pi, fe good on fe	path/tunnel to FE, not FE VM
wan_max_ms >> wan_avg_ms or high mdev	jitter / bufferbloat
conntrack>80%	edge table full
neigh_fe not REACHABLE	ARP/L2 toward FE
route_fe=tun0 / ppp0 on Pi	metric is tunnel quality
ping good, HTTPS slow	DNAT/nginx (not measured here)

From outside LAN, still run occasionally:

mtr -rwzc 50 134.17.26.161

monstro-netdiag.sh (bottleneck + iptables + processes)

Run on the Debian edge router (10.7.6.228) when infra feels stuck. Complements wan-snapshot with on-demand depth.

sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh

Cron (optional; pair with wan-snapshot on router only):

*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1

When sluggish:

sudo monstro-netdiag.sh --full
sudo monstro-netdiag.sh --watch 30
monstro-netdiag.sh --tail 48

Why intermittent (excluding ISP)?

Cause	What to look for
conntrack table full	conntrack>80% in wan-snapshot or monstro
Router CPU / softirq	high cpu_busy, softnet dropped in --full
iptables rule cost	FORWARD/NAT rules with huge packet counts
FE/BE VM host busy	FE ping high from router, low load1 on router
ARP/route flap	neigh_fe not REACHABLE
Bufferbloat on LAN	high mdev / max, 0% loss

Good vs bad: save one --full output on a rocketship evening and diff when stuck.

Safe checks on monstro (read-only)

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
iptables -L FORWARD -n -v --line-numbers | head
ip neigh show 10.7.0.10

Do not change iptables/NAT on monstro without a rollback plan (per infra policy).