4.8 KiB
Edge network probes
Scripts for router, FE, and external vantage (e.g. Raspberry Pi) — not BE app hosts.
wan-snapshot.sh (v2)
Periodic ping comparison plus lightweight host/path context.
| Target | Default | Meaning |
|---|---|---|
| WAN | 1.1.1.1 |
ISP/upstream leg |
| FE | 10.7.0.10 |
Path to Gentoo FE |
| BE | 10.7.16.128 |
Deeper LAN (disable: BE_PING_TARGET=none) |
| router | none |
Optional modem ping (ROUTER_PING_TARGET on Pi) |
| gw | auto | Default gateway ping |
Each target logs: loss, min, avg, max, mdev (jitter).
Extras (WAN_SNAPSHOT_EXTRAS=1): load1, cpu_busy, conntrack, route_wan/route_fe, via_wan, neigh_fe, non-zero drops.
Log format: ver=2 role=pi|fe|router host=… — older lines without ver= still work in --report / --correlate.
Install (any host)
sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
sudo cp examples/network/wan-snapshot.env.example /etc/conf.d/wan-snapshot
# edit /etc/conf.d/wan-snapshot for role-specific targets
Cron (every 15 minutes — one entry per host)
*/15 * * * * root . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
The script uses flock on /var/run/wan-snapshot.lock so duplicate crontab lines do not double-log.
Log: /var/log/wan-quality.log (override with WAN_SNAPSHOT_LOG).
Suggested /etc/conf.d/wan-snapshot
Raspberry Pi (path into infra matters most):
WAN_SNAPSHOT_ROLE=pi
ROUTER_PING_TARGET=10.7.6.228
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
Gentoo FE:
WAN_SNAPSHOT_ROLE=fe
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
ROUTER_PING_TARGET=10.7.6.228
Debian router:
WAN_SNAPSHOT_ROLE=router
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
Usage
# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh
# today vs yesterday (mean, p50, p90, bad>=50ms counts)
/usr/local/bin/wan-snapshot.sh --report
# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48
# merge Pi + FE (+ router) logs by 15-min slot
/usr/local/bin/wan-snapshot.sh --correlate \
tmp/raspberrypi/var/log/wan-quality.log \
tmp/FE_gentoo/var/log/wan-quality.log
Example v2 line:
ts=2026-06-12T15:30:01+03:00 ver=2 role=fe host=wg0 wan=1.1.1.1 wan_loss=0% wan_min_ms=9.1 wan_avg_ms=10.6 wan_max_ms=12.4 wan_mdev_ms=0.8 fe=10.7.0.10 fe_loss=0% fe_min_ms=0.05 fe_avg_ms=0.07 fe_max_ms=0.10 fe_mdev_ms=0.01 be=10.7.16.128 be_loss=0% be_avg_ms=0.20 load1=0.12 cpu_busy=4% route_wan=eno0 route_fe=lo neigh_fe=REACHABLE
Reading results
| Pattern | Likely cause |
|---|---|
wan bad, fe good (on router) |
ISP/upstream |
wan + fe bad on router |
modem CPU, conntrack, LAN |
fe bad on pi, fe good on fe |
path/tunnel to FE, not FE VM |
wan_max_ms >> wan_avg_ms or high mdev |
jitter / bufferbloat |
conntrack>80% |
edge table full |
neigh_fe not REACHABLE |
ARP/L2 toward FE |
route_fe=tun0 / ppp0 on Pi |
metric is tunnel quality |
| ping good, HTTPS slow | DNAT/nginx (not measured here) |
From outside LAN, still run occasionally:
mtr -rwzc 50 134.17.26.161
monstro-netdiag.sh (bottleneck + iptables + processes)
Run on the Debian edge router (10.7.6.228) when infra feels stuck. Complements wan-snapshot with on-demand depth.
sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh
Cron (optional; pair with wan-snapshot on router only):
*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1
When sluggish:
sudo monstro-netdiag.sh --full
sudo monstro-netdiag.sh --watch 30
monstro-netdiag.sh --tail 48
Why intermittent (excluding ISP)?
| Cause | What to look for |
|---|---|
| conntrack table full | conntrack>80% in wan-snapshot or monstro |
| Router CPU / softirq | high cpu_busy, softnet dropped in --full |
| iptables rule cost | FORWARD/NAT rules with huge packet counts |
| FE/BE VM host busy | FE ping high from router, low load1 on router |
| ARP/route flap | neigh_fe not REACHABLE |
| Bufferbloat on LAN | high mdev / max, 0% loss |
Good vs bad: save one --full output on a rocketship evening and diff when stuck.
Safe checks on monstro (read-only)
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
iptables -L FORWARD -n -v --line-numbers | head
ip neigh show 10.7.0.10
Do not change iptables/NAT on monstro without a rollback plan (per infra policy).