# Edge network probes

Scripts for **router**, **FE**, and **external vantage** (e.g. Raspberry Pi) — not BE app hosts.

## `wan-snapshot.sh` (v2)

Periodic ping comparison plus lightweight host/path context.

| Target | Default | Meaning |
|--------|---------|---------|
| **WAN** | `1.1.1.1` | ISP/upstream leg |
| **FE** | `10.7.0.10` | Path to Gentoo FE |
| **BE** | `10.7.16.128` | Deeper LAN (disable: `BE_PING_TARGET=none`) |
| **router** | `none` | Optional modem ping (`ROUTER_PING_TARGET` on Pi) |
| **gw** | auto | Default gateway ping |

Each target logs: `loss`, `min`, `avg`, `max`, `mdev` (jitter).

Extras (`WAN_SNAPSHOT_EXTRAS=1`): `load1`, `cpu_busy`, `conntrack`, `route_wan`/`route_fe`, `via_wan`, `neigh_fe`, non-zero `drops`.

Log format: `ver=2 role=pi|fe|router host=…` — older lines without `ver=` still work in `--report` / `--correlate`.

### Install (any host)

```bash
sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
sudo cp examples/network/wan-snapshot.env.example /etc/conf.d/wan-snapshot
# edit /etc/conf.d/wan-snapshot for role-specific targets
```

### Cron (every 15 minutes — **one** entry per host)

```cron
*/15 * * * * root . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
```

The script uses `flock` on `/var/run/wan-snapshot.lock` so duplicate crontab lines do not double-log.

Log: `/var/log/wan-quality.log` (override with `WAN_SNAPSHOT_LOG`).

### Suggested `/etc/conf.d/wan-snapshot`

**Raspberry Pi** (path into infra matters most):

```sh
WAN_SNAPSHOT_ROLE=pi
ROUTER_PING_TARGET=10.7.6.228
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```

**Gentoo FE**:

```sh
WAN_SNAPSHOT_ROLE=fe
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
ROUTER_PING_TARGET=10.7.6.228
```

**Debian router**:

```sh
WAN_SNAPSHOT_ROLE=router
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```

### Usage

```bash
# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh

# today vs yesterday (mean, p50, p90, bad>=50ms counts)
/usr/local/bin/wan-snapshot.sh --report

# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48

# merge Pi + FE (+ router) logs by 15-min slot
/usr/local/bin/wan-snapshot.sh --correlate \
  tmp/raspberrypi/var/log/wan-quality.log \
  tmp/FE_gentoo/var/log/wan-quality.log
```

Example v2 line:

```text
ts=2026-06-12T15:30:01+03:00 ver=2 role=fe host=wg0 wan=1.1.1.1 wan_loss=0% wan_min_ms=9.1 wan_avg_ms=10.6 wan_max_ms=12.4 wan_mdev_ms=0.8 fe=10.7.0.10 fe_loss=0% fe_min_ms=0.05 fe_avg_ms=0.07 fe_max_ms=0.10 fe_mdev_ms=0.01 be=10.7.16.128 be_loss=0% be_avg_ms=0.20 load1=0.12 cpu_busy=4% route_wan=eno0 route_fe=lo neigh_fe=REACHABLE
```

### Reading results

| Pattern | Likely cause |
|---------|----------------|
| `wan` bad, `fe` good (on router) | ISP/upstream |
| `wan` + `fe` bad on router | modem CPU, conntrack, LAN |
| `fe` bad on **pi**, `fe` good on **fe** | path/tunnel to FE, not FE VM |
| `wan_max_ms` >> `wan_avg_ms` or high `mdev` | jitter / bufferbloat |
| `conntrack>80%` | edge table full |
| `neigh_fe` not `REACHABLE` | ARP/L2 toward FE |
| `route_fe=tun0` / `ppp0` on Pi | metric is tunnel quality |
| ping good, HTTPS slow | DNAT/nginx (not measured here) |

From **outside** LAN, still run occasionally:

```bash
mtr -rwzc 50 134.17.26.161
```

## `monstro-netdiag.sh` (bottleneck + iptables + processes)

Run on the **Debian edge router** (`10.7.6.228`) when infra feels stuck. Complements `wan-snapshot` with on-demand depth.

```bash
sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh
```

**Cron** (optional; pair with wan-snapshot on router only):

```cron
*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1
```

**When sluggish**:

```bash
sudo monstro-netdiag.sh --full
sudo monstro-netdiag.sh --watch 30
monstro-netdiag.sh --tail 48
```

### Why intermittent (excluding ISP)?

| Cause | What to look for |
|-------|------------------|
| **conntrack table full** | `conntrack>80%` in wan-snapshot or monstro |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` in `--full` |
| **iptables rule cost** | FORWARD/NAT rules with huge packet counts |
| **FE/BE VM host busy** | FE ping high from router, low `load1` on router |
| **ARP/route flap** | `neigh_fe` not REACHABLE |
| **Bufferbloat on LAN** | high `mdev` / `max`, 0% loss |

Good vs bad: save one `--full` output on a rocketship evening and diff when stuck.

### Safe checks on monstro (read-only)

```bash
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
iptables -L FORWARD -n -v --line-numbers | head
ip neigh show 10.7.0.10
```

Do **not** change iptables/NAT on monstro without a rollback plan (per infra policy).