# Edge network probes

Scripts for the **Debian router** (`134.17.26.161` / `10.7.6.228`) — not FE/BE app hosts.

## `wan-snapshot.sh`

Periodic ping comparison:

| Target | Default | Meaning |
|--------|---------|---------|
| **WAN** | `1.1.1.1` | ISP/upstream leg (outbound from router) |
| **FE** | `10.7.0.10` | LAN leg router → Gentoo FE |
| **BE** | `10.7.16.128` | Optional deeper LAN (disable: `BE_PING_TARGET=none`) |

### Install (router)

```bash
sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
```

### Cron (every 15 minutes)

```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
```

Log: `/var/log/wan-quality.log` (override with `WAN_SNAPSHOT_LOG`).

### Usage

```bash
# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh

# compare tonight vs yesterday
/usr/local/bin/wan-snapshot.sh --report

# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48
```

Example log line:

```text
ts=2026-06-04T21:15:01+02:00 wan=1.1.1.1 wan_loss=0% wan_avg_ms=11.2 fe=10.7.0.10 fe_loss=0% fe_avg_ms=0.3 be=10.7.16.128 be_loss=0% be_avg_ms=0.5
```

### Reading results

- **`wan_avg_ms` high, `fe_avg_ms` low** → ISP/upstream (your “rocketship vs gap” bet).
- **Both high** → router CPU, conntrack, or local link.
- **Ping good, HTTPS/git slow** → DNAT or service layer (not measured here).

From **outside** LAN, still run occasionally:

```bash
mtr -rwzc 50 134.17.26.161
```

That catches path issues ping from the router cannot see.

## `monstro-netdiag.sh` (bottleneck + iptables + processes)

Run on the **Debian edge router** (`10.7.6.228`, same host as `wan-snapshot.sh`) when infra feels stuck.

```bash
sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh
```

**Cron** (pair with wan-snapshot):

```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1
```

**When sluggish** (run immediately, compare to a “good” capture):

```bash
sudo monstro-netdiag.sh --full          # ping + conntrack + iptables -v + top CPU
sudo monstro-netdiag.sh --watch 30      # stream one-liners every 30s
monstro-netdiag.sh --tail 48
```

One-line log (`/var/log/monstro-netdiag.log`) includes: `load1`, `cpu_busy`, `conntrack=cur/max (%)`, `top=process`, `fe_avg_ms`, `wan_avg_ms`, iface drops.

### Why intermittent (excluding ISP)?

Your data showed **WAN ~9 ms** but **FE ~230 ms** — classic **intra-LAN** flapping:

| Cause | What to look for in `--full` |
|-------|------------------------------|
| **conntrack table full** | `conntrack` >80%, new flows stall |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` |
| **iptables rule cost** | FORWARD/NAT rules with huge packet counts |
| **FE/BE VM host busy** | FE ping high from router, low load on router |
| **ARP/route flap** | `ip neigh` INCOMPLETE/FAILED for 10.7.0.10 |
| **Bufferbloat on LAN** | rising `fe_avg_ms` under load, 0% loss |

Good vs bad: save one `--full` output on a rocketship evening and diff when stuck.

### Safe checks on monstro (read-only)

```bash
# conntrack pressure
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# hottest iptables rules (no changes)
iptables -L FORWARD -n -v --line-numbers | sort -k1 -rn | head
iptables -t nat -L PREROUTING -n -v --line-numbers | head -20

# who eats CPU during slowness
ps -eo pcpu,pmem,comm --sort=-pcpu | head
```

Do **not** change iptables/NAT on monstro without a rollback plan (per infra policy).