ac/android_cast
1
0
Fork 0
mirror of git://f0xx.org/android_cast synced 2026-07-29 03:57:50 +03:00
Code Issues Packages Projects Releases Wiki Activity

small sync

Browse Source
This commit is contained in:
Anton Afanasyeu
2026-06-12 18:48:03 +02:00
parent 170211ef1f
commit 539341d8f1
8 changed files with 653 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
examples/network/README.md
View File

@@ -1,56 +1,108 @@
# Edge network probes
Scripts for the **Debian router** (`134.17.26.161` / `10.7.6.228`) — not FE/BE app hosts.
Scripts for **router**, **FE**, and **external vantage** (e.g. Raspberry Pi) — not BE app hosts.
## `wan-snapshot.sh`
## `wan-snapshot.sh` (v2)
Periodic ping comparison:
Periodic ping comparison plus lightweight host/path context.
| Target | Default | Meaning |
|--------|---------|---------|
| **WAN** | `1.1.1.1` | ISP/upstream leg (outbound from router) |
| **FE** | `10.7.0.10` | LAN leg router → Gentoo FE |
| **BE** | `10.7.16.128` | Optional deeper LAN (disable: `BE_PING_TARGET=none`) |
| **WAN** | `1.1.1.1` | ISP/upstream leg |
| **FE** | `10.7.0.10` | Path to Gentoo FE |
| **BE** | `10.7.16.128` | Deeper LAN (disable: `BE_PING_TARGET=none`) |
| **router** | `none` | Optional modem ping (`ROUTER_PING_TARGET` on Pi) |
| **gw** | auto | Default gateway ping |
### Install (router)
Each target logs: `loss`, `min`, `avg`, `max`, `mdev` (jitter).
Extras (`WAN_SNAPSHOT_EXTRAS=1`): `load1`, `cpu_busy`, `conntrack`, `route_wan`/`route_fe`, `via_wan`, `neigh_fe`, non-zero `drops`.
Log format: `ver=2 role=pi|fe|router host=…` — older lines without `ver=` still work in `--report` / `--correlate`.
### Install (any host)
```bash
sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
sudo cp examples/network/wan-snapshot.env.example /etc/conf.d/wan-snapshot
# edit /etc/conf.d/wan-snapshot for role-specific targets
```
### Cron (every 15 minutes)
### Cron (every 15 minutes — **one** entry per host)
```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
*/15 * * * * root . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
```
The script uses `flock` on `/var/run/wan-snapshot.lock` so duplicate crontab lines do not double-log.
Log: `/var/log/wan-quality.log` (override with `WAN_SNAPSHOT_LOG`).
### Suggested `/etc/conf.d/wan-snapshot`
**Raspberry Pi** (path into infra matters most):
```sh
WAN_SNAPSHOT_ROLE=pi
ROUTER_PING_TARGET=10.7.6.228
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
**Gentoo FE**:
```sh
WAN_SNAPSHOT_ROLE=fe
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
ROUTER_PING_TARGET=10.7.6.228
```
**Debian router**:
```sh
WAN_SNAPSHOT_ROLE=router
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
### Usage
```bash
# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh
# compare tonight vs yesterday
# today vs yesterday (mean, p50, p90, bad>=50ms counts)
/usr/local/bin/wan-snapshot.sh --report
# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48
# merge Pi + FE (+ router) logs by 15-min slot
/usr/local/bin/wan-snapshot.sh --correlate \
tmp/raspberrypi/var/log/wan-quality.log \
tmp/FE_gentoo/var/log/wan-quality.log
```
Example log line:
Example v2 line:
```text
ts=2026-06-04T21:15:01+02:00 wan=1.1.1.1 wan_loss=0% wan_avg_ms=11.2 fe=10.7.0.10 fe_loss=0% fe_avg_ms=0.3 be=10.7.16.128 be_loss=0% be_avg_ms=0.5
ts=2026-06-12T15:30:01+03:00 ver=2 role=fe host=wg0 wan=1.1.1.1 wan_loss=0% wan_min_ms=9.1 wan_avg_ms=10.6 wan_max_ms=12.4 wan_mdev_ms=0.8 fe=10.7.0.10 fe_loss=0% fe_min_ms=0.05 fe_avg_ms=0.07 fe_max_ms=0.10 fe_mdev_ms=0.01 be=10.7.16.128 be_loss=0% be_avg_ms=0.20 load1=0.12 cpu_busy=4% route_wan=eno0 route_fe=lo neigh_fe=REACHABLE
```
### Reading results
- **`wan_avg_ms` high, `fe_avg_ms` low** → ISP/upstream (your “rocketship vs gap” bet).
- **Both high** → router CPU, conntrack, or local link.
- **Ping good, HTTPS/git slow** → DNAT or service layer (not measured here).
| Pattern | Likely cause |
|---------|----------------|
| `wan` bad, `fe` good (on router) | ISP/upstream |
| `wan` + `fe` bad on router | modem CPU, conntrack, LAN |
| `fe` bad on **pi**, `fe` good on **fe** | path/tunnel to FE, not FE VM |
| `wan_max_ms` >> `wan_avg_ms` or high `mdev` | jitter / bufferbloat |
| `conntrack>80%` | edge table full |
| `neigh_fe` not `REACHABLE` | ARP/L2 toward FE |
| `route_fe=tun0` / `ppp0` on Pi | metric is tunnel quality |
| ping good, HTTPS slow | DNAT/nginx (not measured here) |
From **outside** LAN, still run occasionally:
@@ -58,62 +110,49 @@ From **outside** LAN, still run occasionally:
mtr -rwzc 50 134.17.26.161
```
That catches path issues ping from the router cannot see.
## `monstro-netdiag.sh` (bottleneck + iptables + processes)
Run on the **Debian edge router** (`10.7.6.228`, same host as `wan-snapshot.sh`) when infra feels stuck.
Run on the **Debian edge router** (`10.7.6.228`) when infra feels stuck. Complements `wan-snapshot` with on-demand depth.
```bash
sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh
```
**Cron** (pair with wan-snapshot):
**Cron** (optional; pair with wan-snapshot on router only):
```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1
```
**When sluggish** (run immediately, compare to a “good” capture):
**When sluggish**:
```bash
sudo monstro-netdiag.sh --full # ping + conntrack + iptables -v + top CPU
sudo monstro-netdiag.sh --watch 30 # stream one-liners every 30s
sudo monstro-netdiag.sh --full
sudo monstro-netdiag.sh --watch 30
monstro-netdiag.sh --tail 48
```
One-line log (`/var/log/monstro-netdiag.log`) includes: `load1`, `cpu_busy`, `conntrack=cur/max (%)`, `top=process`, `fe_avg_ms`, `wan_avg_ms`, iface drops.
### Why intermittent (excluding ISP)?
Your data showed **WAN ~9 ms** but **FE ~230 ms** — classic **intra-LAN** flapping:
| Cause | What to look for in `--full` |
|-------|------------------------------|
| **conntrack table full** | `conntrack` >80%, new flows stall |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` |
| Cause | What to look for |
|-------|------------------|
| **conntrack table full** | `conntrack>80%` in wan-snapshot or monstro |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` in `--full` |
| **iptables rule cost** | FORWARD/NAT rules with huge packet counts |
| **FE/BE VM host busy** | FE ping high from router, low load on router |
| **ARP/route flap** | `ip neigh` INCOMPLETE/FAILED for 10.7.0.10 |
| **Bufferbloat on LAN** | rising `fe_avg_ms` under load, 0% loss |
| **FE/BE VM host busy** | FE ping high from router, low `load1` on router |
| **ARP/route flap** | `neigh_fe` not REACHABLE |
| **Bufferbloat on LAN** | high `mdev` / `max`, 0% loss |
Good vs bad: save one `--full` output on a rocketship evening and diff when stuck.
### Safe checks on monstro (read-only)
```bash
# conntrack pressure
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# hottest iptables rules (no changes)
iptables -L FORWARD -n -v --line-numbers | sort -k1 -rn | head
iptables -t nat -L PREROUTING -n -v --line-numbers | head -20
# who eats CPU during slowness
ps -eo pcpu,pmem,comm --sort=-pcpu | head
iptables -L FORWARD -n -v --line-numbers | head
ip neigh show 10.7.0.10
```
Do **not** change iptables/NAT on monstro without a rollback plan (per infra policy).

27
examples/network/wan-snapshot.env.example Normal file
View File

@@ -0,0 +1,27 @@
# Copy to /etc/conf.d/wan-snapshot on each host (optional).
# Source from cron: . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh
# --- Raspberry Pi (external / home vantage) ---
# WAN_SNAPSHOT_ROLE=pi
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# ROUTER_PING_TARGET=10.7.6.228
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128
# --- Gentoo FE ---
# WAN_SNAPSHOT_ROLE=fe
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128
# ROUTER_PING_TARGET=10.7.6.228
# --- Debian edge router (monstro) ---
# WAN_SNAPSHOT_ROLE=router
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128
# Shared
# WAN_SNAPSHOT_EXTRAS=1
# PING_COUNT=10
# PING_DEADLINE_S=15

415
examples/network/wan-snapshot.sh
View File

@@ -1,43 +1,54 @@
#!/bin/sh
#
# Log WAN vs LAN ping quality on the Debian edge router.
# Log WAN vs LAN ping quality from any vantage (router, FE, Pi, …).
# Use to compare "rocketship" evenings vs slow periods (ISP/upstream vs local).
#
# Topology (see docs/INFRA.md):
# world -> 134.17.26.161 (router 10.7.6.228) -> 10.7.0.10 (FE) -> 10.7.0.0/8 (BE, …)
#
# Install on router:
# Install:
# sudo cp wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
# sudo chmod +x /usr/local/bin/wan-snapshot.sh
# sudo mkdir -p /var/log
#
# Cron (every 15 minutes):
# */15 * * * * root /usr/local/bin/wan-snapshot.sh
# Cron (every 15 minutes — ONE entry only; script uses flock to skip overlaps):
# */15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
#
# Env overrides:
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# WAN_SNAPSHOT_ROLE=auto|router|fe|pi (default auto from hostname)
# WAN_SNAPSHOT_EXTRAS=1 (0 = ping only, compact v1-style line)
# WAN_SNAPSHOT_LOCK=/var/run/wan-snapshot.lock
# WAN_PING_TARGET=1.1.1.1
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128 (set to "none" to skip)
# BE_PING_TARGET=10.7.16.128 (set to "none" to skip)
# ROUTER_PING_TARGET=none|10.7.6.228 (optional; useful on Pi)
# GW_PING_TARGET=auto|none|<ip> (default auto = default gateway)
# PING_COUNT=10
# PING_DEADLINE_S=15
#
set -eu
WAN_SNAPSHOT_LOG="${WAN_SNAPSHOT_LOG:-/var/log/wan-quality.log}"
WAN_SNAPSHOT_ROLE="${WAN_SNAPSHOT_ROLE:-auto}"
WAN_SNAPSHOT_EXTRAS="${WAN_SNAPSHOT_EXTRAS:-1}"
WAN_SNAPSHOT_LOCK="${WAN_SNAPSHOT_LOCK:-/var/run/wan-snapshot.lock}"
WAN_PING_TARGET="${WAN_PING_TARGET:-1.1.1.1}"
FE_PING_TARGET="${FE_PING_TARGET:-10.7.0.10}"
BE_PING_TARGET="${BE_PING_TARGET-10.7.16.128}"
ROUTER_PING_TARGET="${ROUTER_PING_TARGET:-none}"
GW_PING_TARGET="${GW_PING_TARGET:-auto}"
PING_COUNT="${PING_COUNT:-10}"
PING_DEADLINE_S="${PING_DEADLINE_S:-15}"
usage() {
sed -n '2,22p' "$0"
sed -n '2,28p' "$0"
printf '\nCommands:\n'
printf ' (default) append one snapshot line to the log\n'
printf ' --report summarize today vs yesterday averages\n'
printf ' --tail [N] print last N log lines (default 24)\n'
printf ' --help this text\n'
printf ' (default) append one snapshot line to the log\n'
printf ' --report summarize today vs yesterday\n'
printf ' --tail [N] print last N log lines (default 24)\n'
printf ' --correlate FILE… merge logs by 15-min slot (role= or path basename)\n'
printf ' --help this text\n'
}
need_ping() {
@@ -47,41 +58,187 @@ need_ping() {
}
}
# ping_summary HOST -> prints: loss_pct avg_ms (or loss_pct=-1 avg_ms=-1 on total failure)
detect_role() {
_role="$WAN_SNAPSHOT_ROLE"
if [ "$_role" != "auto" ]; then
printf '%s' "$_role"
return 0
fi
_hn="$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo unknown)"
case "$_hn" in
*raspberry*|*rpi*) printf 'pi' ;;
*monstro*|*router*|*gw*) printf 'router' ;;
*gentoo*|*wg0*|*fe*) printf 'fe' ;;
*) printf 'host' ;;
esac
}
# ping_summary HOST -> loss_pct min_ms avg_ms max_ms mdev_ms (-1 for missing)
ping_summary() {
_host="$1"
_out=""
_out="$(ping -c "$PING_COUNT" -w "$PING_DEADLINE_S" "$_host" 2>/dev/null)" || true
if [ -z "$_out" ]; then
printf '%s %s' "-1" "-1"
printf '%s %s %s %s %s' "-1" "-1" "-1" "-1" "-1"
return 0
fi
_loss="$(printf '%s\n' "$_out" | awk '/packet loss/ {
for (i = 1; i <= NF; i++) if ($i ~ /%$/) { gsub(/%/, "", $i); print $i; exit }
}')"
_avg="$(printf '%s\n' "$_out" | awk -F'[=/ ]' '/min\/avg\/max/ { print $8; exit }')"
_min="-1"
_avg="-1"
_max="-1"
_mdev="-1"
_rtt="$(printf '%s\n' "$_out" | sed -n 's/.*= \([0-9.]*\)\/\([0-9.]*\)\/\([0-9.]*\)\/\([0-9.]*\) ms.*/\1 \2 \3 \4/p' | head -1)"
if [ -n "$_rtt" ]; then
_min="${_rtt%% *}"
_rest="${_rtt#* }"
_avg="${_rest%% *}"
_rest="${_rest#* }"
_max="${_rest%% *}"
_mdev="${_rest#* }"
else
_avg="$(printf '%s\n' "$_out" | awk -F'[=/ ]' '/min\/avg\/max/ { print $8; exit }')"
[ -n "$_avg" ] || _avg="-1"
fi
[ -n "$_loss" ] || _loss="-1"
[ -n "$_avg" ] || _avg="-1"
printf '%s %s' "$_loss" "$_avg"
printf '%s %s %s %s %s' "$_loss" "$_min" "$_avg" "$_max" "$_mdev"
}
append_ping_fields() {
_line="$1"
_prefix="$2"
_host="$3"
_sum="$(ping_summary "$_host")"
_loss="${_sum%% *}"
_rest="${_sum#* }"
_min="${_rest%% *}"
_rest="${_rest#* }"
_avg="${_rest%% *}"
_rest="${_rest#* }"
_max="${_rest%% *}"
_mdev="${_rest#* }"
printf '%s %s=%s %s_loss=%s%% %s_min_ms=%s %s_avg_ms=%s %s_max_ms=%s %s_mdev_ms=%s' \
"$_line" "$_prefix" "$_host" "$_prefix" "$_loss" "$_prefix" "$_min" \
"$_prefix" "$_avg" "$_prefix" "$_max" "$_prefix" "$_mdev"
}
route_dev() {
_host="$1"
ip route get "$_host" 2>/dev/null | awk '{for (i = 1; i <= NF; i++) if ($i == "dev") { print $(i + 1); exit }}'
}
route_gw() {
_host="$1"
ip route get "$_host" 2>/dev/null | awk '{for (i = 1; i <= NF; i++) if ($i == "via") { print $(i + 1); exit }}'
}
default_gw() {
ip route show default 2>/dev/null | awk '{print $3; exit}'
}
load_1m() {
awk '{print $1}' /proc/loadavg 2>/dev/null || echo "n/a"
}
cpu_busy_pct() {
awk '/^cpu / {
idle = $5 + $6; total = 0
for (i = 2; i <= NF; i++) total += $i
if (total > 0) printf "%.0f", 100 * (total - idle) / total
}' /proc/stat 2>/dev/null || echo "n/a"
}
conntrack_pct() {
_cur="$(cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null)" || _cur=""
_max="$(cat /proc/sys/net/netfilter/nf_conntrack_max 2>/dev/null)" || _max=""
if [ -n "$_cur" ] && [ -n "$_max" ] && [ "$_max" -gt 0 ] 2>/dev/null; then
_pct=$((_cur * 100 / _max))
printf 'conntrack=%s/%s(%s%%)' "$_cur" "$_max" "$_pct"
fi
}
neigh_state() {
_host="$1"
ip neigh show "$_host" 2>/dev/null | awk '{print $6; exit}'
}
iface_drops_compact() {
ip -s link 2>/dev/null | awk '
/^[0-9]+:/ {
if (iface != "" && iface !~ /^lo$/) {
if (rx_drop + tx_drop + rx_err + tx_err > 0)
printf "%s:rd=%d,td=%d,re=%d,te=%d ", iface, rx_drop, tx_drop, rx_err, tx_err
}
iface = $2; gsub(":", "", iface)
rx_drop = tx_drop = rx_err = tx_err = 0
}
/RX:/ { getline; rx_err = $3; rx_drop = $4 }
/TX:/ { getline; tx_err = $3; tx_drop = $4 }
END {
if (iface != "" && iface !~ /^lo$/ && rx_drop + tx_drop + rx_err + tx_err > 0)
printf "%s:rd=%d,td=%d,re=%d,te=%d", iface, rx_drop, tx_drop, rx_err, tx_err
}
' | sed 's/ $//'
}
acquire_lock() {
command -v flock >/dev/null 2>&1 || return 0
_lock="$WAN_SNAPSHOT_LOCK"
_dir="$(dirname "$_lock")"
if ! mkdir -p "$_dir" 2>/dev/null || ! : >"$_lock" 2>/dev/null; then
_lock="${TMPDIR:-/tmp}/wan-snapshot.$(id -u 2>/dev/null || echo 0).lock"
fi
exec 9>"$_lock" 2>/dev/null || return 0
flock -n 9 || {
echo "wan-snapshot: skip (already running, lock=$_lock)" >&2
exit 0
}
}
snapshot_once() {
need_ping
_ts="$(date -Iseconds 2>/dev/null || date '+%Y-%m-%dT%H:%M:%S%z')"
_wan="$(ping_summary "$WAN_PING_TARGET")"
_wan_loss="${_wan%% *}"
_wan_avg="${_wan#* }"
_fe="$(ping_summary "$FE_PING_TARGET")"
_fe_loss="${_fe%% *}"
_fe_avg="${_fe#* }"
acquire_lock
_line="ts=$_ts wan=$WAN_PING_TARGET wan_loss=${_wan_loss}% wan_avg_ms=${_wan_avg} fe=$FE_PING_TARGET fe_loss=${_fe_loss}% fe_avg_ms=${_fe_avg}"
_ts="$(date -Iseconds 2>/dev/null || date '+%Y-%m-%dT%H:%M:%S%z')"
_role="$(detect_role)"
_host="$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo unknown)"
_line="ts=$_ts ver=2 role=$_role host=$_host"
_line="$(append_ping_fields "$_line" "wan" "$WAN_PING_TARGET")"
_line="$(append_ping_fields "$_line" "fe" "$FE_PING_TARGET")"
if [ -n "$BE_PING_TARGET" ] && [ "$BE_PING_TARGET" != "none" ]; then
_be="$(ping_summary "$BE_PING_TARGET")"
_be_loss="${_be%% *}"
_be_avg="${_be#* }"
_line="$_line be=$BE_PING_TARGET be_loss=${_be_loss}% be_avg_ms=${_be_avg}"
_line="$(append_ping_fields "$_line" "be" "$BE_PING_TARGET")"
fi
if [ -n "$ROUTER_PING_TARGET" ] && [ "$ROUTER_PING_TARGET" != "none" ]; then
_line="$(append_ping_fields "$_line" "router" "$ROUTER_PING_TARGET")"
fi
_gw_target="$GW_PING_TARGET"
if [ "$_gw_target" = "auto" ]; then
_gw_target="$(default_gw)"
[ -n "$_gw_target" ] || _gw_target="none"
fi
if [ -n "$_gw_target" ] && [ "$_gw_target" != "none" ]; then
_line="$(append_ping_fields "$_line" "gw" "$_gw_target")"
fi
if [ "$WAN_SNAPSHOT_EXTRAS" = "1" ]; then
_line="$_line load1=$(load_1m) cpu_busy=$(cpu_busy_pct)%"
_ct="$(conntrack_pct)"
[ -n "$_ct" ] && _line="$_line $_ct"
_wdev="$(route_dev "$WAN_PING_TARGET")"
_fdev="$(route_dev "$FE_PING_TARGET")"
[ -n "$_wdev" ] && _line="$_line route_wan=$_wdev"
[ -n "$_fdev" ] && _line="$_line route_fe=$_fdev"
_wgw="$(route_gw "$WAN_PING_TARGET")"
[ -n "$_wgw" ] && _line="$_line via_wan=$_wgw"
_nfe="$(neigh_state "$FE_PING_TARGET")"
[ -n "$_nfe" ] && _line="$_line neigh_fe=$_nfe"
_drops="$(iface_drops_compact)"
[ -n "$_drops" ] && _line="$_line drops=\"$_drops\""
fi
_dir="$(dirname "$WAN_SNAPSHOT_LOG")"
@@ -90,8 +247,25 @@ snapshot_once() {
printf '%s\n' "$_line"
}
# Average wan_avg_ms / fe_avg_ms for lines matching DATE prefix (YYYY-MM-DD).
day_avg() {
# Extract numeric field from log line (supports wan_avg_ms and legacy layouts).
field_value() {
_line="$1"
_field="$2"
printf '%s\n' "$_line" | awk -v f="$_field" '
{
for (i = 1; i <= NF; i++) {
if ($i ~ "^" f "=") {
split($i, a, "=")
v = a[2]
gsub(/%/, "", v)
if (v + 0 == v) { print v; exit }
}
}
}
'
}
day_stats() {
_log="$1"
_date="$2"
_field="$3"
@@ -103,13 +277,21 @@ day_avg() {
split($i, a, "=")
v = a[2]
gsub(/%/, "", v)
if (v + 0 == v && v >= 0) { n++; s += v }
if (v + 0 == v && v >= 0) {
n++; s += v; if (v > max) max = v
if (v >= 50) bad++
vals[n] = v
}
}
}
}
END {
if (n > 0) printf "%.2f", s / n
else print "n/a"
if (n == 0) { print "n/a n/a n/a n/a"; exit }
asort(vals)
p50 = vals[int((n + 1) * 0.5)]
p90 = vals[int(n * 0.9)]
if (p90 == "") p90 = vals[n]
printf "%.2f %.2f %.2f %.0f %d/%d", s / n, p50, p90, max, bad + 0, n
}
' "$_log"
}
@@ -137,6 +319,30 @@ day_max_loss() {
' "$_log"
}
report_day_block() {
_log="$1"
_label="$2"
_date="$3"
_samples="$(grep -c "^ts=$_date" "$_log" 2>/dev/null || echo 0)"
printf '%s=%s samples=%s\n' "$_label" "$_date" "$_samples"
for _pfx in wan fe be router gw; do
_stats="$(day_stats "$_log" "$_date" "${_pfx}_avg_ms")"
_mean="${_stats%% *}"
_rest="${_stats#* }"
_p50="${_rest%% *}"
_rest="${_rest#* }"
_p90="${_rest%% *}"
_rest="${_rest#* }"
_max="${_rest%% *}"
_bad="${_rest#* }"
_loss="$(day_max_loss "$_log" "$_date" "${_pfx}_loss")"
if [ "$_mean" != "n/a" ]; then
printf ' %s: mean=%s p50=%s p90=%s max=%s bad(>=50ms)=%s loss_max=%s%%\n' \
"$_pfx" "$_mean" "$_p50" "$_p90" "$_max" "$_bad" "$_loss"
fi
done
}
report() {
_log="$WAN_SNAPSHOT_LOG"
_today="$(date '+%Y-%m-%d')"
@@ -147,31 +353,22 @@ report() {
exit 1
fi
_samples_today="$(grep -c "^ts=$_today" "$_log" 2>/dev/null || echo 0)"
printf 'log=%s\n' "$_log"
printf 'today=%s samples=%s\n' "$_today" "$_samples_today"
printf ' wan_avg_ms=%s wan_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_today" "wan_avg_ms")" \
"$(day_max_loss "$_log" "$_today" "wan_loss")"
printf ' fe_avg_ms=%s fe_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_today" "fe_avg_ms")" \
"$(day_max_loss "$_log" "$_today" "fe_loss")"
_role="$(detect_role)"
printf 'log=%s role=%s\n' "$_log" "$_role"
report_day_block "$_log" "today" "$_today"
if [ -n "$_yesterday" ] && grep -q "^ts=$_yesterday" "$_log" 2>/dev/null; then
_samples_y="$(grep -c "^ts=$_yesterday" "$_log" 2>/dev/null || echo 0)"
printf 'yesterday=%s samples=%s\n' "$_yesterday" "$_samples_y"
printf ' wan_avg_ms=%s wan_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_yesterday" "wan_avg_ms")" \
"$(day_max_loss "$_log" "$_yesterday" "wan_loss")"
printf ' fe_avg_ms=%s fe_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_yesterday" "fe_avg_ms")" \
"$(day_max_loss "$_log" "$_yesterday" "fe_loss")"
report_day_block "$_log" "yesterday" "$_yesterday"
fi
printf '\nInterpretation:\n'
printf ' wan bad + fe good -> likely ISP/upstream (world -> 134.17.26.161)\n'
printf ' wan + fe bad -> router or LAN (10.7.6.228 -> 10.7.0.10)\n'
printf ' wan + fe good, apps slow -> DNAT/nginx above ping path\n'
printf ' wan bad + fe good -> ISP/upstream or route_dev issue\n'
printf ' wan + fe bad -> router/LAN/tunnel path\n'
printf ' fe bad (pi) + fe ok (fe) -> path to FE, not FE host CPU\n'
printf ' high mdev / max vs avg -> jitter (bufferbloat, conntrack, VPN)\n'
printf ' conntrack>80%% -> edge table pressure (router)\n'
printf ' neigh_fe not REACHABLE -> ARP/L2 flap toward FE\n'
printf ' ping good, HTTPS slow -> DNAT/nginx above this probe\n'
}
tail_log() {
@@ -183,6 +380,116 @@ tail_log() {
tail -n "$_n" "$WAN_SNAPSHOT_LOG"
}
# Guess role label for correlate when log line has no role=.
log_role_label() {
_path="$1"
_line="$2"
_role="$(field_value "$_line" "role")"
if [ -n "$_role" ] && [ "$_role" != "0" ]; then
printf '%s' "$_role"
return 0
fi
case "$_path" in
*raspberry*|*rpi*) printf 'pi' ;;
*FE_gentoo*|*gentoo*) printf 'fe' ;;
*monstro*|*router*) printf 'router' ;;
*) printf '%s' "$(basename "$_path" .log)" ;;
esac
}
# Bucket timestamp to 15-min wall-clock slot: YYYY-MM-DDTHH:00|15|30|45
slot_key() {
_ts="$1"
_date="${_ts%%T*}"
_rest="${_ts#*T}"
_hour="${_rest%%:*}"
_min="${_rest#*:}"
_min="${_min%%:*}"
_min="${_min%%+*}"
_min="${_min%%-*}"
_bucket=$(( (_min / 15) * 15 ))
printf '%sT%s:%02d\n' "$_date" "$_hour" "$_bucket"
}
correlate_logs() {
if [ "$#" -lt 1 ]; then
echo "wan-snapshot: --correlate needs at least one log file" >&2
exit 1
fi
_tmp="$(mktemp "${TMPDIR:-/tmp}/wan-correlate.XXXXXX")"
# shellcheck disable=SC2064
trap 'rm -f "$_tmp"' EXIT INT HUP
for _path in "$@"; do
[ -r "$_path" ] || {
echo "wan-snapshot: unreadable: $_path" >&2
continue
}
_label="$(basename "$_path")"
while IFS= read -r _line || [ -n "$_line" ]; do
case "$_line" in
ts=*) ;;
*) continue ;;
esac
_ts="${_line#ts=}"
_ts="${_ts%% *}"
_slot="$(slot_key "$_ts")"
_role="$(log_role_label "$_path" "$_line")"
printf '%s\t%s\t%s\t%s\t%s\n' "$_slot" "$_role" "$_ts" "$_label" "$_line" >> "$_tmp"
done < "$_path"
done
printf 'slot\t'
_roles="$(awk -F'\t' '{print $2}' "$_tmp" | sort -u | tr '\n' ' ')"
for _role in $_roles; do
printf '%s_wan\t%s_fe\t%s_be\t' "$_role" "$_role" "$_role"
done
printf '\n'
awk -F'\t' '
{
slot = $1; role = $2; ts = $3; line = $5
key = slot SUBSEP role
if (!(key in best_ts) || ts > best_ts[key]) {
best_ts[key] = ts
wan = fe = be = "-"
for (i = 1; i <= split(line, a, " "); i++) {
if (a[i] ~ /^wan_avg_ms=/) { split(a[i], b, "="); wan = b[2] }
if (a[i] ~ /^fe_avg_ms=/) { split(a[i], b, "="); fe = b[2] }
if (a[i] ~ /^be_avg_ms=/) { split(a[i], b, "="); be = b[2] }
}
W[slot, role, "wan"] = wan
W[slot, role, "fe"] = fe
W[slot, role, "be"] = be
slots[slot] = 1
roles[role] = 1
}
}
END {
nroles = 0
for (r in roles) rolelist[++nroles] = r
asort(rolelist)
nslots = 0
for (s in slots) slotlist[++nslots] = s
asort(slotlist)
for (si = 1; si <= nslots; si++) {
s = slotlist[si]
printf "%s", s
for (ri = 1; ri <= nroles; ri++) {
r = rolelist[ri]
printf "\t%s\t%s\t%s", W[s, r, "wan"] + 0 ? W[s, r, "wan"] : "-", \
W[s, r, "fe"] + 0 ? W[s, r, "fe"] : "-", \
W[s, r, "be"] + 0 ? W[s, r, "be"] : "-"
}
printf "\n"
}
}
' "$_tmp" | while IFS= read -r _row; do
printf '%s\n' "$_row"
done
}
case "${1:-}" in
-h|--help|help)
usage
@@ -193,6 +500,10 @@ case "${1:-}" in
--tail|tail)
tail_log "${2:-24}"
;;
--correlate|correlate)
shift
correlate_logs "$@"
;;
"")
snapshot_once
;;