ac/android_cast
1
0
Fork 0
mirror of git://f0xx.org/android_cast synced 2026-07-29 02:18:40 +03:00
Code Issues Packages Projects Releases Wiki Activity

small sync

Browse Source
This commit is contained in:
Anton Afanasyeu
2026-06-12 18:48:03 +02:00
parent 170211ef1f
commit 539341d8f1
8 changed files with 653 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/drafts/20100612_1_scaling.txt Normal file
View File

@@ -0,0 +1,47 @@
The problem:
Now we have a small VM instance which serves not so much of things. A short list of services include:
- own web service (nginx+php) for UI/UX and some control panels, serving issues, tickets, graphs, build control (orchestration)
- git UI (gitea)
- database service - data store
- minimalistic OTA service for mobile app
- minimalistic WireGuard provider
- url shortener (nginx+php)
- build service (docker, orchestration)
The problem is, sooner or later we'll need to expand the amount of VM instances, delegate some features to separate virtual machines, most probably migrate to the clouds (GCP, Azure...), proper scaling with zero downtime with the ability of seamlessdeploy/update, add new services - and all this bundle should work ultra-stable, durable environments, splitting everything to the clusters of dev, staging, prod in a way that, for example, one deploys the unstable features to dev cluster, then the initial testing occurs, after decision making - move to the staging for beta tests, and finally to production once the decision is that staging behaves stable.
The how:
The proposed way is:
- one, let's say, cluster (let's call it dev cluster for example), which has N virtual machines, each of:
** web frontent VM(s) + OTA (linked together with common datastore VMs)
** database VM(s)
** data store VM(s)
** WireGuard VM(s)
** URL shortener VM + gitea for developers (1 small instance should be enough I believe)
** build service VM(s)
** N VMs for new services and features (starting from 0 and depend on the features)
- more tighter (money-wise) approach
- other
TODO: consult with AI for the others and the best practices of building scalable systems like proposed one depend on the current services and future needs
The simulation:
In theory, sysop (system operator aka project owner as for now) may simulate any (meaningful) number of HVMs for cluster simulation based on something lightweight like alpine linux distro plus sets of packages pre-installed.
The solution:
Ask AI to collect the information, make its decision based on all that we currently have plus look deeper into the future. The solution might be balanced between cost and providing services, for example, we don't need CPU or RAM resources for data storage but CPU/RAM is important for build service, etc.
Also, the matter of seamless scaling is super important, better to have the load balancing ready to use just from cloud provider.
Also, let AI think of how many simulatltaneous connections can we have per instance - the question is for RSSH implementation and future improvements
The future key role requirements:
- multipoint casting and routing outside the intra network, usage of 3rdparty or own multipoint solutions on the backends(MCU, SFU...)
- new RBAC control, new roles - now we're thinking as developers and deliver the max of services to developers, but we will migrate to the other model sooner or later
- both realtime and non-realtime data streaming solutions, including streams re-compression (from any input to any output that ffmpeg can do for us, so ffmpeg will be involved)
- casts storage - we'll keep the records on the backends and provide them on demand if the user pays for it
- VPN capacities (question for all the resources), anonymous realtime VPN provider services (payed)
- payment systems integration (should cost near 0 for the developer's point of view of scaling/growth, it's more about marketing)
- potential payed add-on services (not decided yet on which side, on the backends or on mobiles) - user face masks, background extraction and replacement, etc.
- in-app money transfers between the app users (payed service)
- DNS control if possible - meaning selling the subdomains to the users, provide simplified own site builders accessible by own domains...
- 3rd party streaming providers integrations: realtime and non-realtime, let them stream and cast using our platform (I see ffmpeg active usage there plus cast handlers: low delay SFU-alike, mid-high delay using m3u playlists or similar hls-alike technologies)
AI should take everything described there plus the current project structure into account, weight everything, prioritize, use beautiful graphs and visual diagrams, and create SPEC/DR/ documents out of this draft using our common templates.

103
examples/crash_reporter/backend/scripts/ensure_url_shortener_prod_config.php Executable file
View File

@@ -0,0 +1,103 @@
#!/usr/bin/env php81
<?php
/**
* Restore url-shortener prod config after rsync/submodule deploy (config.php is gitignored).
*
* On BE (as user that can read crashes config.php):
* php81 examples/crash_reporter/backend/scripts/ensure_url_shortener_prod_config.php
*
* - Writes backend/url-shortener/config/config.php (via apps/s symlink when present)
* - Ensures crashes config.php has url_shortener.enabled + db block
*/
declare(strict_types=1);
$repoRoot = realpath(dirname(__DIR__, 3)) ?: dirname(__DIR__, 3);
$crashConfigPath = $repoRoot . '/examples/crash_reporter/backend/config/config.php';
$urlConfigDir = $repoRoot . '/backend/url-shortener/config';
$urlConfigPath = $urlConfigDir . '/config.php';
$sLink = '/var/www/localhost/htdocs/apps/s';
if (is_link($sLink)) {
$resolved = realpath($sLink);
if ($resolved !== false) {
$urlConfigDir = $resolved . '/config';
$urlConfigPath = $urlConfigDir . '/config.php';
}
}
function fail(string $msg): void {
fwrite(STDERR, "ensure_url_shortener_prod_config: $msg\n");
exit(1);
}
if (!is_readable($crashConfigPath)) {
fail("missing crashes config: $crashConfigPath");
}
/** @var array<string,mixed> $crash */
$crash = require $crashConfigPath;
$m = $crash['db']['mysql'] ?? null;
if (!is_array($m) || ($m['password'] ?? '') === '') {
fail('crashes db.mysql.password not set in config.php');
}
$user = (string) ($m['username'] ?? 'androidcast');
$pass = (string) $m['password'];
$socket = (string) ($m['socket'] ?? '/run/mysqld/mysqld.sock');
try {
$pdo = new PDO(
"mysql:unix_socket={$socket};dbname=url_shortener;charset=utf8mb4",
$user,
$pass,
[PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]
);
$pdo->query('SELECT 1');
} catch (Throwable $e) {
fail('url_shortener DB ping failed: ' . $e->getMessage());
}
$urlBlock = [
'db' => [
'driver' => 'mysql',
'mysql' => [
'host' => '127.0.0.1',
'port' => 3306,
'socket' => $socket,
'database' => 'url_shortener',
'username' => $user,
'password' => $pass,
'charset' => 'utf8mb4',
],
],
'public_base' => 'https://s.f0xx.org',
];
if (!is_dir($urlConfigDir)) {
fail("url-shortener config dir missing: $urlConfigDir");
}
file_put_contents($urlConfigPath, "<?php\nreturn " . var_export($urlBlock, true) . ";\n");
chmod($urlConfigPath, 0644);
$crashUrl = [
'enabled' => true,
'public_base' => 'https://s.f0xx.org',
'db' => [
'mysql' => [
'host' => '127.0.0.1',
'port' => 3306,
'socket' => $socket,
'database' => 'url_shortener',
'username' => $user,
'password' => $pass,
'charset' => 'utf8mb4',
],
],
];
$crash['url_shortener'] = $crashUrl;
file_put_contents($crashConfigPath, "<?php\nreturn " . var_export($crash, true) . ";\n");
echo "ok: $urlConfigPath\n";
echo "ok: crashes url_shortener.enabled=true\n";

30
examples/crash_reporter/backend/scripts/gitea/gitea_attach_remote.sh
View File

@@ -106,25 +106,7 @@ resolve_gitea_owner() {
}
resolve_push_token() {
if [ -n "${GITEA_ADMIN_TOKEN:-}" ]; then
printf '%s' "$GITEA_ADMIN_TOKEN"
return 0
fi
_token=""
_token="$(gitea_cli "admin" "user" "generate-access-token" \
"--username" "$GITEA_OWNER" \
"--token-name" "attach-$$" \
"--scopes" "all" \
"--raw" 2>/dev/null)" && [ -n "$_token" ] && printf '%s' "$_token" && return 0
_admin="$(gitea_pick_owner_username)"
if [ -n "$_admin" ] && [ "$_admin" != "$GITEA_OWNER" ]; then
_token="$(gitea_cli "admin" "user" "generate-access-token" \
"--username" "$_admin" \
"--token-name" "attach-$$" \
"--scopes" "all" \
"--raw" 2>/dev/null)" && [ -n "$_token" ] && printf '%s' "$_token" && return 0
fi
return 1
gitea_resolve_push_token
}
create_repo_via_cli() {
@@ -173,15 +155,7 @@ create_repo_via_api() {
}
gitea_owner_uid() {
need_cmd curl
_token="${GITEA_ADMIN_TOKEN:-}"
[ -n "$_token" ] || _token="$(resolve_push_token)" || return 1
_api="$(gitea_api_base)"
_uid="$(curl -sS "$_api/users/$GITEA_OWNER" \
-H "Authorization: token $_token" \
| awk -F'"' '/"id"/ { print $4; exit }')"
[ -n "$_uid" ] || return 1
printf '%s' "$_uid"
gitea_entity_uid_for "$GITEA_OWNER"
}
migrate_via_api() {

21
examples/crash_reporter/backend/scripts/gitea/gitea_attach_url_shortener.sh
View File

@@ -66,17 +66,24 @@ for _from in $GITEA_TRANSFER_SOURCES; do
fi
done
# 3) Pull mirror (attach when missing).
# 3) Pull mirror (attach when missing). Resolve PAT once for all API calls this run.
_gitea_token="$(gitea_resolve_push_token)" || true
[ -n "$_gitea_token" ] && GITEA_ADMIN_TOKEN="$_gitea_token" && export GITEA_ADMIN_TOKEN
if gitea_repo_exists_api "$GITEA_OWNER" "$GITEA_REPO"; then
log "repo exists — trigger mirror sync"
gitea_sync_mirror_api "$GITEA_OWNER" "$GITEA_REPO" || warn "mirror sync API failed — try gitea_sync_mirror.sh"
else
log "attach pull mirror via gitea_attach_remote.sh ..."
GITEA_OWNER="$GITEA_OWNER" GITEA_REPO="$GITEA_REPO" \
GITEA_MIRROR_URL="$GITEA_MIRROR_URL" \
GITEA_MIRROR_INTERVAL="$GITEA_MIRROR_INTERVAL" \
GITEA_PRIVATE="$GITEA_PRIVATE" \
sh "$SCRIPT_DIR/gitea_attach_remote.sh" "$GITEA_MIRROR_URL" || exit 1
log "attach pull mirror (API migrate) ..."
if ! gitea_mirror_pull_repo "$GITEA_MIRROR_URL" "$GITEA_REPO" "$GITEA_OWNER" \
"$GITEA_MIRROR_INTERVAL" ""; then
warn "API migrate failed — trying gitea_attach_remote.sh fallback ..."
GITEA_OWNER="$GITEA_OWNER" GITEA_REPO="$GITEA_REPO" \
GITEA_MIRROR_URL="$GITEA_MIRROR_URL" \
GITEA_MIRROR_INTERVAL="$GITEA_MIRROR_INTERVAL" \
GITEA_PRIVATE="$GITEA_PRIVATE" \
sh "$SCRIPT_DIR/gitea_attach_remote.sh" "$GITEA_MIRROR_URL" || exit 1
fi
gitea_sync_mirror_api "$GITEA_OWNER" "$GITEA_REPO" || warn "initial sync pending — check [migrations] / git daemon"
fi

20
examples/crash_reporter/backend/scripts/gitea/gitea_mirror_lib.sh
View File

@@ -42,20 +42,32 @@ gitea_rm_path() {
rm -rf "$1" 2>/dev/null || true
}
# First numeric "id" from Gitea JSON (handles "id":2 and "id":"2").
gitea_json_extract_id() {
printf '%s' "$1" | grep -o '"id":[0-9]*' | head -1 | grep -o '[0-9]*'
}
gitea_resolve_push_token() {
if [ -n "${GITEA_ADMIN_TOKEN:-}" ]; then
printf '%s' "$GITEA_ADMIN_TOKEN"
return 0
fi
# One PAT per script run (re-generating with the same --token-name fails).
if [ -n "${_GITEA_RESOLVED_TOKEN:-}" ]; then
printf '%s' "$_GITEA_RESOLVED_TOKEN"
return 0
fi
_admin="$(gitea_pick_owner_username)"
[ -n "$_admin" ] || return 1
_token=""
_token="$(gitea_cli "admin" "user" "generate-access-token" \
"--username" "$_admin" \
"--token-name" "mirror-$$" \
"--token-name" "mirror-$$-$(date +%s)-${RANDOM:-0}" \
"--scopes" "all" \
"--raw" 2>/dev/null)" && [ -n "$_token" ] && printf '%s' "$_token" && return 0
return 1
"--raw" 2>/dev/null)" && [ -n "$_token" ] || return 1
_GITEA_RESOLVED_TOKEN="$_token"
export _GITEA_RESOLVED_TOKEN GITEA_ADMIN_TOKEN="$_token"
printf '%s' "$_token"
}
gitea_owner_uid_for() {
@@ -76,7 +88,7 @@ gitea_entity_uid_for() {
else
_json="$(curl -sS "$_api/users/$_owner" -H "Authorization: token $_token")"
fi
_uid="$(printf '%s' "$_json" | awk -F'"' '/"id":/ { print $4; exit }')"
_uid="$(gitea_json_extract_id "$_json")"
[ -n "$_uid" ] || return 1
printf '%s' "$_uid"
}

123
examples/network/README.md
View File

@@ -1,56 +1,108 @@
# Edge network probes
Scripts for the **Debian router** (`134.17.26.161` / `10.7.6.228`) — not FE/BE app hosts.
Scripts for **router**, **FE**, and **external vantage** (e.g. Raspberry Pi) — not BE app hosts.
## `wan-snapshot.sh`
## `wan-snapshot.sh` (v2)
Periodic ping comparison:
Periodic ping comparison plus lightweight host/path context.
| Target | Default | Meaning |
|--------|---------|---------|
| **WAN** | `1.1.1.1` | ISP/upstream leg (outbound from router) |
| **FE** | `10.7.0.10` | LAN leg router → Gentoo FE |
| **BE** | `10.7.16.128` | Optional deeper LAN (disable: `BE_PING_TARGET=none`) |
| **WAN** | `1.1.1.1` | ISP/upstream leg |
| **FE** | `10.7.0.10` | Path to Gentoo FE |
| **BE** | `10.7.16.128` | Deeper LAN (disable: `BE_PING_TARGET=none`) |
| **router** | `none` | Optional modem ping (`ROUTER_PING_TARGET` on Pi) |
| **gw** | auto | Default gateway ping |
### Install (router)
Each target logs: `loss`, `min`, `avg`, `max`, `mdev` (jitter).
Extras (`WAN_SNAPSHOT_EXTRAS=1`): `load1`, `cpu_busy`, `conntrack`, `route_wan`/`route_fe`, `via_wan`, `neigh_fe`, non-zero `drops`.
Log format: `ver=2 role=pi|fe|router host=…` — older lines without `ver=` still work in `--report` / `--correlate`.
### Install (any host)
```bash
sudo cp examples/network/wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
sudo chmod +x /usr/local/bin/wan-snapshot.sh
sudo cp examples/network/wan-snapshot.env.example /etc/conf.d/wan-snapshot
# edit /etc/conf.d/wan-snapshot for role-specific targets
```
### Cron (every 15 minutes)
### Cron (every 15 minutes — **one** entry per host)
```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
*/15 * * * * root . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
```
The script uses `flock` on `/var/run/wan-snapshot.lock` so duplicate crontab lines do not double-log.
Log: `/var/log/wan-quality.log` (override with `WAN_SNAPSHOT_LOG`).
### Suggested `/etc/conf.d/wan-snapshot`
**Raspberry Pi** (path into infra matters most):
```sh
WAN_SNAPSHOT_ROLE=pi
ROUTER_PING_TARGET=10.7.6.228
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
**Gentoo FE**:
```sh
WAN_SNAPSHOT_ROLE=fe
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
ROUTER_PING_TARGET=10.7.6.228
```
**Debian router**:
```sh
WAN_SNAPSHOT_ROLE=router
FE_PING_TARGET=10.7.0.10
BE_PING_TARGET=10.7.16.128
```
### Usage
```bash
# one line appended + printed
sudo /usr/local/bin/wan-snapshot.sh
# compare tonight vs yesterday
# today vs yesterday (mean, p50, p90, bad>=50ms counts)
/usr/local/bin/wan-snapshot.sh --report
# recent samples
/usr/local/bin/wan-snapshot.sh --tail 48
# merge Pi + FE (+ router) logs by 15-min slot
/usr/local/bin/wan-snapshot.sh --correlate \
tmp/raspberrypi/var/log/wan-quality.log \
tmp/FE_gentoo/var/log/wan-quality.log
```
Example log line:
Example v2 line:
```text
ts=2026-06-04T21:15:01+02:00 wan=1.1.1.1 wan_loss=0% wan_avg_ms=11.2 fe=10.7.0.10 fe_loss=0% fe_avg_ms=0.3 be=10.7.16.128 be_loss=0% be_avg_ms=0.5
ts=2026-06-12T15:30:01+03:00 ver=2 role=fe host=wg0 wan=1.1.1.1 wan_loss=0% wan_min_ms=9.1 wan_avg_ms=10.6 wan_max_ms=12.4 wan_mdev_ms=0.8 fe=10.7.0.10 fe_loss=0% fe_min_ms=0.05 fe_avg_ms=0.07 fe_max_ms=0.10 fe_mdev_ms=0.01 be=10.7.16.128 be_loss=0% be_avg_ms=0.20 load1=0.12 cpu_busy=4% route_wan=eno0 route_fe=lo neigh_fe=REACHABLE
```
### Reading results
- **`wan_avg_ms` high, `fe_avg_ms` low** → ISP/upstream (your “rocketship vs gap” bet).
- **Both high** → router CPU, conntrack, or local link.
- **Ping good, HTTPS/git slow** → DNAT or service layer (not measured here).
| Pattern | Likely cause |
|---------|----------------|
| `wan` bad, `fe` good (on router) | ISP/upstream |
| `wan` + `fe` bad on router | modem CPU, conntrack, LAN |
| `fe` bad on **pi**, `fe` good on **fe** | path/tunnel to FE, not FE VM |
| `wan_max_ms` >> `wan_avg_ms` or high `mdev` | jitter / bufferbloat |
| `conntrack>80%` | edge table full |
| `neigh_fe` not `REACHABLE` | ARP/L2 toward FE |
| `route_fe=tun0` / `ppp0` on Pi | metric is tunnel quality |
| ping good, HTTPS slow | DNAT/nginx (not measured here) |
From **outside** LAN, still run occasionally:
@@ -58,62 +110,49 @@ From **outside** LAN, still run occasionally:
mtr -rwzc 50 134.17.26.161
```
That catches path issues ping from the router cannot see.
## `monstro-netdiag.sh` (bottleneck + iptables + processes)
Run on the **Debian edge router** (`10.7.6.228`, same host as `wan-snapshot.sh`) when infra feels stuck.
Run on the **Debian edge router** (`10.7.6.228`) when infra feels stuck. Complements `wan-snapshot` with on-demand depth.
```bash
sudo cp examples/network/monstro-netdiag.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/monstro-netdiag.sh
```
**Cron** (pair with wan-snapshot):
**Cron** (optional; pair with wan-snapshot on router only):
```cron
*/15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
*/15 * * * * root /usr/local/bin/monstro-netdiag.sh --snapshot >>/var/log/monstro-netdiag.cron.log 2>&1
```
**When sluggish** (run immediately, compare to a “good” capture):
**When sluggish**:
```bash
sudo monstro-netdiag.sh --full # ping + conntrack + iptables -v + top CPU
sudo monstro-netdiag.sh --watch 30 # stream one-liners every 30s
sudo monstro-netdiag.sh --full
sudo monstro-netdiag.sh --watch 30
monstro-netdiag.sh --tail 48
```
One-line log (`/var/log/monstro-netdiag.log`) includes: `load1`, `cpu_busy`, `conntrack=cur/max (%)`, `top=process`, `fe_avg_ms`, `wan_avg_ms`, iface drops.
### Why intermittent (excluding ISP)?
Your data showed **WAN ~9 ms** but **FE ~230 ms** — classic **intra-LAN** flapping:
| Cause | What to look for in `--full` |
|-------|------------------------------|
| **conntrack table full** | `conntrack` >80%, new flows stall |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` |
| Cause | What to look for |
|-------|------------------|
| **conntrack table full** | `conntrack>80%` in wan-snapshot or monstro |
| **Router CPU / softirq** | high `cpu_busy`, `softnet dropped` in `--full` |
| **iptables rule cost** | FORWARD/NAT rules with huge packet counts |
| **FE/BE VM host busy** | FE ping high from router, low load on router |
| **ARP/route flap** | `ip neigh` INCOMPLETE/FAILED for 10.7.0.10 |
| **Bufferbloat on LAN** | rising `fe_avg_ms` under load, 0% loss |
| **FE/BE VM host busy** | FE ping high from router, low `load1` on router |
| **ARP/route flap** | `neigh_fe` not REACHABLE |
| **Bufferbloat on LAN** | high `mdev` / `max`, 0% loss |
Good vs bad: save one `--full` output on a rocketship evening and diff when stuck.
### Safe checks on monstro (read-only)
```bash
# conntrack pressure
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# hottest iptables rules (no changes)
iptables -L FORWARD -n -v --line-numbers | sort -k1 -rn | head
iptables -t nat -L PREROUTING -n -v --line-numbers | head -20
# who eats CPU during slowness
ps -eo pcpu,pmem,comm --sort=-pcpu | head
iptables -L FORWARD -n -v --line-numbers | head
ip neigh show 10.7.0.10
```
Do **not** change iptables/NAT on monstro without a rollback plan (per infra policy).

27
examples/network/wan-snapshot.env.example Normal file
View File

@@ -0,0 +1,27 @@
# Copy to /etc/conf.d/wan-snapshot on each host (optional).
# Source from cron: . /etc/conf.d/wan-snapshot 2>/dev/null; /usr/local/bin/wan-snapshot.sh
# --- Raspberry Pi (external / home vantage) ---
# WAN_SNAPSHOT_ROLE=pi
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# ROUTER_PING_TARGET=10.7.6.228
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128
# --- Gentoo FE ---
# WAN_SNAPSHOT_ROLE=fe
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128
# ROUTER_PING_TARGET=10.7.6.228
# --- Debian edge router (monstro) ---
# WAN_SNAPSHOT_ROLE=router
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128
# Shared
# WAN_SNAPSHOT_EXTRAS=1
# PING_COUNT=10
# PING_DEADLINE_S=15

415
examples/network/wan-snapshot.sh
View File

@@ -1,43 +1,54 @@
#!/bin/sh
#
# Log WAN vs LAN ping quality on the Debian edge router.
# Log WAN vs LAN ping quality from any vantage (router, FE, Pi, …).
# Use to compare "rocketship" evenings vs slow periods (ISP/upstream vs local).
#
# Topology (see docs/INFRA.md):
# world -> 134.17.26.161 (router 10.7.6.228) -> 10.7.0.10 (FE) -> 10.7.0.0/8 (BE, …)
#
# Install on router:
# Install:
# sudo cp wan-snapshot.sh /usr/local/bin/wan-snapshot.sh
# sudo chmod +x /usr/local/bin/wan-snapshot.sh
# sudo mkdir -p /var/log
#
# Cron (every 15 minutes):
# */15 * * * * root /usr/local/bin/wan-snapshot.sh
# Cron (every 15 minutes — ONE entry only; script uses flock to skip overlaps):
# */15 * * * * root /usr/local/bin/wan-snapshot.sh >>/var/log/wan-quality.cron.log 2>&1
#
# Env overrides:
# WAN_SNAPSHOT_LOG=/var/log/wan-quality.log
# WAN_SNAPSHOT_ROLE=auto|router|fe|pi (default auto from hostname)
# WAN_SNAPSHOT_EXTRAS=1 (0 = ping only, compact v1-style line)
# WAN_SNAPSHOT_LOCK=/var/run/wan-snapshot.lock
# WAN_PING_TARGET=1.1.1.1
# FE_PING_TARGET=10.7.0.10
# BE_PING_TARGET=10.7.16.128 (set to "none" to skip)
# BE_PING_TARGET=10.7.16.128 (set to "none" to skip)
# ROUTER_PING_TARGET=none|10.7.6.228 (optional; useful on Pi)
# GW_PING_TARGET=auto|none|<ip> (default auto = default gateway)
# PING_COUNT=10
# PING_DEADLINE_S=15
#
set -eu
WAN_SNAPSHOT_LOG="${WAN_SNAPSHOT_LOG:-/var/log/wan-quality.log}"
WAN_SNAPSHOT_ROLE="${WAN_SNAPSHOT_ROLE:-auto}"
WAN_SNAPSHOT_EXTRAS="${WAN_SNAPSHOT_EXTRAS:-1}"
WAN_SNAPSHOT_LOCK="${WAN_SNAPSHOT_LOCK:-/var/run/wan-snapshot.lock}"
WAN_PING_TARGET="${WAN_PING_TARGET:-1.1.1.1}"
FE_PING_TARGET="${FE_PING_TARGET:-10.7.0.10}"
BE_PING_TARGET="${BE_PING_TARGET-10.7.16.128}"
ROUTER_PING_TARGET="${ROUTER_PING_TARGET:-none}"
GW_PING_TARGET="${GW_PING_TARGET:-auto}"
PING_COUNT="${PING_COUNT:-10}"
PING_DEADLINE_S="${PING_DEADLINE_S:-15}"
usage() {
sed -n '2,22p' "$0"
sed -n '2,28p' "$0"
printf '\nCommands:\n'
printf ' (default) append one snapshot line to the log\n'
printf ' --report summarize today vs yesterday averages\n'
printf ' --tail [N] print last N log lines (default 24)\n'
printf ' --help this text\n'
printf ' (default) append one snapshot line to the log\n'
printf ' --report summarize today vs yesterday\n'
printf ' --tail [N] print last N log lines (default 24)\n'
printf ' --correlate FILE… merge logs by 15-min slot (role= or path basename)\n'
printf ' --help this text\n'
}
need_ping() {
@@ -47,41 +58,187 @@ need_ping() {
}
}
# ping_summary HOST -> prints: loss_pct avg_ms (or loss_pct=-1 avg_ms=-1 on total failure)
detect_role() {
_role="$WAN_SNAPSHOT_ROLE"
if [ "$_role" != "auto" ]; then
printf '%s' "$_role"
return 0
fi
_hn="$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo unknown)"
case "$_hn" in
*raspberry*|*rpi*) printf 'pi' ;;
*monstro*|*router*|*gw*) printf 'router' ;;
*gentoo*|*wg0*|*fe*) printf 'fe' ;;
*) printf 'host' ;;
esac
}
# ping_summary HOST -> loss_pct min_ms avg_ms max_ms mdev_ms (-1 for missing)
ping_summary() {
_host="$1"
_out=""
_out="$(ping -c "$PING_COUNT" -w "$PING_DEADLINE_S" "$_host" 2>/dev/null)" || true
if [ -z "$_out" ]; then
printf '%s %s' "-1" "-1"
printf '%s %s %s %s %s' "-1" "-1" "-1" "-1" "-1"
return 0
fi
_loss="$(printf '%s\n' "$_out" | awk '/packet loss/ {
for (i = 1; i <= NF; i++) if ($i ~ /%$/) { gsub(/%/, "", $i); print $i; exit }
}')"
_avg="$(printf '%s\n' "$_out" | awk -F'[=/ ]' '/min\/avg\/max/ { print $8; exit }')"
_min="-1"
_avg="-1"
_max="-1"
_mdev="-1"
_rtt="$(printf '%s\n' "$_out" | sed -n 's/.*= \([0-9.]*\)\/\([0-9.]*\)\/\([0-9.]*\)\/\([0-9.]*\) ms.*/\1 \2 \3 \4/p' | head -1)"
if [ -n "$_rtt" ]; then
_min="${_rtt%% *}"
_rest="${_rtt#* }"
_avg="${_rest%% *}"
_rest="${_rest#* }"
_max="${_rest%% *}"
_mdev="${_rest#* }"
else
_avg="$(printf '%s\n' "$_out" | awk -F'[=/ ]' '/min\/avg\/max/ { print $8; exit }')"
[ -n "$_avg" ] || _avg="-1"
fi
[ -n "$_loss" ] || _loss="-1"
[ -n "$_avg" ] || _avg="-1"
printf '%s %s' "$_loss" "$_avg"
printf '%s %s %s %s %s' "$_loss" "$_min" "$_avg" "$_max" "$_mdev"
}
append_ping_fields() {
_line="$1"
_prefix="$2"
_host="$3"
_sum="$(ping_summary "$_host")"
_loss="${_sum%% *}"
_rest="${_sum#* }"
_min="${_rest%% *}"
_rest="${_rest#* }"
_avg="${_rest%% *}"
_rest="${_rest#* }"
_max="${_rest%% *}"
_mdev="${_rest#* }"
printf '%s %s=%s %s_loss=%s%% %s_min_ms=%s %s_avg_ms=%s %s_max_ms=%s %s_mdev_ms=%s' \
"$_line" "$_prefix" "$_host" "$_prefix" "$_loss" "$_prefix" "$_min" \
"$_prefix" "$_avg" "$_prefix" "$_max" "$_prefix" "$_mdev"
}
route_dev() {
_host="$1"
ip route get "$_host" 2>/dev/null | awk '{for (i = 1; i <= NF; i++) if ($i == "dev") { print $(i + 1); exit }}'
}
route_gw() {
_host="$1"
ip route get "$_host" 2>/dev/null | awk '{for (i = 1; i <= NF; i++) if ($i == "via") { print $(i + 1); exit }}'
}
default_gw() {
ip route show default 2>/dev/null | awk '{print $3; exit}'
}
load_1m() {
awk '{print $1}' /proc/loadavg 2>/dev/null || echo "n/a"
}
cpu_busy_pct() {
awk '/^cpu / {
idle = $5 + $6; total = 0
for (i = 2; i <= NF; i++) total += $i
if (total > 0) printf "%.0f", 100 * (total - idle) / total
}' /proc/stat 2>/dev/null || echo "n/a"
}
conntrack_pct() {
_cur="$(cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null)" || _cur=""
_max="$(cat /proc/sys/net/netfilter/nf_conntrack_max 2>/dev/null)" || _max=""
if [ -n "$_cur" ] && [ -n "$_max" ] && [ "$_max" -gt 0 ] 2>/dev/null; then
_pct=$((_cur * 100 / _max))
printf 'conntrack=%s/%s(%s%%)' "$_cur" "$_max" "$_pct"
fi
}
neigh_state() {
_host="$1"
ip neigh show "$_host" 2>/dev/null | awk '{print $6; exit}'
}
iface_drops_compact() {
ip -s link 2>/dev/null | awk '
/^[0-9]+:/ {
if (iface != "" && iface !~ /^lo$/) {
if (rx_drop + tx_drop + rx_err + tx_err > 0)
printf "%s:rd=%d,td=%d,re=%d,te=%d ", iface, rx_drop, tx_drop, rx_err, tx_err
}
iface = $2; gsub(":", "", iface)
rx_drop = tx_drop = rx_err = tx_err = 0
}
/RX:/ { getline; rx_err = $3; rx_drop = $4 }
/TX:/ { getline; tx_err = $3; tx_drop = $4 }
END {
if (iface != "" && iface !~ /^lo$/ && rx_drop + tx_drop + rx_err + tx_err > 0)
printf "%s:rd=%d,td=%d,re=%d,te=%d", iface, rx_drop, tx_drop, rx_err, tx_err
}
' | sed 's/ $//'
}
acquire_lock() {
command -v flock >/dev/null 2>&1 || return 0
_lock="$WAN_SNAPSHOT_LOCK"
_dir="$(dirname "$_lock")"
if ! mkdir -p "$_dir" 2>/dev/null || ! : >"$_lock" 2>/dev/null; then
_lock="${TMPDIR:-/tmp}/wan-snapshot.$(id -u 2>/dev/null || echo 0).lock"
fi
exec 9>"$_lock" 2>/dev/null || return 0
flock -n 9 || {
echo "wan-snapshot: skip (already running, lock=$_lock)" >&2
exit 0
}
}
snapshot_once() {
need_ping
_ts="$(date -Iseconds 2>/dev/null || date '+%Y-%m-%dT%H:%M:%S%z')"
_wan="$(ping_summary "$WAN_PING_TARGET")"
_wan_loss="${_wan%% *}"
_wan_avg="${_wan#* }"
_fe="$(ping_summary "$FE_PING_TARGET")"
_fe_loss="${_fe%% *}"
_fe_avg="${_fe#* }"
acquire_lock
_line="ts=$_ts wan=$WAN_PING_TARGET wan_loss=${_wan_loss}% wan_avg_ms=${_wan_avg} fe=$FE_PING_TARGET fe_loss=${_fe_loss}% fe_avg_ms=${_fe_avg}"
_ts="$(date -Iseconds 2>/dev/null || date '+%Y-%m-%dT%H:%M:%S%z')"
_role="$(detect_role)"
_host="$(hostname -s 2>/dev/null || hostname 2>/dev/null || echo unknown)"
_line="ts=$_ts ver=2 role=$_role host=$_host"
_line="$(append_ping_fields "$_line" "wan" "$WAN_PING_TARGET")"
_line="$(append_ping_fields "$_line" "fe" "$FE_PING_TARGET")"
if [ -n "$BE_PING_TARGET" ] && [ "$BE_PING_TARGET" != "none" ]; then
_be="$(ping_summary "$BE_PING_TARGET")"
_be_loss="${_be%% *}"
_be_avg="${_be#* }"
_line="$_line be=$BE_PING_TARGET be_loss=${_be_loss}% be_avg_ms=${_be_avg}"
_line="$(append_ping_fields "$_line" "be" "$BE_PING_TARGET")"
fi
if [ -n "$ROUTER_PING_TARGET" ] && [ "$ROUTER_PING_TARGET" != "none" ]; then
_line="$(append_ping_fields "$_line" "router" "$ROUTER_PING_TARGET")"
fi
_gw_target="$GW_PING_TARGET"
if [ "$_gw_target" = "auto" ]; then
_gw_target="$(default_gw)"
[ -n "$_gw_target" ] || _gw_target="none"
fi
if [ -n "$_gw_target" ] && [ "$_gw_target" != "none" ]; then
_line="$(append_ping_fields "$_line" "gw" "$_gw_target")"
fi
if [ "$WAN_SNAPSHOT_EXTRAS" = "1" ]; then
_line="$_line load1=$(load_1m) cpu_busy=$(cpu_busy_pct)%"
_ct="$(conntrack_pct)"
[ -n "$_ct" ] && _line="$_line $_ct"
_wdev="$(route_dev "$WAN_PING_TARGET")"
_fdev="$(route_dev "$FE_PING_TARGET")"
[ -n "$_wdev" ] && _line="$_line route_wan=$_wdev"
[ -n "$_fdev" ] && _line="$_line route_fe=$_fdev"
_wgw="$(route_gw "$WAN_PING_TARGET")"
[ -n "$_wgw" ] && _line="$_line via_wan=$_wgw"
_nfe="$(neigh_state "$FE_PING_TARGET")"
[ -n "$_nfe" ] && _line="$_line neigh_fe=$_nfe"
_drops="$(iface_drops_compact)"
[ -n "$_drops" ] && _line="$_line drops=\"$_drops\""
fi
_dir="$(dirname "$WAN_SNAPSHOT_LOG")"
@@ -90,8 +247,25 @@ snapshot_once() {
printf '%s\n' "$_line"
}
# Average wan_avg_ms / fe_avg_ms for lines matching DATE prefix (YYYY-MM-DD).
day_avg() {
# Extract numeric field from log line (supports wan_avg_ms and legacy layouts).
field_value() {
_line="$1"
_field="$2"
printf '%s\n' "$_line" | awk -v f="$_field" '
{
for (i = 1; i <= NF; i++) {
if ($i ~ "^" f "=") {
split($i, a, "=")
v = a[2]
gsub(/%/, "", v)
if (v + 0 == v) { print v; exit }
}
}
}
'
}
day_stats() {
_log="$1"
_date="$2"
_field="$3"
@@ -103,13 +277,21 @@ day_avg() {
split($i, a, "=")
v = a[2]
gsub(/%/, "", v)
if (v + 0 == v && v >= 0) { n++; s += v }
if (v + 0 == v && v >= 0) {
n++; s += v; if (v > max) max = v
if (v >= 50) bad++
vals[n] = v
}
}
}
}
END {
if (n > 0) printf "%.2f", s / n
else print "n/a"
if (n == 0) { print "n/a n/a n/a n/a"; exit }
asort(vals)
p50 = vals[int((n + 1) * 0.5)]
p90 = vals[int(n * 0.9)]
if (p90 == "") p90 = vals[n]
printf "%.2f %.2f %.2f %.0f %d/%d", s / n, p50, p90, max, bad + 0, n
}
' "$_log"
}
@@ -137,6 +319,30 @@ day_max_loss() {
' "$_log"
}
report_day_block() {
_log="$1"
_label="$2"
_date="$3"
_samples="$(grep -c "^ts=$_date" "$_log" 2>/dev/null || echo 0)"
printf '%s=%s samples=%s\n' "$_label" "$_date" "$_samples"
for _pfx in wan fe be router gw; do
_stats="$(day_stats "$_log" "$_date" "${_pfx}_avg_ms")"
_mean="${_stats%% *}"
_rest="${_stats#* }"
_p50="${_rest%% *}"
_rest="${_rest#* }"
_p90="${_rest%% *}"
_rest="${_rest#* }"
_max="${_rest%% *}"
_bad="${_rest#* }"
_loss="$(day_max_loss "$_log" "$_date" "${_pfx}_loss")"
if [ "$_mean" != "n/a" ]; then
printf ' %s: mean=%s p50=%s p90=%s max=%s bad(>=50ms)=%s loss_max=%s%%\n' \
"$_pfx" "$_mean" "$_p50" "$_p90" "$_max" "$_bad" "$_loss"
fi
done
}
report() {
_log="$WAN_SNAPSHOT_LOG"
_today="$(date '+%Y-%m-%d')"
@@ -147,31 +353,22 @@ report() {
exit 1
fi
_samples_today="$(grep -c "^ts=$_today" "$_log" 2>/dev/null || echo 0)"
printf 'log=%s\n' "$_log"
printf 'today=%s samples=%s\n' "$_today" "$_samples_today"
printf ' wan_avg_ms=%s wan_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_today" "wan_avg_ms")" \
"$(day_max_loss "$_log" "$_today" "wan_loss")"
printf ' fe_avg_ms=%s fe_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_today" "fe_avg_ms")" \
"$(day_max_loss "$_log" "$_today" "fe_loss")"
_role="$(detect_role)"
printf 'log=%s role=%s\n' "$_log" "$_role"
report_day_block "$_log" "today" "$_today"
if [ -n "$_yesterday" ] && grep -q "^ts=$_yesterday" "$_log" 2>/dev/null; then
_samples_y="$(grep -c "^ts=$_yesterday" "$_log" 2>/dev/null || echo 0)"
printf 'yesterday=%s samples=%s\n' "$_yesterday" "$_samples_y"
printf ' wan_avg_ms=%s wan_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_yesterday" "wan_avg_ms")" \
"$(day_max_loss "$_log" "$_yesterday" "wan_loss")"
printf ' fe_avg_ms=%s fe_max_loss%%=%s\n' \
"$(day_avg "$_log" "$_yesterday" "fe_avg_ms")" \
"$(day_max_loss "$_log" "$_yesterday" "fe_loss")"
report_day_block "$_log" "yesterday" "$_yesterday"
fi
printf '\nInterpretation:\n'
printf ' wan bad + fe good -> likely ISP/upstream (world -> 134.17.26.161)\n'
printf ' wan + fe bad -> router or LAN (10.7.6.228 -> 10.7.0.10)\n'
printf ' wan + fe good, apps slow -> DNAT/nginx above ping path\n'
printf ' wan bad + fe good -> ISP/upstream or route_dev issue\n'
printf ' wan + fe bad -> router/LAN/tunnel path\n'
printf ' fe bad (pi) + fe ok (fe) -> path to FE, not FE host CPU\n'
printf ' high mdev / max vs avg -> jitter (bufferbloat, conntrack, VPN)\n'
printf ' conntrack>80%% -> edge table pressure (router)\n'
printf ' neigh_fe not REACHABLE -> ARP/L2 flap toward FE\n'
printf ' ping good, HTTPS slow -> DNAT/nginx above this probe\n'
}
tail_log() {
@@ -183,6 +380,116 @@ tail_log() {
tail -n "$_n" "$WAN_SNAPSHOT_LOG"
}
# Guess role label for correlate when log line has no role=.
log_role_label() {
_path="$1"
_line="$2"
_role="$(field_value "$_line" "role")"
if [ -n "$_role" ] && [ "$_role" != "0" ]; then
printf '%s' "$_role"
return 0
fi
case "$_path" in
*raspberry*|*rpi*) printf 'pi' ;;
*FE_gentoo*|*gentoo*) printf 'fe' ;;
*monstro*|*router*) printf 'router' ;;
*) printf '%s' "$(basename "$_path" .log)" ;;
esac
}
# Bucket timestamp to 15-min wall-clock slot: YYYY-MM-DDTHH:00|15|30|45
slot_key() {
_ts="$1"
_date="${_ts%%T*}"
_rest="${_ts#*T}"
_hour="${_rest%%:*}"
_min="${_rest#*:}"
_min="${_min%%:*}"
_min="${_min%%+*}"
_min="${_min%%-*}"
_bucket=$(( (_min / 15) * 15 ))
printf '%sT%s:%02d\n' "$_date" "$_hour" "$_bucket"
}
correlate_logs() {
if [ "$#" -lt 1 ]; then
echo "wan-snapshot: --correlate needs at least one log file" >&2
exit 1
fi
_tmp="$(mktemp "${TMPDIR:-/tmp}/wan-correlate.XXXXXX")"
# shellcheck disable=SC2064
trap 'rm -f "$_tmp"' EXIT INT HUP
for _path in "$@"; do
[ -r "$_path" ] || {
echo "wan-snapshot: unreadable: $_path" >&2
continue
}
_label="$(basename "$_path")"
while IFS= read -r _line || [ -n "$_line" ]; do
case "$_line" in
ts=*) ;;
*) continue ;;
esac
_ts="${_line#ts=}"
_ts="${_ts%% *}"
_slot="$(slot_key "$_ts")"
_role="$(log_role_label "$_path" "$_line")"
printf '%s\t%s\t%s\t%s\t%s\n' "$_slot" "$_role" "$_ts" "$_label" "$_line" >> "$_tmp"
done < "$_path"
done
printf 'slot\t'
_roles="$(awk -F'\t' '{print $2}' "$_tmp" | sort -u | tr '\n' ' ')"
for _role in $_roles; do
printf '%s_wan\t%s_fe\t%s_be\t' "$_role" "$_role" "$_role"
done
printf '\n'
awk -F'\t' '
{
slot = $1; role = $2; ts = $3; line = $5
key = slot SUBSEP role
if (!(key in best_ts) || ts > best_ts[key]) {
best_ts[key] = ts
wan = fe = be = "-"
for (i = 1; i <= split(line, a, " "); i++) {
if (a[i] ~ /^wan_avg_ms=/) { split(a[i], b, "="); wan = b[2] }
if (a[i] ~ /^fe_avg_ms=/) { split(a[i], b, "="); fe = b[2] }
if (a[i] ~ /^be_avg_ms=/) { split(a[i], b, "="); be = b[2] }
}
W[slot, role, "wan"] = wan
W[slot, role, "fe"] = fe
W[slot, role, "be"] = be
slots[slot] = 1
roles[role] = 1
}
}
END {
nroles = 0
for (r in roles) rolelist[++nroles] = r
asort(rolelist)
nslots = 0
for (s in slots) slotlist[++nslots] = s
asort(slotlist)
for (si = 1; si <= nslots; si++) {
s = slotlist[si]
printf "%s", s
for (ri = 1; ri <= nroles; ri++) {
r = rolelist[ri]
printf "\t%s\t%s\t%s", W[s, r, "wan"] + 0 ? W[s, r, "wan"] : "-", \
W[s, r, "fe"] + 0 ? W[s, r, "fe"] : "-", \
W[s, r, "be"] + 0 ? W[s, r, "be"] : "-"
}
printf "\n"
}
}
' "$_tmp" | while IFS= read -r _row; do
printf '%s\n' "$_row"
done
}
case "${1:-}" in
-h|--help|help)
usage
@@ -193,6 +500,10 @@ case "${1:-}" in
--tail|tail)
tail_log "${2:-24}"
;;
--correlate|correlate)
shift
correlate_logs "$@"
;;
"")
snapshot_once
;;