# Platform scaling and multi-environment architecture — design review (DR)

<!-- doc-meta:start -->
| Field | Value |
|---|---|
| Author | Anton Afanasyeu |
| Revision | R1 |
| Creation date | 2026-06-12 |
| Last modification date | 2026-06-12 |
| Co-authored |  |
| Severity | medium |
| State | pending review |
| Document type | DR |
<!-- doc-meta:end -->

\newpage
\newpage

---


**Document type:** DR (Design Review)  
**Source draft:** [docs/drafts/20100612_1_scaling.txt](../drafts/20100612_1_scaling.txt)  
**Specification:** [docs/specs/20100612_1_scaling.md](../specs/20100612_1_scaling.md)  
**PDF:** [20100612_1_scaling.pdf](20100612_1_scaling.pdf) · Regenerate: `bash scripts/build-all-docs-pdf.sh` (Mermaid → PNG diagrams; requires `npx`)  
**Date:** 2026-06-12  
**Status:** Reviewed — planning / brainstorming (no implementation sprint yet)  
**Severity:** Strategic

**Documentation index:** [README.md](../README.md)

---

---

---

---


**Documentation index:** [README.md](../README.md)

---


---

---


---

## Table of contents

<!-- toc -->
- [1. Executive summary](#1-executive-summary)
- [2. Problem statement](#2-problem-statement)
- [3. Architecture options](#3-architecture-options)
  - [3.1 Option A — Dev cluster (draft default)](#31-option-a-dev-cluster-draft-default)
  - [3.2 Option B — Tighter cost (minimal split)](#32-option-b-tighter-cost-minimal-split)
  - [3.3 Option C — Cloud-first hybrid](#33-option-c-cloud-first-hybrid)
  - [3.4 Comparison matrix](#34-comparison-matrix)
- [4. DR decision — recommended path](#4-dr-decision-recommended-path)
- [5. Current vs target diagrams](#5-current-vs-target-diagrams)
  - [Today (single BE)](#today-single-be)
  - [Target phase B (minimal split + HA web)](#target-phase-b-minimal-split-ha-web)
- [6. Service split rationale](#6-service-split-rationale)
- [7. Environment promotion model](#7-environment-promotion-model)
- [8. RSSH and connection capacity](#8-rssh-and-connection-capacity)
  - [Assumptions](#assumptions)
  - [Estimates](#estimates)
- [8b. Network bandwidth (DR)](#8b-network-bandwidth-dr)
  - [Baseline risk (today)](#baseline-risk-today)
  - [DR sizing decisions](#dr-sizing-decisions)
  - [Planning numbers (single-site prod, until CDN)](#planning-numbers-single-site-prod-until-cdn)
- [8c. Global availability and GEO segments (DR)](#8c-global-availability-and-geo-segments-dr)
  - [Policy model (DR decision)](#policy-model-dr-decision)
  - [Phasing](#phasing)
  - [Example use cases](#example-use-cases)
- [9. Future capabilities — priority weighting](#9-future-capabilities-priority-weighting)
- [10. Simulation lab design](#10-simulation-lab-design)
- [11. Cloud provider notes (GCP / Azure)](#11-cloud-provider-notes-gcp-azure)
- [12. RBAC evolution](#12-rbac-evolution)
- [13. Risks and mitigations](#13-risks-and-mitigations)
- [14. Estimates](#14-estimates)
- [15. Task dependency graph](#15-task-dependency-graph)
- [16. Resolved open questions](#16-resolved-open-questions)
- [17. Remaining open items (PO)](#17-remaining-open-items-po)
- [18. Suggestions (PO / dev discretion)](#18-suggestions-po-dev-discretion)
- [19. Changelog](#19-changelog)
- [Related docs](#related-docs)
<!-- /toc -->

**Documentation index:** [README.md](README.md)

---

## 1. Executive summary

The project outgrew **one Alpine BE VM** that hosts consoles, APIs, MariaDB, Gitea, OTA, WireGuard, RSSH, URL shortener, and Docker builds. The draft asks for **clusters**, **dev/staging/prod**, **zero-downtime deploys**, optional **cloud**, and headroom for **RT cast**, **VPN commercialization**, **payments**, and **ffmpeg** pipelines.

**DR outcome:**

| Area | Decision |
|------|----------|
| **Timing** | **Post-alpha** strategic track; does not block alpha soak |
| **Architecture** | **Option B → A** phased: minimal VM split first, then full dev cluster simulation |
| **Edge** | Keep **Gentoo FE** as TLS/DNAT anchor; add LB upstream to N BE web nodes |
| **Data** | **Dedicated MariaDB VM** before scaling anything else CPU-heavy |
| **Build** | **First split candidate** — Docker off shared PHP VM |
| **Cloud** | **Hybrid** — object store + staging dress rehearsal before prod migration |
| **RSSH capacity** | Plan **100 concurrent SSH** / **5k heartbeats** per bastion pair before commercial VPN |
| **Bandwidth** | Document **WAN / FE→BE / per-service** Mbps & GB-month budgets ([SPEC §12.3](../specs/20100612_1_scaling.md#123-network-bandwidth-and-egress)); CDN before horizontal web scale for OTA |
| **Global + GEO** | **Worldwide reach** default; **deny segments** per content (e.g. no serve in `ZM`) via layered enforcement ([SPEC §14](../specs/20100612_1_scaling.md#14-global-availability-and-geo-policy)) |
| **Future slots** | Weighted backlog in [§9](#9-future-capabilities-priority-weighting); SFU before ffmpeg farm; **F10 GEO** with F3/F5 |

---

## 2. Problem statement

| Pain | Evidence today |
|------|----------------|
| **Blast radius** | Submodule sync wiped gitignored configs; one BE restart affects all services |
| **Resource contention** | Docker build spikes CPU/RAM during PHP console use |
| **No promotion path** | Single prod-like env; staging OTA path still pending ([20260608_BE_SERVICES_and_infra.md](../20260608_BE_SERVICES_and_infra.md)) |
| **Scaling ceiling** | RSSH + WG + future SFU on same VM as MariaDB |
| **Bandwidth ceiling** | Single FE WAN + FE→BE path; OTA/APK egress competes with consoles ([SPEC §12.3](../specs/20100612_1_scaling.md#123-network-bandwidth-and-egress)) |
| **GEO gap** | No per-content region policy yet; required before global recordings/commerce ([SPEC §14](../specs/20100612_1_scaling.md#14-global-availability-and-geo-policy)) |
| **Operational risk** | Topology changes on FE/router are hard to roll back ([bottomline_reminder.txt](../../bottomline_reminder.txt)) |

**Success criteria:** PO can deploy unstable work to **dev**, beta-test on **staging**, and promote to **prod** with rollback per service — without rewriting mobile apps or public URLs each time.

---

## 3. Architecture options

### 3.1 Option A — Dev cluster (draft default)

Draft proposal: **N VMs per environment**, each role class (web+OTA, DB, object store, WG, url shortener+Gitea, build, +N for new features).

```mermaid
flowchart TB
    subgraph Cluster["One environment cluster (e.g. dev)"]
        W["Web + OTA VM(s)"]
        D["Database VM(s)"]
        O["Object store VM(s)"]
        V["WireGuard VM(s)"]
        U["URL shortener + Gitea VM"]
        B["Build VM(s)"]
        X["Future service VM(s)"]
    end
    W --> D
    W --> O
    U --> D
    B --> O
    V --> W
```

| Pros | Cons |
|------|------|
| Clear isolation; matches future SFU/ffmpeg | Highest VM count / cost |
| Easy to simulate on FE XEN | Ops overhead for small team |
| Aligns with cloud MIG later | Overkill before alpha exit |

### 3.2 Option B — Tighter cost (minimal split)

**Phase 1 prod shape** (recommended start):

| VM | Bundled services |
|----|------------------|
| **be-web** | nginx, PHP (hub, crashes, graphs, builder **UI**, OTA **metadata**) |
| **be-data** | MariaDB only |
| **be-ops** | Gitea, url-shortener |
| **be-build** | Docker builder |
| **be-vpn** | WireGuard + RSSH bastion |

```mermaid
flowchart LR
    FE["Gentoo FE"] --> WEB["be-web"]
    WEB --> DATA["be-data"]
    WEB --> VPN["be-vpn"]
    OPS["be-ops"] --> DATA
    BUILD["be-build"] --> DATA
```

| Pros | Cons |
|------|------|
| ~5 VMs vs 7+; fits “money-wise” draft | Gitea + shortener still coupled |
| Unblocks build/DB contention immediately | Second web node needed later for true HA |
| Low risk to current URLs | WG still single point until pool |

### 3.3 Option C — Cloud-first hybrid

Move **staging** entirely to GCP/Azure; keep **prod** on FE→BE until hybrid LB proven.

| Pros | Cons |
|------|------|
| Managed LB, DB, object store day one | Egress cost; UDP/WG complexity |
| Dress rehearsal for full cloud | PO learning curve; vendor lock-in mild |
| ffmpeg/SFU elastic scale | **Violates** “safest topology” unless staged |

### 3.4 Comparison matrix

Scores 1 (poor) – 5 (excellent) for **current team size** and **alpha proximity**:

| Criterion | A — Full cluster | B — Minimal split | C — Cloud-first |
|-----------|------------------|-------------------|-----------------|
| Cost now | 2 | **4** | 3 |
| Time to first value | 2 | **5** | 3 |
| Zero-downtime readiness | 4 | 3 | **5** |
| Topology safety | 4 | **5** | 2 |
| SFU/ffmpeg fit | **5** | 3 | **5** |
| Simulation fidelity | **5** | 4 | 2 |
| **Global CDN / GEO** | 3 | 3 | **5** |
| **Weighted total** | 25 | **27** | 25 |

**Weights used:** safety and cost now > cloud elasticity pre-alpha.

---

## 4. DR decision — recommended path

```text
Now ──► Option B (minimal split) on-prem HVMs
          │
          ├── Simulate Option A as DEV cluster (3–7 Alpine HVMs)
          │
          ├── Add be-web-2 + FE upstream when metrics say so
          │
          └── Option C for staging only → then hybrid prod
```

| Step | Action | Owner |
|------|--------|-------|
| 1 | Document service boundaries (this SPEC/DR) | Agent / PO review |
| 2 | Move Docker builder to **be-build-1** | Sysop |
| 3 | Move MariaDB to **be-data-1** (hardest cutover) | Sysop |
| 4 | Stand up **dev cluster** simulation on intra network | Sysop |
| 5 | Staging promotion runbook + DNS prefix | PO + DEV |
| 6 | Cloud object store for OTA/APK | Sysop |
| 7 | **CDN + bandwidth dashboards** (SPEC §12.3) | Sysop |
| 8 | SFU SPEC + media workers (separate track) | PO spec |
| 9 | **GEO policy schema + edge rules** (SPEC §14) | PO + DEV |

**Not chosen now:** full Option A in prod; full Option C prod cutover.

---

## 5. Current vs target diagrams

### Today (single BE)

```mermaid
flowchart TB
    R["Router 134.17.26.161"] --> FE["FE Gentoo 10.7.0.10"]
    FE --> BE["Single Alpine BE 10.7.16.128"]
    BE --> S1["PHP consoles + APIs"]
    BE --> S2["MariaDB"]
    BE --> S3["Gitea + url-shortener"]
    BE --> S4["Docker build"]
    BE --> S5["WG + RSSH"]
    FE -->|"UDP 45340 DNAT"| S5
```

### Target phase B (minimal split + HA web)

```mermaid
flowchart TB
    R["Router"] --> FE["Gentoo FE"]
    FE --> LB["nginx upstream / cloud LB"]
    LB --> W1["be-web-1"]
    LB --> W2["be-web-2"]
    W1 --> DB["be-data MariaDB"]
    W2 --> DB
    OPS["be-ops Gitea+shortener"] --> DB
    BUILD["be-build Docker"] --> DB
    VPN1["be-vpn-1"] --> DB
    VPN2["be-vpn-2 optional"]
    FE -->|"UDP DNAT pool"| VPN1
    FE --> VPN2
```

---

## 6. Service split rationale

Draft asked to balance **cost vs CPU/RAM needs**. DR sizing rules:

| Workload | CPU | RAM | Disk | Pin to |
|----------|-----|-----|------|--------|
| PHP consoles | Medium | 2–4 GiB | Small | be-web* |
| MariaDB | Medium | **8+ GiB** | **Fast SSD** | be-data |
| Gitea mirrors | Low | 2 GiB | Medium | be-ops |
| URL shortener | Low | 512 MiB | Tiny | be-ops |
| Docker build | **Burst high** | **16 GiB** | Large ephemeral | be-build |
| WireGuard | Low | 1 GiB | Tiny | be-vpn |
| RSSH sshd | Low–med | 2 GiB | Tiny | be-vpn |
| ffmpeg (future) | **Sustained high** | 8–32 GiB | Temp SSD | be-media* |
| SFU (future) | **High** | 8+ GiB | Low | be-media* |
| Recordings (future) | Low | — | **Large** | object store |

**Anti-pattern rejected:** MariaDB + Docker build + ffmpeg on one VM (draft implied bundle risk).

---

## 7. Environment promotion model

| Gate | dev → staging | staging → prod |
|------|---------------|----------------|
| Automated tests | `./gradlew :app:testDebugUnitTest` + BE smoke | + soak checklist |
| Manual | PO feature acceptance | PO sign-off + tag |
| Data | Synthetic | Anonymized copy optional |
| Rollback | Re-deploy previous image | Git revert + DB restore runbook |

Align app releases with [GIT_FLOW.md](../GIT_FLOW.md); infra releases may cadence independently with **config version pins** per environment.

---

## 8. RSSH and connection capacity

Draft question: *how many simultaneous connections per instance?*

### Assumptions

- Heartbeat poll **1–7 min** ([REMOTE_ACCESS_IMPL](../REMOTE_ACCESS_IMPL.md))
- Average SSH session **15 min** operator work
- Alpine **be-vpn-1**: 2 vCPU, 2 GiB RAM

### Estimates

| Resource | Limiting factor | Conservative | With tuning |
|----------|-----------------|--------------|-------------|
| **Concurrent SSH** | `sshd` `MaxSessions` / CPU | 50 | **100** |
| **Devices polling heartbeat** | PHP-FPM + MariaDB writes | 2 000 | **5 000** |
| **New SSH/min** | `MaxStartups` | 10 | 30 |
| **WG peers (lab)** | kernel + UDP | 200 | 500 / 2 nodes |

**Tuning knobs:** separate **heartbeat** PHP pool to **be-web** (off bastion); keep sshd on **be-vpn** only; connection pooling to MariaDB; read replicas for heartbeat read-heavy phases.

**DR recommendation:** Instrument **active SSH** and **heartbeat QPS** in graphs before selling commercial VPN (draft F4).

---

## 8b. Network bandwidth (DR)

Draft rev.2 adds **throughput** alongside connection counts.

### Baseline risk (today)

| Path | Risk |
|------|------|
| Single **WAN** uplink | OTA release or viral short link saturates all services |
| **FE→BE** proxy | All HTTPS doubles through FE NIC |
| No **CDN** | Every APK download hits origin BE |
| **Media future** | SFU + ffmpeg on same uplink = hard fail without PoP/CDN |

### DR sizing decisions

| Decision | Rationale |
|----------|-----------|
| **CDN before be-web-N** for OTA/static | Fixes egress Mbps cheaper than more PHP VMs |
| **Separate media NIC/subnet** when SFU lands | Isolate UDP/WebRTC from MariaDB LAN |
| **Egress budget alerts** at 80% ISP/cloud quota | PO notified before hard cap |
| **Simulate release day** on staging | e.g. 1000 concurrent OTA downloads |

### Planning numbers (single-site prod, until CDN)

| Traffic | Peak Mbps (conservative) | Mitigation |
|---------|--------------------------|------------|
| Steady HTTPS (API + consoles) | 5–20 | Normal |
| OTA drop (1000 devices × 40 MB APK) | **200–400** burst 10 min | **CDN required** |
| Recording download (F3) | 50–200 per hot file | Object store + CDN signed URLs |
| SFU fan-out 100 viewers | 50–500+ | Regional PoP; not on FE WAN |

Full table: [SPEC §12.3](../specs/20100612_1_scaling.md#123-network-bandwidth-and-egress).

---

## 8c. Global availability and GEO segments (DR)

Draft rev.2: **global reach** + **segment isolation** (example: deny publish/serve in **Zambia `ZM`**).

### Policy model (DR decision)

| Rule | Choice |
|------|--------|
| **Default** | **Allow** worldwide for control plane and public APIs |
| **Content objects** | Attach optional `geo_policy` with `deny_publish` / `deny_serve` / `allow_only` |
| **Enforcement order** | CDN/WAF → BE API (trusted geo header) → object signed URL → app entitlement |
| **Operator UI** | Hub/console editor for geo segments when F3/F10 ship — not alpha |
| **Sanctions** | PO-maintained deny lists; not auto-synced from external API in v1 |

```mermaid
flowchart LR
    REQ["Client request"] --> EDGE["CDN / FE edge"]
    EDGE --> GEO{"Country in deny_serve?"}
    GEO -->|yes| DENY["451 / blocked manifest"]
    GEO -->|no| BE["BE + geo_policy DB"]
    BE --> OK["200 / stream / redirect"]
```

### Phasing

| Phase | GEO capability |
|-------|----------------|
| **Now** | Implicit worldwide; no per-content rules |
| **CDN phase** | Edge geo block for **coarse** country bans |
| **F3 storage** | Per-recording `geo_policy` in MariaDB + object metadata |
| **F5 payments** | Entitlement includes `geo_allowed` |
| **Multi-PoP** | GeoDNS routes to nearest **allowed** PoP only |

### Example use cases

| Use case | Policy |
|----------|--------|
| OTA `stable` globally | `default: allow` |
| Beta APK EU-only | `allow_only: ["DE","FR",…]` |
| Recording licensed EU/US only | `deny_serve: ["ZM",…]` + `allow_only` optional |
| Short link to blocked catalog | BE returns 451 at redirect time |

**Open for PO:** whether **VPN exit** in denied country circumvents policy (DR: log + ToS; technical block on **VPN product** separate from HTTP geo).

---

## 9. Future capabilities — priority weighting

Weighted for **platform dependency** and **revenue enablement** (PO may override):

| Rank | Slot | Weight | Rationale |
|------|------|--------|-----------|
| **1** | F1 Multipoint cast (SFU/MCU) | Critical path | Unblocks WAN cast; already on [OPEN_TASKS_GRAPH.md](../OPEN_TASKS_GRAPH.md) |
| **2** | F2 ffmpeg transcode | High | Shared worker pool for F3, F9 |
| **3** | F3 Cast storage | High | Needs F1+F2 + object store |
| **4** | F5 Payments | Medium | Enables F3, F4, F7 |
| **5** | F4 Commercial VPN | Medium | RSSH/WG scale must be proven first |
| **6** | F9 3rd-party streaming | Medium | Uses F2 cast handlers |
| **7** | F6 Mobile add-ons | Lower | Can ship on-device first |
| **8** | F7 In-app transfers | Lower | Compliance heavy |
| **9** | F8 DNS / subdomains | Exploratory | Largest scope creep |
| **10** | F10 GEO enforcement | Medium | Required for F3/F5 global commerce; draft rev.2 |

```mermaid
flowchart TB
    SFU["F1 SFU / multipoint"] --> STORE["F3 Cast storage"]
    FF["F2 ffmpeg workers"] --> STORE
    FF --> EXT["F9 3rd-party streams"]
    SFU --> EXT
    PAY["F5 Payments"] --> STORE
    PAY --> VPN["F4 Commercial VPN"]
    PAY --> XFER["F7 In-app transfers"]
    PAY --> ADD["F6 Add-ons"]
    DNS["F8 DNS / site builder"] --> PAY
    GEO["F10 GEO policy"] --> STORE
    GEO --> PAY
```

---

## 10. Simulation lab design

Sysop simulates **Option A** under dev without WAN changes:

| HVM | Alpine packages | Role |
|-----|-----------------|------|
| `dev-web-1` | nginx, php81-fpm | Web tier |
| `dev-data-1` | mariadb | DB |
| `dev-ops-1` | git, gitea (optional) | Gitea + shortener |
| `dev-build-1` | docker | Builder |
| `dev-vpn-1` | wireguard-tools, openssh | VPN lab |

**Network:** `10.7.0.0/8` only; FE routes `dev.apps.f0xx.org` to dev-web when ready.

**Deliverable suggestion:** `docs/SIMULATION_CLUSTER.md` + scripts in phase 1 — not blocking this DR.

---

## 11. Cloud provider notes (GCP / Azure)

| Concern | GCP | Azure |
|---------|-----|-------|
| **Managed LB** | ILB + NEG | App Gateway / Front Door |
| **Object store** | GCS | Blob |
| **Managed MariaDB** | Cloud SQL | Azure Database for MariaDB |
| **UDP / WG** | harder | harder — keep on-prem FE DNAT longer |
| **ffmpeg burst** | MIG (CPU) | VMSS |
| **Egress cost** | Watch OTA/APK | Same |
| **CDN + geo WAF** | Cloud CDN / Cloud Armor | Azure Front Door + rules |
| **Multi-region** | GCS dual-region; Cloud CDN PoPs | Blob + AFD PoPs |

**DR:** Use cloud for **staging cluster first** (Option C partial) — validates IaC without prod risk. **CDN geo rules** prototype on staging before prod F3.

---

## 12. RBAC evolution

| Stage | Who | Access |
|-------|-----|--------|
| **Alpha (now)** | Developers = operators | Max privileges; shared session |
| **Beta** | Split **developer** vs **support operator** | Tickets/RA without build admin |
| **Commercial** | Customer admin + platform admin | Tenant isolation; audit export |

New roles MUST NOT require mobile app changes until F5 entitlements API exists.

---

## 13. Risks and mitigations

| Risk | Impact | Mitigation |
|------|--------|------------|
| MariaDB migration downtime | Prod outage | Replica + cutover window; practice on dev |
| Config loss on sync | 503 (seen url-shortener) | `ensure_*` scripts; secrets outside tree |
| FE upstream misconfig | Partial outage | `nginx -t`; weighted rollout |
| WG UDP cloud move | VPN breaks | Keep DNAT path until PO signs UDP plan |
| Over-scoping F5–F9 | Never ship SFU | [§9](#9-future-capabilities-priority-weighting) order |
| WAN saturation on OTA | Global outage perception | CDN + [§8b](#8b-network-bandwidth-dr) budgets |
| Geo policy bypass via VPN | Compliance / licensing | Separate VPN egress policy; audit logs |
| Wrong deny list | Legitimate users blocked | Staging test + `X-Debug-Country`; PO approval workflow |
| Simulation sprawl | Sysop time | Cap dev HVMs at 7; destroy scripts |

---

## 14. Estimates

Person-days (**single sysop + DEV support**), planning accuracy ±40%:

| Phase | Work | Days |
|-------|------|------|
| **P0** | SPEC/DR + simulation design (this doc) | 2 |
| **P1** | Split Docker build VM | 2 |
| **P2** | MariaDB dedicated VM + migration | 3 |
| **P3** | be-ops split (Gitea + shortener) | 1 |
| **P4** | Dev cluster HVM simulation (5 VMs) | 4 |
| **P5** | Staging env + promotion runbook | 3 |
| **P6** | be-web-2 + FE upstream HA | 2 |
| **P7** | Cloud object store for OTA | 2 |
| **P7b** | CDN + bandwidth metrics on graphs | 2 |
| **P8** | GEO policy schema + staging CDN geo test | 3 |
| **P9** | SFU/media (separate SPEC) | TBD |
| **Total (through P7b)** | | **~21 days** |

**Severity:** Strategic — parallel to alpha; no alpha gate.

---

## 15. Task dependency graph

Legend: **1** = highest priority (PO: post-alpha infra). Assignee **Sysop** = PO today; **Agent** = documentation/automation; **DEV** = app/config changes.

```mermaid
flowchart TB
    T1["1. SPEC+DR scaling docs<br/>Agent ✓"]
    T2["2. Split be-build Docker<br/>Sysop"]
    T3["3. Dedicated be-data MariaDB<br/>Sysop"]
    T4["4. ensure_* config hardening all services<br/>Agent"]
    T5["5. Dev cluster simulation HVMs<br/>Sysop"]
    T6["6. Staging env + DNS prefix<br/>Sysop + PO"]
    T7["7. Promotion runbook dev→stg→prod<br/>PO"]
    T8["8. be-web-2 + FE upstream<br/>Sysop"]
    T9["9. Cloud object store OTA<br/>Sysop"]
    T9b["9b. CDN + bandwidth metrics<br/>Sysop"]
    T10["10. GEO policy + edge rules<br/>PO+DEV"]
    T11["11. SFU owner SPEC<br/>PO"]
    T12["12. ffmpeg worker pool<br/>DEV post-SFU"]
    T13["13. Payments / entitlements SPEC<br/>PO"]

    T1 --> T2
    T1 --> T4
    T2 --> T3
    T3 --> T5
    T4 --> T6
    T5 --> T6
    T6 --> T7
    T7 --> T8
    T3 --> T8
    T8 --> T9
    T9 --> T9b
    T9b --> T10
    T10 --> T11
    T11 --> T12
    T12 --> T13

    style T1 fill:#86efac,color:#14532d
    style T11 fill:#e5e7eb,color:#374151
    style T12 fill:#e5e7eb,color:#374151
    style T13 fill:#e5e7eb,color:#374151
```

| ID | Task | Assignee | Blocks |
|----|------|----------|--------|
| 1 | Scaling SPEC/DR | Agent | — |
| 2 | Docker → be-build | Sysop | DB split |
| 3 | MariaDB → be-data | Sysop | HA web |
| 4 | Post-sync config scripts (all services) | Agent | staging |
| 5 | Dev HVM simulation | Sysop | staging realism |
| 6 | Staging cluster | Sysop | promotion |
| 7 | Promotion runbook | PO | prod multi-node |
| 8 | Second web node | Sysop | cloud OTA |
| 9 | Object store OTA | Sysop | CDN |
| 9b | CDN + bandwidth dashboards | Sysop | GEO test |
| 10 | GEO policy + edge | PO+DEV | F3 commerce |
| 11 | SFU SPEC | PO | media workers |
| 12 | ffmpeg pool | DEV | commercial storage |
| 13 | Payments SPEC | PO | F6–F8 |

**Parallel now (non-blocking):** alpha soak, url-shortener ops, Gitea mirrors, ticket **SVC-OTA**.

---

## 16. Resolved open questions

| # | Draft question | DR answer |
|---|----------------|-----------|
| Q1 | How many VMs in a cluster? | **Option B:** 5 prod VMs to start; **dev simulation:** 3–7 ([SPEC §6.3](../specs/20100612_1_scaling.md#63-reference-cluster-layout-prod)) |
| Q2 | “Other” architecture approaches? | **Option C** hybrid for staging; not prod-first ([§3.3](#33-option-c-cloud-first-hybrid)) |
| Q3 | Cost vs CPU/RAM balance? | Split by [§6](#6-service-split-rationale) profile table |
| Q4 | Seamless scaling / LB? | FE upstream → N PHP nodes; cloud managed LB in phase 5 ([SPEC §9](../specs/20100612_1_scaling.md#9-zero-downtime-and-load-balancing)) |
| Q5 | Simultaneous connections / RSSH? | [§8](#8-rssh-and-connection-capacity) — 100 SSH, 5k heartbeats per bastion pair (tuned) |
| Q6 | Alpine simulation? | [§10](#10-simulation-lab-design) — intra `10.7.0.0/8` HVMs only |
| Q7 | Future MCU/SFU/ffmpeg/storage/VPN/payments? | Platform **slots** F1–F9; priority [§9](#9-future-capabilities-priority-weighting) |
| Q8 | RBAC model change? | [§12](#12-rbac-evolution) three-stage |
| Q9 | Keep current topology safe? | **Yes** — [SPEC §18](../specs/20100612_1_scaling.md#18-safety-constraints-current-topology) |
| Q10 | Network bandwidth metrics? | [§8b](#8b-network-bandwidth-dr) + [SPEC §12.3](../specs/20100612_1_scaling.md#123-network-bandwidth-and-egress) — CDN before web horizontal scale |
| Q11 | Global availability + GEO deny (e.g. Zambia)? | [§8c](#8c-global-availability-and-geo-segments-dr) + [SPEC §14](../specs/20100612_1_scaling.md#14-global-availability-and-geo-policy) — default allow, per-content deny |

---

## 17. Remaining open items (PO)

| ID | Question | Needed for |
|----|----------|------------|
| O1 | Target **monthly infra budget** for phase B (5 VMs)? | Option A vs B timing |
| O2 | **Staging URL** pattern: `staging.apps.f0xx.org` vs path prefix? | DNS + TLS |
| O3 | **Cloud vendor** preference GCP vs Azure vs none for 12 months? | P7–P9 |
| O4 | Accept **maintenance window** for MariaDB cutover (minutes of read-only)? | P2 schedule |
| O5 | **Commercial VPN** — same WG infra or separate product stack? | F4 design |
| O6 | **SFU** — Janus on `:8089` vs greenfield mediasoup? | F1 SPEC ([20260608_BE_SERVICES_and_infra.md](../20260608_BE_SERVICES_and_infra.md)) |
| O7 | **Payments** — Stripe-only vs multi-provider? | F5 SPEC |
| O8 | **ISP/cloud egress cap** (GB/month) on FE WAN? | [§8b](#8b-network-bandwidth-dr) budgets |
| O9 | **CDN provider** (Cloudflare vs cloud-native vs none 12 mo)? | OTA + GEO WAF |
| O10 | **GEO deny lists** — PO-owned static JSON vs admin UI first? | F10 rollout |
| O11 | Recordings **451** vs silent 404 for denied regions? | UX + compliance |

*When blocked on O1–O7, append to `tmp/YYYYMMDD_implementation_summary.pdf` per bottomline_reminder.*

---

## 18. Suggestions (PO / dev discretion)

| # | Suggestion |
|---|------------|
| S1 | Add **infra smoke** to `validate_be_services.sh` per VM role after split |
| S2 | Version **nginx upstream** fragments in `examples/crash_reporter/backend/deploy/` |
| S3 | Use **Docker** for PHP-FPM pools only after MariaDB split — avoid double complexity |
| S4 | Keep **Gitea** on be-ops until mirror count > 10 repos |
| S5 | Document **connection pool** sizes when RSSH device count > 500 |
| S6 | Re-read this DR when **SFU SPEC** lands — revise [§9](#9-future-capabilities-priority-weighting) |
| S7 | Add **WAN Mbps** graph series on FE before next OTA wide release |
| S8 | Staging test matrix: `X-Debug-Country: ZM` for geo deny paths |

---

## 19. Changelog

| Date | Change |
|------|--------|
| 2026-06-12 | Initial DR from draft `20100612_1_scaling.txt`; options matrix; task graph |
| 2026-06-12 | Rev. 2 — bandwidth (§8b), global/GEO (§8c), F10, task graph + open items |

---

## Related docs

- [SPEC — Platform scaling](../specs/20100612_1_scaling.md)
- [INFRA.md](../INFRA.md)
- [OPEN_TASKS_GRAPH.md](../OPEN_TASKS_GRAPH.md)
- [20260608_ALPHA_PRIORITIES.md](../20260608_ALPHA_PRIORITIES.md)