ac/android_cast
1
0
Fork 0
mirror of git://f0xx.org/android_cast synced 2026-07-29 08:57:44 +03:00
Code Issues Packages Projects Releases Wiki Activity

all sync, docs mostly

Browse Source
This commit is contained in:
Anton Afanasyeu
2026-06-18 16:39:49 +02:00
parent 017b6ddf25
commit 55cdcb49bf
40 changed files with 2329 additions and 127 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/DRs/20100611_3_url_shortener.pdf
View File

@@ -279,7 +279,7 @@ endobj
endobj
46 0 obj
<<
/Author (Android Cast project) /CreationDate (D:20260616121122+02'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20260616121122+02'00') /Producer (ReportLab PDF Library - \(opensource\))
/Author (Android Cast project) /CreationDate (D:20260617220446+02'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20260617220446+02'00') /Producer (ReportLab PDF Library - \(opensource\))
/Subject (\(unspecified\)) /Title (URL shortener service \204 design review \(DR\)) /Trapped /False
>>
endobj
@@ -553,7 +553,7 @@ xref
trailer
<<
/ID
[<df66ecc393a2b137e06a511c1fbe529e><df66ecc393a2b137e06a511c1fbe529e>]
[<3a7c0ee4bbf3482849eeadc12de9b9c8><3a7c0ee4bbf3482849eeadc12de9b9c8>]
% ReportLab generated PDF document -- digest (opensource)
/Info 46 0 R

4
docs/DRs/20100612_1_scaling.pdf
View File

@@ -498,7 +498,7 @@ endobj
endobj
79 0 obj
<<
/Author (Android Cast project) /CreationDate (D:20260616121123+02'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20260616121123+02'00') /Producer (ReportLab PDF Library - \(opensource\))
/Author (Android Cast project) /CreationDate (D:20260617220446+02'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20260617220446+02'00') /Producer (ReportLab PDF Library - \(opensource\))
/Subject (\(unspecified\)) /Title (Platform scaling and multi-environment architecture \204 design review \(DR\)) /Trapped /False
>>
endobj
@@ -933,7 +933,7 @@ xref
trailer
<<
/ID
[<6faf855138e92333540e6bef33a787f8><6faf855138e92333540e6bef33a787f8>]
[<cc552ef08a2fe35d84129cecaf25d843><cc552ef08a2fe35d84129cecaf25d843>]
% ReportLab generated PDF document -- digest (opensource)
/Info 79 0 R

675
docs/DRs/20260616_rssh_routed_egress.pdf Normal file
View File

File diff suppressed because one or more lines are too long

375
docs/DRs/20260617_codec2_ulbw_voice.md Normal file
View File

@@ -0,0 +1,375 @@
# Codec2 / ultra-low-bandwidth voice — design review (DR)
<!-- doc-meta:start -->
| Field | Value |
|---|---|
| Author | Anton Afanasyeu |
| Revision | R0 |
| Creation date | 2026-06-17 |
| Last modification date | 2026-06-17 |
| Co-authored | Cursor Agent (project assistant) |
| Severity | low |
| State | approved for planning (implementation post-alpha) |
| Document type | DR |
| Pre-requisite to | [ROADMAP.md](../ROADMAP.md) §6.1 Opus/Speex JNI encode/decode; F1 SFU SPEC (post-alpha) |
<!-- doc-meta:end -->
\newpage
\newpage
---
**Document type:** DR (Design Review)
**Source draft:** [docs/drafts/20260617_codec2_ulbw_voice_integration.txt](../drafts/20260617_codec2_ulbw_voice_integration.txt)
**PDF:** [20260617_codec2_ulbw_voice.pdf](20260617_codec2_ulbw_voice.pdf) · Regenerate: `bash scripts/build-all-docs-pdf.sh`
**Status:** PO-reviewed (inline draft comments incorporated) — **no native implementation** until Opus/Speex JNI sinks land
**Scope:** Voice-only / ULBW audio cast on P2P (LAN, BT, mesh); future MCU/SFU relay hooks
**Related:** [OPUS_SPEEX_VALIDATION.md](../OPUS_SPEEX_VALIDATION.md) · [ndk/README.md](../../ndk/README.md) · [20260608_BE_SERVICES_and_infra.md](../20260608_BE_SERVICES_and_infra.md) § SFU · [specs/20100612_1_scaling.md](../specs/20100612_1_scaling.md) F1
**Documentation index:** [README.md](../README.md)
---
---
---
## Table of contents
<!-- toc -->
- [1. Executive summary](#1-executive-summary)
- [2. Problem statement](#2-problem-statement)
- [3. Vocabulary](#3-vocabulary)
- [4. Current state (mobile + BE)](#4-current-state-mobile--be)
- [5. Codec landscape and PO decisions](#5-codec-landscape-and-po-decisions)
- [6. Goals and non-goals](#6-goals-and-non-goals)
- [7. Mobile peer negotiation (today)](#7-mobile-peer-negotiation-today)
- [8. Proposed mobile integration](#8-proposed-mobile-integration)
- [9. Future backend: MCU / SFU](#9-future-backend-mcu--sfu)
- [10. Protocol extensions](#10-protocol-extensions)
- [11. Third-party and licenses](#11-third-party-and-licenses)
- [12. Testing and validation](#12-testing-and-validation)
- [13. Task dependency graph](#13-task-dependency-graph)
- [14. Risks](#14-risks)
- [15. Open questions (remaining)](#15-open-questions-remaining)
- [16. Changelog](#16-changelog)
<!-- /toc -->
---
## 1. Executive summary
**Question:** Should androidcast integrate **Codec2** for ultra-low-bandwidth (ULBW) **voice-only** casts when normal AAC/Opus paths are too heavy for the link?
**Short answer:** **Yes, as a post-alpha product slice** — not a lab experiment. End users on marginal links (BT P2P, slow mesh, very poor LTE) should get **intelligible audio** (“something rather than nothing”). **Codec2 @ 3200 bit/s** (and lower modes where needed) fits that goal better than Opus narrowband @ ~6 kbps + FEC in PO’s practical experience (lower sustained bandwidth, less audible noise on those links).
**DR decision (R0):**
| Item | Decision |
|------|----------|
| **Primary use case** | **Voice-only** ULBW cast; not the audio leg of full screen cast (8 kHz telephony; no lip-sync with HD video) |
| **Codec2 modes** | **3200 bit/s** default target; **700C** for extreme BT/mesh; user/dev sub-mode selection |
| **MELP / TWELP** | **Out of scope** (patent / slim OSS path) |
| **AAC** | Keep as platform MediaCodec path; no license conflict for screen cast |
| **Opus / Speex** | Finish **JNI encode/decode** first ([ROADMAP.md](../ROADMAP.md) — cross-compile done, sinks **TODO**); negotiation/scaffolding exists today but wire path is **stub → AAC** without `libopus.a` / encode JNI |
| **Codec2 timing** | After **6.1** JNI baseline; behind dev flag until E2E validated |
| **SFU / MCU** | Plan **transcode at ingress** (Codec2 → Opus) for mixed WebRTC rooms; separate F1 SPEC — signaling reuses extended audio caps |
| **MSG_AUDIO_SELECTED** | Add in handshake when Codec2 implementation starts (SETTINGS_VERSION bump) — aligns P2P multicast and SFU data channel |
---
## 2. Problem statement
Screen cast defaults to **AAC** (32–128 kbps class). Negotiation already advertises **Opus** and **Speex**, but without native encode/decode the sender often still ships **AAC** while labeling Opus in diagnostics.
For **ULBW** links:
- Opus @ 6 kbps + FEC/NACK can exceed the sustained budget PO has seen on BT/mesh and adds noise versus Codec2-class vocoders.
- Users still want **audio cast** (commentary, coordination, voice relay) when video is impossible — a deliberate **voice-only** mode.
Future **multipoint** (F1 SFU/MCU) cannot rely on pairwise discovery bitmasks alone; room-level **negotiated_audio** must be defined before Codec2 legs join a relay.
---
## 3. Vocabulary
| Term | Meaning |
|------|---------|
| **ULBW** | Ultra-low bandwidth; sustained **&lt; 3–4 kbps** audio payload target on worst links |
| **Voice-only cast** | No video track; discovery may send `videoCodecs = 0` |
| **Radio mode** | Dev/user profile preferring Codec2 in AUTO when both peers advertise `AUDIO_CODEC2` |
| **P2P negotiation** | `CastProtoHeader.audioCodecs` bitmask + `AudioNegotiator` (today) |
| **Room negotiation** | Signaling API + server-published codec (SFU future) |
| **SFU** | Selective forwarding unit — relay without per-subscriber transcode |
| **MCU** | Mix decode → PCM → encode; needed for heterogeneous legs |
---
## 4. Current state (mobile + BE)
### 4.1 Audio codecs in app
| Layer | Opus / Speex today | Notes |
|-------|-------------------|-------|
| `CastCodecFlags` | `AUDIO_OPUS=4`, `AUDIO_SPEEX=8` | Masks advertised in discovery |
| `AudioNegotiator` | Works | AUTO scoring via `CodecPriorityCatalog` |
| `NativeCodecBridge` | **Probe only** (`nativeIsOpusAvailable`) | No encode/decode JNI in `opus_bridge.c` / `speex_bridge.c` |
| Wire path | **OPUS_STUB / SPEEX_STUB → AAC** | `PassthroughCodecPolicy`, `AudioCodecFactory` |
| Build | `scripts/build-native-codecs.sh` can link `.a` | [ROADMAP.md](../ROADMAP.md): JNI sinks **TODO** |
**Clarification for PO:** Prior agent work completed **negotiation, settings UI, priority catalog, and native autolink scaffolding** — not full Opus/Speex on-the-wire encoding. Task **6.1** remains open for JNI sinks + E2E per [OPUS_SPEEX_VALIDATION.md](../OPUS_SPEEX_VALIDATION.md).
### 4.2 Video vs audio negotiation asymmetry
```mermaid
sequenceDiagram
participant S as Sender
participant R as Receiver
Note over S,R: Discovery: audioCodecs bitmask in header
S->>R: MSG_HELLO, CAST_SETTINGS
S->>R: MSG_CODEC_CAPS (video MIME list)
R->>S: MSG_RECEIVER_CAST_SETTINGS
R->>S: MSG_CODEC_CAPS
R->>S: MSG_CODEC_SELECTED (video MIME only)
Note over S: AudioNegotiator.resolveForSender(peerMask)
Note over R: No MSG_AUDIO_SELECTED today
```
### 4.3 Backend (gray)
- `SfuRelayGate.PREVIEW_ENABLED = false`; `api/sfu_health.php` returns disabled.
- Janus on **:8089** is **other stack** — androidcast signaling stays on `apps.f0xx.org` PHP ([INFRA.md](../INFRA.md)).
---
## 5. Codec landscape and PO decisions
| Codec | Bitrate | Role in androidcast | DR decision |
|-------|---------|---------------------|-------------|
| **Codec2** | 700–3200 bps | ULBW voice-only | **Integrate** (post-6.1) |
| **Opus SILK NB** | ~6–12 kbps | General narrowband when link allows | Finish JNI; keep in AUTO above Codec2 only when bandwidth sufficient |
| **Speex NB** | ~2–15 kbps | Legacy / fallback | Finish JNI |
| **AAC-LC** | 32+ kbps | Screen cast default | Unchanged |
| **MELP/TWELP** | ~2.4 kbps | — | **Rejected** |
**PO rationale (2026-06-17):** Opus @ 6k + FEC is **heavier and noisier** than Codec2 on target links; Codec2 **3200 bps** meets product goals; intent is **end-user value**, not codec shootout lab work.
**Maintainer risk:** Codec2 upstream is in active re-development; pin submodule tag (e.g. 1.2.0) and mirror on Gitea like `third-party/opus`.
---
## 6. Goals and non-goals
### Goals
- Voice-only cast mode with **Codec2 encode/decode** on Android (NDK).
- Extend **peer negotiation** (`AUDIO_CODEC2 = 16`, optional `MSG_AUDIO_SELECTED`).
- Document **SFU ingress transcode** and room caps for F1 SPEC.
- LGPL compliance + license files.
### Non-goals (R0)
- MELP/TWELP integration.
- Codec2 as default for **screen cast with video**.
- SFU passthrough-only design (closed Android mesh edge case only).
- Alpha ship — hidden until E2E + PO QA.
---
## 7. Mobile peer negotiation (today)
| Step | Mechanism | Codec2 impact |
|------|-----------|---------------|
| Advertise | `CastCodecFlags.localAudioMask()` in discovery | Add bit 16 when native probe passes |
| Resolve | `AudioNegotiator.negotiate(senderMask, receiverMask, pref, settings)` | Add `CODEC2` to enum + `CodecPriorityId` |
| Score | `codecs.json` + `CodecPriorityCatalog` | ULBW profile: CODEC2 lowest `bitrate_score` |
| Wire | Audio frames (today AAC ADTS) | New codec tag or `MSG_AUDIO_FRAME` subtype |
**Gap:** Receiver does not receive explicit audio selection — sender-resolved only. **Must fix** before multicast + SFU (§10).
---
## 8. Proposed mobile integration
### 8.1 Settings and enum
```text
CastSettings.AudioCodec.CODEC2
CastSettings.codec2Mode: 3200 | 1300 | 700C (default 3200 per PO)
CastSettings.VoiceOnlyMode: boolean
```
### 8.2 Native stack
1. `third-party/codec2` submodule + `scripts/init-third-party-submodules.sh`
2. `scripts/build-native-codecs.sh` → `libcodec2.a`, `ANDROIDCAST_HAVE_CODEC2`
3. JNI: `isCodec2Available()`, encode/decode **8 kHz mono** frames
4. Link via existing `libandroidcast_codecs.so`
### 8.3 Priority (ULBW profile)
Suggested AUTO order when `NetworkAdoption.ESTIMATED` or **radio mode** on:
**CODEC2 → OPUS → SPEEX → AAC → PCM**
### 8.4 Milestones
| # | Deliverable | Depends on |
|---|-------------|------------|
| M1 | Opus/Speex JNI encode/decode + E2E | 6.1 |
| M2 | Codec2 probe + round-trip unit test | M1 |
| M3 | Voice-only UI slice (dev gate) | M2 |
| M4 | `MSG_AUDIO_SELECTED` + SETTINGS_VERSION | M2 |
| M5 | SFU signaling fields in F1 SPEC | Owner SFU spec |
---
## 9. Future backend: MCU / SFU
```mermaid
flowchart TB
subgraph clients [Android clients]
A[Sender A Codec2]
B[Sender B Opus]
C[Receiver WebRTC]
end
subgraph be [BE post-alpha]
SIG[Signaling API PHP]
SFU[Janus / media worker]
end
A --> SIG
B --> SIG
SIG --> SFU
SFU -->|transcode| C
SFU -->|fan-out Opus| B
```
### 9.1 Negotiation shift (P2P → room)
| Layer | P2P today | SFU future |
|-------|-----------|------------|
| Caps | Header bitmask | `join` JSON: `audio_caps.bitmask`, `codec2_modes[]` |
| Pick | `AudioNegotiator` local | Server `negotiated_audio` |
| Auth | PIN / LAN | Session RBAC + room token |
### 9.2 Codec2 on media plane (DR preference)
| Mode | When | CPU on be-media-N |
|------|------|-------------------|
| **Transcode ingress** | Mixed room (Android + browser) | **Recommended** — decode Codec2 → Opus egress |
| Passthrough | All Codec2 clients | Low CPU; no WebRTC viewers |
| MCU mix | Conference voice | High CPU; radio bridge legs only |
**Signaling sketch** (F1 SPEC — not implemented):
- `POST /api/sfu/room` — `ingress_caps: [opus, codec2]`, `egress_caps: [opus]`
- `POST /api/sfu/room/{id}/join` — returns transport params + `negotiated_audio`
Mobile: `SfuSignalingClient` fills `CastSettings.resolvedAudioCodec` from server; **LAN fallback** if signaling down.
---
## 10. Protocol extensions
### 10.1 `CastCodecFlags`
```text
AUDIO_CODEC2 = 16 // next free bit after SPEEX=8
```
### 10.2 `MSG_AUDIO_SELECTED` (recommended at M4)
```text
payload: uint32 audio_flag (little-endian) + uint8 codec2_mode_id
```
Symmetric with `MSG_CODEC_SELECTED`; reusable on WebRTC data channel.
### 10.3 `CastSettings.SETTINGS_VERSION`
Bump when adding `sfu_room_id`, `relay_mode`, `server_negotiated_audio` (SFU phase).
---
## 11. Third-party and licenses
| Component | License | Action |
|-----------|---------|--------|
| codec2 (libcodec2) | LGPL-2.1 | Submodule + Gitea mirror; dynamic link in `.so` |
| Opus / Speex | BSD | Existing |
| ffmpeg (SFU transcode) | LGPL/GPL | BE media worker SPEC only |
Update mobile `LICENSES` / notices when submodule added ([bottomline_reminder.txt](../../bottomline_reminder.txt)).
---
## 12. Testing and validation
| Phase | Tests |
|-------|-------|
| Unit | `AudioNegotiatorTest` CODEC2 mask; `CastProtoHeader` round-trip |
| Native | JNI c2enc/c2dec parity arm64-v8a |
| E2E | Extend [OPUS_SPEEX_VALIDATION.md](../OPUS_SPEEX_VALIDATION.md) — voice-only, UDP/TCP, diagnostics kbps |
| SFU | Post-alpha: 2× Codec2 Android + 1× WebRTC viewer via transcode |
---
## 13. Task dependency graph
```mermaid
flowchart TB
A61["6.1 Opus/Speex JNI sinks<br/>Developer · P5"]
C2["Codec2 NDK + voice-only<br/>Agent · post-alpha"]
AUDIO_HS["MSG_AUDIO_SELECTED<br/>Agent"]
SFU["F1 SFU SPEC + signaling<br/>PO · post-alpha"]
TRANS["SFU Codec2 transcode worker<br/>BE media"]
A61 --> C2
C2 --> AUDIO_HS
AUDIO_HS --> SFU
SFU --> TRANS
style A61 fill:#b45309,color:#fff
style C2 fill:#7c3aed,color:#fff
style SFU fill:#e5e7eb,color:#374151
```
**Numbered priority (PO overrides AI):**
1. **6.1** — Opus/Speex JNI (alpha LAN quality path; unblocks honest ULBW comparison)
2. **Codec2 ULBW** — this DR (parallel after M1 green)
3. **F1 SFU SPEC** — room negotiation + transcode
4. Alpha blockers **5.1 / 5.2 / 4.rssh** remain above on [OPEN_TASKS_GRAPH.md](../OPEN_TASKS_GRAPH.md)
---
## 14. Risks
| Risk | Mitigation |
|------|------------|
| Codec2 upstream churn | Pin tag; Gitea mirror |
| 8 kHz-only confuses users expecting music | Voice-only mode UX; no video leg |
| SFU CPU cost | Ingress transcode budget in scaling SPEC |
| Opus/Speex “done” confusion | ROADMAP + this DR state JNI sinks explicitly |
| BT latency 150–300 ms | Document; jitter buffer 80–120 ms |
---
## 15. Open questions (remaining)
| # | Question | Owner |
|---|----------|-------|
| 1 | SFU commercial story: always transcode vs optional passthrough mesh? | PO |
| 2 | Ship Codec2 in AUTO on ULBW profile or explicit “radio mode” only? | PO + UX |
| 3 | `700C` exposed to end users or dev-only? | PO |
**Resolved by PO (2026-06-17):** voice-only ULBW product intent; Codec2 3200 OK; skip MELP/TWELP; Opus @ 6k insufficient for target links; AAC platform use OK.
---
## 16. Changelog
| Rev | Date | Change |
|-----|------|--------|
| R0 | 2026-06-17 | Initial DR from draft; PO inline comments; Opus/Speex JNI status clarified |

669
docs/DRs/20260617_codec2_ulbw_voice.pdf Normal file
View File

File diff suppressed because one or more lines are too long