ac/ac-docs
1
0
Fork 0
mirror of git://f0xx.org/ac/ac-docs synced 2026-07-29 05:18:46 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
311a759a26b429000b8f233c902a59928724d9ec
ac-docs/DRs/20260617_codec2_ulbw_voice.md
Anton Afanasyeu 69a448f156 initial
2026-06-23 12:20:43 +02:00

14 KiB
Raw Blame History

Codec2 / ultra-low-bandwidth voice — design review (DR)

Field Value
Author Anton Afanasyeu
Revision R0
Creation date 2026-06-17
Last modification date 2026-06-17
Co-authored Cursor Agent (project assistant)
Severity low
State approved for planning (implementation post-alpha)
Document type DR
Pre-requisite to ROADMAP.md §6.1 Opus/Speex JNI encode/decode; F1 SFU SPEC (post-alpha)

\newpage \newpage

Document type: DR (Design Review)
Source draft: docs/drafts/20260617_codec2_ulbw_voice_integration.txt
PDF: 20260617_codec2_ulbw_voice.pdf · Regenerate: bash scripts/build-all-docs-pdf.sh
Status: PO-reviewed (inline draft comments incorporated) — no native implementation until Opus/Speex JNI sinks land
Scope: Voice-only / ULBW audio cast on P2P (LAN, BT, mesh); future MCU/SFU relay hooks

Related: OPUS_SPEEX_VALIDATION.md · ndk/README.md · 20260608_BE_SERVICES_and_infra.md § SFU · specs/20100612_1_scaling.md F1

Documentation index: README.md

Table of contents

1. Executive summary

Question: Should androidcast integrate Codec2 for ultra-low-bandwidth (ULBW) voice-only casts when normal AAC/Opus paths are too heavy for the link?

Short answer: Yes, as a post-alpha product slice — not a lab experiment. End users on marginal links (BT P2P, slow mesh, very poor LTE) should get intelligible audio (“something rather than nothing”). Codec2 @ 3200 bit/s (and lower modes where needed) fits that goal better than Opus narrowband @ ~6 kbps + FEC in PO’s practical experience (lower sustained bandwidth, less audible noise on those links).

DR decision (R0):

Item Decision
Primary use case Voice-only ULBW cast; not the audio leg of full screen cast (8 kHz telephony; no lip-sync with HD video)
Codec2 modes 3200 bit/s default target; 700C for extreme BT/mesh; user/dev sub-mode selection
MELP / TWELP Out of scope (patent / slim OSS path)
AAC Keep as platform MediaCodec path; no license conflict for screen cast
Opus / Speex Finish JNI encode/decode first (ROADMAP.md — cross-compile done, sinks TODO); negotiation/scaffolding exists today but wire path is stub → AAC without libopus.a / encode JNI
Codec2 timing After 6.1 JNI baseline; behind dev flag until E2E validated
SFU / MCU Plan transcode at ingress (Codec2 → Opus) for mixed WebRTC rooms; separate F1 SPEC — signaling reuses extended audio caps
MSG_AUDIO_SELECTED Add in handshake when Codec2 implementation starts (SETTINGS_VERSION bump) — aligns P2P multicast and SFU data channel

2. Problem statement

Screen cast defaults to AAC (32–128 kbps class). Negotiation already advertises Opus and Speex, but without native encode/decode the sender often still ships AAC while labeling Opus in diagnostics.

For ULBW links:

  • Opus @ 6 kbps + FEC/NACK can exceed the sustained budget PO has seen on BT/mesh and adds noise versus Codec2-class vocoders.
  • Users still want audio cast (commentary, coordination, voice relay) when video is impossible — a deliberate voice-only mode.

Future multipoint (F1 SFU/MCU) cannot rely on pairwise discovery bitmasks alone; room-level negotiated_audio must be defined before Codec2 legs join a relay.

3. Vocabulary

Term Meaning
ULBW Ultra-low bandwidth; sustained < 3–4 kbps audio payload target on worst links
Voice-only cast No video track; discovery may send videoCodecs = 0
Radio mode Dev/user profile preferring Codec2 in AUTO when both peers advertise AUDIO_CODEC2
P2P negotiation CastProtoHeader.audioCodecs bitmask + AudioNegotiator (today)
Room negotiation Signaling API + server-published codec (SFU future)
SFU Selective forwarding unit — relay without per-subscriber transcode
MCU Mix decode → PCM → encode; needed for heterogeneous legs

4. Current state (mobile + BE)

4.1 Audio codecs in app

Layer Opus / Speex today Notes
CastCodecFlags AUDIO_OPUS=4, AUDIO_SPEEX=8 Masks advertised in discovery
AudioNegotiator Works AUTO scoring via CodecPriorityCatalog
NativeCodecBridge Probe only (nativeIsOpusAvailable) No encode/decode JNI in opus_bridge.c / speex_bridge.c
Wire path OPUS_STUB / SPEEX_STUB → AAC PassthroughCodecPolicy, AudioCodecFactory
Build scripts/build-native-codecs.sh can link .a ROADMAP.md: JNI sinks TODO

Clarification for PO: Prior agent work completed negotiation, settings UI, priority catalog, and native autolink scaffolding — not full Opus/Speex on-the-wire encoding. Task 6.1 remains open for JNI sinks + E2E per OPUS_SPEEX_VALIDATION.md.

4.2 Video vs audio negotiation asymmetry

sequenceDiagram
    participant S as Sender
    participant R as Receiver
    Note over S,R: Discovery: audioCodecs bitmask in header
    S->>R: MSG_HELLO, CAST_SETTINGS
    S->>R: MSG_CODEC_CAPS (video MIME list)
    R->>S: MSG_RECEIVER_CAST_SETTINGS
    R->>S: MSG_CODEC_CAPS
    R->>S: MSG_CODEC_SELECTED (video MIME only)
    Note over S: AudioNegotiator.resolveForSender(peerMask)
    Note over R: No MSG_AUDIO_SELECTED today

4.3 Backend (gray)

  • SfuRelayGate.PREVIEW_ENABLED = false; api/sfu_health.php returns disabled.
  • Janus on :8089 is other stack — androidcast signaling stays on apps.f0xx.org PHP (INFRA.md).

5. Codec landscape and PO decisions

Codec Bitrate Role in androidcast DR decision
Codec2 700–3200 bps ULBW voice-only Integrate (post-6.1)
Opus SILK NB ~6–12 kbps General narrowband when link allows Finish JNI; keep in AUTO above Codec2 only when bandwidth sufficient
Speex NB ~2–15 kbps Legacy / fallback Finish JNI
AAC-LC 32+ kbps Screen cast default Unchanged
MELP/TWELP ~2.4 kbps — Rejected

PO rationale (2026-06-17): Opus @ 6k + FEC is heavier and noisier than Codec2 on target links; Codec2 3200 bps meets product goals; intent is end-user value, not codec shootout lab work.

Maintainer risk: Codec2 upstream is in active re-development; pin submodule tag (e.g. 1.2.0) and mirror on Gitea like third-party/opus.

6. Goals and non-goals

Goals

  • Voice-only cast mode with Codec2 encode/decode on Android (NDK).
  • Extend peer negotiation (AUDIO_CODEC2 = 16, optional MSG_AUDIO_SELECTED).
  • Document SFU ingress transcode and room caps for F1 SPEC.
  • LGPL compliance + license files.

Non-goals (R0)

  • MELP/TWELP integration.
  • Codec2 as default for screen cast with video.
  • SFU passthrough-only design (closed Android mesh edge case only).
  • Alpha ship — hidden until E2E + PO QA.

7. Mobile peer negotiation (today)

Step Mechanism Codec2 impact
Advertise CastCodecFlags.localAudioMask() in discovery Add bit 16 when native probe passes
Resolve AudioNegotiator.negotiate(senderMask, receiverMask, pref, settings) Add CODEC2 to enum + CodecPriorityId
Score codecs.json + CodecPriorityCatalog ULBW profile: CODEC2 lowest bitrate_score
Wire Audio frames (today AAC ADTS) New codec tag or MSG_AUDIO_FRAME subtype

Gap: Receiver does not receive explicit audio selection — sender-resolved only. Must fix before multicast + SFU (§10).

8. Proposed mobile integration

8.1 Settings and enum

CastSettings.AudioCodec.CODEC2
CastSettings.codec2Mode: 3200 | 1300 | 700C  (default 3200 per PO)
CastSettings.VoiceOnlyMode: boolean

8.2 Native stack

  1. third-party/codec2 submodule + scripts/init-third-party-submodules.sh
  2. scripts/build-native-codecs.sh → libcodec2.a, ANDROIDCAST_HAVE_CODEC2
  3. JNI: isCodec2Available(), encode/decode 8 kHz mono frames
  4. Link via existing libandroidcast_codecs.so

8.3 Priority (ULBW profile)

Suggested AUTO order when NetworkAdoption.ESTIMATED or radio mode on:

CODEC2 → OPUS → SPEEX → AAC → PCM

8.4 Milestones

# Deliverable Depends on
M1 Opus/Speex JNI encode/decode + E2E 6.1
M2 Codec2 probe + round-trip unit test M1
M3 Voice-only UI slice (dev gate) M2
M4 MSG_AUDIO_SELECTED + SETTINGS_VERSION M2
M5 SFU signaling fields in F1 SPEC Owner SFU spec

9. Future backend: MCU / SFU

flowchart TB
    subgraph clients [Android clients]
        A[Sender A Codec2]
        B[Sender B Opus]
        C[Receiver WebRTC]
    end
    subgraph be [BE post-alpha]
        SIG[Signaling API PHP]
        SFU[Janus / media worker]
    end
    A --> SIG
    B --> SIG
    SIG --> SFU
    SFU -->|transcode| C
    SFU -->|fan-out Opus| B

9.1 Negotiation shift (P2P → room)

Layer P2P today SFU future
Caps Header bitmask join JSON: audio_caps.bitmask, codec2_modes[]
Pick AudioNegotiator local Server negotiated_audio
Auth PIN / LAN Session RBAC + room token

9.2 Codec2 on media plane (DR preference)

Mode When CPU on be-media-N
Transcode ingress Mixed room (Android + browser) Recommended — decode Codec2 → Opus egress
Passthrough All Codec2 clients Low CPU; no WebRTC viewers
MCU mix Conference voice High CPU; radio bridge legs only

Signaling sketch (F1 SPEC — not implemented):

  • POST /api/sfu/room — ingress_caps: [opus, codec2], egress_caps: [opus]
  • POST /api/sfu/room/{id}/join — returns transport params + negotiated_audio

Mobile: SfuSignalingClient fills CastSettings.resolvedAudioCodec from server; LAN fallback if signaling down.

10. Protocol extensions

10.1 CastCodecFlags

AUDIO_CODEC2 = 16   // next free bit after SPEEX=8

10.2 MSG_AUDIO_SELECTED (recommended at M4)

payload: uint32 audio_flag (little-endian) + uint8 codec2_mode_id

Symmetric with MSG_CODEC_SELECTED; reusable on WebRTC data channel.

10.3 CastSettings.SETTINGS_VERSION

Bump when adding sfu_room_id, relay_mode, server_negotiated_audio (SFU phase).

11. Third-party and licenses

Component License Action
codec2 (libcodec2) LGPL-2.1 Submodule + Gitea mirror; dynamic link in .so
Opus / Speex BSD Existing
ffmpeg (SFU transcode) LGPL/GPL BE media worker SPEC only

Update mobile LICENSES / notices when submodule added (bottomline_reminder.txt).

12. Testing and validation

Phase Tests
Unit AudioNegotiatorTest CODEC2 mask; CastProtoHeader round-trip
Native JNI c2enc/c2dec parity arm64-v8a
E2E Extend OPUS_SPEEX_VALIDATION.md — voice-only, UDP/TCP, diagnostics kbps
SFU Post-alpha: 2× Codec2 Android + 1× WebRTC viewer via transcode

13. Task dependency graph

flowchart TB
    A61["6.1 Opus/Speex JNI sinks<br/>Developer · P5"]
    C2["Codec2 NDK + voice-only<br/>Agent · post-alpha"]
    AUDIO_HS["MSG_AUDIO_SELECTED<br/>Agent"]
    SFU["F1 SFU SPEC + signaling<br/>PO · post-alpha"]
    TRANS["SFU Codec2 transcode worker<br/>BE media"]

    A61 --> C2
    C2 --> AUDIO_HS
    AUDIO_HS --> SFU
    SFU --> TRANS

    style A61 fill:#b45309,color:#fff
    style C2 fill:#7c3aed,color:#fff
    style SFU fill:#e5e7eb,color:#374151

Numbered priority (PO overrides AI):

  1. 6.1 — Opus/Speex JNI (alpha LAN quality path; unblocks honest ULBW comparison)
  2. Codec2 ULBW — this DR (parallel after M1 green)
  3. F1 SFU SPEC — room negotiation + transcode
  4. Alpha blockers 5.1 / 5.2 / 4.rssh remain above on OPEN_TASKS_GRAPH.md

14. Risks

Risk Mitigation
Codec2 upstream churn Pin tag; Gitea mirror
8 kHz-only confuses users expecting music Voice-only mode UX; no video leg
SFU CPU cost Ingress transcode budget in scaling SPEC
Opus/Speex “done” confusion ROADMAP + this DR state JNI sinks explicitly
BT latency 150–300 ms Document; jitter buffer 80–120 ms

15. Open questions (remaining)

# Question Owner
1 SFU commercial story: always transcode vs optional passthrough mesh? PO
2 Ship Codec2 in AUTO on ULBW profile or explicit “radio mode” only? PO + UX
3 700C exposed to end users or dev-only? PO

Resolved by PO (2026-06-17): voice-only ULBW product intent; Codec2 3200 OK; skip MELP/TWELP; Opus @ 6k insufficient for target links; AAC platform use OK.

16. Changelog

Rev Date Change
R0 2026-06-17 Initial DR from draft; PO inline comments; Opus/Speex JNI status clarified
Reference in New Issue View Git Blame Copy Permalink