Codec2 / ultra-low-bandwidth (ULBW) voice — research draft
R0-pre
severity: low
state: research (post-alpha candidate; blocked on Opus/Speex JNI baseline)
author: PO + agent
date: 2026-06-17

Problem:
- PO asked whether Codec2 (or MELP/TWELP-class codecs) should be integrated for voice-only or
  audio-primary paths on very slow links: mesh/BT P2P, marginal LTE, future multipoint relay.
- Today screen cast defaults to AAC; Opus/Speex are advertised in discovery masks but native
  encode/decode JNI is still TODO (stub → AAC on wire). See ndk/README.md, OPUS_SPEEX_VALIDATION.md.
- Future F1 multipoint cast (MCU/SFU on BE) will need a single negotiated audio format per room
  or per leg — P2P bitmask negotiation alone will not scale.
[foxx comment]: it's surprise to me to hear that Opus/Speex is still TODO - reported to be finished by AI
[AI note]: Negotiation/UI/autolink scaffolding is done; ROADMAP.md lists "Opus/Speex JNI encode/decode sinks" as TODO.
  opus_bridge.c only exposes nativeIsOpusAvailable — wire path is OPUS_STUB→AAC without libopus.a + encode JNI.
  See DR docs/DRs/20260617_codec2_ulbw_voice.md §4.1.

Scope of this draft:
- Research + integration sketch only. No Codec2 until Opus/Speex JNI encode/decode lands (ROADMAP 6.1).
- Covers mobile peer negotiation (today + extensions) and backend MCU/SFU hooks (gray / post-alpha).

──────────────────────────────────────────────────────────────────────────────
1. Codec landscape (skeptical first)
──────────────────────────────────────────────────────────────────────────────

| Codec        | Bitrate      | Sample rate | Quality / role              | License   | Android fit        |
|--------------|-------------|-------------|-----------------------------|-----------|--------------------|
| Opus SILK NB | ~6–12 kbps  | 8 kHz       | Good comms; FEC; wide use   | BSD       | Submodule exists   |
| Speex NB     | ~2–15 kbps  | 8 kHz       | Legacy narrowband           | BSD       | Submodule exists   |
| Codec2       | 700–3200 bps| 8 kHz       | HF radio; intelligible only | LGPL-2.1  | New submodule      |
| MELP / TWELP | ~2400 bps   | 8 kHz       | Govt vocoder; patent encumbrance | varies | Poor OSS story |
| AAC-LC       | 32–128 kbps | 44.1/48 kHz | Screen cast default         | patent    | MediaCodec HW      |

Skeptical PO questions (answer before building Codec2):
- Is Codec2 needed if Opus @ 6 kbps + existing FEC/NACK already fits the slow-link budget?
- Is the use case **voice-only cast** (no video) or **audio leg of screen cast**? Codec2 is 8 kHz
  telephony — downmixing stereo app audio loses music/UI cues; fine for push-to-talk, poor for
  “share my meeting audio”.
- Codec2 maintainer note (2023+): major re-dev; many modes “not actively maintained”; only 3200 for
  M17 called out as exception. Fork/maintenance risk vs stable Opus.
- LGPL-2.1: dynamic link via libandroidcast_codecs.so is OK; document in mobile + BE licenses;
  Gitea mirror like opus/speex.
[foxx comment]: from my past practical experience, Opus@6k+FEC gives a lot more hearable noise and network bandwidth rather than Codec2, so everything just fits its hole and goals; we use AAC legally as a part of Android platform, no conflict there; Codec2 3200bps looks also OK for the goals we want to achieve; MELP/TWELP - skipping that, out of scope because of many slim points using it

When Codec2 still wins:
- Sub-3 kbps **sustained** (not burst): BT Classic SPP mesh, APRS-style relay, satellite backhaul.
- Symmetric **voice-only** sessions where both peers opt into “radio mode” (dev/PO setting).
- Lab demos comparing Speex vs Codec2 700C on identical packet loss (after Opus baseline exists).
[foxx comment]: the intent of using Codec2 is not the lab comparison but the desire from end-users to have audio casts on ultra-low bandwidth (better to have something than have nothing, let's say)

──────────────────────────────────────────────────────────────────────────────
2. Current mobile negotiation (P2P LAN / TCP / UDP)
──────────────────────────────────────────────────────────────────────────────

Two parallel paths today — video and audio are **not** negotiated the same way.

A) Discovery + wire header (both peers, pre-session):
   - CastProtoHeader.audioCodecs / videoCodecs bitmasks (48-byte header, every packet).
   - CastCodecFlags: AUDIO_PCM=1, AAC=2, OPUS=4, SPEEX=8 (room for 16, 32, …).
   - DiscoveryManager advertises localAudioMask() from CastCodecFlags.localAudioMask().
   - Receiver list in UI carries peer audioCodecs → sender picks target.

B) TCP handshake (video only, CastSession):
   - MSG_CODEC_CAPS → MIME lists (H.264, VP8, …).
   - MSG_CODEC_SELECTED → single negotiated video MIME.
   - Receiver runs CodecNegotiator; sender logs CodecPriorityCatalog decision.
   - **No MSG_AUDIO_SELECTED** — audio is not on this wire sequence.

C) Audio resolution (sender side, after target known):
   - AudioNegotiator.resolveForSender(settings, peerAudioMask):
       negotiate(localMask, peerMask, user preference, settings).
   - AUTO → CodecPriorityCatalog.decideBestAudioCodec (scores from codecs.json).
   - Explicit pref → mask intersection; fallback to AUTO → AAC.
   - Result stored in CastSettings.resolvedAudioCodec (transient).
   - Wired from ScreenCastService / MultiCastCoordinator.

D) Stream protection (UDP):
   - Separate post-handshake negotiation (FEC/NACK); independent of codec choice.

Implications for Codec2:
- Adding Codec2 requires a new flag bit (e.g. AUDIO_CODEC2 = 16) in CastCodecFlags + proto docs.
- Must update: localAudioMask(), flagForAudioPreference(), isLocallyAvailable(), CodecPriorityId,
  CastSettings.AudioCodec enum, codecs.json, manualAudioPreferences() order.
- **Do not** advertise CODEC2 in mask until NativeCodecBridge probe passes (same rule as Opus).
- Unit tests: AudioNegotiatorTest, CastProtoHeaderTest round-trip, mask intersection cases.

Gap (intentional for alpha, painful for SFU):
- Audio choice is **sender-resolved** from discovery masks; receiver is not sent MSG_AUDIO_SELECTED.
- Receiver must infer from CastSettings in session or mirror sender logic on first audio frame
  (today: implicit via shared settings blob / same negotiation inputs). Document before SFU.

──────────────────────────────────────────────────────────────────────────────
3. Proposed Codec2 mobile integration (minimal diff)
──────────────────────────────────────────────────────────────────────────────

Enum / settings:
- CastSettings.AudioCodec.CODEC2 (or CODEC2_1300 / CODEC2_700C sub-modes in dev settings).
- Optional CastSettings.VoiceOnlyMode flag (no video track; skips video handshake).

Native:
- third-party/codec2 submodule → scripts/init-third-party-submodules.sh + Gitea mirror.
- Extend scripts/build-native-codecs.sh + ndk CMake: ANDROIDCAST_HAVE_CODEC2.
- JNI in NativeCodecBridge: isCodec2Available(), encode/decode 20 ms frames @ 8 kHz mono.
- Frame format on wire: length-prefixed Codec2 blobs (mode id in session meta or first frame);
  **not** AAC ADTS — new CastProtocol MSG_AUDIO_FRAME subtype or codec tag in existing payload.

Priority catalog (codecs.json):
- Default order suggestion if ULBW profile enabled: CODEC2 < OPUS < SPEEX < AAC < PCM.
- bitrate_score: CODEC2 lowest; internal_score: higher delay penalty (see §5).
- Profile hook: CastSettings.NetworkAdoption.ESTIMATED or new ULBW_PRESET gates AUTO toward CODEC2
  only when both masks include it AND user/dev “radio mode” on.

Voice-only cast (optional product slice):
- Capture: AudioRecord 8 kHz mono (or resample from 48 kHz graph).
- Skip video encoder; discovery still sends videoCodecs=0 or VIDEO_UNKNOWN.
- BT / mesh: same negotiation masks over RFCOMM or UDP wrapper — framing spec TBD (§6).

Milestone order:
1. Opus + Speex JNI encode/decode (ROADMAP 6.1) — validates pipeline end-to-end.
2. Codec2 probe + encode/decode behind dev flag.
3. Voice-only mode + E2E on two devices.
4. SFU signaling extensions (§4) — only when F1 spec approved.

──────────────────────────────────────────────────────────────────────────────
4. Future backend: MCU / SFU integration
──────────────────────────────────────────────────────────────────────────────

References:
- docs/20260608_BE_SERVICES_and_infra.md § Planned SFU relay (Janus-class)
- docs/specs/20100612_1_scaling.md F1 Multipoint cast, be-media-N tier
- docs/INFRA.md: port 8089 = Janus/other — **not** androidcast PHP vhost; do not mix nginx paths
- App: SfuRelayGate.PREVIEW_ENABLED = false; CastConfig.ALPHA_FEATURE_FREEZE

Architecture (target, gray):

    [Sender A]──┐                      ┌──[Receiver B]
                ├──► Signaling (PHP) ──►├── Janus/mediasoup SFU (:8089 or dedicated)
    [Sender C]──┘         │            └── transcode / fan-out
                          RBAC/session
                          room lifecycle

Negotiation shifts from pairwise bitmask to **room capability contract**:

| Layer              | P2P today                    | SFU future                                      |
|--------------------|------------------------------|-------------------------------------------------|
| Capability advert  | Discovery header masks       | Join-room: client caps JSON + bitmask           |
| Audio pick         | AudioNegotiator local        | Server publishes room_audio_codec or per-leg    |
| Video pick         | MSG_CODEC_SELECTED           | WebRTC SDP / simulcast layers                   |
| Auth               | PIN / LAN                    | Existing session/RBAC + room token              |
| Transport          | UDP/TCP cast                 | WebRTC (+ optional TURN); LAN cast unchanged    |

Codec2 on SFU — three realistic modes:

1) **Passthrough (hardest):** SFU relays Codec2 packets without decode. All clients must speak
   Codec2; no WebRTC browser viewers. Fits closed Android mesh only.

2) **Transcode at SFU (likely):** Android sends Codec2 or Opus; media worker (ffmpeg/Janus plugin)
   decodes → Opus/VP8 for WebRTC subscribers. PO must budget CPU on be-media-N (scaling SPEC).
   Negotiation: room advertises sfu_audio_ingress: [opus, codec2], egress: opus.

3) **MCU mix (voice conference):** Decode all legs → PCM mix → encode Opus (or Codec2 for radio
   leg). Codec2 only on “radio bridge” participants, not main room.

Signaling API sketch (owner spec TBD — not implemented):
- POST /api/sfu/room — create; body includes max_participants, ingress_caps, egress_caps.
- POST /api/sfu/room/{id}/join — returns Janus session/handle or mediasoup transport params.
- Client payload extension (parallel to CastSettings):
    audio_caps: { bitmask, codec2_modes: ["700C","1300"], max_bitrate_bps }
    video_caps: { bitmask, mime_list }
    preference: { audio: "AUTO", network_profile: "ULBW" }
- Server response:
    negotiated_audio: "opus" | "codec2_1300" | ...
    negotiated_video: mime or null for voice-only
    relay_mode: "p2p" | "sfu" | "mcu"

Mobile changes when SFU lands:
- New transport enum value TRANSPORT_WEBRTC (frozen in alpha UI).
- Before cast: if SfuRelayGate.isAvailable() && room URL → skip LAN discovery masks;
  run SfuSignalingClient.negotiate(roomCaps) → populate CastSettings.resolvedAudioCodec from server.
- Fallback: LAN P2P unchanged if signaling unreachable (local-first).
- AudioNegotiator.negotiate() reused for **pre-signaling** pairwise preview; server may override.

Janus / 8089 caution:
- FE nginx location /janus/ exists for other stack — androidcast signaling stays on apps.f0xx.org
  PHP API; media UDP ports on dedicated media subnet per scaling DR. No coupling in alpha.

MCU vs SFU choice (for PO):
- SFU: lower server CPU if passthrough/simulcast; N subscribers see one encoded stream.
- MCU: needed for true multiparty voice mix or heterogeneous codecs (Codec2 leg + Opus leg).
- F1 backlog item — separate SFU SPEC before any Codec2-on-BE work.

──────────────────────────────────────────────────────────────────────────────
5. Peer negotiation — extensions (mobile + protocol)
──────────────────────────────────────────────────────────────────────────────

Short term (Codec2 on P2P, no proto version bump if possible):
- Reserve CastCodecFlags.AUDIO_CODEC2 = 16.
- Add optional CastSettings.codec2Mode (1300, 700C, 3200) — must match on both sides; if mismatch,
  AudioNegotiator falls back to next scored codec (same as explicit OPUS unavailable).
- Consider MSG_AUDIO_SELECTED in TCP handshake (symmetric with video):
    payload: uint32 audio_flag + uint8 mode
  Benefits: receiver logs definitive choice; multi-receiver multicast gets same codec;
  SFU migration reuses message type over WebRTC data channel.
  Cost: proto doc + CastSession serverHandshake/clientHandshake change — schedule with SETTINGS_VERSION bump.

Medium term (multipoint / SFU):
- Extend CastSettings SETTINGS_VERSION when adding sfu_room_id, relay_mode, server_negotiated_audio.
- Discovery header insufficient for 3+ peers — room state lives on signaling server.
- Per-peer legs: senderMask/receiverMask become N-vectors or server-issued single codec.
- AudioNegotiator refactor: negotiateRoom(RoomCaps, List<PeerCaps>, preference) → server or elected
  “master” receiver (first joiner) picks; align with CodecNegotiator pattern for video.

BT / mesh P2P (no BE):
- Same bitmask in link-local discovery (mDNS or BLE advertisement) — reuse CastProtoHeader layout.
- Frame timing: Codec2 1600-mode ~ 40 ms frames; BT SPP throughput ~1–3 kbps practical — profile
  700C only; add jitter buffer 80–120 ms (document in AV_QUALITY_QA_SESSION follow-up).

Delay budget (manage expectations):
- Codec2 encode+decode + BT latency → 150–300 ms one-way plausible; unacceptable for lip-sync
  with video — reinforce voice-only or audio-only cast for ULBW profile.

──────────────────────────────────────────────────────────────────────────────
6. Testing & validation (when implemented)
──────────────────────────────────────────────────────────────────────────────

Automated:
- AudioNegotiatorTest: CODEC2 in both masks → selected when scores favor; one side missing → fallback.
- CastCodecFlags round-trip AUDIO_CODEC2 bit.
- Native round-trip: c2enc/c2dec parity via JNI on arm64-v8a CI (optional, after submodule).

Manual E2E (extend OPUS_SPEEX_VALIDATION.md matrix):
- Voice-only, CODEC2 explicit, two devices, UDP then TCP.
- AUTO + ESTIMATED network preset on marginal Wi‑Fi (throttle via dev settings).
- Record: resolvedAudioCodec in diagnostics, audio kbps, subjective MOS notes.

SFU (post-alpha):
- Room with 2 Android + 1 ingress Codec2 → verify transcode to Opus for third WebRTC viewer.
- sfu_health.php enabled; SfuRelayGate preview on dev builds only.

──────────────────────────────────────────────────────────────────────────────
7. Open questions for PO
──────────────────────────────────────────────────────────────────────────────

1. Primary use case: voice-only P2P, or audio leg of screen cast on ULBW?
2. Accept 8 kHz telephony quality for “radio mode”, or require wideband Opus for anything with video?
3. Codec2 mode default: 1300 (balance) vs 700C (minimum bits) vs user-selectable?
4. Proceed with Codec2 at all after Opus @ 6 kbps measured on target BT/mesh hardware?
5. SFU: passthrough Codec2 vs transcode-only at ingress — which matches commercial story?
6. MSG_AUDIO_SELECTED in handshake: worth SETTINGS_VERSION bump now, or defer until SFU spec?
7. MELP/TWELP: drop — insufficient OSS path unless PO has licensed binaries.

──────────────────────────────────────────────────────────────────────────────
8. Recommendation (agent opinion)
──────────────────────────────────────────────────────────────────────────────

Defer Codec2 implementation until:
- Opus/Speex JNI is real on wire (not stub).
- PO picks voice-only vs AV-leg scope.
- One measured session on slow link proves Opus+FEC insufficient.

If proceeding:
- Treat Codec2 as dev/ULBW profile codec, not AUTO default.
- Design bitmask + optional MSG_AUDIO_SELECTED now so SFU room negotiation reuses the same types.
- Plan BE transcode path (ffmpeg decode Codec2 → Opus) in F1 SFU SPEC — do not assume SFU passthrough.

Next doc steps:
- DR: docs/DRs/20260617_codec2_ulbw_voice.md + PDF (done 2026-06-17)
- SPEC slice in scaling or dedicated docs/specs/…_sfu_audio_negotiation.md when F1 unblocks
- Ticket: “ULBW voice” → implementation epic after 6.1 green

Related files:
- app/.../CastCodecFlags.java, AudioNegotiator.java, CodecPriorityCatalog.java
- app/.../network/CastSession.java, CastProtoHeader.java, CastProtocol.java
- app/.../sfu/SfuRelayGate.java
- ndk/README.md, app/src/main/assets/codecs.json
- third-party/opus, third-party/speex (pattern for codec2 submodule)
