Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/guide/audio/audio-devices.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,15 @@ AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio);
PeerConnectionFactory factory = new PeerConnectionFactory(audioModule);
```

This is useful for applications that only need video functionality, when you want to implement your own custom audio handling, or for headless modes where neither audio nor video is required. Using the dummy audio layer is particularly valuable in server-side or automated testing environments where no physical audio devices are available.
This is useful for applications that only need video functionality, or for headless modes where neither audio nor video is required. Using the dummy audio layer is particularly valuable in server-side or automated testing environments where no physical audio devices are available.

::: warning
The dummy layer disables audio in both directions. Nothing is captured, and nothing is rendered, so an `AudioTrackSink` added to a received audio track is never called. To push your own audio, or to receive remote audio without opening a real device, use the [Headless Audio Device Module](/guide/audio/headless-audio) instead.
:::

::: warning
Do not call `setAudioSink` or `setAudioSource` on a module you passed to a `PeerConnectionFactory`. Both replace the audio transport WebRTC installed on the module, which silently stops audio flowing to and from every peer connection of that factory. Use them on a module of your own, as [AudioRecorder](/tools/audio/audio-recorder) and [AudioPlayer](/tools/audio/audio-player) do.
:::

## Additional Features

Expand Down
99 changes: 95 additions & 4 deletions docs/guide/audio/custom-audio-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,26 @@ PeerConnectionFactory factory = new PeerConnectionFactory();
AudioTrack audioTrack = factory.createAudioTrack("audio-track-id", audioSource);
```

::: warning One kind of audio input per factory
A factory sends audio either from its `AudioDeviceModule`, which is what tracks created with `createAudioSource(AudioOptions)` send, or from audio you push. WebRTC feeds device-captured audio into **every** audio sender of a factory, so a sender that is fed both ways is fed by two threads at once. That is a race inside WebRTC, and it aborts the whole process:

```
Fatal error in: ../../audio/audio_send_stream.cc
Check failed: !race_checker.RaceDetected()
```

The factory therefore commits to the kind used first and throws an `IllegalStateException` if you ask for the other. The same applies to a track received from a remote peer and forwarded on, since its audio is pushed as well. If you need both, create a second `PeerConnectionFactory`.

While a factory sends pushed audio it never opens the recording device, so no microphone is used. Playout of received audio is unaffected.
:::

### Pushing Audio Data

The key feature of `CustomAudioSource` is the ability to push audio data directly to the WebRTC pipeline:

```java
// Parameters for the audio data
int bitsPerSample = 16; // Common values: 8, 16, 32
int bitsPerSample = 16; // Must be 16; WebRTC reads the samples as 16-bit PCM
int sampleRate = 48000; // Common values: 8000, 16000, 44100, 48000
int channels = 2; // 1 for mono, 2 for stereo
int frameCount = 480; // For 10ms of audio at 48kHz
Expand All @@ -68,14 +81,23 @@ byte[] audioData = new byte[frameCount * channels * bytesPerSample];
audioSource.pushAudio(audioData, bitsPerSample, sampleRate, channels, frameCount);
```

The call checks its arguments and throws `IllegalArgumentException` rather than letting a bad chunk reach WebRTC, where it would read past your array or abort the process:

- Samples must be **16-bit** signed PCM in the platform byte order, with the channels interleaved.
- The array must hold at least `frameCount * channels * 2` bytes. A longer array is fine; only that many bytes are read.
- One call carries one chunk, and **10 ms** is the size WebRTC works with. A chunk may hold at most `CustomAudioSource.MAX_SAMPLES_PER_PUSH` samples counting every channel, so push in small chunks rather than handing over a whole file or a growing buffer.
- `frameCount` counts frames, not samples and not bytes. A frame holds one sample per channel, so 10 ms at 48 kHz is 480 frames whether it is mono or stereo.

::: warning Push from one thread
The audio is handed to the track's senders on the thread that calls `pushAudio`, so call it from a single thread. A scheduled executor with one thread, as shown below, is the simplest way to do that.
:::

## Audio Format Considerations

When pushing audio data, you need to consider the following parameters:

### Bits Per Sample
- **8-bit**: Lower quality, smaller data size
- **16-bit**: Standard quality for most applications
- **32-bit**: Higher quality, larger data size
- **16-bit**: the only width WebRTC accepts. Convert audio of any other width before pushing it.

### Sample Rate
- **8000 Hz**: Telephone quality
Expand Down Expand Up @@ -183,6 +205,75 @@ audioSource.dispose();
audioStreamer.stop();
```

## Audio processing

Pushed audio goes straight to the track's senders and does **not** pass through WebRTC's audio processing module. Echo cancellation, noise suppression and gain control apply to device-captured audio only, so they have no effect on a `CustomAudioSource`. Apply any processing you need before you push.

The standalone [`AudioProcessing`](/guide/audio/audio-processing) class wraps the same processing module, so you can run each chunk through it right before `pushAudio`. It takes the same 10 ms frames of 16-bit PCM that `pushAudio` takes, so no reformatting is needed in between.

### Noise suppression and gain control

```java
int sampleRate = 48000;
int channels = 1;
int frameCount = sampleRate / 100; // 10 ms

AudioProcessingConfig config = new AudioProcessingConfig();
config.noiseSuppression.enabled = true;
config.gainControllerDigital.enabled = true;
config.gainControllerDigital.adaptiveDigital.enabled = true;

AudioProcessing processing = new AudioProcessing();
processing.applyConfig(config);

// Input and output use the same format here; the module can also resample or
// down-mix between the two, see the audio processing guide.
AudioProcessingStreamConfig format = new AudioProcessingStreamConfig(sampleRate, channels);

byte[] raw = new byte[frameCount * channels * 2];
byte[] processed = new byte[processing.getTargetBufferSize(format, format)];

// On the single push thread, once per 10 ms:
fillWithAudio(raw); // your source
int result = processing.processStream(raw, format, format, processed);

if (result == 0) {
audioSource.pushAudio(processed, 16, sampleRate, channels, frameCount);
}
```

Dispose the `AudioProcessing` instance together with the source once you stop pushing.

### Echo cancellation

Echo cancellation needs the far-end signal as a reference. Device capture gets it for free, since WebRTC feeds every rendered frame back into the module itself. For a `CustomAudioSource` you provide it: feed the audio you play out through `processReverseStream`, and tell the module how far apart the two streams are in time.

```java
config.echoCanceller.enabled = true;
processing.applyConfig(config);

// The delay between a far-end frame reaching processReverseStream and the
// echo of it reaching processStream. Measure it for your setup; a value that
// is roughly right is enough for the canceller to lock on.
processing.setStreamDelayMs(50);

// Feed every far-end frame you play out. A sink on the received track gets
// them in the format below. WebRTC calls it on its own thread, which is fine:
// processStream and processReverseStream may run concurrently.
remoteAudioTrack.addSink((data, bitsPerSample, rate, ch, frames) -> {
AudioProcessingStreamConfig farEnd = new AudioProcessingStreamConfig(rate, ch);
byte[] reverse = new byte[processing.getTargetBufferSize(farEnd, farEnd)];

processing.processReverseStream(data, farEnd, farEnd, reverse);
});
```

The canceller can only remove echo of audio that actually went through `processReverseStream`. If your application plays out the received audio through a path the sink does not see, or plays other sounds alongside it, that audio comes back uncancelled.

::: tip
If a factory sends only pushed audio, the processing module WebRTC creates for that factory does no work. Enabling its features through `AudioOptions` or the factory's `AudioProcessing` changes nothing for a `CustomAudioSource`; only the instance you apply yourself does.
:::

## Conclusion

The `CustomAudioSource` provides a flexible way to integrate external audio sources with WebRTC. By understanding the audio format parameters and properly managing the audio data flow, you can create applications that use custom audio from virtually any source.
Expand Down
33 changes: 20 additions & 13 deletions docs/guide/audio/headless-audio.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Headless Audio

The `HeadlessAudioDeviceModule` is a convenience implementation of the `AudioDeviceModule` that uses WebRTC's dummy audio layer. It avoids touching real OS audio devices while still enabling the WebRTC audio pipeline to pull and render audio frames (headless playout) and to simulate capture (recording path).
The `HeadlessAudioDeviceModule` is a convenience implementation of the `AudioDeviceModule` that uses WebRTC's dummy audio layer. It avoids touching real OS audio devices while still driving the WebRTC render pipeline, which is what delivers received audio to an `AudioTrackSink`. It has no capture path; audio you want to send goes through a `CustomAudioSource`.

This is ideal for:
- Server-side or CI environments without audio hardware
Expand All @@ -11,7 +11,7 @@ This is ideal for:
## Key characteristics
- Uses dummy audio drivers; no real system devices are opened
- Exposes at least one dummy playout and recording device to allow initialization
- Supports playout and recording initialization and start/stop lifecycle
- Supports the playout and recording start/stop lifecycle, though recording carries no audio
- Intended primarily for headless scenarios where you want the WebRTC audio pipelines to run without touching physical devices

---
Expand Down Expand Up @@ -55,21 +55,26 @@ finally {
:::


## Recording path (capture)
## Sending audio

The headless module also implements a recording path that simulates a microphone. When started, it periodically pulls 10 ms of PCM from the registered AudioTransport (your Java audio source) and feeds it into WebRTC’s capture pipeline. This is particularly useful in tests or server-side senders.
The module has no capture path. There is no device to capture from, so `initRecording()` and `startRecording()` only move the module through the recording lifecycle and deliver no audio. Audio you want to send goes through a `CustomAudioSource`, which hands it to the track's senders directly.

::: info
Earlier versions pulled the render mix back through the module and handed it to WebRTC as captured audio. That made a peer connection send the audio it had just received from the remote peer straight back to it, and it fed the send stream a second time alongside the pushed audio, which aborted the process inside WebRTC. Calling `startRecording()` is now harmless but does nothing.
:::

Typical steps:

```java
HeadlessAudioDeviceModule adm = new HeadlessAudioDeviceModule();

// Initialize and start the recording pipeline (capture)
adm.initRecording();
adm.startRecording();

PeerConnectionFactory factory = new PeerConnectionFactory(adm);

// Playout drives the receive side of every peer connection of this factory,
// so start it even when the application only sends.
adm.initPlayout();
adm.startPlayout();

// Use a custom or built-in AudioSource to provide audio frames
CustomAudioSource source = new CustomAudioSource();
AudioTrack senderTrack = factory.createAudioTrack("audio0", source);
Expand All @@ -80,31 +85,33 @@ byte[] pcm = new byte[480 /* frames */ * 2 /* ch */ * 2 /* bytes */];
source.pushAudio(pcm, 16, 48000, 2, 480);

// ... later, stop
adm.stopRecording();
adm.stopPlayout();
// addTrack() returns an RTCRtpSender that is not owned by the peer
// connection, so dispose it explicitly.
sender.dispose();
source.dispose();
adm.dispose();
factory.dispose();
```

::: info
- Initialization order matters: call `initRecording()` before `startRecording()`.
- Push one chunk of 16-bit PCM per call, with 10 ms being the size WebRTC works with. See [Custom Audio Source](/guide/audio/custom-audio-source) for the format rules.
- Push from a single thread. The audio reaches the sender on the thread that calls `pushAudio`.
- The module exposes one virtual recording device; selection calls succeed with index 0.
- Stereo can be enabled/disabled via the standard ADM methods; by default 1 channel is used.
- If no AudioTransport is registered (no source), silence is injected to keep timings consistent.
:::

## When to use HeadlessAudioDeviceModule vs. dummy audio layer on AudioDeviceModule

- Prefer `HeadlessAudioDeviceModule` when you need to receive remote audio frames in a headless environment and consume them via `AudioTrack.addSink(AudioSink)`, or when you need to send audio from a custom source without touching physical devices. The headless module drives both playout and recording pipelines while no real system audio device is opened.
- Prefer `HeadlessAudioDeviceModule` when you need to receive remote audio frames in a headless environment and consume them via `AudioTrack.addSink(AudioSink)`, or when you need to send audio from a custom source without touching physical devices. Its render thread drives the receive side of every peer connection of the factory, while no real system audio device is opened.
- Using a standard `AudioDeviceModule` with `AudioLayer.kDummyAudio` disables actual audio I/O; the audio pipeline is not started for playout and sinks will typically not receive audio frame callbacks. Use this only when you intentionally do not want any audio delivery (e.g., video‑only or fully custom audio).

Related guides:
- [Audio Device Selection](/guide/audio/audio-devices)
- [Custom Audio Source](/guide/audio/custom-audio-source)

## Limitations and notes
- No real audio is played or captured; playout frames are pulled from the render pipeline and discarded, and capture frames are pulled from your source (or zeroed) and delivered into WebRTC.
- No real audio is played or captured. Playout frames are pulled from the render pipeline and discarded, and nothing is captured.
- Always follow the lifecycles: `initPlayout()` before `startPlayout()`, and `initRecording()` before `startRecording()`. Stop before dispose.
- Playout may be started before or after the peer connections are created; either order works.
- The library handles native loading internally; instantiate and use the module as shown above.
4 changes: 2 additions & 2 deletions webrtc-jni/src/main/cpp/include/JNI_CustomAudioSource.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 22 additions & 6 deletions webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading