diff --git a/docs/guide/audio/audio-devices.md b/docs/guide/audio/audio-devices.md index 16d66698..70501c4e 100644 --- a/docs/guide/audio/audio-devices.md +++ b/docs/guide/audio/audio-devices.md @@ -85,7 +85,15 @@ AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); PeerConnectionFactory factory = new PeerConnectionFactory(audioModule); ``` -This is useful for applications that only need video functionality, when you want to implement your own custom audio handling, or for headless modes where neither audio nor video is required. Using the dummy audio layer is particularly valuable in server-side or automated testing environments where no physical audio devices are available. +This is useful for applications that only need video functionality, or for headless modes where neither audio nor video is required. Using the dummy audio layer is particularly valuable in server-side or automated testing environments where no physical audio devices are available. + +::: warning +The dummy layer disables audio in both directions. Nothing is captured, and nothing is rendered, so an `AudioTrackSink` added to a received audio track is never called. To push your own audio, or to receive remote audio without opening a real device, use the [Headless Audio Device Module](/guide/audio/headless-audio) instead. +::: + +::: warning +Do not call `setAudioSink` or `setAudioSource` on a module you passed to a `PeerConnectionFactory`. Both replace the audio transport WebRTC installed on the module, which silently stops audio flowing to and from every peer connection of that factory. Use them on a module of your own, as [AudioRecorder](/tools/audio/audio-recorder) and [AudioPlayer](/tools/audio/audio-player) do. +::: ## Additional Features diff --git a/docs/guide/audio/custom-audio-source.md b/docs/guide/audio/custom-audio-source.md index 33d90a67..bdc5d0ab 100644 --- a/docs/guide/audio/custom-audio-source.md +++ b/docs/guide/audio/custom-audio-source.md @@ -45,13 +45,26 @@ PeerConnectionFactory factory = new PeerConnectionFactory(); AudioTrack audioTrack = factory.createAudioTrack("audio-track-id", audioSource); ``` +::: warning One kind of audio input per factory +A factory sends audio either from its `AudioDeviceModule`, which is what tracks created with `createAudioSource(AudioOptions)` send, or from audio you push. WebRTC feeds device-captured audio into **every** audio sender of a factory, so a sender that is fed both ways is fed by two threads at once. That is a race inside WebRTC, and it aborts the whole process: + +``` +Fatal error in: ../../audio/audio_send_stream.cc +Check failed: !race_checker.RaceDetected() +``` + +The factory therefore commits to the kind used first and throws an `IllegalStateException` if you ask for the other. The same applies to a track received from a remote peer and forwarded on, since its audio is pushed as well. If you need both, create a second `PeerConnectionFactory`. + +While a factory sends pushed audio it never opens the recording device, so no microphone is used. Playout of received audio is unaffected. +::: + ### Pushing Audio Data The key feature of `CustomAudioSource` is the ability to push audio data directly to the WebRTC pipeline: ```java // Parameters for the audio data -int bitsPerSample = 16; // Common values: 8, 16, 32 +int bitsPerSample = 16; // Must be 16; WebRTC reads the samples as 16-bit PCM int sampleRate = 48000; // Common values: 8000, 16000, 44100, 48000 int channels = 2; // 1 for mono, 2 for stereo int frameCount = 480; // For 10ms of audio at 48kHz @@ -68,14 +81,23 @@ byte[] audioData = new byte[frameCount * channels * bytesPerSample]; audioSource.pushAudio(audioData, bitsPerSample, sampleRate, channels, frameCount); ``` +The call checks its arguments and throws `IllegalArgumentException` rather than letting a bad chunk reach WebRTC, where it would read past your array or abort the process: + +- Samples must be **16-bit** signed PCM in the platform byte order, with the channels interleaved. +- The array must hold at least `frameCount * channels * 2` bytes. A longer array is fine; only that many bytes are read. +- One call carries one chunk, and **10 ms** is the size WebRTC works with. A chunk may hold at most `CustomAudioSource.MAX_SAMPLES_PER_PUSH` samples counting every channel, so push in small chunks rather than handing over a whole file or a growing buffer. +- `frameCount` counts frames, not samples and not bytes. A frame holds one sample per channel, so 10 ms at 48 kHz is 480 frames whether it is mono or stereo. + +::: warning Push from one thread +The audio is handed to the track's senders on the thread that calls `pushAudio`, so call it from a single thread. A scheduled executor with one thread, as shown below, is the simplest way to do that. +::: + ## Audio Format Considerations When pushing audio data, you need to consider the following parameters: ### Bits Per Sample -- **8-bit**: Lower quality, smaller data size -- **16-bit**: Standard quality for most applications -- **32-bit**: Higher quality, larger data size +- **16-bit**: the only width WebRTC accepts. Convert audio of any other width before pushing it. ### Sample Rate - **8000 Hz**: Telephone quality @@ -183,6 +205,75 @@ audioSource.dispose(); audioStreamer.stop(); ``` +## Audio processing + +Pushed audio goes straight to the track's senders and does **not** pass through WebRTC's audio processing module. Echo cancellation, noise suppression and gain control apply to device-captured audio only, so they have no effect on a `CustomAudioSource`. Apply any processing you need before you push. + +The standalone [`AudioProcessing`](/guide/audio/audio-processing) class wraps the same processing module, so you can run each chunk through it right before `pushAudio`. It takes the same 10 ms frames of 16-bit PCM that `pushAudio` takes, so no reformatting is needed in between. + +### Noise suppression and gain control + +```java +int sampleRate = 48000; +int channels = 1; +int frameCount = sampleRate / 100; // 10 ms + +AudioProcessingConfig config = new AudioProcessingConfig(); +config.noiseSuppression.enabled = true; +config.gainControllerDigital.enabled = true; +config.gainControllerDigital.adaptiveDigital.enabled = true; + +AudioProcessing processing = new AudioProcessing(); +processing.applyConfig(config); + +// Input and output use the same format here; the module can also resample or +// down-mix between the two, see the audio processing guide. +AudioProcessingStreamConfig format = new AudioProcessingStreamConfig(sampleRate, channels); + +byte[] raw = new byte[frameCount * channels * 2]; +byte[] processed = new byte[processing.getTargetBufferSize(format, format)]; + +// On the single push thread, once per 10 ms: +fillWithAudio(raw); // your source +int result = processing.processStream(raw, format, format, processed); + +if (result == 0) { + audioSource.pushAudio(processed, 16, sampleRate, channels, frameCount); +} +``` + +Dispose the `AudioProcessing` instance together with the source once you stop pushing. + +### Echo cancellation + +Echo cancellation needs the far-end signal as a reference. Device capture gets it for free, since WebRTC feeds every rendered frame back into the module itself. For a `CustomAudioSource` you provide it: feed the audio you play out through `processReverseStream`, and tell the module how far apart the two streams are in time. + +```java +config.echoCanceller.enabled = true; +processing.applyConfig(config); + +// The delay between a far-end frame reaching processReverseStream and the +// echo of it reaching processStream. Measure it for your setup; a value that +// is roughly right is enough for the canceller to lock on. +processing.setStreamDelayMs(50); + +// Feed every far-end frame you play out. A sink on the received track gets +// them in the format below. WebRTC calls it on its own thread, which is fine: +// processStream and processReverseStream may run concurrently. +remoteAudioTrack.addSink((data, bitsPerSample, rate, ch, frames) -> { + AudioProcessingStreamConfig farEnd = new AudioProcessingStreamConfig(rate, ch); + byte[] reverse = new byte[processing.getTargetBufferSize(farEnd, farEnd)]; + + processing.processReverseStream(data, farEnd, farEnd, reverse); +}); +``` + +The canceller can only remove echo of audio that actually went through `processReverseStream`. If your application plays out the received audio through a path the sink does not see, or plays other sounds alongside it, that audio comes back uncancelled. + +::: tip +If a factory sends only pushed audio, the processing module WebRTC creates for that factory does no work. Enabling its features through `AudioOptions` or the factory's `AudioProcessing` changes nothing for a `CustomAudioSource`; only the instance you apply yourself does. +::: + ## Conclusion The `CustomAudioSource` provides a flexible way to integrate external audio sources with WebRTC. By understanding the audio format parameters and properly managing the audio data flow, you can create applications that use custom audio from virtually any source. diff --git a/docs/guide/audio/headless-audio.md b/docs/guide/audio/headless-audio.md index c2d9292d..aa044ed0 100644 --- a/docs/guide/audio/headless-audio.md +++ b/docs/guide/audio/headless-audio.md @@ -1,6 +1,6 @@ # Headless Audio -The `HeadlessAudioDeviceModule` is a convenience implementation of the `AudioDeviceModule` that uses WebRTC's dummy audio layer. It avoids touching real OS audio devices while still enabling the WebRTC audio pipeline to pull and render audio frames (headless playout) and to simulate capture (recording path). +The `HeadlessAudioDeviceModule` is a convenience implementation of the `AudioDeviceModule` that uses WebRTC's dummy audio layer. It avoids touching real OS audio devices while still driving the WebRTC render pipeline, which is what delivers received audio to an `AudioTrackSink`. It has no capture path; audio you want to send goes through a `CustomAudioSource`. This is ideal for: - Server-side or CI environments without audio hardware @@ -11,7 +11,7 @@ This is ideal for: ## Key characteristics - Uses dummy audio drivers; no real system devices are opened - Exposes at least one dummy playout and recording device to allow initialization -- Supports playout and recording initialization and start/stop lifecycle +- Supports the playout and recording start/stop lifecycle, though recording carries no audio - Intended primarily for headless scenarios where you want the WebRTC audio pipelines to run without touching physical devices --- @@ -55,21 +55,26 @@ finally { ::: -## Recording path (capture) +## Sending audio -The headless module also implements a recording path that simulates a microphone. When started, it periodically pulls 10 ms of PCM from the registered AudioTransport (your Java audio source) and feeds it into WebRTC’s capture pipeline. This is particularly useful in tests or server-side senders. +The module has no capture path. There is no device to capture from, so `initRecording()` and `startRecording()` only move the module through the recording lifecycle and deliver no audio. Audio you want to send goes through a `CustomAudioSource`, which hands it to the track's senders directly. + +::: info +Earlier versions pulled the render mix back through the module and handed it to WebRTC as captured audio. That made a peer connection send the audio it had just received from the remote peer straight back to it, and it fed the send stream a second time alongside the pushed audio, which aborted the process inside WebRTC. Calling `startRecording()` is now harmless but does nothing. +::: Typical steps: ```java HeadlessAudioDeviceModule adm = new HeadlessAudioDeviceModule(); -// Initialize and start the recording pipeline (capture) -adm.initRecording(); -adm.startRecording(); - PeerConnectionFactory factory = new PeerConnectionFactory(adm); +// Playout drives the receive side of every peer connection of this factory, +// so start it even when the application only sends. +adm.initPlayout(); +adm.startPlayout(); + // Use a custom or built-in AudioSource to provide audio frames CustomAudioSource source = new CustomAudioSource(); AudioTrack senderTrack = factory.createAudioTrack("audio0", source); @@ -80,24 +85,25 @@ byte[] pcm = new byte[480 /* frames */ * 2 /* ch */ * 2 /* bytes */]; source.pushAudio(pcm, 16, 48000, 2, 480); // ... later, stop -adm.stopRecording(); +adm.stopPlayout(); // addTrack() returns an RTCRtpSender that is not owned by the peer // connection, so dispose it explicitly. sender.dispose(); +source.dispose(); adm.dispose(); factory.dispose(); ``` ::: info -- Initialization order matters: call `initRecording()` before `startRecording()`. +- Push one chunk of 16-bit PCM per call, with 10 ms being the size WebRTC works with. See [Custom Audio Source](/guide/audio/custom-audio-source) for the format rules. +- Push from a single thread. The audio reaches the sender on the thread that calls `pushAudio`. - The module exposes one virtual recording device; selection calls succeed with index 0. - Stereo can be enabled/disabled via the standard ADM methods; by default 1 channel is used. -- If no AudioTransport is registered (no source), silence is injected to keep timings consistent. ::: ## When to use HeadlessAudioDeviceModule vs. dummy audio layer on AudioDeviceModule -- Prefer `HeadlessAudioDeviceModule` when you need to receive remote audio frames in a headless environment and consume them via `AudioTrack.addSink(AudioSink)`, or when you need to send audio from a custom source without touching physical devices. The headless module drives both playout and recording pipelines while no real system audio device is opened. +- Prefer `HeadlessAudioDeviceModule` when you need to receive remote audio frames in a headless environment and consume them via `AudioTrack.addSink(AudioSink)`, or when you need to send audio from a custom source without touching physical devices. Its render thread drives the receive side of every peer connection of the factory, while no real system audio device is opened. - Using a standard `AudioDeviceModule` with `AudioLayer.kDummyAudio` disables actual audio I/O; the audio pipeline is not started for playout and sinks will typically not receive audio frame callbacks. Use this only when you intentionally do not want any audio delivery (e.g., video‑only or fully custom audio). Related guides: @@ -105,6 +111,7 @@ Related guides: - [Custom Audio Source](/guide/audio/custom-audio-source) ## Limitations and notes -- No real audio is played or captured; playout frames are pulled from the render pipeline and discarded, and capture frames are pulled from your source (or zeroed) and delivered into WebRTC. +- No real audio is played or captured. Playout frames are pulled from the render pipeline and discarded, and nothing is captured. - Always follow the lifecycles: `initPlayout()` before `startPlayout()`, and `initRecording()` before `startRecording()`. Stop before dispose. +- Playout may be started before or after the peer connections are created; either order works. - The library handles native loading internally; instantiate and use the module as shown above. diff --git a/webrtc-jni/src/main/cpp/include/JNI_CustomAudioSource.h b/webrtc-jni/src/main/cpp/include/JNI_CustomAudioSource.h index 2355fe2d..d68c37a4 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_CustomAudioSource.h +++ b/webrtc-jni/src/main/cpp/include/JNI_CustomAudioSource.h @@ -33,10 +33,10 @@ extern "C" { /* * Class: dev_onvoid_webrtc_media_audio_CustomAudioSource - * Method: pushAudio + * Method: pushAudioInternal * Signature: ([BIIII)V */ - JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_CustomAudioSource_pushAudio + JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_CustomAudioSource_pushAudioInternal (JNIEnv *, jobject, jbyteArray, jint, jint, jint, jint); #ifdef __cplusplus diff --git a/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h b/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h index c73fba14..eeff1838 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h +++ b/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h @@ -9,20 +9,36 @@ extern "C" { #endif /* * Class: dev_onvoid_webrtc_PeerConnectionFactory - * Method: createAudioSource + * Method: createAudioSourceInternal * Signature: (Ldev/onvoid/webrtc/media/audio/AudioOptions;)Ldev/onvoid/webrtc/media/audio/AudioTrackSource; */ - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioSource + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioSourceInternal (JNIEnv *, jobject, jobject); /* * Class: dev_onvoid_webrtc_PeerConnectionFactory - * Method: createAudioTrack + * Method: createAudioTrackInternal * Signature: (Ljava/lang/String;Ldev/onvoid/webrtc/media/audio/AudioTrackSource;)Ldev/onvoid/webrtc/media/audio/AudioTrack; */ - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioTrack + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioTrackInternal (JNIEnv *, jobject, jstring, jobject); + /* + * Class: dev_onvoid_webrtc_PeerConnectionFactory + * Method: setDeviceCaptureEnabled + * Signature: (Z)V + */ + JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_setDeviceCaptureEnabled + (JNIEnv *, jobject, jboolean); + + /* + * Class: dev_onvoid_webrtc_PeerConnectionFactory + * Method: isSinkFedAudioTrack + * Signature: (Ldev/onvoid/webrtc/media/MediaStreamTrack;)Z + */ + JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_isSinkFedAudioTrack + (JNIEnv *, jobject, jobject); + /* * Class: dev_onvoid_webrtc_PeerConnectionFactory * Method: createVideoTrack @@ -33,10 +49,10 @@ extern "C" { /* * Class: dev_onvoid_webrtc_PeerConnectionFactory - * Method: createPeerConnection + * Method: createPeerConnectionInternal * Signature: (Ldev/onvoid/webrtc/RTCConfiguration;Ldev/onvoid/webrtc/PeerConnectionObserver;)Ldev/onvoid/webrtc/RTCPeerConnection; */ - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createPeerConnection + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createPeerConnectionInternal (JNIEnv *, jobject, jobject, jobject); /* diff --git a/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h b/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h index bdf79e04..fd3aaea9 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h +++ b/webrtc-jni/src/main/cpp/include/JNI_RTCPeerConnection.h @@ -33,10 +33,10 @@ extern "C" { /* * Class: dev_onvoid_webrtc_RTCPeerConnection - * Method: addTrack + * Method: addTrackInternal * Signature: (Ldev/onvoid/webrtc/media/MediaStreamTrack;Ljava/util/List;)Ldev/onvoid/webrtc/RTCRtpSender; */ - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTrack + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTrackInternal (JNIEnv *, jobject, jobject, jobject); /* @@ -49,10 +49,10 @@ extern "C" { /* * Class: dev_onvoid_webrtc_RTCPeerConnection - * Method: addTransceiver + * Method: addTransceiverInternal * Signature: (Ldev/onvoid/webrtc/media/MediaStreamTrack;Ldev/onvoid/webrtc/RTCRtpTransceiverInit;)Ldev/onvoid/webrtc/RTCRtpTransceiver; */ - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceiver + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceiverInternal (JNIEnv *, jobject, jobject, jobject); /* diff --git a/webrtc-jni/src/main/cpp/include/api/HeadlessAudioDeviceModule.h b/webrtc-jni/src/main/cpp/include/api/HeadlessAudioDeviceModule.h index 8d2c7e63..cc63f59f 100644 --- a/webrtc-jni/src/main/cpp/include/api/HeadlessAudioDeviceModule.h +++ b/webrtc-jni/src/main/cpp/include/api/HeadlessAudioDeviceModule.h @@ -23,10 +23,7 @@ #include #include -#include "api/environment/environment.h" -#include "api/environment/environment_factory.h" #include "api/make_ref_counted.h" -#include "modules/audio_device/audio_device_buffer.h" #include "modules/audio_device/include/audio_device.h" #include "modules/audio_device/include/audio_device_defines.h" #include "rtc_base/buffer.h" @@ -39,22 +36,24 @@ namespace jni { // A headless AudioDeviceModule that drives the render pipeline by pulling - // 10 ms PCM chunks from AudioTransport and discarding them, and simulates - // a microphone by pulling 10 ms PCM chunks from the registered AudioTransport - // and feeding them into the WebRTC capture pipeline. + // 10 ms PCM chunks from AudioTransport and discarding them. + // + // It has no capture path. Recording is a state this module reports and + // nothing more, so that the recording lifecycle of the AudioDeviceModule + // interface still works. There is no device to capture from, and audio a + // headless application wants to send goes through a CustomAudioSource. class HeadlessAudioDeviceModule : public webrtc::AudioDeviceModule { public: static webrtc::scoped_refptr Create( - const webrtc::Environment & env, int sample_rate_hz = 48000, size_t channels = 1) { return webrtc::make_ref_counted( - env, sample_rate_hz, channels); + sample_rate_hz, channels); } - HeadlessAudioDeviceModule(const webrtc::Environment & env, int sample_rate_hz, size_t channels); + HeadlessAudioDeviceModule(int sample_rate_hz, size_t channels); ~HeadlessAudioDeviceModule() override; // ----- AudioDeviceModule interface ----- @@ -148,7 +147,6 @@ namespace jni private: bool PlayThreadProcess(); - bool CaptureThreadProcess(); // State bool initialized_ = false; @@ -162,22 +160,25 @@ namespace jni size_t channels_ = 1; webrtc::BufferT play_buffer_; - webrtc::BufferT record_buffer_; size_t playoutFramesIn10MS_; - size_t recordingFramesIn10MS_; // Absolute wall-clock deadline (ms) of the next 10 ms tick. Advanced by a // fixed +10 each tick so scheduling/wake-up latency is corrected against the // grid rather than accumulating into the frame period. int64_t nextPlayoutMillis_; - int64_t nextRecordMillis_; + // Guards the module state above. Never held while calling into the + // transport, so a pull in progress does not stall other calls on + // this module and cannot deadlock on anything the pull does. mutable webrtc::Mutex mutex_; - std::unique_ptr audio_device_buffer_ RTC_GUARDED_BY(mutex_); - webrtc::AudioTransport * audio_callback_; + + // Guards audio_callback_ and is held for the whole of a pull, so + // that after RegisterAudioCallback() returns no call on the old + // transport is still in flight. + webrtc::Mutex callback_mutex_; + webrtc::AudioTransport * audio_callback_ RTC_GUARDED_BY(callback_mutex_); webrtc::PlatformThread render_thread_; - webrtc::PlatformThread capture_thread_; }; } diff --git a/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h b/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h new file mode 100644 index 00000000..b2d64c36 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h @@ -0,0 +1,209 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JNI_WEBRTC_API_PROXY_AUDIO_DEVICE_MODULE_H_ +#define JNI_WEBRTC_API_PROXY_AUDIO_DEVICE_MODULE_H_ + +#include "api/audio/audio_device.h" +#include "api/audio/audio_device_defines.h" +#include "api/scoped_refptr.h" + +#include +#include +#include + +namespace jni +{ + // Wraps the AudioDeviceModule a PeerConnectionFactory is created with and + // forwards every call to it. Its purpose is to gate the device capture path. + // + // WebRTC's AudioState starts the module's recording as soon as any audio + // send stream starts and fans every recorded frame out to every send stream + // of the factory (AudioTransportImpl::SendProcessedData). A send stream whose + // track is backed by a sink-fed source (jni::CustomAudioSource, or a + // forwarded remote track) already receives its audio through WebRTC's + // LocalAudioSinkAdapter on the pushing thread. Feeding it from the capture + // thread as well trips RTC_CHECK_RUNS_SERIALIZED in + // AudioSendStream::SendAudioData and aborts the process, and interleaves + // microphone frames with the pushed audio whenever it does not. + // + // While capture is disabled, InitRecording() and StartRecording() fail so + // AudioState never opens the device, and recorded data that still arrives + // from the wrapped module (from a recording the application started through + // the Java AudioDeviceModule, which holds the wrapped module directly) is + // dropped before it reaches WebRTC. Playout is never affected. + // + // Capture starts disabled and is enabled only once the application asks for + // device-captured audio, which it can only do through + // PeerConnectionFactory.createAudioSource(). A factory that sends nothing + // but sink-fed audio therefore never opens the recording device, and one + // that sends no audio at all never opens it either. + class ProxyAudioDeviceModule : public webrtc::AudioDeviceModule + { + public: + static webrtc::scoped_refptr Create( + webrtc::scoped_refptr delegate); + + explicit ProxyAudioDeviceModule(webrtc::scoped_refptr delegate); + ~ProxyAudioDeviceModule() override; + + // Enables or disables the device capture path. May be called from any + // thread. Callers that must not interleave with AudioState starting a + // recording should run this on the factory's worker thread. + void SetCaptureEnabled(bool enabled); + bool CaptureEnabled() const; + + // webrtc::AudioDeviceModule implementation. + int32_t ActiveAudioLayer(AudioLayer * audioLayer) const override; + int32_t RegisterAudioCallback(webrtc::AudioTransport * audioCallback) override; + int32_t Init() override; + int32_t Terminate() override; + bool Initialized() const override; + + int16_t PlayoutDevices() override; + int16_t RecordingDevices() override; + int32_t PlayoutDeviceName(uint16_t index, + char name[webrtc::kAdmMaxDeviceNameSize], + char guid[webrtc::kAdmMaxGuidSize]) override; + int32_t RecordingDeviceName(uint16_t index, + char name[webrtc::kAdmMaxDeviceNameSize], + char guid[webrtc::kAdmMaxGuidSize]) override; + + int32_t SetPlayoutDevice(uint16_t index) override; + int32_t SetPlayoutDevice(WindowsDeviceType device) override; + int32_t SetRecordingDevice(uint16_t index) override; + int32_t SetRecordingDevice(WindowsDeviceType device) override; + + int32_t PlayoutIsAvailable(bool * available) override; + int32_t InitPlayout() override; + bool PlayoutIsInitialized() const override; + int32_t RecordingIsAvailable(bool * available) override; + int32_t InitRecording() override; + bool RecordingIsInitialized() const override; + + int32_t StartPlayout() override; + int32_t StopPlayout() override; + bool Playing() const override; + int32_t StartRecording() override; + int32_t StopRecording() override; + bool Recording() const override; + + int32_t InitSpeaker() override; + bool SpeakerIsInitialized() const override; + int32_t InitMicrophone() override; + bool MicrophoneIsInitialized() const override; + + int32_t SpeakerVolumeIsAvailable(bool * available) override; + int32_t SetSpeakerVolume(uint32_t volume) override; + int32_t SpeakerVolume(uint32_t * volume) const override; + int32_t MaxSpeakerVolume(uint32_t * maxVolume) const override; + int32_t MinSpeakerVolume(uint32_t * minVolume) const override; + + int32_t MicrophoneVolumeIsAvailable(bool * available) override; + int32_t SetMicrophoneVolume(uint32_t volume) override; + int32_t MicrophoneVolume(uint32_t * volume) const override; + int32_t MaxMicrophoneVolume(uint32_t * maxVolume) const override; + int32_t MinMicrophoneVolume(uint32_t * minVolume) const override; + + int32_t SpeakerMuteIsAvailable(bool * available) override; + int32_t SetSpeakerMute(bool enable) override; + int32_t SpeakerMute(bool * enabled) const override; + int32_t MicrophoneMuteIsAvailable(bool * available) override; + int32_t SetMicrophoneMute(bool enable) override; + int32_t MicrophoneMute(bool * enabled) const override; + + int32_t StereoPlayoutIsAvailable(bool * available) const override; + int32_t SetStereoPlayout(bool enable) override; + int32_t StereoPlayout(bool * enabled) const override; + int32_t StereoRecordingIsAvailable(bool * available) const override; + int32_t SetStereoRecording(bool enable) override; + int32_t StereoRecording(bool * enabled) const override; + + int32_t PlayoutDelay(uint16_t * delayMS) const override; + + bool BuiltInAECIsAvailable() const override; + bool BuiltInAGCIsAvailable() const override; + bool BuiltInNSIsAvailable() const override; + int32_t EnableBuiltInAEC(bool enable) override; + int32_t EnableBuiltInAGC(bool enable) override; + int32_t EnableBuiltInNS(bool enable) override; + + int32_t GetPlayoutUnderrunCount() const override; + std::optional GetStats() const override; + + private: + // The AudioTransport registered with the wrapped module in place of + // the one WebRTC registers with this proxy. Render calls are always + // forwarded; recorded data is forwarded only while capture is enabled. + class TransportGate : public webrtc::AudioTransport + { + public: + explicit TransportGate(const std::atomic & captureEnabled); + ~TransportGate() override = default; + + void SetTarget(webrtc::AudioTransport * target); + + // webrtc::AudioTransport implementation. + int32_t RecordedDataIsAvailable(const void * audioSamples, + size_t nSamples, + size_t nBytesPerSample, + size_t nChannels, + uint32_t samplesPerSec, + uint32_t totalDelayMS, + int32_t clockDrift, + uint32_t currentMicLevel, + bool keyPressed, + uint32_t & newMicLevel) override; + int32_t RecordedDataIsAvailable(const void * audioSamples, + size_t nSamples, + size_t nBytesPerSample, + size_t nChannels, + uint32_t samplesPerSec, + uint32_t totalDelayMS, + int32_t clockDrift, + uint32_t currentMicLevel, + bool keyPressed, + uint32_t & newMicLevel, + std::optional estimatedCaptureTimeNS) override; + int32_t NeedMorePlayData(size_t nSamples, + size_t nBytesPerSample, + size_t nChannels, + uint32_t samplesPerSec, + void * audioSamples, + size_t & nSamplesOut, + int64_t * elapsed_time_ms, + int64_t * ntp_time_ms) override; + void PullRenderData(int bits_per_sample, + int sample_rate, + size_t number_of_channels, + size_t number_of_frames, + void * audio_data, + int64_t * elapsed_time_ms, + int64_t * ntp_time_ms) override; + + private: + const std::atomic & captureEnabled_; + std::atomic target_; + }; + + webrtc::scoped_refptr delegate_; + std::atomic captureEnabled_; + TransportGate gate_; + bool gateRegistered_; + }; +} + +#endif diff --git a/webrtc-jni/src/main/cpp/src/JNI_CustomAudioSource.cpp b/webrtc-jni/src/main/cpp/src/JNI_CustomAudioSource.cpp index 84c6a471..b9af2797 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_CustomAudioSource.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_CustomAudioSource.cpp @@ -59,15 +59,15 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_CustomAudioSource_disp source = nullptr; } -JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_CustomAudioSource_pushAudio +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_CustomAudioSource_pushAudioInternal (JNIEnv * env, jobject caller, jbyteArray audioData, jint bits_per_sample, jint sampleRate, jint channels, jint frameCount) { jni::CustomAudioSource * source = GetHandle(env, caller); CHECK_HANDLE(source); + // The caller validated the format, the frame count and the array length. jbyte * data = env->GetByteArrayElements(audioData, nullptr); - jsize length = env->GetArrayLength(audioData); - + if (data != nullptr) { source->PushAudioData(data, bits_per_sample, sampleRate, channels, frameCount); diff --git a/webrtc-jni/src/main/cpp/src/JNI_HeadlessAudioDeviceModule.cpp b/webrtc-jni/src/main/cpp/src/JNI_HeadlessAudioDeviceModule.cpp index 0c4c7b77..52984285 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_HeadlessAudioDeviceModule.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_HeadlessAudioDeviceModule.cpp @@ -18,7 +18,6 @@ #include "Exception.h" #include "JavaError.h" #include "JavaUtils.h" -#include "WebRTCContext.h" #include "api/HeadlessAudioDeviceModule.h" @@ -27,8 +26,7 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_HeadlessAudioDeviceModule_initialize (JNIEnv* env, jobject caller) { - jni::WebRTCContext * context = static_cast(javaContext); - webrtc::scoped_refptr audioModule = jni::HeadlessAudioDeviceModule::Create(context->webrtcEnv); + webrtc::scoped_refptr audioModule = jni::HeadlessAudioDeviceModule::Create(); if (!audioModule) { env->Throw(jni::JavaError(env, "Create HeadlessAudioDeviceModule failed")); diff --git a/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp b/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp index 96e69199..4e3064d8 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp @@ -60,6 +60,13 @@ #include "api/video_codecs/video_encoder_factory.h" #include "api/video_codecs/video_encoder_factory_template.h" +#include "api/ProxyAudioDeviceModule.h" +#include "media/audio/CustomAudioSource.h" + +#include "api/media_stream_interface.h" +#include "rtc_base/logging.h" +#include "rtc_base/thread.h" + #include JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_initialize @@ -133,11 +140,17 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_initialize } } + // Hand WebRTC a proxy of the module so the device capture path can be + // switched off once this factory sends audio from sink-fed sources. See + // ProxyAudioDeviceModule for why both paths must never feed the same + // send stream. + auto proxy = jni::ProxyAudioDeviceModule::Create(adm); + auto factory = webrtc::CreatePeerConnectionFactory( networkThread.get(), workerThread.get(), signalingThread.get(), - adm, + proxy, webrtc::CreateBuiltinAudioEncoderFactory(), webrtc::CreateBuiltinAudioDecoderFactory(), #ifdef __APPLE__ @@ -165,6 +178,7 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_initialize SetHandle(env, caller, "networkThreadHandle", networkThread.release()); SetHandle(env, caller, "signalingThreadHandle", signalingThread.release()); SetHandle(env, caller, "workerThreadHandle", workerThread.release()); + SetHandle(env, caller, "audioModuleHandle", proxy.release()); } else { throw jni::Exception("Create PeerConnectionFactory failed"); @@ -184,6 +198,7 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_dispose webrtc::Thread * networkThread = GetHandle(env, caller, "networkThreadHandle"); webrtc::Thread * signalingThread = GetHandle(env, caller, "signalingThreadHandle"); webrtc::Thread * workerThread = GetHandle(env, caller, "workerThreadHandle"); + jni::ProxyAudioDeviceModule * audioModule = GetHandle(env, caller, "audioModuleHandle"); webrtc::RefCountReleaseStatus status = factory->Release(); @@ -208,13 +223,45 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_dispose workerThread->Stop(); delete workerThread; } + if (audioModule) { + // The factory dropped its reference above; this releases the last one + // and, with it, the proxy's reference on the wrapped module. + webrtc::RefCountReleaseStatus admStatus = audioModule->Release(); + + if (admStatus != webrtc::RefCountReleaseStatus::kDroppedLastRef) { + RTC_LOG(LS_WARNING) << "ProxyAudioDeviceModule was not deleted. A reference is still around somewhere."; + } + + SetHandle(env, caller, "audioModuleHandle", nullptr); + } + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_setDeviceCaptureEnabled +(JNIEnv * env, jobject caller, jboolean enabled) +{ + jni::ProxyAudioDeviceModule * audioModule = GetHandle(env, caller, "audioModuleHandle"); + CHECK_HANDLE(audioModule); + + webrtc::Thread * workerThread = GetHandle(env, caller, "workerThreadHandle"); + CHECK_HANDLE(workerThread); + + try { + // AudioState starts and stops the module's recording on the worker thread. + // Switching the capture path there keeps the two from interleaving. + workerThread->BlockingCall([audioModule, enabled]() { + audioModule->SetCaptureEnabled(enabled == JNI_TRUE); + }); } catch (...) { ThrowCxxJavaException(env); } } -JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioSource +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioSourceInternal (JNIEnv * env, jobject caller, jobject jAudioOptions) { if (jAudioOptions == nullptr) { @@ -237,7 +284,7 @@ JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAud return jni::JavaFactories::create(env, audioSource.release()).release(); } -JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioTrack +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioTrackInternal (JNIEnv * env, jobject caller, jstring jlabel, jobject jsource) { if (jlabel == nullptr) { @@ -262,6 +309,34 @@ JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAud return jni::JavaFactories::create(env, audioTrack.release()).release(); } +JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_isSinkFedAudioTrack +(JNIEnv * env, jobject caller, jobject jTrack) +{ + if (jTrack == nullptr) { + return JNI_FALSE; + } + + webrtc::MediaStreamTrackInterface * track = GetHandle(env, jTrack); + + if (track == nullptr || track->kind() != webrtc::MediaStreamTrackInterface::kAudioKind) { + return JNI_FALSE; + } + + webrtc::AudioSourceInterface * source = static_cast(track)->GetSource(); + + if (source == nullptr) { + return JNI_FALSE; + } + + // A remote source hands the track's sinks the audio it decodes, and a + // CustomAudioSource hands them the audio the application pushes. Either way + // a sender of that track is fed through the track rather than by the audio + // device module. + bool sinkFed = source->remote() || dynamic_cast(source) != nullptr; + + return sinkFed ? JNI_TRUE : JNI_FALSE; +} + JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createVideoTrack (JNIEnv * env, jobject caller, jstring jlabel, jobject jsource) { @@ -288,7 +363,7 @@ JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createVid return jni::JavaFactories::create(env, videoTrack.release()).release(); } -JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createPeerConnection +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createPeerConnectionInternal (JNIEnv * env, jobject caller, jobject jConfig, jobject jobserver) { if (jConfig == nullptr) { diff --git a/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp b/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp index 455306b6..f41af06a 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_RTCPeerConnection.cpp @@ -99,7 +99,7 @@ JNIEXPORT jobjectArray JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_getTrans return objectArray.release(); } -JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTrack +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTrackInternal (JNIEnv * env, jobject caller, jobject jTrack, jobject jStreamIds) { if (jTrack == nullptr) { @@ -154,7 +154,7 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_removeTrack } } -JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceiver +JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_RTCPeerConnection_addTransceiverInternal (JNIEnv * env, jobject caller, jobject jTrack, jobject jTransceiverInit) { if (jTrack == nullptr) { diff --git a/webrtc-jni/src/main/cpp/src/api/HeadlessAudioDeviceModule.cpp b/webrtc-jni/src/main/cpp/src/api/HeadlessAudioDeviceModule.cpp index 93da7a3a..60738f64 100644 --- a/webrtc-jni/src/main/cpp/src/api/HeadlessAudioDeviceModule.cpp +++ b/webrtc-jni/src/main/cpp/src/api/HeadlessAudioDeviceModule.cpp @@ -18,18 +18,13 @@ namespace jni { - HeadlessAudioDeviceModule::HeadlessAudioDeviceModule(const webrtc::Environment & env, - int sample_rate_hz, - size_t channels): + HeadlessAudioDeviceModule::HeadlessAudioDeviceModule(int sample_rate_hz, size_t channels): sample_rate_hz_(sample_rate_hz), channels_(channels ? channels : 1), playoutFramesIn10MS_(0), - recordingFramesIn10MS_(0), nextPlayoutMillis_(0), - nextRecordMillis_(0), audio_callback_(nullptr) { - audio_device_buffer_ = std::make_unique(env); } HeadlessAudioDeviceModule::~HeadlessAudioDeviceModule() @@ -49,9 +44,14 @@ namespace jni int32_t HeadlessAudioDeviceModule::RegisterAudioCallback(webrtc::AudioTransport * audioCallback) { - webrtc::MutexLock lock(&mutex_); + // The render thread holds callback_mutex_ across a pull, so this waits + // for a pull in progress and no call reaches the old transport after + // this returns. Unlike AudioDeviceBuffer, the swap is accepted while + // playout runs: WebRTC registers its transport only once it builds the + // voice engine, and an application may well have started playout by + // then. + webrtc::MutexLock lock(&callback_mutex_); audio_callback_ = audioCallback; - audio_device_buffer_->RegisterAudioCallback(audioCallback); return 0; } @@ -144,10 +144,9 @@ namespace jni return -1; } - playoutFramesIn10MS_ = static_cast(sample_rate_hz_ / 100); + webrtc::MutexLock lock(&mutex_); - audio_device_buffer_->SetPlayoutSampleRate(static_cast(sample_rate_hz_)); - audio_device_buffer_->SetPlayoutChannels(static_cast(channels_)); + playoutFramesIn10MS_ = static_cast(sample_rate_hz_ / 100); const size_t total_samples = channels_ * playoutFramesIn10MS_; if (play_buffer_.size() != total_samples) { @@ -178,15 +177,7 @@ namespace jni return -1; } - recordingFramesIn10MS_ = static_cast(sample_rate_hz_ / 100); - - audio_device_buffer_->SetRecordingSampleRate(static_cast(sample_rate_hz_)); - audio_device_buffer_->SetRecordingChannels(static_cast(channels_)); - - const size_t total_samples = channels_ * recordingFramesIn10MS_; - if (record_buffer_.size() != total_samples) { - record_buffer_.SetSize(total_samples); - } + webrtc::MutexLock lock(&mutex_); recording_initialized_ = true; return 0; @@ -212,7 +203,6 @@ namespace jni return 0; // already playing } playing_ = true; - audio_device_buffer_->StartPlayout(); } // Launch 10ms render pull thread. @@ -242,10 +232,6 @@ namespace jni render_thread_.Finalize(); } - { - webrtc::MutexLock lock(&mutex_); - audio_device_buffer_->StopPlayout(); - } return 0; } @@ -263,46 +249,25 @@ namespace jni return -1; } - { - webrtc::MutexLock lock(&mutex_); - if (recording_) { - return 0; // already recording - } - recording_ = true; - audio_device_buffer_->StartRecording(); - } + // Recording is a state on this module and nothing more: there is no + // device to capture from, and no capture thread runs. An earlier version + // pulled the render mix through AudioTransport and handed it back as + // recorded audio, which made a peer connection send the audio it had just + // received from the remote peer straight back to it. Audio a headless + // application wants to send goes through a CustomAudioSource instead. + webrtc::MutexLock lock(&mutex_); - // Launch 10ms capture push thread. - capture_thread_ = webrtc::PlatformThread::SpawnJoinable( - [this] { - while (CaptureThreadProcess()) { - } - }, - "webrtc_audio_module_capture_thread", - webrtc::ThreadAttributes().SetPriority(webrtc::ThreadPriority::kRealtime)); + recording_ = true; return 0; } int32_t HeadlessAudioDeviceModule::StopRecording() { - { - webrtc::MutexLock lock(&mutex_); - if (!recording_) { - // Already stopped. - return 0; - } - recording_ = false; - } + webrtc::MutexLock lock(&mutex_); - if (!capture_thread_.empty()) { - capture_thread_.Finalize(); - } + recording_ = false; - { - webrtc::MutexLock lock(&mutex_); - audio_device_buffer_->StopRecording(); - } return 0; } @@ -442,16 +407,15 @@ namespace jni return -1; } - channels_ = enable ? 2u : 1u; - // Propagate channel change to AudioDeviceBuffer if playout is initialized. webrtc::MutexLock lock(&mutex_); + channels_ = enable ? 2u : 1u; + const size_t total_samples = channels_ * playoutFramesIn10MS_; if (play_buffer_.size() != total_samples) { play_buffer_.SetSize(total_samples); } - audio_device_buffer_->SetPlayoutChannels(static_cast(channels_)); return 0; } @@ -478,13 +442,10 @@ namespace jni if (recording_initialized_) { return -1; } - channels_ = enable ? 2u : 1u; webrtc::MutexLock lock(&mutex_); - const size_t total_samples = channels_ * recordingFramesIn10MS_; - if (record_buffer_.size() != total_samples) { - record_buffer_.SetSize(total_samples); - } - audio_device_buffer_->SetRecordingChannels(static_cast(channels_)); + + channels_ = enable ? 2u : 1u; + return 0; } @@ -543,118 +504,77 @@ namespace jni bool HeadlessAudioDeviceModule::PlayThreadProcess() { - { - webrtc::MutexLock lock(&mutex_); - if (!playing_) { - return false; - } - } - - int64_t currentTime = webrtc::TimeMillis(); - mutex_.Lock(); - - // Seed the grid on the first tick. - if (nextPlayoutMillis_ == 0) { - nextPlayoutMillis_ = currentTime; - } - - if (currentTime >= nextPlayoutMillis_) { - mutex_.Unlock(); - audio_device_buffer_->RequestPlayoutData(playoutFramesIn10MS_); - mutex_.Lock(); - - audio_device_buffer_->GetPlayoutData(play_buffer_.data()); - - // Advance the grid by a fixed 10 ms rather than re-anchoring to currentTime, - // so wake-up latency is corrected on the next tick instead of accumulating - // into the frame period (which otherwise pulls the effective rate below 100 Hz). - nextPlayoutMillis_ += 10; - - // If we fell far behind (e.g. the thread was descheduled), resync to now - // instead of bursting frames to catch up. - if (nextPlayoutMillis_ < currentTime - 100) { - nextPlayoutMillis_ = currentTime; - } - } + const int64_t currentTime = webrtc::TimeMillis(); - int64_t sleepMillis = nextPlayoutMillis_ - webrtc::TimeMillis(); - mutex_.Unlock(); + // Decide under the state lock whether this tick pulls, and snapshot + // the format the pull needs, so the lock is not held during the pull. + bool pull = false; + size_t frames = 0; + size_t channels = 0; + uint32_t sampleRate = 0; + int64_t sleepMillis = 0; - if (sleepMillis > 0) { - webrtc::Thread::SleepMs(sleepMillis); - } - - return true; - } - - bool HeadlessAudioDeviceModule::CaptureThreadProcess() - { - webrtc::AudioTransport* callback = nullptr; { webrtc::MutexLock lock(&mutex_); - if (!recording_) { + + if (!playing_) { return false; } - callback = audio_callback_; - } - - int64_t currentTime = webrtc::TimeMillis(); - mutex_.Lock(); - - // Seed the grid on the first tick. - if (nextRecordMillis_ == 0) { - nextRecordMillis_ = currentTime; - } - if (currentTime >= nextRecordMillis_) { - size_t nSamplesOut = 0; - const size_t nBytesPerSample = sizeof(int16_t); - const size_t nChannels = channels_; - const uint32_t samplesPerSec = static_cast(sample_rate_hz_); - int64_t elapsed_time_ms = 0; - int64_t ntp_time_ms = 0; - - if (callback) { - // Pull 10 ms of audio from the registered AudioTransport (Java AudioSource). - callback->NeedMorePlayData(recordingFramesIn10MS_ * nChannels, - nBytesPerSample, - nChannels, - samplesPerSec, - record_buffer_.data(), - nSamplesOut, - &elapsed_time_ms, - &ntp_time_ms); - } - else { - nSamplesOut = recordingFramesIn10MS_ * nChannels; - std::memset(record_buffer_.data(), 0, nSamplesOut * nBytesPerSample); + // Seed the grid on the first tick. + if (nextPlayoutMillis_ == 0) { + nextPlayoutMillis_ = currentTime; } - if (nChannels > 0) { - // Feed the captured buffer to WebRTC. - audio_device_buffer_->SetRecordedBuffer(record_buffer_.data(), recordingFramesIn10MS_); - audio_device_buffer_->SetVQEData(/*play_delay_ms*/ 0, /*rec_delay_ms*/ 0); + if (currentTime >= nextPlayoutMillis_) { + pull = true; + frames = playoutFramesIn10MS_; + channels = channels_; + sampleRate = static_cast(sample_rate_hz_); // Advance the grid by a fixed 10 ms rather than re-anchoring to currentTime, // so wake-up latency is corrected on the next tick instead of accumulating // into the frame period (which otherwise pulls the effective rate below 100 Hz). - nextRecordMillis_ += 10; + nextPlayoutMillis_ += 10; // If we fell far behind (e.g. the thread was descheduled), resync to now // instead of bursting frames to catch up. - if (nextRecordMillis_ < currentTime - 100) { - nextRecordMillis_ = currentTime; + if (nextPlayoutMillis_ < currentTime - 100) { + nextPlayoutMillis_ = currentTime; } + } - mutex_.Unlock(); - audio_device_buffer_->DeliverRecordedData(); - mutex_.Lock(); + sleepMillis = nextPlayoutMillis_ - webrtc::TimeMillis(); + } + + if (pull) { + // Pull 10 ms of rendered audio straight from the transport and drop + // it; there is no device to play it on. The pull is what drives the + // receive side of every peer connection of the factory this module + // belongs to, so remote audio only reaches an AudioTrack sink while + // this thread runs. + // + // Only callback_mutex_ is held here. It keeps the transport alive + // for the duration of the call (see RegisterAudioCallback) without + // serialising the pull against the rest of this module. + webrtc::MutexLock lock(&callback_mutex_); + + if (audio_callback_) { + size_t samplesOut = 0; + int64_t elapsedTimeMillis = -1; + int64_t ntpTimeMillis = -1; + + audio_callback_->NeedMorePlayData(frames, + sizeof(int16_t) * channels, + channels, + sampleRate, + play_buffer_.data(), + samplesOut, + &elapsedTimeMillis, + &ntpTimeMillis); } } - int64_t sleepMillis = nextRecordMillis_ - webrtc::TimeMillis(); - mutex_.Unlock(); - if (sleepMillis > 0) { webrtc::Thread::SleepMs(sleepMillis); } diff --git a/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp b/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp new file mode 100644 index 00000000..314e274b --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp @@ -0,0 +1,511 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "api/ProxyAudioDeviceModule.h" + +#include "api/make_ref_counted.h" +#include "rtc_base/logging.h" + +#include +#include + +namespace jni +{ + webrtc::scoped_refptr ProxyAudioDeviceModule::Create( + webrtc::scoped_refptr delegate) + { + return webrtc::make_ref_counted(std::move(delegate)); + } + + ProxyAudioDeviceModule::ProxyAudioDeviceModule(webrtc::scoped_refptr delegate) : + delegate_(std::move(delegate)), + captureEnabled_(false), + gate_(captureEnabled_), + gateRegistered_(false) + { + } + + ProxyAudioDeviceModule::~ProxyAudioDeviceModule() + { + // WebRTC unregisters its transport before releasing the factory. Should + // the wrapped module still point at the gate, detach it so the module + // does not keep a dangling pointer once this proxy is gone. A module + // rejects this while its playout or recording is running, so warn rather + // than stop media the application started itself. + if (gateRegistered_ && delegate_->RegisterAudioCallback(nullptr) != 0) { + RTC_LOG(LS_WARNING) << "ProxyAudioDeviceModule: the AudioDeviceModule kept its audio callback. " + "Stop its playout and recording before disposing the PeerConnectionFactory."; + } + } + + void ProxyAudioDeviceModule::SetCaptureEnabled(bool enabled) + { + captureEnabled_.store(enabled, std::memory_order_release); + + RTC_LOG(LS_INFO) << "ProxyAudioDeviceModule: device capture " << (enabled ? "enabled" : "disabled"); + + if (!enabled && delegate_->Recording()) { + // A recording that is already running was started by the application + // through the Java AudioDeviceModule, which holds the wrapped module + // and does not pass through this proxy. Leave it running; its frames + // are dropped instead. WebRTC cannot have started it, because it only + // starts a recording while capture is enabled. + RTC_LOG(LS_INFO) << "ProxyAudioDeviceModule: dropping recorded audio, send streams are sink-fed"; + } + } + + bool ProxyAudioDeviceModule::CaptureEnabled() const + { + return captureEnabled_.load(std::memory_order_acquire); + } + + int32_t ProxyAudioDeviceModule::ActiveAudioLayer(AudioLayer * audioLayer) const + { + return delegate_->ActiveAudioLayer(audioLayer); + } + + int32_t ProxyAudioDeviceModule::RegisterAudioCallback(webrtc::AudioTransport * audioCallback) + { + gate_.SetTarget(audioCallback); + + int32_t result = delegate_->RegisterAudioCallback(audioCallback ? &gate_ : nullptr); + + if (result == 0) { + gateRegistered_ = audioCallback != nullptr; + } + + return result; + } + + int32_t ProxyAudioDeviceModule::Init() + { + return delegate_->Init(); + } + + int32_t ProxyAudioDeviceModule::Terminate() + { + return delegate_->Terminate(); + } + + bool ProxyAudioDeviceModule::Initialized() const + { + return delegate_->Initialized(); + } + + int16_t ProxyAudioDeviceModule::PlayoutDevices() + { + return delegate_->PlayoutDevices(); + } + + int16_t ProxyAudioDeviceModule::RecordingDevices() + { + return delegate_->RecordingDevices(); + } + + int32_t ProxyAudioDeviceModule::PlayoutDeviceName(uint16_t index, + char name[webrtc::kAdmMaxDeviceNameSize], + char guid[webrtc::kAdmMaxGuidSize]) + { + return delegate_->PlayoutDeviceName(index, name, guid); + } + + int32_t ProxyAudioDeviceModule::RecordingDeviceName(uint16_t index, + char name[webrtc::kAdmMaxDeviceNameSize], + char guid[webrtc::kAdmMaxGuidSize]) + { + return delegate_->RecordingDeviceName(index, name, guid); + } + + int32_t ProxyAudioDeviceModule::SetPlayoutDevice(uint16_t index) + { + return delegate_->SetPlayoutDevice(index); + } + + int32_t ProxyAudioDeviceModule::SetPlayoutDevice(WindowsDeviceType device) + { + return delegate_->SetPlayoutDevice(device); + } + + int32_t ProxyAudioDeviceModule::SetRecordingDevice(uint16_t index) + { + return delegate_->SetRecordingDevice(index); + } + + int32_t ProxyAudioDeviceModule::SetRecordingDevice(WindowsDeviceType device) + { + return delegate_->SetRecordingDevice(device); + } + + int32_t ProxyAudioDeviceModule::PlayoutIsAvailable(bool * available) + { + return delegate_->PlayoutIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::InitPlayout() + { + return delegate_->InitPlayout(); + } + + bool ProxyAudioDeviceModule::PlayoutIsInitialized() const + { + return delegate_->PlayoutIsInitialized(); + } + + int32_t ProxyAudioDeviceModule::RecordingIsAvailable(bool * available) + { + if (!CaptureEnabled()) { + if (!available) { + return -1; + } + *available = false; + return 0; + } + + return delegate_->RecordingIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::InitRecording() + { + if (!CaptureEnabled()) { + return -1; + } + + return delegate_->InitRecording(); + } + + bool ProxyAudioDeviceModule::RecordingIsInitialized() const + { + return delegate_->RecordingIsInitialized(); + } + + int32_t ProxyAudioDeviceModule::StartPlayout() + { + return delegate_->StartPlayout(); + } + + int32_t ProxyAudioDeviceModule::StopPlayout() + { + return delegate_->StopPlayout(); + } + + bool ProxyAudioDeviceModule::Playing() const + { + return delegate_->Playing(); + } + + int32_t ProxyAudioDeviceModule::StartRecording() + { + if (!CaptureEnabled()) { + return -1; + } + + return delegate_->StartRecording(); + } + + int32_t ProxyAudioDeviceModule::StopRecording() + { + return delegate_->StopRecording(); + } + + bool ProxyAudioDeviceModule::Recording() const + { + return delegate_->Recording(); + } + + int32_t ProxyAudioDeviceModule::InitSpeaker() + { + return delegate_->InitSpeaker(); + } + + bool ProxyAudioDeviceModule::SpeakerIsInitialized() const + { + return delegate_->SpeakerIsInitialized(); + } + + int32_t ProxyAudioDeviceModule::InitMicrophone() + { + return delegate_->InitMicrophone(); + } + + bool ProxyAudioDeviceModule::MicrophoneIsInitialized() const + { + return delegate_->MicrophoneIsInitialized(); + } + + int32_t ProxyAudioDeviceModule::SpeakerVolumeIsAvailable(bool * available) + { + return delegate_->SpeakerVolumeIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::SetSpeakerVolume(uint32_t volume) + { + return delegate_->SetSpeakerVolume(volume); + } + + int32_t ProxyAudioDeviceModule::SpeakerVolume(uint32_t * volume) const + { + return delegate_->SpeakerVolume(volume); + } + + int32_t ProxyAudioDeviceModule::MaxSpeakerVolume(uint32_t * maxVolume) const + { + return delegate_->MaxSpeakerVolume(maxVolume); + } + + int32_t ProxyAudioDeviceModule::MinSpeakerVolume(uint32_t * minVolume) const + { + return delegate_->MinSpeakerVolume(minVolume); + } + + int32_t ProxyAudioDeviceModule::MicrophoneVolumeIsAvailable(bool * available) + { + return delegate_->MicrophoneVolumeIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::SetMicrophoneVolume(uint32_t volume) + { + return delegate_->SetMicrophoneVolume(volume); + } + + int32_t ProxyAudioDeviceModule::MicrophoneVolume(uint32_t * volume) const + { + return delegate_->MicrophoneVolume(volume); + } + + int32_t ProxyAudioDeviceModule::MaxMicrophoneVolume(uint32_t * maxVolume) const + { + return delegate_->MaxMicrophoneVolume(maxVolume); + } + + int32_t ProxyAudioDeviceModule::MinMicrophoneVolume(uint32_t * minVolume) const + { + return delegate_->MinMicrophoneVolume(minVolume); + } + + int32_t ProxyAudioDeviceModule::SpeakerMuteIsAvailable(bool * available) + { + return delegate_->SpeakerMuteIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::SetSpeakerMute(bool enable) + { + return delegate_->SetSpeakerMute(enable); + } + + int32_t ProxyAudioDeviceModule::SpeakerMute(bool * enabled) const + { + return delegate_->SpeakerMute(enabled); + } + + int32_t ProxyAudioDeviceModule::MicrophoneMuteIsAvailable(bool * available) + { + return delegate_->MicrophoneMuteIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::SetMicrophoneMute(bool enable) + { + return delegate_->SetMicrophoneMute(enable); + } + + int32_t ProxyAudioDeviceModule::MicrophoneMute(bool * enabled) const + { + return delegate_->MicrophoneMute(enabled); + } + + int32_t ProxyAudioDeviceModule::StereoPlayoutIsAvailable(bool * available) const + { + return delegate_->StereoPlayoutIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::SetStereoPlayout(bool enable) + { + return delegate_->SetStereoPlayout(enable); + } + + int32_t ProxyAudioDeviceModule::StereoPlayout(bool * enabled) const + { + return delegate_->StereoPlayout(enabled); + } + + int32_t ProxyAudioDeviceModule::StereoRecordingIsAvailable(bool * available) const + { + return delegate_->StereoRecordingIsAvailable(available); + } + + int32_t ProxyAudioDeviceModule::SetStereoRecording(bool enable) + { + return delegate_->SetStereoRecording(enable); + } + + int32_t ProxyAudioDeviceModule::StereoRecording(bool * enabled) const + { + return delegate_->StereoRecording(enabled); + } + + int32_t ProxyAudioDeviceModule::PlayoutDelay(uint16_t * delayMS) const + { + return delegate_->PlayoutDelay(delayMS); + } + + bool ProxyAudioDeviceModule::BuiltInAECIsAvailable() const + { + return delegate_->BuiltInAECIsAvailable(); + } + + bool ProxyAudioDeviceModule::BuiltInAGCIsAvailable() const + { + return delegate_->BuiltInAGCIsAvailable(); + } + + bool ProxyAudioDeviceModule::BuiltInNSIsAvailable() const + { + return delegate_->BuiltInNSIsAvailable(); + } + + int32_t ProxyAudioDeviceModule::EnableBuiltInAEC(bool enable) + { + return delegate_->EnableBuiltInAEC(enable); + } + + int32_t ProxyAudioDeviceModule::EnableBuiltInAGC(bool enable) + { + return delegate_->EnableBuiltInAGC(enable); + } + + int32_t ProxyAudioDeviceModule::EnableBuiltInNS(bool enable) + { + return delegate_->EnableBuiltInNS(enable); + } + + int32_t ProxyAudioDeviceModule::GetPlayoutUnderrunCount() const + { + return delegate_->GetPlayoutUnderrunCount(); + } + + std::optional ProxyAudioDeviceModule::GetStats() const + { + return delegate_->GetStats(); + } + + // TransportGate + + ProxyAudioDeviceModule::TransportGate::TransportGate(const std::atomic & captureEnabled) : + captureEnabled_(captureEnabled), + target_(nullptr) + { + } + + void ProxyAudioDeviceModule::TransportGate::SetTarget(webrtc::AudioTransport * target) + { + target_.store(target, std::memory_order_release); + } + + int32_t ProxyAudioDeviceModule::TransportGate::RecordedDataIsAvailable(const void * audioSamples, + size_t nSamples, + size_t nBytesPerSample, + size_t nChannels, + uint32_t samplesPerSec, + uint32_t totalDelayMS, + int32_t clockDrift, + uint32_t currentMicLevel, + bool keyPressed, + uint32_t & newMicLevel) + { + return RecordedDataIsAvailable(audioSamples, nSamples, nBytesPerSample, nChannels, + samplesPerSec, totalDelayMS, clockDrift, currentMicLevel, + keyPressed, newMicLevel, std::nullopt); + } + + int32_t ProxyAudioDeviceModule::TransportGate::RecordedDataIsAvailable(const void * audioSamples, + size_t nSamples, + size_t nBytesPerSample, + size_t nChannels, + uint32_t samplesPerSec, + uint32_t totalDelayMS, + int32_t clockDrift, + uint32_t currentMicLevel, + bool keyPressed, + uint32_t & newMicLevel, + std::optional estimatedCaptureTimeNS) + { + newMicLevel = currentMicLevel; + + webrtc::AudioTransport * target = target_.load(std::memory_order_acquire); + + if (!target || !captureEnabled_.load(std::memory_order_acquire)) { + // Drop the frame. Send streams of this factory are fed by their + // tracks' sources, not by the device. + return 0; + } + + return target->RecordedDataIsAvailable(audioSamples, nSamples, nBytesPerSample, nChannels, + samplesPerSec, totalDelayMS, clockDrift, currentMicLevel, + keyPressed, newMicLevel, estimatedCaptureTimeNS); + } + + int32_t ProxyAudioDeviceModule::TransportGate::NeedMorePlayData(size_t nSamples, + size_t nBytesPerSample, + size_t nChannels, + uint32_t samplesPerSec, + void * audioSamples, + size_t & nSamplesOut, + int64_t * elapsed_time_ms, + int64_t * ntp_time_ms) + { + webrtc::AudioTransport * target = target_.load(std::memory_order_acquire); + + if (!target) { + // Set safe values for all out parameters, as the interface requires. + std::memset(audioSamples, 0, nSamples * nBytesPerSample); + nSamplesOut = nSamples; + if (elapsed_time_ms) { + *elapsed_time_ms = -1; + } + if (ntp_time_ms) { + *ntp_time_ms = -1; + } + return 0; + } + + return target->NeedMorePlayData(nSamples, nBytesPerSample, nChannels, samplesPerSec, + audioSamples, nSamplesOut, elapsed_time_ms, ntp_time_ms); + } + + void ProxyAudioDeviceModule::TransportGate::PullRenderData(int bits_per_sample, + int sample_rate, + size_t number_of_channels, + size_t number_of_frames, + void * audio_data, + int64_t * elapsed_time_ms, + int64_t * ntp_time_ms) + { + webrtc::AudioTransport * target = target_.load(std::memory_order_acquire); + + if (!target) { + std::memset(audio_data, 0, number_of_frames * number_of_channels * (bits_per_sample / 8)); + if (elapsed_time_ms) { + *elapsed_time_ms = -1; + } + if (ntp_time_ms) { + *ntp_time_ms = -1; + } + return; + } + + target->PullRenderData(bits_per_sample, sample_rate, number_of_channels, number_of_frames, + audio_data, elapsed_time_ms, ntp_time_ms); + } +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java b/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java index b02f857f..4938e0ef 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java @@ -25,15 +25,33 @@ import dev.onvoid.webrtc.media.audio.AudioProcessing; import dev.onvoid.webrtc.media.audio.AudioTrackSource; import dev.onvoid.webrtc.media.audio.AudioTrack; +import dev.onvoid.webrtc.media.audio.CustomAudioSource; import dev.onvoid.webrtc.media.video.VideoTrackSource; import dev.onvoid.webrtc.media.video.VideoTrack; import java.util.Map; +import java.util.Objects; /** * The PeerConnectionFactory is the main entry point for a WebRTC application. * It provides factory methods for {@link RTCPeerConnection} and audio/video * {@link MediaStreamTrack}s. + *

+ * A factory sends audio from exactly one kind of input. Either its + * {@link AudioDeviceModuleBase audio device module} captures the audio, which + * is what tracks created from {@link #createAudioSource(AudioOptions)} send, + * or the audio is pushed into the senders, which is what a + * {@link CustomAudioSource} and a track forwarded from a remote peer do. + * WebRTC feeds device-captured audio into every audio sender of a factory, so + * a sender that is fed both ways hits a race inside WebRTC that aborts the + * process. The factory therefore commits to the kind that is used first and + * rejects the other with an {@link IllegalStateException}. Applications that + * need both create a second factory. + *

+ * The recording device is opened only once + * {@link #createAudioSource(AudioOptions)} is called, since that is the only + * way to send device-captured audio. A factory that sends pushed audio, or no + * audio at all, never opens it. Playout of received audio is never affected. * * @author Alex Andres */ @@ -48,6 +66,19 @@ public class PeerConnectionFactory extends DisposableNativeObject { } } + /** + * The kind of input that provides the audio this factory sends. + */ + private enum AudioInputMode { + + /** The audio device module captures the audio. */ + DEVICE, + + /** The application pushes the audio through a {@link CustomAudioSource}. */ + CUSTOM + + } + @SuppressWarnings("unused") private long networkThreadHandle; @@ -58,6 +89,15 @@ public class PeerConnectionFactory extends DisposableNativeObject { @SuppressWarnings("unused") private long workerThreadHandle; + @SuppressWarnings("unused") + private long audioModuleHandle; + + /** Guards {@link #audioInputMode}. */ + private final Object audioInputLock = new Object(); + + /** Set the first time either kind of audio input is used; never reset. */ + private AudioInputMode audioInputMode; + /** * Creates an instance of PeerConnectionFactory. @@ -154,27 +194,113 @@ public PeerConnectionFactory(Map fieldTrials, } /** - * Creates an {@link AudioTrackSource}. The audio source may be used by one - * or more {@link AudioTrack}s. + * Creates an {@link AudioTrackSource} whose audio is captured by this + * factory's audio device module. The audio source may be used by one or + * more {@link AudioTrack}s. + *

+ * Calling this commits the factory to device-captured audio and lets it open + * the recording device; see the class description. * * @param options Audio options to control the audio processing. * * @return The created audio source. + * + * @throws IllegalStateException If this factory already sends audio from a + * {@link CustomAudioSource}. */ - public native AudioTrackSource createAudioSource(AudioOptions options); + public AudioTrackSource createAudioSource(AudioOptions options) { + Objects.requireNonNull(options, "AudioOptions is null"); + + requireAudioInputMode(AudioInputMode.DEVICE); + + return createAudioSourceInternal(options); + } /** * Creates an new {@link AudioTrack}. The audio track can be added to the * {@link RTCPeerConnection} using the {@link RTCPeerConnection#addTrack * addTrack} or {@link RTCPeerConnection#addTransceiver addTransceiver} * methods. + *

+ * Passing a {@link CustomAudioSource} commits the factory to pushed audio; + * see the class description. * * @param label The identifier string of the audio track. * @param source The audio source that provides audio data. * * @return The created audio track. + * + * @throws IllegalStateException If the source is a {@link CustomAudioSource} + * and this factory already sends audio + * captured by its audio device module. + */ + public AudioTrack createAudioTrack(String label, AudioTrackSource source) { + Objects.requireNonNull(label, "Audio track label is null"); + Objects.requireNonNull(source, "AudioTrackSource is null"); + + if (source instanceof CustomAudioSource) { + requireAudioInputMode(AudioInputMode.CUSTOM); + } + + return createAudioTrackInternal(label, source); + } + + /** + * Commits this factory to the given kind of audio input, or verifies that + * it is already committed to it. Committing to {@link AudioInputMode#CUSTOM} + * switches off capture from the audio device module. + * + * @param mode The kind of audio input about to be used. + * + * @throws IllegalStateException If the factory is committed to the other + * kind. + */ + private void requireAudioInputMode(AudioInputMode mode) { + synchronized (audioInputLock) { + if (audioInputMode == mode) { + return; + } + if (audioInputMode == AudioInputMode.CUSTOM) { + throw new IllegalStateException("This PeerConnectionFactory already sends audio pushed through a " + + "CustomAudioSource. WebRTC feeds the audio captured by a factory's AudioDeviceModule into " + + "every audio sender of that factory, so device-captured and custom audio sources cannot be " + + "combined in one factory. Create a separate PeerConnectionFactory for device-captured audio."); + } + if (audioInputMode == AudioInputMode.DEVICE) { + throw new IllegalStateException("This PeerConnectionFactory already sends audio captured by its " + + "AudioDeviceModule. WebRTC feeds that audio into every audio sender of the factory, so " + + "device-captured and custom audio sources cannot be combined in one factory. Create a " + + "separate PeerConnectionFactory for CustomAudioSource tracks."); + } + + // The recording device stays closed until the application asks for + // device-captured audio, which is what keeps captured frames away + // from senders fed by a custom source or by a forwarded remote track. + setDeviceCaptureEnabled(mode == AudioInputMode.DEVICE); + + audioInputMode = mode; + } + } + + /** + * Commits this factory to pushed audio if the given track is an audio track + * whose audio is pushed into its sender rather than captured by the audio + * device module. Tracks backed by a {@link CustomAudioSource} and tracks + * received from a remote peer and forwarded on are of that kind. Called + * before a track becomes a sender of one of this factory's peer + * connections; any other track is ignored. + * + * @param track The track about to be sent, may be {@code null}. + * + * @throws IllegalStateException If the track is of that kind and this + * factory already sends audio captured by its + * audio device module. */ - public native AudioTrack createAudioTrack(String label, AudioTrackSource source); + void commitAudioInput(MediaStreamTrack track) { + if (track != null && isSinkFedAudioTrack(track)) { + requireAudioInputMode(AudioInputMode.CUSTOM); + } + } /** * Creates a new {@link VideoTrack}. The video track can be added to the @@ -198,8 +324,19 @@ public PeerConnectionFactory(Map fieldTrials, * * @return The created peer connection. */ - public native RTCPeerConnection createPeerConnection( - RTCConfiguration config, PeerConnectionObserver observer); + public RTCPeerConnection createPeerConnection(RTCConfiguration config, + PeerConnectionObserver observer) { + RTCPeerConnection peerConnection = createPeerConnectionInternal(config, + observer); + + if (peerConnection != null) { + // The connection reports back which kind of audio its senders are + // about to send, so this factory can reject a mix of both kinds. + peerConnection.setFactory(this); + } + + return peerConnection; + } /** * Returns the capabilities of the system for receiving media of the given @@ -229,4 +366,32 @@ public native RTCPeerConnection createPeerConnection( private native void initialize(Map fieldTrials, AudioDeviceModuleBase audioModule, AudioProcessing audioProcessing); + private native AudioTrackSource createAudioSourceInternal(AudioOptions options); + + private native AudioTrack createAudioTrackInternal(String label, AudioTrackSource source); + + private native RTCPeerConnection createPeerConnectionInternal( + RTCConfiguration config, PeerConnectionObserver observer); + + /** + * Returns whether the given track is an audio track whose audio reaches a + * sender through the track's sinks instead of through the audio device + * module. That is the case for a {@link CustomAudioSource} and for a track + * received from a remote peer. + * + * @param track The track to classify, must not be {@code null}. + * + * @return True if the track is audio and its source feeds sinks. + */ + private native boolean isSinkFedAudioTrack(MediaStreamTrack track); + + /** + * Enables or disables the capture path of the audio device module WebRTC + * uses for this factory. While disabled, WebRTC cannot start a recording + * and recorded audio never reaches the factory's audio senders. + * + * @param enabled Whether device-captured audio may reach the audio senders. + */ + private native void setDeviceCaptureEnabled(boolean enabled); + } diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java index 80c2af10..e9a81eee 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java @@ -16,6 +16,9 @@ package dev.onvoid.webrtc; +import static java.util.Objects.isNull; +import static java.util.Objects.nonNull; + import dev.onvoid.webrtc.internal.NativeObject; import dev.onvoid.webrtc.media.MediaStreamTrack; @@ -37,6 +40,12 @@ public class RTCPeerConnection extends NativeObject { @SuppressWarnings("unused") private long observerHandle; + /** + * The factory that created this connection. Set right after construction; + * null only if a connection was obtained some other way. + */ + private PeerConnectionFactory factory; + /** * Constructor used by the native api. @@ -45,6 +54,16 @@ private RTCPeerConnection() { } + /** + * Sets the factory that created this connection, so that tracks added to it + * can be reported back to that factory. + * + * @param factory The creating factory. + */ + void setFactory(PeerConnectionFactory factory) { + this.factory = factory; + } + /** * Returns an array of {@link RTCRtpSender} objects representing the RTP * senders that belong to non-stopped {@link RTCRtpTransceiver} objects @@ -81,9 +100,21 @@ private RTCPeerConnection() { * to. * * @return The RTCRtpSender which will be used to transmit the media data. - */ - public native RTCRtpSender addTrack(MediaStreamTrack track, - List streamIds); + * + * @throws IllegalStateException If the track is an audio track that pushes + * its audio, for example one received from a + * remote peer, while the creating factory + * already sends audio captured by its audio + * device module. See {@link + * PeerConnectionFactory} for why a factory + * sends only one kind of audio input. + */ + public RTCRtpSender addTrack(MediaStreamTrack track, + List streamIds) { + commitAudioInput(track); + + return addTrackInternal(track, streamIds); + } /** * Stops sending media from sender. The RTCRtpSender will still appear in @@ -107,9 +138,41 @@ public native RTCRtpSender addTrack(MediaStreamTrack track, * * @return The RTCRtpTransceiver which will be used to transmit and receive * the media data. + * + * @throws IllegalStateException If the transceiver sends and the track is an + * audio track that pushes its audio, for + * example one received from a remote peer, + * while the creating factory already sends + * audio captured by its audio device module. + * See {@link PeerConnectionFactory} for why a + * factory sends only one kind of audio input. + */ + public RTCRtpTransceiver addTransceiver(MediaStreamTrack track, + RTCRtpTransceiverInit init) { + // A transceiver that only receives never sends the track's audio. Its + // direction can be changed later through RTCRtpTransceiver.setDirection, + // which is not covered here. + if (isNull(init) || isNull(init.direction) + || init.direction == RTCRtpTransceiverDirection.SEND_RECV + || init.direction == RTCRtpTransceiverDirection.SEND_ONLY) { + commitAudioInput(track); + } + + return addTransceiverInternal(track, init); + } + + /** + * Reports a track that is about to be sent to the factory that created this + * connection, which rejects a track whose audio is pushed while the factory + * already sends audio captured by its audio device module. + * + * @param track The track about to be sent, may be {@code null}. */ - public native RTCRtpTransceiver addTransceiver(MediaStreamTrack track, - RTCRtpTransceiverInit init); + private void commitAudioInput(MediaStreamTrack track) { + if (nonNull(factory)) { + factory.commitAudioInput(track); + } + } /** * Creates a new RTCDataChannel object with the given label. The @@ -356,4 +419,10 @@ public int hashCode() { return handle == 0 ? System.identityHashCode(this) : Long.hashCode(handle); } + private native RTCRtpSender addTrackInternal(MediaStreamTrack track, + List streamIds); + + private native RTCRtpTransceiver addTransceiverInternal( + MediaStreamTrack track, RTCRtpTransceiverInit init); + } diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/CustomAudioSource.java b/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/CustomAudioSource.java index fdb3a8d5..b4ab39df 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/CustomAudioSource.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/CustomAudioSource.java @@ -16,6 +16,8 @@ package dev.onvoid.webrtc.media.audio; +import static java.util.Objects.requireNonNull; + import dev.onvoid.webrtc.media.SyncClock; /** @@ -26,6 +28,23 @@ */ public class CustomAudioSource extends AudioTrackSource { + /** The only sample format WebRTC accepts from a source. */ + private static final int BITS_PER_SAMPLE = 16; + + /** + * The most samples, counting every channel, that one pushed chunk may hold. + * WebRTC copies a chunk into a fixed-size audio frame, so this mirrors + * {@code AudioFrame::kMaxDataSizeSamples} in the native library. + */ + public static final int MAX_SAMPLES_PER_PUSH = 7680; + + /** + * The most channels one pushed chunk may hold. Mirrors + * {@code kMaxNumberOfAudioChannels} in the native library. + */ + public static final int MAX_CHANNELS = 24; + + /** * Constructs a new CustomAudioSource instance. */ @@ -47,16 +66,78 @@ public CustomAudioSource(SyncClock clock) { } /** - * Pushes audio data to be processed by this audio source. + * Pushes audio data to be processed by this audio source. The data is + * handed to the source's sinks on the calling thread, so it must be pushed + * from one thread at a time. + *

+ * The samples must be 16-bit signed PCM in the platform byte order, with + * the channels interleaved. A call carries one chunk of audio, for which + * 10 ms is the size WebRTC works with; at most + * {@value #MAX_SAMPLES_PER_PUSH} samples, counting every channel, fit in + * one chunk. * - * @param audioData The raw audio data bytes to process. - * @param bits_per_sample The number of bits per sample (e.g., 8, 16, 32). + * @param audioData The raw audio data bytes to process. Must hold at + * least {@code frameCount * channels * 2} bytes. + * @param bits_per_sample The number of bits per sample, which must be 16. * @param sampleRate The sample rate of the audio in Hz (e.g., 44100, 48000). * @param channels The number of audio channels (1 for mono, 2 for stereo). - * @param frameCount The number of frames in the provided audio data. + * @param frameCount The number of frames in the provided audio data. A + * frame holds one sample for each channel, so 10 ms + * at 48 kHz is 480 frames whatever the channel count. + * + * @throws NullPointerException If the audio data is {@code null}. + * @throws IllegalArgumentException If the audio format is not 16-bit PCM, + * if a value is not positive, if the chunk + * is larger than WebRTC can take, or if the + * array is too short for the frames it is + * said to hold. */ - public native void pushAudio(byte[] audioData, int bits_per_sample, - int sampleRate, int channels, int frameCount); + public void pushAudio(byte[] audioData, int bits_per_sample, int sampleRate, + int channels, int frameCount) { + requireNonNull(audioData, "audioData must not be null"); + + if (bits_per_sample != BITS_PER_SAMPLE) { + throw new IllegalArgumentException(String.format( + "Audio must be %d-bit PCM, got %d bits per sample", + BITS_PER_SAMPLE, bits_per_sample)); + } + if (sampleRate <= 0) { + throw new IllegalArgumentException( + "Sample rate must be positive, got " + sampleRate); + } + if (channels <= 0 || channels > MAX_CHANNELS) { + throw new IllegalArgumentException(String.format( + "Channel count must be between 1 and %d, got %d", + MAX_CHANNELS, channels)); + } + if (frameCount <= 0) { + throw new IllegalArgumentException( + "Frame count must be positive, got " + frameCount); + } + + // WebRTC copies the chunk into a fixed-size audio frame and aborts the + // process if it does not fit, so reject an oversized chunk here. + long samples = (long) frameCount * channels; + + if (samples > MAX_SAMPLES_PER_PUSH) { + throw new IllegalArgumentException(String.format( + "A chunk holds at most %d samples across all channels, got %d " + + "(%d frames x %d channels). Push shorter chunks, 10 ms each.", + MAX_SAMPLES_PER_PUSH, samples, frameCount, channels)); + } + + // Native code reads this many bytes out of the array, whatever its size. + long required = samples * (BITS_PER_SAMPLE / 8); + + if (audioData.length < required) { + throw new IllegalArgumentException(String.format( + "Audio data holds %d bytes, but %d frames of %d channels need %d", + audioData.length, frameCount, channels, required)); + } + + pushAudioInternal(audioData, bits_per_sample, sampleRate, channels, + frameCount); + } /** * Disposes of any native resources held by this audio source. @@ -65,6 +146,19 @@ public native void pushAudio(byte[] audioData, int bits_per_sample, */ public native void dispose(); + /** + * Hands the validated audio data to the native source. + * + * @param audioData The raw audio data bytes to process. + * @param bits_per_sample The number of bits per sample. + * @param sampleRate The sample rate of the audio in Hz. + * @param channels The number of audio channels. + * @param frameCount The number of frames in the provided audio data. + */ + private native void pushAudioInternal(byte[] audioData, int bits_per_sample, + int sampleRate, int channels, + int frameCount); + /** * Initializes the native resources required by this audio source. */ diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java b/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java index 467c89ca..62ab2008 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java @@ -153,6 +153,198 @@ void createAudioTrack() { assertTrue(audioTrack.isEnabled()); } + @Test + void customAudioSourceAfterDeviceAudioSourceIsRejected() { + // WebRTC feeds device-captured audio into every audio sender of a + // factory. A CustomAudioSource track would be fed twice, which aborts + // the process inside WebRTC, so the factory must refuse it up front. + AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); + CustomAudioSource customSource = new CustomAudioSource(); + + try { + AudioTrackSource deviceSource = audioFactory.createAudioSource(new AudioOptions()); + + IllegalStateException e = assertThrows(IllegalStateException.class, () -> { + audioFactory.createAudioTrack("customTrack", customSource); + }); + assertTrue(e.getMessage().contains("CustomAudioSource"), e.getMessage()); + + // The device path stays usable. + AudioTrack deviceTrack = audioFactory.createAudioTrack("deviceTrack", deviceSource); + assertNotNull(deviceTrack); + + deviceTrack.dispose(); + deviceSource.dispose(); + } + finally { + customSource.dispose(); + audioFactory.dispose(); + audioModule.dispose(); + } + } + + @Test + void deviceAudioSourceAfterCustomAudioSourceIsRejected() { + AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); + CustomAudioSource customSource = new CustomAudioSource(); + CustomAudioSource otherCustomSource = new CustomAudioSource(); + + try { + AudioTrack customTrack = audioFactory.createAudioTrack("customTrack", customSource); + assertNotNull(customTrack); + + IllegalStateException e = assertThrows(IllegalStateException.class, () -> { + audioFactory.createAudioSource(new AudioOptions()); + }); + assertTrue(e.getMessage().contains("AudioDeviceModule"), e.getMessage()); + + // Any number of custom sources may share the factory. + AudioTrack otherCustomTrack = audioFactory.createAudioTrack("otherCustomTrack", otherCustomSource); + assertNotNull(otherCustomTrack); + + otherCustomTrack.dispose(); + customTrack.dispose(); + } + finally { + otherCustomSource.dispose(); + customSource.dispose(); + audioFactory.dispose(); + audioModule.dispose(); + } + } + + @Test + void sinkFedTrackAddedToDeviceAudioFactoryIsRejected() { + // A track whose audio is pushed rather than captured, here one backed by + // a CustomAudioSource but equally one forwarded from a remote peer, may + // not become a sender of a factory that sends device-captured audio. + AudioDeviceModule deviceModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + AudioDeviceModule customModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + PeerConnectionFactory deviceFactory = new PeerConnectionFactory(deviceModule); + PeerConnectionFactory customFactory = new PeerConnectionFactory(customModule); + CustomAudioSource customSource = new CustomAudioSource(); + + try { + AudioTrackSource deviceSource = deviceFactory.createAudioSource(new AudioOptions()); + AudioTrack deviceTrack = deviceFactory.createAudioTrack("deviceTrack", deviceSource); + AudioTrack customTrack = customFactory.createAudioTrack("customTrack", customSource); + + RTCPeerConnection peerConnection = deviceFactory.createPeerConnection( + new RTCConfiguration(), candidate -> { }); + + // The device-captured track is what this factory sends. + RTCRtpSender sender = peerConnection.addTrack(deviceTrack, + Collections.singletonList("stream0")); + + assertNotNull(sender); + + IllegalStateException e = assertThrows(IllegalStateException.class, () -> { + peerConnection.addTrack(customTrack, Collections.singletonList("stream1")); + }); + assertTrue(e.getMessage().contains("AudioDeviceModule"), e.getMessage()); + + // A transceiver that sends is rejected for the same reason. + RTCRtpTransceiverInit sendRecv = new RTCRtpTransceiverInit(); + + assertThrows(IllegalStateException.class, () -> { + peerConnection.addTransceiver(customTrack, sendRecv); + }); + + // A receive-only transceiver never sends the track, so it is allowed. + RTCRtpTransceiverInit recvOnly = new RTCRtpTransceiverInit(); + recvOnly.direction = RTCRtpTransceiverDirection.RECV_ONLY; + + RTCRtpTransceiver transceiver = peerConnection.addTransceiver(customTrack, recvOnly); + + assertNotNull(transceiver); + + transceiver.dispose(); + sender.dispose(); + peerConnection.close(); + deviceTrack.dispose(); + customTrack.dispose(); + deviceSource.dispose(); + } + finally { + customSource.dispose(); + customFactory.dispose(); + deviceFactory.dispose(); + customModule.dispose(); + deviceModule.dispose(); + } + } + + @Test + void sinkFedTrackAddedFirstCommitsFactoryToPushedAudio() { + // Adding a track whose audio is pushed settles the question for the + // factory that owns the connection, even though that factory did not + // create the track. A forwarded remote track reaches a factory this way. + AudioDeviceModule receivingModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + AudioDeviceModule sendingModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + PeerConnectionFactory receivingFactory = new PeerConnectionFactory(receivingModule); + PeerConnectionFactory sendingFactory = new PeerConnectionFactory(sendingModule); + CustomAudioSource customSource = new CustomAudioSource(); + + try { + AudioTrack customTrack = sendingFactory.createAudioTrack("customTrack", customSource); + + RTCPeerConnection peerConnection = receivingFactory.createPeerConnection( + new RTCConfiguration(), candidate -> { }); + + RTCRtpSender sender = peerConnection.addTrack(customTrack, + Collections.singletonList("stream0")); + + assertNotNull(sender); + + // The receiving factory now sends pushed audio, so its own device + // capture is off limits. + assertThrows(IllegalStateException.class, () -> { + receivingFactory.createAudioSource(new AudioOptions()); + }); + + sender.dispose(); + peerConnection.close(); + customTrack.dispose(); + } + finally { + customSource.dispose(); + sendingFactory.dispose(); + receivingFactory.dispose(); + sendingModule.dispose(); + receivingModule.dispose(); + } + } + + @Test + void customAudioTrackNullParamsDoNotCommitFactory() { + AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); + PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); + CustomAudioSource customSource = new CustomAudioSource(); + + try { + assertThrows(NullPointerException.class, () -> { + audioFactory.createAudioTrack(null, customSource); + }); + assertThrows(NullPointerException.class, () -> { + audioFactory.createAudioSource(null); + }); + + // The rejected custom-track call must not have committed the factory + // to pushed audio, so a device-captured source is still allowed. + AudioTrackSource deviceSource = audioFactory.createAudioSource(new AudioOptions()); + assertNotNull(deviceSource); + + deviceSource.dispose(); + } + finally { + customSource.dispose(); + audioFactory.dispose(); + audioModule.dispose(); + } + } + @Test void createVideoTrackNullParams() { assertThrows(NullPointerException.class, () -> { diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/CustomAudioSourceTest.java b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/CustomAudioSourceTest.java index 1a0033a9..04497e9c 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/CustomAudioSourceTest.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/CustomAudioSourceTest.java @@ -44,6 +44,65 @@ void dispose() { } + @Test + void pushAudioRejectsBadArguments() { + byte[] data = new byte[480 * 2 * 2]; + + assertThrows(NullPointerException.class, + () -> customAudioSource.pushAudio(null, 16, 48000, 2, 480)); + + // WebRTC reads the samples as 16-bit PCM whatever is declared here, so + // any other width would be read as the wrong number of bytes. + assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 8, 48000, 2, 480)); + assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 32, 48000, 2, 480)); + + assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 16, 0, 2, 480)); + assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 16, 48000, 0, 480)); + assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 16, 48000, 2, 0)); + assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 16, 48000, -1, 480)); + } + + @Test + void pushAudioRejectsArrayShorterThanTheFramesItClaims() { + // Native code reads frameCount * channels * 2 bytes out of the array. + // Without this check a short array is read past its end. + byte[] data = new byte[480 * 2 * 2]; + + IllegalArgumentException e = assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 16, 48000, 2, 481)); + + assertTrue(e.getMessage().contains("1924"), e.getMessage()); + + // Exactly the required size is fine, and so is a longer array. + customAudioSource.pushAudio(data, 16, 48000, 2, 480); + customAudioSource.pushAudio(new byte[8192], 16, 48000, 2, 480); + } + + @Test + void pushAudioRejectsChunkLargerThanWebRtcTakes() { + // WebRTC copies a chunk into a fixed-size frame and aborts the process + // when it does not fit, so an oversized chunk must not reach it. + int frames = CustomAudioSource.MAX_SAMPLES_PER_PUSH / 2 + 1; + byte[] data = new byte[frames * 2 * 2]; + + IllegalArgumentException e = assertThrows(IllegalArgumentException.class, + () -> customAudioSource.pushAudio(data, 16, 48000, 2, frames)); + + assertTrue(e.getMessage().contains(String.valueOf( + CustomAudioSource.MAX_SAMPLES_PER_PUSH)), e.getMessage()); + + // The largest chunk that still fits is accepted. + int maxFrames = CustomAudioSource.MAX_SAMPLES_PER_PUSH / 2; + + customAudioSource.pushAudio(new byte[maxFrames * 2 * 2], 16, 48000, 2, maxFrames); + } + @Test void stateAfterCreation() { assertEquals(MediaSource.State.LIVE, customAudioSource.getState()); @@ -116,10 +175,12 @@ void concurrentAddRemoveSinkDoesNotCrash() throws InterruptedException { @Test void pushAudioWithDifferentFormats() { - testAudioFormat(8, 8000, 1, 80); // 8-bit, 8kHz, mono, 10ms - testAudioFormat(16, 16000, 1, 160); // 16-bit, 16kHz, mono, 10ms - testAudioFormat(16, 44100, 2, 441); // 16-bit, 44.1kHz, stereo, 10ms - testAudioFormat(16, 48000, 2, 480); // 16-bit, 48kHz, stereo, 10ms + // Every rate and channel count is passed through unchanged. Only 16-bit + // samples are accepted, since that is what WebRTC reads. + testAudioFormat(16, 8000, 1, 80); // 8kHz, mono, 10ms + testAudioFormat(16, 16000, 1, 160); // 16kHz, mono, 10ms + testAudioFormat(16, 44100, 2, 441); // 44.1kHz, stereo, 10ms + testAudioFormat(16, 48000, 2, 480); // 48kHz, stereo, 10ms } @Test diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/HeadlessADMIntegrationTest.java b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/HeadlessADMIntegrationTest.java index eb3b1a1a..cbf2b671 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/HeadlessADMIntegrationTest.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/HeadlessADMIntegrationTest.java @@ -39,10 +39,17 @@ void audioReceivedOnSink() throws Exception { HeadlessAudioDeviceModule adm = new HeadlessAudioDeviceModule(); PeerConnectionFactory factory = new PeerConnectionFactory(adm); - // Ensure the playout pipeline is started (headless output). + // Playout starts before either peer connection exists, which is the + // order the guide gives. WebRTC hands the module its audio transport + // only when it builds its voice engine, which happens later, when the + // first peer connection is created. The module's render thread is what + // pulls the receive side of both connections, so if it stopped pulling + // over that ordering, no remote audio would reach a sink below. adm.initPlayout(); adm.startPlayout(); + // Recording is a state on the module and feeds nothing, so it must + // neither deliver audio nor keep remote audio from arriving. adm.initRecording(); adm.startRecording(); @@ -120,7 +127,10 @@ public void onConnectionChange(RTCPeerConnectionState state) { receiverPcRef.set(receiverPc); // Add an explicit receive-only audio transceiver on the receiver side. - AudioTrackSource rxSource = factory.createAudioSource(new AudioOptions()); + // Its track never sends, so the source kind does not matter; a custom + // source keeps this factory on pushed audio, which is what the sender + // below uses. A factory sends audio from only one kind of input. + CustomAudioSource rxSource = new CustomAudioSource(); AudioTrack receiverTrack = factory.createAudioTrack("rx-audio", rxSource); RTCRtpTransceiverInit recvOnlyInit = new RTCRtpTransceiverInit(); recvOnlyInit.direction = RTCRtpTransceiverDirection.RECV_ONLY; @@ -171,7 +181,13 @@ public void onConnectionChange(RTCPeerConnectionState state) { assertTrue(sinkAddedLatch.await(5, TimeUnit.SECONDS), "Audio sink was not added in time"); - for (int i = 0; i < 10; i++) { // ~100ms of audio + // The module's recording is running, so its capture thread delivers + // frames at 100 Hz while this thread pushes at the same rate. The + // factory must keep the captured frames away from the send stream; + // otherwise both threads enter AudioSendStream::SendAudioData and + // WebRTC aborts the process (issue #217). Two seconds give the race + // ample opportunity to show up. + for (int i = 0; i < 200; i++) { customSource.pushAudio(silence, bitsPerSample, sampleRate, channels, frameCount); Thread.sleep(10); } @@ -196,6 +212,7 @@ public void onConnectionChange(RTCPeerConnectionState state) { // senderTrack.dispose(); // receiverTrack.dispose(); customSource.dispose(); + rxSource.dispose(); factory.dispose();