Papers

Field

Top tools here

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-video generation for interactive systems2 papers raise it · 2 address it
Interactive world models and applications need low-latency, streaming-capable audio-video synthesis to respond to user input in real time.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation and generation1 paper raises it · 1 addresses it
Generating or separating individual sound sources from a mix is needed for dynamic scene rendering and editing, but most models output only a single mixed audio track.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Evaluation of complex audio-video consistency1 paper raises it · 1 addresses it
Existing metrics fail to assess multi-event, multi-layer, or asymmetric cross-modal relations in generated scenes, limiting progress in joint generation.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Scalable training for high-resolution joint video-audio models1 paper raises it · 1 addresses it
Full attention becomes computationally infeasible at higher resolutions, and sparse attention methods often ignore cross-modal interactions in joint data.
- Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Trainingraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

LongeyeSeries A · Oct 21$20M
VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M

Recently funded

Topics in this field

text speech · automatic speech · speech models274 papers42 this week
audio language models · language models audio · large audio language76 papers16 this week
visual speech · speech gesture generation · audio driven78 papers15 this week
duplex speech models · duplex spoken language · duplex speech79 papers12 this week
spatial audio visual · audio visual dataset · audiovisual69 papers12 this week
music models · text music generation · music annotation47 papers6 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CLSep 24
Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition
Asmee Mishra, Mengjie Qian, Brechtje Post, Kate Knill
Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA…
PDF arXiv
arXiv cs.CLSep 24
Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languages
Stephen E. Moore, Akwasi Asare, Mich-Seth Owusu, Paul Azunre, Joel Budu +1
This paper presents an end-to-end study of automatic speech recognition (ASR) for adolescent health communication in three Ghanaian languages (Twi, Dagbani, and Ewe). The work proceeds in three connected stages; First, we benchmark five ASR systems (three language-specific Wav2Vec2 models and two multimodal LLMs, Gemma 3n and Gemma 4) on a general-domain Bible corpus and a Youth Adolescent Sexual and Reproductive Health (ASRH) Domain ASR dataset, using Character and Word Error Rate (CER, WER). Second, guided by the benchmark, we perform supervised domain adaptation: although Gemma 4 was the strongest zero-shot candidate, fine-tuning it proved computationally infeasible, so we pivoted to the compact Qwen3-ASR-0.6B, fine-tuned on a large Ghana Bible corpus (~90k samples) and evaluated strictly on held-out human-collected in-domain audio. Fine-tuning reduced WER on every language, most dramatically for Ewe (WER from 109.3% to 64.8%, a drop of 44.5 pp; CER from 65.1% to 24.9%). Third, we validate the work through KasaHealth, a live voice-first ASRH application deployed in all three languages, complemented by Senti-Check, a technical evaluation harness. KasaHealth was tested by 50…
PDF arXiv
arXiv eess.ASSep 24
Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion
Nhat-Nam Nguyen, Pierre-Andre Vuissoz, Yves Laprie
Cross-speaker acoustic-to-articulatory inversion requires accounting for anatomical differences between speakers. We propose a geometric adaptation framework that uses anatomical landmarks, primarily on vertebrae and dental structures,to transfer predictions from a fixed inversion model to unseen speakers. An affine transformation followed by thin-plate spline (TPS) deformation maps the predicted contours of 10 vocal-tract structures into each target speaker's geometry without retraining. Landmarks are identified in one selected /u/ frame per speaker as a common phonetic reference without assuming identical articulatory configurations across speakers, and the resulting mapping is reused across recordings. We train the model on a single-speaker rt-MRI database and evaluate adaptation on eight speakers from a separate multi-speaker rt-MRI database. We compare affine and TPS configurations using 12 or 14 landmarks. Affine12+TPS14 achieves the lowest mean point-to-closest-point error of 3.19mm. These results support the combined value of anatomical landmark information and nonrigid alignment.
PDF arXiv
arXiv cs.SDSep 24
Estimation of Room Impulse Responses from Handclaps
Shih-Yu Lai, Kyung Yun Lee, Nils Meyer-Kahlen, Eloi Moliner, Bing-Yu Chen +1
Handclaps provide an equipment-free excitation for room acoustics, but their unknown and variable source waveform makes room impulse response (RIR) estimation challenging. In this work, we investigate whether RIRs can be estimated directly from handclaps. To this end, we introduce an anechoic handclap dataset containing 2,540 claps from 17 participants, designed to capture variability across natural claps and different hand configurations. We first establish the performance attainable when the excitation clap is known using regularized deconvolution, and show that approximating the unknown excitation by windowing the direct sound from the reverberant recording is insufficient. To estimate the RIR without a known excitation, we propose using the anechoic handclap recordings to train a deep neural network with a supervised regression objective. Evaluated on a controlled synthetic benchmark, the proposed neural regressor significantly outperforms windowing-based baselines across all instrumental metrics. Furthermore, we test the proposed method on handclap recordings measured in real acoustic spaces, showing that the inferred RIR spectra are consistent across different handclap…
PDF arXiv
arXiv cs.CLSep 24
YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech
William Chen, Shinnosuke Takamichi, Sayaka Shiota, Satoru Fukayama, Samuele Cornell +1
We present YODAS v3, a weakly-labeled speech corpus containing over 1.1 million hours of 48kHz multi-channel audio in 147 languages, released under a CC BY 3.0 license. YODAS v3 is not only the largest open speech dataset to date, but also the first truly large-scale speech corpus with high-fidelity stereo audio. We first provide the collection methodology for the corpus, where we introduce new techniques for gathering language-balanced speech data. The effectiveness of our approach is shown by the language distribution of the crawled data: 22 languages in YODAS v3 have over 10K hours and 73 languages have over 5K hours of data. We then conduct extensive analyses on the composition of the data, such as the distribution of languages, audio quality, and transcription quality. Finally, we train baseline speech recognition and neural codec models to show the effectiveness of the dataset. Download at https://huggingface.co/datasets/espnet/yodas3.
PDF arXiv
arXiv cs.CVSep 24
Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition
Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh
Head-pose variation introduces substantial appearance transformations in visual speech recognition (VSR), making pose-aware feature modulation desirable. However, performance degradation and unwanted feature interactions may result from using numerous Feature-wise Linear Modulation (FiLM) circuits with fixed modulation intensity. We propose a Pose Adaptive Dynamic FiLM framework with a Dynamic Residual FiLM (DR-FiLM) modulator that predicts input-dependent weights to adaptively control the strength of pose-conditioned modulation. Experiments on LRS2 and LRS3 demonstrate that unweighted multi-pathway modulation substantially degrades phoneme recognition, increasing PER to 20.33% and 29.42%, respectively, compared with 16.20% and 20.96% for the single ResFiLM configuration. In contrast, the proposed DR-FiLM with dynamic Deep-Res weighting reduces PER to 15.74% on LRS2 and 23.91% on LRS3, substantially mitigating the adverse effects of unweighted modulation. The analysis of the learned weights further reveals a consistent tendency to assign greater weight to the deeper FiLM pathway as head-pose variation increases. These results show that merging pose-conditioned FiLM circuits is…
PDF arXiv

Papers

Topics in this field

text speech · automatic speech · speech models274 papers42 this week
audio language models · language models audio · large audio language76 papers16 this week
visual speech · speech gesture generation · audio driven78 papers15 this week
duplex speech models · duplex spoken language · duplex speech79 papers12 this week
spatial audio visual · audio visual dataset · audiovisual69 papers12 this week
music models · text music generation · music annotation47 papers6 this week

Active contributors · 90 days

Recently funded

Market

LongeyeSeries A · Oct 21$20M
VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M

Activity · 12 weeks

Recent papers

All papers in Research

arXiv cs.CLSep 24
Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition
Asmee Mishra, Mengjie Qian, Brechtje Post, Kate Knill
Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA…
PDF arXiv
arXiv cs.CLSep 24
Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languages
Stephen E. Moore, Akwasi Asare, Mich-Seth Owusu, Paul Azunre, Joel Budu +1
This paper presents an end-to-end study of automatic speech recognition (ASR) for adolescent health communication in three Ghanaian languages (Twi, Dagbani, and Ewe). The work proceeds in three connected stages; First, we benchmark five ASR systems (three language-specific Wav2Vec2 models and two multimodal LLMs, Gemma 3n and Gemma 4) on a general-domain Bible corpus and a Youth Adolescent Sexual and Reproductive Health (ASRH) Domain ASR dataset, using Character and Word Error Rate (CER, WER). Second, guided by the benchmark, we perform supervised domain adaptation: although Gemma 4 was the strongest zero-shot candidate, fine-tuning it proved computationally infeasible, so we pivoted to the compact Qwen3-ASR-0.6B, fine-tuned on a large Ghana Bible corpus (~90k samples) and evaluated strictly on held-out human-collected in-domain audio. Fine-tuning reduced WER on every language, most dramatically for Ewe (WER from 109.3% to 64.8%, a drop of 44.5 pp; CER from 65.1% to 24.9%). Third, we validate the work through KasaHealth, a live voice-first ASRH application deployed in all three languages, complemented by Senti-Check, a technical evaluation harness. KasaHealth was tested by 50…
PDF arXiv
arXiv eess.ASSep 24
Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion
Nhat-Nam Nguyen, Pierre-Andre Vuissoz, Yves Laprie
Cross-speaker acoustic-to-articulatory inversion requires accounting for anatomical differences between speakers. We propose a geometric adaptation framework that uses anatomical landmarks, primarily on vertebrae and dental structures,to transfer predictions from a fixed inversion model to unseen speakers. An affine transformation followed by thin-plate spline (TPS) deformation maps the predicted contours of 10 vocal-tract structures into each target speaker's geometry without retraining. Landmarks are identified in one selected /u/ frame per speaker as a common phonetic reference without assuming identical articulatory configurations across speakers, and the resulting mapping is reused across recordings. We train the model on a single-speaker rt-MRI database and evaluate adaptation on eight speakers from a separate multi-speaker rt-MRI database. We compare affine and TPS configurations using 12 or 14 landmarks. Affine12+TPS14 achieves the lowest mean point-to-closest-point error of 3.19mm. These results support the combined value of anatomical landmark information and nonrigid alignment.
PDF arXiv
arXiv cs.SDSep 24
Estimation of Room Impulse Responses from Handclaps
Shih-Yu Lai, Kyung Yun Lee, Nils Meyer-Kahlen, Eloi Moliner, Bing-Yu Chen +1
Handclaps provide an equipment-free excitation for room acoustics, but their unknown and variable source waveform makes room impulse response (RIR) estimation challenging. In this work, we investigate whether RIRs can be estimated directly from handclaps. To this end, we introduce an anechoic handclap dataset containing 2,540 claps from 17 participants, designed to capture variability across natural claps and different hand configurations. We first establish the performance attainable when the excitation clap is known using regularized deconvolution, and show that approximating the unknown excitation by windowing the direct sound from the reverberant recording is insufficient. To estimate the RIR without a known excitation, we propose using the anechoic handclap recordings to train a deep neural network with a supervised regression objective. Evaluated on a controlled synthetic benchmark, the proposed neural regressor significantly outperforms windowing-based baselines across all instrumental metrics. Furthermore, we test the proposed method on handclap recordings measured in real acoustic spaces, showing that the inferred RIR spectra are consistent across different handclap…
PDF arXiv
arXiv cs.CLSep 24
YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech
William Chen, Shinnosuke Takamichi, Sayaka Shiota, Satoru Fukayama, Samuele Cornell +1
We present YODAS v3, a weakly-labeled speech corpus containing over 1.1 million hours of 48kHz multi-channel audio in 147 languages, released under a CC BY 3.0 license. YODAS v3 is not only the largest open speech dataset to date, but also the first truly large-scale speech corpus with high-fidelity stereo audio. We first provide the collection methodology for the corpus, where we introduce new techniques for gathering language-balanced speech data. The effectiveness of our approach is shown by the language distribution of the crawled data: 22 languages in YODAS v3 have over 10K hours and 73 languages have over 5K hours of data. We then conduct extensive analyses on the composition of the data, such as the distribution of languages, audio quality, and transcription quality. Finally, we train baseline speech recognition and neural codec models to show the effectiveness of the dataset. Download at https://huggingface.co/datasets/espnet/yodas3.
PDF arXiv
arXiv cs.CVSep 24
Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition
Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh
Head-pose variation introduces substantial appearance transformations in visual speech recognition (VSR), making pose-aware feature modulation desirable. However, performance degradation and unwanted feature interactions may result from using numerous Feature-wise Linear Modulation (FiLM) circuits with fixed modulation intensity. We propose a Pose Adaptive Dynamic FiLM framework with a Dynamic Residual FiLM (DR-FiLM) modulator that predicts input-dependent weights to adaptively control the strength of pose-conditioned modulation. Experiments on LRS2 and LRS3 demonstrate that unweighted multi-pathway modulation substantially degrades phoneme recognition, increasing PER to 20.33% and 29.42%, respectively, compared with 16.20% and 20.96% for the single ResFiLM configuration. In contrast, the proposed DR-FiLM with dynamic Deep-Res weighting reduces PER to 15.74% on LRS2 and 23.91% on LRS3, substantially mitigating the adverse effects of unweighted modulation. The analysis of the learned weights further reveals a consistent tendency to assign greater weight to the deeper FiLM pathway as head-pose variation increases. These results show that merging pose-conditioned FiLM circuits is…
PDF arXiv

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-video generation for interactive systems2 papers raise it · 2 address it
Interactive world models and applications need low-latency, streaming-capable audio-video synthesis to respond to user input in real time.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation and generation1 paper raises it · 1 addresses it
Generating or separating individual sound sources from a mix is needed for dynamic scene rendering and editing, but most models output only a single mixed audio track.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Evaluation of complex audio-video consistency1 paper raises it · 1 addresses it
Existing metrics fail to assess multi-event, multi-layer, or asymmetric cross-modal relations in generated scenes, limiting progress in joint generation.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Scalable training for high-resolution joint video-audio models1 paper raises it · 1 addresses it
Full attention becomes computationally infeasible at higher resolutions, and sparse attention methods often ignore cross-modal interactions in joint data.
- Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Trainingraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 22m ago

Voice AI

Speech recognition, speech synthesis, voice agents, spoken dialogue and audio language models.

103papers this week−12.0% vs last week

All papers · 623 With code · 16

Newest Most relevant

568 papers this month