Papers

Field

Top tools here

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-video generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation in generation1 paper raises it · 1 addresses it
Accessing individual sound sources is necessary for dynamic scene rendering and editing, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Evaluation of complex audio-video consistency1 paper raises it · 1 addresses it
Current metrics fail to assess multi-event, multi-layer, or asymmetric cross-modal scenes, limiting progress in advanced generation.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Reference accents often contaminate target speech, reducing naturalness and speaker similarity in multilingual synthesis.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Recently funded

Topics in this field

text speech · automatic speech · speech models268 papers38 this week
visual speech · speech gesture generation · audio driven77 papers16 this week
audio language models · language models audio · large audio language75 papers15 this week
duplex speech models · duplex spoken language · duplex speech78 papers12 this week
spatial audio visual · audio visual dataset · audiovisual68 papers11 this week
music models · text music generation · music annotation45 papers6 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.AIOct 4
Offline AI Modules: Voice-First Offline Architecture, Hardware Reference Stack, Quantization and Benchmarking
Sunday Afariogun, Odunolaoluwa Jenrola, Zeinab Nezami
The Offline AI Modules workstream enables practical, low-power, and community-accessible deployment of voice-first AI systems that operate fully offline. Designed for African language communities where speech is the dominant mode of interaction and internet connectivity is unreliable or absent, the workstream delivers three reinforcing components: a modular voice-first offline architecture, a low-cost hardware reference bill of materials, and a reproducible quantization and a reproducible quantization and benchmarking pipeline for instruction-tuned language models in the 2-5B parameter class. This paper presents the first end-to-end benchmark evaluation of the stack across two hardware tiers: an NVIDIA Jetson Orin NX (TierB) and a Raspberry Pi5 (TierA). Three instruction-tuned models are evaluated across four quantization formats, assessed for deployment metrics (decode throughput, chat latency, memory, power) and multilingual quality (topic classification accuracy on MasakhaNEWS across English, Hausa, Igbo, Nigerian Pidgin, and Yoruba; per-language perplexity drift). Speech recognition is evaluated using Ethio-ASR on Amharic and Oromo across both tiers. The principal finding is…
PDF arXiv
arXiv cs.CVOct 4
BossouChimpanzee: Long-term Chimpanzee Video Dataset
Daniel Schofield, Susana Carvalho, Vladimir Iashin, Andrew Zisserman, Max Bain +8
We describe the BossouChimpanzee video dataset, a unique long-term visual record of wild chimpanzees at an outdoor laboratory for field experiments in Bossou, Guinea, spanning three decades (1988-2018) and comprising over 1,200 hours of continuous video recordings collected through collaborative fieldwork and research. In this paper, we outline the history and scientific contributions of the experimental paradigm and video archive, provide key statistics and details on the structure of the main video dataset, and release an initial ~74h snapshot, BossouChimpanzee70h, covering 23 identified individuals focused on chimpanzee individual and action recognition, ahead of the full video resource. This dataset represents a valuable resource for cognitive and behavioural research in ethology and a rich benchmark for training and evaluating machine learning models on audiovisual data from the wild.
PDF arXiv
arXiv cs.SDOct 4
Task-Aware Joint Pruning and Distillation for Efficient Audio Deepfake Detection
Miao He, Peng Cheng, Zhongjie Ba, Qing Wen, Li Lu +2
Advances in speech synthesis have made deepfake speeches increasingly convincing, posing growing threats to security. While self-supervised learning (SSL) based detectors achieve state-of-the-art performance, their computational demands (typically 300M+ parameters) prevent deployment on resource-constrained devices. Existing compression methods, designed mainly for content-centric tasks, struggle to maintain competitive performance when directly adapted to deepfake detection. We propose a Task-Aware Joint Pruning and Distillation framework that combines cross-domain knowledge distillation with movement-guided structured pruning to transfer forgery-discriminative knowledge and preserve critical structures under aggressive compression. Our framework reduces the model to 31.9M parameters with 6.3$\times$ FLOPs reduction, with an average performance drop of only 1.30\% across multiple datasets compared to the uncompressed baseline, demonstrating strong potential for on-device deployment.
PDF arXiv
arXiv cs.SDOct 4
Where Does the Audio Jailbreak Live? A Controlled Frequency-Depth Audit of AdvWave-P on Qwen2-Audio
Boyuan Chen, Minseok Kim, Sohaila Abdulsattar, Minghao Shao, Siddharth Garg +2
We audit frequency and decoder-depth claims for AdvWave-P, an additive audio jailbreak, on Qwen2-Audio. The protocol masks frequency components of the perturbation in the short-time Fourier transform (STFT) domain and measures attack success and audio-span representations. On 520 AdvBench prompts, the primary judge labels 76.7% of adversarial inputs as jailbreaks. A condition-blind, single-annotator validation yields a Rogan-Gladen sensitivity estimate of 0.87 for this condition (about 0.83-0.95 with validation-rate uncertainty); this correction is not applied to masked conditions. The apparent frequency ranking depends on the partition: energy share alone predicts the standard eight-band ranking (Spearman's rho = 0.95), and equal-Hz and equal-energy partitions show that masking any tested band can sharply reduce attack success. At a finer 16-band equal-energy resolution, however, masking the narrow 7520-7960 Hz band leaves ASR at 0.10, which remains unresolved without a matched control. Matched-energy scattered-removal tests show that contiguous removal is more damaging in lower bands, while both forms approach the floor in upper bands. Global rescaling leaves ASR near baseline…
PDF arXiv
arXiv cs.SDOct 4
Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech
Hongfei Du, Jiacheng Shi, Yanfu Zhang, Ye Gao
LLM-based text-to-speech (TTS) models can generate emotionally expressive speech, but how reference emotion is routed through the model and realized in decoded speech remains unclear. We introduce two emotion-sensitive metrics for matched neutral and emotional syntheses---a codec trajectory score and a late residual direction score---and use them to score activation-patching interventions. Under controlled matched-reference conditions, this analysis identifies a sparse source-to-readout component-level circuit: 23--27 attention heads and MLPs per emotion, roughly 5% of the components considered, recover or suppress 74--88% of the late emotion-readout shift on held-out cases. The circuit combines a shared component backbone with emotion-specific components; cross-emotion activation swaps reduce the target readout in 47 of 48 cases. In decoded speech, the same intervention produces consistent changes in pitch, energy, and spectral brightness over 24 matched pairs per emotion. A readout-matched residual-direction baseline produces only 17--27% of the intervention's pitch effect, showing that internal readout movement alone does not explain the decoded acoustic changes. These results…
PDF arXiv
arXiv cs.CLOct 4
AraYoungVoices: A Diverse L1/L2 Corpus of Arabic Child and Adolescent Speech
Shammur Absar Chowdhury, Zien Sheikh Ali, Houssam Eddine-Othman Lachemat, Hamdy Mubarak
State-of-the-art ASR systems primarily target native adult speech, leading to substantial performance gaps for children, adolescents, and L2 speakers. We introduce AraYoungVoices, a 151.72-hour Arabic read-speech corpus from 286 speakers aged 7--18, comprising AraKids (7--12) and AraTeens (13--18). The corpus includes 146 native Arabic (L1) and 140 second-language (L2) speakers, with native speakers spanning Egyptian, Gulf, Levantine, and North African dialectal backgrounds and L2 speakers representing diverse linguistic backgrounds across the Americas, Asia, Africa, and Europe. We benchmark four pretrained ASR models under zero-shot and fine-tuned settings using unseen-speaker-$\&$-unseen-prompt (USUP) and unseen-speaker-$\&$-seen-prompt (USSP) evaluations. Results show that L2 speech remains substantially more challenging than L1 speech, with the largest errors observed mainly for younger L2 speakers. Age-specific fine-tuning improves the matched age group, while joint fine-tuning provides a stronger balance across populations. ASR hypotheses are also consistently closer to the standard reading prompt than to the verbatim transcription, particularly for L2 speech, suggesting…
PDF arXiv

Papers

Topics in this field

text speech · automatic speech · speech models268 papers38 this week
visual speech · speech gesture generation · audio driven77 papers16 this week
audio language models · language models audio · large audio language75 papers15 this week
duplex speech models · duplex spoken language · duplex speech78 papers12 this week
spatial audio visual · audio visual dataset · audiovisual68 papers11 this week
music models · text music generation · music annotation45 papers6 this week

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.AIOct 4
Offline AI Modules: Voice-First Offline Architecture, Hardware Reference Stack, Quantization and Benchmarking
Sunday Afariogun, Odunolaoluwa Jenrola, Zeinab Nezami
The Offline AI Modules workstream enables practical, low-power, and community-accessible deployment of voice-first AI systems that operate fully offline. Designed for African language communities where speech is the dominant mode of interaction and internet connectivity is unreliable or absent, the workstream delivers three reinforcing components: a modular voice-first offline architecture, a low-cost hardware reference bill of materials, and a reproducible quantization and a reproducible quantization and benchmarking pipeline for instruction-tuned language models in the 2-5B parameter class. This paper presents the first end-to-end benchmark evaluation of the stack across two hardware tiers: an NVIDIA Jetson Orin NX (TierB) and a Raspberry Pi5 (TierA). Three instruction-tuned models are evaluated across four quantization formats, assessed for deployment metrics (decode throughput, chat latency, memory, power) and multilingual quality (topic classification accuracy on MasakhaNEWS across English, Hausa, Igbo, Nigerian Pidgin, and Yoruba; per-language perplexity drift). Speech recognition is evaluated using Ethio-ASR on Amharic and Oromo across both tiers. The principal finding is…
PDF arXiv
arXiv cs.CVOct 4
BossouChimpanzee: Long-term Chimpanzee Video Dataset
Daniel Schofield, Susana Carvalho, Vladimir Iashin, Andrew Zisserman, Max Bain +8
We describe the BossouChimpanzee video dataset, a unique long-term visual record of wild chimpanzees at an outdoor laboratory for field experiments in Bossou, Guinea, spanning three decades (1988-2018) and comprising over 1,200 hours of continuous video recordings collected through collaborative fieldwork and research. In this paper, we outline the history and scientific contributions of the experimental paradigm and video archive, provide key statistics and details on the structure of the main video dataset, and release an initial ~74h snapshot, BossouChimpanzee70h, covering 23 identified individuals focused on chimpanzee individual and action recognition, ahead of the full video resource. This dataset represents a valuable resource for cognitive and behavioural research in ethology and a rich benchmark for training and evaluating machine learning models on audiovisual data from the wild.
PDF arXiv
arXiv cs.SDOct 4
Task-Aware Joint Pruning and Distillation for Efficient Audio Deepfake Detection
Miao He, Peng Cheng, Zhongjie Ba, Qing Wen, Li Lu +2
Advances in speech synthesis have made deepfake speeches increasingly convincing, posing growing threats to security. While self-supervised learning (SSL) based detectors achieve state-of-the-art performance, their computational demands (typically 300M+ parameters) prevent deployment on resource-constrained devices. Existing compression methods, designed mainly for content-centric tasks, struggle to maintain competitive performance when directly adapted to deepfake detection. We propose a Task-Aware Joint Pruning and Distillation framework that combines cross-domain knowledge distillation with movement-guided structured pruning to transfer forgery-discriminative knowledge and preserve critical structures under aggressive compression. Our framework reduces the model to 31.9M parameters with 6.3$\times$ FLOPs reduction, with an average performance drop of only 1.30\% across multiple datasets compared to the uncompressed baseline, demonstrating strong potential for on-device deployment.
PDF arXiv
arXiv cs.SDOct 4
Where Does the Audio Jailbreak Live? A Controlled Frequency-Depth Audit of AdvWave-P on Qwen2-Audio
Boyuan Chen, Minseok Kim, Sohaila Abdulsattar, Minghao Shao, Siddharth Garg +2
We audit frequency and decoder-depth claims for AdvWave-P, an additive audio jailbreak, on Qwen2-Audio. The protocol masks frequency components of the perturbation in the short-time Fourier transform (STFT) domain and measures attack success and audio-span representations. On 520 AdvBench prompts, the primary judge labels 76.7% of adversarial inputs as jailbreaks. A condition-blind, single-annotator validation yields a Rogan-Gladen sensitivity estimate of 0.87 for this condition (about 0.83-0.95 with validation-rate uncertainty); this correction is not applied to masked conditions. The apparent frequency ranking depends on the partition: energy share alone predicts the standard eight-band ranking (Spearman's rho = 0.95), and equal-Hz and equal-energy partitions show that masking any tested band can sharply reduce attack success. At a finer 16-band equal-energy resolution, however, masking the narrow 7520-7960 Hz band leaves ASR at 0.10, which remains unresolved without a matched control. Matched-energy scattered-removal tests show that contiguous removal is more damaging in lower bands, while both forms approach the floor in upper bands. Global rescaling leaves ASR near baseline…
PDF arXiv
arXiv cs.SDOct 4
Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech
Hongfei Du, Jiacheng Shi, Yanfu Zhang, Ye Gao
LLM-based text-to-speech (TTS) models can generate emotionally expressive speech, but how reference emotion is routed through the model and realized in decoded speech remains unclear. We introduce two emotion-sensitive metrics for matched neutral and emotional syntheses---a codec trajectory score and a late residual direction score---and use them to score activation-patching interventions. Under controlled matched-reference conditions, this analysis identifies a sparse source-to-readout component-level circuit: 23--27 attention heads and MLPs per emotion, roughly 5% of the components considered, recover or suppress 74--88% of the late emotion-readout shift on held-out cases. The circuit combines a shared component backbone with emotion-specific components; cross-emotion activation swaps reduce the target readout in 47 of 48 cases. In decoded speech, the same intervention produces consistent changes in pitch, energy, and spectral brightness over 24 matched pairs per emotion. A readout-matched residual-direction baseline produces only 17--27% of the intervention's pitch effect, showing that internal readout movement alone does not explain the decoded acoustic changes. These results…
PDF arXiv
arXiv cs.CLOct 4
AraYoungVoices: A Diverse L1/L2 Corpus of Arabic Child and Adolescent Speech
Shammur Absar Chowdhury, Zien Sheikh Ali, Houssam Eddine-Othman Lachemat, Hamdy Mubarak
State-of-the-art ASR systems primarily target native adult speech, leading to substantial performance gaps for children, adolescents, and L2 speakers. We introduce AraYoungVoices, a 151.72-hour Arabic read-speech corpus from 286 speakers aged 7--18, comprising AraKids (7--12) and AraTeens (13--18). The corpus includes 146 native Arabic (L1) and 140 second-language (L2) speakers, with native speakers spanning Egyptian, Gulf, Levantine, and North African dialectal backgrounds and L2 speakers representing diverse linguistic backgrounds across the Americas, Asia, Africa, and Europe. We benchmark four pretrained ASR models under zero-shot and fine-tuned settings using unseen-speaker-$\&$-unseen-prompt (USUP) and unseen-speaker-$\&$-seen-prompt (USSP) evaluations. Results show that L2 speech remains substantially more challenging than L1 speech, with the largest errors observed mainly for younger L2 speakers. Age-specific fine-tuning improves the matched age group, while joint fine-tuning provides a stronger balance across populations. ASR hypotheses are also consistently closer to the standard reading prompt than to the verbatim transcription, particularly for L2 speech, suggesting…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 27m ago

Voice AI

Speech recognition, speech synthesis, voice agents, spoken dialogue and audio language models.

98papers this week−14.8% vs last week

All papers · 611 With code · 16

Newest Most relevant

559 papers this month

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-video generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation in generation1 paper raises it · 1 addresses it
Accessing individual sound sources is necessary for dynamic scene rendering and editing, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Evaluation of complex audio-video consistency1 paper raises it · 1 addresses it
Current metrics fail to assess multi-event, multi-layer, or asymmetric cross-modal scenes, limiting progress in advanced generation.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Reference accents often contaminate target speech, reducing naturalness and speaker similarity in multilingual synthesis.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.