Papers

Field

Top tools here

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-video generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation in generation1 paper raises it · 1 addresses it
Accessing individual sound sources is necessary for dynamic scene rendering and editing, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Evaluation of complex audio-video consistency1 paper raises it · 1 addresses it
Current metrics fail to assess multi-event, multi-layer, or asymmetric cross-modal scenes, limiting progress in advanced generation.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Reference accents often contaminate target speech, reducing naturalness and speaker similarity in multilingual synthesis.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Recently funded

Topics in this field

text speech · automatic speech · speech models268 papers40 this week
visual speech · speech gesture generation · audio driven77 papers16 this week
audio language models · language models audio · large audio language75 papers15 this week
duplex speech models · duplex spoken language · duplex speech78 papers12 this week
spatial audio visual · audio visual dataset · audiovisual68 papers12 this week
music models · text music generation · music annotation45 papers6 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CVOct 8
Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion
Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal, Neela Sawant
Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex…
PDF arXiv
arXiv cs.LGOct 8
Self-Supervised Speech Representations for Cross-Speaker Dysarthria Detection During Awake Craniotomy
Kanthila Chinmayi, Abdallah Nassib, Misy Harrison, Panheleux Celine, Saliou Vanessa +2
Detecting intra-operative speech impairment during awake craniotomy is essential for preserving language function. However, automated detection remains challenging because operating-room recordings contain substantial acoustic interference, clinically relevant speech events are rare, and available cohorts are small and heterogeneous across speakers. This study presents a systematic component-wise evaluation of a pipeline for distinguishing dysarthric from no-trouble speech in the DATABRASE corpus of awake-craniotomy recordings. The pipeline incorporates speaker diarization to isolate patient speech, a multi-view representation combining handcrafted acoustic descriptors with multilayer wav2vec 2.0 embeddings, speaker-conditional normalization and transferability-based feature selection to improve cross-speaker robustness, and a cascaded classifier comprising a gradient-boosted first stage and a neural second stage. Evaluation was conducted under strict speaker-independent conditions using leave-one-speaker-out cross-validation. The results show that cross-speaker performance is influenced more strongly by the speech representation than by classifier choice. The AUCs of three…
PDF arXiv
arXiv cs.MMOct 8
AuraLuxMuse: Adaptive Fusion Modeling for Aesthetic Stage Lighting Design with Music and Expert Guidance
Junyu Deng, Jiale Cao, Mengtian Li, Zhongxia Ji, Ruhua Chen +3
We present AuraLuxMuse, a novel system for automated aesthetic stage lighting design that integrates expert knowledge, representation learning, and preference-adaptive modeling. Lighting design in live performance settings requires the seamless translation of musical features into dynamic lighting behaviors. However, traditional workflows remain time-consuming, labor-intensive, and difficult to transfer. AuraLuxMuse encodes music and professional cue sequences into a shared retrieval space, estimates cue-event density, and retargets selected fixture commands to the destination stage. It assists pre-production authoring by returning editable cues rather than replacing the designer with an unconstrained generator. At the heart of AuraLuxMuse are two key modules: Lighting-Aligned Music Pretraining (LAMP), which performs contrastive learning between audio and lighting cues for alignment, and Preference-Adaptive Mixture of Experts (PAMoE), which conditions preference-aware cue retrieval and adaptation on designers' intent through a gated ensemble of style-specific expert networks. To support training and evaluation, we introduce Musilux, the first dataset of paired musical audio and…
PDF arXiv
arXiv cs.CVOct 8
Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation
Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak
Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations.…
PDF arXiv
arXiv cs.SDOct 8
Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation
Jia Li, Yinfeng Yu
Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of \textbf{Dynamic Audio-Visual Navigation (D-AVN)}. Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel \textbf{Two-Stage Multi-modal Learning} framework for D-AVN. In the Physical Alignment stage, the proposed \textbf{Intensity-Calibrated Geometry Encoder (ICGE)} performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based \textbf{Cross-Modal Scene Aggregator (CMSA)} dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly…
PDF arXiv
arXiv cs.CLOct 8
Phonologically Informed Tokenization for German Speech Recognition: A Cross-Domain Study
Christopher Witzl, Tobias Bocklet, Korbinian Riedhammer
German is a morphologically rich language whose syllable structure is exceptionally well-predicted by the Knuth--Liang hyphenation algorithm. We ask whether phonologically informed tokenization can serve as a competitive target for end-to-end speech recognition. We compare three tokenizer families on the Omnilingual ASR wav2vec 2.0 backbone fine-tuned with CTC: the pretrained multilingual character inventory, a data-driven Byte-Pair Encoding (BPE) over orthography, and phonologically informed units from Pyphen syllabification and grapheme-to-phoneme conversion. Across 40 fine-tunes, we evaluate on three German test sets spanning orthogonal shifts: in-domain read speech, dialectal spontaneous speech, and standard-German spontaneous speech. In-domain, all phonologically informed tokenizers match BPE and the multilingual character baseline on both WER and CER. Under domain shift the picture splits along vocabulary size rather than the linguistic axis of variation: at small vocabularies, syllable-aware tokenization improves on dialectal speech, where phonetic surface forms vary but syllable structure is preserved, and stays ahead on spontaneous speech, where new word-forms violate…
PDF arXiv

Papers

Topics in this field

text speech · automatic speech · speech models268 papers40 this week
visual speech · speech gesture generation · audio driven77 papers16 this week
audio language models · language models audio · large audio language75 papers15 this week
duplex speech models · duplex spoken language · duplex speech78 papers12 this week
spatial audio visual · audio visual dataset · audiovisual68 papers12 this week
music models · text music generation · music annotation45 papers6 this week

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.CVOct 8
Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion
Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal, Neela Sawant
Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex…
PDF arXiv
arXiv cs.LGOct 8
Self-Supervised Speech Representations for Cross-Speaker Dysarthria Detection During Awake Craniotomy
Kanthila Chinmayi, Abdallah Nassib, Misy Harrison, Panheleux Celine, Saliou Vanessa +2
Detecting intra-operative speech impairment during awake craniotomy is essential for preserving language function. However, automated detection remains challenging because operating-room recordings contain substantial acoustic interference, clinically relevant speech events are rare, and available cohorts are small and heterogeneous across speakers. This study presents a systematic component-wise evaluation of a pipeline for distinguishing dysarthric from no-trouble speech in the DATABRASE corpus of awake-craniotomy recordings. The pipeline incorporates speaker diarization to isolate patient speech, a multi-view representation combining handcrafted acoustic descriptors with multilayer wav2vec 2.0 embeddings, speaker-conditional normalization and transferability-based feature selection to improve cross-speaker robustness, and a cascaded classifier comprising a gradient-boosted first stage and a neural second stage. Evaluation was conducted under strict speaker-independent conditions using leave-one-speaker-out cross-validation. The results show that cross-speaker performance is influenced more strongly by the speech representation than by classifier choice. The AUCs of three…
PDF arXiv
arXiv cs.MMOct 8
AuraLuxMuse: Adaptive Fusion Modeling for Aesthetic Stage Lighting Design with Music and Expert Guidance
Junyu Deng, Jiale Cao, Mengtian Li, Zhongxia Ji, Ruhua Chen +3
We present AuraLuxMuse, a novel system for automated aesthetic stage lighting design that integrates expert knowledge, representation learning, and preference-adaptive modeling. Lighting design in live performance settings requires the seamless translation of musical features into dynamic lighting behaviors. However, traditional workflows remain time-consuming, labor-intensive, and difficult to transfer. AuraLuxMuse encodes music and professional cue sequences into a shared retrieval space, estimates cue-event density, and retargets selected fixture commands to the destination stage. It assists pre-production authoring by returning editable cues rather than replacing the designer with an unconstrained generator. At the heart of AuraLuxMuse are two key modules: Lighting-Aligned Music Pretraining (LAMP), which performs contrastive learning between audio and lighting cues for alignment, and Preference-Adaptive Mixture of Experts (PAMoE), which conditions preference-aware cue retrieval and adaptation on designers' intent through a gated ensemble of style-specific expert networks. To support training and evaluation, we introduce Musilux, the first dataset of paired musical audio and…
PDF arXiv
arXiv cs.CVOct 8
Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation
Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak
Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations.…
PDF arXiv
arXiv cs.SDOct 8
Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation
Jia Li, Yinfeng Yu
Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of \textbf{Dynamic Audio-Visual Navigation (D-AVN)}. Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel \textbf{Two-Stage Multi-modal Learning} framework for D-AVN. In the Physical Alignment stage, the proposed \textbf{Intensity-Calibrated Geometry Encoder (ICGE)} performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based \textbf{Cross-Modal Scene Aggregator (CMSA)} dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly…
PDF arXiv
arXiv cs.CLOct 8
Phonologically Informed Tokenization for German Speech Recognition: A Cross-Domain Study
Christopher Witzl, Tobias Bocklet, Korbinian Riedhammer
German is a morphologically rich language whose syllable structure is exceptionally well-predicted by the Knuth--Liang hyphenation algorithm. We ask whether phonologically informed tokenization can serve as a competitive target for end-to-end speech recognition. We compare three tokenizer families on the Omnilingual ASR wav2vec 2.0 backbone fine-tuned with CTC: the pretrained multilingual character inventory, a data-driven Byte-Pair Encoding (BPE) over orthography, and phonologically informed units from Pyphen syllabification and grapheme-to-phoneme conversion. Across 40 fine-tunes, we evaluate on three German test sets spanning orthogonal shifts: in-domain read speech, dialectal spontaneous speech, and standard-German spontaneous speech. In-domain, all phonologically informed tokenizers match BPE and the multilingual character baseline on both WER and CER. Under domain shift the picture splits along vocabulary size rather than the linguistic axis of variation: at small vocabularies, syllable-aware tokenization improves on dialectal speech, where phonetic surface forms vary but syllable structure is preserved, and stays ahead on spontaneous speech, where new word-forms violate…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 24m ago

Voice AI

Speech recognition, speech synthesis, voice agents, spoken dialogue and audio language models.

101papers this week−13.7% vs last week

All papers · 611 With code · 16

Newest Most relevant

564 papers this month

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-video generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation in generation1 paper raises it · 1 addresses it
Accessing individual sound sources is necessary for dynamic scene rendering and editing, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Evaluation of complex audio-video consistency1 paper raises it · 1 addresses it
Current metrics fail to assess multi-event, multi-layer, or asymmetric cross-modal scenes, limiting progress in advanced generation.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Reference accents often contaminate target speech, reducing naturalness and speaker similarity in multilingual synthesis.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.