Papers

Field

Top tools here

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-visual generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation and generation1 paper raises it · 1 addresses it
Rendering dynamic scenes requires individual control over sound sources, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Event-level audio-video consistency evaluation1 paper raises it · 1 addresses it
Complex scenes with multi-event composition and asymmetric cross-modal relations need fine-grained metrics beyond whole-clip assessments.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Zero-shot cross-lingual TTS often inherits the reference speaker's accent, degrading naturalness in the target language.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Recently funded

Topics in this field

text speech · automatic speech · speech models255 papers33 this week
visual speech · speech gesture generation · audio driven76 papers17 this week
audio language models · language models audio · large audio language71 papers13 this week
duplex speech models · duplex spoken language · duplex speech76 papers11 this week
spatial audio visual · audio visual dataset · audiovisual64 papers8 this week
music models · text music generation · music annotation45 papers6 this week

Topics in this field

Active contributors · 90 days

Shinji WatanabeLong Listening Thoughts: Eliciting Open Auditory Reasoning with Deliberative Perception and Cognitive Refinement5
Zhizheng WuConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence5
Joon Son ChungJAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents5
Chao ZhangFull-Duplex Interaction with Speech, Expressions and Gestures5
Dinesh ManochaMultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue5
Petr MotlicekBeyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS4

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.SDOct 18
DF-PALM: Dynamic Fusion Prompt Learning for Audio--Language Models
Yulin Sun, Shenghua Li, Yong Dou, Xinwang Liu, Cheng Yang +1
Few-shot audio classification requires audio--language models to distinguish variable acoustic events from limited labeled data. PALM's main formulation adapts encoded class representations without changing pretrained encoders, but its class representations remain input-independent and classification uses audio features without class-conditioned refinement. We introduce DF-PALM, a dynamic feature-space prompting framework that combines audio-conditioned class representations with asymmetric cross-attention for audio refinement. Class representations are conditioned after text encoding, held fixed across fusion layers for each input, and reused for final scoring. Both pretrained encoders remain frozen, avoiding audio-conditioned text-encoder passes. With PENGI as the frozen backbone, the main 16-shot evaluation covered 11 benchmarks and six baselines. DF-PALM achieved 80.69% mean accuracy and 80.32% macro-averaged F1, exceeding PALM by 4.11 and 4.30 percentage points, respectively. Paired benchmark-level tests supported both gains after Holm correction. Across four shot settings, DF-PALM achieved the highest mean accuracy and macro-averaged F1 among the four evaluated…
PDF arXiv
arXiv cs.CVOct 17
Predicting Individual Differences in Lipreading From Eye-Tracking and Pupillometry During Silent Visual Speech
Puneet Kumar, Satyarth Malik, Gao Zheng, Guoying Zhao, Xiaobai Li
Lipreading ability varies widely, including among people who are hard of hearing. Although mouth-directed attention is linked to speechreading outcomes, it remains unclear how much lipreading skill can be inferred from viewing behavior and which signals carry this information. We examine whether individual differences in lipreading ability are reflected in viewing behavior during silent speech and quantify the contributions of gaze dynamics, pupil-linked signals, and spatial viewing allocation. We introduce the Lipreading Perception Study (LiPS) dataset and a tracker-independent preprocessing pipeline that synchronizes SR Research EyeLink gaze and pupil recordings with the stimuli. Twenty-four adults (age 32 $\pm$ 10 years) completed a four-alternative forced-choice silent lipreading task. An extreme-groups protocol yielded 18 participants labeled as stronger or weaker lipreaders. Each participant-trial included four aligned views: lip-region images, frame-level gaze, pupil summaries, and spatial viewing proportions. A four-branch neural network with cross-modal attention predicted labels using participant-disjoint splits and systematic ablations. The full model reached accuracy…
PDF arXiv
arXiv eess.ASOct 17
Dual Task Vector Merging: Leveraging Emotion and ASR Knowledge for Low-Resource Alzheimer's Disease Detection
Yuan-Chao Li, Yin-Long Liu, Liu He, Sizhe Liu, Ji-Xuan Deng +3
Alzheimer's Disease (AD) detection from speech benefits from both paralinguistic and linguistic cues, yet model training is frequently constrained by limited data. Fine-tuning a large pretrained speech encoder can easily overfit, while a frozen probe leaves the encoder's task-relevant knowledge underutilized. Therefore, we propose Dual Task Vector Merging, a parameter-efficient method that adapts a frozen wav2vec 2.0 backbone using task knowledge learned outside the AD task. Specifically, we inject two task vectors, derived as weight deltas from emotion- and ASR-fine-tuned wav2vec 2.0 models, into each transformer layer. During AD training, only layer-wise merging coefficients and a linear classification head are optimized, updating less than 0.1% of the parameters required for full fine-tuning. Speaker-independent experiments on ADReSS and ADReSSo show that dual merging improves over frozen probes, LoRA, and full fine-tuning, in the main binary settings, and gives the best results when combined with LoRA. Evaluation on PROCESS further shows that our advantages transfer to three-class HC/MCI/dementia classification when paired with LoRA. Experiments under reduced training data…
PDF arXiv
arXiv cs.CLOct 17
Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors
Sankalp Parashar, Preethi Jyothi, Sunita Sarawagi
Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during streaming ASR permits a recourse mechanism where the speaker can be asked to repeat the affected content under cleaner acoustic conditions. We propose \sysname, a new framework for online error localization, that identifies spans likely to benefit from this intervention. Our supervision uses clean audio to estimate how the recognizer's output might change under improved acoustic conditions. We probabilistically align noisy and clean decodes using a pair-HMM to obtain soft supervision. These soft targets train a counterfactual gain scorer by combining a suite of acoustic and language features to produce token-level scores, that are further grouped into spans for targeted recourse. An LLM uses the ASR transcript of the repetition to revise only the selected span. At a precision of $70\%$, \sysname achieves the highest recall by identifying up to 52\% more erroneous tokens compared to the strongest baseline. Targeted repetition yields relative WER reductions of up to 24\% under a budget that accounts for both interruption frequency and the…
PDF arXiv
arXiv cs.CVCodeOct 17▲ 7
UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling
Kien T. Pham, Yuhao Liu, Chong Hou Choi, Zhenqi He, Yanghao Wang +4
Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing. Consequently, they are spent learning single-branch generative and editing capabilities that abundant unimodal data could readily teach, rather than on the cross-modal coupling. Furthermore, prevailing systems apply a single shared noise schedule to both modalities, neglecting two critical limitations. First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise. Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained. We present UniAVC, which addresses both obstacles. It first trains each branch on unimodal dataset to full generative and editing competence, and subsequently dedicates paired…
PDF arXiv Code
arXiv cs.SDOct 17
ASR Personalization in Pathological Voice: Accuracy, Efficiency, and Duration Effects
Giuseppe Giaimis, Sergei Katkov, Muhammad Azfar Yaqub, Antonio Liotta
Automatic Speech recognition (ASR) struggles with Pathological speech due to persistent speaker-specific mismatch. In clinical and accessibility scenarios, only limited target-speaker data is typically available, making full model retraining impractical. We present a component-level analysis of LoRA adaptation in Whisper, investigating how adapter placement interacts with the model size, training efficiency and utterance-duration in single-speaker settings. Our experimental comparative study shows that while scaling reduces errors, it does not eliminate mismatch. Furthermore, the full attention LoRa provides the most reliable accuracy, and the best time--accuracy trade-off. Improvements vary across utterance length, with medium-length recordings benefiting the most, while short and long utterances remain challenging. These findings highlight the importance of adaptation placement and duration-dependent evaluation, under challenging, data-scarce personalization scenarios.
PDF arXiv

Papers

Topics in this field

text speech · automatic speech · speech models255 papers33 this week
visual speech · speech gesture generation · audio driven76 papers17 this week
audio language models · language models audio · large audio language71 papers13 this week
duplex speech models · duplex spoken language · duplex speech76 papers11 this week
spatial audio visual · audio visual dataset · audiovisual64 papers8 this week
music models · text music generation · music annotation45 papers6 this week

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Active contributors · 90 days

Shinji WatanabeLong Listening Thoughts: Eliciting Open Auditory Reasoning with Deliberative Perception and Cognitive Refinement5
Zhizheng WuConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence5
Joon Son ChungJAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents5
Chao ZhangFull-Duplex Interaction with Speech, Expressions and Gestures5
Dinesh ManochaMultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue5
Petr MotlicekBeyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS4

Recent papers

All in Research →

arXiv cs.SDOct 18
DF-PALM: Dynamic Fusion Prompt Learning for Audio--Language Models
Yulin Sun, Shenghua Li, Yong Dou, Xinwang Liu, Cheng Yang +1
Few-shot audio classification requires audio--language models to distinguish variable acoustic events from limited labeled data. PALM's main formulation adapts encoded class representations without changing pretrained encoders, but its class representations remain input-independent and classification uses audio features without class-conditioned refinement. We introduce DF-PALM, a dynamic feature-space prompting framework that combines audio-conditioned class representations with asymmetric cross-attention for audio refinement. Class representations are conditioned after text encoding, held fixed across fusion layers for each input, and reused for final scoring. Both pretrained encoders remain frozen, avoiding audio-conditioned text-encoder passes. With PENGI as the frozen backbone, the main 16-shot evaluation covered 11 benchmarks and six baselines. DF-PALM achieved 80.69% mean accuracy and 80.32% macro-averaged F1, exceeding PALM by 4.11 and 4.30 percentage points, respectively. Paired benchmark-level tests supported both gains after Holm correction. Across four shot settings, DF-PALM achieved the highest mean accuracy and macro-averaged F1 among the four evaluated…
PDF arXiv
arXiv cs.CVOct 17
Predicting Individual Differences in Lipreading From Eye-Tracking and Pupillometry During Silent Visual Speech
Puneet Kumar, Satyarth Malik, Gao Zheng, Guoying Zhao, Xiaobai Li
Lipreading ability varies widely, including among people who are hard of hearing. Although mouth-directed attention is linked to speechreading outcomes, it remains unclear how much lipreading skill can be inferred from viewing behavior and which signals carry this information. We examine whether individual differences in lipreading ability are reflected in viewing behavior during silent speech and quantify the contributions of gaze dynamics, pupil-linked signals, and spatial viewing allocation. We introduce the Lipreading Perception Study (LiPS) dataset and a tracker-independent preprocessing pipeline that synchronizes SR Research EyeLink gaze and pupil recordings with the stimuli. Twenty-four adults (age 32 $\pm$ 10 years) completed a four-alternative forced-choice silent lipreading task. An extreme-groups protocol yielded 18 participants labeled as stronger or weaker lipreaders. Each participant-trial included four aligned views: lip-region images, frame-level gaze, pupil summaries, and spatial viewing proportions. A four-branch neural network with cross-modal attention predicted labels using participant-disjoint splits and systematic ablations. The full model reached accuracy…
PDF arXiv
arXiv eess.ASOct 17
Dual Task Vector Merging: Leveraging Emotion and ASR Knowledge for Low-Resource Alzheimer's Disease Detection
Yuan-Chao Li, Yin-Long Liu, Liu He, Sizhe Liu, Ji-Xuan Deng +3
Alzheimer's Disease (AD) detection from speech benefits from both paralinguistic and linguistic cues, yet model training is frequently constrained by limited data. Fine-tuning a large pretrained speech encoder can easily overfit, while a frozen probe leaves the encoder's task-relevant knowledge underutilized. Therefore, we propose Dual Task Vector Merging, a parameter-efficient method that adapts a frozen wav2vec 2.0 backbone using task knowledge learned outside the AD task. Specifically, we inject two task vectors, derived as weight deltas from emotion- and ASR-fine-tuned wav2vec 2.0 models, into each transformer layer. During AD training, only layer-wise merging coefficients and a linear classification head are optimized, updating less than 0.1% of the parameters required for full fine-tuning. Speaker-independent experiments on ADReSS and ADReSSo show that dual merging improves over frozen probes, LoRA, and full fine-tuning, in the main binary settings, and gives the best results when combined with LoRA. Evaluation on PROCESS further shows that our advantages transfer to three-class HC/MCI/dementia classification when paired with LoRA. Experiments under reduced training data…
PDF arXiv
arXiv cs.CLOct 17
Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors
Sankalp Parashar, Preethi Jyothi, Sunita Sarawagi
Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during streaming ASR permits a recourse mechanism where the speaker can be asked to repeat the affected content under cleaner acoustic conditions. We propose \sysname, a new framework for online error localization, that identifies spans likely to benefit from this intervention. Our supervision uses clean audio to estimate how the recognizer's output might change under improved acoustic conditions. We probabilistically align noisy and clean decodes using a pair-HMM to obtain soft supervision. These soft targets train a counterfactual gain scorer by combining a suite of acoustic and language features to produce token-level scores, that are further grouped into spans for targeted recourse. An LLM uses the ASR transcript of the repetition to revise only the selected span. At a precision of $70\%$, \sysname achieves the highest recall by identifying up to 52\% more erroneous tokens compared to the strongest baseline. Targeted repetition yields relative WER reductions of up to 24\% under a budget that accounts for both interruption frequency and the…
PDF arXiv
arXiv cs.CVCodeOct 17▲ 7
UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling
Kien T. Pham, Yuhao Liu, Chong Hou Choi, Zhenqi He, Yanghao Wang +4
Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing. Consequently, they are spent learning single-branch generative and editing capabilities that abundant unimodal data could readily teach, rather than on the cross-modal coupling. Furthermore, prevailing systems apply a single shared noise schedule to both modalities, neglecting two critical limitations. First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise. Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained. We present UniAVC, which addresses both obstacles. It first trains each branch on unimodal dataset to full generative and editing competence, and subsequently dedicates paired…
PDF arXiv Code
arXiv cs.SDOct 17
ASR Personalization in Pathological Voice: Accuracy, Efficiency, and Duration Effects
Giuseppe Giaimis, Sergei Katkov, Muhammad Azfar Yaqub, Antonio Liotta
Automatic Speech recognition (ASR) struggles with Pathological speech due to persistent speaker-specific mismatch. In clinical and accessibility scenarios, only limited target-speaker data is typically available, making full model retraining impractical. We present a component-level analysis of LoRA adaptation in Whisper, investigating how adapter placement interacts with the model size, training efficiency and utterance-duration in single-speaker settings. Our experimental comparative study shows that while scaling reduces errors, it does not eliminate mismatch. Furthermore, the full attention LoRa provides the most reliable accuracy, and the best time--accuracy trade-off. Improvements vary across utterance length, with medium-length recordings benefiting the most, while short and long utterances remain challenging. These findings highlight the importance of adaptation placement and duration-dependent evaluation, under challenging, data-scarce personalization scenarios.
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 52m ago

Voice AI

Speech recognition, speech synthesis, voice agents, spoken dialogue and audio language models.

88papers this week−24.8% vs last week

All papers · 587 With code · 14

Newest Most relevant

553 papers this month

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-visual generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation and generation1 paper raises it · 1 addresses it
Rendering dynamic scenes requires individual control over sound sources, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Event-level audio-video consistency evaluation1 paper raises it · 1 addresses it
Complex scenes with multi-event composition and asymmetric cross-modal relations need fine-grained metrics beyond whole-clip assessments.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Zero-shot cross-lingual TTS often inherits the reference speaker's accent, degrading naturalness in the target language.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.