Papers

Field

Top tools here

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-visual generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation and generation1 paper raises it · 1 addresses it
Rendering dynamic scenes requires individual control over sound sources, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Event-level audio-video consistency evaluation1 paper raises it · 1 addresses it
Complex scenes with multi-event composition and asymmetric cross-modal relations need fine-grained metrics beyond whole-clip assessments.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Zero-shot cross-lingual TTS often inherits the reference speaker's accent, degrading naturalness in the target language.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Recently funded

Topics in this field

text speech · automatic speech · speech models255 papers34 this week
visual speech · speech gesture generation · audio driven76 papers17 this week
audio language models · language models audio · large audio language71 papers13 this week
duplex speech models · duplex spoken language · duplex speech76 papers11 this week
spatial audio visual · audio visual dataset · audiovisual64 papers8 this week
music models · text music generation · music annotation45 papers6 this week

Topics in this field

Active contributors · 90 days

Shinji WatanabeLong Listening Thoughts: Eliciting Open Auditory Reasoning with Deliberative Perception and Cognitive Refinement5
Zhizheng WuConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence5
Joon Son ChungJAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents5
Chao ZhangFull-Duplex Interaction with Speech, Expressions and Gestures5
Dinesh ManochaMultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue5
Petr MotlicekBeyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS4

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.SDOct 19
Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification
Zezhong Jin, Xiaoyu Wang, Zhe Li, Chong-Xin Gan, Siqing Qin +2
Large self-supervised speech models provide powerful representations for speaker verification, but full fine-tuning is expensive. Most conventional adapters insert small trainable modules to modify the output of each layer, while keeping the original residual connections unchanged. Building on our preliminary study of manifold-constrained Hyper-Connections in speaker encoders, we explore parameter-efficient adaptation of both residual paths and feature selection across layers. We first propose a Manifold-Constrained Residual Adapter (MCRA), which splits the hidden dimensions of each Transformer residual state into N groups and uses a small Sinkhorn-normalized matrix to exchange information among them. A learnable gate controls the balance between the original and mixed residual paths. The gate adds only one scalar parameter at each residual connection and initially favors the original path, allowing gradual adaptation. We further propose Frame-Adaptive Layer Aggregation (FALA), an attention-based aggregation method that uses a single learned query shared across all utterances to compute layer weights separately for each speech frame. Because the layer representations depend on…
PDF arXiv
arXiv cs.LGOct 19
Text Dominance in Paralinguistic Conflict: Tracing Suppression Through Audio LLM Circuits
Ankur Banerjee, Vishnu Guntakala, Danity Pike, Advay Roongta, Kevin Yang +1
Audio LLMs deployed in settings such as crisis hotlines and customer support are only useful beyond speech-to-text pipelines if they use acoustic cues rather than the transcript alone. We test whether they prioritise transcript content over paralinguistic cues (emotion, speaker identity, sarcasm) and trace the mechanism when they do. Across six audio LLMs under LISTEN-style conflict conditions, the mean text-dominance rate on mismatched text+audio items is 0.78 (0.86 excluding sarcasm). In the most text-dominant screened model, probing shows speaker identity remains linearly decodable at ceiling at the language model's input, so the failure is not perceptual. Back-patching, reported to repair lexical text dominance, has no measurable effect here; but it also fails on its own source task in our replication (n=300), so we treat its null as uninformative rather than as evidence of non-transfer. Attention knockout partially restores audio-grounded output by blocking the claim from propagating through intermediate tokens during open-ended generation, whereas blocking the answer position's direct connection to the conflicting claim has no effect. The claim's influence therefore travels…
PDF arXiv
arXiv cs.CLOct 18
Quality-Controlled Reply and Summary Data for Post-Training Small Language Models in AI Call-Screening Agents
Jiaqi Gan, Tsang Ng, Ankit Raj, Tommy Duong, Xingyu Shen +3
Small language models have improved to the point where a phone can plausibly screen its own calls, answering on an absent owner's behalf without sending call content off the device, a privacy-respecting alternative to cloud call handling. Realizing that promise takes a model that can do two things well: reply within the owner's authority during the call, and afterward tell the owner what happened and what needs attention. Released small models do neither reliably - they raise unwarranted alarms about ordinary callers and write summaries that drop material conditions or report a caller's claims as fact - and real calls supply no reference replies or summaries to learn from. We construct supervision for both tasks from synthetic calls under an information-boundary constraint: a caller writer works from a private scenario brief, a reply writer sees only the conversation up to the current turn, and a summary writer sees the completed transcript. Separate model-based reviews of the full dialogue and the summary decide which calls enter training, yielding a corpus of 3,395 calls with 15,102 targets (11,707 reply, 3,395 summary). Training a single shared LoRA adapter on both tasks, we…
PDF arXiv
arXiv cs.SDOct 18
Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody
Luigi Rivolta
People look up a song by humming it, but also by whistling it or singing it with the wrong words. Published query-by-humming systems and benchmarks focus on hums and sung queries, and no public dataset pairs whistles or wrong-lyrics singing with song recordings. We study both query types with a melody-only retrieval system: a key-normalized pitch track embedded by a 5.2M-parameter contour encoder, with a spectral-peak tracker for whistles. First, a physics-grounded generator renders synthetic whistles and nonsense-syllable singing from the vocal melodies of 11,083 songs. Training on them raises top-1 retrieval of held-out synthetic queries against an 11,050-song gallery from 0.180 to 0.251 for whistles and from 0.193 to 0.264 for wrong-lyrics singing, but the gain does not transfer: on the MLEnd test performers the best synthetic-only model gets 210 of 339 real whistles right at top-1, against 214 for the baseline. Second, a closed-set fine-tune on real MLEnd whistles with performer-disjoint splits raises whistle-to-hum top-1 from 0.631 (95% CI 0.581-0.681) to 0.673 (0.622-0.723), on known songs only. Third, two further generations of synthetic queries also fail to transfer:…
PDF arXiv
arXiv cs.CVOct 18
CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation
Jaehong Myung, Seungho Eum, Taegon Kim, Hyunjung Chung, Unsang Park
General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.
PDF arXiv
arXiv cs.CVOct 18
Stable Signer V3: 4D Deformable Primitive Splatting Is All You Need for Sign Language Production
Sen Fang, Liwei Che, Dimitris N. Metaxas
We present Stable Signer V3, an interactive high-resolution sign language avatar system built on 4D Deformable Primitive Splatting. It addresses three limitations of existing avatars: restricted facial expression, inaccurate identity-specific body deformation, and loss of fine appearance details. Shared Face Primitive Splatting adds transferable superquadric primitives for richer expression control; Test-Time Body Primitive Splatting learns an identity-specific virtual muscle layer for more faithful motion; and Test-Time Tunneling Super-Resolution uses TunnelingNet to recover identity-specific high-resolution details. Both test-time modules are optimized once per signer identity; inference then runs entirely from the stored identity parameters. Together, they reduce facial geometry error by 71.4%, body-boundary error by 35.1% across ten identities, and high-resolution RGB reconstruction error by 24.2% on held-out evaluations, while retaining interactive streaming.
PDF arXiv

Papers

Topics in this field

text speech · automatic speech · speech models255 papers34 this week
visual speech · speech gesture generation · audio driven76 papers17 this week
audio language models · language models audio · large audio language71 papers13 this week
duplex speech models · duplex spoken language · duplex speech76 papers11 this week
spatial audio visual · audio visual dataset · audiovisual64 papers8 this week
music models · text music generation · music annotation45 papers6 this week

Recently funded

Market

VoicePatrolSeed · Oct 12$3.4M
RifflectPre-seed · Oct 9undisclosed
SoniloSeed · Oct 8$11M
VoccaSeries A · Oct 7$20M
DeepslateSeed · Oct 7$8.6M

Active contributors · 90 days

Shinji WatanabeLong Listening Thoughts: Eliciting Open Auditory Reasoning with Deliberative Perception and Cognitive Refinement5
Zhizheng WuConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence5
Joon Son ChungJAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents5
Chao ZhangFull-Duplex Interaction with Speech, Expressions and Gestures5
Dinesh ManochaMultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue5
Petr MotlicekBeyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS4

Activity · 12 weeks

Coming with more sources: repos

Open problems in Voice AI

Spatial audio alignment in joint video-audio generation3 papers raise it · 3 address it
Immersive applications like AR/VR require stereo audio that dynamically matches visual source motion, which current models often lack.
- Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationraises · addresses
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Real-time joint audio-visual generation2 papers raise it · 2 address it
Interactive applications need low-latency synthesis to keep pace with user input and camera motion.
- WorldSonus: Bringing Sound to Worldsraises · addresses
- HelixWorld: A Real-time Interactive Audio-Visual World Modelraises · addresses
Multi-stem audio separation and generation1 paper raises it · 1 addresses it
Rendering dynamic scenes requires individual control over sound sources, but most models output only mixed audio.
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelraises · addresses
Event-level audio-video consistency evaluation1 paper raises it · 1 addresses it
Complex scenes with multi-event composition and asymmetric cross-modal relations need fine-grained metrics beyond whole-clip assessments.
- AVC-Reward: Event-Centric Audio-Video Consistency Evaluationraises · addresses
Accent leakage in cross-lingual voice cloning1 paper raises it · 1 addresses it
Zero-shot cross-lingual TTS often inherits the reference speaker's accent, degrading naturalness in the target language.
- Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloningraises · addresses

Named by an AI from these papers' abstracts.

Recent papers

All in Research →

arXiv cs.SDOct 19
Manifold-Constrained Residual Adaptation with Frame-Adaptive Layer Aggregation for Speaker Verification
Zezhong Jin, Xiaoyu Wang, Zhe Li, Chong-Xin Gan, Siqing Qin +2
Large self-supervised speech models provide powerful representations for speaker verification, but full fine-tuning is expensive. Most conventional adapters insert small trainable modules to modify the output of each layer, while keeping the original residual connections unchanged. Building on our preliminary study of manifold-constrained Hyper-Connections in speaker encoders, we explore parameter-efficient adaptation of both residual paths and feature selection across layers. We first propose a Manifold-Constrained Residual Adapter (MCRA), which splits the hidden dimensions of each Transformer residual state into N groups and uses a small Sinkhorn-normalized matrix to exchange information among them. A learnable gate controls the balance between the original and mixed residual paths. The gate adds only one scalar parameter at each residual connection and initially favors the original path, allowing gradual adaptation. We further propose Frame-Adaptive Layer Aggregation (FALA), an attention-based aggregation method that uses a single learned query shared across all utterances to compute layer weights separately for each speech frame. Because the layer representations depend on…
PDF arXiv
arXiv cs.LGOct 19
Text Dominance in Paralinguistic Conflict: Tracing Suppression Through Audio LLM Circuits
Ankur Banerjee, Vishnu Guntakala, Danity Pike, Advay Roongta, Kevin Yang +1
Audio LLMs deployed in settings such as crisis hotlines and customer support are only useful beyond speech-to-text pipelines if they use acoustic cues rather than the transcript alone. We test whether they prioritise transcript content over paralinguistic cues (emotion, speaker identity, sarcasm) and trace the mechanism when they do. Across six audio LLMs under LISTEN-style conflict conditions, the mean text-dominance rate on mismatched text+audio items is 0.78 (0.86 excluding sarcasm). In the most text-dominant screened model, probing shows speaker identity remains linearly decodable at ceiling at the language model's input, so the failure is not perceptual. Back-patching, reported to repair lexical text dominance, has no measurable effect here; but it also fails on its own source task in our replication (n=300), so we treat its null as uninformative rather than as evidence of non-transfer. Attention knockout partially restores audio-grounded output by blocking the claim from propagating through intermediate tokens during open-ended generation, whereas blocking the answer position's direct connection to the conflicting claim has no effect. The claim's influence therefore travels…
PDF arXiv
arXiv cs.CLOct 18
Quality-Controlled Reply and Summary Data for Post-Training Small Language Models in AI Call-Screening Agents
Jiaqi Gan, Tsang Ng, Ankit Raj, Tommy Duong, Xingyu Shen +3
Small language models have improved to the point where a phone can plausibly screen its own calls, answering on an absent owner's behalf without sending call content off the device, a privacy-respecting alternative to cloud call handling. Realizing that promise takes a model that can do two things well: reply within the owner's authority during the call, and afterward tell the owner what happened and what needs attention. Released small models do neither reliably - they raise unwarranted alarms about ordinary callers and write summaries that drop material conditions or report a caller's claims as fact - and real calls supply no reference replies or summaries to learn from. We construct supervision for both tasks from synthetic calls under an information-boundary constraint: a caller writer works from a private scenario brief, a reply writer sees only the conversation up to the current turn, and a summary writer sees the completed transcript. Separate model-based reviews of the full dialogue and the summary decide which calls enter training, yielding a corpus of 3,395 calls with 15,102 targets (11,707 reply, 3,395 summary). Training a single shared LoRA adapter on both tasks, we…
PDF arXiv
arXiv cs.SDOct 18
Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody
Luigi Rivolta
People look up a song by humming it, but also by whistling it or singing it with the wrong words. Published query-by-humming systems and benchmarks focus on hums and sung queries, and no public dataset pairs whistles or wrong-lyrics singing with song recordings. We study both query types with a melody-only retrieval system: a key-normalized pitch track embedded by a 5.2M-parameter contour encoder, with a spectral-peak tracker for whistles. First, a physics-grounded generator renders synthetic whistles and nonsense-syllable singing from the vocal melodies of 11,083 songs. Training on them raises top-1 retrieval of held-out synthetic queries against an 11,050-song gallery from 0.180 to 0.251 for whistles and from 0.193 to 0.264 for wrong-lyrics singing, but the gain does not transfer: on the MLEnd test performers the best synthetic-only model gets 210 of 339 real whistles right at top-1, against 214 for the baseline. Second, a closed-set fine-tune on real MLEnd whistles with performer-disjoint splits raises whistle-to-hum top-1 from 0.631 (95% CI 0.581-0.681) to 0.673 (0.622-0.723), on known songs only. Third, two further generations of synthetic queries also fail to transfer:…
PDF arXiv
arXiv cs.CVOct 18
CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation
Jaehong Myung, Seungho Eum, Taegon Kim, Hyunjung Chung, Unsang Park
General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.
PDF arXiv
arXiv cs.CVOct 18
Stable Signer V3: 4D Deformable Primitive Splatting Is All You Need for Sign Language Production
Sen Fang, Liwei Che, Dimitris N. Metaxas
We present Stable Signer V3, an interactive high-resolution sign language avatar system built on 4D Deformable Primitive Splatting. It addresses three limitations of existing avatars: restricted facial expression, inaccurate identity-specific body deformation, and loss of fine appearance details. Shared Face Primitive Splatting adds transferable superquadric primitives for richer expression control; Test-Time Body Primitive Splatting learns an identity-specific virtual muscle layer for more faithful motion; and Test-Time Tunneling Super-Resolution uses TunnelingNet to recover identity-specific high-resolution details. Both test-time modules are optimized once per signer identity; inference then runs entirely from the stored identity parameters. Together, they reduce facial geometry error by 71.4%, body-boundary error by 35.1% across ten identities, and high-resolution RGB reconstruction error by 24.2% on held-out evaluations, while retaining interactive streaming.
PDF arXiv

Fields · AI · updated 45m ago

Voice AI

Speech recognition, speech synthesis, voice agents, spoken dialogue and audio language models.

89papers this week−23.9% vs last week

All papers · 587 With code · 14

Newest Most relevant

555 papers this month