Papers

Field

Top tools here

All tools

NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%
BentoML+34% downloads · 46k/wk+34.2%
LMDeploy+36% downloads · 7k/wk+35.5%
MLX+22% downloads · 356.8k/wk+21.7%
LiteLLM17 releases this week

Top tools here

Open problems in Inference & serving

Efficient long-context inference with limited memory3 papers raise it · 3 address it
Long trajectories or sequences strain KV cache and recurrent state memory, making serving costly or infeasible.
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in existing systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent or looped models1 paper raises it · 1 addresses it
Models trained on short sequences often fail to extrapolate to much longer sequences, limiting practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Quantization-induced accuracy loss in recurrent states2 papers raise it · 2 address it
Low-precision quantization of persistent states degrades accuracy due to error propagation over time.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Adaptive computation allocation for parallel token generation1 paper raises it · 1 addresses it
Uniform computation depth in diffusion models wastes resources on easy tokens and under-serves hard ones.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Arda IndustriesSeries A · Oct 26$160M
AilyticsSeries A · Oct 26$7.4M
Chaoqiumian TechnologySeed · Oct 26undisclosed
Beijing Token GongchangStage not given · Oct 26undisclosed
XiaoshanshijichengStage not given · Oct 26undisclosed

Recently funded

Topics in this field

diffusion language models · models diffusion language · diffusion language model168 papers30 this week
catastrophic forgetting · continual learning · forgetting182 papers25 this week
kv cache compression · kv cache · kv caches267 papers20 this week
low bit quantization · quantization aware training · training quantization176 papers18 this week
sparse autoencoders saes · sparse autoencoders · sparse autoencoder140 papers17 this week
low rank adaptation · rank adaptation lora · rank adaptation127 papers16 this week

Topics in this field

Active contributors · 90 days

Yu WangRewriting Scaling Laws for Synthetic Data with Compute and Data Constraints7
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.IRSep 25
KuaFu: Compressing Long User Behavior into Understanding at Billion Scale
Jiahao Hui, Lin Zhu, Yishen Hu, Jingdong Shu, Zetai Jiang +6
Conversational agents, generative recommenders, and personalized advertising all rest on one capability: understanding each user from raw behavior. Prevailing industrial practice is task-specific: for each task, a relevant subsequence is extracted from the full history and a dedicated model trained on it. In production it hits two bottlenecks. First, even after filtering, a single-task sequence stays extremely long: content-interest summarization reads several hundred items per user, tens of thousands of tokens once serialized as prompt text. Second, profiles are refreshed routinely: a billion users weekly, roughly 100K QPM in aggregate, which under a fixed GPU budget sets a hard throughput floor. Compression is therefore mandatory, yet truncation or coarse compression can silently distort the profile, introducing four hallucination types (fabrication, omission, date misattribution, broken logic) that, with no way to evaluate the compressed representation itself, surface only as diffuse degradation in downstream metrics. We present KuaFu, a unified behavior-compression layer whose minimal unit is one behavior item. A two-axis projector compresses each item into 2-4 tokens of…
PDF arXiv
arXiv cs.LGSep 25
Metacognitive Selective Ensemble for Mobile Systems
Sungmin Lee, Kichang Lee, Joonhee Lee, JaeYeon Park, Songkuk Kim +1
Deep ensembles improve robustness in mobile sensing, but repeatedly executing many models over continuous sensor streams is costly. Selecting only a few members reduces this cost, yet adaptive selection often requires additional model execution to obtain reliable evidence about inactive candidates. We present MetaSE, an active ensemble framework that exploits short-term persistence in per-model reliability. MetaSE maintains a small active set across windows, uses post-execution evidence to reject unreliable members, and invokes lightweight routing only when replacement is needed. This stateful design accesses the diversity of a larger pool without repeated full-pool evaluation. Across four HAR datasets and four model architectures, MetaSE consistently improves over a fixed three-model ensemble and achieves accuracy comparable to substantially more expensive adaptive and full-ensemble inference. On a Raspberry Pi 4B, MetaSE is 2.7x faster and uses 69% less memory than full ten-model inference.
PDF arXiv
arXiv cs.CLSep 25
G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation
Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai +6
Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G$^2$PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G$^2$PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model…
PDF arXiv
arXiv cs.LGSep 25
Does Uniform Discrete Diffusion Need Time?
Chunsan Hong, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Jong Chul Ye +1
Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning, but we find that it can often be unnecessary in practice. In this paper, we first show that the population-optimal UDM predictor generally depends on time: time controls how much the model should trust the observed context. We then show that this dependence can become negligible in finite-data settings relevant to language. When a corrupted training sequence remains much closer to its original clean sequence than to competing training sequences, the empirical-optimal predictor is nearly insensitive to time over most of the diffusion trajectory, where the guarantee weakens toward the high-noise endpoint. Empirically, trained language UDMs exhibit limited time sensitivity over most of the trajectory, while time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives. These results challenge the use of explicit time conditioning in UDMs: although the population optimum depends on time, explicitly conditioning on it may often be unnecessary in practice.
PDF arXiv
arXiv cs.LGSep 25
A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields
Tiexin Ding
Pooled statistics of Transformer weights obscure how magnitude is distributed across functional channels, while individual weights are too numerous to compare directly. We study the mesoscopic level between them: row and column scale fields, the median-centred log-RMS profiles of a weight matrix over its channels, which together with a global scale and a full balanced core represent the matrix exactly. Across public Pythia checkpoints at four sizes and controlled runs from three initialization families, balancing reveals similar measured core magnitude profiles. A mixture bridge, with its form fixed before the analysis and its coefficients fitted, predicts the pooled-shape departure from field width on held-out runs and data arms of the controlled grid. The indexed fields retain further structure: they align across projections that share a functional channel, and query/key profiles follow reassigned RoPE frequencies rather than fixed matrix coordinates. Training trajectories show early field formation followed by component-dependent broadening or recession. Extending the channel-based analysis to AdamW's second moment reveals related functional organization in its log-space row…
PDF arXiv
arXiv cs.LGSep 25
Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Miao Lu, Zeyu Bian, Kaiyue Wen, Beining Wu, Siyu Chen +2
The empirical success of pretraining large language models has inspired a deeper investigation into the underlying loss landscapes and the optimization dynamics. Recent empirical and theoretical study suggest that the training loss landscape often exhibits a "river-valley" structure, which features a low-loss manifold (river) flanked by sharp orthogonal directions with higher loss (mountains). In the long term, the optimization progress is determined primarily by the progress along the river. Within such a landscape, gradient descent with large learning rates can move faster along the river despite high apparent loss due to vertical oscillations, while a subsequent sharp decay in the learning rate suppresses these oscillations, revealing genuine optimization progress. This explains the recent success of warmup-stable-decay (WSD) learning rate scheduler which, unlike cosine scheduling, keeps stable high learning rate and decays before producing intermediate checkpoints. Building on this foundation, in this work we take a step further and study the role of momentum within such a loss landscape. We establish theoretical analysis that characterizes how momentum accelerates…
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%
BentoML+34% downloads · 46k/wk+34.2%
LMDeploy+36% downloads · 7k/wk+35.5%
MLX+22% downloads · 356.8k/wk+21.7%
LiteLLM17 releases this week

Recently funded

Market

Arda IndustriesSeries A · Oct 26$160M
AilyticsSeries A · Oct 26$7.4M
Chaoqiumian TechnologySeed · Oct 26undisclosed
Beijing Token GongchangStage not given · Oct 26undisclosed
XiaoshanshijichengStage not given · Oct 26undisclosed

Topics in this field

diffusion language models · models diffusion language · diffusion language model168 papers30 this week
catastrophic forgetting · continual learning · forgetting182 papers25 this week
kv cache compression · kv cache · kv caches267 papers20 this week
low bit quantization · quantization aware training · training quantization176 papers18 this week
sparse autoencoders saes · sparse autoencoders · sparse autoencoder140 papers17 this week
low rank adaptation · rank adaptation lora · rank adaptation127 papers16 this week

Active contributors · 90 days

Yu WangRewriting Scaling Laws for Synthetic Data with Compute and Data Constraints7
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Recent papers

All papers in Research

arXiv cs.IRSep 25
KuaFu: Compressing Long User Behavior into Understanding at Billion Scale
Jiahao Hui, Lin Zhu, Yishen Hu, Jingdong Shu, Zetai Jiang +6
Conversational agents, generative recommenders, and personalized advertising all rest on one capability: understanding each user from raw behavior. Prevailing industrial practice is task-specific: for each task, a relevant subsequence is extracted from the full history and a dedicated model trained on it. In production it hits two bottlenecks. First, even after filtering, a single-task sequence stays extremely long: content-interest summarization reads several hundred items per user, tens of thousands of tokens once serialized as prompt text. Second, profiles are refreshed routinely: a billion users weekly, roughly 100K QPM in aggregate, which under a fixed GPU budget sets a hard throughput floor. Compression is therefore mandatory, yet truncation or coarse compression can silently distort the profile, introducing four hallucination types (fabrication, omission, date misattribution, broken logic) that, with no way to evaluate the compressed representation itself, surface only as diffuse degradation in downstream metrics. We present KuaFu, a unified behavior-compression layer whose minimal unit is one behavior item. A two-axis projector compresses each item into 2-4 tokens of…
PDF arXiv
arXiv cs.LGSep 25
Metacognitive Selective Ensemble for Mobile Systems
Sungmin Lee, Kichang Lee, Joonhee Lee, JaeYeon Park, Songkuk Kim +1
Deep ensembles improve robustness in mobile sensing, but repeatedly executing many models over continuous sensor streams is costly. Selecting only a few members reduces this cost, yet adaptive selection often requires additional model execution to obtain reliable evidence about inactive candidates. We present MetaSE, an active ensemble framework that exploits short-term persistence in per-model reliability. MetaSE maintains a small active set across windows, uses post-execution evidence to reject unreliable members, and invokes lightweight routing only when replacement is needed. This stateful design accesses the diversity of a larger pool without repeated full-pool evaluation. Across four HAR datasets and four model architectures, MetaSE consistently improves over a fixed three-model ensemble and achieves accuracy comparable to substantially more expensive adaptive and full-ensemble inference. On a Raspberry Pi 4B, MetaSE is 2.7x faster and uses 69% less memory than full ten-model inference.
PDF arXiv
arXiv cs.CLSep 25
G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation
Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai +6
Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G$^2$PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G$^2$PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model…
PDF arXiv
arXiv cs.LGSep 25
Does Uniform Discrete Diffusion Need Time?
Chunsan Hong, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Jong Chul Ye +1
Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning, but we find that it can often be unnecessary in practice. In this paper, we first show that the population-optimal UDM predictor generally depends on time: time controls how much the model should trust the observed context. We then show that this dependence can become negligible in finite-data settings relevant to language. When a corrupted training sequence remains much closer to its original clean sequence than to competing training sequences, the empirical-optimal predictor is nearly insensitive to time over most of the diffusion trajectory, where the guarantee weakens toward the high-noise endpoint. Empirically, trained language UDMs exhibit limited time sensitivity over most of the trajectory, while time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives. These results challenge the use of explicit time conditioning in UDMs: although the population optimum depends on time, explicitly conditioning on it may often be unnecessary in practice.
PDF arXiv
arXiv cs.LGSep 25
A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields
Tiexin Ding
Pooled statistics of Transformer weights obscure how magnitude is distributed across functional channels, while individual weights are too numerous to compare directly. We study the mesoscopic level between them: row and column scale fields, the median-centred log-RMS profiles of a weight matrix over its channels, which together with a global scale and a full balanced core represent the matrix exactly. Across public Pythia checkpoints at four sizes and controlled runs from three initialization families, balancing reveals similar measured core magnitude profiles. A mixture bridge, with its form fixed before the analysis and its coefficients fitted, predicts the pooled-shape departure from field width on held-out runs and data arms of the controlled grid. The indexed fields retain further structure: they align across projections that share a functional channel, and query/key profiles follow reassigned RoPE frequencies rather than fixed matrix coordinates. Training trajectories show early field formation followed by component-dependent broadening or recession. Extending the channel-based analysis to AdamW's second moment reveals related functional organization in its log-space row…
PDF arXiv
arXiv cs.LGSep 25
Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
Miao Lu, Zeyu Bian, Kaiyue Wen, Beining Wu, Siyu Chen +2
The empirical success of pretraining large language models has inspired a deeper investigation into the underlying loss landscapes and the optimization dynamics. Recent empirical and theoretical study suggest that the training loss landscape often exhibits a "river-valley" structure, which features a low-loss manifold (river) flanked by sharp orthogonal directions with higher loss (mountains). In the long term, the optimization progress is determined primarily by the progress along the river. Within such a landscape, gradient descent with large learning rates can move faster along the river despite high apparent loss due to vertical oscillations, while a subsequent sharp decay in the learning rate suppresses these oscillations, revealing genuine optimization progress. This explains the recent success of warmup-stable-decay (WSD) learning rate scheduler which, unlike cosine scheduling, keeps stable high learning rate and decays before producing intermediate checkpoints. Building on this foundation, in this work we take a step further and study the role of momentum within such a loss landscape. We establish theoretical analysis that characterizes how momentum accelerates…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 22m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

216papers this week−36.3% vs last week

All papers · 1.9k With code · 67

Newest Most relevant

1.6k papers this month

Open problems in Inference & serving

Efficient long-context inference with limited memory3 papers raise it · 3 address it
Long trajectories or sequences strain KV cache and recurrent state memory, making serving costly or infeasible.
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in existing systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent or looped models1 paper raises it · 1 addresses it
Models trained on short sequences often fail to extrapolate to much longer sequences, limiting practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Quantization-induced accuracy loss in recurrent states2 papers raise it · 2 address it
Low-precision quantization of persistent states degrades accuracy due to error propagation over time.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Adaptive computation allocation for parallel token generation1 paper raises it · 1 addresses it
Uniform computation depth in diffusion models wastes resources on easy tokens and under-serves hard ones.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses

Named by an AI from these papers' abstracts.