Papers

Field

Top tools here

All tools

NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%
BentoML+34% downloads · 46k/wk+34.2%
LMDeploy+36% downloads · 7k/wk+35.5%
MLX+22% downloads · 356.8k/wk+21.7%
LiteLLM17 releases this week

Top tools here

Open problems in Inference & serving

Efficient long-context inference with limited memory3 papers raise it · 3 address it
Long trajectories or sequences strain KV cache and recurrent state memory, making serving costly or infeasible.
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in existing systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent or looped models1 paper raises it · 1 addresses it
Models trained on short sequences often fail to extrapolate to much longer sequences, limiting practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Quantization-induced accuracy loss in recurrent states2 papers raise it · 2 address it
Low-precision quantization of persistent states degrades accuracy due to error propagation over time.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Adaptive computation allocation for parallel token generation1 paper raises it · 1 addresses it
Uniform computation depth in diffusion models wastes resources on easy tokens and under-serves hard ones.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Arda IndustriesSeries A · Oct 26$160M
InevitableStage not given · Oct 26$16.8M
INFIFORCEStage not given · Oct 26$15M
AilyticsSeries A · Oct 26$7.4M
Symx.aiStage not given · Oct 26$6M

Recently funded

Topics in this field

diffusion language models · models diffusion language · diffusion language model168 papers26 this week
catastrophic forgetting · continual learning · forgetting182 papers25 this week
low bit quantization · quantization aware training · training quantization176 papers18 this week
sparse autoencoders saes · sparse autoencoders · sparse autoencoder140 papers16 this week
kv cache compression · kv cache · kv caches267 papers14 this week
low rank adaptation · rank adaptation lora · rank adaptation127 papers14 this week

Topics in this field

Active contributors · 90 days

Yu WangRewriting Scaling Laws for Synthetic Data with Compute and Data Constraints7
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGSep 23
CFD Correction of Open Tip Clearance Flow in a Compressor Cascade Using VAE Latent Space Adaptation
Xiang Zuo, Hefang Deng, Caiyan Chen, Honglin He, Mingmin Zhu +2
CFD predictions of open tip clearance flow in compressor cascades are subject to discrepancies relative to experiments, while experimental observations are sparse and high-resolution experimental ground truth is unavailable. This study proposes a non-intrusive correction method based on a variational autoencoder (VAE) and latent-space adaptation. A VAE is first trained using a dataset of 166 parametrically sampled CFD total pressure loss fields to learn a low-dimensional statistical representation of these fields. The VAE is then frozen, and a low-rank latent-space adapter is trained using only 12 paired CFD--experiment operating conditions. An observation operator maps the corrected high-resolution fields to the experimental observation space, allowing supervision to be applied only at the available measurement locations and within the measured pitchwise windows. In the current 12-fold cross-validation, the mean absolute error decreases from 0.1335 to 0.0473, the root mean square error from 0.1717 to 0.0621, and the relative $L_2$ error from 0.5108 to 0.1871. These results indicate that the method improves agreement between CFD predictions and sparse experimental observations of…
PDF arXiv
arXiv cs.LGSep 23
Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters
Egor Romanyukov, Timofey Novikov, Timur Shokarov, Elizaveta Zorkina, Anastasia Palienko +1
Step Law gives power-law formulas for the optimal peak learning rate eta* and batch size B* when pre-training language models. It was calibrated on models between 59M and 1B parameters; the small-model regime N < 59M was never tested empirically by its authors. This regime matters for single-GPU training, interpretability research, educational experiments, and settings where larger models are infeasible on memory or cost grounds. We test whether Step Law transfers to small language models. We consider three outcomes: H1, the original coefficients work directly; H2, the power-law form holds but with different coefficients; and H3, a power law does not describe the optima in this regime. All experiments use a single nanoGPT/TinyStories pipeline with a 2048-token BPE vocabulary, AdamW, and a warmup-cosine schedule. The optimum for each (N, D) cell is extracted from the loss surface L(eta, B) via a local quadratic approximation in log-log coordinates over the smoothed training loss. The final dataset contains 29 unique (N, D) cells and 935 analysis-ready runs. The main refit uses 25 cells (815 runs) in the working range 4 <= D/N <= 600. On the pooled data we accept H2: the functional…
PDF arXiv
arXiv cs.LGSep 23
PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
Shanfeng Huang, Rongqian Zhang, Zhou Fang, Song Xiao, Hai Du
Mixture-of-Experts (MoE) models have emerged as an effective paradigm for scaling large language models, yet their massive parameter footprint poses severe memory challenges for deployment. Existing compression methods typically treat expert pruning and merging as disjoint operations and rely on greedy heuristics, often leading to irreversible information loss or representational pollution. In this paper, we propose PRIME, a unified optimization framework for MoE compression via adaptive PRunIng and MErging. PRIME casts the discrete prune-merge assignment step as a global Integer Linear Programming over a frequency-weighted surrogate objective, jointly selecting pruning and merging decisions while accounting for expert usage. We motivate the frequency-aware objective through an error-bound analysis under fixed representative assignments, and show that the formulation favors high-frequency experts as centroids, promoting routing stability. Extensive experiments on Qwen3-30B-A3B and Qwen3.5-122B-A10B demonstrate that PRIME achieves strong accuracy retention across knowledge, coding, math, and long-context tasks, while serving benchmarks of merged models quantify the memory and…
PDF arXiv
arXiv cs.LGSep 23
EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management
Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao +10
Embodied reinforcement learning (RL) improves model capabilities with a pipeline of environment simulation, action generation, and model updates. These stages show heterogeneous CPU and GPU demands, making efficient resource utilization difficult. Recent systems overlap rollout (simulation and generation) with training for efficiency, but exclusive GPU allocation and synchronized barrier in rollout still leave substantial hardware resource waste. In this paper, we present EBRL, an asynchronous embodied RL training system with two core techniques. The asynchronous pipelined scheduler overlaps rollout and training, pipelines simulation and generation across environment groups, and carries out each environment independently, eliminating synchronization stalls. The fine-grained resource manager pools CPU cores and GPU streaming multiprocessors, and uses stage profiles and runtime feedback to adjust resource quotas and batch sizes to meet the shifting demands among stages. We implement EBRL on RLinf and evaluate it with four embodied policies and four simulation benchmarks across heterogeneous GPU testbeds. Experiments show that EBRL achieves 1.30-3.47 times the end-to-end rollout…
PDF arXiv
arXiv cs.CVSep 23
DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video
Taeyoun Kwon, Seungjin Kim, Hyeonyu Kim, Moon Hwan Kim
Recent video-language models increasingly adopt hybrid architectures that interleave linear and full attention layers for efficient long-context processing. While the recurrent state of linear attention remains fixed in size, the KV cache of full attention continues to grow with the video stream, making eviction necessary under a bounded memory budget. The key challenge in streaming is that eviction must occur before the question arrives, so what to retain has to be decided without the question. Existing eviction methods derive token scores from the KV cache itself, using position, attention, or key-value representations, and attention-based scores further require proxy queries or extra computation. Hybrid backbones offer another source of signal. In gated-delta linear attention, the recurrent state is updated by the residual between each input and what can already be retrieved from the state, so its change over a chunk of frames reflects how much new information the chunk brings. We propose DeltaS, a query-agnostic, training-free method that retains video chunks inducing larger normalized state change, or state drift. In a controlled comparison with the budget and retention…
PDF arXiv
arXiv cs.CLSep 23
When Parallel Drafter Meets Parallel Speculative Decoding
Fuliang Liu, Xue Li, Kun Qian, Zhibin Wang, Wanchun Dou +2
DSpark-style parallel drafters have made speculative decoding highly effective, yet their draft phase remains serialized on the critical path of every round. Parallel speculative decoding (PSD) overlaps drafting with verification, yet existing methods must guess the accepted prefix and bonus token in advance: a wrong guess reverts the whole batch to serial drafting. We present DPara, a PSD framework that reuses effective parallel drafters yet guarantees backbone--verification overlap in every round, thereby eliminating this probabilistic fallback altogether. While the target verifies, DPara's diffusion backbone precomputes draft representations for every acceptance boundary with the bonus left unspecified; a lightweight autoregressive head then combines the revealed verification outcome with the matching precomputed representation to emit the next round's draft tokens almost instantly---fully parallelizing the dominant backbone forward with verification and leaving only the negligible head cost serial. Experiments on Qwen3-8B and Qwen3-14B across seven math, coding, and chat benchmarks show that DPara achieves average speedups of $3.21\times$ and $3.52\times$ over autoregressive…
PDF arXiv

Papers

Recently funded

Market

Arda IndustriesSeries A · Oct 26$160M
InevitableStage not given · Oct 26$16.8M
INFIFORCEStage not given · Oct 26$15M
AilyticsSeries A · Oct 26$7.4M
Symx.aiStage not given · Oct 26$6M

Top tools here

All tools

NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%
BentoML+34% downloads · 46k/wk+34.2%
LMDeploy+36% downloads · 7k/wk+35.5%
MLX+22% downloads · 356.8k/wk+21.7%
LiteLLM17 releases this week

Topics in this field

diffusion language models · models diffusion language · diffusion language model168 papers26 this week
catastrophic forgetting · continual learning · forgetting182 papers25 this week
low bit quantization · quantization aware training · training quantization176 papers18 this week
sparse autoencoders saes · sparse autoencoders · sparse autoencoder140 papers16 this week
kv cache compression · kv cache · kv caches267 papers14 this week
low rank adaptation · rank adaptation lora · rank adaptation127 papers14 this week

Active contributors · 90 days

Yu WangRewriting Scaling Laws for Synthetic Data with Compute and Data Constraints7
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Recent papers

All papers in Research

arXiv cs.LGSep 23
CFD Correction of Open Tip Clearance Flow in a Compressor Cascade Using VAE Latent Space Adaptation
Xiang Zuo, Hefang Deng, Caiyan Chen, Honglin He, Mingmin Zhu +2
CFD predictions of open tip clearance flow in compressor cascades are subject to discrepancies relative to experiments, while experimental observations are sparse and high-resolution experimental ground truth is unavailable. This study proposes a non-intrusive correction method based on a variational autoencoder (VAE) and latent-space adaptation. A VAE is first trained using a dataset of 166 parametrically sampled CFD total pressure loss fields to learn a low-dimensional statistical representation of these fields. The VAE is then frozen, and a low-rank latent-space adapter is trained using only 12 paired CFD--experiment operating conditions. An observation operator maps the corrected high-resolution fields to the experimental observation space, allowing supervision to be applied only at the available measurement locations and within the measured pitchwise windows. In the current 12-fold cross-validation, the mean absolute error decreases from 0.1335 to 0.0473, the root mean square error from 0.1717 to 0.0621, and the relative $L_2$ error from 0.5108 to 0.1871. These results indicate that the method improves agreement between CFD predictions and sparse experimental observations of…
PDF arXiv
arXiv cs.LGSep 23
Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters
Egor Romanyukov, Timofey Novikov, Timur Shokarov, Elizaveta Zorkina, Anastasia Palienko +1
Step Law gives power-law formulas for the optimal peak learning rate eta* and batch size B* when pre-training language models. It was calibrated on models between 59M and 1B parameters; the small-model regime N < 59M was never tested empirically by its authors. This regime matters for single-GPU training, interpretability research, educational experiments, and settings where larger models are infeasible on memory or cost grounds. We test whether Step Law transfers to small language models. We consider three outcomes: H1, the original coefficients work directly; H2, the power-law form holds but with different coefficients; and H3, a power law does not describe the optima in this regime. All experiments use a single nanoGPT/TinyStories pipeline with a 2048-token BPE vocabulary, AdamW, and a warmup-cosine schedule. The optimum for each (N, D) cell is extracted from the loss surface L(eta, B) via a local quadratic approximation in log-log coordinates over the smoothed training loss. The final dataset contains 29 unique (N, D) cells and 935 analysis-ready runs. The main refit uses 25 cells (815 runs) in the working range 4 <= D/N <= 600. On the pooled data we accept H2: the functional…
PDF arXiv
arXiv cs.LGSep 23
PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
Shanfeng Huang, Rongqian Zhang, Zhou Fang, Song Xiao, Hai Du
Mixture-of-Experts (MoE) models have emerged as an effective paradigm for scaling large language models, yet their massive parameter footprint poses severe memory challenges for deployment. Existing compression methods typically treat expert pruning and merging as disjoint operations and rely on greedy heuristics, often leading to irreversible information loss or representational pollution. In this paper, we propose PRIME, a unified optimization framework for MoE compression via adaptive PRunIng and MErging. PRIME casts the discrete prune-merge assignment step as a global Integer Linear Programming over a frequency-weighted surrogate objective, jointly selecting pruning and merging decisions while accounting for expert usage. We motivate the frequency-aware objective through an error-bound analysis under fixed representative assignments, and show that the formulation favors high-frequency experts as centroids, promoting routing stability. Extensive experiments on Qwen3-30B-A3B and Qwen3.5-122B-A10B demonstrate that PRIME achieves strong accuracy retention across knowledge, coding, math, and long-context tasks, while serving benchmarks of merged models quantify the memory and…
PDF arXiv
arXiv cs.LGSep 23
EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management
Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao +10
Embodied reinforcement learning (RL) improves model capabilities with a pipeline of environment simulation, action generation, and model updates. These stages show heterogeneous CPU and GPU demands, making efficient resource utilization difficult. Recent systems overlap rollout (simulation and generation) with training for efficiency, but exclusive GPU allocation and synchronized barrier in rollout still leave substantial hardware resource waste. In this paper, we present EBRL, an asynchronous embodied RL training system with two core techniques. The asynchronous pipelined scheduler overlaps rollout and training, pipelines simulation and generation across environment groups, and carries out each environment independently, eliminating synchronization stalls. The fine-grained resource manager pools CPU cores and GPU streaming multiprocessors, and uses stage profiles and runtime feedback to adjust resource quotas and batch sizes to meet the shifting demands among stages. We implement EBRL on RLinf and evaluate it with four embodied policies and four simulation benchmarks across heterogeneous GPU testbeds. Experiments show that EBRL achieves 1.30-3.47 times the end-to-end rollout…
PDF arXiv
arXiv cs.CVSep 23
DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video
Taeyoun Kwon, Seungjin Kim, Hyeonyu Kim, Moon Hwan Kim
Recent video-language models increasingly adopt hybrid architectures that interleave linear and full attention layers for efficient long-context processing. While the recurrent state of linear attention remains fixed in size, the KV cache of full attention continues to grow with the video stream, making eviction necessary under a bounded memory budget. The key challenge in streaming is that eviction must occur before the question arrives, so what to retain has to be decided without the question. Existing eviction methods derive token scores from the KV cache itself, using position, attention, or key-value representations, and attention-based scores further require proxy queries or extra computation. Hybrid backbones offer another source of signal. In gated-delta linear attention, the recurrent state is updated by the residual between each input and what can already be retrieved from the state, so its change over a chunk of frames reflects how much new information the chunk brings. We propose DeltaS, a query-agnostic, training-free method that retains video chunks inducing larger normalized state change, or state drift. In a controlled comparison with the budget and retention…
PDF arXiv
arXiv cs.CLSep 23
When Parallel Drafter Meets Parallel Speculative Decoding
Fuliang Liu, Xue Li, Kun Qian, Zhibin Wang, Wanchun Dou +2
DSpark-style parallel drafters have made speculative decoding highly effective, yet their draft phase remains serialized on the critical path of every round. Parallel speculative decoding (PSD) overlaps drafting with verification, yet existing methods must guess the accepted prefix and bonus token in advance: a wrong guess reverts the whole batch to serial drafting. We present DPara, a PSD framework that reuses effective parallel drafters yet guarantees backbone--verification overlap in every round, thereby eliminating this probabilistic fallback altogether. While the target verifies, DPara's diffusion backbone precomputes draft representations for every acceptance boundary with the bonus left unspecified; a lightweight autoregressive head then combines the revealed verification outcome with the matching precomputed representation to emit the next round's draft tokens almost instantly---fully parallelizing the dominant backbone forward with verification and leaving only the negligible head cost serial. Experiments on Qwen3-8B and Qwen3-14B across seven math, coding, and chat benchmarks show that DPara achieves average speedups of $3.21\times$ and $3.52\times$ over autoregressive…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 47m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

189papers this week−43.9% vs last week

All papers · 1.9k With code · 67

Newest Most relevant

1.6k papers this month

Open problems in Inference & serving

Efficient long-context inference with limited memory3 papers raise it · 3 address it
Long trajectories or sequences strain KV cache and recurrent state memory, making serving costly or infeasible.
- Cache Engineering for Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in existing systems.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Length generalization in recurrent or looped models1 paper raises it · 1 addresses it
Models trained on short sequences often fail to extrapolate to much longer sequences, limiting practical use.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
Quantization-induced accuracy loss in recurrent states2 papers raise it · 2 address it
Low-precision quantization of persistent states degrades accuracy due to error propagation over time.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Adaptive computation allocation for parallel token generation1 paper raises it · 1 addresses it
Uniform computation depth in diffusion models wastes resources on easy tokens and under-serves hard ones.
- ALoDLM: Adaptively Looped Diffusion Language Modelsraises · addresses

Named by an AI from these papers' abstracts.