Papers

Field

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

QuestorAIStage not given · Oct 25$3M
Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M

Recently funded

Topics in this field

diffusion language models · models diffusion language · diffusion language model165 papers31 this week
kv cache compression · kv cache · kv caches264 papers18 this week
catastrophic forgetting · continual learning · forgetting173 papers18 this week
low bit quantization · quantization aware training · training quantization172 papers15 this week
low rank adaptation · rank adaptation lora · rank adaptation123 papers13 this week
recurrent depth · looped language models · computational depth105 papers13 this week

Topics in this field

Active contributors · 90 days

Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Yu WangCache Engineering for Agents6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGSep 26
Age of Learning: Temporal Persistence of Prediction Errors as a Learning Signal
Chenyang Wang, Stefan Forsström, Roger Olsson, Di Yuan, Qing He
Current machine learning algorithms primarily rely on instantaneous signals such as loss, margin, and prediction confidence to characterize model behavior. These signals indicate how difficult a prediction is at the current optimization step, but they do not capture how long the model has remained incorrect. We study this temporal dimension of learning and introduce Age of Learning (AoL), a learning-state variable that measures the persistence of prediction errors over time. AoL increases while an error remains unresolved and resets when a correct prediction is achieved, thereby distinguishing persistent under-learning from transient mistakes. We develop AoL-based training strategies for both offline and streaming settings. In offline learning, sample-level AoL is accumulated over training and aggregated into class-level states that guide adaptive reweighting and resampling. In streaming learning, where full historical access is unavailable, we maintain lightweight class-level AoL states using current and buffered observations. Across long-tailed classification settings, AoL improves or matches standard training baselines, with larger benefits when learning difficulty persists…
PDF arXiv
arXiv cs.CLSep 26
HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion
Junxiang Qiu, Zhengsu Chen, Xinting Hu, Shuo Wang, Hengheng Zhang +4
Mixture-of-Experts (MoE) architectures have become a standard way to scale model capacity while keeping computation sparse, yet routing remains a key determinant of MoE quality and training behavior. Prior empirical studies suggest that MoE routing reflects input semantics and upstream computation across depth, but standard routers do not explicitly use the routing distributions produced by preceding layers. We propose HERO-MoE, Historical Expert ROuting with Scale-Preserving Fusion, a routing framework that injects historical routing priors into MoE routers by reusing detached, dense routing distributions collected from preceding MoE layers. The key idea is simple: HERO-MoE preserves the original token-conditioned routing branch and adds a residual historical routing contribution before the standard softmax and top-$k$ dispatch. To stabilize this historical signal, HERO-MoE introduces a scale-preserving fusion mechanism that matches the magnitude of historical routing memory to the current hidden representation and accounts for the number of visible historical layers, without introducing an auxiliary routing loss or a fusion-specific tuning parameter. By reusing routing…
PDF arXiv
arXiv cs.LGSep 26
DimPO: Dimensionality Reduction for Attention using Preference Optimization
Vojtěch Lanz, Yufei Cui, Prasanna Parthasarathi
A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model behavior. We ask whether preferences over keys and attention mass on the highest-weighted keys provide a better signal than matching the full attention distribution with KL divergence, especially in long-context settings. We introduce DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity. DimPO is trained offline from the attention patterns of a frozen language model, with one map per layer, shared by the query and the keys and trained separately from the other layers. Across LLaMA3.2-3B, LLaMA3.1-8B, Qwen2.5-7B, and Qwen3-4B Instruct models, pairwise preference objectives outperform the triplet baseline and retain 98% of the original score on short-context tasks when projecting to half the dimension on the last 40% of the layers. With more projected layers or on long-context RULER, they degrade rapidly. In contrast, KL and DimPO, which use every key during training, retain about 95% of the original RULER 4k score on the 8B model when…
PDF arXiv
arXiv cs.LGSep 26
ReLOBGen: Replayable Limit Order Book Message Generation
Junoh Kang, Kiseop Lee, Bohyung Han
We propose ReLOBGen, a method for generating limit order book (LOB) messages that are replayable by construction. Replayability is required for closed-loop market simulation, yet existing LOB message generators may produce non-replayable raw messages, i.e., messages inconsistent with the current market state. These generators therefore rely on post-hoc correction or rejection followed by resampling, which may alter the replayed message distribution or increase inference cost. ReLOBGen instead ensures replayability during generation: it selects the referenced order from the resting orders in the current LOB and then generates the remaining message fields to be consistent with that order and the market state. For realistic reference selection, ReLOBGen samples from a learned distribution over eligible resting orders, efficiently computed from cached order representations and a context-dependent query. It then enforces the consistency of the remaining fields by masking out invalid tokens. Together, these components enable efficient generation of realistic messages without post-hoc correction or resampling. In 500-message rollouts, ReLOBGen achieves 100% replayability, improves…
PDF arXiv
arXiv cs.CVSep 26
DepthBench: Measuring How Residual Connections Enable More Computational Depth
Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu +1
Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (\text{e.g.}, LayerNorm Scaling) or residual connections (\text{e.g.}, mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce \textbf{DepthBench}, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width--depth aspect ratio ($d_{\text{model}}/n_{\text{layer}}$) from shallow--wide to deep--narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper…
PDF arXiv
arXiv cs.LGSep 26
Activation Flow: Manufacturing Activations for Steering
Hong Kiat Tan, Linh Le, David Williams-King
Difference-in-means steering requires activations recorded while a model shows the desired behavior, which a sandbagging model withholds by deliberately underperforming. We introduce Activation Flow (ActFlow), which manufactures these activations from $k$ correct labels without fine-tuning. ActFlow sets target logits that rank each labeled item's correct answer first, and moves the logits toward them by adding one vector $x$ to all $k$ residual streams at one layer. ActFlow is a family of ordinary differential equations for $x$, one for each rule that maps the required logit change to the velocity of $x$. The smallest-norm rule lands exactly on the targets, while the others keep only the top singular directions of the Jacobian. We test ActFlow on three instruction-tuned models, each locked by a sandbagging prompt and by a password-locked LoRA. At $k=40$, ActFlow keeping five singular directions raises the mean held-out ARC-Easy accuracy over the six locked models from $0.05$ to $0.85$, against $0.88$ for rank-$16$ LoRA fine-tuning, which trains about $3{,}000$ times as many parameters, and $0.92$ for the honest models. Furthermore, it scores higher than the smallest-norm rule in…
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Topics in this field

diffusion language models · models diffusion language · diffusion language model165 papers31 this week
kv cache compression · kv cache · kv caches264 papers18 this week
catastrophic forgetting · continual learning · forgetting173 papers18 this week
low bit quantization · quantization aware training · training quantization172 papers15 this week
low rank adaptation · rank adaptation lora · rank adaptation123 papers13 this week
recurrent depth · looped language models · computational depth105 papers13 this week

Recently funded

Market

QuestorAIStage not given · Oct 25$3M
Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M

Active contributors · 90 days

Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes7
Jungseob LeeUnderstanding Target Probability Reuse in Speculative Decoding6
Yu WangCache Engineering for Agents6
Wei WangFuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper6
Shiwei LiuDepth Memory6
Hao WangBudgeted KV-Cache Repair for Evolving Contexts6

Recent papers

All papers in Research

arXiv cs.LGSep 26
Age of Learning: Temporal Persistence of Prediction Errors as a Learning Signal
Chenyang Wang, Stefan Forsström, Roger Olsson, Di Yuan, Qing He
Current machine learning algorithms primarily rely on instantaneous signals such as loss, margin, and prediction confidence to characterize model behavior. These signals indicate how difficult a prediction is at the current optimization step, but they do not capture how long the model has remained incorrect. We study this temporal dimension of learning and introduce Age of Learning (AoL), a learning-state variable that measures the persistence of prediction errors over time. AoL increases while an error remains unresolved and resets when a correct prediction is achieved, thereby distinguishing persistent under-learning from transient mistakes. We develop AoL-based training strategies for both offline and streaming settings. In offline learning, sample-level AoL is accumulated over training and aggregated into class-level states that guide adaptive reweighting and resampling. In streaming learning, where full historical access is unavailable, we maintain lightweight class-level AoL states using current and buffered observations. Across long-tailed classification settings, AoL improves or matches standard training baselines, with larger benefits when learning difficulty persists…
PDF arXiv
arXiv cs.CLSep 26
HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion
Junxiang Qiu, Zhengsu Chen, Xinting Hu, Shuo Wang, Hengheng Zhang +4
Mixture-of-Experts (MoE) architectures have become a standard way to scale model capacity while keeping computation sparse, yet routing remains a key determinant of MoE quality and training behavior. Prior empirical studies suggest that MoE routing reflects input semantics and upstream computation across depth, but standard routers do not explicitly use the routing distributions produced by preceding layers. We propose HERO-MoE, Historical Expert ROuting with Scale-Preserving Fusion, a routing framework that injects historical routing priors into MoE routers by reusing detached, dense routing distributions collected from preceding MoE layers. The key idea is simple: HERO-MoE preserves the original token-conditioned routing branch and adds a residual historical routing contribution before the standard softmax and top-$k$ dispatch. To stabilize this historical signal, HERO-MoE introduces a scale-preserving fusion mechanism that matches the magnitude of historical routing memory to the current hidden representation and accounts for the number of visible historical layers, without introducing an auxiliary routing loss or a fusion-specific tuning parameter. By reusing routing…
PDF arXiv
arXiv cs.LGSep 26
DimPO: Dimensionality Reduction for Attention using Preference Optimization
Vojtěch Lanz, Yufei Cui, Prasanna Parthasarathi
A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model behavior. We ask whether preferences over keys and attention mass on the highest-weighted keys provide a better signal than matching the full attention distribution with KL divergence, especially in long-context settings. We introduce DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity. DimPO is trained offline from the attention patterns of a frozen language model, with one map per layer, shared by the query and the keys and trained separately from the other layers. Across LLaMA3.2-3B, LLaMA3.1-8B, Qwen2.5-7B, and Qwen3-4B Instruct models, pairwise preference objectives outperform the triplet baseline and retain 98% of the original score on short-context tasks when projecting to half the dimension on the last 40% of the layers. With more projected layers or on long-context RULER, they degrade rapidly. In contrast, KL and DimPO, which use every key during training, retain about 95% of the original RULER 4k score on the 8B model when…
PDF arXiv
arXiv cs.LGSep 26
ReLOBGen: Replayable Limit Order Book Message Generation
Junoh Kang, Kiseop Lee, Bohyung Han
We propose ReLOBGen, a method for generating limit order book (LOB) messages that are replayable by construction. Replayability is required for closed-loop market simulation, yet existing LOB message generators may produce non-replayable raw messages, i.e., messages inconsistent with the current market state. These generators therefore rely on post-hoc correction or rejection followed by resampling, which may alter the replayed message distribution or increase inference cost. ReLOBGen instead ensures replayability during generation: it selects the referenced order from the resting orders in the current LOB and then generates the remaining message fields to be consistent with that order and the market state. For realistic reference selection, ReLOBGen samples from a learned distribution over eligible resting orders, efficiently computed from cached order representations and a context-dependent query. It then enforces the consistency of the remaining fields by masking out invalid tokens. Together, these components enable efficient generation of realistic messages without post-hoc correction or resampling. In 500-message rollouts, ReLOBGen achieves 100% replayability, improves…
PDF arXiv
arXiv cs.CVSep 26
DepthBench: Measuring How Residual Connections Enable More Computational Depth
Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu +1
Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (\text{e.g.}, LayerNorm Scaling) or residual connections (\text{e.g.}, mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce \textbf{DepthBench}, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width--depth aspect ratio ($d_{\text{model}}/n_{\text{layer}}$) from shallow--wide to deep--narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper…
PDF arXiv
arXiv cs.LGSep 26
Activation Flow: Manufacturing Activations for Steering
Hong Kiat Tan, Linh Le, David Williams-King
Difference-in-means steering requires activations recorded while a model shows the desired behavior, which a sandbagging model withholds by deliberately underperforming. We introduce Activation Flow (ActFlow), which manufactures these activations from $k$ correct labels without fine-tuning. ActFlow sets target logits that rank each labeled item's correct answer first, and moves the logits toward them by adding one vector $x$ to all $k$ residual streams at one layer. ActFlow is a family of ordinary differential equations for $x$, one for each rule that maps the required logit change to the velocity of $x$. The smallest-norm rule lands exactly on the targets, while the others keep only the top singular directions of the Jacobian. We test ActFlow on three instruction-tuned models, each locked by a sandbagging prompt and by a password-locked LoRA. At $k=40$, ActFlow keeping five singular directions raises the mean held-out ARC-Easy accuracy over the six locked models from $0.05$ to $0.85$, against $0.88$ for rank-$16$ LoRA fine-tuning, which trains about $3{,}000$ times as many parameters, and $0.92$ for the honest models. Furthermore, it scores higher than the smallest-norm rule in…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 46m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

186papers this week−44.8% vs last week

All papers · 1.8k With code · 67

Newest Most relevant

1.6k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.