Papers

Field

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M
CygridPre-seed · Oct 23$3M

Recently funded

Topics in this field

agent memory · memory construction · term memory270 papers33 this week
diffusion language models · language models diffusion · models diffusion language164 papers30 this week
gpus · gpu · throughput234 papers24 this week
continual learning · catastrophic forgetting · forgetting249 papers22 this week
speculative decoding accelerates · self speculative decoding · speculative decoding97 papers18 this week
looped language models · recurrent depth · loops98 papers17 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGSep 28
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Bowen Dong, Yilong Fan, Tengyu Pan, Yike Zhang, Zhenyu Li +4
Mixture-of-Depths (MoD) enables conditional computation across Transformer depth by routing only a subset of tokens through selected layers, but its original one-sparse--one-dense alternation tightly couples total capacity to active capacity and limits sparse-depth scaling. We introduce X-MoD, a scalable sparse-depth architecture that decouples token sparsity from anchor stride, allowing total parameter count to grow while keeping active-equivalent capacity nearly fixed. To make deep sparse routing trainable, X-MoD combines dense anchors with variance-scaled layer-wise gating and depth-wise token balancing. To make this regime analyzable and usable, we formulate sparse-depth routing as a conditional architecture-design problem: given compute, context length, and active-equivalent backbone size, how should the routing configuration be chosen? We develop a practical scaling-law framework by fitting X-MoD relative to FLOP-matched dense baselines, yielding an interpretable law that decomposes performance into sparse-capacity gain, sparse-context correction, and anchor-stride interaction. The law predicts validation loss across routing configurations and reveals how context length,…
PDF arXiv
arXiv cs.AISep 28
CASS: Contribution-Aware Structured Sparsity for Model Merging
Yan Li, Guiping Cao, Meng Xu, Tao Jiang, Yaguang Song +3
Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose \textbf{C}ontribution-\textbf{A}ware \textbf{S}tructured \textbf{S}parsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering…
PDF arXiv
arXiv cs.AISep 28
ReplayLens: Auditing Agents' Use of Outcomes
Dong Xu, Zhangfan Yang, Jiantao Wu, Shipeng Zhang, Zexuan Zhu +3
When an agent reuses logged experience, a changed decision may reflect the recorded score, the action's name, or the record's position in storage. Standard memory evaluations do not reveal which relationship drives that change. We introduce ReplayLens, a black-box audit that changes one relationship in the stored history at a time, holds the remaining interface fixed, and measures the resulting decision. Four interventions target four relationships. Outcome reassignment swaps which scores belong to which actions. Pair transport moves intact action-score pairs to new record slots. Consistent renaming relabels actions in both history and menu. Key-slot reassignment changes both score attachment and position. A constructive separation shows why the audit is needed: two memory writers with identical endpoint accuracy respond differently to the same replay, so conventional evaluation cannot resolve the underlying dependence. On black-box LLM interfaces, swapping scores changes decisions while moving intact pairs does not, separating score attachment from record order. A bounded-memory study exposes ingestion-order sensitivity that endpoint comparison misses. In sequential experiment…
PDF arXiv
arXiv cs.LGSep 28
GradLev: Token-Parallel Test-Time Training Via Costate Prediction
Bo Liu, Qiang Liu
Test-time training (TTT) allows a model to improve its predictions at inference time by updating weights after every observed token. However, sequential gra- dient writes make parallel training difficult. We observe that, given layer inputs and activation gradients (costates), online gradient descent admits exact parallel scans for both forward evaluation and reverse backpropagation. GradLev lever- ages this duality: a causal auxiliary network predicts costates across all tokens in parallel; associative scans compute the adapted weights and forward activations and propagate gradients backward; and the resulting gradient targets supervise the predictor via a consistency loss. Exact consistency guarantees exact recovery of the sequential online learner. At deployment, the auxiliary predictor is discarded, and the model updates natively via token-by-token forward and backward passes.
PDF arXiv
arXiv cs.DCSep 28
SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
Ruijia Yang, Shiyuan Lin, Yulong Ao, Zhiyu Li, Yingli Zhao +3
Host-resident layer streaming enables full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks compete for shared host resources. Replicated transfers amplify traffic, while strong scaling can expose host work as computation windows shrink. We present SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems. It maintains one authoritative host state, decouples communication routes from state layout, and pipelines parameter delivery, gradient aggregation, and CPU updates across ranks and chunks. An analytical step-time model characterizes resource bottlenecks and pipeline exposure; runtime measurements guide communication, chunking, and activation policies under a GPU memory budget. In matched-batch sweeps, SlideDP achieves geometric-mean throughput ratios of 1.46-2.64$\times$ over SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s, SlideDP approaches GPU-resident FSDP2 throughput for Qwen3-14B at a smaller batch size. With a larger batch, it processes over 1M tokens per step and exceeds FSDP2's measured peak throughput by 11.2%. Separately, it supports 256K-token sequences for the same model and fine-tunes Qwen2.5-72B on four RTX 4090…
PDF arXiv
arXiv cs.AISep 28
RoutePrism: Tracing Construction Order Effects in Agent Memory
Dong Xu, Zhangfan Yang, Jiantao Wu, Shipeng Zhang, Zexuan Zhu +3
Processing the same records in a different order can discard different evidence, yet endpoint accuracy alone cannot reveal what changed or whether it mattered. We introduce RoutePrism, a diagnostic protocol that builds memory twice from the same source pool in two processing orders, then traces which sources, compiled contexts, and answers differ. Because record content, timestamps, policy, and the answer model all stay fixed, any observed difference is localized to the memory construction step. A matched four-condition intervention tests whether a record displaced by reordering actually carried task-relevant evidence: restoring that single record recovers over 60 percentage points of lost accuracy, while substituting a non-supporting record of equal length does not. We evaluate the protocol on PersonaMem-32K (63 primary queries, 29 users) and 470 LongMemEval-S questions with histories spanning 38 to 62 sessions, replicating the core intervention across five answer models. Survivor selection, defined as the choice of which record a cluster retains, drives most source-level changes, while different memory policies (compaction, bounded recency, MemoChat-style summarization, A-MEM)…
PDF arXiv

Papers

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Recently funded

Market

Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M
CygridPre-seed · Oct 23$3M

Topics in this field

agent memory · memory construction · term memory270 papers33 this week
diffusion language models · language models diffusion · models diffusion language164 papers30 this week
gpus · gpu · throughput234 papers24 this week
continual learning · catastrophic forgetting · forgetting249 papers22 this week
speculative decoding accelerates · self speculative decoding · speculative decoding97 papers18 this week
looped language models · recurrent depth · loops98 papers17 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.LGSep 28
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Bowen Dong, Yilong Fan, Tengyu Pan, Yike Zhang, Zhenyu Li +4
Mixture-of-Depths (MoD) enables conditional computation across Transformer depth by routing only a subset of tokens through selected layers, but its original one-sparse--one-dense alternation tightly couples total capacity to active capacity and limits sparse-depth scaling. We introduce X-MoD, a scalable sparse-depth architecture that decouples token sparsity from anchor stride, allowing total parameter count to grow while keeping active-equivalent capacity nearly fixed. To make deep sparse routing trainable, X-MoD combines dense anchors with variance-scaled layer-wise gating and depth-wise token balancing. To make this regime analyzable and usable, we formulate sparse-depth routing as a conditional architecture-design problem: given compute, context length, and active-equivalent backbone size, how should the routing configuration be chosen? We develop a practical scaling-law framework by fitting X-MoD relative to FLOP-matched dense baselines, yielding an interpretable law that decomposes performance into sparse-capacity gain, sparse-context correction, and anchor-stride interaction. The law predicts validation loss across routing configurations and reveals how context length,…
PDF arXiv
arXiv cs.AISep 28
CASS: Contribution-Aware Structured Sparsity for Model Merging
Yan Li, Guiping Cao, Meng Xu, Tao Jiang, Yaguang Song +3
Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose \textbf{C}ontribution-\textbf{A}ware \textbf{S}tructured \textbf{S}parsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering…
PDF arXiv
arXiv cs.AISep 28
ReplayLens: Auditing Agents' Use of Outcomes
Dong Xu, Zhangfan Yang, Jiantao Wu, Shipeng Zhang, Zexuan Zhu +3
When an agent reuses logged experience, a changed decision may reflect the recorded score, the action's name, or the record's position in storage. Standard memory evaluations do not reveal which relationship drives that change. We introduce ReplayLens, a black-box audit that changes one relationship in the stored history at a time, holds the remaining interface fixed, and measures the resulting decision. Four interventions target four relationships. Outcome reassignment swaps which scores belong to which actions. Pair transport moves intact action-score pairs to new record slots. Consistent renaming relabels actions in both history and menu. Key-slot reassignment changes both score attachment and position. A constructive separation shows why the audit is needed: two memory writers with identical endpoint accuracy respond differently to the same replay, so conventional evaluation cannot resolve the underlying dependence. On black-box LLM interfaces, swapping scores changes decisions while moving intact pairs does not, separating score attachment from record order. A bounded-memory study exposes ingestion-order sensitivity that endpoint comparison misses. In sequential experiment…
PDF arXiv
arXiv cs.LGSep 28
GradLev: Token-Parallel Test-Time Training Via Costate Prediction
Bo Liu, Qiang Liu
Test-time training (TTT) allows a model to improve its predictions at inference time by updating weights after every observed token. However, sequential gra- dient writes make parallel training difficult. We observe that, given layer inputs and activation gradients (costates), online gradient descent admits exact parallel scans for both forward evaluation and reverse backpropagation. GradLev lever- ages this duality: a causal auxiliary network predicts costates across all tokens in parallel; associative scans compute the adapted weights and forward activations and propagate gradients backward; and the resulting gradient targets supervise the predictor via a consistency loss. Exact consistency guarantees exact recovery of the sequential online learner. At deployment, the auxiliary predictor is discarded, and the model updates natively via token-by-token forward and backward passes.
PDF arXiv
arXiv cs.DCSep 28
SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
Ruijia Yang, Shiyuan Lin, Yulong Ao, Zhiyu Li, Yingli Zhao +3
Host-resident layer streaming enables full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks compete for shared host resources. Replicated transfers amplify traffic, while strong scaling can expose host work as computation windows shrink. We present SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems. It maintains one authoritative host state, decouples communication routes from state layout, and pipelines parameter delivery, gradient aggregation, and CPU updates across ranks and chunks. An analytical step-time model characterizes resource bottlenecks and pipeline exposure; runtime measurements guide communication, chunking, and activation policies under a GPU memory budget. In matched-batch sweeps, SlideDP achieves geometric-mean throughput ratios of 1.46-2.64$\times$ over SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s, SlideDP approaches GPU-resident FSDP2 throughput for Qwen3-14B at a smaller batch size. With a larger batch, it processes over 1M tokens per step and exceeds FSDP2's measured peak throughput by 11.2%. Separately, it supports 256K-token sequences for the same model and fine-tunes Qwen2.5-72B on four RTX 4090…
PDF arXiv
arXiv cs.AISep 28
RoutePrism: Tracing Construction Order Effects in Agent Memory
Dong Xu, Zhangfan Yang, Jiantao Wu, Shipeng Zhang, Zexuan Zhu +3
Processing the same records in a different order can discard different evidence, yet endpoint accuracy alone cannot reveal what changed or whether it mattered. We introduce RoutePrism, a diagnostic protocol that builds memory twice from the same source pool in two processing orders, then traces which sources, compiled contexts, and answers differ. Because record content, timestamps, policy, and the answer model all stay fixed, any observed difference is localized to the memory construction step. A matched four-condition intervention tests whether a record displaced by reordering actually carried task-relevant evidence: restoring that single record recovers over 60 percentage points of lost accuracy, while substituting a non-supporting record of equal length does not. We evaluate the protocol on PersonaMem-32K (63 primary queries, 29 users) and 470 LongMemEval-S questions with histories spanning 38 to 62 sessions, replicating the core intervention across five answer models. Survivor selection, defined as the choice of which record a cluster retains, drives most source-level changes, while different memory policies (compaction, bounded recency, MemoChat-style summarization, A-MEM)…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 27m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

199papers this week−33.0% vs last week

All papers · 1.6k With code · 70

Newest Most relevant

1.4k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.