Papers

Field

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Top tools here

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M
CygridPre-seed · Oct 23$3M

Recently funded

Topics in this field

agent memory · memory construction · term memory270 papers36 this week
diffusion language models · language models diffusion · models diffusion language164 papers30 this week
gpus · gpu · throughput234 papers25 this week
continual learning · catastrophic forgetting · forgetting249 papers23 this week
looped language models · recurrent depth · loops98 papers20 this week
speculative decoding accelerates · self speculative decoding · speculative decoding97 papers19 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.DCSep 28
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
Aaryam Sharma
Pipeline parallelism can improve prefill throughput by processing multiple request chunks concurrently across different stages of the model. However, keeping the pipeline fully utilized requires efficient scheduling and request preparation. In systems where stages retain and evict cache state independently, a local cache hit does not guarantee that the same prefix can be reused across the pipeline. Here, coordination overhead can impede request admission cadence and thus reduce overall throughput. In this paper, we present WavePP, a prefill runtime built on top of TensorRT-LLM that addresses these challenges by overlapping request admission with pipeline execution. WavePP asynchronously finds a prefix that can be reused across all stages, protects the cached state, and reserves space for the remaining input while earlier requests continue to execute. It subsequently plans the chunk sizes of each request dynamically to maximize pipeline fill. Each stage then completes the local preparation before executing the request. In the same system and pipeline topology, WavePP improves TensorRT-LLM's prefill throughput in 37 of 40 tested settings on GLM 5.2 and MiniMax M2.7. At concurrency…
PDF arXiv
arXiv cs.AISep 28
EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents
Fengzhou Sun, Yuan Zhang, Xintong Yu, Jinyao Yan
Large language model (LLM) agents face critical privacy risks when acting as delegates in human-agent-human communication. To prevent such breaches, agents must understand users' social relationships and adhere to context-dependent social information disclosure boundaries. Current studies on agent memory privacy focus on instantaneous interactions, leaving the long-term relational disclosure problem unexplored. In this paper, we propose EP-Mem, an Elastic Privacy Memory architecture that reframes privacy as user-owned boundary control across social roles. EP-Mem introduces (1) token-level memory driven by user-configurable a privacy policy that stratifies persons and events, combining domain-level default circulation rules with fact-level whitelist/blacklist exceptions; and (2) a pluggable sidecar with a privacy engine that aligns disclosure controls with memory across summary, detail, and boundary granularities, enforced throughout generation, storage, and retrieval. We construct EP-Bench, to our knowledge the first long-term multi-party benchmark with cross-session correlated events for policy-conditioned relational disclosure. Experiments show that EP-Mem achieves 94.0%…
PDF arXiv
arXiv cs.LGSep 28
CADENCE: A Confidence-Adaptive Dual-Expert Network for Fast and Accurate Time Series Classification
Onisa Mapunda
Time series classification (TSC) exhibits a sharp trade-off between accuracy and computational scalability. Meta-ensembles like HIVE-COTE 2.0 reach state-of-the-art accuracy but require extensive compute, whereas ultra-fast random convolutional transforms (e.g., MiniRocket, Hydra) run in seconds but struggle with phase-independent distributions, signal kinematics, and decision tree fragmentation on large class counts. In this work, we present CADENCE (Confidence-Adaptive Dual-Expert Network for time series Classification Excellence), a unified, CPU-native dual-expert architecture. CADENCE decouples representation learning into two specialized pathways: (i) a Convolutional Linear Expert pairing 10,000 deterministic dilated features with closed-form L2-regularized Woodbury ridge classification, and (ii) a Distributional Interval Expert pairing competing dilated kernels (Hydra) with dyadic Cornish-Fisher moment approximations across signal kinematics and FFT spectral bands, fitted with an ExtraTrees ensemble. An internal validation meta-router with rare-class preservation dynamically selects between pure expert routing and confidence-weighted soft blending, followed by a full refit…
PDF arXiv
arXiv cs.AISep 28
Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference
Suwesh Prasad Sah
Autoregressive large language model inference repeatedly invokes the target model to generate one token at a time, making generation sensitive to GPU memory movement and sequential execution. This study evaluates two-token multi-token prediction (MTP) against autoregressive decoding in a controlled single-request deployment on an NVIDIA A10G GPU. A 360-request benchmark covered plain-text, reasoning-intensive, and tool-calling workloads, while runtime telemetry, Nsight Systems, PyTorch Profiler, and selected Nsight Compute measurements were used to explain the observed performance. MTP increased output throughput by $1.91\times$ to $2.19\times$ across all prompts and reduced time to first output by 10.0--14.2\%. Median mean acceptance length ranged from 2.370 to 2.595 tokens per verification iteration. Profiling showed that MTP introduced a longer and more complex execution path, including proposal, sampling, attention, gathering, and reduction operations. However, it required 56.4--78.1\% fewer executions of the selected repeating CUDA Graph per generated token. The dominant MTP GEMM kernel was not faster than the dominant autoregressive GEMV kernel, and selected instances…
PDF arXiv
arXiv cs.LGSep 28
EdgeCraft: Automated Model Crafting for Edge IoT
Genglin Wang, Kaiwei Liu, Liekang Zeng, Wangsong Yin, Shangcheng Jin +2
Machine learning (ML) increasingly powers Internet of Things (IoT) applications at the edge. Yet producing a deployable edge ML artifact for a specific scenario requires navigating a huge search space spanning data representation, model design, training on domain-specific data, and runtime customization. This workflow is fragmented and difficult to scale across diverse edge applications. We present EdgeCraft, an LLM-driven system that turns high-level intent into deployable edge ML artifacts. Building such a system raises two challenges: (1) How can an LLM be guided to find high-quality solutions that meet dynamic SLOs for task quality, latency, and energy? (2) How can trustworthy target-device verification be obtained at low cost? EdgeCraft addresses these challenges with two designs. (1) A constraint-aware synthesis tree explores alternative candidates and uses measured SLO gaps to guide each improvement. (2) A multi-fidelity verifier progressively combines low-cost checks with full target-device verification to reduce verification cost while preserving reliable verification results. It also records verified failures for reuse, avoiding repeated device work. To support…
PDF arXiv
arXiv cs.LGSep 28
Learning to Re-Draft: A Variational Stackelberg Game for Discrete Diffusion
Dmitrii Moor, Federico Tomasi, Paul N. Bennett, Alice Wang, Mounia Lalmas
Discrete diffusion models offer the ability to re-draft, revisiting and correcting earlier tokens throughout generation. This capability depends on the forward corruption process that defines what the denoiser learns to correct. Masked diffusion models fix tokens once they are unmasked, while uniform diffusion permits revisions but relies on uniformly random token substitutions. We instead learn which substitutions are most useful for training the denoiser to re-draft. We introduce Variational Stackelberg Discrete Diffusion (VSDD), a framework for learning a semantically aware corruption process. VSDD formulates training as a leader-follower game: the leader defines a Markovian corruption process parameterized by the denoiser's token embeddings, while the follower optimizes a variational denoising objective with the corruption process held fixed. The leader rewards corruptions based on how much the denoiser improves after learning from them, rather than on how easily the current denoiser can reconstruct them. We measure this improvement under a fixed reference corruption process, approximate the follower's response with a one-step gradient update, and optimize the leader using a…
PDF arXiv

Papers

Recently funded

Market

Jiutian RuixinStage not given · Oct 23$149.9M
TractianSeries B+ · Oct 23$130M
Mach9Series A · Oct 23$34M
Youzhi ChuangxinSeed · Oct 23$15M
CygridPre-seed · Oct 23$3M

Topics in this field

agent memory · memory construction · term memory270 papers36 this week
diffusion language models · language models diffusion · models diffusion language164 papers30 this week
gpus · gpu · throughput234 papers25 this week
continual learning · catastrophic forgetting · forgetting249 papers23 this week
looped language models · recurrent depth · loops98 papers20 this week
speculative decoding accelerates · self speculative decoding · speculative decoding97 papers19 this week

Top tools here

All tools

NVIDIA Dynamo+79% downloads · 17.5k/wk+79.1%
BentoML+31% downloads · 44.8k/wk+31.1%
MLX+23% downloads · 356.4k/wk+22.8%
ort+6% downloads · 945.7k/wk+6.4%
Ollama4 releases this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.DCSep 28
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
Aaryam Sharma
Pipeline parallelism can improve prefill throughput by processing multiple request chunks concurrently across different stages of the model. However, keeping the pipeline fully utilized requires efficient scheduling and request preparation. In systems where stages retain and evict cache state independently, a local cache hit does not guarantee that the same prefix can be reused across the pipeline. Here, coordination overhead can impede request admission cadence and thus reduce overall throughput. In this paper, we present WavePP, a prefill runtime built on top of TensorRT-LLM that addresses these challenges by overlapping request admission with pipeline execution. WavePP asynchronously finds a prefix that can be reused across all stages, protects the cached state, and reserves space for the remaining input while earlier requests continue to execute. It subsequently plans the chunk sizes of each request dynamically to maximize pipeline fill. Each stage then completes the local preparation before executing the request. In the same system and pipeline topology, WavePP improves TensorRT-LLM's prefill throughput in 37 of 40 tested settings on GLM 5.2 and MiniMax M2.7. At concurrency…
PDF arXiv
arXiv cs.AISep 28
EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents
Fengzhou Sun, Yuan Zhang, Xintong Yu, Jinyao Yan
Large language model (LLM) agents face critical privacy risks when acting as delegates in human-agent-human communication. To prevent such breaches, agents must understand users' social relationships and adhere to context-dependent social information disclosure boundaries. Current studies on agent memory privacy focus on instantaneous interactions, leaving the long-term relational disclosure problem unexplored. In this paper, we propose EP-Mem, an Elastic Privacy Memory architecture that reframes privacy as user-owned boundary control across social roles. EP-Mem introduces (1) token-level memory driven by user-configurable a privacy policy that stratifies persons and events, combining domain-level default circulation rules with fact-level whitelist/blacklist exceptions; and (2) a pluggable sidecar with a privacy engine that aligns disclosure controls with memory across summary, detail, and boundary granularities, enforced throughout generation, storage, and retrieval. We construct EP-Bench, to our knowledge the first long-term multi-party benchmark with cross-session correlated events for policy-conditioned relational disclosure. Experiments show that EP-Mem achieves 94.0%…
PDF arXiv
arXiv cs.LGSep 28
CADENCE: A Confidence-Adaptive Dual-Expert Network for Fast and Accurate Time Series Classification
Onisa Mapunda
Time series classification (TSC) exhibits a sharp trade-off between accuracy and computational scalability. Meta-ensembles like HIVE-COTE 2.0 reach state-of-the-art accuracy but require extensive compute, whereas ultra-fast random convolutional transforms (e.g., MiniRocket, Hydra) run in seconds but struggle with phase-independent distributions, signal kinematics, and decision tree fragmentation on large class counts. In this work, we present CADENCE (Confidence-Adaptive Dual-Expert Network for time series Classification Excellence), a unified, CPU-native dual-expert architecture. CADENCE decouples representation learning into two specialized pathways: (i) a Convolutional Linear Expert pairing 10,000 deterministic dilated features with closed-form L2-regularized Woodbury ridge classification, and (ii) a Distributional Interval Expert pairing competing dilated kernels (Hydra) with dyadic Cornish-Fisher moment approximations across signal kinematics and FFT spectral bands, fitted with an ExtraTrees ensemble. An internal validation meta-router with rare-class preservation dynamically selects between pure expert routing and confidence-weighted soft blending, followed by a full refit…
PDF arXiv
arXiv cs.AISep 28
Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference
Suwesh Prasad Sah
Autoregressive large language model inference repeatedly invokes the target model to generate one token at a time, making generation sensitive to GPU memory movement and sequential execution. This study evaluates two-token multi-token prediction (MTP) against autoregressive decoding in a controlled single-request deployment on an NVIDIA A10G GPU. A 360-request benchmark covered plain-text, reasoning-intensive, and tool-calling workloads, while runtime telemetry, Nsight Systems, PyTorch Profiler, and selected Nsight Compute measurements were used to explain the observed performance. MTP increased output throughput by $1.91\times$ to $2.19\times$ across all prompts and reduced time to first output by 10.0--14.2\%. Median mean acceptance length ranged from 2.370 to 2.595 tokens per verification iteration. Profiling showed that MTP introduced a longer and more complex execution path, including proposal, sampling, attention, gathering, and reduction operations. However, it required 56.4--78.1\% fewer executions of the selected repeating CUDA Graph per generated token. The dominant MTP GEMM kernel was not faster than the dominant autoregressive GEMV kernel, and selected instances…
PDF arXiv
arXiv cs.LGSep 28
EdgeCraft: Automated Model Crafting for Edge IoT
Genglin Wang, Kaiwei Liu, Liekang Zeng, Wangsong Yin, Shangcheng Jin +2
Machine learning (ML) increasingly powers Internet of Things (IoT) applications at the edge. Yet producing a deployable edge ML artifact for a specific scenario requires navigating a huge search space spanning data representation, model design, training on domain-specific data, and runtime customization. This workflow is fragmented and difficult to scale across diverse edge applications. We present EdgeCraft, an LLM-driven system that turns high-level intent into deployable edge ML artifacts. Building such a system raises two challenges: (1) How can an LLM be guided to find high-quality solutions that meet dynamic SLOs for task quality, latency, and energy? (2) How can trustworthy target-device verification be obtained at low cost? EdgeCraft addresses these challenges with two designs. (1) A constraint-aware synthesis tree explores alternative candidates and uses measured SLO gaps to guide each improvement. (2) A multi-fidelity verifier progressively combines low-cost checks with full target-device verification to reduce verification cost while preserving reliable verification results. It also records verified failures for reuse, avoiding repeated device work. To support…
PDF arXiv
arXiv cs.LGSep 28
Learning to Re-Draft: A Variational Stackelberg Game for Discrete Diffusion
Dmitrii Moor, Federico Tomasi, Paul N. Bennett, Alice Wang, Mounia Lalmas
Discrete diffusion models offer the ability to re-draft, revisiting and correcting earlier tokens throughout generation. This capability depends on the forward corruption process that defines what the denoiser learns to correct. Masked diffusion models fix tokens once they are unmasked, while uniform diffusion permits revisions but relies on uniformly random token substitutions. We instead learn which substitutions are most useful for training the denoiser to re-draft. We introduce Variational Stackelberg Discrete Diffusion (VSDD), a framework for learning a semantically aware corruption process. VSDD formulates training as a leader-follower game: the leader defines a Markovian corruption process parameterized by the denoiser's token embeddings, while the follower optimizes a variational denoising objective with the corruption process held fixed. The leader rewards corruptions based on how much the denoiser improves after learning from them, rather than on how easily the current denoiser can reconstruct them. We measure this improvement under a fixed reference corruption process, approximate the follower's response with a one-step gradient update, and optimize the leader using a…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 25m ago

Inference & serving

Making models fast and cheap to run: serving systems, kernels, KV-cache, speculative decoding, hardware.

213papers this week−25.8% vs last week

All papers · 1.6k With code · 70

Newest Most relevant

1.5k papers this month

Open problems in Inference & serving

Efficient long-horizon context management3 papers raise it · 4 address it
Long agent trajectories exceed context windows, degrading performance without compacting or retaining key information.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Cache Engineering for Agentsraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- MemT-1.0: Towards Memory-Native Evolutionaddresses
Length generalization in recurrent or looped models2 papers raise it · 2 address it
Models trained on short sequences fail to extrapolate to much longer sequences, limiting their use in multi-step tasks.
- PAR: Progress-Aligned Recurrence for Length Generalization in State Trackingraises · addresses
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationraises · addresses
Token-level routing efficiency in serving systems1 paper raises it · 1 addresses it
Fine-grained routing improves cost-quality trade-offs but causes desynchronization and high implementation complexity in serving.
- TokenRouter: Efficient Serving System for Token-Level LLM Routingraises · addresses
Sparse attention computation overhead1 paper raises it · 1 addresses it
Sparse attention reduces compute but still requires scoring all keys, limiting speedups.
- DoSA: Dots Sparse Attentionraises · addresses
Memory-native model training and inference1 paper raises it · 1 addresses it
Foundation models lack intrinsic memory, making it hard to accumulate and reuse information across long interactions.
- MemT-1.0: Towards Memory-Native Evolutionraises · addresses

Named by an AI from these papers' abstracts.