Papers

Field

Top tools here

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantizing training and rollout paths independently in RL leads to mismatched behavior and reduced performance.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not diagnose the unique challenges of graph-based decision tasks for non-language models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Non-compositional sparse autoencoder features1 paper raises it · 1 addresses it
Sparse autoencoder features often exhibit higher-order interactions, limiting interpretability and control.
- Behavioral Compositionality of Sparse Autoencoder Representationsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

gradient descent · learning rate · neural networks221 papers43 this week
neural networks · neural network · arnold networks149 papers36 this week
graph learning · graph neural · graph neural networks263 papers35 this week
memory efficient · optimizers · low rank adaptation157 papers25 this week
bit quantization · quantization aware training · quantization aware152 papers20 this week
linear attention · softmax attention · transformer attention106 papers13 this week

Topics in this field

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.LGOct 16
A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability
Quan Nguyen, Robin Yadav, Param Mody, Mark Schmidt
Gradient descent (GD) with large learning rates can successfully optimize neural networks even though classical smoothness-based learning rate conditions are violated. Yet theory for this regime largely covers linear models or linearly separable data. We analyze a minimal setting outside both regimes. Specifically, a two-neuron ReLU network trained with logistic loss on 1D non-linearly separable data. The objective is nonconvex and nonsmooth, and GD exhibits edge-of-stability (EoS) behavior. We identify an active cone in parameter space that contains every perfect classifier, where the problem becomes piecewise logistic regression with a varying effective dataset. Within this cone, we prove a finite-time loss bound for constant learning rates and show that GD converges in direction to an explicit max-margin vector, while the orthogonal component converges to the unique minimizer of a non-strongly convex potential. We further establish catapult results where exponentially small loss is reached after a single loss spike. Furthermore, we propose an adaptive schedule that lets learning rates grow to infinity without leaving the active cone. We prove that this schedule achieves a loss…
PDF arXiv
arXiv cs.LGOct 16
A Label Noise Robust Algorithm for Graph Data using Neighborhood Label Distribution
Pintu Kumar, Nandyala Hemachandra
Graph Convolutional Networks (GCNs) have demonstrated exceptional performance on graph-structured data. However, their success heavily relies on the availability of clean labels, an assumption that often fails in real-world scenarios. The learnability of GCNs is intrinsically linked to the neighborhood structure of nodes, making it critical to understand how this structure changes in the presence of label noise. In this work, we investigate how the neighborhood label distribution of a node changes under label noise. We mathematically derive that label noise smooths this distribution, which inherently increases its entropy and severely decreases interclass distinguishability. To address this challenge, we propose Leave-one-out Entropy Node Screening (LENS), an end-to-end framework that identifies noisy nodes by measuring the reduction in neighborhood entropy when a specific node is removed. LENS further incorporates a relabeling mechanism to recover and preserve valuable feature information instead of entirely discarding the identified noisy nodes. Crucially, the end-to-end computational overhead of LENS is remarkably low. Extensive evaluations demonstrate that our algorithm…
PDF arXiv
arXiv cs.LGOct 16
Scalable Levenberg-Marquardt Training Framework for Deep Learning
Xianghao Wang, Johan Barthélemy, John Taylor
The Levenberg-Marquardt (LMA) algorithm is a widely used nonlinear least-squares solver known for its rapid convergence. For small-scale neural networks, LMA provides better convergence and architectural robustness than first-order optimizers like Adam. However, LMA imposes prohibitive memory and computational costs. These costs are primarily from the Jacobian matrix, whose dimensions scale with both the number of samples and parameters. This bottleneck prevents its use in large-scale neural networks. Recent efforts scale LMA using preconditioned conjugate gradient (PCG) solvers to avoid materializing the Jacobian. However, PCG only approximates the solution, requiring many network propagations and a task-specific preconditioner. Other studies optimize LMA to reduce costs, but they restrict their methods to specific tasks or network architectures. As a result, LMA remains impractical for training general large-scale neural networks. In this paper, we propose LeMA, a distributed LMA algorithm for large-scale neural network training on multi-GPU platforms. Our contributions are twofold: First, an exact dual formulation of LMA removes the memory dependence on the parameter…
PDF arXiv
arXiv cs.LGOct 16
Dynamical Kernel Attention: Estimating the Present from a Record of the Past
Peter Racioppo
Standard self-attention compares tokens by similarity but does not account for how much of a past state survives to the present. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state evolving under stochastic dynamics. As a past state decays toward equilibrium under the dynamics, the information it carries about the present vanishes, bounding the total evidence a head can draw. Unlike recency biases, DKA can abstain when relevant evidence is weak and guarantees a bounded error when old tokens are discarded. DKA preserves the parallel structure and computational cost of standard attention. We present the model and its theory; empirical evaluation is deferred to a subsequent version.
PDF arXiv
arXiv cs.AIOct 16
ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning
Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Longtao Huang +2
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prevalent strategy for adapting large pre-trained models. Among PEFT techniques, reparameterization-based methods, particularly those that employ low-rank or sparsity structures, have attracted significant attention. However, these approaches often underperform full fine-tuning. Specifically, low-rank methods constrain the update rank, which is misaligned with the inherently high-rank nature of full fine-tuning. In contrast, sparsity-based techniques limit the flexibility of the parameter space and compromise connectivity across weights. To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints. A straightforward approach is to update weights via a product of Toeplitz matrices. However, similar to RNNs, long multiplicative chains often lead to gradient instability. To address this issue, we propose \textbf{ToPA}, a more refined version of this idea that replaces standard Toeplitz matrices with block-wise Toeplitz, greatly increasing each factor's capacity and…
PDF arXiv
arXiv cs.LGOct 16
Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training
Xinyu Luo, Jincheng Zhou, Mengbo Wang, Zilin Shen, Nadia Atallah Lanman +3
A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace and map the update back to the full model. Most existing work either recomputes the gradient subspace at certain intervals, or performs post-hoc adaptation to catch up to the moving subspace. In this work, we explicitly identify in practice the slow, stable manner in which LLM gradient subspaces drift across time. To this end, we propose Ori, a novel low-rank optimizer for training LLMs that captures the shifting principal gradient directions over time to achieve state-of-the-art accuracy under stringent low-rank constraints. Ori employs a streaming $k$-PCA algorithm inspired by Oja++, whose low per-step cost enables high-frequency, on-the-fly subspace tracking. In practice, Ori reduces the performance gap of the state-of-the-art low-rank optimizers to full-rank optimizers by up to 69.3\% on a 1B LLaMA pretraining task, and achieves the best performance on 10 out of 16 task scenarios in Roberta-Base fine-tuning. Together, these contributions allow Ori to further push the performance and efficiency…
PDF arXiv

Papers

Topics in this field

gradient descent · learning rate · neural networks221 papers43 this week
neural networks · neural network · arnold networks149 papers36 this week
graph learning · graph neural · graph neural networks263 papers35 this week
memory efficient · optimizers · low rank adaptation157 papers25 this week
bit quantization · quantization aware training · quantization aware152 papers20 this week
linear attention · softmax attention · transformer attention106 papers13 this week

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Recent papers

All in Research →

arXiv cs.LGOct 16
A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability
Quan Nguyen, Robin Yadav, Param Mody, Mark Schmidt
Gradient descent (GD) with large learning rates can successfully optimize neural networks even though classical smoothness-based learning rate conditions are violated. Yet theory for this regime largely covers linear models or linearly separable data. We analyze a minimal setting outside both regimes. Specifically, a two-neuron ReLU network trained with logistic loss on 1D non-linearly separable data. The objective is nonconvex and nonsmooth, and GD exhibits edge-of-stability (EoS) behavior. We identify an active cone in parameter space that contains every perfect classifier, where the problem becomes piecewise logistic regression with a varying effective dataset. Within this cone, we prove a finite-time loss bound for constant learning rates and show that GD converges in direction to an explicit max-margin vector, while the orthogonal component converges to the unique minimizer of a non-strongly convex potential. We further establish catapult results where exponentially small loss is reached after a single loss spike. Furthermore, we propose an adaptive schedule that lets learning rates grow to infinity without leaving the active cone. We prove that this schedule achieves a loss…
PDF arXiv
arXiv cs.LGOct 16
A Label Noise Robust Algorithm for Graph Data using Neighborhood Label Distribution
Pintu Kumar, Nandyala Hemachandra
Graph Convolutional Networks (GCNs) have demonstrated exceptional performance on graph-structured data. However, their success heavily relies on the availability of clean labels, an assumption that often fails in real-world scenarios. The learnability of GCNs is intrinsically linked to the neighborhood structure of nodes, making it critical to understand how this structure changes in the presence of label noise. In this work, we investigate how the neighborhood label distribution of a node changes under label noise. We mathematically derive that label noise smooths this distribution, which inherently increases its entropy and severely decreases interclass distinguishability. To address this challenge, we propose Leave-one-out Entropy Node Screening (LENS), an end-to-end framework that identifies noisy nodes by measuring the reduction in neighborhood entropy when a specific node is removed. LENS further incorporates a relabeling mechanism to recover and preserve valuable feature information instead of entirely discarding the identified noisy nodes. Crucially, the end-to-end computational overhead of LENS is remarkably low. Extensive evaluations demonstrate that our algorithm…
PDF arXiv
arXiv cs.LGOct 16
Scalable Levenberg-Marquardt Training Framework for Deep Learning
Xianghao Wang, Johan Barthélemy, John Taylor
The Levenberg-Marquardt (LMA) algorithm is a widely used nonlinear least-squares solver known for its rapid convergence. For small-scale neural networks, LMA provides better convergence and architectural robustness than first-order optimizers like Adam. However, LMA imposes prohibitive memory and computational costs. These costs are primarily from the Jacobian matrix, whose dimensions scale with both the number of samples and parameters. This bottleneck prevents its use in large-scale neural networks. Recent efforts scale LMA using preconditioned conjugate gradient (PCG) solvers to avoid materializing the Jacobian. However, PCG only approximates the solution, requiring many network propagations and a task-specific preconditioner. Other studies optimize LMA to reduce costs, but they restrict their methods to specific tasks or network architectures. As a result, LMA remains impractical for training general large-scale neural networks. In this paper, we propose LeMA, a distributed LMA algorithm for large-scale neural network training on multi-GPU platforms. Our contributions are twofold: First, an exact dual formulation of LMA removes the memory dependence on the parameter…
PDF arXiv
arXiv cs.LGOct 16
Dynamical Kernel Attention: Estimating the Present from a Record of the Past
Peter Racioppo
Standard self-attention compares tokens by similarity but does not account for how much of a past state survives to the present. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state evolving under stochastic dynamics. As a past state decays toward equilibrium under the dynamics, the information it carries about the present vanishes, bounding the total evidence a head can draw. Unlike recency biases, DKA can abstain when relevant evidence is weak and guarantees a bounded error when old tokens are discarded. DKA preserves the parallel structure and computational cost of standard attention. We present the model and its theory; empirical evaluation is deferred to a subsequent version.
PDF arXiv
arXiv cs.AIOct 16
ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning
Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Longtao Huang +2
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prevalent strategy for adapting large pre-trained models. Among PEFT techniques, reparameterization-based methods, particularly those that employ low-rank or sparsity structures, have attracted significant attention. However, these approaches often underperform full fine-tuning. Specifically, low-rank methods constrain the update rank, which is misaligned with the inherently high-rank nature of full fine-tuning. In contrast, sparsity-based techniques limit the flexibility of the parameter space and compromise connectivity across weights. To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints. A straightforward approach is to update weights via a product of Toeplitz matrices. However, similar to RNNs, long multiplicative chains often lead to gradient instability. To address this issue, we propose \textbf{ToPA}, a more refined version of this idea that replaces standard Toeplitz matrices with block-wise Toeplitz, greatly increasing each factor's capacity and…
PDF arXiv
arXiv cs.LGOct 16
Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training
Xinyu Luo, Jincheng Zhou, Mengbo Wang, Zilin Shen, Nadia Atallah Lanman +3
A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace and map the update back to the full model. Most existing work either recomputes the gradient subspace at certain intervals, or performs post-hoc adaptation to catch up to the moving subspace. In this work, we explicitly identify in practice the slow, stable manner in which LLM gradient subspaces drift across time. To this end, we propose Ori, a novel low-rank optimizer for training LLMs that captures the shifting principal gradient directions over time to achieve state-of-the-art accuracy under stringent low-rank constraints. Ori employs a streaming $k$-PCA algorithm inspired by Oja++, whose low per-step cost enables high-frequency, on-the-fly subspace tracking. In practice, Ori reduces the performance gap of the state-of-the-art low-rank optimizers to full-rank optimizers by up to 69.3\% on a 1B LLaMA pretraining task, and achieves the best performance on 10 out of 16 task scenarios in Roberta-Base fine-tuning. Together, these contributions allow Ori to further push the performance and efficiency…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 34m ago

Graph learning

Found by the topic model: graph learning, gradient descent, bit quantization.

172papers this week−24.2% vs last week

All papers · 1k With code · 11

Newest Most relevant

1k papers this month

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantizing training and rollout paths independently in RL leads to mismatched behavior and reduced performance.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not diagnose the unique challenges of graph-based decision tasks for non-language models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Non-compositional sparse autoencoder features1 paper raises it · 1 addresses it
Sparse autoencoder features often exhibit higher-order interactions, limiting interpretability and control.
- Behavioral Compositionality of Sparse Autoencoder Representationsraises · addresses

Named by an AI from these papers' abstracts.