Papers

Field

Top tools here

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantizing training and rollout paths independently in RL leads to mismatched behavior and reduced performance.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not diagnose the unique challenges of graph-based decision tasks for non-language models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Non-compositional sparse autoencoder features1 paper raises it · 1 addresses it
Sparse autoencoder features often exhibit higher-order interactions, limiting interpretability and control.
- Behavioral Compositionality of Sparse Autoencoder Representationsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

gradient descent · learning rate · neural networks221 papers43 this week
neural networks · neural network · arnold networks149 papers37 this week
graph learning · graph neural · graph neural networks263 papers35 this week
memory efficient · optimizers · low rank adaptation157 papers25 this week
bit quantization · quantization aware training · quantization aware152 papers20 this week
linear attention · softmax attention · transformer attention106 papers14 this week

Topics in this field

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.LGOct 16
Anisotropic Representation Dynamics in Transformers
Nguyen Do, My T. Thai
Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preserved and how sensitive the model is to perturbations. Yet it remains unclear when changes in attention weights can turn a representation difference that would otherwise shrink into one that grows. We develop a \textbf{Mixture-Gated Theory of Anisotropic Representation Dynamics} that separates self-attention into value transport, query gating, and key gating. Our analysis reveals how the geometry of attended representations constrains gating and how interactions between these mechanisms can reverse contraction induced by value transport. We formalize this phenomenon as gating-induced regime switching, where a representation difference contracts under fixed attention routing but expands under full attention. We derive a necessary and sufficient spectral criterion that certifies the existence or absence of switching and identifies switching directions without exhaustive search. Beyond existence, our theory characterizes maximum relative switching strength, certifies the severity of contraction that gating can overcome, and determines the…
PDF arXiv
arXiv cs.AROct 16
qGAMMA: A Framework for Optimal DNN Mapping on Multi-Precision Accelerators
Leandro Fiorin, Luigi Altamura, Cristina Silvano
This work describes qGAMMA, a flexible framework for exploring and optimizing the mapping of mixed-precision Deep Neural Network (DNN) models onto general-purpose spatial accelerators that support a range of fixed- and floating-point precisions. qGAMMA navigates the design space using a domain-specific genetic algorithm and an analytical performance model based on hardware synthesis results. The framework identifies the optimal mapping and accelerator configuration that minimizes target metrics. Extensive evaluations across DNN workloads show that qGAMMA offers valuable insights for designing mixed-precision applications, enabling better trade-offs between performance, energy efficiency, and resource utilization.
PDF arXiv
arXiv cs.LGOct 16
Local proxies for memory-efficient training
Wu S. Zihan, Brian A. Yuan, Guillaume Bellec
Local learning algorithms can reshape resource requirements of artificial intelligence. The memory bottleneck of standard end-to-end backpropagation lies in the need to store all network activations until the loss gradient has been propagated back through. Local learning avoids this, but typically at a cost in accuracy. We study how to achieve accurate and memory efficient local learning in transformers. For each transformer block, we train a local proxy network to predict the global network output via distillation. Gradients can be independently propagated through this local proxy while estimating well the backpropagation gradients. In theory, this approach is exact for deep linear networks; in non-linear residual models, the gradient error can be bounded when using the appropriate residual proxy. We verify this numerically and demonstrate that optimizing the distillation loss improves gradient estimation. Evaluating the performance of proxy propagation on CIFAR-100 in vision transformers, we find that it achieves a better accuracy/memory trade-off compared to other local algorithms. Finally, we show that the method is further relevant for fine-tuning large language models on a…
PDF arXiv
arXiv cs.LGOct 16
Towards a GraphNARX model for structural dynamics
George Tsialiamanis, Keith Worden
The current paper introduces the GraphNARX model, a graph-based nonlinear autoregressive exogenous model for structural dynamics. The model is built to account for the geometry of structures and to model dynamic behaviour as the interaction between elements of the structure. Similar to other types of NARX models, GraphNARX is used for system identification. Nodal attributes of the graph encode past displacements and nodal forces, and the output of the model is the next-timestep displacement. The graph formulation of the model allows it to generalise to structures of different dimensions and degrees of freedom. Moreover, because graph-learning is performed locally between the nodes, the GraphNARX model can also learn to infer displacements of nodes for which data are not available, i.e.\ to perform virtual sensing. The proposed model's performance is evaluated in a system-identification framework and a virtual-sensing framework, and achieves good accuracy and exhibits generalisation capabilities to structures of different sizes.
PDF arXiv
arXiv stat.MLOct 16
Distinguishability and Recovery for a Doubly Sparse Spiked Wigner Model
Ioana Dumitriu, JD Flynn, Zhichao Wang
We study distinguishability and recovery of a sparse rank-one signal in a spiked Wigner model with independently sparse noise through a variety of algorithms and parameter regimes. The signal and noise have separate sparsity parameters, while the signal-to-noise ratio (SNR) remains of constant order as the matrix size grows. For sub-Gaussian noise and spikes whose nonzero coordinates are bounded away from zero, a leave-one-out quadratic thresholding procedure (a generalization of Ding et al., 2024) identifies the support. Sign synchronization yields exact recovery for a Rademacher spike, and spectral refinement yields strong recovery. The quadratic thresholding search parameter gives an explicit tradeoff between signal sparsity, noise sparsity, and computational cost. We also study extensions to sub-Gaussian spike priors through diagonal and quadratic thresholding. For Rademacher spikes with continuous nonzero noise, we develop an identical search algorithm that exploits unblurred signal entries, the existence of which permits distinguishability; connectivity of the graph they reveal provides a sufficient condition for exact recovery. These guarantees do not require moment…
PDF arXiv
arXiv cs.LGOct 16
"Deep lifts": Neural networks can generalize even with the dumbest optimizer
Sunil Madhow, Yu-Xiang Wang
Given a labeled dataset $\{(x_i, y_i)\}_{i = 1}^n$, consider drawing a parameter vector $\hatθ$ uniformly at random from those that interpolate the data within a width-$W$ depth-$D$ ReLU architecture with continuous weights bounded within $[-M, M]$. We study the generalization properties of $\hatθ$ in the case where the dataset can, in fact, be explained by a smaller $w$-by-$d$ ReLU network. Despite the obliviousness of the selection rule for $\hatθ$, it adaptively generalizes at the rate $\tilde O\left(\frac{wdWD}{n}\right)$, rather than being governed by the VC dimension of the full $W$-by-$D$ hypothesis class. The reason is due to a particular inductive bias in ReLU networks: the $w$-by-$d$ parameter *witnesses* an exponential volume of $W$-by-$D$ interpolating parameters, which dwarfs the volume of poorly generalizing parameters (conditioned on interpolation). We explicitly construct this set of near-equivalent $W$-by-$D$ parameters by a proof technique called *deep lifting*, and then tie the achievable volume to stability properties of the $w$-by-$d$ witness. Examples show that these techniques allow us to convert approximation-theoretic results into generalization…
PDF arXiv

Papers

Topics in this field

gradient descent · learning rate · neural networks221 papers43 this week
neural networks · neural network · arnold networks149 papers37 this week
graph learning · graph neural · graph neural networks263 papers35 this week
memory efficient · optimizers · low rank adaptation157 papers25 this week
bit quantization · quantization aware training · quantization aware152 papers20 this week
linear attention · softmax attention · transformer attention106 papers14 this week

Active contributors · 90 days

Pietro LiòLocaLeJEPA: Maximum Entropy Representations under Geometric Priors8
Dan AlistarhWhat Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes6
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach6
Rebekka BurkholzAC/DC on a Budget - Alternating Sparse Phases5
Haggai MaronGraph Metanetworks for Transformer Based Language Models5

Activity · 12 weeks

Coming with more sources: repos

Recent papers

All in Research →

arXiv cs.LGOct 16
Anisotropic Representation Dynamics in Transformers
Nguyen Do, My T. Thai
Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preserved and how sensitive the model is to perturbations. Yet it remains unclear when changes in attention weights can turn a representation difference that would otherwise shrink into one that grows. We develop a \textbf{Mixture-Gated Theory of Anisotropic Representation Dynamics} that separates self-attention into value transport, query gating, and key gating. Our analysis reveals how the geometry of attended representations constrains gating and how interactions between these mechanisms can reverse contraction induced by value transport. We formalize this phenomenon as gating-induced regime switching, where a representation difference contracts under fixed attention routing but expands under full attention. We derive a necessary and sufficient spectral criterion that certifies the existence or absence of switching and identifies switching directions without exhaustive search. Beyond existence, our theory characterizes maximum relative switching strength, certifies the severity of contraction that gating can overcome, and determines the…
PDF arXiv
arXiv cs.AROct 16
qGAMMA: A Framework for Optimal DNN Mapping on Multi-Precision Accelerators
Leandro Fiorin, Luigi Altamura, Cristina Silvano
This work describes qGAMMA, a flexible framework for exploring and optimizing the mapping of mixed-precision Deep Neural Network (DNN) models onto general-purpose spatial accelerators that support a range of fixed- and floating-point precisions. qGAMMA navigates the design space using a domain-specific genetic algorithm and an analytical performance model based on hardware synthesis results. The framework identifies the optimal mapping and accelerator configuration that minimizes target metrics. Extensive evaluations across DNN workloads show that qGAMMA offers valuable insights for designing mixed-precision applications, enabling better trade-offs between performance, energy efficiency, and resource utilization.
PDF arXiv
arXiv cs.LGOct 16
Local proxies for memory-efficient training
Wu S. Zihan, Brian A. Yuan, Guillaume Bellec
Local learning algorithms can reshape resource requirements of artificial intelligence. The memory bottleneck of standard end-to-end backpropagation lies in the need to store all network activations until the loss gradient has been propagated back through. Local learning avoids this, but typically at a cost in accuracy. We study how to achieve accurate and memory efficient local learning in transformers. For each transformer block, we train a local proxy network to predict the global network output via distillation. Gradients can be independently propagated through this local proxy while estimating well the backpropagation gradients. In theory, this approach is exact for deep linear networks; in non-linear residual models, the gradient error can be bounded when using the appropriate residual proxy. We verify this numerically and demonstrate that optimizing the distillation loss improves gradient estimation. Evaluating the performance of proxy propagation on CIFAR-100 in vision transformers, we find that it achieves a better accuracy/memory trade-off compared to other local algorithms. Finally, we show that the method is further relevant for fine-tuning large language models on a…
PDF arXiv
arXiv cs.LGOct 16
Towards a GraphNARX model for structural dynamics
George Tsialiamanis, Keith Worden
The current paper introduces the GraphNARX model, a graph-based nonlinear autoregressive exogenous model for structural dynamics. The model is built to account for the geometry of structures and to model dynamic behaviour as the interaction between elements of the structure. Similar to other types of NARX models, GraphNARX is used for system identification. Nodal attributes of the graph encode past displacements and nodal forces, and the output of the model is the next-timestep displacement. The graph formulation of the model allows it to generalise to structures of different dimensions and degrees of freedom. Moreover, because graph-learning is performed locally between the nodes, the GraphNARX model can also learn to infer displacements of nodes for which data are not available, i.e.\ to perform virtual sensing. The proposed model's performance is evaluated in a system-identification framework and a virtual-sensing framework, and achieves good accuracy and exhibits generalisation capabilities to structures of different sizes.
PDF arXiv
arXiv stat.MLOct 16
Distinguishability and Recovery for a Doubly Sparse Spiked Wigner Model
Ioana Dumitriu, JD Flynn, Zhichao Wang
We study distinguishability and recovery of a sparse rank-one signal in a spiked Wigner model with independently sparse noise through a variety of algorithms and parameter regimes. The signal and noise have separate sparsity parameters, while the signal-to-noise ratio (SNR) remains of constant order as the matrix size grows. For sub-Gaussian noise and spikes whose nonzero coordinates are bounded away from zero, a leave-one-out quadratic thresholding procedure (a generalization of Ding et al., 2024) identifies the support. Sign synchronization yields exact recovery for a Rademacher spike, and spectral refinement yields strong recovery. The quadratic thresholding search parameter gives an explicit tradeoff between signal sparsity, noise sparsity, and computational cost. We also study extensions to sub-Gaussian spike priors through diagonal and quadratic thresholding. For Rademacher spikes with continuous nonzero noise, we develop an identical search algorithm that exploits unblurred signal entries, the existence of which permits distinguishability; connectivity of the graph they reveal provides a sufficient condition for exact recovery. These guarantees do not require moment…
PDF arXiv
arXiv cs.LGOct 16
"Deep lifts": Neural networks can generalize even with the dumbest optimizer
Sunil Madhow, Yu-Xiang Wang
Given a labeled dataset $\{(x_i, y_i)\}_{i = 1}^n$, consider drawing a parameter vector $\hatθ$ uniformly at random from those that interpolate the data within a width-$W$ depth-$D$ ReLU architecture with continuous weights bounded within $[-M, M]$. We study the generalization properties of $\hatθ$ in the case where the dataset can, in fact, be explained by a smaller $w$-by-$d$ ReLU network. Despite the obliviousness of the selection rule for $\hatθ$, it adaptively generalizes at the rate $\tilde O\left(\frac{wdWD}{n}\right)$, rather than being governed by the VC dimension of the full $W$-by-$D$ hypothesis class. The reason is due to a particular inductive bias in ReLU networks: the $w$-by-$d$ parameter *witnesses* an exponential volume of $W$-by-$D$ interpolating parameters, which dwarfs the volume of poorly generalizing parameters (conditioned on interpolation). We explicitly construct this set of near-equivalent $W$-by-$D$ parameters by a proof technique called *deep lifting*, and then tie the achievable volume to stability properties of the $w$-by-$d$ witness. Examples show that these techniques allow us to convert approximation-theoretic results into generalization…
PDF arXiv

Open problems in Graph learning

Quantization error propagation in recurrent states1 paper raises it · 1 addresses it
Low-precision recurrent states in linear attention degrade accuracy due to error accumulation over time and across dimensions.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
Train-rollout discrepancy in low-precision RL1 paper raises it · 1 addresses it
Quantizing training and rollout paths independently in RL leads to mismatched behavior and reduced performance.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in decoding-aware pruning1 paper raises it · 1 addresses it
Pruning guided by natural sequences fails to account for self-generated token distributions during decoding, hurting performance.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Lack of benchmarks for graph decision-making in System One models1 paper raises it · 1 addresses it
Existing benchmarks do not diagnose the unique challenges of graph-based decision tasks for non-language models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses
Non-compositional sparse autoencoder features1 paper raises it · 1 addresses it
Sparse autoencoder features often exhibit higher-order interactions, limiting interpretability and control.
- Behavioral Compositionality of Sparse Autoencoder Representationsraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 35m ago

Graph learning

Found by the topic model: graph learning, gradient descent, bit quantization.

174papers this week−23.3% vs last week

All papers · 1k With code · 11

Newest Most relevant

1k papers this month