Papers

Field

Top tools here

Open problems in Graph learning

Accurate low-bit quantization of recurrent states in linear attention2 papers raise it · 2 address it
Enables efficient concurrent serving of many sequences without accuracy loss in graph-based or sequential models.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent Statesraises · addresses
Train-rollout discrepancy in low-precision RL for MoE models1 paper raises it · 1 addresses it
Reducing this gap improves the stability and efficiency of reinforcement learning with quantized models.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in pruning for decoding efficiency1 paper raises it · 1 addresses it
Pruning guided by natural sequences degrades performance during self-generated token decoding, limiting speedups.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Calibration-free quantization for diffusion language models1 paper raises it · 1 addresses it
Avoids reliance on calibration data, which is impractical for models with shifting activation distributions.
- SoloQ: Calibration-Free Quantization for Diffusion Language Modelsraises · addresses
Benchmarking System One models on graph tasks1 paper raises it · 1 addresses it
Lacks standardized evaluation to assess graph understanding and decision-making capabilities of non-LLM models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

gradient descent · learning rate · neural networks241 papers41 this week
graph learning · graph neural · graph neural networks277 papers32 this week
neural networks · neural network · arnold networks159 papers25 this week
memory efficient · optimizers · low rank adaptation166 papers21 this week
bit quantization · quantization aware training · quantization aware163 papers19 this week
linear attention · softmax attention · transformer attention109 papers8 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGSep 28
Temporal Heterogeneous Graph Pretraining for Relational Deep Learning
Yixin Peng, Er Jin, Diego Collarana, Stefan Decker
Relational deep learning models database rows and foreign-key links as a heterogeneous graph for prediction from record attributes and relational context. These graphs contain two distinct temporal signals: record age changes with the prediction cutoff, while intervals between observed records remain fixed. Prior work often treats time as a single signal or studies temporal representation and pretraining separately. We investigate how explicitly encoding both signals affects temporal pretraining for downstream tasks. Our framework combines Multi-scale Time Encoding, which captures record age using learnable time scales and type-specific projections, with Rotary Time Encoding, which represents signed inter-record intervals through rotary transformations during graph propagation. We pair these encodings with three self-supervised objectives: historical relation recovery, horizon-aware future relation activity prediction, and temporal subgraph contrast. All inputs respect their observation cutoffs. Pretraining proceeds in two stages: subgraph contrast first learns neighborhood representations, followed by refinement through either relation recovery or future activity prediction. We…
PDF arXiv
arXiv cs.LGSep 28
Subgroup Rank-1 Lattice for Practical High-dimensional Black-box Integral Approximation
Yueming Lyu
Estimating integrals of black-box, high-dimensional functions, from expectations and kernel mean embeddings to the softmax kernel in self-attention, is a basic subroutine in machine learning. Rank-1 lattice rules suit this setting: they query the integrand only at a fixed point set and need no gradients. When the $n$ points serve as a design matrix $X\in\mathbb{R}^{n\times d}$ for a feature map, however, computing $Ψ(X)^\top v$ or $Ψ(X)w$ for an elementwise nonlinearity $Ψ$ costs $O(nd)$ time and memory for any standard quasi-Monte Carlo point set. We study subgroup rank-1 lattices, whose Korobov generator $(1,t,\dots,t^{d-1})$ uses a scalar $t$ of fixed multiplicative order $m$. Splitting $\mathbb{F}_n^\times$ into cosets of $\langle t\rangle$ reduces both maps to short cyclic correlations evaluated by FFT, giving exact results for arbitrary $Ψ$ in $O(n\log m)$ time and $O(n)$ memory, without forming $X$. Since fixing $m$ falls outside classical component-by-component theory, we prove convergence directly: via resultants with the cyclotomic polynomial $Φ_m$, the squared worst-case error in the Korobov space decays as $O(n^{-(α-1)/(m-1)})$ for prime $m\ge d+1$, and this threshold…
PDF arXiv
arXiv cs.LGSep 28
Explaining Hyperbolic Neural Networks via Geometry-Aware Relevance Propagation
Ping Xiong, Shanglin Li, Yi Ding, Thomas Schnake, Shinichi Nakajima
Hyperbolic neural networks introduce geometric operations that require explicit treatment in relevance propagation. Equivalent geometric realizations can produce different feature attributions, even when local relevance is conserved. We study this problem through Geometric Representation Invariance (GRI), a specialization of Implementation Invariance, and zero-curvature consistency, which requires identity relevance propagation when a geometric module approaches the identity. We propose LRP-radial-all for origin-centered radial modules, treating geometric scaling as modulation and assigning relevance entirely to the signal branch. The rule conserves relevance, is invariant to equivalent radial factorizations, and satisfies zero-curvature consistency, yielding GRI for a specified Poincaré-Lorentz logarithmic-map construction. In contrast, a conservative LRP-half baseline can violate both consistency criteria. Experiments on hyperbolic MNIST, sEEG, and CIFAR-10 classifiers assess attribution fidelity, qualitative explanations, and runtime. LRP-radial-all achieves competitive attribution fidelity across datasets with runtime comparable to Gradient$\times$Input and substantially…
PDF arXiv
arXiv cs.LGSep 28
Propagate, Then Sharpen: Post-Hoc Refinement of Frozen Node Classifiers
Preben Johnsen Bentdal, Nello Blaser, Xue-Cheng Tai
We study post-hoc refinement of frozen node classifiers: given only the graph $G$ and class distributions $Q$ predicted by a frozen model, can we improve accuracy without access to node features, model parameters, or gradients? APPNP answers this by propagating logits with a restart towards the initial predictions, minimizing the anchored Dirichlet energy. Instead, we consider the Potts energy, and decompose it into a Dirichlet term, which penalizes disagreement between neighbouring nodes, and a Gini term, which penalizes indecision within each node. This decomposition motivates Propagate, Then Sharpen (PtS), which alternates between propagation of class probabilities and node-wise, mass-preserving sharpening, with only one additional hyperparameter selected using labelled validation nodes. Across nine homophilic graphs, with a frozen MLP backbone, PtS improves mean test accuracy over independently tuned APPNP by $1.71$ percentage points on clean inputs and $3.90$ under severe Gaussian feature corruption. Gains over APPNP become smaller, but remain positive with frozen GCN and GraphSAGE backbones. Sharpening also removes most of the accuracy loss of deep propagation: on clean…
PDF arXiv
arXiv cs.LGSep 28
Interrelating Fruchterman-Reingold Graph Visualization and Agglomerative Clustering
Alexandre Benatti, Luciano da F. Costa
Graph visualization methods and agglomerative clustering have been frequently considered in data analysis and pattern recognition. Because these approaches are interrelated and complementary, it is of particular interest to investigate their associations. In this work, we study the possible relationship between the Fruchterman-Reingold graph visualization method and four types of agglomerative clustering adopting single- and complete-linkage, average, and Ward's linkage criteria. Three types of datasets have been considered in 2 and 10 dimensions, as well as the PCA projection of the latter to two dimensions. The results obtained suggest that the relationship between the methods considered did not vary much for the three types of data mentioned above. At the same time, the agglomerative methods tended to yield results that are mostly similar to each other, while presenting moderate similarity with the original data. The Fruchterman-Reingold visualization resulted similar to the original data, but exhibited relatively smaller similarity to the agglomerative methods.
PDF arXiv
arXiv cs.LGSep 28
Universality and Generalization of Causal Transformers Across Context Lengths
Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré
Long contexts are central to modern transformer systems, but most expressivity results choose a different network for each fixed sequence length. We study whether one masked transformer can approximate causal token-to-token maps uniformly over sequences of arbitrary length sampling a fixed normalized horizon. To relate sampling resolutions, we model tokens by $α$-Hölder sequences or, more generally, a common modulus of continuity. Our notion of continuity across resolutions characterizes the causal families admitting uniform approximation on these compact input classes by a single transformer with length-independent parameters. The result extends to the infinite-length mean-field limit, where tokens form continuous curves and masked attention becomes a causal time integral. For bounded regression with target maps satisfying a $β$-smooth stability condition defined using regular test functions, quantitative approximation yields a generalization bound: exact empirical risk minimization over suitably sized bounded-weight transformers gives root mean-square prediction error $O((\log\log N/\log N)^{β/(d+2)})$ from $N$ iid labeled sequences. The bound holds at fixed confidence on the…
PDF arXiv

Papers

Topics in this field

gradient descent · learning rate · neural networks241 papers41 this week
graph learning · graph neural · graph neural networks277 papers32 this week
neural networks · neural network · arnold networks159 papers25 this week
memory efficient · optimizers · low rank adaptation166 papers21 this week
bit quantization · quantization aware training · quantization aware163 papers19 this week
linear attention · softmax attention · transformer attention109 papers8 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.LGSep 28
Temporal Heterogeneous Graph Pretraining for Relational Deep Learning
Yixin Peng, Er Jin, Diego Collarana, Stefan Decker
Relational deep learning models database rows and foreign-key links as a heterogeneous graph for prediction from record attributes and relational context. These graphs contain two distinct temporal signals: record age changes with the prediction cutoff, while intervals between observed records remain fixed. Prior work often treats time as a single signal or studies temporal representation and pretraining separately. We investigate how explicitly encoding both signals affects temporal pretraining for downstream tasks. Our framework combines Multi-scale Time Encoding, which captures record age using learnable time scales and type-specific projections, with Rotary Time Encoding, which represents signed inter-record intervals through rotary transformations during graph propagation. We pair these encodings with three self-supervised objectives: historical relation recovery, horizon-aware future relation activity prediction, and temporal subgraph contrast. All inputs respect their observation cutoffs. Pretraining proceeds in two stages: subgraph contrast first learns neighborhood representations, followed by refinement through either relation recovery or future activity prediction. We…
PDF arXiv
arXiv cs.LGSep 28
Subgroup Rank-1 Lattice for Practical High-dimensional Black-box Integral Approximation
Yueming Lyu
Estimating integrals of black-box, high-dimensional functions, from expectations and kernel mean embeddings to the softmax kernel in self-attention, is a basic subroutine in machine learning. Rank-1 lattice rules suit this setting: they query the integrand only at a fixed point set and need no gradients. When the $n$ points serve as a design matrix $X\in\mathbb{R}^{n\times d}$ for a feature map, however, computing $Ψ(X)^\top v$ or $Ψ(X)w$ for an elementwise nonlinearity $Ψ$ costs $O(nd)$ time and memory for any standard quasi-Monte Carlo point set. We study subgroup rank-1 lattices, whose Korobov generator $(1,t,\dots,t^{d-1})$ uses a scalar $t$ of fixed multiplicative order $m$. Splitting $\mathbb{F}_n^\times$ into cosets of $\langle t\rangle$ reduces both maps to short cyclic correlations evaluated by FFT, giving exact results for arbitrary $Ψ$ in $O(n\log m)$ time and $O(n)$ memory, without forming $X$. Since fixing $m$ falls outside classical component-by-component theory, we prove convergence directly: via resultants with the cyclotomic polynomial $Φ_m$, the squared worst-case error in the Korobov space decays as $O(n^{-(α-1)/(m-1)})$ for prime $m\ge d+1$, and this threshold…
PDF arXiv
arXiv cs.LGSep 28
Explaining Hyperbolic Neural Networks via Geometry-Aware Relevance Propagation
Ping Xiong, Shanglin Li, Yi Ding, Thomas Schnake, Shinichi Nakajima
Hyperbolic neural networks introduce geometric operations that require explicit treatment in relevance propagation. Equivalent geometric realizations can produce different feature attributions, even when local relevance is conserved. We study this problem through Geometric Representation Invariance (GRI), a specialization of Implementation Invariance, and zero-curvature consistency, which requires identity relevance propagation when a geometric module approaches the identity. We propose LRP-radial-all for origin-centered radial modules, treating geometric scaling as modulation and assigning relevance entirely to the signal branch. The rule conserves relevance, is invariant to equivalent radial factorizations, and satisfies zero-curvature consistency, yielding GRI for a specified Poincaré-Lorentz logarithmic-map construction. In contrast, a conservative LRP-half baseline can violate both consistency criteria. Experiments on hyperbolic MNIST, sEEG, and CIFAR-10 classifiers assess attribution fidelity, qualitative explanations, and runtime. LRP-radial-all achieves competitive attribution fidelity across datasets with runtime comparable to Gradient$\times$Input and substantially…
PDF arXiv
arXiv cs.LGSep 28
Propagate, Then Sharpen: Post-Hoc Refinement of Frozen Node Classifiers
Preben Johnsen Bentdal, Nello Blaser, Xue-Cheng Tai
We study post-hoc refinement of frozen node classifiers: given only the graph $G$ and class distributions $Q$ predicted by a frozen model, can we improve accuracy without access to node features, model parameters, or gradients? APPNP answers this by propagating logits with a restart towards the initial predictions, minimizing the anchored Dirichlet energy. Instead, we consider the Potts energy, and decompose it into a Dirichlet term, which penalizes disagreement between neighbouring nodes, and a Gini term, which penalizes indecision within each node. This decomposition motivates Propagate, Then Sharpen (PtS), which alternates between propagation of class probabilities and node-wise, mass-preserving sharpening, with only one additional hyperparameter selected using labelled validation nodes. Across nine homophilic graphs, with a frozen MLP backbone, PtS improves mean test accuracy over independently tuned APPNP by $1.71$ percentage points on clean inputs and $3.90$ under severe Gaussian feature corruption. Gains over APPNP become smaller, but remain positive with frozen GCN and GraphSAGE backbones. Sharpening also removes most of the accuracy loss of deep propagation: on clean…
PDF arXiv
arXiv cs.LGSep 28
Interrelating Fruchterman-Reingold Graph Visualization and Agglomerative Clustering
Alexandre Benatti, Luciano da F. Costa
Graph visualization methods and agglomerative clustering have been frequently considered in data analysis and pattern recognition. Because these approaches are interrelated and complementary, it is of particular interest to investigate their associations. In this work, we study the possible relationship between the Fruchterman-Reingold graph visualization method and four types of agglomerative clustering adopting single- and complete-linkage, average, and Ward's linkage criteria. Three types of datasets have been considered in 2 and 10 dimensions, as well as the PCA projection of the latter to two dimensions. The results obtained suggest that the relationship between the methods considered did not vary much for the three types of data mentioned above. At the same time, the agglomerative methods tended to yield results that are mostly similar to each other, while presenting moderate similarity with the original data. The Fruchterman-Reingold visualization resulted similar to the original data, but exhibited relatively smaller similarity to the agglomerative methods.
PDF arXiv
arXiv cs.LGSep 28
Universality and Generalization of Causal Transformers Across Context Lengths
Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré
Long contexts are central to modern transformer systems, but most expressivity results choose a different network for each fixed sequence length. We study whether one masked transformer can approximate causal token-to-token maps uniformly over sequences of arbitrary length sampling a fixed normalized horizon. To relate sampling resolutions, we model tokens by $α$-Hölder sequences or, more generally, a common modulus of continuity. Our notion of continuity across resolutions characterizes the causal families admitting uniform approximation on these compact input classes by a single transformer with length-independent parameters. The result extends to the infinite-length mean-field limit, where tokens form continuous curves and masked attention becomes a causal time integral. For bounded regression with target maps satisfying a $β$-smooth stability condition defined using regular test functions, quantitative approximation yields a generalization bound: exact empirical risk minimization over suitably sized bounded-weight transformers gives root mean-square prediction error $O((\log\log N/\log N)^{β/(d+2)})$ from $N$ iid labeled sequences. The bound holds at fixed confidence on the…
PDF arXiv

Activity · 12 weeks

Open problems in Graph learning

Accurate low-bit quantization of recurrent states in linear attention2 papers raise it · 2 address it
Enables efficient concurrent serving of many sequences without accuracy loss in graph-based or sequential models.
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantizationraises · addresses
- PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent Statesraises · addresses
Train-rollout discrepancy in low-precision RL for MoE models1 paper raises it · 1 addresses it
Reducing this gap improves the stability and efficiency of reinforcement learning with quantized models.
- TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Modelsraises · addresses
Distribution shift in pruning for decoding efficiency1 paper raises it · 1 addresses it
Pruning guided by natural sequences degrades performance during self-generated token decoding, limiting speedups.
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inferenceraises · addresses
Calibration-free quantization for diffusion language models1 paper raises it · 1 addresses it
Avoids reliance on calibration data, which is impractical for models with shifting activation distributions.
- SoloQ: Calibration-Free Quantization for Diffusion Language Modelsraises · addresses
Benchmarking System One models on graph tasks1 paper raises it · 1 addresses it
Lacks standardized evaluation to assess graph understanding and decision-making capabilities of non-LLM models.
- GraphDecisionBench: Benchmarking System One Models on Graph Tasksraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 2h ago

Graph learning

Found by the topic model: graph learning, gradient descent, bit quantization.

146papers this week−35.4% vs last week

All papers · 1.1k With code · 11

Newest Most relevant

1k papers this month