Papers

Field

Top tools here

Open problems in Physics informed neural

Accurate long-term PDE dynamics prediction1 paper raises it · 1 addresses it
Reconstruction-based learning often fails to capture the evolution of physical fields over time, limiting predictive accuracy for parametric PDEs.
- PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEsraises · addresses
Numerical instability in softmax and cross-entropy1 paper raises it · 1 addresses it
Standard implementations suffer from gradient collapse and non-zero gradients when logits match, harming training stability and convergence.
- A Fix for Softmax and Cross-Entropyraises · addresses
Token collapse in graph soft prompting1 paper raises it · 0 address it
Graph-specific tokens in LLM prompts often become indistinguishable, failing to encode meaningful graph information.
- On the Token Collapse Phenomenon in Textual Graph Soft Promptingraises
Efficient sparse GNN training with dynamic topology1 paper raises it · 1 addresses it
Reusing a fixed sparse graph locks training to a static topology, while dynamic sampling is computationally expensive.
- EDiS: Edge Disjoint Subgraph Sparsification Framework for Graph Neural Networksraises · addresses
Automated diagnosis and improvement of numerical solvers1 paper raises it · 1 addresses it
Execution feedback reveals poor performance but rarely explains the cause or how to fix it, hindering solver optimization.
- Training Numerical Intelligence via Auto-Diagnosis and Skill Discoveryraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

graph neural · graph learning · graph neural networks266 papers32 this week
physics informed neural · informed neural networks · neural operators303 papers29 this week
relu networks · deep neural · gradient descent163 papers25 this week
kernel · svm · gaussian126 papers14 this week
arnold networks · symmetries · parameter symmetries86 papers11 this week
muon optimizer · practical muon · gradient descent49 papers7 this week

Topics in this field

Active contributors · 90 days

Pietro LiòOn the Token Collapse Phenomenon in Textual Graph Soft Prompting9
Anima AnandkumarNeural Operator Splitting for Fluid Dynamics in Complex Geometry5
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Alessio BorgiLocaLeJEPA: Maximum Entropy Representations under Geometric Priors4
Jing XuSame Target, Different Origins: Aligning Latent Rollout for Consistent PDE Emulation4

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGOct 20
AnchorRM: Parallel Intra-Batch State Reconstruction with Anchor-Based Residual Memory for Temporal Graph Neural Networks
Haroon Gharwi, Kai Shu, Yue Dai
Memory-based temporal graph neural networks (TGN) rely on recurrent node memories to capture evolving interaction histories. However, temporal mini-batching introduces a trade-off between computational efficiency and temporal fidelity: when a node appears multiple times within the same batch, later interactions are evaluated using stale pre-batch memory because the intermediate memory updates have not yet been committed. This problem becomes more severe as temporal batch size increases. To overcome this challenge, we propose AnchorRM, an anchor-based residual memory mechanism that reconstructs event-specific intra-batch node states without executing sequential recurrent updates. AnchorRM treats the last committed node memory at the beginning of each batch as a pre-batch fixed anchor and uses causally preceding same-batch interactions to infer the missing state evolution relative to that anchor. The learned residual then calibrates the anchor to reconstruct an event-specific node memory for each interaction. Because each event-specific residual is predicted directly from the fixed anchor and its causal history, reconstructed states do not depend on previously reconstructed states…
PDF arXiv
arXiv math.NAOct 20
Soliton Profiles in 1D: Enhanced Neural-Network Solvers and Constrained Deep Ritz Methods
Chandler Haight, Alex D Rodriguez, Svetlana Roudenko
We develop and compare enhanced neural-network solvers for one-dimensional ground-state profiles, which are solutions of $$ Q^{\prime\prime}=bQ-γQ^{p},\quad Q>0,\quad p>1, \quad b, γ>0, $$ the stationary profile equation for the nonlinear Schrödinger, generalized Korteweg-de Vries, and nonlinear Klein-Gordon equations. Its explicit sech-type solutions serve as exact reference for error computation and benchmark hyperparameter selection. We improve on the standard Adam-trained physics-informed neural network (PINN) baseline by introducing higher-order optimization via Gauss-Newton/Levenberg-Marquardt and L-BFGS refinements, as well as Fourier-feature PINNs, and constrained Deep Ritz (DR) formulations. In DR approaches we introduce two new neural-network formulations, which implement two classical variational characterizations: a Nehari-constrained Deep Ritz method, in which the height is learned by projecting onto the Nehari/Pokhozhaev manifold, and a Weinstein Deep Ritz method, in which a normalized profile minimizes the Weinstein quotient. The Deep Ritz formulations perform best overall, in particular, the Weinstein variant gives the best error-runtime balance. The…
PDF arXiv
arXiv cs.LGOct 20
When Parameter Gradient Flows Close in Representation Space
Shengye Tao
The represented gradient flow of a parameterized model depends on a mobility that can retain hidden parameter information. We study when one entire represented mobility map works for every autonomous loss from one fixed initialization. For two-factor matrix representations wider than both output dimensions, with individually full-row-rank factors, we classify all admissible joint ranks: closure occurs precisely at effective scalar-balanced minimal joint rank or a two-value signed spectral family at full joint rank. All intermediate ranks are excluded; positive represented flows belong to known square-root families. For general representations with constant ambient representation and gradient-Lie ranks, ordinary closure is equivalent to a local invariant-level metric in the analytic case and specified smooth regimes. An endpoint section forces the initially unrestricted mobility map to admit a locally smooth representative on the represented image. We also separate testing families: linear losses locally admit closure at every submersion, whereas one fixed translated strongly convex quadratic family fails it on a delayed-bracket example. A fixed output readout gives an ordinary…
PDF arXiv
arXiv cs.LGOct 20
Scaling Graph Reasoning in Language Models with Learnable Graph Positional Encodings
Javier Porras-Valenzuela, Arush Arora, Alejandro Ribeiro
Solving reasoning tasks supported on graphs typically requires providing a Language Reasoning Model (LRM) with a serialized description of the graph. This approach results in the input length growing with the number of edges, which limits the size of the problems that can be solved. Furthermore, even when graphs fit in context, performance sharply decays as the graphs grow. We propose to address this by augmenting LRMs with scalable and permutation equivariant graph positional encodings that modulate attention logits. An additional post-fusion-style module incorporates structural information into hidden states. We compute these positional encodings using a GNN followed by a Graph Transformer. This eliminates the need for serialized adjacencies, reducing the context burden to $O(N)$. We jointly fine-tune the positional encoder and the LRM LoRA adapters using optimal solutions generated by graph-theoretic solvers, resulting in a computationally efficient method. Trained on small graphs, the model exhibits improved size-generalization across graph-theoretic tasks on graphs with up to 1,000 nodes, and produces fewer tokens than a conventional LRM. Additionally, we observe that our…
PDF arXiv
arXiv cs.LGOct 19
Sample-efficient learning with Dense Associative Memories
Andrew Lucas, Dmitry Krotov
We analytically study the evolution of the energy function of Dense Associative Memories (DenseAMs) as they train on randomly sampled codewords of random expander low-density parity-check codes (a.k.a. solutions of random XORSAT problems). We identify two distinct regimes and a transition between them, reminiscent of grokking in neural networks. A regime of small training set size is where the model memorizes a few training examples but cannot discover the underlying rules of the entire codeword dictionary generative process. The opposite regime, corresponding to a larger training set size, is where the model has uncovered the sparse constraints that generated the entire data set after seeing only polynomially many patterns. We also design a novel class of DenseAMs with a thresholding learning rule, and show that any amount of thresholding enables the model to perform simplified sparse Hamiltonian learning, which exponentially reduces the number of training examples needed to learn the true underlying data set. In this regime, the model learns how to discover $\exp[\mathrm{O}(n)]$ solutions from only $\mathrm{O}(\log n)$ training samples, which is asymptotically optimal for any…
PDF arXiv
arXiv nlin.CDOct 19
Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability
Alexander Itin
Gradient descent with momentum can be viewed as a particle moving with friction in the loss landscape. In deep learning the learning rate is usually large, so training is a discrete dynamical system: in many full-batch experiments the sharpness grows to the optimizer's stability limit, the edge of stability (EoS), and training continues there. Physics-inspired optimizers change this motion; Velocity-Regularized Adam (VRAdam), for example, shrinks the step at high speed. We study kinetic maps, in which the step is the gradient of a kinetic energy of the momentum, and ask what they do beyond the EoS. (i) In a reduced model, a kinetic map creates a symmetric period-two orbit beyond the edge, strictly linearly stable if and only if the map's elasticity (the log-log slope of step length versus momentum) lies between 0 and 1; on it, training sits at an effective edge of stability. VRAdam's damper fails above twice the critical learning rate, where its kinetic energy is not convex; the Legendre dual of VRAdam's own Lagrangian is convex and passes at every learning rate. (ii) Because the sharpness rises gradually, the loss of stability is delayed, as in Neishtadt's theory of slowly swept…
PDF arXiv

Papers

Topics in this field

graph neural · graph learning · graph neural networks266 papers32 this week
physics informed neural · informed neural networks · neural operators303 papers29 this week
relu networks · deep neural · gradient descent163 papers25 this week
kernel · svm · gaussian126 papers14 this week
arnold networks · symmetries · parameter symmetries86 papers11 this week
muon optimizer · practical muon · gradient descent49 papers7 this week

Active contributors · 90 days

Pietro LiòOn the Token Collapse Phenomenon in Textual Graph Soft Prompting9
Anima AnandkumarNeural Operator Splitting for Fluid Dynamics in Complex Geometry5
Rong-Hua LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Xunkai LiToward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach5
Alessio BorgiLocaLeJEPA: Maximum Entropy Representations under Geometric Priors4
Jing XuSame Target, Different Origins: Aligning Latent Rollout for Consistent PDE Emulation4

Recent papers

All papers in Research

arXiv cs.LGOct 20
AnchorRM: Parallel Intra-Batch State Reconstruction with Anchor-Based Residual Memory for Temporal Graph Neural Networks
Haroon Gharwi, Kai Shu, Yue Dai
Memory-based temporal graph neural networks (TGN) rely on recurrent node memories to capture evolving interaction histories. However, temporal mini-batching introduces a trade-off between computational efficiency and temporal fidelity: when a node appears multiple times within the same batch, later interactions are evaluated using stale pre-batch memory because the intermediate memory updates have not yet been committed. This problem becomes more severe as temporal batch size increases. To overcome this challenge, we propose AnchorRM, an anchor-based residual memory mechanism that reconstructs event-specific intra-batch node states without executing sequential recurrent updates. AnchorRM treats the last committed node memory at the beginning of each batch as a pre-batch fixed anchor and uses causally preceding same-batch interactions to infer the missing state evolution relative to that anchor. The learned residual then calibrates the anchor to reconstruct an event-specific node memory for each interaction. Because each event-specific residual is predicted directly from the fixed anchor and its causal history, reconstructed states do not depend on previously reconstructed states…
PDF arXiv
arXiv math.NAOct 20
Soliton Profiles in 1D: Enhanced Neural-Network Solvers and Constrained Deep Ritz Methods
Chandler Haight, Alex D Rodriguez, Svetlana Roudenko
We develop and compare enhanced neural-network solvers for one-dimensional ground-state profiles, which are solutions of $$ Q^{\prime\prime}=bQ-γQ^{p},\quad Q>0,\quad p>1, \quad b, γ>0, $$ the stationary profile equation for the nonlinear Schrödinger, generalized Korteweg-de Vries, and nonlinear Klein-Gordon equations. Its explicit sech-type solutions serve as exact reference for error computation and benchmark hyperparameter selection. We improve on the standard Adam-trained physics-informed neural network (PINN) baseline by introducing higher-order optimization via Gauss-Newton/Levenberg-Marquardt and L-BFGS refinements, as well as Fourier-feature PINNs, and constrained Deep Ritz (DR) formulations. In DR approaches we introduce two new neural-network formulations, which implement two classical variational characterizations: a Nehari-constrained Deep Ritz method, in which the height is learned by projecting onto the Nehari/Pokhozhaev manifold, and a Weinstein Deep Ritz method, in which a normalized profile minimizes the Weinstein quotient. The Deep Ritz formulations perform best overall, in particular, the Weinstein variant gives the best error-runtime balance. The…
PDF arXiv
arXiv cs.LGOct 20
When Parameter Gradient Flows Close in Representation Space
Shengye Tao
The represented gradient flow of a parameterized model depends on a mobility that can retain hidden parameter information. We study when one entire represented mobility map works for every autonomous loss from one fixed initialization. For two-factor matrix representations wider than both output dimensions, with individually full-row-rank factors, we classify all admissible joint ranks: closure occurs precisely at effective scalar-balanced minimal joint rank or a two-value signed spectral family at full joint rank. All intermediate ranks are excluded; positive represented flows belong to known square-root families. For general representations with constant ambient representation and gradient-Lie ranks, ordinary closure is equivalent to a local invariant-level metric in the analytic case and specified smooth regimes. An endpoint section forces the initially unrestricted mobility map to admit a locally smooth representative on the represented image. We also separate testing families: linear losses locally admit closure at every submersion, whereas one fixed translated strongly convex quadratic family fails it on a delayed-bracket example. A fixed output readout gives an ordinary…
PDF arXiv
arXiv cs.LGOct 20
Scaling Graph Reasoning in Language Models with Learnable Graph Positional Encodings
Javier Porras-Valenzuela, Arush Arora, Alejandro Ribeiro
Solving reasoning tasks supported on graphs typically requires providing a Language Reasoning Model (LRM) with a serialized description of the graph. This approach results in the input length growing with the number of edges, which limits the size of the problems that can be solved. Furthermore, even when graphs fit in context, performance sharply decays as the graphs grow. We propose to address this by augmenting LRMs with scalable and permutation equivariant graph positional encodings that modulate attention logits. An additional post-fusion-style module incorporates structural information into hidden states. We compute these positional encodings using a GNN followed by a Graph Transformer. This eliminates the need for serialized adjacencies, reducing the context burden to $O(N)$. We jointly fine-tune the positional encoder and the LRM LoRA adapters using optimal solutions generated by graph-theoretic solvers, resulting in a computationally efficient method. Trained on small graphs, the model exhibits improved size-generalization across graph-theoretic tasks on graphs with up to 1,000 nodes, and produces fewer tokens than a conventional LRM. Additionally, we observe that our…
PDF arXiv
arXiv cs.LGOct 19
Sample-efficient learning with Dense Associative Memories
Andrew Lucas, Dmitry Krotov
We analytically study the evolution of the energy function of Dense Associative Memories (DenseAMs) as they train on randomly sampled codewords of random expander low-density parity-check codes (a.k.a. solutions of random XORSAT problems). We identify two distinct regimes and a transition between them, reminiscent of grokking in neural networks. A regime of small training set size is where the model memorizes a few training examples but cannot discover the underlying rules of the entire codeword dictionary generative process. The opposite regime, corresponding to a larger training set size, is where the model has uncovered the sparse constraints that generated the entire data set after seeing only polynomially many patterns. We also design a novel class of DenseAMs with a thresholding learning rule, and show that any amount of thresholding enables the model to perform simplified sparse Hamiltonian learning, which exponentially reduces the number of training examples needed to learn the true underlying data set. In this regime, the model learns how to discover $\exp[\mathrm{O}(n)]$ solutions from only $\mathrm{O}(\log n)$ training samples, which is asymptotically optimal for any…
PDF arXiv
arXiv nlin.CDOct 19
Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability
Alexander Itin
Gradient descent with momentum can be viewed as a particle moving with friction in the loss landscape. In deep learning the learning rate is usually large, so training is a discrete dynamical system: in many full-batch experiments the sharpness grows to the optimizer's stability limit, the edge of stability (EoS), and training continues there. Physics-inspired optimizers change this motion; Velocity-Regularized Adam (VRAdam), for example, shrinks the step at high speed. We study kinetic maps, in which the step is the gradient of a kinetic energy of the momentum, and ask what they do beyond the EoS. (i) In a reduced model, a kinetic map creates a symmetric period-two orbit beyond the edge, strictly linearly stable if and only if the map's elasticity (the log-log slope of step length versus momentum) lies between 0 and 1; on it, training sits at an effective edge of stability. VRAdam's damper fails above twice the critical learning rate, where its kinetic energy is not convex; the Legendre dual of VRAdam's own Lagrangian is convex and passes at every learning rate. (ii) Because the sharpness rises gradually, the loss of stability is delayed, as in Neishtadt's theory of slowly swept…
PDF arXiv

Activity · 12 weeks

Open problems in Physics informed neural

Accurate long-term PDE dynamics prediction1 paper raises it · 1 addresses it
Reconstruction-based learning often fails to capture the evolution of physical fields over time, limiting predictive accuracy for parametric PDEs.
- PDE-JEPA: Predictive Representation Learning of Latent Dynamics Modeling for Parametric PDEsraises · addresses
Numerical instability in softmax and cross-entropy1 paper raises it · 1 addresses it
Standard implementations suffer from gradient collapse and non-zero gradients when logits match, harming training stability and convergence.
- A Fix for Softmax and Cross-Entropyraises · addresses
Token collapse in graph soft prompting1 paper raises it · 0 address it
Graph-specific tokens in LLM prompts often become indistinguishable, failing to encode meaningful graph information.
- On the Token Collapse Phenomenon in Textual Graph Soft Promptingraises
Efficient sparse GNN training with dynamic topology1 paper raises it · 1 addresses it
Reusing a fixed sparse graph locks training to a static topology, while dynamic sampling is computationally expensive.
- EDiS: Edge Disjoint Subgraph Sparsification Framework for Graph Neural Networksraises · addresses
Automated diagnosis and improvement of numerical solvers1 paper raises it · 1 addresses it
Execution feedback reveals poor performance but rarely explains the cause or how to fix it, hindering solver optimization.
- Training Numerical Intelligence via Auto-Diagnosis and Skill Discoveryraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 2m ago

Physics informed neural

Found by the topic model: physics informed neural, graph neural, relu networks.

118papers this week−28.0% vs last week

All papers · 993 With code · 3

Newest Most relevant

832 papers this month