Papers

Field

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
LMDeploy+109% downloads · 10.8k/wk+109.0%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
Vercel AI SDK300 releases this week

Top tools here

Open problems in Frontier LLMs

Evaluating decision models beyond accuracy1 paper raises it · 4 address it
Aggregate accuracy hides failures on controlled variations, calibration, and confidence reliability, which are critical for safe deployment in LLM systems.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- Jeva: System One Models from First Principlesaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient, structured decision-making for agents3 papers raise it · 3 address it
Generative reasoning is slow and unreliable for short, repeated decisions, while fixed-depth decision models lack adaptability.
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsraises · addresses
Long-horizon context retention and reuse2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and textual summaries alone may not preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web search and evidence integration1 paper raises it · 2 address it
Agents struggle to autonomously explore, access, and integrate heterogeneous, interactive web sources to answer complex factual queries.
- Deep Search in Real-World Web Environmentsraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Cost-effective fact-checking of review comments1 paper raises it · 1 addresses it
Frontier LLMs can verify review comments but require reading entire papers for each check, making the process expensive and slow.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Chaojinhua IntelligenceSeed · Oct 28undisclosed
Chaoqiumian TechnologySeed · Oct 26undisclosed
HypersphereSeed · Oct 26undisclosed
Star TherapeuticsSeries B+ · Oct 23$195M
OpenAIStage not given · Oct 22$30B

Recently funded

Topics in this field

retrieval augmented generation · knowledge graphs · retrieval augmented280 papers42 this week
language models llms · large language models · language models301 papers38 this week
memory construction · persistent memory · long term memory285 papers32 this week
large language models · machine translation · multilingual236 papers26 this week
generative recommendation · sequential recommendation · personalized171 papers26 this week
deep research · scientific literature · citations155 papers22 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGOct 1
Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability
Chuqin Geng, Li Zhang, Haolin Ye, Mark Zhang, Luke Zhang +1
Mechanistic interpretability aims to recover the internal computations responsible for model behavior. Progress in automated circuit discovery is often framed as a search problem: better attribution or optimization should identify better mechanisms. This assumes that the evaluation objective can recognize a better circuit once it is found. Yet standard intervention-based evaluation replaces signals outside a circuit to isolate its effect, changing the computational environment in which the retained components operate. We call this change context distortion. This raises the question of whether faithfulness rewards recovery of model behavior or compatibility with the altered context. We show that intervention-based faithfulness can prefer an equally sized circuit that reproduces the model's behavior less well. We call this reversal misranking, an objective-level recovery gap. To isolate evaluation from discovery, we construct equally sized candidates through controlled edits to human-identified circuits in pretrained language models, including IOI and Greater-Than, and benchmark circuits with known mechanisms. We find misranking even without a discovery algorithm. Misranking also…
PDF arXiv
arXiv cs.CLOct 1▲ 15
LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them
Yinheng Li, Justin Wagle
Jev-style decision models have gained surging traction by making decision-making dramatically faster and more efficient, returning categorical probability distributions over predefined options without generating free-form text. However, because Jev's architecture and training methodologies remain proprietary, how to replicate these capabilities effectively remains an open question. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM-as-Jev, an architecture-preserving framework offering both a training-free inference recipe and a fine-tuning objective with KL-divergence anchors. Across Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: off the shelf, frozen models match or exceed existing training-free readouts on the same 4B backbone, scale to arbitrary option counts, and natively handle multimodal decisions. They also seamlessly inherit the backbone's thinking mode: while Jev was introduced as a System 1 model, "Thinking Jev" leverages budgeted test-time reasoning to boost accuracy on challenging tasks without…
PDF arXiv
arXiv cs.AIOct 1
Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval
Arman Behnam, Binghui Wang
Memory-augmented large language models must decide which memories to retain, and recent systems do so by estimating each memory's effect on task performance. However, these estimates rely entirely on retrieved memories. When a memory is never retrieved, store-level interventions produce identical outcomes, leaving its utility unidentified. This is a retrieval-level positivity violation, invisible to diagnostics that examine only memory operations. We introduce Causal Memory Policy (CMP), a causal framework that restores identification by intervening on retrieval itself, reserving a fixed number of context slots for memories sampled with known propensities. CMP estimates memory utility by self-normalized inverse propensity weighting under a balanced assignment design. We prove the causal factorization of memory utility through retrieval, the unbiasedness and exact variance of the estimator, and the optimal decision rule under irreversible operations. Empirically, identification fails for 54% of required memories on LongMemEval and 67% on LoCoMo, and the failure persists in a deployed memory system. CMP improves discrimination between required and non-required memories from 0.54 to…
PDF arXiv
arXiv cs.CLOct 1
HakemBench: A Turkish Benchmark of Typed Decisions
Sait Furkan Teke
HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0, with 2,346 items and 4,275 choice, yes/no and score questions in seven tracks (fact-check triage, education, guardrails, legal routing, moderation, spam and phishing, and customer support). One harness scores decision quality (macro F1), calibration (from the normalised Brier score) and selective automation (from the normalised area under the generalised risk-coverage curve), combines them by a geometric mean and reports intervals from 2,000 bootstrap draws; probes for option order, paraphrase, English translation and substituted names are reported alongside. Most gold labels come from blind passes of one AI model family compared with the votes of a panel of large language models from other model families; they are not human-verified. On a board of 16 rows the leader scores a composite of 0.888 and the lab's own model is 7th at 0.660. Its numbers are not blind. Earlier runs' test results shaped its training data, so its guardrail, moderation and…
PDF arXiv
arXiv cs.CLOct 1
Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages
Nadine El-Naggar, Tatsuki Kuribayashi, Ted Briscoe
Some properties of languages, e.g., subject-object-verb (SOV) word order, are more prevalent than others among the thousands of attested natural languages (NLs). Such typological commonality is often attributed to learning biases. Computational simulations, recently with language models (LMs), have facilitated the exploration of this theory. In this paper, we extend existing analyses of the relationship between LMs' learning biases and typological commonality on both data and model sides, focusing on: (i) cross-serial dependencies, the upper limit of attested syntactic complexity, and (ii) stack-based LMs (SLMs), potentially facilitating learning of hierarchical patterns. We first evaluate generalization of SLMs on cross-serial dependencies across diverse artificial languages and confirm that they struggle with such constructions. However, SLMs with limited working memory generalize better suggesting a possible basis for such inductive bias and thus the typological commonality of some word order configurations.
PDF arXiv
arXiv cs.CLOct 1
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue +1
AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that seemingly small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty…
PDF arXiv

Papers

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
LMDeploy+109% downloads · 10.8k/wk+109.0%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
Vercel AI SDK300 releases this week

Recently funded

Market

Chaojinhua IntelligenceSeed · Oct 28undisclosed
Chaoqiumian TechnologySeed · Oct 26undisclosed
HypersphereSeed · Oct 26undisclosed
Star TherapeuticsSeries B+ · Oct 23$195M
OpenAIStage not given · Oct 22$30B

Topics in this field

retrieval augmented generation · knowledge graphs · retrieval augmented280 papers42 this week
language models llms · large language models · language models301 papers38 this week
memory construction · persistent memory · long term memory285 papers32 this week
large language models · machine translation · multilingual236 papers26 this week
generative recommendation · sequential recommendation · personalized171 papers26 this week
deep research · scientific literature · citations155 papers22 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.LGOct 1
Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability
Chuqin Geng, Li Zhang, Haolin Ye, Mark Zhang, Luke Zhang +1
Mechanistic interpretability aims to recover the internal computations responsible for model behavior. Progress in automated circuit discovery is often framed as a search problem: better attribution or optimization should identify better mechanisms. This assumes that the evaluation objective can recognize a better circuit once it is found. Yet standard intervention-based evaluation replaces signals outside a circuit to isolate its effect, changing the computational environment in which the retained components operate. We call this change context distortion. This raises the question of whether faithfulness rewards recovery of model behavior or compatibility with the altered context. We show that intervention-based faithfulness can prefer an equally sized circuit that reproduces the model's behavior less well. We call this reversal misranking, an objective-level recovery gap. To isolate evaluation from discovery, we construct equally sized candidates through controlled edits to human-identified circuits in pretrained language models, including IOI and Greater-Than, and benchmark circuits with known mechanisms. We find misranking even without a discovery algorithm. Misranking also…
PDF arXiv
arXiv cs.CLOct 1▲ 15
LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them
Yinheng Li, Justin Wagle
Jev-style decision models have gained surging traction by making decision-making dramatically faster and more efficient, returning categorical probability distributions over predefined options without generating free-form text. However, because Jev's architecture and training methodologies remain proprietary, how to replicate these capabilities effectively remains an open question. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM-as-Jev, an architecture-preserving framework offering both a training-free inference recipe and a fine-tuning objective with KL-divergence anchors. Across Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: off the shelf, frozen models match or exceed existing training-free readouts on the same 4B backbone, scale to arbitrary option counts, and natively handle multimodal decisions. They also seamlessly inherit the backbone's thinking mode: while Jev was introduced as a System 1 model, "Thinking Jev" leverages budgeted test-time reasoning to boost accuracy on challenging tasks without…
PDF arXiv
arXiv cs.AIOct 1
Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval
Arman Behnam, Binghui Wang
Memory-augmented large language models must decide which memories to retain, and recent systems do so by estimating each memory's effect on task performance. However, these estimates rely entirely on retrieved memories. When a memory is never retrieved, store-level interventions produce identical outcomes, leaving its utility unidentified. This is a retrieval-level positivity violation, invisible to diagnostics that examine only memory operations. We introduce Causal Memory Policy (CMP), a causal framework that restores identification by intervening on retrieval itself, reserving a fixed number of context slots for memories sampled with known propensities. CMP estimates memory utility by self-normalized inverse propensity weighting under a balanced assignment design. We prove the causal factorization of memory utility through retrieval, the unbiasedness and exact variance of the estimator, and the optimal decision rule under irreversible operations. Empirically, identification fails for 54% of required memories on LongMemEval and 67% on LoCoMo, and the failure persists in a deployed memory system. CMP improves discrimination between required and non-required memories from 0.54 to…
PDF arXiv
arXiv cs.CLOct 1
HakemBench: A Turkish Benchmark of Typed Decisions
Sait Furkan Teke
HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0, with 2,346 items and 4,275 choice, yes/no and score questions in seven tracks (fact-check triage, education, guardrails, legal routing, moderation, spam and phishing, and customer support). One harness scores decision quality (macro F1), calibration (from the normalised Brier score) and selective automation (from the normalised area under the generalised risk-coverage curve), combines them by a geometric mean and reports intervals from 2,000 bootstrap draws; probes for option order, paraphrase, English translation and substituted names are reported alongside. Most gold labels come from blind passes of one AI model family compared with the votes of a panel of large language models from other model families; they are not human-verified. On a board of 16 rows the leader scores a composite of 0.888 and the lab's own model is 7th at 0.660. Its numbers are not blind. Earlier runs' test results shaped its training data, so its guardrail, moderation and…
PDF arXiv
arXiv cs.CLOct 1
Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages
Nadine El-Naggar, Tatsuki Kuribayashi, Ted Briscoe
Some properties of languages, e.g., subject-object-verb (SOV) word order, are more prevalent than others among the thousands of attested natural languages (NLs). Such typological commonality is often attributed to learning biases. Computational simulations, recently with language models (LMs), have facilitated the exploration of this theory. In this paper, we extend existing analyses of the relationship between LMs' learning biases and typological commonality on both data and model sides, focusing on: (i) cross-serial dependencies, the upper limit of attested syntactic complexity, and (ii) stack-based LMs (SLMs), potentially facilitating learning of hierarchical patterns. We first evaluate generalization of SLMs on cross-serial dependencies across diverse artificial languages and confirm that they struggle with such constructions. However, SLMs with limited working memory generalize better suggesting a possible basis for such inductive bias and thus the typological commonality of some word order configurations.
PDF arXiv
arXiv cs.CLOct 1
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue +1
AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that seemingly small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 1h ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

322papers this week−34.8% vs last week

All papers · 2.6k With code · 74

Newest Most relevant

1.9k papers this month

Open problems in Frontier LLMs

Evaluating decision models beyond accuracy1 paper raises it · 4 address it
Aggregate accuracy hides failures on controlled variations, calibration, and confidence reliability, which are critical for safe deployment in LLM systems.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- Jeva: System One Models from First Principlesaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient, structured decision-making for agents3 papers raise it · 3 address it
Generative reasoning is slow and unreliable for short, repeated decisions, while fixed-depth decision models lack adaptability.
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsraises · addresses
Long-horizon context retention and reuse2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and textual summaries alone may not preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web search and evidence integration1 paper raises it · 2 address it
Agents struggle to autonomously explore, access, and integrate heterogeneous, interactive web sources to answer complex factual queries.
- Deep Search in Real-World Web Environmentsraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Cost-effective fact-checking of review comments1 paper raises it · 1 addresses it
Frontier LLMs can verify review comments but require reading entire papers for each check, making the process expensive and slow.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses

Named by an AI from these papers' abstracts.