Papers

Field

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
Agno+17% downloads · 519.1k/wk+16.8%
BentoML+30% downloads · 41k/wk+29.7%
n8n+25% downloads · 148.3k/wk+25.0%
LangGraph7 releases this week

Top tools here

Open problems in Frontier LLMs

Structural decision-making in LLMs1 paper raises it · 3 address it
Current LLMs lack native support for decision problems with structural requirements like null-awareness, permutation invariance, and scalability.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Adaptive computation for decision models1 paper raises it · 1 addresses it
Fixed-depth decision models waste computation on easy questions and cannot scale depth dynamically.
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
Reliable no-thinking behavior in LLMs1 paper raises it · 1 addresses it
LLMs often ignore or misinterpret no-thinking instructions, leading to unwanted reasoning traces.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses
Memory consolidation in context-augmented LLMs1 paper raises it · 1 addresses it
Gains from context are lost after its removal, and models struggle to retain learned knowledge without forgetting.
- Cortex: Consolidating Context into Parametric Memoryraises · addresses
Sustaining self-evolution in reasoning models1 paper raises it · 1 addresses it
Self-generated training data degrades over rounds due to invalid or repetitive questions, causing performance collapse.
- Questioning the Questions: Sustaining Self-Evolution in Reasoning Modelsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

InfinigenceSeries B+ · Oct 12$343.6M
Lean SuperIntelligencePre-seed · Oct 9undisclosed
VinciSeries B+ · Oct 6$250M
Siena AISeries A · Oct 6$17M
Ocular AIPre-seed · Oct 5$2M

Recently funded

Topics in this field

reasoning traces · reasoning models · latent reasoning372 papers68 this week
attention heads · large language models · language models218 papers49 this week
dense retrieval · retrieval augmented generation · retrieval246 papers47 this week
large language models · attribution · language models llms196 papers37 this week
large language models · large language · language models178 papers35 this week
typed decision models · typed decision model · typed decisions129 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CLOct 19
How Benchmark Graph Construction Shapes Multi-Hop KG Navigation: A Controlled Audit of PathQuestion and PathQuestion-Large
Eduin E. Hernandez, Sergio A. Diaz, Luis F. Garcia, Nurassyl Askar, Stefano Rini
Multi-hop knowledge-graph question answering (KGQA) benchmarks are often treated as fixed language-and-reasoning tasks, but for navigation-based methods, difficulty also depends on the graph in which those tasks are instantiated. This makes graph construction a consequential but often entangled part of the navigation task. To isolate this effect, we audit PathQuestion (PQ) and PathQuestion-Large (PQL) as graph-navigation benchmarks under a controlled intervention: we hold questions, topic entities, released answers, released path annotations, and family-level splits fixed while replacing each released task-filtered graph with the larger source KG distributed with the benchmark. Source restoration expands the effective navigation search space and graph-consistent answer and path multiplicity while sharply reducing unguided reachability to released answers. We evaluate whether these structural changes persist across navigation policies using structural references, frozen small-language-model navigation policies, and trained MINERVA policies. Across both frozen and trained policies, restoration substantially reduces released-answer accuracy and trajectory fidelity, while…
PDF arXiv
arXiv cs.AIOct 19
Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization
Yupeng Chang, Linyuan Gao, Zhiyuan Li, Yuan Wu, Yi Chang
An inference budget is useful only when the system delivers an answer within it. We study a fixed-budget policy combining first-valid-box stopping with a charged finalization cue. On 100 fresh, independently audited GSM8K test questions per model, it improves delivered endpoint accuracy over first-box stopping by 8.0 percentage points for Qwen3-4B-Thinking-2507 and 7.0 for DeepSeek-R1-Distill-Llama-8B. A prelocked repeat on the same questions gives 7.0 and 10.0 points; neither seed shows a gross harm relative to first-box stopping. The point estimates of incremental physical-slot savings are below one percent. A finalization-only ablation separates cue recovery from stopping, and a fixed scalar reader tests extraction as an alternative explanation. The cue is prior art; the contribution is a component and readout analysis with charged injections and verified paths. Results cover two checkpoints, one task, one prompt family, and two fixed seed conditions.
PDF arXiv
arXiv cs.AIOct 19
BOSS: Token-Budgeted Online Sampling and Selection for LLM Reasoning
Yupeng Chang, Yahan Li, Chenlu Guo, Diana Inkpen, Yuan Wu +1
Test-time scaling can improve LLM reasoning, but existing approaches typically adapt generation or apply answer selection after a predetermined generation schedule. We study within-instance allocation: under a hard per-instance prompt-plus-completion token cap, when should inference stop, generate another candidate, or spend residual tokens adjudicating candidates already observed? BOSS is a training-free policy for this three-way decision, with worst-case budget checks before every generation and judge call. Its key contribution is online generation-adjudication allocation under one shared token ledger. Across two Llama-3.1 models and four reasoning benchmarks, BOSS attains the highest accuracy in 22 of 24 model-task-budget cells under the common hard cap. At Llama-3.1-8B and B=4k, it reaches 60.5 macro-average accuracy versus 58.6 for Adaptive-Consistency, 59.1 for TALE-EP, and 58.9 for fixed-schedule League. On identical Stage-I candidate pools, randomized A/B judging with repeated voting improves plurality from 58.4 to 60.3, with the full BOSS selector reaching 60.5. Approximately matched-cost controls further characterize how the benefit of adjudication changes with…
PDF arXiv
arXiv stat.MLOct 19
Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons
Motti Goldberger, Nils Rudi
This paper considers a decision-maker tasked with identifying the $k$ best items from a large pool of candidates, such as job applicants, LLMs, award nominees, or new product ideas. The decision-maker learns about the items' quality by soliciting human evaluations. Each evaluation can be costly, so the decision-maker aims to use the limited evaluation budget efficiently by allocating evaluations adaptively. A key consideration is the form of evaluation to solicit, which should be something evaluators are comfortable providing. Two popular forms are ratings and pairwise comparisons, and we allow for both, as they may be especially useful in combination given the limited budget. Ratings can be used early on to get a coarse picture of which items are good and which are not. As that picture forms, comparisons can refine the ordering of the good items, narrowing in on the ultimate top-$k$ selection. To solve the allocation problem, we develop an adaptive algorithm built from two allocation policies. The first, our primary contribution, is based on a surrogate objective that is naturally forward-looking, valuing evaluations that help inform what to evaluate in future rounds, not just…
PDF arXiv
arXiv cs.CLOct 19
Large-Scale Biomedical RAG: Corpus Scaling, Document Representation, and Efficiency Trade-offs
Adrián Sánchez-Freire, Gonzalo Pérez-Gómez, Aitziber Atutxa, Ander Barrena
We present a controlled, large-scale evaluation of how corpus scale, document representation, and reranking affect the effectiveness and efficiency of biomedical Retrieval-Augmented Generation (RAG) systems. Retrieval collections were progressively expanded while retaining the relevant source documents, enabling consistent comparisons across corpus sizes. We compared abstract, flat full-text, and hierarchical full-text representations across first-stage retrieval, reranking, and end-to-end multiple-choice question answering. Configurations selected on development data were evaluated on five external biomedical benchmarks and a held-out test set, while retrieval latency and index storage were measured to quantify operational cost. Performance patterns observed with small or controlled collections did not consistently persist as collections grew, and the usefulness of reranking depended on the corpus and retrieval setting. RAG improved three of four model configurations across the external benchmarks, with mean gains of up to 9.6 percentage points, and improved all four configurations on the held-out test set by 10.2-30.3 points. Abstract retrieval performed best when reference…
PDF arXiv
arXiv cs.LGOct 19
Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors
Abdul Rahman Iqbal, Muhammad Fawad Nasim, Amir Saleem
Typed decision models (TDMs) return a probability over caller-defined options, and Jev is described by its vendor as calibrated. But in deployment class frequencies change, so calibration measured on a fixed test set need not hold where thresholds are applied. Existing evaluations leave two gaps: the gains of vendor decision training have not been separated from those of the backbone under an identical readout, and TDMs have been tested only at the test set's class distribution. We ask whether decision training improves a model over its own backbone, and whether a TDM's calibration and thresholds withstand prevalence (label) shift as well as a fine-tuned classifier's. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and two trained classifiers on identical decisions from five public datasets under simulated shift, with 35 paired tests fixed before any test was computed. First, decision training is not uniformly beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12.0 Macro-F1 points worse on financial sentiment. Second, under moderate shift Jev's expected calibration error (ECE) and threshold risk…
PDF arXiv

Papers

Top tools here

Coming with more sources: stars history

All tools

NVIDIA Dynamo+60% downloads · 15.8k/wk+60.0%
Agno+17% downloads · 519.1k/wk+16.8%
BentoML+30% downloads · 41k/wk+29.7%
n8n+25% downloads · 148.3k/wk+25.0%
LangGraph7 releases this week

Recently funded

Market

InfinigenceSeries B+ · Oct 12$343.6M
Lean SuperIntelligencePre-seed · Oct 9undisclosed
VinciSeries B+ · Oct 6$250M
Siena AISeries A · Oct 6$17M
Ocular AIPre-seed · Oct 5$2M

Topics in this field

reasoning traces · reasoning models · latent reasoning372 papers68 this week
attention heads · large language models · language models218 papers49 this week
dense retrieval · retrieval augmented generation · retrieval246 papers47 this week
large language models · attribution · language models llms196 papers37 this week
large language models · large language · language models178 papers35 this week
typed decision models · typed decision model · typed decisions129 papers32 this week

Active contributors · 90 days

Activity · 12 weeks

Coming with more sources: repos

Open problems in Frontier LLMs

Structural decision-making in LLMs1 paper raises it · 3 address it
Current LLMs lack native support for decision problems with structural requirements like null-awareness, permutation invariance, and scalability.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Adaptive computation for decision models1 paper raises it · 1 addresses it
Fixed-depth decision models waste computation on easy questions and cannot scale depth dynamically.
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
Reliable no-thinking behavior in LLMs1 paper raises it · 1 addresses it
LLMs often ignore or misinterpret no-thinking instructions, leading to unwanted reasoning traces.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses
Memory consolidation in context-augmented LLMs1 paper raises it · 1 addresses it
Gains from context are lost after its removal, and models struggle to retain learned knowledge without forgetting.
- Cortex: Consolidating Context into Parametric Memoryraises · addresses
Sustaining self-evolution in reasoning models1 paper raises it · 1 addresses it
Self-generated training data degrades over rounds due to invalid or repetitive questions, causing performance collapse.
- Questioning the Questions: Sustaining Self-Evolution in Reasoning Modelsraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 51m ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

430papers this week−7.5% vs last week

All papers · 2.2k With code · 51

Newest Most relevant

2.1k papers this month

Recent papers

All in Research →

arXiv cs.CLOct 19
How Benchmark Graph Construction Shapes Multi-Hop KG Navigation: A Controlled Audit of PathQuestion and PathQuestion-Large
Eduin E. Hernandez, Sergio A. Diaz, Luis F. Garcia, Nurassyl Askar, Stefano Rini
Multi-hop knowledge-graph question answering (KGQA) benchmarks are often treated as fixed language-and-reasoning tasks, but for navigation-based methods, difficulty also depends on the graph in which those tasks are instantiated. This makes graph construction a consequential but often entangled part of the navigation task. To isolate this effect, we audit PathQuestion (PQ) and PathQuestion-Large (PQL) as graph-navigation benchmarks under a controlled intervention: we hold questions, topic entities, released answers, released path annotations, and family-level splits fixed while replacing each released task-filtered graph with the larger source KG distributed with the benchmark. Source restoration expands the effective navigation search space and graph-consistent answer and path multiplicity while sharply reducing unguided reachability to released answers. We evaluate whether these structural changes persist across navigation policies using structural references, frozen small-language-model navigation policies, and trained MINERVA policies. Across both frozen and trained policies, restoration substantially reduces released-answer accuracy and trajectory fidelity, while…
PDF arXiv
arXiv cs.AIOct 19
Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization
Yupeng Chang, Linyuan Gao, Zhiyuan Li, Yuan Wu, Yi Chang
An inference budget is useful only when the system delivers an answer within it. We study a fixed-budget policy combining first-valid-box stopping with a charged finalization cue. On 100 fresh, independently audited GSM8K test questions per model, it improves delivered endpoint accuracy over first-box stopping by 8.0 percentage points for Qwen3-4B-Thinking-2507 and 7.0 for DeepSeek-R1-Distill-Llama-8B. A prelocked repeat on the same questions gives 7.0 and 10.0 points; neither seed shows a gross harm relative to first-box stopping. The point estimates of incremental physical-slot savings are below one percent. A finalization-only ablation separates cue recovery from stopping, and a fixed scalar reader tests extraction as an alternative explanation. The cue is prior art; the contribution is a component and readout analysis with charged injections and verified paths. Results cover two checkpoints, one task, one prompt family, and two fixed seed conditions.
PDF arXiv
arXiv cs.AIOct 19
BOSS: Token-Budgeted Online Sampling and Selection for LLM Reasoning
Yupeng Chang, Yahan Li, Chenlu Guo, Diana Inkpen, Yuan Wu +1
Test-time scaling can improve LLM reasoning, but existing approaches typically adapt generation or apply answer selection after a predetermined generation schedule. We study within-instance allocation: under a hard per-instance prompt-plus-completion token cap, when should inference stop, generate another candidate, or spend residual tokens adjudicating candidates already observed? BOSS is a training-free policy for this three-way decision, with worst-case budget checks before every generation and judge call. Its key contribution is online generation-adjudication allocation under one shared token ledger. Across two Llama-3.1 models and four reasoning benchmarks, BOSS attains the highest accuracy in 22 of 24 model-task-budget cells under the common hard cap. At Llama-3.1-8B and B=4k, it reaches 60.5 macro-average accuracy versus 58.6 for Adaptive-Consistency, 59.1 for TALE-EP, and 58.9 for fixed-schedule League. On identical Stage-I candidate pools, randomized A/B judging with repeated voting improves plurality from 58.4 to 60.3, with the full BOSS selector reaching 60.5. Approximately matched-cost controls further characterize how the benefit of adjudication changes with…
PDF arXiv
arXiv stat.MLOct 19
Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons
Motti Goldberger, Nils Rudi
This paper considers a decision-maker tasked with identifying the $k$ best items from a large pool of candidates, such as job applicants, LLMs, award nominees, or new product ideas. The decision-maker learns about the items' quality by soliciting human evaluations. Each evaluation can be costly, so the decision-maker aims to use the limited evaluation budget efficiently by allocating evaluations adaptively. A key consideration is the form of evaluation to solicit, which should be something evaluators are comfortable providing. Two popular forms are ratings and pairwise comparisons, and we allow for both, as they may be especially useful in combination given the limited budget. Ratings can be used early on to get a coarse picture of which items are good and which are not. As that picture forms, comparisons can refine the ordering of the good items, narrowing in on the ultimate top-$k$ selection. To solve the allocation problem, we develop an adaptive algorithm built from two allocation policies. The first, our primary contribution, is based on a surrogate objective that is naturally forward-looking, valuing evaluations that help inform what to evaluate in future rounds, not just…
PDF arXiv
arXiv cs.CLOct 19
Large-Scale Biomedical RAG: Corpus Scaling, Document Representation, and Efficiency Trade-offs
Adrián Sánchez-Freire, Gonzalo Pérez-Gómez, Aitziber Atutxa, Ander Barrena
We present a controlled, large-scale evaluation of how corpus scale, document representation, and reranking affect the effectiveness and efficiency of biomedical Retrieval-Augmented Generation (RAG) systems. Retrieval collections were progressively expanded while retaining the relevant source documents, enabling consistent comparisons across corpus sizes. We compared abstract, flat full-text, and hierarchical full-text representations across first-stage retrieval, reranking, and end-to-end multiple-choice question answering. Configurations selected on development data were evaluated on five external biomedical benchmarks and a held-out test set, while retrieval latency and index storage were measured to quantify operational cost. Performance patterns observed with small or controlled collections did not consistently persist as collections grew, and the usefulness of reranking depended on the corpus and retrieval setting. RAG improved three of four model configurations across the external benchmarks, with mean gains of up to 9.6 percentage points, and improved all four configurations on the held-out test set by 10.2-30.3 points. Abstract retrieval performed best when reference…
PDF arXiv
arXiv cs.LGOct 19
Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors
Abdul Rahman Iqbal, Muhammad Fawad Nasim, Amir Saleem
Typed decision models (TDMs) return a probability over caller-defined options, and Jev is described by its vendor as calibrated. But in deployment class frequencies change, so calibration measured on a fixed test set need not hold where thresholds are applied. Existing evaluations leave two gaps: the gains of vendor decision training have not been separated from those of the backbone under an identical readout, and TDMs have been tested only at the test set's class distribution. We ask whether decision training improves a model over its own backbone, and whether a TDM's calibration and thresholds withstand prevalence (label) shift as well as a fine-tuned classifier's. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and two trained classifiers on identical decisions from five public datasets under simulated shift, with 35 paired tests fixed before any test was computed. First, decision training is not uniformly beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12.0 Macro-F1 points worse on financial sentiment. Second, under moderate shift Jev's expected calibration error (ECE) and threshold risk…
PDF arXiv