Papers

Field

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
TanStack AI+68% downloads · 1M/wk+67.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%

Top tools here

Open problems in Frontier LLMs

Evaluating and benchmarking decision models for reliability and calibration3 papers raise it · 4 address it
Decision models are increasingly used in LLM systems but lack standardized ways to measure correctness, confidence, and robustness across controlled variations.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient and accurate fact-checking of AI and human reviews1 paper raises it · 1 addresses it
Peer review systems struggle with factual errors in comments, and current methods are either too slow or inaccurate for large-scale use.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Long-horizon agent context management and memory retention2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and current compaction methods fail to preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web-based deep search and evidence integration1 paper raises it · 1 addresses it
Agents cannot reliably perform multi-step research in live, open web environments with diverse, interactive, and heterogeneous information sources.
- Deep Search in Real-World Web Environmentsraises · addresses
User behavior modeling for long-term personalization1 paper raises it · 1 addresses it
LLMs lack the ability to infer and adapt to evolving user preferences and intent from real-world behavior data over time.
- Vela: User Behavior as Supervision Sparks LLM Intelligenceraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Chaoqiumian TechnologySeed · Oct 26undisclosed
HypersphereSeed · Oct 26undisclosed
Star TherapeuticsSeries B+ · Oct 23$195M
OpenAIStage not given · Oct 22$30B
DeepSeekStage not given · Oct 22$15B

Recently funded

Topics in this field

language models llms · large language models · language models278 papers40 this week
retrieval augmented generation · knowledge graphs · retrieval augmented251 papers30 this week
memory construction · persistent memory · long term memory264 papers28 this week
large language models · machine translation · multilingual222 papers24 this week
large language models · language models · brain alignment148 papers17 this week
deep research · scientific literature · citations139 papers16 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CLOct 5
Word-Level Text Unmixing via Evidence-Preserving Ownership Routing with Language Models
Jinglin He, Siyang Jiang, Lixing He, Guoliang Xing, Hongkai Chen
Text from multiple sources can become interleaved into a single sequence when attribution metadata is lost, such as overlapping speech transcripts, document reading flows, or concurrent agent streams. We formalize this challenge as Word-Level Text Unmixing: given an interleaved lexical stream and source count K, recover the original source sequences while preserving every word occurrence and its within-source order exactly. Directly generating separated texts with LLMs can omit, duplicate, or hallucinate words, violating this exact-reconstruction objective. We therefore propose Evidence-Preserving Ownership Routing (EPOR), which decouples source-ownership prediction from reconstruction. EPOR adapts a causal LLM to predict canonical ownership routes conditioned on the mixed stream and prior routing decisions. At inference, completion-safe constrained decoding is combined with deterministic indexed reconstruction, yielding structurally valid K-source partitions that preserve every observed occurrence exactly once. We also introduce UNMIXBENCH, covering controlled synthetic mixtures, timestamp-derived speech from AMI and ICSI, layout-derived document streams from ReadingBank, and…
PDF arXiv
arXiv cs.LGOct 5
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Hashmath Shaik, Gnaneswar Villuri, Alex Doboli
The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.
PDF arXiv
arXiv cs.CLOct 5
Molecules of a Story: Community Detection in PMI-weighted Narrative Networks
Kasper Fyhn, Rebekah Baglini
Automatically extracted narrative networks -- graphs with entities as nodes and their relations as edges -- have proven useful for revealing central narrative structures through salient entities and their connections (Tangherlini et al. 2020; Labatut and Bost 2019). But a narrative is more than those central structures that everything else revolves around. This work is concerned with the everything else: brief sub-plots, small clusters of descriptions, or associations between minor characters that go under the radar at the macro-level. We present an approach to unearth such peripheral structures. They involve rare entities with limited textual presence, overshadowed by dominant entities and lost among each other in the long tail of many but rare entities (Baayen 2001). We leverage the known tendency of pointwise mutual information (PMI, Church and Hanks 1990) to inflate for rare events, turning its weakness into a strength by weighting edges with PMI to foreground peripheral entity configurations. Communities extracted from the resulting network are structural traces of underlying narrative elements. We demonstrate the approach on The Lord of the Rings. From measures of how…
PDF arXiv
arXiv cs.AIOct 5
The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)
Feilian Huang
How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation…
PDF arXiv
arXiv econ.GNOct 5
Synthetic Cultural Agents from Aggregate Anchors
Augusto Gonzalez-Bonorino, Kseniia Biriukova, Monica Capra
Population prompts are widely used to generate synthetic survey responses, but they combine information supplied at inference with associations already encoded during pretraining. We introduce an alternative construction that maps declared aggregate preference anchors into group-indexed choice policies. For each population, the signs of six Global Preferences Survey (GPS) coordinates deterministically label a shared bank of paired synthetic responses, and Direct Preference Optimization fits a parameter-efficient adapter to those comparisons. We evaluate the adapters on candidate World Values Survey (WVS) items using prompts that omit country names and distinguish four questions: recovery of the imposed labels, transfer of the anchor signal to new text, coherence between the GPS anchors and human WVS responses, and agreement between adapter and human scores. The adapters recover the imposed pairwise labels. On a purposively selected sixteen-country development panel, adapter trust scores completely separate the two GPS-sign groups and have a rank correlation of (0.74) with continuous GPS trust scores. Human-GPS and adapter-human associations remain unresolved on the same panel,…
PDF arXiv
arXiv cs.CLOct 5
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Yiqi Liu, Joseph James, Yang Wang, Kun Zhao, Chenghao Xiao +1
When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge--dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by $+0.079$. Applying the same configuration to six additional cells yields a $+0.0393$ mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank $p=0.0091$. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.
PDF arXiv

Papers

Recently funded

Market

Chaoqiumian TechnologySeed · Oct 26undisclosed
HypersphereSeed · Oct 26undisclosed
Star TherapeuticsSeries B+ · Oct 23$195M
OpenAIStage not given · Oct 22$30B
DeepSeekStage not given · Oct 22$15B

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
TanStack AI+68% downloads · 1M/wk+67.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%

Topics in this field

language models llms · large language models · language models278 papers40 this week
retrieval augmented generation · knowledge graphs · retrieval augmented251 papers30 this week
memory construction · persistent memory · long term memory264 papers28 this week
large language models · machine translation · multilingual222 papers24 this week
large language models · language models · brain alignment148 papers17 this week
deep research · scientific literature · citations139 papers16 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CLOct 5
Word-Level Text Unmixing via Evidence-Preserving Ownership Routing with Language Models
Jinglin He, Siyang Jiang, Lixing He, Guoliang Xing, Hongkai Chen
Text from multiple sources can become interleaved into a single sequence when attribution metadata is lost, such as overlapping speech transcripts, document reading flows, or concurrent agent streams. We formalize this challenge as Word-Level Text Unmixing: given an interleaved lexical stream and source count K, recover the original source sequences while preserving every word occurrence and its within-source order exactly. Directly generating separated texts with LLMs can omit, duplicate, or hallucinate words, violating this exact-reconstruction objective. We therefore propose Evidence-Preserving Ownership Routing (EPOR), which decouples source-ownership prediction from reconstruction. EPOR adapts a causal LLM to predict canonical ownership routes conditioned on the mixed stream and prior routing decisions. At inference, completion-safe constrained decoding is combined with deterministic indexed reconstruction, yielding structurally valid K-source partitions that preserve every observed occurrence exactly once. We also introduce UNMIXBENCH, covering controlled synthetic mixtures, timestamp-derived speech from AMI and ICSI, layout-derived document streams from ReadingBank, and…
PDF arXiv
arXiv cs.LGOct 5
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Hashmath Shaik, Gnaneswar Villuri, Alex Doboli
The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.
PDF arXiv
arXiv cs.CLOct 5
Molecules of a Story: Community Detection in PMI-weighted Narrative Networks
Kasper Fyhn, Rebekah Baglini
Automatically extracted narrative networks -- graphs with entities as nodes and their relations as edges -- have proven useful for revealing central narrative structures through salient entities and their connections (Tangherlini et al. 2020; Labatut and Bost 2019). But a narrative is more than those central structures that everything else revolves around. This work is concerned with the everything else: brief sub-plots, small clusters of descriptions, or associations between minor characters that go under the radar at the macro-level. We present an approach to unearth such peripheral structures. They involve rare entities with limited textual presence, overshadowed by dominant entities and lost among each other in the long tail of many but rare entities (Baayen 2001). We leverage the known tendency of pointwise mutual information (PMI, Church and Hanks 1990) to inflate for rare events, turning its weakness into a strength by weighting edges with PMI to foreground peripheral entity configurations. Communities extracted from the resulting network are structural traces of underlying narrative elements. We demonstrate the approach on The Lord of the Rings. From measures of how…
PDF arXiv
arXiv cs.AIOct 5
The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)
Feilian Huang
How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation…
PDF arXiv
arXiv econ.GNOct 5
Synthetic Cultural Agents from Aggregate Anchors
Augusto Gonzalez-Bonorino, Kseniia Biriukova, Monica Capra
Population prompts are widely used to generate synthetic survey responses, but they combine information supplied at inference with associations already encoded during pretraining. We introduce an alternative construction that maps declared aggregate preference anchors into group-indexed choice policies. For each population, the signs of six Global Preferences Survey (GPS) coordinates deterministically label a shared bank of paired synthetic responses, and Direct Preference Optimization fits a parameter-efficient adapter to those comparisons. We evaluate the adapters on candidate World Values Survey (WVS) items using prompts that omit country names and distinguish four questions: recovery of the imposed labels, transfer of the anchor signal to new text, coherence between the GPS anchors and human WVS responses, and agreement between adapter and human scores. The adapters recover the imposed pairwise labels. On a purposively selected sixteen-country development panel, adapter trust scores completely separate the two GPS-sign groups and have a rank correlation of (0.74) with continuous GPS trust scores. Human-GPS and adapter-human associations remain unresolved on the same panel,…
PDF arXiv
arXiv cs.CLOct 5
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Yiqi Liu, Joseph James, Yang Wang, Kun Zhao, Chenghao Xiao +1
When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge--dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by $+0.079$. Applying the same configuration to six additional cells yields a $+0.0393$ mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank $p=0.0091$. Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 1h ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

262papers this week−50.6% vs last week

All papers · 2.4k With code · 64

Newest Most relevant

1.9k papers this month

Open problems in Frontier LLMs

Evaluating and benchmarking decision models for reliability and calibration3 papers raise it · 4 address it
Decision models are increasingly used in LLM systems but lack standardized ways to measure correctness, confidence, and robustness across controlled variations.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient and accurate fact-checking of AI and human reviews1 paper raises it · 1 addresses it
Peer review systems struggle with factual errors in comments, and current methods are either too slow or inaccurate for large-scale use.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Long-horizon agent context management and memory retention2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and current compaction methods fail to preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web-based deep search and evidence integration1 paper raises it · 1 addresses it
Agents cannot reliably perform multi-step research in live, open web environments with diverse, interactive, and heterogeneous information sources.
- Deep Search in Real-World Web Environmentsraises · addresses
User behavior modeling for long-term personalization1 paper raises it · 1 addresses it
LLMs lack the ability to infer and adapt to evolving user preferences and intent from real-world behavior data over time.
- Vela: User Behavior as Supervision Sparks LLM Intelligenceraises · addresses

Named by an AI from these papers' abstracts.