Papers

Field

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
TanStack AI+68% downloads · 1M/wk+67.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%

Top tools here

Open problems in Frontier LLMs

Evaluating and benchmarking decision models for reliability and calibration3 papers raise it · 4 address it
Decision models are increasingly used in LLM systems but lack standardized ways to measure correctness, confidence, and robustness across controlled variations.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient and accurate fact-checking of AI and human reviews1 paper raises it · 1 addresses it
Peer review systems struggle with factual errors in comments, and current methods are either too slow or inaccurate for large-scale use.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Long-horizon agent context management and memory retention2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and current compaction methods fail to preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web-based deep search and evidence integration1 paper raises it · 1 addresses it
Agents cannot reliably perform multi-step research in live, open web environments with diverse, interactive, and heterogeneous information sources.
- Deep Search in Real-World Web Environmentsraises · addresses
User behavior modeling for long-term personalization1 paper raises it · 1 addresses it
LLMs lack the ability to infer and adapt to evolving user preferences and intent from real-world behavior data over time.
- Vela: User Behavior as Supervision Sparks LLM Intelligenceraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Chaoqiumian TechnologySeed · Oct 26undisclosed
Star TherapeuticsSeries B+ · Oct 23$194.9M
OpenAIStage not given · Oct 22$30B
DeepSeekStage not given · Oct 22$15B
Xinxin HangtuStage not given · Oct 22$749.6M

Recently funded

Topics in this field

language models llms · large language models · language models278 papers45 this week
retrieval augmented generation · knowledge graphs · retrieval augmented251 papers39 this week
memory construction · persistent memory · long term memory264 papers34 this week
large language models · machine translation · multilingual222 papers31 this week
generative recommendation · sequential recommendation · personalized151 papers20 this week
activation steering · attention heads · attention168 papers18 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGOct 6
Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs
Ankush Checkervarty
Activation probes that monitor deployed language models are trained on synthetic conversations, and how many a probe needs is open. We trace learning curves over 10-590 synthetic samples for three monitoring concepts, high-stakes situations, replies harmful to a person, and replies that do not follow the user's instruction, on fourteen held-out evaluation distributions and four probe models, varying the generator LLM and the prompt's detail. The need is set by what is monitored: probes for high-stakes and harmful are within a few hundredths of their plateau from 80 samples on Gemma-3-27B-IT, instruction probes need several times as many, and the ordering holds on three smaller probe models and on real samples (from dev set). Prior work advises spending a generation budget on breadth, more kinds of data, over depth, more of each kind. We read the depth a concept needs as the half-gain size of a fitted curve, the number of samples at which half the gain is in hand. Concept and distribution account for 42-45% of its variance, the generator, probe model, and prompt detail for under 10%. What sets the value of the half-gain size is coverage, not per-kind difficulty: the number of…
PDF arXiv
arXiv cs.CLOct 6
Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch
Szymon Kocur
Historical Polish is well documented as a language but annotated in machine-readable form only to about a million words for the period this paper covers; the rest sits behind optical character recognition of variable quality. We present Wieszcz-XIX, a corpus of 6.75 billion tokens (about 3.1 billion words) in 294,369 documents, most of them periodical issues, of Polish published from 1800 to 1918, assembled from Wolne Lektury and the Internet Archive by a pipeline that filters, deduplicates, audits for post-1918 leakage and splits at the document level. It is over three orders of magnitude larger than the annotated corpus of the same period, and we quantify its defects: recognition corruption against a false-positive floor, near-identical duplication, which is removed, and post-1918 leakage, which is excluded from the training corpus itself down to a known residue of 0.04 to 0.38% of its bytes, found in the transcribed source, so the published corpus is the trained one document for document. On a hand-corrected sample the character error rate is 0.68% where the text is legible, and 45% of the sampled passages cannot be corrected. On it we train a ladder of decoder-only models…
PDF arXiv
arXiv cs.AIOct 6
Agent-Controlled Forgetting for Tool-Using Agents: Reversible Context Curation in Practice
Jan-Peter Franke
Tool-using agents repeatedly carry observations whose useful content can be much smaller than their original payload. We study agent-controlled forgetting: the acting model selects previously observed tool results, replaces each with a short note at its original position, and retains the exact original in a recoverable archive. A Python harness exposes batch archival and explicit recovery without task-specific model training, while protecting user instructions and assistant messages from these operations. In an exploratory OpenTelemetry debugging case followed by an unrelated implementation task, the method ended with 231,951 provider-reported prompt tokens versus 912,492 under retained history, used 50% fewer cumulative input tokens, and had an estimated API cost of USD 1.28-1.44 versus approximately USD 4.38. Both arms passed the two-case primary behavioral oracle; neither fully satisfied the follow-up evaluation. The method made more requests and took 17% longer. A contrasting application-development pair produced no context or cost saving, and an earlier continuation exhibited lower manually assessed quality despite reduced context. These observations demonstrate substantial…
PDF arXiv
arXiv cs.CLOct 6
IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas
Ziyu Chen, Yilun Zhao, Jiashuo Sun, Yiling Ma, Manasi Patwardhan +1
Scientific research often begins by synthesizing ideas from a set of related papers to identify gaps and formulate new directions. However, training language models to perform this form of literature-grounded ideation remains challenging, as existing approaches based on prompting or feedback lack structured supervision for how papers should be synthesized. We introduce IdeaAnchor, a paradigm for training LLMs to perform research ideation using structured specifications as privileged signals. Each IdeaAnchor instance encodes how each input paper should be synthesized into a successful idea, including their functional roles, relationships, and target synthesis criteria. We build this paradigm by mining instances from published papers, capturing how real ideas emerge from prior literature. We then train models via demonstration, self-distillation, and reinforcement learning, and further enhance generation with retrieval at inference time. Experiments show consistent improvements in ideation quality. Our analysis reveals a functional decomposition: anchor-based training strengthens creative synthesis, retrieval enhances detail elaboration, and combining both yields the best…
PDF arXiv
arXiv cs.CLOct 6
The Missing Minimal Pair: Stereotype Evaluation in LLMs
Nataliya Stepanova, Ivan Titov, Emily Allaway, Björn Ross
A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation. First, we present a data-augmentation framework that fills critical gaps in existing stereotype datasets by generating paraphrases and alternate attributes. We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes. Second, we introduce two evaluation metrics tailored to the dual minimal pair setup. One of these metrics provides a new perspective on bias by modeling the mutual information (MI) between social groups and stereotyped attributes. This MI-based metric is better suited for aggregation and enables more robust comparisons of stereotype strength across different languages and models. Our code is available at https://github.com/stepanat/missing-minimal-pair/.
PDF arXiv
arXiv cs.IROct 6
A Systematic Study of Semantic ID Spaces for Generative Information Retrieval
Alexia Allal, Hicham Randrianarivo, Sylvain Lamprier
Generative Information Retrieval (GIR) has emerged as a transformative paradigm, shifting document retrieval from a traditional "retrieve-and-rank" workflow to sequence-to-sequence generation, where a model directly predicts document identifiers (DocIDs). While the semantic design of these DocIDs is known to be critical for performance, a fundamental question remains under-explored: what makes a good DocID? Current approaches rely heavily on computationally expensive downstream evaluations, hindering systematic analysis and rapid iteration. In this work, we address this challenge by presenting a comprehensive study on the properties, metrics, and trade-offs that define effective numerical DocIDs. Specifically, our contributions are threefold: First, we propose a unified framework that unifies Product Quantization (PQ) and Residual Quantization (RQ), and their hybrid variants within a single design space. This enables us to systematically study key DocID properties, such as hierarchy versus parallelism, as well as the impact of hyperparameters like DocID length and codebook size. Second, we define a suite of training-free, intrinsic metrics, to quantify DocID quality and evaluate…
PDF arXiv

Papers

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
TanStack AI+68% downloads · 1M/wk+67.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%

Topics in this field

language models llms · large language models · language models278 papers45 this week
retrieval augmented generation · knowledge graphs · retrieval augmented251 papers39 this week
memory construction · persistent memory · long term memory264 papers34 this week
large language models · machine translation · multilingual222 papers31 this week
generative recommendation · sequential recommendation · personalized151 papers20 this week
activation steering · attention heads · attention168 papers18 this week

Recently funded

Market

Chaoqiumian TechnologySeed · Oct 26undisclosed
Star TherapeuticsSeries B+ · Oct 23$194.9M
OpenAIStage not given · Oct 22$30B
DeepSeekStage not given · Oct 22$15B
Xinxin HangtuStage not given · Oct 22$749.6M

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.LGOct 6
Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needs
Ankush Checkervarty
Activation probes that monitor deployed language models are trained on synthetic conversations, and how many a probe needs is open. We trace learning curves over 10-590 synthetic samples for three monitoring concepts, high-stakes situations, replies harmful to a person, and replies that do not follow the user's instruction, on fourteen held-out evaluation distributions and four probe models, varying the generator LLM and the prompt's detail. The need is set by what is monitored: probes for high-stakes and harmful are within a few hundredths of their plateau from 80 samples on Gemma-3-27B-IT, instruction probes need several times as many, and the ordering holds on three smaller probe models and on real samples (from dev set). Prior work advises spending a generation budget on breadth, more kinds of data, over depth, more of each kind. We read the depth a concept needs as the half-gain size of a fitted curve, the number of samples at which half the gain is in hand. Concept and distribution account for 42-45% of its variance, the generator, probe model, and prompt detail for under 10%. What sets the value of the half-gain size is coverage, not per-kind difficulty: the number of…
PDF arXiv
arXiv cs.CLOct 6
Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch
Szymon Kocur
Historical Polish is well documented as a language but annotated in machine-readable form only to about a million words for the period this paper covers; the rest sits behind optical character recognition of variable quality. We present Wieszcz-XIX, a corpus of 6.75 billion tokens (about 3.1 billion words) in 294,369 documents, most of them periodical issues, of Polish published from 1800 to 1918, assembled from Wolne Lektury and the Internet Archive by a pipeline that filters, deduplicates, audits for post-1918 leakage and splits at the document level. It is over three orders of magnitude larger than the annotated corpus of the same period, and we quantify its defects: recognition corruption against a false-positive floor, near-identical duplication, which is removed, and post-1918 leakage, which is excluded from the training corpus itself down to a known residue of 0.04 to 0.38% of its bytes, found in the transcribed source, so the published corpus is the trained one document for document. On a hand-corrected sample the character error rate is 0.68% where the text is legible, and 45% of the sampled passages cannot be corrected. On it we train a ladder of decoder-only models…
PDF arXiv
arXiv cs.AIOct 6
Agent-Controlled Forgetting for Tool-Using Agents: Reversible Context Curation in Practice
Jan-Peter Franke
Tool-using agents repeatedly carry observations whose useful content can be much smaller than their original payload. We study agent-controlled forgetting: the acting model selects previously observed tool results, replaces each with a short note at its original position, and retains the exact original in a recoverable archive. A Python harness exposes batch archival and explicit recovery without task-specific model training, while protecting user instructions and assistant messages from these operations. In an exploratory OpenTelemetry debugging case followed by an unrelated implementation task, the method ended with 231,951 provider-reported prompt tokens versus 912,492 under retained history, used 50% fewer cumulative input tokens, and had an estimated API cost of USD 1.28-1.44 versus approximately USD 4.38. Both arms passed the two-case primary behavioral oracle; neither fully satisfied the follow-up evaluation. The method made more requests and took 17% longer. A contrasting application-development pair produced no context or cost saving, and an earlier continuation exhibited lower manually assessed quality despite reduced context. These observations demonstrate substantial…
PDF arXiv
arXiv cs.CLOct 6
IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas
Ziyu Chen, Yilun Zhao, Jiashuo Sun, Yiling Ma, Manasi Patwardhan +1
Scientific research often begins by synthesizing ideas from a set of related papers to identify gaps and formulate new directions. However, training language models to perform this form of literature-grounded ideation remains challenging, as existing approaches based on prompting or feedback lack structured supervision for how papers should be synthesized. We introduce IdeaAnchor, a paradigm for training LLMs to perform research ideation using structured specifications as privileged signals. Each IdeaAnchor instance encodes how each input paper should be synthesized into a successful idea, including their functional roles, relationships, and target synthesis criteria. We build this paradigm by mining instances from published papers, capturing how real ideas emerge from prior literature. We then train models via demonstration, self-distillation, and reinforcement learning, and further enhance generation with retrieval at inference time. Experiments show consistent improvements in ideation quality. Our analysis reveals a functional decomposition: anchor-based training strengthens creative synthesis, retrieval enhances detail elaboration, and combining both yields the best…
PDF arXiv
arXiv cs.CLOct 6
The Missing Minimal Pair: Stereotype Evaluation in LLMs
Nataliya Stepanova, Ivan Titov, Emily Allaway, Björn Ross
A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation. First, we present a data-augmentation framework that fills critical gaps in existing stereotype datasets by generating paraphrases and alternate attributes. We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes. Second, we introduce two evaluation metrics tailored to the dual minimal pair setup. One of these metrics provides a new perspective on bias by modeling the mutual information (MI) between social groups and stereotyped attributes. This MI-based metric is better suited for aggregation and enables more robust comparisons of stereotype strength across different languages and models. Our code is available at https://github.com/stepanat/missing-minimal-pair/.
PDF arXiv
arXiv cs.IROct 6
A Systematic Study of Semantic ID Spaces for Generative Information Retrieval
Alexia Allal, Hicham Randrianarivo, Sylvain Lamprier
Generative Information Retrieval (GIR) has emerged as a transformative paradigm, shifting document retrieval from a traditional "retrieve-and-rank" workflow to sequence-to-sequence generation, where a model directly predicts document identifiers (DocIDs). While the semantic design of these DocIDs is known to be critical for performance, a fundamental question remains under-explored: what makes a good DocID? Current approaches rely heavily on computationally expensive downstream evaluations, hindering systematic analysis and rapid iteration. In this work, we address this challenge by presenting a comprehensive study on the properties, metrics, and trade-offs that define effective numerical DocIDs. Specifically, our contributions are threefold: First, we propose a unified framework that unifies Product Quantization (PQ) and Residual Quantization (RQ), and their hybrid variants within a single design space. This enables us to systematically study key DocID properties, such as hierarchy versus parallelism, as well as the impact of hyperparameters like DocID length and codebook size. Second, we define a suite of training-free, intrinsic metrics, to quantify DocID quality and evaluate…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 2m ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

320papers this week−37.9% vs last week

All papers · 2.4k With code · 62

Newest Most relevant

1.9k papers this month

Open problems in Frontier LLMs

Evaluating and benchmarking decision models for reliability and calibration3 papers raise it · 4 address it
Decision models are increasingly used in LLM systems but lack standardized ways to measure correctness, confidence, and robustness across controlled variations.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient and accurate fact-checking of AI and human reviews1 paper raises it · 1 addresses it
Peer review systems struggle with factual errors in comments, and current methods are either too slow or inaccurate for large-scale use.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Long-horizon agent context management and memory retention2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and current compaction methods fail to preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web-based deep search and evidence integration1 paper raises it · 1 addresses it
Agents cannot reliably perform multi-step research in live, open web environments with diverse, interactive, and heterogeneous information sources.
- Deep Search in Real-World Web Environmentsraises · addresses
User behavior modeling for long-term personalization1 paper raises it · 1 addresses it
LLMs lack the ability to infer and adapt to evolving user preferences and intent from real-world behavior data over time.
- Vela: User Behavior as Supervision Sparks LLM Intelligenceraises · addresses

Named by an AI from these papers' abstracts.