Papers

Field

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
LMDeploy+151% downloads · 12.7k/wk+150.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
Vercel AI SDK383 releases this week

Top tools here

Open problems in Frontier LLMs

Evaluating long-horizon agents in real-world web environments1 paper raises it · 1 addresses it
Current benchmarks do not test agents on live, open web tasks with diverse modalities and interactive access, limiting real-world applicability.
- Deep Search in Real-World Web Environmentsraises · addresses
Net value of agent memory over direct retrieval1 paper raises it · 1 addresses it
Memory systems often underperform naive retrieval, and their trade-offs between gains and losses (memory tax) are poorly understood.
- Memory Tax: Dissecting the Net Value of Agent Memoryraises · addresses
Structural requirements for decision-native models1 paper raises it · 3 address it
Existing models lack null-awareness, permutation invariance, option scalability, and cacheability, limiting their use in structured decision-making.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient fact-checking of review comments1 paper raises it · 1 addresses it
Frontier LLMs are accurate but slow and costly for fact-checking, while cheaper models struggle with complex assertions.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Inferring user habits from weak longitudinal evidence1 paper raises it · 2 address it
Agents struggle to infer and apply user habits from sparse or noisy historical data, limiting personalization.
- HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memoryraises · addresses
- Vela: User Behavior as Supervision Sparks LLM Intelligenceaddresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Raodong ZhixinStage not given · Oct 30undisclosed
CohereSeries B+ · Oct 29$7B
VinciSeries B+ · Oct 29$250M
&TruthSeed · Oct 29$70M
KineDBPre-seed · Oct 29$946k

Recently funded

Topics in this field

retrieval augmented generation · knowledge graphs · retrieval augmented291 papers41 this week
language models llms · large language models · language models314 papers38 this week
memory construction · persistent memory · long term memory293 papers30 this week
generative recommendation · sequential recommendation · personalized180 papers30 this week
deep research · scientific literature · citations163 papers27 this week
large language models · machine translation · multilingual242 papers24 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGSep 29
From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning
Yupeng Chang, Wenxuan Zhang, Yuan Wu
Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fixed-budget comparisons do not by themselves distinguish three empirical claims: whether more validation data improve checkpoint selection, whether a selection rule outperforms validation-loss selection, and whether it improves over simply retaining the final checkpoint. We therefore treat checkpoint selection as a finite-information decision problem. Holding completed training trajectories, candidate checkpoints, and independent test items fixed, we vary the validation budget and separately measure improvement from additional validation data, gain over negative log-likelihood (NLL) selection, and gain over the final checkpoint. Across 60 mathematical SFT trajectories and 19 configurations, increasing the validation budget from 32 to 305-313 examples raises independent-test accuracy by 0.32 percentage points (pp) for generated-accuracy selection and 0.29 pp for checkpoint agreement, with 95% configuration-bootstrap CIs of [0.10, 0.56] and [0.11, 0.50], respectively. At the full validation budget, the two generation-based rules…
PDF arXiv
arXiv cs.CLSep 29
Grounded Revision vs. Prior Injection: Probing Retrieval-Augmented Patent Claim Amendment
Josepha Michiko Leo, Hyun-seok Min, Yehoon Jang, Irvan Zidny, Jin-Woo Chung +1
Retrieval-augmented generation is widely used in professional writing, yet whether retrieval grounds revision or merely injects templates is rarely tested where "correct" has a definable meaning. Patent claim amendment supplies that signal: the examiner names the attacked limitation and cites prior art, providing per-case ground truth. We release three artifacts: (i) a corpus of 7,385 USPTO prosecution cases with XML-aligned pre/post claims, rejection, and cited prior art; (ii) a seven-probe battery comparing random and structural-match retrieval as two policies under a fixed prompt scaffold; (iii) a deterministic five-channel metric (C1-C3 and C5 in main, C4 supplementary) requiring no LLM evaluation. Across 9,600 pre-registered calls on four frontier LLMs (Claude Sonnet 4, Claude Haiku 4.5, GPT-5.4, GPT-4o-mini), no tested model exhibits detectable classical prior-injection behavior; retrieval effects are small and direction-inconsistent between random and structural retrieval, and the null is unchanged under a dense (semantic) retriever, across retrieval depths k in {1,3,5,10}, and under a paraphrase-sensitive grounding metric. Revision locality reveals a model-specific…
PDF arXiv
arXiv cs.CLSep 29
Retrieval Sensitivity to Identity Signals in Queries
Andrew Tang, Nicholas Deas, Kathleen McKeown, Vishal Misra
Dense retrievers decide which documents reach users and the language models that use them, yet they are typically evaluated with neutral queries. We ask whether the identity signals that real users express in their queries---political ideology and dialect---bias what a retriever returns. We design evaluations in two domains, political news and consumer-health questions, each pairing a controlled synthetic set that varies only the identity signal with naturalistic queries. Across five dense retrievers and a sparse baseline, every retriever (i) retrieves articles that align with the query's own political lean and (ii) performs worse for questions written in African American Language (AAL) than in White Mainstream English (WME). Two analyses tie these gaps to queries' identity signals beyond surface vocabulary: partialling out an aggregate lexical-asymmetry score leaves the synthetic gaps largely intact, and linear probes recover lean and dialect from the retrievers' query embeddings beyond token-level features. Left unaddressed, such retrieval biases risk contributing to polarization and reinforcing the health disparities already faced by AAL speakers. Code is available at…
PDF arXiv
arXiv cs.CLSep 29
Large-scale factor analysis shows machine intelligence is only partially interpretable
Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad Ardi
A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The $g$ factor is not dominated by any common theme, and…
PDF arXiv
arXiv cs.LGSep 29
AdaptArena: Evaluating Test-Time Personalization of Web Agents
Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala, Tomás Vergara Browne +5
Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing benchmarks. To address this gap, we introduce AdaptArena, a benchmark for evaluating test-time personalization of web agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and double-preference scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle agents with access to ground-truth preferences achieve an 82.92% success rate, the evaluated LLM agents using our framework reach at most…
PDF arXiv
arXiv cs.CLSep 29
Memory Consolidation Flattens the Temporal Shape of User Facts
Sugam Panthi, Muhaiminul Yeamin, Siyan Luo, Rabab Abdelfattah
Long-term memory systems turn conversations into short stored notes. A note can keep a user fact while losing evidence about whether the fact still holds. For example, "I am driving a Peugeot" can become "The user drives a Peugeot," which drops the cue that the activity is ongoing. We call this aspectual flattening and measure it with LAPSE, a benchmark of matched user statements that differ only in temporal form. We find that memory writers flatten aspect selectively. Three writer models flattened the progressive statement but kept its simple-present match in 244 of 381 pairs, never the reverse. The asymmetry holds in all 11 model configurations tested and in the installed pipelines mem0, Graphiti, and Letta. The lost cue matters to later readers. In exploratory tests, changing only the stored verb shifted all three readers' estimates that a fact still holds. When readers could ask the user before acting, two of three acted without asking more often on flattened notes. Our planned memory-use task could not detect this, because readers there acted on almost every stored fact, even expired ones. Memory writing can thus remove evidence that later models use to decide whether to act.
PDF arXiv

Papers

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
LMDeploy+151% downloads · 12.7k/wk+150.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
Vercel AI SDK383 releases this week

Topics in this field

retrieval augmented generation · knowledge graphs · retrieval augmented291 papers41 this week
language models llms · large language models · language models314 papers38 this week
memory construction · persistent memory · long term memory293 papers30 this week
generative recommendation · sequential recommendation · personalized180 papers30 this week
deep research · scientific literature · citations163 papers27 this week
large language models · machine translation · multilingual242 papers24 this week

Active contributors · 90 days

Recently funded

Market

Raodong ZhixinStage not given · Oct 30undisclosed
CohereSeries B+ · Oct 29$7B
VinciSeries B+ · Oct 29$250M
&TruthSeed · Oct 29$70M
KineDBPre-seed · Oct 29$946k

Recent papers

All papers in Research

arXiv cs.LGSep 29
From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning
Yupeng Chang, Wenxuan Zhang, Yuan Wu
Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fixed-budget comparisons do not by themselves distinguish three empirical claims: whether more validation data improve checkpoint selection, whether a selection rule outperforms validation-loss selection, and whether it improves over simply retaining the final checkpoint. We therefore treat checkpoint selection as a finite-information decision problem. Holding completed training trajectories, candidate checkpoints, and independent test items fixed, we vary the validation budget and separately measure improvement from additional validation data, gain over negative log-likelihood (NLL) selection, and gain over the final checkpoint. Across 60 mathematical SFT trajectories and 19 configurations, increasing the validation budget from 32 to 305-313 examples raises independent-test accuracy by 0.32 percentage points (pp) for generated-accuracy selection and 0.29 pp for checkpoint agreement, with 95% configuration-bootstrap CIs of [0.10, 0.56] and [0.11, 0.50], respectively. At the full validation budget, the two generation-based rules…
PDF arXiv
arXiv cs.CLSep 29
Grounded Revision vs. Prior Injection: Probing Retrieval-Augmented Patent Claim Amendment
Josepha Michiko Leo, Hyun-seok Min, Yehoon Jang, Irvan Zidny, Jin-Woo Chung +1
Retrieval-augmented generation is widely used in professional writing, yet whether retrieval grounds revision or merely injects templates is rarely tested where "correct" has a definable meaning. Patent claim amendment supplies that signal: the examiner names the attacked limitation and cites prior art, providing per-case ground truth. We release three artifacts: (i) a corpus of 7,385 USPTO prosecution cases with XML-aligned pre/post claims, rejection, and cited prior art; (ii) a seven-probe battery comparing random and structural-match retrieval as two policies under a fixed prompt scaffold; (iii) a deterministic five-channel metric (C1-C3 and C5 in main, C4 supplementary) requiring no LLM evaluation. Across 9,600 pre-registered calls on four frontier LLMs (Claude Sonnet 4, Claude Haiku 4.5, GPT-5.4, GPT-4o-mini), no tested model exhibits detectable classical prior-injection behavior; retrieval effects are small and direction-inconsistent between random and structural retrieval, and the null is unchanged under a dense (semantic) retriever, across retrieval depths k in {1,3,5,10}, and under a paraphrase-sensitive grounding metric. Revision locality reveals a model-specific…
PDF arXiv
arXiv cs.CLSep 29
Retrieval Sensitivity to Identity Signals in Queries
Andrew Tang, Nicholas Deas, Kathleen McKeown, Vishal Misra
Dense retrievers decide which documents reach users and the language models that use them, yet they are typically evaluated with neutral queries. We ask whether the identity signals that real users express in their queries---political ideology and dialect---bias what a retriever returns. We design evaluations in two domains, political news and consumer-health questions, each pairing a controlled synthetic set that varies only the identity signal with naturalistic queries. Across five dense retrievers and a sparse baseline, every retriever (i) retrieves articles that align with the query's own political lean and (ii) performs worse for questions written in African American Language (AAL) than in White Mainstream English (WME). Two analyses tie these gaps to queries' identity signals beyond surface vocabulary: partialling out an aggregate lexical-asymmetry score leaves the synthetic gaps largely intact, and linear probes recover lean and dialect from the retrievers' query embeddings beyond token-level features. Left unaddressed, such retrieval biases risk contributing to polarization and reinforcing the health disparities already faced by AAL speakers. Code is available at…
PDF arXiv
arXiv cs.CLSep 29
Large-scale factor analysis shows machine intelligence is only partially interpretable
Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad Ardi
A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The $g$ factor is not dominated by any common theme, and…
PDF arXiv
arXiv cs.LGSep 29
AdaptArena: Evaluating Test-Time Personalization of Web Agents
Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala, Tomás Vergara Browne +5
Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing benchmarks. To address this gap, we introduce AdaptArena, a benchmark for evaluating test-time personalization of web agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and double-preference scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle agents with access to ground-truth preferences achieve an 82.92% success rate, the evaluated LLM agents using our framework reach at most…
PDF arXiv
arXiv cs.CLSep 29
Memory Consolidation Flattens the Temporal Shape of User Facts
Sugam Panthi, Muhaiminul Yeamin, Siyan Luo, Rabab Abdelfattah
Long-term memory systems turn conversations into short stored notes. A note can keep a user fact while losing evidence about whether the fact still holds. For example, "I am driving a Peugeot" can become "The user drives a Peugeot," which drops the cue that the activity is ongoing. We call this aspectual flattening and measure it with LAPSE, a benchmark of matched user statements that differ only in temporal form. We find that memory writers flatten aspect selectively. Three writer models flattened the progressive statement but kept its simple-present match in 244 of 381 pairs, never the reverse. The asymmetry holds in all 11 model configurations tested and in the installed pipelines mem0, Graphiti, and Letta. The lost cue matters to later readers. In exploratory tests, changing only the stored verb shifted all three readers' estimates that a fact still holds. When readers could ask the user before acting, two of three acted without asking more often on flattened notes. Our planned memory-use task could not detect this, because readers there acted on almost every stored fact, even expired ones. Memory writing can thus remove evidence that later models use to decide whether to act.
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 21m ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

325papers this week−30.9% vs last week

All papers · 2.7k With code · 78

Newest Most relevant

1.8k papers this month

Open problems in Frontier LLMs

Evaluating long-horizon agents in real-world web environments1 paper raises it · 1 addresses it
Current benchmarks do not test agents on live, open web tasks with diverse modalities and interactive access, limiting real-world applicability.
- Deep Search in Real-World Web Environmentsraises · addresses
Net value of agent memory over direct retrieval1 paper raises it · 1 addresses it
Memory systems often underperform naive retrieval, and their trade-offs between gains and losses (memory tax) are poorly understood.
- Memory Tax: Dissecting the Net Value of Agent Memoryraises · addresses
Structural requirements for decision-native models1 paper raises it · 3 address it
Existing models lack null-awareness, permutation invariance, option scalability, and cacheability, limiting their use in structured decision-making.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient fact-checking of review comments1 paper raises it · 1 addresses it
Frontier LLMs are accurate but slow and costly for fact-checking, while cheaper models struggle with complex assertions.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Inferring user habits from weak longitudinal evidence1 paper raises it · 2 address it
Agents struggle to infer and apply user habits from sparse or noisy historical data, limiting personalization.
- HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memoryraises · addresses
- Vela: User Behavior as Supervision Sparks LLM Intelligenceaddresses

Named by an AI from these papers' abstracts.