Papers

Field

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
TanStack AI+68% downloads · 1M/wk+67.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%

Top tools here

Open problems in Frontier LLMs

Evaluating and benchmarking decision models for reliability and calibration3 papers raise it · 4 address it
Decision models are increasingly used in LLM systems but lack standardized ways to measure correctness, confidence, and robustness across controlled variations.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient and accurate fact-checking of AI and human reviews1 paper raises it · 1 addresses it
Peer review systems struggle with factual errors in comments, and current methods are either too slow or inaccurate for large-scale use.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Long-horizon agent context management and memory retention2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and current compaction methods fail to preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web-based deep search and evidence integration1 paper raises it · 1 addresses it
Agents cannot reliably perform multi-step research in live, open web environments with diverse, interactive, and heterogeneous information sources.
- Deep Search in Real-World Web Environmentsraises · addresses
User behavior modeling for long-term personalization1 paper raises it · 1 addresses it
LLMs lack the ability to infer and adapt to evolving user preferences and intent from real-world behavior data over time.
- Vela: User Behavior as Supervision Sparks LLM Intelligenceraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Chaoqiumian TechnologySeed · Oct 26undisclosed
OpenAIStage not given · Oct 22$30B
DeepSeekStage not given · Oct 22$15B
Xinxin HangtuStage not given · Oct 22$749.6M
Flapping AirplanesSeries A · Oct 21$1B

Recently funded

Topics in this field

language models llms · large language models · language models278 papers47 this week
retrieval augmented generation · knowledge graphs · retrieval augmented251 papers39 this week
memory construction · persistent memory · long term memory264 papers35 this week
large language models · machine translation · multilingual222 papers33 this week
activation steering · attention heads · attention168 papers20 this week
generative recommendation · sequential recommendation · personalized151 papers20 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.LGOct 7
From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis
Raviraja G, Viraj Bagal, Prabhath Chellingi
Organizations increasingly use frontier language models to analyze customer feedback, but answer quality also depends on how that feedback is organized and made available. We define a \emph{customer context graph} as a unified model of customer and business context. Typed relationships connect customer objects (feedback, conversations, users, and accounts), operational objects (tickets, support agents, opportunities, and competitors), and analytical or action objects (taxonomy concepts, evidence, insights, work items, and outcomes). This lets an agent investigate not only what customers say, but why, who is affected, what action followed, who owns it, and whether it was resolved. For this experiment, the graph is populated from public Cursor feedback; the same architecture can support any type of feedback source. We compare Agentic RAG, a Deep Research Agent, and a Customer Context Graph-backed Agent on the same 9,432 public Cursor feedback records using 30 realistic product, incident, comparison, and metadata questions. Without exhaustive ground truth, we jointly score responses on answer quality (coverage and organization), analytical depth (specificity and decomposition), and…
PDF arXiv
arXiv cs.IROct 7
From Chunks to Functional Evidence: Function-Aware Retrieval for EDA Documentation QA
Xiaotian Qiu, Kairui Liu, Shi Chenyi, Jinyuan Deng, Qi Sun +1
Retrieval-Augmented Generation (RAG) is widely used to ground answers in documents. For complex technical documentation, however, the primary bottleneck is often not model reasoning but a mismatch between a query and the way knowledge is organized for retrieval. This mismatch is pronounced in Electronic Design Automation (EDA) documentation, where the information needed for an answer is scattered across heterogeneous yet tightly coupled artifacts. We therefore redesign the basic retrieval unit of RAG. Instead of operating on isolated chunks or binary relations, we collect typed artifacts into EDA functional units. Each unit is recorded as a hyperedge with links to its source chunks. We then train an encoder to align queries with functional units and combine unit retrieval with direct chunk retrieval. After mapping the selected units back to their sources, a unified reranker chooses the evidence given to the generator. On the newly constructed EDADocEval-QA dataset, our method improves ROUGE-L by 37.1% over Chunk RAG and 55.6% over the strongest graph baseline. On the public ORD-MMBench benchmark, it improves ROUGE-L by 30.0% over the strongest baseline. These results support…
PDF arXiv
arXiv cs.AIOct 7
Relevance Is Not Sufficiency: What Actually Closes the Evidence Gap in Long-Term Memory QA
Yufeng Li, Shuxin Li, Zhenhua Xu, Junxian Li, Peng Zeng +5
LLM agents that interact with a user across many sessions accumulate histories that exceed their context window, so they store past interactions in an external memory and answer each question from a small set of retrieved records. Existing memory systems rank records by lexical or embedding relevance, yet the top-ranked memories can each be relevant while jointly omitting a complementary fact that the answer requires, especially for multi-session and temporal questions. Drawing on the distinction between relevance and sufficiency in legal evidence scholarship, we recast memory retrieval as constructing a sufficient memory set. To operationalize this view, we introduce a blinded LLM judgment over the retrieved set, together with Gold Hit and Turn Hit as evidence-coverage proxies. We then propose Budgeted Flat Reconstruction (BFR), which builds sufficient sets over a fixed flat memory store in two stages. Specifically, we first apply Formal Concept Analysis for Memory Selection (FCA-MS) to decompose the question into information requirements and select a compact candidate subset that jointly covers them. Then, we repeatedly acquire unseen records through deeper text search or…
PDF arXiv
arXiv cs.AIOct 7
The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules
Litao Hu, Yutong Tang
Self-improving LLM systems propose changes to themselves and keep those that score better on a small evaluation set. We treat this keep-if-better step as selection under measurement noise, model the correlated errors of the candidates in a single decision, and study empirically what happens when the evaluation set is reused. In runs where Qwen models rewrite their own instructions and every candidate is also scored on 600 held-out items, most proposals after the first are harmful, and the model gives the size of the winner's curse of a generation's best candidate. With a prior from a separate pilot, it matches the average overstatement of first-generation commits in native loops, though not setting by setting. In a pre-registered study, the final selection-set score of greedy loops exceeded held-out accuracy by 13 to 20 points with 16 selection items and by 1 to 5 points with 256. Held-out gains grew with the selection set on TREC but not on GSM8K, and the tested acceptance rules did not beat greedy acceptance over whole runs. Gains measured on the selection set also exceeded held-out gains when a current model refined a competent instruction, and in the validation scores of GEPA…
PDF arXiv
arXiv cs.AIOct 6
Trajectory Abstraction for the Science of Language Agent Behavior
Tianqiang Yan
Scientific studies of language agents need behavioral variables that support hypotheses across tasks and models. We formulate this research problem as learning and testing a hierarchy of trajectory abstractions. A concrete recursive procedure first measures role- and phase-indexed events, proposes temporally constrained relations, and tests their stability across conditions. It then constructs episode-level motif variables from selected relations and repeats the analysis on those variables. Explicit measurement functions connect every abstraction level to the original trajectories. Observations and randomized protocol experiments assess the resulting hypotheses, while comparisons between intervention realizations determine whether an abstraction should be retained, refined, or restricted. We derive a finite-depth bound for accepted reductions, identify protocol effects on fixed abstractions, and characterize realization disagreement and composition of abstraction error. A finite-sample test makes projected intervention consistency operational, and constructed examples illustrate motif construction and abstraction refinement. The formulation distinguishes this experimental…
PDF arXiv
arXiv cs.LGOct 6
Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions
Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin +1
LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We introduce \textbf{Conditional Accuracy Profiling} (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality. CAP is benchmark-agnostic: it can be applied directly when a benchmark provides the required annotations, approximately through task-subset proxies, or through controlled augmentation when perturbation pairs can be generated. We instantiate CAP on seven LLM judges across six pairwise judging benchmarks, including \textsc{judgerEva-Standard}, a controlled testbed we created to support all eight conditions. CAP exposes profile differences hidden by aggregate accuracy: on \textsc{judgerEva}'s judge-independent Hard-Constructed subset, the two judges most sensitive to omitted qualifications rank in the bottom three of seven by overall accuracy, so omission sensitivity is not predicted by aggregate accuracy. Across benchmarks, Position…
PDF arXiv

Papers

Top tools here

All tools

billion-context+259% downloads · 1.1M/wk+258.8%
jcode+233% downloads · 25.5k/wk+233.1%
TanStack AI+68% downloads · 1M/wk+67.7%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
NVIDIA Dynamo+83% downloads · 18.4k/wk+83.4%

Recently funded

Market

Chaoqiumian TechnologySeed · Oct 26undisclosed
OpenAIStage not given · Oct 22$30B
DeepSeekStage not given · Oct 22$15B
Xinxin HangtuStage not given · Oct 22$749.6M
Flapping AirplanesSeries A · Oct 21$1B

Topics in this field

language models llms · large language models · language models278 papers47 this week
retrieval augmented generation · knowledge graphs · retrieval augmented251 papers39 this week
memory construction · persistent memory · long term memory264 papers35 this week
large language models · machine translation · multilingual222 papers33 this week
activation steering · attention heads · attention168 papers20 this week
generative recommendation · sequential recommendation · personalized151 papers20 this week

Active contributors · 90 days

Activity · 12 weeks

Fields · AI · updated 5m ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

330papers this week−35.5% vs last week

All papers · 2.4k With code · 61

Newest Most relevant

1.9k papers this month

Recent papers

All papers in Research

arXiv cs.LGOct 7
From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis
Raviraja G, Viraj Bagal, Prabhath Chellingi
Organizations increasingly use frontier language models to analyze customer feedback, but answer quality also depends on how that feedback is organized and made available. We define a \emph{customer context graph} as a unified model of customer and business context. Typed relationships connect customer objects (feedback, conversations, users, and accounts), operational objects (tickets, support agents, opportunities, and competitors), and analytical or action objects (taxonomy concepts, evidence, insights, work items, and outcomes). This lets an agent investigate not only what customers say, but why, who is affected, what action followed, who owns it, and whether it was resolved. For this experiment, the graph is populated from public Cursor feedback; the same architecture can support any type of feedback source. We compare Agentic RAG, a Deep Research Agent, and a Customer Context Graph-backed Agent on the same 9,432 public Cursor feedback records using 30 realistic product, incident, comparison, and metadata questions. Without exhaustive ground truth, we jointly score responses on answer quality (coverage and organization), analytical depth (specificity and decomposition), and…
PDF arXiv
arXiv cs.IROct 7
From Chunks to Functional Evidence: Function-Aware Retrieval for EDA Documentation QA
Xiaotian Qiu, Kairui Liu, Shi Chenyi, Jinyuan Deng, Qi Sun +1
Retrieval-Augmented Generation (RAG) is widely used to ground answers in documents. For complex technical documentation, however, the primary bottleneck is often not model reasoning but a mismatch between a query and the way knowledge is organized for retrieval. This mismatch is pronounced in Electronic Design Automation (EDA) documentation, where the information needed for an answer is scattered across heterogeneous yet tightly coupled artifacts. We therefore redesign the basic retrieval unit of RAG. Instead of operating on isolated chunks or binary relations, we collect typed artifacts into EDA functional units. Each unit is recorded as a hyperedge with links to its source chunks. We then train an encoder to align queries with functional units and combine unit retrieval with direct chunk retrieval. After mapping the selected units back to their sources, a unified reranker chooses the evidence given to the generator. On the newly constructed EDADocEval-QA dataset, our method improves ROUGE-L by 37.1% over Chunk RAG and 55.6% over the strongest graph baseline. On the public ORD-MMBench benchmark, it improves ROUGE-L by 30.0% over the strongest baseline. These results support…
PDF arXiv
arXiv cs.AIOct 7
Relevance Is Not Sufficiency: What Actually Closes the Evidence Gap in Long-Term Memory QA
Yufeng Li, Shuxin Li, Zhenhua Xu, Junxian Li, Peng Zeng +5
LLM agents that interact with a user across many sessions accumulate histories that exceed their context window, so they store past interactions in an external memory and answer each question from a small set of retrieved records. Existing memory systems rank records by lexical or embedding relevance, yet the top-ranked memories can each be relevant while jointly omitting a complementary fact that the answer requires, especially for multi-session and temporal questions. Drawing on the distinction between relevance and sufficiency in legal evidence scholarship, we recast memory retrieval as constructing a sufficient memory set. To operationalize this view, we introduce a blinded LLM judgment over the retrieved set, together with Gold Hit and Turn Hit as evidence-coverage proxies. We then propose Budgeted Flat Reconstruction (BFR), which builds sufficient sets over a fixed flat memory store in two stages. Specifically, we first apply Formal Concept Analysis for Memory Selection (FCA-MS) to decompose the question into information requirements and select a compact candidate subset that jointly covers them. Then, we repeatedly acquire unseen records through deeper text search or…
PDF arXiv
arXiv cs.AIOct 7
The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules
Litao Hu, Yutong Tang
Self-improving LLM systems propose changes to themselves and keep those that score better on a small evaluation set. We treat this keep-if-better step as selection under measurement noise, model the correlated errors of the candidates in a single decision, and study empirically what happens when the evaluation set is reused. In runs where Qwen models rewrite their own instructions and every candidate is also scored on 600 held-out items, most proposals after the first are harmful, and the model gives the size of the winner's curse of a generation's best candidate. With a prior from a separate pilot, it matches the average overstatement of first-generation commits in native loops, though not setting by setting. In a pre-registered study, the final selection-set score of greedy loops exceeded held-out accuracy by 13 to 20 points with 16 selection items and by 1 to 5 points with 256. Held-out gains grew with the selection set on TREC but not on GSM8K, and the tested acceptance rules did not beat greedy acceptance over whole runs. Gains measured on the selection set also exceeded held-out gains when a current model refined a competent instruction, and in the validation scores of GEPA…
PDF arXiv
arXiv cs.AIOct 6
Trajectory Abstraction for the Science of Language Agent Behavior
Tianqiang Yan
Scientific studies of language agents need behavioral variables that support hypotheses across tasks and models. We formulate this research problem as learning and testing a hierarchy of trajectory abstractions. A concrete recursive procedure first measures role- and phase-indexed events, proposes temporally constrained relations, and tests their stability across conditions. It then constructs episode-level motif variables from selected relations and repeats the analysis on those variables. Explicit measurement functions connect every abstraction level to the original trajectories. Observations and randomized protocol experiments assess the resulting hypotheses, while comparisons between intervention realizations determine whether an abstraction should be retained, refined, or restricted. We derive a finite-depth bound for accepted reductions, identify protocol effects on fixed abstractions, and characterize realization disagreement and composition of abstraction error. A finite-sample test makes projected intervention consistency operational, and constructed examples illustrate motif construction and abstraction refinement. The formulation distinguishes this experimental…
PDF arXiv
arXiv cs.LGOct 6
Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions
Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin +1
LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We introduce \textbf{Conditional Accuracy Profiling} (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality. CAP is benchmark-agnostic: it can be applied directly when a benchmark provides the required annotations, approximately through task-subset proxies, or through controlled augmentation when perturbation pairs can be generated. We instantiate CAP on seven LLM judges across six pairwise judging benchmarks, including \textsc{judgerEva-Standard}, a controlled testbed we created to support all eight conditions. CAP exposes profile differences hidden by aggregate accuracy: on \textsc{judgerEva}'s judge-independent Hard-Constructed subset, the two judges most sensitive to omitted qualifications rank in the bottom three of seven by overall accuracy, so omission sensitivity is not predicted by aggregate accuracy. Across benchmarks, Position…
PDF arXiv

Open problems in Frontier LLMs

Evaluating and benchmarking decision models for reliability and calibration3 papers raise it · 4 address it
Decision models are increasingly used in LLM systems but lack standardized ways to measure correctness, confidence, and robustness across controlled variations.
- DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systemsraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsraises · addresses
- Jeva: System One Models from First Principlesraises · addresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient and accurate fact-checking of AI and human reviews1 paper raises it · 1 addresses it
Peer review systems struggle with factual errors in comments, and current methods are either too slow or inaccurate for large-scale use.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses
Long-horizon agent context management and memory retention2 papers raise it · 3 address it
Agents lose critical information when context windows are exceeded, and current compaction methods fail to preserve decision-relevant details.
- REMORY: Learning Residual Memory for Context Compactionraises · addresses
- Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentsraises · addresses
- Cortex: Consolidating Context into Parametric Memoryaddresses
Real-world web-based deep search and evidence integration1 paper raises it · 1 addresses it
Agents cannot reliably perform multi-step research in live, open web environments with diverse, interactive, and heterogeneous information sources.
- Deep Search in Real-World Web Environmentsraises · addresses
User behavior modeling for long-term personalization1 paper raises it · 1 addresses it
LLMs lack the ability to infer and adapt to evolving user preferences and intent from real-world behavior data over time.
- Vela: User Behavior as Supervision Sparks LLM Intelligenceraises · addresses

Named by an AI from these papers' abstracts.