Papers

Field

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
LMDeploy+136% downloads · 12.3k/wk+135.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
Vercel AI SDK300 releases this week

Top tools here

Open problems in Frontier LLMs

Evaluating long-horizon agents in real-world web environments1 paper raises it · 1 addresses it
Current benchmarks do not test agents on live, open, and heterogeneous web tasks with complex interactions.
- Deep Search in Real-World Web Environmentsraises · addresses
Net value of agent memory over direct retrieval1 paper raises it · 1 addresses it
Many memory systems underperform naive retrieval, and their trade-offs are poorly understood.
- Memory Tax: Dissecting the Net Value of Agent Memoryraises · addresses
Inferring user habits from weak longitudinal evidence1 paper raises it · 1 addresses it
Agents struggle to learn and apply user habits from sparse or noisy historical interactions.
- HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memoryraises · addresses
Structural requirements for decision-native models1 paper raises it · 3 address it
Current decision models often lack null-awareness, permutation invariance, option scalability, and cacheability.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient fact-checking of review comments1 paper raises it · 1 addresses it
Frontier LLMs are accurate but slow and expensive for checking implicit assertions in reviews.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Jingshuo TechnologySeed · Oct 29undisclosed
Kaola YouranSeries B+ · Oct 29undisclosed
Chaojinhua IntelligenceSeed · Oct 28undisclosed
Chaoqiumian TechnologySeed · Oct 26undisclosed
HypersphereSeed · Oct 26undisclosed

Recently funded

Topics in this field

language models llms · large language models · language models301 papers37 this week
retrieval augmented generation · knowledge graphs · retrieval augmented280 papers37 this week
memory construction · persistent memory · long term memory285 papers29 this week
large language models · machine translation · multilingual236 papers26 this week
generative recommendation · sequential recommendation · personalized171 papers26 this week
deep research · scientific literature · citations155 papers21 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv stat.MLOct 1
Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening
Jie Tang, Chuanlong Xie, Lixing Zhu
As artificial intelligence is increasingly deployed, algorithmic unfairness has raised growing concerns and intensified demands for transparent fairness auditing. In practice, the tolerable degree of algorithmic unfairness depends on the specific legal, ethical, or application context. Given a prespecified tolerance threshold, an important statistical question is how to determine whether a group disparity exceeds the allowable tolerance across different auditing objectives. To address this problem, we develop a unified tolerance-based fairness auditing framework for two complementary auditing objectives: violation certification, which prioritizes control of false violation declarations, and sensitivity screening, which prioritizes reducing missed violations. For the first objective, we develop a constrained empirical likelihood test for formal settings that uses least-favorable-point calibration and can be combined with false flagging rate control for simultaneous subgroup auditing. For the second objective, we develop split empirical likelihood and adjusted split empirical likelihood tests using an adaptive boundary-proxy principle for early-warning settings. Numerical…
PDF arXiv
arXiv cs.AIOct 1
What Can Analogy Tell Us About Artificial Consciousness?
Keith J. Holyoak, Martin M. Monti
Who or what is conscious? Because subjective experience is directly accessible only in the first person, judgments about consciousness in other entities depend partly on analogy. Historically, such inferences have focused on nonhuman animals, but advances in artificial intelligence have raised the possibility of conscious AI. Here we develop a causal framework for evaluating such evidential analogies. The key distinction is between similarities in factors plausibly involved in generating consciousness and similarities in downstream behavioural or cognitive effects. Our framework weights source-target similarity by causal relevance while allowing for unknown causes, disabling differences and alternative routes to consciousness. Applied to biological systems, it explains why analogical support generally weakens with increasing causal distance from humans. Applied to contemporary AI, it suggests that behavioural similarity provides only limited evidence for consciousness because relevant causal correspondences remain poorly established. The framework also clarifies what evidence would strengthen claims of artificial consciousness.
PDF arXiv
arXiv cs.AIOct 1
Evaluating LLM-Generated Preference Distributions
Fan Huang, Minsuk Kim, C. Tyler Diggans, Filippo Radicchi
Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settings where real-world data are unavailable. Yet, the structure and reliability of the distributions they produce remain understudied. Here, we systematically analyze LLM-generated distributions of preferences for air travel, restaurants, and consumer products. Encouragingly, all models considered in our analysis exhibit self-coherence, with the most probable outcomes stabilizing rapidly under repeated sampling. At the same time, we observe substantial discordance across both model families and scales, with little consensus even among their most probable outcomes. These patterns hold across nine open-weight models, three choice domains, and show robustness under temperature changes, greedy decoding, and perturbations of prompt and ordering. Our findings indicate that outcomes are influenced more by the choice of model than by the wording of the prompt, challenging the common assumption that sufficiently capable LLMs produce similar preference distributions when used as stand-ins for survey respondents.
PDF arXiv
arXiv cs.CLOct 1
Exploring the Design Space of Permutation Diffusion LLMs
Zhe Xu, Mingfu Liang, Kavosh Asadi, Jean-Pascal Billaud, Tianyi Chen +14
Permutation generation with language models requires both text-conditioned ranking and a global one-to-one assignment between candidates and output positions. We study this problem through three diffusion-LLM parameterizations: hard-assignment prediction over shuffled candidate-index sequences, soft-assignment prediction over doubly stochastic matrices, and direct ordering-score prediction followed by sorting. The three families place permutation constraints at different stages of prediction and decoding and therefore support different training objectives and sampling procedures. We evaluate these designs on recommendation reranking across three Amazon domains with candidate sets of size 10 and 20. At one function evaluation, the hard- and soft-assignment variants achieve Recall@1 comparable to a constrained masked-diffusion baseline, whereas direct ordering-score prediction guarantees valid permutations without Hungarian assignment or Sinkhorn normalization. Additional function evaluations do not consistently improve ranking quality, and their effect depends strongly on the state representation and training objective. These results show that corruption, prediction…
PDF arXiv
arXiv stat.MLOct 1
The Price of Correlated Tests: How Strict Should a Model Release Gate Be?
Marco Pollanen
Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many models that would have served users well, and it does not say how trustworthy a passing model actually is. We treat the release gate as a design problem: choose how many tests a model must pass so that cleared models meet a stated reliability target, while keeping as many good models as possible. A two-class latent-factor model makes both costs explicit and reduces each calculation to a one-dimensional integral. We prove that when both classes share the same latent correlation, a stricter gate always raises reliability, so the gate that keeps the most good models is the most lenient one that still meets the target. Under pass-all gating, any reliability target short of perfection is attainable within the model, but the share of good models kept tends to zero as the suite grows. Correlation between tests sets the price. In one configuration, a 99 percent target needs 8 independent tests, but 74 tests at a latent correlation of 0.3 and 5,182 at 0.5, where the gate keeps fewer than one good model in ten. We also give a validation…
PDF arXiv
arXiv cs.CLOct 1
A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models
Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu, Wuqiong Luo +2
Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generating standalone claims, users increasingly prefer grounded analyses that pair claims with verifiable citations from source documents to enable independent validation. However, evaluating such analytical claims typically requires extensive expert annotation, which is costly and difficult to scale, and real-world financial analysis commonly involves long context-question-answer triplets, further increasing task complexity. To address these challenges and benchmark the current landscape of grounded analysis by LLMs, we propose a numeric evidence evaluation method that enables groundedness assessment without reliance on expert annotation. We also introduce an automated dataset construction pipeline and construct ECTs-100 from the top 100 constituents of the S&P 500 to support benchmark of both groundedness and correctness. In addition, we examine conscious incompetence, a practical failure mode in financial analysis in which LLMs must detect when available evidence is insufficient and refrain from producing unsupported hallucinations.…
PDF arXiv

Papers

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
LMDeploy+136% downloads · 12.3k/wk+135.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
Vercel AI SDK300 releases this week

Recently funded

Market

Jingshuo TechnologySeed · Oct 29undisclosed
Kaola YouranSeries B+ · Oct 29undisclosed
Chaojinhua IntelligenceSeed · Oct 28undisclosed
Chaoqiumian TechnologySeed · Oct 26undisclosed
HypersphereSeed · Oct 26undisclosed

Topics in this field

language models llms · large language models · language models301 papers37 this week
retrieval augmented generation · knowledge graphs · retrieval augmented280 papers37 this week
memory construction · persistent memory · long term memory285 papers29 this week
large language models · machine translation · multilingual236 papers26 this week
generative recommendation · sequential recommendation · personalized171 papers26 this week
deep research · scientific literature · citations155 papers21 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv stat.MLOct 1
Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening
Jie Tang, Chuanlong Xie, Lixing Zhu
As artificial intelligence is increasingly deployed, algorithmic unfairness has raised growing concerns and intensified demands for transparent fairness auditing. In practice, the tolerable degree of algorithmic unfairness depends on the specific legal, ethical, or application context. Given a prespecified tolerance threshold, an important statistical question is how to determine whether a group disparity exceeds the allowable tolerance across different auditing objectives. To address this problem, we develop a unified tolerance-based fairness auditing framework for two complementary auditing objectives: violation certification, which prioritizes control of false violation declarations, and sensitivity screening, which prioritizes reducing missed violations. For the first objective, we develop a constrained empirical likelihood test for formal settings that uses least-favorable-point calibration and can be combined with false flagging rate control for simultaneous subgroup auditing. For the second objective, we develop split empirical likelihood and adjusted split empirical likelihood tests using an adaptive boundary-proxy principle for early-warning settings. Numerical…
PDF arXiv
arXiv cs.AIOct 1
What Can Analogy Tell Us About Artificial Consciousness?
Keith J. Holyoak, Martin M. Monti
Who or what is conscious? Because subjective experience is directly accessible only in the first person, judgments about consciousness in other entities depend partly on analogy. Historically, such inferences have focused on nonhuman animals, but advances in artificial intelligence have raised the possibility of conscious AI. Here we develop a causal framework for evaluating such evidential analogies. The key distinction is between similarities in factors plausibly involved in generating consciousness and similarities in downstream behavioural or cognitive effects. Our framework weights source-target similarity by causal relevance while allowing for unknown causes, disabling differences and alternative routes to consciousness. Applied to biological systems, it explains why analogical support generally weakens with increasing causal distance from humans. Applied to contemporary AI, it suggests that behavioural similarity provides only limited evidence for consciousness because relevant causal correspondences remain poorly established. The framework also clarifies what evidence would strengthen claims of artificial consciousness.
PDF arXiv
arXiv cs.AIOct 1
Evaluating LLM-Generated Preference Distributions
Fan Huang, Minsuk Kim, C. Tyler Diggans, Filippo Radicchi
Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settings where real-world data are unavailable. Yet, the structure and reliability of the distributions they produce remain understudied. Here, we systematically analyze LLM-generated distributions of preferences for air travel, restaurants, and consumer products. Encouragingly, all models considered in our analysis exhibit self-coherence, with the most probable outcomes stabilizing rapidly under repeated sampling. At the same time, we observe substantial discordance across both model families and scales, with little consensus even among their most probable outcomes. These patterns hold across nine open-weight models, three choice domains, and show robustness under temperature changes, greedy decoding, and perturbations of prompt and ordering. Our findings indicate that outcomes are influenced more by the choice of model than by the wording of the prompt, challenging the common assumption that sufficiently capable LLMs produce similar preference distributions when used as stand-ins for survey respondents.
PDF arXiv
arXiv cs.CLOct 1
Exploring the Design Space of Permutation Diffusion LLMs
Zhe Xu, Mingfu Liang, Kavosh Asadi, Jean-Pascal Billaud, Tianyi Chen +14
Permutation generation with language models requires both text-conditioned ranking and a global one-to-one assignment between candidates and output positions. We study this problem through three diffusion-LLM parameterizations: hard-assignment prediction over shuffled candidate-index sequences, soft-assignment prediction over doubly stochastic matrices, and direct ordering-score prediction followed by sorting. The three families place permutation constraints at different stages of prediction and decoding and therefore support different training objectives and sampling procedures. We evaluate these designs on recommendation reranking across three Amazon domains with candidate sets of size 10 and 20. At one function evaluation, the hard- and soft-assignment variants achieve Recall@1 comparable to a constrained masked-diffusion baseline, whereas direct ordering-score prediction guarantees valid permutations without Hungarian assignment or Sinkhorn normalization. Additional function evaluations do not consistently improve ranking quality, and their effect depends strongly on the state representation and training objective. These results show that corruption, prediction…
PDF arXiv
arXiv stat.MLOct 1
The Price of Correlated Tests: How Strict Should a Model Release Gate Be?
Marco Pollanen
Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many models that would have served users well, and it does not say how trustworthy a passing model actually is. We treat the release gate as a design problem: choose how many tests a model must pass so that cleared models meet a stated reliability target, while keeping as many good models as possible. A two-class latent-factor model makes both costs explicit and reduces each calculation to a one-dimensional integral. We prove that when both classes share the same latent correlation, a stricter gate always raises reliability, so the gate that keeps the most good models is the most lenient one that still meets the target. Under pass-all gating, any reliability target short of perfection is attainable within the model, but the share of good models kept tends to zero as the suite grows. Correlation between tests sets the price. In one configuration, a 99 percent target needs 8 independent tests, but 74 tests at a latent correlation of 0.3 and 5,182 at 0.5, where the gate keeps fewer than one good model in ten. We also give a validation…
PDF arXiv
arXiv cs.CLOct 1
A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models
Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu, Wuqiong Luo +2
Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generating standalone claims, users increasingly prefer grounded analyses that pair claims with verifiable citations from source documents to enable independent validation. However, evaluating such analytical claims typically requires extensive expert annotation, which is costly and difficult to scale, and real-world financial analysis commonly involves long context-question-answer triplets, further increasing task complexity. To address these challenges and benchmark the current landscape of grounded analysis by LLMs, we propose a numeric evidence evaluation method that enables groundedness assessment without reliance on expert annotation. We also introduce an automated dataset construction pipeline and construct ECTs-100 from the top 100 constituents of the S&P 500 to support benchmark of both groundedness and correctness. In addition, we examine conscious incompetence, a practical failure mode in financial analysis in which LLMs must detect when available evidence is insufficient and refrain from producing unsupported hallucinations.…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 1h ago

Frontier LLMs

The largest general-purpose language models, their releases, capabilities, scaling and training at frontier scale.

301papers this week−38.2% vs last week

All papers · 2.6k With code · 74

Newest Most relevant

1.9k papers this month

Open problems in Frontier LLMs

Evaluating long-horizon agents in real-world web environments1 paper raises it · 1 addresses it
Current benchmarks do not test agents on live, open, and heterogeneous web tasks with complex interactions.
- Deep Search in Real-World Web Environmentsraises · addresses
Net value of agent memory over direct retrieval1 paper raises it · 1 addresses it
Many memory systems underperform naive retrieval, and their trade-offs are poorly understood.
- Memory Tax: Dissecting the Net Value of Agent Memoryraises · addresses
Inferring user habits from weak longitudinal evidence1 paper raises it · 1 addresses it
Agents struggle to learn and apply user habits from sparse or noisy historical interactions.
- HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memoryraises · addresses
Structural requirements for decision-native models1 paper raises it · 3 address it
Current decision models often lack null-awareness, permutation invariance, option scalability, and cacheability.
- Jeva: System One Models from First Principlesraises · addresses
- LEV: Thinking Fast and Deep with Looped Decision Modelsaddresses
- SearchJev: A Fast and Calibrated System-1 Model for Search Agentsaddresses
Efficient fact-checking of review comments1 paper raises it · 1 addresses it
Frontier LLMs are accurate but slow and expensive for checking implicit assertions in reviews.
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviewsraises · addresses

Named by an AI from these papers' abstracts.