Spotlight

Papers

Field

Top tools here

Open problems in Classifier & causal discovery

Lack of standardized benchmarks for agentic black-box optimization1 paper raises it · 1 addresses it
Makes it hard to compare methods and isolate the effects of design choices across domains.
- A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimizationraises · addresses
Identifiability of individual world latents without reconstruction1 paper raises it · 1 addresses it
Current methods often rely on reconstruction or auxiliary supervision, limiting their applicability.
- DSReg: Provably Recovering Individual World Latents without Reconstructionraises · addresses
Handling noisy complementary labels in classification1 paper raises it · 1 addresses it
Complementary labels can be cheap but are prone to errors that harm model training.
- CAVA: Complementary-Label-Aware Data Valuationraises · addresses
Calibration in class-incremental learning1 paper raises it · 1 addresses it
Confidence calibration degrades asymmetrically across tasks as models adapt to new data.
- MT-CC: Multi-Group Temperature Scaling for Asymmetric Calibration Behavior in Class-Incremental Learningraises · addresses
Training with ML-labeled data under label noise1 paper raises it · 1 addresses it
ML-labeled data can introduce errors that degrade model performance if not addressed.
- Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimizationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

classifier · supervised · machine learning175 papers37 this week
causal discovery · causal inference · causal models118 papers16 this week
bayesian optimization · bayesian optimization bo · black box optimization91 papers12 this week
conformal prediction · split conformal prediction · aware conformal67 papers12 this week

Topics in this field

Active contributors · 90 days

Yujia ZhengDSReg: Provably Recovering Individual World Latents without Reconstruction4
Gunwoong ParkLocal Fisher Information Enables Sparse Causal Discovery4
Isabel ValeraDeDiCa: A Scalable Deconfounding Discrete Causal Generative Model3
Eyke HüllermeierUncertainty Quantification in Pairwise Difference Learning for Classification3
Yuheng JiaCalibrated Disambiguation for Partial Multi-label Learning3
Sébastien DesterckeInterval-valued SHAP in Tree-Based Models3

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

★ Spotlight this month

arXiv cs.AIOct 15

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe, Khuushi Maheshwari

Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recalibration), a reliability framework that separates confidence repair from deciding whether to deploy the repair and auditing the errors that remain. Under squared loss, we decompose prediction risk into outcome noise, information hidden by the score, and mapping error; restricting repair to order-preserving maps additionally introduces a pooling gap. For repair, we propose Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts. PAIC's excess risk depends on the probe's score-conditional bias and fitting error rather than its overall accuracy. For adoption, we derive a paired procedure with finite-sample control of false improvements. For auditing, we develop a prompt-clustered kernel test that detects feature-aligned residual errors while accounting for dependent answers from the same prompt. Across multiple LLM checkpoints, benchmarks, and distribution shifts, richer…

PDF arXiv page Details

Spotlight

Recent papers

All in Research →

arXiv cs.LGOct 19
Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization
Maria-Eleni Sfyraki, Jun-Kun Wang
Training a machine learning (ML) model is an optimization process that typically requires a set of examples with ground-truth labels beforehand. Obtaining these ground truths usually requires laborious human effort and can be costly. The scalability of human labeling becomes a concern when there are a large number of data points that need to be labeled. To address this practical issue, we consider an optimization setting where a subset of samples has ground-truth labels, while the remaining data points were labeled by an ML predictor, which could potentially assign incorrect labels to those points. We propose both first-order and zeroth-order stochastic optimization methods for training on datasets consisting of a mix of true-labeled and ML-labeled samples. We further conduct comprehensive empirical studies from training a linear regression model to fine-tuning a large language model (LLM) to showcase the effectiveness of the proposed algorithms for training with ML-labeled data.
PDF arXiv
arXiv cs.LGOct 19
CAVA: Complementary-Label-Aware Data Valuation
Tan-Ha Mai, Hsuan-Tien Lin, Vu Nguyen
Complementary-label learning (CLL) trains a classifier from labels that name a class an instance does not belong to. Such labels are cheap to collect from crowdworkers or vision-language models (VLMs), making them attractive for large-scale annotation. However, both sources can mistakenly assign the true class as the complementary label. Such annotation errors are particularly harmful in CLL because the model is explicitly trained to suppress the correct answer. Optimal transport (OT) data valuation identifies harmful samples by comparing training data with a clean validation set, but transferring it directly to CLL creates a semantic mismatch: label $c$ means not class $c$ in the training but class $c$ in the validation. We show that this makes the OT label groups noisy mixtures whose behavior depends on the noise rate. The transferred score is informative only at sufficiently high noise, becomes uninformative at the crossover rate $1/M$ for $M$ classes, and reverses below it, where practical CLL annotation typically operates. We propose CAVA, which introduces a complementary label cost and a label margin that isolates the contribution of each complementary label to the…
PDF arXiv
arXiv cs.AIOct 19
Local data augmentation for imbalanced binary classification
Harold Gu{é}neau, Alain Celisse, Pascal Delange
Facing imbalanced binary classification, we aim at maximizing the recall while limiting the proportion of false detections in the minority class. For doing that, the present work studies the effect of several rebalancing strategies, from the uniform subsampling of the observations from the minority class, to a more involved local data augmentation scheme. To be more specific, we establish that only rebalancing class proportions cannot modify the classification performance, whereas local data augmentation is proved to have an impact on the final results. The local augmentation scheme is introduced by means of a two-step procedure. The first step randomly subsamples anchor points among the training observations. These anchor points are used in the second step, for drawing new augmentation data following an ''augmentation kernel'' to be specified. This enables us to decouple the impact of the anchor points subsampling from the one of drawing augmentation data accounting (or not) for the local geometry of the support. The experiments on synthetic data illustrate the practical performance of several strategies for subsampling anchor points combined with either a Gaussian or a…
PDF arXiv
arXiv cs.LGOct 19
Optimal Multi-Kernel Combination for Online Multi-Label Classification
Tingting Zhai, Wei Liu, Renjian Yu
Multi-label classification is particularly challenging in streaming scenarios, where data arrives sequentially. Online kernel methods offer a promising solution. However, prevailing methods rely on a single preselected kernel, making them prone to suboptimal choices since the global data distribution is unknown in advance in online settings. Although the recent method, Online Kernel Selection for Online Multi-label Classification (OKS-OMC), enables data-driven kernel selection, it fails to fully harness the complementary information from multiple kernels and encounters the curse of kernelization. To address these limitations, this paper introduces a novel online multi-kernel combination method for large-scale multi-label classification. First, an efficient and robust online single-kernel multi-label method is designed. Second, learning an optimal convex combination of multiple multi-label kernel classifiers is formalized as a structural risk minimization problem. An improved alternating optimization procedure is developed, in which the combination coefficients are updated by directly minimizing the objective function, while each kernel classifier is updated by minimizing a…
PDF arXiv
arXiv cs.AIOct 19
On the Benefits of Uncertainty Estimation for Species Classification: A Case Study on Optimizing Expert re-Annotation
Kamenan N'Gadi, Martin Laviale, C{é}dric Pradalier, J{é}r{é}my Fix
Automated identification of diatoms from microscopic images is a cornerstone of large-scale freshwater biomonitoring, yet the high diversity of species and strong class imbalance pose significant challenges for reliable classification. In this study, we develop a ResNet-18 baseline for a 759-class diatom dataset and investigate how calibrated confidence estimates support selective classification to reduce expert workload without sacrificing taxonomic integrity. While the initial model exhibits overconfidence with an Expected Calibration Error (ECE) of 0.117, post-hoc temperature scaling (T = 2.12) dramatically improves reliability, lowering the ECE to 0.055.By leveraging these calibrated probabilities within a triage framework, we demonstrate that a Selective Macro-F1 score of 99.01% can be maintained at 66.2% coverage (rejection threshold $τ$ = 0.934). Qualitative analysis reveals that the system effectively functions as an intelligent filter, consistently deferring rare taxa-such as AAMB and AASI-and ambiguous morphologies for expert review. Our findings, supported by the torch-uncertainty framework, highlight the potential of simple calibration techniques to make automated…
PDF arXiv
arXiv stat.MLOct 19
Fair Regression under Approximate Demographic Parity
Weiwei Wang, Xianyang Zhang, Raymond K. W. Wong
Exact demographic parity requires the same prediction distribution across sensitive groups. In regression, this can be too restrictive and reduce predictive accuracy. We instead bound every pairwise Wasserstein distance between group-specific prediction distributions by a chosen tolerance, allowing the groups to differ while keeping fairness tunable. On the real line, the prediction and constrained transport problems have the same infimum. If the transport problem attains this value and suitable Monge maps exist, a deterministic transport-form predictor realizes it. Under a standard one-dimensional condition, the problem becomes a bounded quantile optimization that is convex when the conditional expected loss is convex in prediction. Under squared loss with two positively weighted groups, the $W_1$, $W_2$, and $W_\infty$ constraints yield soft-thresholding, shrinkage, and clipping. Under the stated conditions, we establish rates for the known-cost plug-in estimator. When the cost is unavailable, we describe an observed-loss computational proxy. Synthetic and California 2018 ACS studies illustrate the tunable fairness-accuracy trade-off and distinguish calibration feasibility from…
PDF arXiv

Papers

Topics in this field

classifier · supervised · machine learning175 papers37 this week
causal discovery · causal inference · causal models118 papers16 this week
bayesian optimization · bayesian optimization bo · black box optimization91 papers12 this week
conformal prediction · split conformal prediction · aware conformal67 papers12 this week

Active contributors · 90 days

Yujia ZhengDSReg: Provably Recovering Individual World Latents without Reconstruction4
Gunwoong ParkLocal Fisher Information Enables Sparse Causal Discovery4
Isabel ValeraDeDiCa: A Scalable Deconfounding Discrete Causal Generative Model3
Eyke HüllermeierUncertainty Quantification in Pairwise Difference Learning for Classification3
Yuheng JiaCalibrated Disambiguation for Partial Multi-label Learning3
Sébastien DesterckeInterval-valued SHAP in Tree-Based Models3

Activity · 12 weeks

Coming with more sources: repos

Open problems in Classifier & causal discovery

Lack of standardized benchmarks for agentic black-box optimization1 paper raises it · 1 addresses it
Makes it hard to compare methods and isolate the effects of design choices across domains.
- A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimizationraises · addresses
Identifiability of individual world latents without reconstruction1 paper raises it · 1 addresses it
Current methods often rely on reconstruction or auxiliary supervision, limiting their applicability.
- DSReg: Provably Recovering Individual World Latents without Reconstructionraises · addresses
Handling noisy complementary labels in classification1 paper raises it · 1 addresses it
Complementary labels can be cheap but are prone to errors that harm model training.
- CAVA: Complementary-Label-Aware Data Valuationraises · addresses
Calibration in class-incremental learning1 paper raises it · 1 addresses it
Confidence calibration degrades asymmetrically across tasks as models adapt to new data.
- MT-CC: Multi-Group Temperature Scaling for Asymmetric Calibration Behavior in Class-Incremental Learningraises · addresses
Training with ML-labeled data under label noise1 paper raises it · 1 addresses it
ML-labeled data can introduce errors that degrade model performance if not addressed.
- Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimizationraises · addresses

Named by an AI from these papers' abstracts.

★ Spotlight this month

arXiv cs.AIOct 15

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe, Khuushi Maheshwari

Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recalibration), a reliability framework that separates confidence repair from deciding whether to deploy the repair and auditing the errors that remain. Under squared loss, we decompose prediction risk into outcome noise, information hidden by the score, and mapping error; restricting repair to order-preserving maps additionally introduces a pooling gap. For repair, we propose Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts. PAIC's excess risk depends on the probe's score-conditional bias and fitting error rather than its overall accuracy. For adoption, we derive a paired procedure with finite-sample control of false improvements. For auditing, we develop a prompt-clustered kernel test that detects feature-aligned residual errors while accounting for dependent answers from the same prompt. Across multiple LLM checkpoints, benchmarks, and distribution shifts, richer…

PDF arXiv page Details

Fields · AI · updated 39m ago

Classifier & causal discovery

Found by the topic model: classifier, causal discovery, bayesian optimization.

77papers this week−28.7% vs last week

All papers · 451 With code · 2

Newest Most relevant

429 papers this month

Recent papers

All in Research →

arXiv cs.LGOct 19
Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization
Maria-Eleni Sfyraki, Jun-Kun Wang
Training a machine learning (ML) model is an optimization process that typically requires a set of examples with ground-truth labels beforehand. Obtaining these ground truths usually requires laborious human effort and can be costly. The scalability of human labeling becomes a concern when there are a large number of data points that need to be labeled. To address this practical issue, we consider an optimization setting where a subset of samples has ground-truth labels, while the remaining data points were labeled by an ML predictor, which could potentially assign incorrect labels to those points. We propose both first-order and zeroth-order stochastic optimization methods for training on datasets consisting of a mix of true-labeled and ML-labeled samples. We further conduct comprehensive empirical studies from training a linear regression model to fine-tuning a large language model (LLM) to showcase the effectiveness of the proposed algorithms for training with ML-labeled data.
PDF arXiv
arXiv cs.LGOct 19
CAVA: Complementary-Label-Aware Data Valuation
Tan-Ha Mai, Hsuan-Tien Lin, Vu Nguyen
Complementary-label learning (CLL) trains a classifier from labels that name a class an instance does not belong to. Such labels are cheap to collect from crowdworkers or vision-language models (VLMs), making them attractive for large-scale annotation. However, both sources can mistakenly assign the true class as the complementary label. Such annotation errors are particularly harmful in CLL because the model is explicitly trained to suppress the correct answer. Optimal transport (OT) data valuation identifies harmful samples by comparing training data with a clean validation set, but transferring it directly to CLL creates a semantic mismatch: label $c$ means not class $c$ in the training but class $c$ in the validation. We show that this makes the OT label groups noisy mixtures whose behavior depends on the noise rate. The transferred score is informative only at sufficiently high noise, becomes uninformative at the crossover rate $1/M$ for $M$ classes, and reverses below it, where practical CLL annotation typically operates. We propose CAVA, which introduces a complementary label cost and a label margin that isolates the contribution of each complementary label to the…
PDF arXiv
arXiv cs.AIOct 19
Local data augmentation for imbalanced binary classification
Harold Gu{é}neau, Alain Celisse, Pascal Delange
Facing imbalanced binary classification, we aim at maximizing the recall while limiting the proportion of false detections in the minority class. For doing that, the present work studies the effect of several rebalancing strategies, from the uniform subsampling of the observations from the minority class, to a more involved local data augmentation scheme. To be more specific, we establish that only rebalancing class proportions cannot modify the classification performance, whereas local data augmentation is proved to have an impact on the final results. The local augmentation scheme is introduced by means of a two-step procedure. The first step randomly subsamples anchor points among the training observations. These anchor points are used in the second step, for drawing new augmentation data following an ''augmentation kernel'' to be specified. This enables us to decouple the impact of the anchor points subsampling from the one of drawing augmentation data accounting (or not) for the local geometry of the support. The experiments on synthetic data illustrate the practical performance of several strategies for subsampling anchor points combined with either a Gaussian or a…
PDF arXiv
arXiv cs.LGOct 19
Optimal Multi-Kernel Combination for Online Multi-Label Classification
Tingting Zhai, Wei Liu, Renjian Yu
Multi-label classification is particularly challenging in streaming scenarios, where data arrives sequentially. Online kernel methods offer a promising solution. However, prevailing methods rely on a single preselected kernel, making them prone to suboptimal choices since the global data distribution is unknown in advance in online settings. Although the recent method, Online Kernel Selection for Online Multi-label Classification (OKS-OMC), enables data-driven kernel selection, it fails to fully harness the complementary information from multiple kernels and encounters the curse of kernelization. To address these limitations, this paper introduces a novel online multi-kernel combination method for large-scale multi-label classification. First, an efficient and robust online single-kernel multi-label method is designed. Second, learning an optimal convex combination of multiple multi-label kernel classifiers is formalized as a structural risk minimization problem. An improved alternating optimization procedure is developed, in which the combination coefficients are updated by directly minimizing the objective function, while each kernel classifier is updated by minimizing a…
PDF arXiv
arXiv cs.AIOct 19
On the Benefits of Uncertainty Estimation for Species Classification: A Case Study on Optimizing Expert re-Annotation
Kamenan N'Gadi, Martin Laviale, C{é}dric Pradalier, J{é}r{é}my Fix
Automated identification of diatoms from microscopic images is a cornerstone of large-scale freshwater biomonitoring, yet the high diversity of species and strong class imbalance pose significant challenges for reliable classification. In this study, we develop a ResNet-18 baseline for a 759-class diatom dataset and investigate how calibrated confidence estimates support selective classification to reduce expert workload without sacrificing taxonomic integrity. While the initial model exhibits overconfidence with an Expected Calibration Error (ECE) of 0.117, post-hoc temperature scaling (T = 2.12) dramatically improves reliability, lowering the ECE to 0.055.By leveraging these calibrated probabilities within a triage framework, we demonstrate that a Selective Macro-F1 score of 99.01% can be maintained at 66.2% coverage (rejection threshold $τ$ = 0.934). Qualitative analysis reveals that the system effectively functions as an intelligent filter, consistently deferring rare taxa-such as AAMB and AASI-and ambiguous morphologies for expert review. Our findings, supported by the torch-uncertainty framework, highlight the potential of simple calibration techniques to make automated…
PDF arXiv
arXiv stat.MLOct 19
Fair Regression under Approximate Demographic Parity
Weiwei Wang, Xianyang Zhang, Raymond K. W. Wong
Exact demographic parity requires the same prediction distribution across sensitive groups. In regression, this can be too restrictive and reduce predictive accuracy. We instead bound every pairwise Wasserstein distance between group-specific prediction distributions by a chosen tolerance, allowing the groups to differ while keeping fairness tunable. On the real line, the prediction and constrained transport problems have the same infimum. If the transport problem attains this value and suitable Monge maps exist, a deterministic transport-form predictor realizes it. Under a standard one-dimensional condition, the problem becomes a bounded quantile optimization that is convex when the conditional expected loss is convex in prediction. Under squared loss with two positively weighted groups, the $W_1$, $W_2$, and $W_\infty$ constraints yield soft-thresholding, shrinkage, and clipping. Under the stated conditions, we establish rates for the known-cost plug-in estimator. When the cost is unavailable, we describe an observed-loss computational proxy. Synthetic and California 2018 ACS studies illustrate the tunable fairness-accuracy trade-off and distinguish calibration feasibility from…
PDF arXiv