Sparse Regression Distilled from a Single Robust Fit

Wooyoung Shin, Seunghwan Park

Robust linear fits can resist response contamination yet remain too dense or unstable for useful global explanations. We propose penalized distillation, which fits a smoothly clipped absolute deviation (SCAD) estimator to a robust initial estimator's empirical fitted surface along a safeguarded coordinate-descent path and evaluates candidate states separately for fidelity, parsimony, perturbation stability, and held-out prediction. The new results attach to the states the algorithm actually computes. Conditional on a fixed uncontaminated design, deterministic bounds transfer response-replacement boundedness from the initial fit to every retained path state. Turning to fixed dimension, we characterize the oracle-support branch by its empirical-Gram projection and influence function, give conditions for covariance-weighted least-squares approximation equivalence, and establish a path-conditional generalized information criterion. By contrast, at large dimension-to-sample ratios the full-coordinate robust fit collapses without warning, and screening restores the construction. Under a sure-screening framework, the robustness bound and the support and selection guarantees transfer to…