Skip to Content

Publications Search - Abstract View

Title: Empirical Performance of Cross-Validation With Oracle Methods in a Genomics Context.
Authors: Martinez JG,  Carroll RJ,  Müller S,  Sampson JN,  Chatterjee N
Journal: Am Stat
Date: 2011 Nov 1
Branches: BB
PubMed ID: 22347720
PMC ID: PMC3281424
Abstract: When employing model selection methods with oracle properties such as the smoothly clipped absolute deviation (SCAD) and the Adaptive Lasso, it is typical to estimate the smoothing parameter by m-fold cross-validation, for example, m = 10. In problems where the true regression function is sparse and the signals large, such cross-validation typically works well. However, in regression modeling of genomic studies involving Single Nucleotide Polymorphisms (SNP), the true regression functions, while thought to be sparse, do not have large signals. We demonstrate empirically that in such problems, the number of selected variables using SCAD and the Adaptive Lasso, with 10-fold cross-validation, is a random variable that has considerable and surprising variation. Similar remarks apply to non-oracle methods such as the Lasso. Our study strongly questions the suitability of performing only a single run of m-fold cross-validation with any oracle method, and not just the SCAD and Adaptive Lasso.