One standard error rule for model selection using cross-validation

How to use cross-validation to pick best model?

For each model, we use k fold cross-validation to get its evaluation result (RMSE for example). Usually we will choose the model with minimum RMSE. 

What is one standard error rule?

One standard error rule says we are not going to pick the actual minimum of RMSEs. The reason is as follows. 

For each model, its RMSE from k fold cross-validation is the mean of RMSE from each of k folds. And we should acknowledge that each RMSE from cross validation has variation because they are random like the data is. Actually they are the functions of the data. So it means that we can get standard error of the mean of k folds's RMSE for each model. 

Since the minimum RMSE also has variation, one standard error rule says that rather than picking a model that has minimum RMSE, we will pick the simplest model that has RMSE within one standard error of the minimum RMSE, because if a model's RMSE is within one standard error of each other's RMSE, we can't really tell them apart based on the sample data we have.

To sum up, one standard error rule says we should first calculate the standard error of the minimum error and the find the best model that has the error within one standard error of that minimum error.