validation set以及cross validation的常见做法

如果给定的样本充足，进行模型选择的一种简单方法是随机地将数据集切分成三部分，分为训练集（training set）、验证集（validation set）和测试集（testing set）。训练集用来训练模型，验证集用于模型的选择，而测试集用于最终对学习方法评估。在学习到的不同复杂度的模型中，选择对验证集有最小预测误差的模型。由于验证集有足够多的数据，用它对模型进行选择也是有效的。

在许多实际应用中数据是不充足的，为了选择好的模型，可以采用交叉验证方法。

k折交叉验证(k-fold cross validation)：首先随机地将已给数据切分为k个互不相交的大小相同的子集，然后利用k-1个子集的数据训练模型，利用余下的子集测试数据，将这一过程对可能的k中选择重复进行，最后选出k次评测中平均测试误差最小的模型。

总结：实际使用时，我们通过训练集学习到参数，再计算交叉验证集上的error，再选择一个在验证集上error最小的模型，最后再在测试集上估计模型的泛化误差。

注意k-fold cross validation的目的不是为了选择模型，而是先是有了一个模型，对这个模型进行精度评定。此处不同的模型指的“generally when we say 'a model' we refer to a particular method for describing how some input data relates to what we are trying to predict. We don't generally refer to particular instances of that method as different models. So you might say 'I have a linear regression model' but you wouldn't call two different sets of the trained coefficients different models. ”

say we have two models, say a linear regression model and a neural network. How can we say which model is better? We can do K-fold cross-validation and see which one proves better at predicting the test set points. But once we have used cross-validation to select the better performing model, we train that model (whether it be the linear regression or the neural network) on all the data. We don't use the actual model instances we trained during cross-validation for our final predictive model.

---

除了上述选择模型的功能之外，validation set还用来防止过拟合。

To make sure you dont overfit the network you need to input the validation dataset to the network and check if the error is within some range. Because the validation set is not being using directly to adjust the weights of the netowork, therefore a good error for the validation and also the test set indicates that the network predicts well for the train set examples, also it is expected to perform well when new example are presented to the network which was not used in the training process.

Training set与Validation set都是在模型的training过程中使用的，训练过程的workflow：

for each epoch
    for each training data instance
        propagate error through the network
        adjust the weights
        calculate the accuracy over training data
    for each validation data instance
        calculate the accuracy over the validation data
    if the threshold validation accuracy is met
        exit training
    else
        continue training

---

1、I don't do a separate final training on all the training data, instead I average the responses of the 10 folded models on the test data as my final results. Which may make for better CV results, as you're guaranteed to know you're using the same models you've got CV results for.

2、I use a holdout sample (usually it was ids 0-5k), but I occasionally change the holdout sample. My hardware isn't that great so 5 fold cv is a bit time-consuming. It matched reasonably well with lb. I also use a watchlist of 20% of data to get the number of rounds before retraining, so I sort of have 3 holdout sets - lb, holdout, and watchlist.

3、I like to draw alot of samples with n = size of private-LB data to get an estimate for the private LB score (if train & test share same distribution) or n = size of public-LB to check the correlations between local score and public LB score.

4、Using a fixed hold out-set is rarely a good idea, because it's very prone to overfitting. Also take a look at the std of your CV, not just the mean. What you can do is to monitor how your k-fold scores are varying together and how your LB scores behave with respesct to that. You will like always see some patterns there, which can be used to draw some conclusions.

edit: "rarely a good idea" is misleading. Should be something like that: on datasets, where (stratified) k-fold cv is applicable, it is the safer bet compared to a single hold-out set.

5、My understanding of "out-of-fold" prediction, is that you do the following:

1. Run K-fold CV, and for each run generate n*(1/K) predictions from training data with size n .
2. Aggregate the K set of n*(1/K) predictions, so that you have n prediction, and this is what is referred as "out-of-fold" prediction

And what you suggest, is to sample over this out-of-fold prediction to calculate with error rate.

6、I don't wanted to state, that single hold-out sets are a no-go, they have their applications. Forecasting problems are probably the best example for that. In competitions like that (e. g. Rossmann Store Sales), k-fold cv does not work very well, because the data is not iid. Gert mentions some other examples, like splits by geo-locations. In this case, stratified-CV could be bad, but often you can still define more than one hold-out set, which have the desired distribution. Another application is to detect leakage. If you don't want to waste submissions in order to be sure that your pre-processing is leakage-free, you can create a local private test: putting some training data and treat those as test data. So don't look at this set for data exploration and do not use its labels for any pre-processing.

My point is, that a single hold-out gets overfitted faster and hence it's more dangerous to use, if you do not have a good rapport with the god of overfitting. It's easy to do the wrong things, after you got an overfitting-occured response from the LB. Besides, you do not have information regarding variance with a single hold-out. So, if you are unexperienced with all the overfitting caveats, I would suggest to prefer k-fold cv over single hold-out if applicable.

7、Nevertheless, it seems 10-fold CV with out-of-fold prediction is a very much an adequate solution

注意：

kaggle上面public leaderboards are based on validating the submissions against a random fraction of the test set and the private one's are validated against the rest of the test set. I was just going to add that private one's are released after the competition is over and the final ranking is determined based on the private leaderboard.. People can do well in public leaderboard, yet do really bad in the private one because of overfitting.

参考：

http://cvrs.whu.edu.cn/blogs/?p=154

https://www.kaggle.com/c/telstra-recruiting-network/discussion/19277