Grid competition

Sutton wanted to win the competition. She knew the only path forward was to dedicate as much time as possible to tune her model and squeeze as much performance as possible from it.

Sutton used a library to tune her model’s hyperparameters on the test set. She used Grid Search to try to cover as many combinations as possible and chose the hyperparameters that gave her the lowest training error.

What do you think about Sutton’s chances of winning the competition?

1. Sutton’s approach gives her a fighting chance.

2. Sutton’s approach will produce a model that fails to run.

3. Sutton’s approach will likely result in an overfit model.

4. We need more information to determine whether Sutton has a chance of winning the competition.

Expand to see the answer

3

Sutton used the test set to choose the best hyperparameters, which could lead to a model that overfits to the testing dataset.

This model will have trouble with unseen data because Sutton optimized it to work well on the test dataset. Instead, she should have used a validation set for hyperparameter tuning.

The test set is helpful for a final evaluation of the model, but anytime we use it to make changes to the model, we risk overfitting it.

Recommended reading

• “Overview of hyperparameter tuning” is a great introduction to hyperparameters and the process of finding their optimal value.
• “Train, Validation, Test Split for Machine Learning” goes into detail about the importance of each split.