Eloise found that adding Batch Normalization was the best approach to improve the training speed of her deep neural network.
But everything comes at a price. Batch Normalization introduces some regularization to the network, and Eloise knows this will affect the Dropout she is currently using.
How should Eloise adjust the Dropout her network is using?
After adding Batch Normalization, Eloise should remove the Dropout she is currently using.
After adding Batch Normalization, Eloise should slightly decrease the amount of Dropout she is currently using.
After adding Batch Normalization, Eloise should slightly increase the amount of Dropout she is currently using.
After adding Batch Normalization, Eloise should significantly increase the amount of Dropout she is using.
2
Batch Normalization is a popular technique used to train deep neural networks. It normalizes the input to a layer during every training iteration using a mini-batch of data. It smooths and simplifies the optimization function leading to a more stable and faster training.
Batch Normalization acts as a regularizer. The mean of a mini-batch of data is a noisier version of the true mean of the population, and when used to normalize the data, it adds randomness to the optimization process. Eloise will need to adjust the Dropout she is using to accommodate this extra regularization.
Keep in mind that Batch Normalization doesn’t introduce a lot of regularization, so it’s unlikely that Eloise can remove all of the Dropout she is using. She should, however, slightly decrease the Dropout she is using to account for the extra regularization.
Recommended reading