Analysis on the Effect of Dropout as a Regularization Technique in Deep Averaging Network

Deep neural networks are powerful machine learning systems and many deep learning models for natural language processing tasks focus on learning the compositionality of their inputs. DAN model relies on both simple vector operations and neural network-based models for learning the compositionality. The depth of the model allows it to capture subtle variations in the input even though the composition is unordered. However, overfitting is a serious problem in any deep neural network. Dropout is a technique for addressing overfitting in large neural networks. The idea is to randomly drop neurons and their connections from the neural network during training phase. This prevents neurons from co-adapting. DAN includes a variant of dropout where individual words are dropped rather than individual neurons of the feed forward network.But since this technique has the potential to drop critical words it may have significant impact on the performance of the model in text classification tasks. This paper deliberates on this drawback and the impact of dropping individual neurons rather than word-level dropout.