Activation function, initializer function, etc., Effects for neural networks for face detection
There are various activation functions: sigmoid, tanh, etc. There are also several initializer functions: Nguyen and Widrow, random, normalized, constants, zero, etc. So they have a big impact on the outcome of a neural network specializing in face detection? Right now I'm using the Tanh activation function and just randomizing all weights from -0.5 to 0.5. I have no idea if this is the best approach, albeit with 4 hours to train the network each time, I would rather ask here than experiment!
a source to share
Take a few hundred cases of data and look at the mean and standard deviation of the activation values of your devices. You want to exit the sine tuna saturation mode.
I doubt that various sane initialization schemes will have a big impact on the quality of your decisions. This is probably good enough to just initialize the weights to be uniform over the interval [-1 / sqrt (N), + 1 / sqrt (N)], where N is the number of incoming connections.
Thus, what is usually of great importance is a preliminary estimate of the weight of the network, either as RBM or as an autoencoder. This can be useful even for neural networks with a single hidden layer, although it is much more important for deeper networks. You do not mention the architecture you are using, this information will allow a more useful answer to your question.
There is even a new init rule that seems to be well covered in this article: http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/447 The doc also mentions some of the symptoms bad initialization, which I mentioned above, which you can easily check.
To summarize, Uniform on [-1 / sqrt (N), + 1 / sqrt (N)] is not too bad and is not mentioned in the document I am linking to. Don't worry too much about it if you are using one of these. What's very important is the preview of the weights as an auto-encoder (or a limited Boltzmann machine) that you should look out for even if you only have one hidden layer.
If you want to pre-prepare the weight as RBM, you can switch to logistic sigmoids and even initialize the scales from a small Gaussian standard deviation without starting up, to trouble.
a source to share