Neural network with softmax activation
edit:
More sharply asked question: What is the derivative of softmax that will be used in my gradient descent?
This is more or less a research project for the course and my understanding of NN is very / rather limited, so please be patient :)
I am currently in the process of building a neural network that tries to examine an input dataset and outputs the likelihood / likelihood of each classification (there are 5 different classifications). Naturally, the sum of all output nodes must be up to 1.
I currently have two levels and I have set a hidden layer containing 10 nodes.
I came up with two different types of implementations
- Logistic sigmoid to activate the hidden layer, softmax to activate the output
- Softmax to activate both hidden level and exit
I am using gradient descent to find local maxima to adjust the weights of the hidden nodes and the weights of the output nodes. I'm pretty sure I have it correct for the sigmoid. I'm less sure about softmax (or maybe I can use gradient descent at all), after a bit of research I couldn't find an answer and decided to calculate the derivative myself and got it softmax'(x) = softmax(x) - softmax(x)^2
(this returns a column vector of size n). I also looked into the MATLAB NN toolkit, the softmax derivative provided by the toolkit returned a square matrix of size nxn where the diagonal matches the math expression softmax '(x), which I calculated by hand; and I'm not sure how to interpret the output matrix.
I ran each implementation with a training rate of 0.001 and 1000 backpropagation iterations. However, my NN returns 0.2 (evenly distributed) for all five output nodes for any subset of the input dataset.
My findings:
- I'm pretty sure my descent gradient is wrong, but I don't know how to fix it.
- I may not be using enough hidden nodes
- Maybe I need to increase the number of layers
Any help would be greatly appreciated!
The dataset I'm working with can be found here (processed by Cleveland): http://archive.ics.uci.edu/ml/datasets/Heart+Disease
a source to share
The gradient you are using actually matches the squared error: output - target. This may seem unexpected at first, but the trick is that the other error function is minimized:
(- \sum^N_{n=1}\sum^K_{k=1} t_{kn} log(y_{kn}))
where log is the natural logarithm, N represents the number of training examples, and K is the number of classes (and therefore units in the output layer). t_kn depicts the binary encoding (0 or 1) of the kth class in the nth training example. y_kn is the corresponding network output.
Showing the correct gradient might be a good exercise, but I didn't do it myself.
To your problem: you can check the correctness of your gradient by numerical differentiation. Let's say you have a function f and an implementation of f and f '. Then the following should be done:
(f'(x) = \frac{f(x - \epsilon) - f(x + \epsilon)}{2\epsilon} + O(\epsilon^2))
a source to share
check out sites.google.com/site/gatmkorn for the open source Desire simulator. For the Windows version, the / mydesire / neural folder has several softmax classifiers, some with the softmax gradient descent algorithm.
In the examples, this works well for the simplemcharacter recognition task.
ASee also
Korn, GA: Advanced Dynamic System Modeling, Wiley 2007
SAC
a source to share
look at the link: http://www.youtube.com/watch?v=UOt3M5IuD5s derivative softmax: dyi / dzi = yi * (1,0 - yi);
a source to share