Have Geek Brain

Strengthening Learning and POMDP

  • I am trying to use multilayer NN to implement a probability function in a partially observable Markov process.
  • I thought the inputs to NN would be: current state, selected action, result state; The output is the probability in [0,1] (a problem performing the selected action in the current state will result in a result state)
  • In the course of training, I fed the inputs mentioned earlier in NN and I taught him to output = 1.0 for every case that already happened.

Problem:
For almost the entire test case, the probability of an exit is about 0.95 .. no exit was lower than 0.9! Even for almost impossible results, he gave this lofty problem.

PS: I think this is because I taught that these were only cases, but not by chance. But at every stage of the episode I cannot teach him the result = 0.0 for every non-random action!

Any suggestions how to deal with this issue? Or could there be another way to use NN or to implement a trial function?

thanks

+2


a source to share


2 answers


The problem is that the sum over all possible following states should be 1. If you create your network as such, this is not guaranteed. Two possible alternatives come to me, where I accept discrete states.

  • When making a forecast, run the network for each possible next state. Then normalize by dividing the sum of all probabilities.
  • Use one output for the next state. Then you can use a softmax layer (as in classification) and interpret the values, which then range from 0 to 1, and sum to 1 as probabilities.


The two are actually roughly mathematically equivalent.

In the case of continuous variables, you will have to take distributions (such as multivariate Gaussian) and use the parameters of that distribution (such as mean and covariance stdev) as outputs.

+2


a source


When setting NN you might want to set a wider range of data, in training is there any data you want to set closer to probability 0? Unless I suspect you can get bad results. As a first step, I would try to select a few different things in the training dataset.

Also how do you teach NN? Have you tried other methods? How about activation functions, maybe experiment using some different ones.



With neural networks, I think some trial and error in choosing a model will help. (Sorry if all this is not enough.)

0


a source






More articles:


All Articles