Code throws std :: bad_alloc, not enough memory or could it be an error?
I'm sorting out using a fairly large grammar (1.1 GB, that's data oriented parsing). The parser I'm using (bitpar) is said to be optimized for very ambiguous grammars. I am getting this error:
1terminate called after throwing an instance of 'std::bad_alloc'
what(): St9bad_alloc
dotest.sh: line 11: 16686 Aborted bitpar -p -b 1 -s top -u unknownwordsm -w pos.dfsa /tmp/gsyntax.pcfg /tmp/gsyntax.lex arbobanko.test arbobanko.results
Is there any hope? Does this mean that he is out of memory? It uses about 15GB before it crashes. The machine I am using has 32GB of RAM and also a swap. It fails before displaying one parse tree; I think it will work after reading the grammar while trying to build a diagram analysis for the first sentence.
The parser is an efficient parser for CYK diagrams using vector bit representations; I suppose this is already quite memory efficient. If in fact it takes too much memory, I could choose from the grammar rules, but that would of course reduce the accuracy of the parsing.
I think the problem is that I have so many non-terminals, I should probably try to find another parser (any suggestions?)
UPDATE: For the sake of posterity, I found the problem a long time ago. The grammar was too large due to an error, so the analyzer could not handle it with the available memory. With the correct grammar (which is an order of magnitude less), it works great.
a source to share
It is possible that the memory is becoming fragmented. This means that your program may not allocate 1KB, even though there is 17GB of free memory, when that 17GB is fragmented into 34 million free 512-byte blocks.
Of course, there is a chance that your program will calculate the memory allocation. A common mistake is to allocate -1 bytes of memory. Since the memory sizes are always positive, this is interpreted as size_t(-1)
more than 32 GB. But in fact, there is no fact that points in this direction.
To fix this problem, you need someone who speaks C ++. If this is indeed memory fragmentation, a good C ++ programmer can tailor the memory allocation strategy to suit your specific needs. Some strategies involve concatenating objects of the same size together and replacing the string with spacers.
a source to share
If your application uses a 32-bit memory model, each process will receive 4 GB of virtual address space. Of these, only 2G is available for user space.
I suspect your parser might try to allocate more available virtual memory. I'm not sure if Parser provides a mechanism for custom memory allocation. If so, you can try using memory mapped files for allocation and push them to memroy only when needed.
a source to share