How do I render a large set of documents?
I have 100 Gb of documents. I would like to characterize it and get a general idea of what topics are prevalent.
The documents are plain text.
I've considered using a tool like Google Desktop for searching, but it's too big to really guess what to look for and take too long to run enough queries to cover the entire set.
Are there freely available tools that will group a large dataset of documents?
Are there any tools that can visualize such clusters?
a source to share
For a basic NLP approach, you can represent each document as a vector based on word frequency, then lay down the document vectors using Bayesian or other techniques (SVM, k-means, etc.).
For relevant answers see this somewhat similar SO question .
a source to share
You need to learn tools that handle natural language. Basically you can quite reliably determine (using statistical tools) the document language (see http://en.wikipedia.org/wiki/N-gram ) and the discourse area (see http://en.wikipedia.org/wiki / Support_vector_machine ). Several tools should be available if you are starting with wikipedia.
a source to share