Have Geek Brain

Automatic creation of keywords from pdf

My app allows a user to download PDF files and save them to a web server for later viewing. I store filename, location, size, upload date, username, etc. In a SQL server database.

I would like to be able to programmatically, immediately after uploading the file, generate a list of keywords (possibly all but general words) and store them in a sql database so that subsequent users can perform keyword searches ...

Suggestions on how to approach this problem? Is there such a routine?

EDIT: Just to clarify my requirements, I wouldn't go into OCR, I don't know the internals of PDF files, but I understand that if it was generated by an application like Word-> PDF Print, the document text is searchable .. so really my first task and the goal of my question is: how do I access the text of a PDF file from an asp.net application? The OCR on the scanned PDF is probably not up to my requirements at the moment.

+1


a source to share


4 answers


As a first step, you should extract all the text from the PDF. ghostscript and pdftotext can do this, PDFBox is another option. There are other tools, of course.

Then you can remove all stopwords and duplicates and write them to the database.



I mentioned that this does not work for scanned PDF documents, but this is only half the truth. On the one hand, there are a lot of scanned PDFs that are additionally added to the text, because that's what some scanner drivers do (Canon CanoScan drivers do OCR and create searchable PDFs). On the other hand, documents created with LaTeX that contain non-ASCCII characters are throwing garbage back into my experience (even when copied and pasted in acrobat).

+2


a source


The only problem I see in capturing every non-generic word is that you will dilute the search results and have to query the database for more PDFs. One website to watch is Scribd , which does something similar to what you say with users uploading files and people who can view them online through a flash app.



0


a source


This is a very interesting topic. The question is how many keywords do you need to define one PDF file. If you say:

  • 3 to 10 - I would test categorization methods for text such as Bayesian classifier or K-NN (this method will group PDF files into similar clusters). I know that similar algorithms are used to filter spam. But this is a system that requires input, for example, if you add keywords in 100 PDF, this system will learn schemas. I'm not an expert, but this is one way to do it.

  • more than 10 - then I would suggest brute force -> general word filter -> get the most frequently used words for a specific document.

I would look into the first option. You should definitely check methods such as "text categorization", "automatic tagging", "text mining", "automatic keyword extraction".

Some links:

http://en.wikipedia.org/wiki/Naive_Bayes_classifier

Extracting keywords using Naive Bayes

0


a source


If you plan to index PDF documents, you should consider using a dedicated text search engine such as Lucene . Lucene provides features that are difficult to implement using just SQL and a relational database. You still need to extract text from PDF documents, but you don't have to worry about filtering common words. By filtering out common words, you will completely lose the ability to perform phrase searches.

0


a source






More articles:


All Articles