Have Geek Brain

Resources for character and text processing (encoding, regular expressions, NLP)

I would like to learn the basics of encodings, characters and text. Understanding this is important for working with a large set of texts, be it log files or a text source for building collective intelligence algorithms. My current knowledge is pretty basic: something like "As long as I'm using UTF-8 I'm fine".

I am not saying that I need to learn about cutting edge topics right away. But I need to know:

  • Knowledge of encodings at the bit and byte level.
  • Symbols and alphabets not used in English.
  • Multibyte encodings. (I understand some Chinese and Japanese, and their analysis is important.)
  • Regular expressions.
  • Text processing algorithm.
  • Natural language analysis.

I also need an understanding of corpus mathematics and linguistics. The current and future network (semantic, intelligent, real-time) requires processing, analysis and analysis of large text.

I'm looking for some resources (books perhaps?) That get me started with some of the bullets. (I find a lot of useful discussion of regular expressions here on Stack Overflow, so you don't need to suggest resources on this topic.)

+2


a source to share


2 answers


  • In addition to the wikipedia, by Joel Spolskys article in coding is also very good.
  • This free character map is a good resource for all Unicode characters.
  • This regex tutorial might be helpful.
  • Specifically NLP and Japanese , you can take a look at this Japanese NLP project.
  • In word processing , this Open Source Project can be helpful.


+3


a source


As usual for most general "I want to know about the X topic" questions, Wikipedia is a good place to start:

http://en.wikipedia.org/wiki/Character_encoding



http://en.wikipedia.org/wiki/Natural_language_processing

0


a source






More articles:


All Articles