Have Geek Brain

Hadoop Map / Shrink - simple use example to do the following

I have a MySQL database where I store the following BLOB (which contains a JSON object) and an ID (for that JSON object). A JSON object contains a lot of different information. Let's say "City: Los Angeles" and "State: California".

At the moment, there are about 500 thousand such records, but they are growing. And each JSON object is pretty big.

My goal is to do a search (in real time) on a MySQL database. Let's say I want to search for all JSON objects that have "state" in "California" and "city" in "San Francisco".

I want to use Hadoop for this task. My idea is that there will be a "job" that takes chunks of, say, 100 records (rows) from MySQL, checks them against the given search criteria, returns those (IDs) that match.

Pros / Cons? I understand that one might think that I should use simple SQL power for this, but the fact is that the structure of JSON objects is quite "heavy", if I put it as SQL schemas, there will be at least three hundred table joins, which (I tried, really) creates quite a headache, and building all the necessary indexes eats up RAM faster than I can think. ;-) And even then every SQL query must be parsed to use the indexes, otherwise it is literally a pain in a full scan. And with this structure, we have the only way "up" with only vertical scaling. But I'm not sure if this is the best option for me, as I can see how JSON objects will grow (data structure) and I can see that their number will grow too. :-)

Help? Can anyone point me to simple examples of how this can be done? Does this make sense at all? Am I missing something important?

Thanks.

+2


a source to share


3 answers


A few pointers to consider:

  • Hadoop (HDFS specifically) distributes data around a cluster of machines. Using MapReduce to parse / manipulate this data requires the data to be stored on HDFS in order to take advantage of the parallel processing capabilities of Hadoop.

  • Hadoop / MapReduce is not near real time. Even when working on small amounts of data, the time it takes for Hadoop to set up a job can be 30+ seconds. This is something that cannot be stopped.



Perhaps something to look out for would be using Lucene to index your JSON objects as documents. You can store the index in solr and easily query whatever you want.

+2


a source


actually you .. because searching one huge field for text will take much longer than indexing the database and finding the correct sql code. The database was built for use with sql and indexes, it doesn't have the ability to parse and index json, so any way to search in json (maybe just hacky string matching) would be much slower. 500k rows is not much for mysql to handle, you really don't need hadoop, just a nice normalized schema, correct indexes and optimized queries



0


a source


It looks like you are trying to recreate CouchDB. CouchDB is built with a sizing map and is designed specifically to work with JSON objects.

0


a source






More articles:


All Articles