Have Geek Brain

Search (and general query) with HBase and / or Cassandra (best practices?)

I have a user model object with quite a few fields (properties, if you like). Say "first name", "last name", "city" and "year of birth". Each user also receives a "unique identifier".

I want to be searchable. How can I do it right? How to do this in general?

My understanding (will work for almost any keystore) - key comes first, then value)

u: 123456789 = serialized_json_object

("u" is a simple prefix for custom keys, 123456789 is "unique identifier").

Now, thinking that I want to be able to search by first and last name, I can save in:

f: Steve = u: 384734807, u: 2398248764, u: 23276263 f: Alex = u: 12324355, u: 121324334

so the key "f" is the prefix for firstnames, and "Steve" is the actual name. For "u: Steve", we store as value all user IDs that are "Steve's".

This makes every search very, very easy. Querying on multiple fields (properties) - say the first (ie "Steve") and the last (ie "L: Anything") is still simple - first get a list of user IDs from "f: Steve" then list from "l: Anything", find cross user ids, here you go.

Problems (and there are quite a few of them):

  • Saving, updating, deleting a user is a pain. It should be atomic and consistent work. Also, if we have a value size limited to some value, then we are in a (potential) problem. And there really isn't an answer here. Only listing a list of user ids? Not too great.

  • In which id we want to add a new search field. Eventually. Say city. We can of course do the same: "c: Los Angeles" = ..., "c: Chicago" = ..., but if we don't anticipate all these "search options" from the beginning, then we will to be able to create some nightly work or whatever it takes for all existing user records and update "c: CITY" for them ... Quite a lot of work!

  • Blocking problems. User "u: 123" updates his name "Alex" and user "u: 456" updates his name "Alex". They both need to update "f: Alex" with their IDs. That means we'll either get down to the overwrite issue or one update will wait for another (and images, if there are many?).

What's the best way to do this? Remember that I want to search across many fields?

PS Please have a question about HBase / Cassandra / NoSQL / Key-Value Stores. Please, please - no advice on using MySQL or "reading about" SELECT; and worry about "later" scaling problems. There is a reason why I asked MY question exactly as I did. :-)

+2


source to share


2 answers


The ability to query properties directly is one of the features you lose when moving away from SQL, so you need a way to keep your own index so you can find records.

If your datastore does not have built-in indexing or atomic list operations, you will need to address the blocking issues you mentioned. However, indexing does not have to be synchronous - maintain a queue of updated records that need to be reindexed and you have solution for 3 that can be reused to solve problem 2.



If the list of indexes for a particular value becomes too large for the system to handle in a single list, you can replace the user list with a list of lists. However, if you have many records with the same value, this is probably not a particularly useful search criterion.

Another option that is useful in some cases is to use a separate indexing system - for example, you can configure lucene to index records in your main data store.

+4


source


I imagine this would be implemented as a MapReduce job that would run on a schedule. Each search word will be a string key with a UID lookup.

RowKey:
Profile UID1 : firstName: Joe
Profile: lastName: Doe
Profile: Nickname: DoeMaster

Rowkey: uid2
Profile: firstName: Jane
Profile: lastName: Doe
User: nick: SuperBabe

MapReduse indexes all searchable properties and adds them with the search word as a string string

Rowkey: Jane
lookup: uid: uid2

Rowkey: Doe
lookup: uid: uid2, uid1

Rowkey: DoeMaster
lookup: uid: uid1

.. etc.



Now, if you need to update the index list on the fly as a user change, you must write the change directly to the index database by removing the uid value from the index and adding rows to the other key. In this case, a simultaneous temporary blocking occurs.

To remove users, an optional attribute indicating the user's state can be used to filter them out of searches.

Adding an additional search word is not very difficult, since its the name you want to index. you can also filter your search by adding a type attribute to your keyword / keyword. ie boston - lookup: type: city.

The idea is to maintain your own string based search index inside hbase.

+1


source






More articles:


All Articles