What's the important part of search technology?
I think the big problem that has yet to be conquered in search is Ontology . We have a very difficult time understanding and classification of results in a friendly environment. Some call this the "semantic web". We have this lack of data and indexes, but we categorize or understand it very poorly. The fault lies in the fact that we do not possess atomic self-promotion in the way that a search (algorithm) can understand it.
a source to share
// Start a sarcastic response
RAM + CPU + Bandwidth + Really Amazing Search Engine
// End of sarcastic response
Seriously, though, something as complex as search technology will be affected by a myriad of challenges, as you pointed out. Trying to attach it to one single silver bullet really doesn't apply.
a source to share
Search engine performance is critical; slow service is dead, but this is not the most important aspect; this badge, I think, is about the relevance of the results.
We ask search engines to do the impossible; take a phrase or a few keywords and somehow return a whole bunch of results that are exactly what we wanted - we may not know exactly what we are looking for, but once we see it, we can admit it. Likewise, we know right away when a search engine returns the "wrong thing".
Ultimately, it's all about getting a dumb machine to do what we mean rather than what we say, which is basically an intractable problem in computing. The best search engine solutions will always be those that have the newest ways to rank results and link them to our original searches. This concept is quite closely related to the idea of ontology, as Wayne Hartman describes in his answer.
a source to share
I built a small search engine as part of my project last year and I can say that the inverted index is an important part of the search engine.
I can also say that the Crawler is an important part.
You can also say that the page rank algorithm is important.
You can argue that clustering is important.
....
What exactly are you looking for? your question seems broad to me.
a source to share