Having two sets of inputs combined in hadoop
I have a fairly simple question that I will try to present with an example
let's say you have a list of lines and a large file, and you want each handler to process part of the file and one of the lines in grep.
how should you do it? My impression is that the number of cartographers is the result of the inputSplits produced. I could run subsequent jobs, one for each line, but it seems like ... messy?
edit: I'm not actually trying to build a scaled down version of the grep map. I used it as an example of having two different inputs for a cartographer. Let's say I am listing A and B and would like the handler to work on 1 item from list A and 1 from list B
So, given that the issue is not a data dependency that would lead to the need for a quest chain, is my only option to somehow split the entire list A across all mappers and then inject 1 item of list B into each handler?
What I am trying to do is build some type of prefixed search structure for my data. So I have a giant text and a bunch of lines. This process has a strong memory bottleneck so I was after 1 piece of text / 1 line per carduper
a source to share
Mappers should be able to work independently and without side effects. parallelism could be that the matcher tries to match a string against all patterns. Each input is processed only once!
Otherwise, you can multiply each line of input by the number of patterns. Process each line with one pattern. Then start the reducer. A ChainMapper
is the solution of choice here. But remember: a string will appear twice if it matches two patterns. Is this what you want?
In my opinion, you should prefer the first scenario: each handler processes the string independently and checks it against all known patterns.
Hint: you can distribute templates using a function DistributedCache
for all cartographers! ;-) The input must be split using InputLineFormat
a source to share
Regarding your editing: In general, the cartographer is not used to process two elements at the same time. It should only handle one element of time. The job needs to be configured in a sense that there can be a cartographer for each input record and it will still work correctly!
It is of course suitable that the cartographer needs some supporting information to process the input. This information can be bypassed using the job configuration (Configuration.setString ()). A wider set of data must be transferred through a distributed cache.
Have you had a look at one of these options? I'm not sure if I fully understood your problem, so please check for yourself if it works; -)
By the way: a grateful vote for my well researched previous answer would be nice; -)
a source to share
a good friend had a big epiphany: what about chain 2 cartographer?
basically, run the job that starts the cartooper (no gear). The input is a list of strings, and we can arrange so that each handler only receives one string.
in turn, the first transformer starts a new job in which the input is text. It can bind a string by setting a variable in context.
a source to share