Hadoop: Iterative MapReduce Performance
Would you say that parallel computation with iterative MapReduce can be justified mainly when the size of the training data is too large for non-parallel computation for the same logic?
I am aware that there is an overhead for running MapReduce jobs. This can be critical for overall runtime when a large number of iterations are required.
I can imagine that sequential computation is faster than parallel computation with iterative MapReduce if the memory allows to store the dataset in many cases.
a source to share
No parallel processing system makes a lot of sense if a single machine does the job most of the time. The complexity associated with most parallelization tasks is significant and requires a good reason to use it.
Even when it becomes apparent that a task cannot be solved without parallel processing in a reasonable time, parallel execution frameworks come in a variety of flavors, from lower-level, science-oriented tools like PVM or MPI to high-level, specialized (e.g. maps / abbreviations ) frameworks like Hadoop.
Among the parameters you should consider are start time and scalability (how close to the linear scale of the system). Hadoop won't be a good choice if you need answers quickly, but it can be a good choice if you can fit your process into a minified map box.
a source to share
You can refer to the HaLoop project ( http://code.google.com/p/haloop ) which solves exactly this problem.
a source to share