MapReduce: word count in three phases
MapReduce is a programming model for processing large data sets on clusters, introduced by Google in 2004. The programmer writes two functions. A map function takes one input record and emits key–value pairs. A reduce function takes one key and all the values collected for it and combines them. The framework handles splitting the data, scheduling work, moving data between machines, and recovering from failures.
Python
from collections import Counter, defaultdict
documents = ["the cat sat", "the dog sat", "the cat ran"]
def map_phase(document):
return [(word, 1) for word in document.split()]
def shuffle(pairs):
groups = defaultdict(list)
for key, value in pairs:
groups[key].append(value)
return groups
def reduce_phase(key, values):
return key, sum(values)
pairs = [pair for document in documents for pair in map_phase(document)]
grouped = shuffle(pairs)
result = dict(reduce_phase(key, values) for key, values in sorted(grouped.items()))
print(result)
assert result == dict(Counter(" ".join(documents).split()))