wordcount
Folders and files
Name | Name | Last commit date | ||
---|---|---|---|---|
parent directory.. | ||||
A distributed word counting example. A hasher shards <word,count> to multiple receives by hash(word). A receiver collects <word,count> from multiple hashers and writes the result to disk. Example run with 3 hashers and 4 receivers: 1. run 4 receivers on 4 machines, namely ip1:port1, ip2:port2, ip3:port3, ip4:port4. a. on ip1, bin/wordcount_receiver port1 3 b. on ip2, bin/wordcount_receiver port2 3 c. on ip3, bin/wordcount_receiver port3 3 d. on ip4, bin/wordcount_receiver port4 3 2. run 3 hashers on 3 machines. a. on ip1, bin/wordcount_hasher 'ip1:port1,ip2:port2,ip3:port3,ip4:port4' input1 b. on ip2, bin/wordcount_hasher 'ip1:port1,ip2:port2,ip3:port3,ip4:port4' input2 c. on ip3, bin/wordcount_hasher 'ip1:port1,ip2:port2,ip3:port3,ip4:port4' input3 input4 3. wait all hashers and receivers exit.