Parallel Word Count in Go

Question

Jacob Östergard introduced this problem:

Write a program that reads text from standard input and returns (prints) the total number of different words found in the text.

How can we solve this problem using parallel programming (preferably in Go, but enough description in English)?

Answer 1

There are several possibilities, but I think you mean "effectively"?

The general idea would be to split the text into manageable chunks, put these chunks in a queue, and several consumers to process the chunks.

Map/Reduce :

          _ Worker_
         /          \
        /            \
Splitter--- Worker ---Aggregator
        \            /
         \_ Worker _/

"" , , , .

Splitter to Workers, , , Aggregator.

Answer 2

Go, .

/*
    The problem: Write a program that reads text from 
    standard-input, and returns (prints) the total 
    number of distinct words found in the text. 

    C versus C++, Jakob Østergaard, February 24th, 2004
    http://unthought.net/c++/c_vs_c++.html
*/

package main

import (
    "bufio"
    "fmt"
    "os"
    "unicode"
)

func main() {
    rdr := bufio.NewReader(os.Stdin)
    words := make(map[string]bool, 1024*1024)
    word := make([]int, 0, 256)
    for {
        r, n, _ := rdr.ReadRune()
        if unicode.IsSpace(r) || n == 0 {
            if len(word) > 0 {
                words[string(word)] = true
                word = word[:0]
            }
            if n == 0 {
                break
            }
        } else {
            word = append(word, r)
        }
    }
    fmt.Println(len(words))
}

" " . .