Pieces of fragments in Go

Question

I have a snippet with ~ 2.1 million lines of log in it, and I would like to create a slice of slices, with the lines being distributed as evenly as possible.

Here is what I still have:

// logs is a slice with ~2.1 million strings in it. var divided = make([][]string, 0) NumCPU := runtime.NumCPU() ChunkSize := len(logs) / NumCPU for i := 0; i < NumCPU; i++ { temp := make([]string, 0) idx := i * ChunkSize end := i * ChunkSize + ChunkSize for x := range logs[idx:end] { temp = append(temp, logs[x]) } if i == NumCPU { for x := range logs[idx:] { temp = append(temp, logs[x]) } } divided = append(divided, temp) } 

idx := i * ChunkSize will give me the current "chunk start" for the logs index, and end := i * ChunkSize + ChunkSize will give me the "end of fragment" or the end of the range of this fragment. I could not find any documentation or examples on how to fragment or split a fragment or iterate over a limited range in Go, so this is what I came up with. However, it only copies the first fragment several times, so it does not work.

How do I (as evenly as possible) cut a fragment in Go?

Answer 1

You do not need to create new fragments, just add logs fragments to the divided fragment.

http://play.golang.org/p/vyihJZlDVy

 var divided [][]string chunkSize := (len(logs) + numCPU - 1) / numCPU for i := 0; i < len(logs); i += chunkSize { end := i + chunkSize if end > len(logs) { end = len(logs) } divided = append(divided, logs[i:end]) } fmt.Printf("%#v\n", divided)