Geek Answers Handbook

The best way to manage a dynamic goroutin pool

Suppose I have a readable channel, it may or may not perform a task (depending on the workload) Specifically, for several hours there may be no work, and then a sudden problem in tasks may occur

I want my goroutine pool to increase from 1 to N, where N is max concurrency when work appears, and then automatically rolls down to 1, where there was no work for goroutine for more than X seconds, to avoid using memory / processor waste.

I could only use a fixed pool, since goroutines are cheap, but I don’t like the idea of ​​having thousands of idle goroutines that I can better use for these resources (mostly a drum, but still)

The sliding part is pretty simple.

for {
    timeoutTimer := time.NewTimer(WORKER_ROUTINE_TIMEOUT)

    select {
    case taskContext, isBatchRunning := <-runner.tasksCh:
        if !isBatchRunning {
            log.Print("task provider is closed, quit worker goroutine")
            return
        }

        runner.job.Process(&taskContext)
    case <-timeoutTimer.C:
        return
    }
}

But I'm not sure how to make the pool dynamically grow, i.e. on what condition does a new one appear

The priority for this pool is that it can respond quickly to increased load and expand to N (max concurrency) goroutines, with the ability to eventually roll up to more reasonable numbers (1 per minute) while reducing workload

PS I saw the package https://github.com/Jeffail/tunny , but it looks like it has nothing like adaptively scaling the current pool size. Did I miss something?

Thanks!

+4
source share
1 answer

Well, I'm not sure you need a pool. Gorotins quickly start up, and you probably don't need to cook them all the time.

. Go, . .

( goroutines), :

  • goroutine
  • goroutine.

. ; .

+1

More articles:


All Articles