What fields in memstats structure apply only to heap, only to stack

Question

Go runtime has many different variables associated with the heap and the stack, and some of the stack numbers are part of the heap numbers, which leads to confusion (for me). For example, in this link . He says

// Stack numbers are part of the heap numbers, separate those out for user consumption stats.StackSys = stats.StackInuse stats.HeapInuse -= stats.StackInuse stats.HeapSys -= stats.StackInuse 

In runtime documents (excerpt from below), it gives 7 different fields associated with the heap (i.e. fields of the memstat structure), without a clear explanation of which ones include the stack, and similarly, which fields of the stack are included in the heap and how it relates to general distributions.

This is a problem because I want to compare the heap with the stack, but I don't want to select the heap variable that includes the stack (obviously).

Questions 1). Does the general distribution field include a heap, a stack, or both? 2), in what fields of the heap are stacks of numbers not included? 3) which heap fields include numbers for the stack? 4) which stack fields do not contain numbers for the heap?

  Alloc uint64 // bytes allocated and still in use TotalAlloc uint64 // bytes allocated (even if freed) Sys uint64 // bytes obtained from system (sum of XxxSys below) Lookups uint64 // number of pointer lookups Mallocs uint64 // number of mallocs Frees uint64 // number of frees // Main allocation heap statistics. HeapAlloc uint64 // bytes allocated and still in use HeapSys uint64 // bytes obtained from system HeapIdle uint64 // bytes in idle spans HeapInuse uint64 // bytes in non-idle span HeapReleased uint64 // bytes released to the OS HeapObjects uint64 // total number of allocated objects // Low-level fixed-size structure allocator statistics. // Inuse is bytes used now. // Sys is bytes obtained from system. StackInuse uint64 // bytes used by stack allocator StackSys uint64 

Answer 1

These questions are a little difficult to answer because goroutine stacks are allocated from the heap. Go does not have a clear separation between the stack and the heap that exists in C.

Does heap, stack, or both enter the general distribution field?

The TotalAlloc field of the MemStats structure includes all of the memory requested by the Go run operation from the OS for the Go heap. It does not include memory allocated for goroutine stacks. Initially, I thought it was, but I was wrong. Sorry for the confusion. I hope this answer will be more accurate.

(To be precise, I must mention that in a program using cgo, each thread (and not goroutine - usually more goroutines than threads) will have a stack allocated by the OS, this stack is not allocated by Go runtime and is not counted in TotalAlloc. It used only for cgo calls.)

, which heap fields do not include stacks of numbers? Which heap fields include numbers for the stack?

These fields include numbers for goroutine strings: HeapIdle, HeapReleased.

These fields do not include numbers for goroutine strings: HeapAlloc, HeapInUse, HeapObjects.

The HeapSys field does not include memory used by the currently active goroutine stacks, but includes memory for goroutine stacks that were once used but then freed.

which stack fields do not contain numbers for the heap?

I do not know how to answer this question in a way that makes sense. Stack fields report goroutine package information.

Answer 2

From the launch (variation) of the test program and viewing in the Go source, I see:

• Alloc and TotalAlloc seem to cover only distributions other than the stack. The allocation of large local residents did not add their size to TotalAlloc, even if this caused stack growth.

• If memory is currently reserved for the goroutine stack, it is counted in the StackX van, and not in the HeapX clock. This is the subtraction that you found in the source. It also implies everything that allocates space for stacks, can reduce HeapSys and HeapIdle, but leave HeapInuse alone.

  • Because you asked: stack fields never include heap allocations — stacks come from the heap, but not vice versa.
  • Variables that you think are on the heap ( ssp := new(SomeStruct) ) can actually be allocated on the stack if evacuation analysis can determine that they are not expecting a function call. This is almost always useful for you, because these vars can be freed when the function exits without creating garbage for the GC. Don't worry too much about it. :)

• Once goroutine completes, its stack space can be returned to the heap. (If its stack is small, it will most likely be cached for reuse as a future goroutine stack.) Then it will not be displayed as stack space and can again be displayed as available heap space. I see this both empirically and in the Go source (proc.c gfput calls runtime · stackfree). This means that exiting goroutines or old stacks returned after stack growth may appear to grow HeapSys and HeapIdle, but it really is just moving the space between uses.

• It seems that the TotalAlloc counter is not working, covering all the pages ever allocated to the stacks. If the goroutine stack is freed and reused, it is counted only once.

• There is definitely no TotalAlloc-style runtime counter covering all the variables associated with the stack. This will require tracking the function call overhead.

Stack issues are relatively rare , as variables allocated by the stacks are freed when the function returns, and large stacks themselves are freed when the goroutine exits. They can happen as if goroutines were leaking (they never go out, even when you create new ones), or if you make huge stack allocations ( var bigLocal [1e7]uint64 or funny deep recursion) in goroutines that do not go out. But it is much more common to have problems with the heap because the material in the heap is not freed up to the GC (tools like the Pool standard help you recycle the elements allocated by the heap to delay the need for GC).

So, in practical terms, I would basically keep an eye on Alloc and TotalAlloc for overuse of the heap, and if some stack space becomes a problem, look at the stack numbers (and maybe check for unexpectedly many goroutines running ).

These observations are implementation-specific (I look at 1.4, not the hint), and I am not an expert in Go source code, so take it like that. This test program is for reference:

 package main import ( "fmt" "reflect" "runtime" "sync" ) var g []byte func usesHeap() { g = make([]byte, 1000) } func usesTempStack() { var l [1000]byte _ = l } func createsGoroutineAndWaits() { wg := new(sync.WaitGroup) wg.Add(1) go func() { usesTempStack() wg.Done() }() wg.Wait() } func createsGoroutine() { go usesTempStack() } func recurse(depth int, max int) { var l [1024]byte _ = l if depth < max { recurse(depth+1, max) } } func growsTheStack() { recurse(0, 1000) } func checkUsageOf(lbl string, f func(), before, after *runtime.MemStats) { _ = new(sync.WaitGroup) runtime.ReadMemStats(before) // using own goroutine so everyone starts w/the same stack size wg := new(sync.WaitGroup) wg.Add(1) // request GC in hopes of a fair start runtime.GC() go func() { runtime.ReadMemStats(before) for i := 0; i < 1000; i++ { f() } runtime.Gosched() runtime.ReadMemStats(after) wg.Done() }() wg.Wait() fmt.Println("Results for", lbl, "\n") beforeVal, afterVal := reflect.ValueOf(*before), reflect.ValueOf(*after) memStatsType := beforeVal.Type() fieldCount := memStatsType.NumField() for i := 0; i < fieldCount; i++ { field := memStatsType.Field(i) if field.Type.Kind() != reflect.Uint64 { continue } beforeStat, afterStat := int64(beforeVal.Field(i).Uint()), int64(afterVal.Field(i).Uint()) if beforeStat == afterStat { continue } fmt.Println(field.Name, "differs by", afterStat-beforeStat) } fmt.Println("\n") } func main() { before, after := new(runtime.MemStats), new(runtime.MemStats) checkUsageOf("growsTheStack", growsTheStack, before, after) checkUsageOf("createsGoroutineAndWaits", createsGoroutine, before, after) checkUsageOf("usesHeap", usesHeap, before, after) checkUsageOf("usesTempStack", usesTempStack, before, after) checkUsageOf("createsGoroutine", createsGoroutine, before, after) }