Can control tests be run dynamically?

Question

I have several different implementations of the interface and many factors that I want to check. The ultimate goal is to create a grid of results for different implementations in different situations.

I could write a test for every possible combination, but it gets tedious:

func Benchmark_ImplA_N100_X300(b *testing.B){ impl := newImplA(100,300) runBenchmark(b,impl) } 

The more combinations I add, the more I have to copy / paste. It quickly becomes bulky.

I would like to do something like:

 tests := []testing.InternalBenchmark{} for _, n := range []int{50,100,500,10000}{ for _,x := range []int{300,200}{ for name, gen := range implementations{ impl = gen(n,x) bm := testing.InternalBenchmark{Name: fmt.Sprint(name,x,n)} bm.F = func(b *testing.B){ runBench(b,impl) } tests = append(tests,bm) } } } testing.RunBenchmarks(anything, tests) 

so that I can update the list of combinations and the tests will be magically executed in all combinations. I am trying to do something similar mainly in TestMain and nothing will work. I'm not sure if I am using it incorrectly, or if the testing package just does something funny.

I really don't care if the go test tool can handle this, or if there is another way.

Answer 1

Yes it is possible. In the test file ( xxx_test.go ) create your own function TestMain() and inside it after assembling dynamic tests (values ​​of struct testing.InternalBenchmark ), call testing.Main() , which correctly analyzes command line flags, creates and installs testing.M and prepares and calls testing.RunBenchmarks() . That way your dynamic tests will still be run with go test .

Notes: testing.Main() will never return as it calls os.Exit() . If you want to do further logging and calculations based on test results, you can also call testing.MainStart() .Run() (this is what testing.Main() ), and you can pass the exit code that M.Run() on os.Exit() .

Below is the full test file, which you can simply run with go test -bench . .

Conclusion: test results of dynamically generated tests (different implementations with different parameters):

 testing: warning: no tests to run PASS main.EngineA[impl=0, n=50, x=300]-4 100000 16716 ns/op main.EngineB[impl=1, n=50, x=300]-4 100000 24788 ns/op main.EngineA[impl=0, n=50, x=200]-4 100000 10764 ns/op main.EngineB[impl=1, n=50, x=200]-4 100000 16415 ns/op main.EngineA[impl=0, n=100, x=300]-4 50000 33426 ns/op main.EngineB[impl=1, n=100, x=300]-4 30000 48466 ns/op main.EngineA[impl=0, n=100, x=200]-4 50000 20452 ns/op main.EngineB[impl=1, n=100, x=200]-4 50000 33134 ns/op main.EngineA[impl=0, n=500, x=300]-4 10000 163087 ns/op main.EngineB[impl=1, n=500, x=300]-4 5000 238043 ns/op main.EngineA[impl=0, n=500, x=200]-4 10000 102662 ns/op main.EngineB[impl=1, n=500, x=200]-4 10000 163113 ns/op main.EngineA[impl=0, n=1000, x=300]-4 5000 319744 ns/op main.EngineB[impl=1, n=1000, x=300]-4 3000 512077 ns/op main.EngineA[impl=0, n=1000, x=200]-4 10000 201036 ns/op main.EngineB[impl=1, n=1000, x=200]-4 5000 325714 ns/op ok _/xxx/src/play 27.307s 

And the source (test file, for example dynbench_test.go ):

 package main import ( "fmt" "testing" ) type Engine interface { Calc() } type EngineA struct{ n, x int } func (e EngineA) Calc() { for i := 0; i < en; i++ { a, b := make([]byte, ex), make([]byte, ex) copy(b, a) } } type EngineB struct{ n, x int } func (e EngineB) Calc() { for i := 0; i < en*2; i++ { a, b := make([]byte, ex/2), make([]byte, ex/2) copy(b, a) } } func TestMain(m *testing.M) { implementations := [](func(n, x int) Engine){ func(n, x int) Engine { return EngineA{n, x} }, func(n, x int) Engine { return EngineB{n, x} }, } benchmarks := []testing.InternalBenchmark{} for _, n := range []int{50, 100, 500, 1000} { for _, x := range []int{300, 200} { for name, gen := range implementations { impl := gen(n, x) bm := testing.InternalBenchmark{ Name: fmt.Sprintf("%T[impl=%d, n=%d, x=%d]", impl, name, n, x)} bm.F = func(b *testing.B) { for i := 0; i < bN; i++ { impl.Calc() } } benchmarks = append(benchmarks, bm) } } } anything := func(pat, str string) (bool, error) { return true, nil } testing.Main(anything, nil, benchmarks, nil) } 

Notes # 2:

testing.Main() , testing.MainStart() and testing.InternalBenchmark can be changed (or removed) in the next version of Go:

Internal function / internal type, but exported because it is a cross-package; part or called by running the "go test" command.

Answer 2

I believe reading documents will help. I forgot

func Benchmark(f func(b *B)) BenchmarkResult

which will run my function without requiring any test harnesses at all.

The benchmark evaluates a single function. Useful for creating custom ones that do not use the "go test" command.

That way, I can iterate through my test cases and create a function for each opportunity, and then run the checklist directly.