Geek Answers Handbook

JMH: using the same static object in all benchmark tests

I have a class that builds some complex data (imagine a large XML or JSON structure - that’s what). It takes time to build it. Therefore, I want to build it once, and then use the same data in all tests. Currently, I basically have an instance of a public static object defined in the class that defines main , and then explicitly references it in the tests (the code is a very simplified example):

 public class Data { // This class constructs some complicated data } public class TestSet { public static final Data PARSE_ME = new Data(...); public static void main(String[] args) throws RunnerException { Options opt = new OptionsBuilder() .include(".*ParserTest") // several tests .forks(1) .build(); new Runner(opt).run(); } } @State(Scope.Thread) public class SomeParserTest { @Setup(Level.Iteration) public void setup() { Parser parser = new Parser(TestSet.PARSE_ME); } @Benchmark public void getId() { parser.getId(123); } }

And this, of course, is terrible ... An equally evil option would be to create a separate class so that it could contain one static object. It would be nice to use something like

 Options opt = new OptionsBuilder() ... .param(/*my Data object comes here*/)

but param only accepts strings, so I'm not sure how to pass the object (and, more importantly, the same instance of the object!) to it.

So, is there something more elegant than the global object described above?

+6
source share
2 answers

Unfortunately, JMH does not provide the ability to exchange data between standards.

On the one hand, this violates the isolation of benchmarks, when one control test can silently change the input data for another test, making the comparison incorrect. This is why you need the @Setup object for each test.

But more importantly, any trick that you create for sharing data between standards (for example, the static field, available from both) will break in default forked mode when JMH runs each test in its own virtual machine. It is noteworthy that what you propose using static final Data TestSet.PARSE_ME will indeed be executed for each @Benchmark , since each new instance of the virtual machine must in some way initialize TestSet ;). Of course, you can turn off forking, but this creates more problems than it solves.

Thus, it might be better to think about investing time in making installation costs more bearable, so it is not excruciatingly painful. For example, deserialize data from disk, rather than calculate it. Or just come up with a faster way to calculate.

+7
source

I would just move all your test methods to one class, define your state object as an inner class, and enter the state for each of them:

 public class ParserBenchmarks { @State(Scope.Thread) public static class StateHolder { Parser parser = null; @Setup(Level.Iteration) public void setup() { parser = new Parser(TestSet.PARSE_ME); } public Parser getParser() { return parser; } } @Benchmark public int getId_123(StateHolder stateHolder) { return stateHolder.getParser().getId(123); } @Benchmark public int getId_456(StateHolder stateHolder) { return stateHolder.getParser().getId(456); } }

Note that all of your test methods must return values, otherwise the compiler could exclude it as dead code.

0
source

More articles:


All Articles