Graphical criteria criteria taking different orders of time

Question

I have a Criterion criterion where each bgroup corresponds to a test, and inside each bgroup there are two bench test values ​​with different parameters. For example:

 main = defaultMain [bgroup "test1" [bench "v1" test1_1, bench "v2" test1_2] ,bgroup "test2" [bench "v1" test2_1, bench "v2" test2_2 -- lots more tests ] 

Inside each bgroup two bench tests are comparable. However, test1 takes 2000 microseconds, and test2 45 microseconds. The overview graph (which is most useful for what I want to do) displays various tests on the same axes, so I can clearly see the differences in test1 , but test2 hard to see.

Is it possible to normalize each bgroup to build a graph? Or show them on separate axes? Or do I need to dump CSV data and record what I want?

Answer 1

This question is definitely one of the shortcomings of Criterion. I have been bitten by the same problem several times.

The standard approach that I use for this is to simply create a separate executable for each comparison block. In recent versions of Cabal, a special benchmark target has been added, so I declare a control target for each comparison block in the .cabal file. Then I can do each comparison using cabal bench [target-name] . Yes, this is far from reassuring, but this is the best I could think of.

Answer 2

I just released the criterion plus library. This is a dome library by the “criterion” that fits the problem you are experiencing among others. It allows you to declare several “confrontations” that generate independent “criterion” report files. Another important problem that he fixes is the ability to exclude the "install / detach" phases from benchmarking, which the "criterion" does not allow.

Here is an example of how this library should be used:

 import CriterionPlus import qualified SomeMySQLLib as MySQL import qualified SomePostgreSQLLib as PostgreSQL main = benchmark $ do standoff "Inserting rows" $ do subject "MySQL" $ do -- Exclude the "setup" phase from measurement: pause connection <- liftIO $ MySQL.openConnection -- Measure what we want: continue liftIO $ MySQL.insertAThousandRows connection -- Exclude the "teardown" phase from measurement: pause liftIO $ MySQL.closeConnection connection subject "PostgreSQL" $ do -- This is how we can exclude the "setup" phase from monad transformers: pause PostgreSQL.runSession $ do lift $ continue PostgreSQL.insertAThousandRows -- Exclude "teardown": lift $ pause -- Each standoff generates an independent report file: standoff "Querying" $ do subject "MySQL" $ error "So on..." subject "PostgreSQL" $ error "So on..."