How to check overall performance while optimizing the entire module

Question

At WWDC 2015, Session 409 is at around 18 minutes. The discussion leads me to the idea that generic tools can be optimized using universal specialization by turning on the optimization mode of the entire module. Unfortunately, my tests, which I'm not sure of, did not show anything useful.

I did some simple tests between the following two methods to see if the performance was the same:

func genericMax<T : Comparable>(x:T, y:T) -> T { return y > x ? y : x } func intMax(x:Int, y:Int) -> Int { return y > x ? y : x } 

Simple XCTest:

 func testPerformanceExample() { self.measureBlock { let x: Int = Int(arc4random_uniform(9999)) let y: Int = Int(arc4random_uniform(9999)) for _ in 0...1000000 { // let _ = genericMax(x, y: y) let _ = intMax(x, y: y) } } } 

What happened

Without optimization, the following tests were reasonably different:

• genericMax: 0.018 s
• intMax: 0.005 sec

However, when optimizing the entire module, the following tests were not similar:

• genericMax: 0.014 s
• intMax: 0.004 sec

What i expected

With module optimization turned on, I expected similar times between two method calls. It makes me think my test is flawed.

Question

Assuming my tests are wrong / poor. How could I better measure how the optimization mode of the entire module is optimized using common specialization?

Answer 1

Your tests are wrong because they measure the performance of the test, not the performance of the application. Tests live in a separate executable file, and they do not benefit from the optimization of the entire module. Because of this, your test always uses a common non-specialized implementation, even in places where your program is not.

If you want to see that the optimization of the whole module is included in your executable file, you need to check the function from your executable file. (You also need to make sure that your tests either use the Release build, or that you turned on WMO in the debug build.)

Adding this function to the executable file:

 func genericIntMax(x: Int, y: Int) -> Int { return genericMax(x, y: y) } 

and using it in tests instead of genericMax in tests, I get identical performance. (Note that this is not a whole module optimization, since the two functions live in the same file, but it also shows the difference between the application code and the test code when it comes to optimization.)