What is the best tool for testing performance regression

Question

Our organization is looking for a performance testing tool in every release. We are delivering a whole bunch of new software, and we want key performance features not to slow down since the latest version. We have code in C # and Java. It could be anything:

• when i run this function, it takes <2 seconds
• when I run this feature set, the sum is <5 seconds
• etc..

Is it better to do this using our basic integration testing of continuous integration (nunit, junit, team city) with manual performance checks or are there special tools that can help focus on this area.

Any suggestions?

Answer 1

In my projects (which typically use Spring), I use AOP and PerformanceMonitorInterceptor .

While you cannot use Spring, this is definitely good code to view and can base your own version. I found AOP ideal for this situation because it does not clutter up the actual function calls. If you have a multi-tier application, you can place these performance monitors at every level. (As a rule, for my web applications, I put it as the level of data access, so I can control the performance of database queries.)

Answer 2

You can try soap ui if your application is accessible remotely:

http://www.soapui.org/userguide/loadtest/index.html

This will give you all kinds of statistics that you after:

http://www.soapui.org/userguide/loadtest/images/loadtest_editor.gif

Answer 3

We use python to write scripts + advanced logging to generate XML logs, which can then be imported into a spreadsheet.

Answer 4

I fiddled with a tool called “Basher” that allows you to write “tasks” that are selected by the system and then run for a custom time (so that, for example, the JVM can warm up) and then runs, run time of the recording job, average values etc.

Version 1.0 was quiet, and if you want to take a look, it is available at http://basher.sourceforge.net

The trunk version contains some improvements - there is maven integration, with the settings specified in pom.xml, the bare basis of the report structure, etc.