Gui tests take too long - what's your approach?

Question

we have a typical web application stack. There are 120 selenium tests (webdriver) that run against the application. It takes about 1 hour. we run them as part of our assembly chain "compile> unit test> test> gui tests". gui tests are time consuming and we are wondering how to structure them better. they are currently "happy and unfortunate." they are quite stable, that is, they are not crashing due to programmer errors.

we want to reduce build time, and most are gui tests. we want to do this on the basis of “customer trips”, that is, to indicate (along with business people) some typical use case and test them (happy journey) instead of testing too much .....

How do you guys structure your gui tests? here are some ideas that came to my mind

• Perform only those bon voyage tests
• perform a "client test test", i.e. perform several successful path tests in one ("click on pages")
• use only the "top 10" listed in the business (mission critical)
• top 10 + "everything else" as a nightly build (once)

I would appreciate your ideas.

thanks Marcel

Answer 1

The night is the perfect time for Selenium tests - you just have to remember to put "Don't disconnect me!" sticky note on your computer :).

In addition, there is always a Selenium Grid when night time becomes too short to run all the tests. With Grid, you can run your tests on multiple machines in parallel!

We have several suite tests that are applicable to various situations. Before a major release (for testing, for preliminary work, for production), everything works. Usually (daily or even hourly during rush hours) only the package “Accelerated normal user path through the application” is launched. And if someone "fixes" a big mistake, then the tests associated with this part of the application are executed.

Answer 2

It seems to me that it seems to me that the hour.

One suggestion might be to decide which tests pass the smoke tests, and they should run every night. That is, tests that show the functionality of your main web application remain intact and work - other more detailed tests can run at different times (once every few days?).

With that said, ours takes about 2 hours - the only problem arises when one test failed, you fixed it, fixed it, but then you have to wait a very long time to verify that it is installed on the CI server.

Answer 3

TeamCity allows you to run assemblies in parallel on one computer, so gui tests should not be in the assembly chain along with unit and integration tests. User interface tests should have a separate database and a separate assembly so that they do not lose time for developers or manual testers or any other interested parties. TeamCity will collect all statistics, send emails about build failures, etc.
The next step is parallelization. As Slanec said you can use a Grid (multiple machines are not required) with Mbunit (C #) or TestNG (java). With the help of Grid, you can reduce the execution time of tests, for example. 10 times, therefore, to complete all your tests it will take only 6 (!) min.
You can also combine some of your tests into larger ones (but this will increase the time to detect the reasons for the failure and make the tests more difficult).
After these steps, Gui tests can be performed after each source commit and provide a quick response to the error state of applications.

Answer 4

Great question, great answers.

An additional consideration is that you can prioritize the 120 gui tests: you can run them in the order that the most important ones or those that are most likely to fail, run first. This will not help reduce assembly time, but it will help you get useful feedback from the assembly faster.

This prioritization (your top 10) does not have to be fixed, but can be changed for release / iteration / completed story / day, etc. For example, you can run the latest gui tests first. Or those that have been changed recently. Or those that cover most of the code that was recently modified.

There are no tools for development and completion that immediately support this, as far as I know, although there are quite a few (academic) studies in the area of ​​prioritizing the test.