Work with Selenium tests that sometimes fail when deployed automatically

Question

We have a C # / ASP.Net web application that is built and deployed by a build server (Jenkins). One of the assembly steps before automatic deployment is to ensure that all automated tests pass, including the functional tests that we use with Selenium 2 WebDriver and NUnit.

Problem: Sometimes these tests are interrupted randomly. They will succeed in 100 builds, and then simply fail. They do not work for various reasons - the .Click () event is simply ignored, the element cannot be found, IE has a bad day, etc. We have a heavy AJAX web application and therefore we rely heavily on WebDriverWaits, but we always take this into account when writing tests, and, as I said, tests pass most of the time.

How can this problem be avoided or eliminated? The couple that crossed my mind:

• Accept a certain amount of crashes (seems like a bad idea)
• Re-testing errors?

Answer 1

I don't like any of the suggestions you mentioned, but I admit that I used them from time to time. The best thing to do is to make sure that when there is an apparently “random” refusal to do everything possible to get all the data on why it really failed. Is this an environmental issue? Has any other process affected the machine for tests? Was it a matter of time that only appears when a website loads painfully slowly or flashes quickly?

One thing you can try is to test your automated tests. Run each of them 100 or more times in the same environment (so you can exclude them as potential points of failure) and find those that do not work occasionally. See if they work in one place or in different places. Typically, when you go through this exercise, you will find some tests that do peel off a bit, and you can remove them from the daily run until they are fixed. You can even include as a registration criterion for any automated test case.

Another useful thing I found that helped me figure out some of the seemingly random failures was to take screenshots on failure. Often you can see other windows or dialogs appear, as a result of which browsers cannot be at the forefront, etc.

Answer 2

Of the two, I would prefer to repeat the tests, or rather, if the test fails, repeat the tests.

If you accept a certain number of test crashes, you get into problems that can cause tests to fail. You will need to have two sets of tests, some of which are allowed, and some are not.

For reuse, I'm not a testing expert with NUnit, but you could control the repetition itself. In JUnit, you can enter a rule, so if the test fails, it will retry a maximum of 3 times. This will probably avoid most of the problems that you have. I do not know how to do this in NUnit, but see my answer to How to rerun unsuccessful JUnit tests? . This will give you a general idea.