What is a systematic approach to debugging intermittently specs?

Question

I have four tests in my Capybara / Rspec suite that continue to fail (a real problem for deploying CI).

Worst of all, these tests are interrupted intermittently, and often only when the entire package is launched, which makes debugging difficult.

All of them are ajax requests, either send a remote form, or are deleted by a remote link, and then expect(page).to have_content 'My Flash Message' .

These tests are even interrupted intermittently during the same test cycle. For example, I have several models that behave the same, so I repeat them for testing.

 eg, ['Country', 'State', 'City'].each do |object| let(:target) { create object.to_sym } it 'runs my frustrating test' do end end 

Sometimes a country fails, sometimes they say, sometimes it all goes away.

I tried adding wait: 30 to the wait statement. I tried adding sleep 30 before the expect statement. I still get intermittent passages.

There is quite a lot of information describing subtle ajax tests, but I have not found much about how to debug and fix such problems.

I am very grateful for any advice or pointers from others before pulling all my hair out!

UPDATE

Thanks for all these excellent answers. It was helpful to see that others were facing similar problems and that I was not alone.

So is there a solution?

Recommendations for using debugging tools, such as the prug, byebug, Poltergeist debug functions (thanks @ Jay-Ar Polidario, @TomWalpole), were useful for confirming what I thought I already knew, namely, and how it was @ BM5K suggested), the functions work sequentially in the browser, and the errors lie in the tests.

I experimented with setting timeouts and retries (@ Jay-Ar Polidario, @ BM5K), and although the improvement was not yet a consistent solution. More importantly, this approach was similar to fixing holes, rather than correct fixing, and therefore I was not completely satisfied.

I ended up going with a census of these tests. This entailed the destruction of multi-stage functions and the setup and testing of each step individually. Although purists may argue that this is not really testing from the user's point of view, there is sufficient agreement between each test, which is convenient for me with the result.

After going through this process, I noticed that all these errors were related to “clicking on things or filling out forms,” as @BoraMa suggested. Although the experience was canceled in this case, we adopted the .trigger('click') syntax because capybara + poltergeist reported errors when clicking on elements using click_link or find(object).click , and these tests were problematic .

To avoid these problems, I removed JS from the tests as much as possible. that is, testing most functions without JS, and then creating very short target JS specifications to test for specific JS responses, functions, or user feedback.

Thus, in fact, there is not a single fix. Great refactoring, which, frankly, probably should have happened, was a valuable exercise. Tests lost some functions, breaking everything down into separate tests, but overall it made reading and maintaining tests easier.

There are several more tests that sometimes show red, and this will require another work. But overall a big improvement.

Thank you all for the great leadership and assured me that interaction in a test environment can be the main reason.

Answer 1

Periodically unsuccessful tests are a pain for troubleshooting, but there are some things you can do to make life easier. First, it would remove any circular or common examples. By explicitly stating that each expectation should make it clearer which combination of combinations fails (or make it even more obvious that it is truly random).

Over the course of a few runs, keep track of which tests fail. Are they all in the same context group?

Do you mix and match javascript tests and tests other than javascript? If so, you may run into database problems (I saw problems caused by enabling the context block for database cleanup environments).

Make sure you think that any parent context blocks the tests.

And if none of this narrows your search, use a gem that lets you repeat failed tests.

I used respec-retry in the past, but recently found it unreliable. I switched to rspec-repeat . I usually leave them in development (configured for 1 attempt) and start with several attempts on CI (usually 3). This way, I can feel which tests are unstable locally, but don't let these tests destroy my assembly (if they don't work sequentially).

TL DR

In most intermittent failure tests that I encounter, there are many moving parts (rails, capybara, database cleaner, factory girl, phantomjs, rspec, to name a few). If the code is tested and often passes specifications, and the function works consistently in the browser, some interaction in your test environment may be the main cause of intermittent crashes. If you cannot track this, repeat unsuccessful specifications several times.

Answer 2

Let me tell a story too :). Recently, we also tried to find and fix problems with our periodic errors with a similar setup (Poltergeist, JS tests). Testing turned out to be more likely if the entire test package was launched separately, but in about one third of the time the entire package failed. This is just a couple of tests from the set, about 10, which accidentally failed, others seemed to work fine all the time.

At first, we made sure that the tests did not fail due to db truncation problems, remaining records, etc. We took screenshots at the time the page validation failed.

After repeated searches, we noticed that all the remaining failed tests relate to clicking on things or filling out forms , while jQuery animations and other dynamic operations are often used on pages. This led us to this Poltergeist question, which helped us a lot in the end. It turns out that Poltergeist, by clicking on a button or dealing with form inputs, tries to imitate the normal user as much as possible, which can lead to problems when animating inputs / links.

The way to admit that this was indeed a problem for us was that we could successfully find to create an element on the page, but the browser could not click on it.

We ended up using a not-so-clean solution - we rewrote capybara's helpers for clicking and interacting with forms to use find and trigger inside:

 # override capybara methods as they react badly with animations # (click/action is not registered then and test fails) # see https://github.com/teampoltergeist/poltergeist/issues/530 def click_button(locator, *options) find_button(locator, *options).trigger(:click) end def click_link(locator, *options) find_link(locator, *options).trigger(:click) end def choose(locator, *options) find(:radio_button, locator, *options).trigger(:click) end def check(locator, *options) find(:checkbox, locator, *options).trigger(:click) end 

This approach can lead to some unexpected problems, because now you can click on things in your tests, even if they are, for example. overlap with a modal div or when they are not fully visible on the page. But, having carefully read the comments on the github problem, we decided that this is the way for us.

Since then, we only have very random test failures that seem to be related to another poltergeist issue. But failures are so rare that we do not feel like looking further - tests are finally reliable enough.

Answer 3

If you are sure that the variable does not change on the server side (Rails) and client (JS). You can try the following if it works. We used this for some similar problem that we had.

Specification / Support / wait_for_ajax.rb

 # ref: https://robots.thoughtbot.com/automatically-wait-for-ajax-with-capybara module WaitForAjax def wait_for_ajax Timeout.timeout(Capybara.default_max_wait_time) do loop until finished_all_ajax_requests? end sleep(1) # ensure just because above doesn't always work end def finished_all_ajax_requests? page.evaluate_script('jQuery.active').zero? end end 

Specifications / Features / YOUR_SPEC.rb

 Rspec.feature 'My Feature Test', type: :feature do ['Country', 'State', 'City'].each do |object| let(:target) { create object.to_sym } it 'runs my frustrating test' do find('#my-div').click wait_for_ajax end end end 

rails_helper.rb

 # .. RSpec.configure do |config| # .. config.include WaitForAjax, type: :feature # .. end # ..