How to choose between different types of tests with SpecFlow, Cucumber or another base for testing BDD acceptance?

Question

I am looking at SpecFlow examples, and this MVC sample contains several alternatives for testing:

• Acceptance tests based on verification of the results generated by the controllers;
• Integration Tests Using MvcIntegrationTestFramework;
• Automated acceptance testing using Selenium;
• Manual acceptance tests when the test is asked to manually check the results.

I have to say that I am very impressed with how well the SpecFlow examples are written (and I managed to run them within a few minutes after loading, I just had to configure the database and install the Selenium Remote Control server). Looking at the test alternatives, I see that most of them are complementary, not an alternative. I can recall the following combinations of these tests:

• The controllers are tested in the TDD style, and not using SpecFlow (I believe that this / when / then the type of tests should be applied at a higher level from end to end, they should provide good code coverage for the corresponding components;
• MvcIntegrationTestFramework is useful when performing integration tests during development sessions; these tests are also part of daily builds;
• Although Selenium-based tests are automated, they are slow and mostly run during QA sessions to quickly verify that the pages and workflow of the site have no broken logic;
• Manual acceptance tests, when the tester is asked to confirm the accuracy of the result, mainly to check the appearance of the page.

If you use SpecFlow, Cucumber or other BDD acceptance testing framework in your web development, can you share your practices regarding the choice between different types of tests.

Thanks in advance.

Answer 1

All this behavior.

Given the specific context when an event occurs (within a certain area), then some result should occur.

A scope can be an entire application, part of a system, or a single class. Even a function behaves this way: input as a context and result as a result (you can also use BDD for a functional language!)

I use Unit modules (NUnit, JUnit, RSpec, etc.) at the class or integration level, because the audience is technical. Sometimes I document this / when / then in the comments.

At the script level, I'm trying to figure out who really wants to help read or write scripts. Even business stakeholders can read text containing several dots and brackets, so the main reason for having a natural language environment, such as MSpec or JBehave, is that they want to write scripts themselves or show them to people who will really be put off with dots and brackets.

After that, I will consider how the infrastructure will work with the build system, and how we will provide the opportunity to read or write to relevant stakeholders.

Here is an example I wrote to show what you can do with scripts using simple DSLs. It is just written in NUnit.

Here is an example in the same code base in which the data is shown, when, then in the comments at the class level.

I abstract the steps and then put screens or pages on them, then on the screens and pages I call any automatic structure that I use - these can be Selenium, Watir, WebRat, Microsoft UI Automation, etc.

The example I have given is itself an automation tool, so the scripts demonstrate the behavior of an automation tool, showing the behavior of a fake gui, in case this is confusing. Hope this helps!

Answer 2

Since acceptance tests are a kind of functional test, the overall goal is to test your application with them to the end. On the other hand, you may need to consider the effectiveness (as necessary for test automation), maintainability, performance, and reliability of test automation. It is also important that test automation can easily fit into the development process, so that it supports a kind of “test-first” approach (for support outside the development process).

So, this is a compromise that may be different for each situation (why we provided alternatives).

I am sure that today the most suitable option is testing at the controller level. (Perhaps later, when the framework for automating the user interface and interface will evolve, this will change.)