How to test a Ruby command line program with Cucumber that has a large amount of output?

Question

I am creating a Ruby command-line program and using Cucumber and Aruba to test it. Aruba includes some really convenient matches, so I can check the output with several lines in the .feature file:

 When I run `myprogram` Then it should pass with: """ my program output """ 

The problem is that my program may contain dozens or even hundreds of lines of output; anything in the .feature file .feature reading and navigating difficult (and this is disgusting). What is the recommended way to check the output in this case?

Answer 1

Short answer: you should not do this.

Cucumber tests should be user-friendly and readable. They describe features. The user does not care about whether the error result matches some known byte of the value for the byte.

You need to ask yourself: what am I testing? Respond to an error message? Probably no. You are testing some features in your application. If all you really want is to make sure that this fails, then what you want in your Cucumber script is the following line:

 Then the exit status should not be 0 

This assumes that the script follows standard conditions, according to which a non-zero exit status signals an error.

If your script requires a specific message to be output, you can add it:

 Then it should fail with """ Some error message """ 

but this is not necessarily the whole conclusion, but only a partial coincidence. (Note that “it should crash for sure:” is defined in Aruba, but I do not recommend using it.)

Change You changed your example to testing, not failure, but my basic tips are the same:

• Separate output specificity from script logic. Using the example in the comments, if you have a test that confirms that you can display one user-created comment, and another test that confirms that you will output the correct 100 comments, then this is enough, you do not need to have 100 comments for the release in the script Cucumber.
• Save Cucumber scripts written from a user perspective. Each script should check what is important to the user. Try to keep them minimal by removing everything that leaks from the implementation when the user doesn't care.
• Use Aruba's built-in constructs that check for partial matches to achieve this. Look for keywords or phrases in the output. Not only are Cucumber tests easier to read, they will be more reliable and immune to unrelated release changes.

Answer 2

To check your program, you have 2 options:

• Check only the relevant part of the message (what you really want to check with this test). Aruba has a built-in stepdef so it is very easy

• Check the full message. If the message is short, you can use Aruba's built-in stepdef . However, if the message is long, you can put it in a separate file. Since Aruba does not contain such a method, you must write this stepdef yourself.

It might look like this:

 # Require aruba/api before that Then /^it should (pass|fail) with message from file "(.*)"$/ do |pass_fail, filename| exact_output = File.read(filename) Aruba::API::assert_exit_status_and_output(pass_fail == "pass", exact_output, true) end 

If you want to write many tests and many messages are similar, you can get many WET messsages. Approving a complete message will make complex tests more difficult in case of changes.

Thus, you can use some kind of template engine to validate these messages to run DRYer tests.

You can design your tests in many different ways.
If you use only the first method, then your tests may not detect errors in the regression, since they do not check everything. If you use the second method, it can be much more difficult to maintain them if there are many repetitions between messages.

Thus, there is a trade-off between the two options. Usually you create some tests using method 1, others with method 2. You should think about what will be better in your case. I do not know the golden rule.

Answer 3

Hundreds of issues in the app are not surprising. And yes, you need to make them easy to read. If you cannot do this in cucumber files, I am afraid there is no better place to replace. Cucumber, after all, is plain English and acts like live documentation.

All you need to do is organize them. Here are some basic tips if you neglect them.

• Write a specific function that covers only one topic for the .feature file .feature . Although I know that your mentioned When I run my program is for demonstration purposes, such a broad topic is not acceptable.

• Put the .feature files in the features/ folder. And, if necessary, divide them into subfolders, for example features/user/user_login.feature .

• Use Scenario Outline and Examples to organize similar outputs in one scenario. Examples will contain lines to clearly represent each of them.

I hope for this help.

Answer 4

I would recommend the following:

 When I run `my_app` Then the exit status should not be 0 And the output should contain: """ some output """ 

Thus, while “some kind of output” is being output from your program, the test passes, but the additional output is ignored, and your test does not have to have all this output.

If you want to approve all this in one step, just make your own, although I think this is more understandable.