[FIXED] Scenario Outline: Using Whitelists & Ranges?

Issue

I have been building up a Cucumber automation framework and have a number of components to test. I have used Scenario Outline's to capture various values and their expected responses.

What I see as a problem: I have to specify every single type of input of data and the error message to go with it. From the example Scenario Outline below you can see I have certain numbers that all are expected to return the one message. If anything does not equal these values the return an error message:

Scenario Outline: Number is or is not valid
    Given I send an event with the "Number" set to <num>
    Then I will receive the following <message>
    Examples:
        | num    | message           |                                                                                     
        | 0      | "Processed event" |                                                        
        | 1      | "Processed event" |                                                                  
        | 2      | "Processed event" |                                                                 
        | 3      | "Processed event" |                                                                 
        | 4      | "Processed event" |                                                                 
        | 5      | "Processed event" |                                                                                                                                 
        | 6      | "Message failed"  |
        | -1     | "Message failed"  |
        | "One"  | "Message failed"  |

What I would like to do: I would basically like to have a "whitelist" of good data defined in the Scenario Outline and if there is any other value input - it returns the the expected error message. Like the following:

Scenario Outline: Number is or is not valid
    Given I send an event with the "Number" set to <num>
    Then I will receive the following <message>
    Examples:
        | num                |  message          |                                                                                     
        | 0-5                | "Processed event" |                                                        
        | Anything else      | "Message failed"  |  

Is the following possible with the code behind it? As you can see it would have benefits of making an automation suite far more concise and maintainable. If so please let me know, keen to discuss.

Thanks!

Kirsty

Solution

Cucumber is a tool to support BDD. This means that it works really well when you have to communicate about behavior. But this particular problem is going towards validating the properties of the event validator, i.e. property based testing. So it might be worth to split the test strategy accordingly.

It appears there is a rule that valid events results are processed and invalid events are rejected. This is something you could test with Cucumber. For example:

Feature: Events
    
    This system accepts events. Events are json messages.
    Examples of well known valid and invalid json messages
    can be found in the supplementary documentation.

    Scenario: The system accepts valid events
      When a well known valid event is send
      Then the system accepts the event
      And responds with "Processed event"

    Scenario: The system rejects invalid events
      When a well known invalid event is send
      Then the system rejects the event
      And responds with "Message failed"

It also appears there is a rule that valid events have a field "Number" set to any value between 0-5. And since sounds like a json object I'm guessing the strings "0", "1", "2", "3", "4", "5" are also valid. Anything else is invalid.

A good way to test this exhaustively is by using property based testing framework. For example JQwik. Given a description of a set of either valid or invalid values it will randomly try a few. For a simplified example:

package my.example.project;

import net.jqwik.api.*;

import static org.assertj.core.api.Assertions.assertThat;

class ValidatorProperties {

    @Provide
    Arbitrary<Object> validValues() {
        Arbitrary<Integer> validNumbers = Arbitraries.integers().between(0, 5);
        Arbitrary<String> validStrings = validNumbers.map(Object::toString);
        return Arbitraries.oneOf(validNumbers, validStrings);
    }

    @Provide
    Arbitrary<Object> invalidValues() {
        Arbitrary<Integer> invalidNumbers = Arbitraries.oneOf(
                Arbitraries.integers().lessOrEqual(-1),
                Arbitraries.integers().greaterOrEqual(6)
        );
        Arbitrary<String> invalidStrings = invalidNumbers.map(Object::toString);
        return Arbitraries.oneOf(
                invalidNumbers, 
                invalidStrings, 
                Arbitraries.just(null)
        );
    }

    @Property
    void accepts0To5(@ForAll("validValues") Object value) {
        Validator validator = new Validator();
        assertThat(validator.isValid(value)).isTrue();
    }

    @Property
    void rejectsAnythingElse(@ForAll("invalidValues") Object value) {
        Validator validator = new Validator();
        assertThat(validator.isValid(value)).isFalse();
    }

    static class Validator {

        boolean isValid(Object event) {
            return event != null && event.toString().matches("^[0-5]$");
        }

    }

}

Split this way the Cucumber tests describe how the system should respond to valid and invalid events. While the JQwik test describe what the properties of a valid and invalid event are. This allows much more clarity on the first and a greater fidelity on the second.

Answered By - M.P. Korstanje
Answer Checked By - Katrina (JavaFixing Volunteer)