[FIXED] Running file-based tests on Junit 5 Platform using Apache Maven Surefire

Issue

I have implemented a custom TestEngine class that is able to discover and run unit tests from *.csv files (not java classes) by checking for FileSelector in the EngineDiscoveryRequest from Junit 5.

I am able to run these tests without problems using the ConsoleLauncher provided by the Junit team, by invoking org.junit.platform.console.ConsoleLauncher -f build/resources/test/testfile_1.csv (this is invoked from a gradle task because gradle does not support resource-based testing but that should not matter here).

Now, I am trying to run the same tests using the same custom test engine but by using maven and the surefire plugin instead of the console launcher. My current pom.xml looks as follows (relevant part):

<project>
    <dependencies>
        <dependency>
            <groupId>my_custom_test_engine</groupId>
            <artifactId>custom_test_engine</artifactId>
            <version>1.0-SNAPSHOT</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-engine</artifactId>
            <version>1.7.0-M1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-console</artifactId>
            <version>1.7.0-M1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.30</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
            </plugin>
        </plugins>
    </build>
</project>

Unfortunately, surefire:test does not appear to even call my test engine for discovery and just finishes with:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.626 s
[INFO] Finished at: 2020-06-20T17:00:01+02:00
[INFO] ------------------------------------------------------------------------

I do not have any test classes in my src folder, just one test file under resources/testfile_1.csv.

Does surefire even support file-based testing with Junit 5? If so, how can I force Surefire to call my custom test engine for test discovery?

-- Update

The discovery part of my TestEngine implementation looks something like this (only relevant part shown):

public class MyTestEngine implements TestEngine {

    @Override
    public TestDescriptor discover(EngineDiscoveryRequest discoveryRequest, UniqueId uniqueId) {
        EngineDescriptor engineDescriptor = new EngineDescriptor(uniqueId, name);

        logger.debug("Discovering tests");

        discoveryRequest.getSelectorsByType(FileSelector.class).forEach(selector -> {
            logger.debug("Discovered: {}", selector.getFile().getAbsolutePath());
                engineDescriptor.addChild(new TestDescriptor (engineDescriptor.getUniqueId(), selector.getFile().getName()));

        });
}

-- Update 2 My test files are located under src/test/resources.

Solution

Figured it out. Credit goes to @johanneslink for pointing me in the right direction.

Surefire indeed does not support resource-based testing (see also this issue). What I did instead (as @johanneslink advised), was to create an empty class, which can be detected by surefire so that the custom TestEngine is invoked:

public class TestInitializer {
}

Then, in my TestEngine, I check for ClassSelector and discover the resource-based tests:

discoveryRequest.getSelectorsByType(ClassSelector.class).forEach(selector -> {
            if (selector.getClassName().equals(TestInitializer.class.getName())) {
                discoverTests(new File(System.getenv("test_dir")), engineDescriptor);
            }
        });

The actual directory where my test files are located is specified with the environment variable test_dir.

Then, for actually using the custom TestEngine, modify your pom.xml as follows:

   <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
                <configuration>
                    <dependenciesToScan>
                        <dependency>
                            my_custom_test_engine:custom_test_engine
                        </dependency>
                    </dependenciesToScan>
                    <includes>
                        <include>**/TestInitializer*
                        </include>
                    </includes>
                    <environmentVariables>
                        <test_dir>target/test-classes
                        </test_dir>
                    </environmentVariables>
                </configuration>
            </plugin>
        </plugins>
    </build>

The dependenciesToScan property will ensure, that surefire scans the dependency containing the TestInitializer class.

Note that the latest version 3.0.0-M5 of maven-surefire-plugin appears to be broken in this regard as the tests will be discovered and run, but surefire will not pick up on the results. Use 2.22.2 instead.

Answered By - mlxyz
Answer Checked By - Timothy Miller (JavaFixing Admin)