[FIXED] Gitlab job is successful despite assertion failure

Issue

I am running SoapUI assertions using maven image in gitlab. Even though the assertion fails the build is successful in gitlab. I have tried using mvn integration-tests -ff and as well as -fae but no luck. Also used allow_failure: false. This did not work either. Please advise as to how to fail the gitlab pipeline job if there is a failure in assertions.

Here is my yml file

T001-0011:
  extends: .ETE -stage
  image: adoptopenjdk/maven-openjdk11
  variables:
   MAVEN_CLI_OPTS: "--fail-fast"
  script:
  - 'mvn -f ./TV001/pom.xml $MAVEN_CLI_OPTS integration-test'
  allow_failure: false
     
  when: always

Here is the gitlab log

   1 error
    09:53:48,937 ERROR [SoapUITestCaseRunner] JDBC_Request failed, exporting to [/builds/gitlab/data/test-team-automation-scripts/./SV321/Warnings/target/surefire-reports/TestSuite_1-AC1-JDBC_Request-0-FAILED.txt]
    09:53:48,938 INFO  [SoapUITestCaseRunner] Finished running SoapUI testcase [AC1], time taken: 904ms, status: FAILED
    09:53:48,953 INFO  [SoapUITestCaseRunner] Running SoapUI testcase [AC2]
    09:53:48,963 INFO  [SoapUITestCaseRunner] running step [IDN220001-Request2]
    09:53:48,966 DEBUG [HttpClientSupport$SoapUIHttpClient] Stale connection check
    09:53:48,968 DEBUG [HttpClientSupport$SoapUIHttpClient] Attempt 1 to execute request
    09:53:48,968 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Sending request: GET /apikey/v1/warnings/waning/IDN22000 HTTP/1.1
    09:53:48,974 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 404 Not Found
    09:53:48,975 DEBUG [HttpClientSupport$SoapUIHttpClient] Connection can be kept alive indefinitely
    09:53:49,018 INFO  [log] HTTP status code: 404
    09:53:49,019 INFO  [SoapUITestCaseRunner] Assertion [Valid HTTP Status Codes] has status UNKNOWN
    09:53:49,019 INFO  [SoapUITestCaseRunner] Assertion [Script Assertion] has status VALID
    09:53:49,019 INFO  [SoapUITestCaseRunner] Finished running SoapUI testcase [AC2], time taken: 8ms, status: FINISHED
    09:53:49,021 INFO  [SoapUITestCaseRunner] Project [DPD-3396] finished with status [FAILED] in 2591ms
    SoapUI 5.3.0 TestCaseRunner Summary
    -----------------------------
    Time Taken: 2599ms
    Total TestSuites: 1
    Total TestCases: 2 (1 failed)
    Total TestSteps: 3
    Total Request Assertions: 5
    Total Failed Assertions: 1
    Total Exported Results: 3

[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:03 min
[INFO] Finished at: 2022-06-15T09:53:54+10:00
[INFO] ------------------------------------------------------------------------
Cleaning up file based variables
00:01
Job succeeded

Solution

The mvn tests fail but the exit code that the command itself returns, probably is zero.

It is saying, I managed to run the tests. But for you this doesn't help since you want to check the result of the tests.

Gitlab in order to fail a job checks the exit code of the commands used. You could force mvn to return an erroneous exit code when the tests fail

You could add the following flag

-Dmaven.test.failure.ignore=false

Answered By - Tolis Gerodimos
Answer Checked By - Mary Flores (JavaFixing Volunteer)