[FIXED] How to compile spark-testing-base in Java project built with maven?

Issue

I'm don't have a lot of experience with Java, but I built a Spark application using Java. I want to write some unit tests for my Spark application. I saw that spark-testing-base is very useful for that purpose. I have added the following to my pom.xml:

<dependency>
    <groupId>com.holdenkarau</groupId>
    <artifactId>spark-testing-base_2.12</artifactId>
    <version>2.4.5_1.0.0</version>
    <scope>test</scope>
</dependency>

I'm using Junit framework and my tests fail when trying to reach jsc(). My current test (Not much logic yet):

public class SparkTest extends SharedJavaSparkContext {

 @Test
 public void verifyMapTest() {
     // Create and run the test
     List<String> input = Arrays.asList("1\tHeart", "2\tDiamonds");
     JavaRDD<String> inputRDD = jsc().parallelize(input);
 }
}

I used the following maven command to build the Jar: maven clean install

It seems like I didn't really compile spark-testing-base. jsc() returns null and I'm getting null pointer exception.

Solution

I figured out the issue for my case.I was using Junit 5 for my test(org.junit.jupiter.api.Test) .And the lib was referring to "org.junit.Test" .If we move to junit4 It will work properly. Raised an issue with the lib ->https://github.com/holdenk/spark-testing-base/issues/340 .

Hope you also had the same issue.

Answered By - aksankar