[FIXED] Find elements in one RDD but not in ther other RDD

Issue

I have two JavaRDD A and B. I want to only keep longs that are in A but not in B. How should I do that? Thanks!

Solution

I am posting a solution in scala. Should be almost similar in Java.

Do a leftOuterJoin which would give all the records in the first rdd alongwith matching records from the second rdd. Like WrappedArray((168,(def,None)), (192,(abc,Some(abc)))). But to keep the record only present in first rdd, we apply a filter over None.

val data = spark.sparkContext.parallelize(Seq((192, "abc"),(168, "def")))
val data2 = spark.sparkContext.parallelize(Seq((192, "abc")))

val result = data
.leftOuterJoin(data2)
.filter(record => record._2._2 == None)

println(result.collect.toSeq)
Output> WrappedArray((168,(def,None)))

Answered By - Hema Jayachandran
Answer Checked By - Candace Johnson (JavaFixing Volunteer)