[FIXED] JDK 11 vs JDK 13 performance

Issue

UPDATE

Throughout comments it turned out that the approach for benchmark I taken was incorrect, therefore results were misleading. After correcting my approach (as in accepted answer) results are as one would expect - JDK 13 performance is just as good as with JDK 11. See the answer for more details.

Original question

I was doing some performance benchmarks on HashSet under Windows 10, using following test code for JMH:

@Benchmark
@BenchmarkMode(Mode.AverageTime)
@Fork(value = 1, warmups = 1)
public void init() {
    HashSet<String> s = new HashSet<>();
    for (int i = 0; i < 1000000; i++) {
        s.add(Math.random() + "");
    }
    s.size();
}

I compiled and run it under different JDK versions and here is what results I got:

I tested it with different heap sizes too (thus 3 different colors for each JDK). JDK 14 is of course a pre-release snapshot from today - just to see ZGC performing under Windows.

I wonder - what happened after JDK 11? (note, for JDK 12 it already starts growing, even though it's not present on the chart above)

Solution

Thank you all for suggestions in comments.

The answer was most likely Math.random() or HashSet, or missing Blackhole::consume or combination of all. I changed the test to simply do i + "aaaaaaaaa" and replaced HashSet with ArrayList pre-initialized with appropriate size to accommodate for all values to be populated. I also added Blackhole::consume at the end to exclude unwanted JIT optimizations.

After all of that, timing drops from JDK 8 to 11 gradually and then stays around the same among JDK 11-13. In JDK 14 it raises slightly, but well - it's not released yet.

Answered By - Googie
Answer Checked By - Pedro (JavaFixing Volunteer)