[FIXED] hazelcast error Reason of failure for node join: Joining node's version 3.12.7 is not compatible with cluster version 3.9

Issue

i have this issue in a spring boot microservices Jhipster project, the issue started without changing any code. the microservers are deployed in a kubernetes microsoft azure enviroment. the issue is related to a hazelcast problem. Stack trace:

2021-08-05T15:36:00.921185708Z 2021-08-05 15:36:00.920 ERROR 7 --- [ration.thread-0] com.hazelcast.security                   : [10.32.0.18]:5701 [dev] [3.12.7] Node could not join cluster. Before join check failed node is going to shutdown now!
2021-08-05T15:36:00.92269528Z 2021-08-05 15:36:00.922 ERROR 7 --- [ration.thread-0] com.hazelcast.security                   : [10.32.0.18]:5701 [dev] [3.12.7] Reason of failure for node join: Joining node's version 3.12.7 is not compatible with cluster version 3.9 (Rolling Member Upgrades are only supported in Hazelcast Enterprise)
2021-08-05T15:36:00.923792359Z 2021-08-05 15:36:00.923  WARN 7 --- [ration.thread-0] com.hazelcast.instance.Node              : [10.32.0.18]:5701 [dev] [3.12.7] Terminating forcefully...
2021-08-05T15:36:00.924224151Z 2021-08-05 15:36:00.924  INFO 7 --- [ration.thread-0] com.hazelcast.instance.Node              : [10.32.0.18]:5701 [dev] [3.12.7] Shutting down connection manager...
2021-08-05T15:36:00.929472551Z 2021-08-05 15:36:00.929  INFO 7 --- [ration.thread-0] com.hazelcast.nio.tcp.TcpIpConnection    : [10.32.0.18]:5701 [dev] [3.12.7] Connection[id=1, /10.32.0.18:49367->/10.40.0.24:5701, qualifier=null, endpoint=[10.40.0.24]:5701, alive=false, type=MEMBER] closed. Reason: EndpointManager is stopping
2021-08-05T15:36:00.937643697Z 2021-08-05 15:36:00.937  INFO 7 --- [ration.thread-0] com.hazelcast.instance.Node              : [10.32.0.18]:5701 [dev] [3.12.7] Shutting down node engine...
2021-08-05T15:36:01.002501471Z 2021-08-05 15:36:01.002  INFO 7 --- [ration.thread-0] com.hazelcast.instance.NodeExtension     : [10.32.0.18]:5701 [dev] [3.12.7] Destroying node NodeExtension.
2021-08-05T15:36:01.002715567Z 2021-08-05 15:36:01.002  INFO 7 --- [ration.thread-0] com.hazelcast.instance.Node              : [10.32.0.18]:5701 [dev] [3.12.7] Hazelcast Shutdown is completed in 79 ms.
2021-08-05T15:36:04.930340712Z 2021-08-05 15:36:04.926 ERROR 7 --- [           main] com.hazelcast.instance.Node              : [10.32.0.18]:5701 [dev] [3.12.7] Could not join cluster. Shutting down now!
2021-08-05T15:36:04.930373712Z 2021-08-05 15:36:04.927  INFO 7 --- [           main] com.hazelcast.core.LifecycleService      : [10.32.0.18]:5701 [dev] [3.12.7] [10.32.0.18]:5701 is SHUTTING_DOWN
2021-08-05T15:36:04.932312175Z 2021-08-05 15:36:04.931  INFO 7 --- [           main] com.hazelcast.instance.Node              : [10.32.0.18]:5701 [dev] [3.12.7] Node is already shutting down... Waiting for shutdown process to complete...
2021-08-05T15:36:04.932877065Z 2021-08-05 15:36:04.932  INFO 7 --- [           main] com.hazelcast.core.LifecycleService      : [10.32.0.18]:5701 [dev] [3.12.7] [10.32.0.18]:5701 is SHUTDOWN
2021-08-05T15:36:05.038848261Z 2021-08-05 15:36:05.038  WARN 7 --- [           main] com.hazelcast.util.PhoneHome             : [10.32.0.18]:5701 [dev] [3.12.7] Could not schedule phone home task! Most probably Hazelcast failed to start.
2021-08-05T15:36:05.04047403Z 2021-08-05 15:36:05.039  INFO 7 --- [           main] com.hazelcast.instance.Node              : [10.32.0.18]:5701 [dev] [3.12.7] Node is already shutting down... Waiting for shutdown process to complete...
2021-08-05T15:36:05.042475292Z 2021-08-05 15:36:05.042  WARN 7 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'servletEndpointRegistrar' defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/ServletEndpointManagementContextConfiguration$WebMvcServletEndpointManagementContextConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]: Factory method 'servletEndpointRegistrar' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'healthEndpoint' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Unsatisfied dependency expressed through method 'healthEndpoint' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'healthContributorRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'hazelcastHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/hazelcast/HazelcastHealthContributorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'hazelcastHealthContributor' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [.../.../.../.../config/CacheConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
2021-08-05T15:36:05.042724688Z 2021-08-05 15:36:05.042  INFO 7 --- [           main] x.x.x.x.config.CacheConfiguration      : Closing Cache Manager
2021-08-05T15:36:05.254458785Z 2021-08-05 15:36:05.253 ERROR 7 --- [           main] o.s.boot.SpringApplication               : Application run failed
2021-08-05T15:36:05.254488484Z 
2021-08-05T15:36:05.254494684Z org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'servletEndpointRegistrar' defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/ServletEndpointManagementContextConfiguration$WebMvcServletEndpointManagementContextConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]: Factory method 'servletEndpointRegistrar' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'healthEndpoint' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Unsatisfied dependency expressed through method 'healthEndpoint' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'healthContributorRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'hazelcastHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/hazelcast/HazelcastHealthContributorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'hazelcastHealthContributor' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [.../.../.../.../config/CacheConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
2021-08-05T15:36:05.254519084Z  at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:156)
2021-08-05T15:36:05.254522883Z  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
2021-08-05T15:36:05.254526583Z  at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
2021-08-05T15:36:05.254530283Z  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
2021-08-05T15:36:05.254534083Z  at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
2021-08-05T15:36:05.254537583Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)

CacheConfiguration class

import io.github.jhipster.config.JHipsterProperties;

import com.hazelcast.config.*;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.Hazelcast;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.boot.autoconfigure.web.ServerProperties;

import org.springframework.cache.CacheManager;

import org.springframework.boot.info.BuildProperties;
import org.springframework.boot.info.GitProperties;
import org.springframework.cache.interceptor.KeyGenerator;
import org.springframework.beans.factory.annotation.Autowired;
import io.github.jhipster.config.cache.PrefixedKeyGenerator;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.discovery.DiscoveryClient;
import org.springframework.cloud.client.serviceregistry.Registration;
import org.springframework.context.annotation.*;
import org.springframework.core.env.Environment;
import org.springframework.core.env.Profiles;

import javax.annotation.PreDestroy;

@Configuration
@EnableCaching
public class CacheConfiguration {
    private GitProperties gitProperties;
    private BuildProperties buildProperties;

    private final Logger log = LoggerFactory.getLogger(CacheConfiguration.class);

    private final Environment env;

    private final ServerProperties serverProperties;

    private final DiscoveryClient discoveryClient;

    private Registration registration;

    public CacheConfiguration(Environment env, ServerProperties serverProperties, DiscoveryClient discoveryClient) {
        this.env = env;
        this.serverProperties = serverProperties;
        this.discoveryClient = discoveryClient;
    }

    @Autowired(required = false)
    public void setRegistration(Registration registration) {
        this.registration = registration;
    }

    @PreDestroy
    public void destroy() {
        log.info("Closing Cache Manager");
        Hazelcast.shutdownAll();
    }

    @Bean
    public CacheManager cacheManager(HazelcastInstance hazelcastInstance) {
        log.debug("Starting HazelcastCacheManager");
        return new com.hazelcast.spring.cache.HazelcastCacheManager(hazelcastInstance);
    }

    @Bean
    public HazelcastInstance hazelcastInstance(JHipsterProperties jHipsterProperties) {
        log.debug("Configuring Hazelcast");
        HazelcastInstance hazelCastInstance = Hazelcast.getHazelcastInstanceByName("bpm");
        if (hazelCastInstance != null) {
            log.debug("Hazelcast already initialized");
            return hazelCastInstance;
        }
        Config config = new Config();
        config.setInstanceName("bpm");
        config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
        if (this.registration == null) {
            log.warn("No discovery service is set up, Hazelcast cannot create a cluster.");
        } else {
            // The serviceId is by default the application's name,
            // see the "spring.application.name" standard Spring property
            String serviceId = registration.getServiceId();
            log.debug("Configuring Hazelcast clustering for instanceId: {}", serviceId);
            // In development, everything goes through 127.0.0.1, with a different port
            if (env.acceptsProfiles(Profiles.of(JHipsterConstants.SPRING_PROFILE_DEVELOPMENT))) {
                log.debug("Application is running with the \"dev\" profile, Hazelcast " +
                          "cluster will only work with localhost instances");

                System.setProperty("hazelcast.local.localAddress", "127.0.0.1");
                config.getNetworkConfig().setPort(serverProperties.getPort() + 5701);
                config.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(true);
                for (ServiceInstance instance : discoveryClient.getInstances(serviceId)) {
                    String clusterMember = "127.0.0.1:" + (instance.getPort() + 5701);
                    log.debug("Adding Hazelcast (dev) cluster member {}", clusterMember);
                    config.getNetworkConfig().getJoin().getTcpIpConfig().addMember(clusterMember);
                }
            } else { // Production configuration, one host per instance all using port 5701
                config.getNetworkConfig().setPort(5701);
                config.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(true);
                for (ServiceInstance instance : discoveryClient.getInstances(serviceId)) {
                    String clusterMember = instance.getHost() + ":5701";
                    log.debug("Adding Hazelcast (prod) cluster member {}", clusterMember);
                    config.getNetworkConfig().getJoin().getTcpIpConfig().addMember(clusterMember);
                }
            }
        }
        config.getMapConfigs().put("default", initializeDefaultMapConfig(jHipsterProperties));

        // Full reference is available at: https://docs.hazelcast.org/docs/management-center/3.9/manual/html/Deploying_and_Starting.html
        config.setManagementCenterConfig(initializeDefaultManagementCenterConfig(jHipsterProperties));
        config.getMapConfigs().put("xxx.xxx.xxx.xxx.domain.*", initializeDomainMapConfig(jHipsterProperties));
        return Hazelcast.newHazelcastInstance(config);
    }

    private ManagementCenterConfig initializeDefaultManagementCenterConfig(JHipsterProperties jHipsterProperties) {
        ManagementCenterConfig managementCenterConfig = new ManagementCenterConfig();
        managementCenterConfig.setEnabled(jHipsterProperties.getCache().getHazelcast().getManagementCenter().isEnabled());
        managementCenterConfig.setUrl(jHipsterProperties.getCache().getHazelcast().getManagementCenter().getUrl());
        managementCenterConfig.setUpdateInterval(jHipsterProperties.getCache().getHazelcast().getManagementCenter().getUpdateInterval());
        return managementCenterConfig;
    }

    private MapConfig initializeDefaultMapConfig(JHipsterProperties jHipsterProperties) {
        MapConfig mapConfig = new MapConfig();

        /*
        Number of backups. If 1 is set as the backup-count for example,
        then all entries of the map will be copied to another JVM for
        fail-safety. Valid numbers are 0 (no backup), 1, 2, 3.
        */
        mapConfig.setBackupCount(jHipsterProperties.getCache().getHazelcast().getBackupCount());

        /*
        Valid values are:
        NONE (no eviction),
        LRU (Least Recently Used),
        LFU (Least Frequently Used).
        NONE is the default.
        */
        mapConfig.setEvictionPolicy(EvictionPolicy.LRU);

        /*
        Maximum size of the map. When max size is reached,
        map is evicted based on the policy defined.
        Any integer between 0 and Integer.MAX_VALUE. 0 means
        Integer.MAX_VALUE. Default is 0.
        */
        mapConfig.setMaxSizeConfig(new MaxSizeConfig(0, MaxSizeConfig.MaxSizePolicy.USED_HEAP_SIZE));

        return mapConfig;
    }

    private MapConfig initializeDomainMapConfig(JHipsterProperties jHipsterProperties) {
        MapConfig mapConfig = new MapConfig();
        mapConfig.setTimeToLiveSeconds(jHipsterProperties.getCache().getHazelcast().getTimeToLiveSeconds());
        return mapConfig;
    }

    @Autowired(required = false)
    public void setGitProperties(GitProperties gitProperties) {
        this.gitProperties = gitProperties;
    }

    @Autowired(required = false)
    public void setBuildProperties(BuildProperties buildProperties) {
        this.buildProperties = buildProperties;
    }

    @Bean
    public KeyGenerator keyGenerator() {
        return new PrefixedKeyGenerator(this.gitProperties, this.buildProperties);
    }
}

pom.xml: versions:

<maven.version>3.3.9</maven.version>
  <java.version>1.8</java.version>

hazelCast dependencies:

     <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
        </dependency>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast-hibernate53</artifactId>
        </dependency>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast-spring</artifactId>
        </dependency>

please help.

Solution

Two processes with the (default) cluster name dev have found each other and attempted to cluster together. Versions are incompatible so this fails.

You could turn off discovery on both processes, but if you don't control the other this may not be viable.

Instead you could use config.getGroupConfig().setName('...') to change the name of your cluster to prevent the join attempt.

Answered By - Neil Stevenson