[FIXED] VertX EventBus not receiving messages in AWS context

Issue

I have a Java service running on 3 different ec2 instances. They form a cluster using Hazelcast. Here's part of my cluster.xml configuration:

 <join>
        <multicast enabled="false"></multicast>
        <tcp-ip enabled="false"></tcp-ip>
        <aws enabled="${AWS_ENABLED}">
            <iam-role>DEFAULT</iam-role>
            <region>us-east-1</region>
            <security-group-name>sec-group-name</security-group-name>
            <hz-port>6100-6110</hz-port>
        </aws>
</join>

Here's the log message that the discovery is successful:

[3.12.2] (This is the hazelcast version)
Members {size:3, ver:31} [
    Member [10.0.3.117]:6100 - f5a9d579-ae9c-4c3d-8126-0e8d3a1ecdb9
    Member [10.0.1.32]:6100 - 5799f451-f122-4886-92de-e351704e6980
    Member [10.0.1.193]:6100 - 626de40a-197a-446e-a44f-ac456a52d118 this
]

vertxInstance.sharedData() is working fine, meaning we can cache data between the instances.

However, the issue is when publishing messages to the instances using the vertx eventbus:

this.vertx.eventBus().publish(EventBusService.TOPIC, memberId);

and having this listener:

eventBus.consumer(TOPIC, event -> {
   logger.warn("Captured message: {}", event.body());
});

This configuration works locally, the consumer get's the messages, but once deployed to AWS it doesn't work.

I have tried setting up the host explicitly just for test, but this does not work either:

   VertxOptions options = new VertxOptions();
   options.setHAEnabled(true);
   options.getEventBusOptions().setClustered(true);
   options.getEventBusOptions().setHost("10.0.1.0"); 

What am I doing wrong and what are my options to debug this issue further?

Solution

eventbus communication does not use the cluster manager, but rather direct tcp connections

Quote from this conversation: https://groups.google.com/g/vertx/c/fCiJpQh66fk

The solution was to explicitly set the public host and port options for the eventbus:

    vertxOptions.getEventBusOptions().setClusterPublicHost(privateIpAddress);
    vertxOptions.getEventBusOptions().setClusterPublicPort(5702);

Answered By - Костадин Поп-Кочев
Answer Checked By - Willingham (JavaFixing Volunteer)