[FIXED] Unable to access the Cassandra from inside the Kubernetes cluster

Issue

I have a Cassandra cluster and a spring boot application in my Kubernetes cluster. They are in the same (default) namespace. The spring boot application needs to connect to Cassandra, but it's unable to do that. During the connection attempt, spring boot application receives the exception below:

Suppressed: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: cassandra/10.111.117.185:32532 Caused by: java.net.ConnectException: Connection refused at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.netty.channel.StacklessClosedChannelException: null

Cassandra yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  type: NodePort
  ports:
  - port: 9042
    targetPort: 9042
    protocol: TCP
    nodePort: 32532
  selector:
    app: cassandra
---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: cassandra:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8SCassandra"
          - name: CASSANDRA_DC
            value: "DC1-K8SCassandra"
          - name: CASSANDRA_RACK
            value: "Rack1-K8SCassandra"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: standard
      resources:
        requests:
          storage: 1Gi

Spring boot application yaml:

apiVersion: v1
kind: Service
metadata:
  name: service-cassandraapp
  labels:
     app: cassandraapp
spec:
  selector:
    app: cassandraapp
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 32588
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-cassandraapp
  labels:
    app: cassandraapp
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: cassandraapp
  template:
    metadata:
      labels:
        app: cassandraapp
    spec:
      containers:
      - name: cassandraapp
        image: ek/cassandraapp:latest
        ports:
          - containerPort: 8080
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests:
            memory: "256Mi"
            cpu: "500m"
        env:
          - name: CONFIG_CASSANDRA_HOST
            value: "cassandra"
          - name: CONFIG_CASSANDRA_PORT
            value: "32532"  

Spring boot application.properties:

spring.data.cassandra.local-datacenter=datacenter1
spring.data.cassandra.keyspace-name=testkeyspace
spring.data.cassandra.port=${CONFIG_CASSANDRA_PORT}
spring.data.cassandra.contact-points=${CONFIG_CASSANDRA_HOST}
spring.data.cassandra.username=cassandra
spring.data.cassandra.password=cassandra
spring.data.cassandra.schema-action=CREATE_IF_NOT_EXISTS

When I check the Cassandra pods, they are at running state. But the spring boot application gets refused. Any help would be greatly appreciated.

Solution

I solved the problem. I assumed that spring.data.cassandra.schema-action=CREATE_IF_NOT_EXISTS would create the key space but it wouldn't. So I created it manually.

My updated springboot application yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-cassandra-cassandraapp
data:
  cassandra-host: "cassandra.default.svc.cluster.local"
  cassandra-port: "9042"
  cassandra-keyspace: "testkeyspace"
  cassandra-datacenter: "datacenter1"
---
apiVersion: v1
kind: Service
metadata:
  name: service-cassandraapp
  labels:
     app: cassandraapp
spec:
  selector:
    app: cassandraapp
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-cassandraapp
  labels:
    app: cassandraapp
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: cassandraapp
  template:
    metadata:
      labels:
        app: cassandraapp
    spec:
      containers:
      - name: cassandraapp
        image: ek/cassandraapp:latest
        ports:
          - containerPort: 8080
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests:
            memory: "256Mi"
            cpu: "500m"
        env:
          - name: CONFIG_CASSANDRA_HOST
            valueFrom:
              configMapKeyRef:
                name: configmap-cassandra-cassandraapp
                key: cassandra-host
          - name: CONFIG_CASSANDRA_PORT
            valueFrom:
              configMapKeyRef:
                name: configmap-cassandra-cassandraapp
                key: cassandra-port
          - name: CONFIG_CASSANDRA_KEYSPACE
            valueFrom:
              configMapKeyRef:
                name: configmap-cassandra-cassandraapp
                key: cassandra-keyspace
          - name: CONFIG_CASSANDRA_DATACENTER
            valueFrom:
              configMapKeyRef:
                name: configmap-cassandra-cassandraapp
                key: cassandra-datacenter

Springboot application.properties

spring.data.cassandra.local-datacenter=${CONFIG_CASSANDRA_DATACENTER}
spring.data.cassandra.keyspace-name=${CONFIG_CASSANDRA_KEYSPACE}
spring.data.cassandra.port=${CONFIG_CASSANDRA_PORT}
spring.data.cassandra.contact-points=${CONFIG_CASSANDRA_HOST}

And lastly I also tried Bitnami Helm Chart. And I recommend it too. [https://bitnami.com/stack/cassandra/helm][1]

Answered By - eftmk
Answer Checked By - David Goodson (JavaFixing Volunteer)