Issue
I am new to Kubernetes and I am deploying an application for the first time on Kubernetes. I want to deploy a postgreSQL statefulset and a simple replicaset of spring boot pods. I created a headless service that will be attached to the following statefulset.
# Headless Service
apiVersion: v1
kind: Service
metadata:
name: ecom-db-h
spec:
ports:
- targetPort: 5432
port: 80
clusterIP: None
selector:
app: ecom-db
---
# PostgreSQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ecomdb-statefulset
labels:
app: ecom-db
spec:
serviceName: ecom-db-h
selector:
matchLabels:
app: postgresql-db
replicas: 2
template:
# Pod Definition
metadata:
labels:
app: postgresql-db
spec:
containers:
- name: db
image: postgres:13
ports:
- name: postgres
containerPort: 5432
# volumeMounts:
# - name: ecom-db
# mountPath: /var/lib/postgresql/data
env:
- name: POSTGRES_DB
value: ecommerce
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD
value: <password>
#volumes:
# - name: ecom-db
# persistentVolumeClaim:
# claimName: ecom-pvc
And here is the replicaset I created for the spring boot application pods :
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: ecom-server
labels:
app: ecom-server
tier: backend
spec:
# modify replicas according to your case
replicas: 2
selector:
matchLabels:
type: backend
template:
# Pod definition
metadata:
labels:
app: ecom-server
type: backend
spec:
containers:
- name: ecommerce-server
image: <my.private.repo.url>/spring-server:kubernetes-test
imagePullPolicy: Always
imagePullSecrets:
- name: regcred
I am using the default namespace and here is the application.properties:
spring.datasource.url=jdbc:postgresql://ecom-db-h.default.svc.cluster.local/ecommerce?useSSL=false
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.username=postgres
spring.datasource.password=<password>
spring.jpa.hibernate.ddl-auto=update
spring.jpa.hibernate.format_sql=true
spring.jpa.hibernate.id.new_generator_mappings=true
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL95Dialect
But my spring boot containers always exit with the following error :
Caused by: java.net.UnknownHostException: ecom-db-h.default.svc.cluster.local
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
at org.postgresql.core.PGStream.<init>(PGStream.java:95)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
... 136 common frames omitted
This means that either the replicaset is not able to find the headless service, or my headless service is not connected to the statefulset pods at all. Did I miss something in the YAML files ? Or am I using the headless service in the wrong way ?
Update:
Thanks to @Thomas and @harsh-manvar I understood that the headless service is not meant to connect to the databases. Instead, I should use a normal ClusterIP service. The problem is I did that and I am still getting the same error. Here is my new YAML files :
application.properties modification:
spring.datasource.url=jdbc:postgresql://db-svc.default.svc.cluster.local/ecommerce?useSSL=false
statefulset.yaml modified
# ClusterIP service instead of the headless service
apiVersion: v1
kind: Service
metadata:
name: db-svc
spec:
ports:
- name: pgql
port: 80
targetPort: 5432
protocol: TCP
selector:
app: db
---
# PostgreSQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db-sts
labels:
app: db-sts
spec:
serviceName: db-svc
selector:
matchLabels:
app: db
replicas: 2
template:
# Pod Definition
metadata:
labels:
app: db
spec:
containers:
- name: db
image: postgres:13
ports:
- name: postgres
containerPort: 5432
# volumeMounts:
# - name: ecom-db
# mountPath: /var/lib/postgresql/data
env:
- name: POSTGRES_DB
value: ecommerce
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD
value: mss#123
#volumes:
# - name: ecom-db
# persistentVolumeClaim:
# claimName: ecom-pvc
Here is the new error :
Caused by: java.net.UnknownHostException: db-svc.default.svc.cluster.local
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
at org.postgresql.core.PGStream.<init>(PGStream.java:95)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
... 136 common frames omitted
My statefulset pods are running smoothly and the service is created :
NAME READY AGE
statefulset.apps/db-sts 2/2 26m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/db-svc ClusterIP 10.108.39.189 <none> 5432/TCP 26m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 30m
NAME READY STATUS RESTARTS AGE
db-sts-0 1/1 Running 0 26m
db-sts-1 1/1 Running 0 26m
ecom-server-5jbs2 0/1 CrashLoopBackOff 9 (44s ago) 25m
ecom-server-rtpmg 0/1 CrashLoopBackOff 8 (4m46s ago) 25m
NAME ENDPOINTS AGE
db-svc 10.244.1.48:5432,10.244.2.85:5432 18h
Second Update :
After running an alpine-shell container in a pod to check on the DNS resolving, I performed an "nslookup db-svc.default.svc.cluster.local" and I got this :
nslookup db-svc.default.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: db-svc.default.svc.cluster.local
Address: 10.107.101.52
which means the service does exist and it can be reached from this container. What's strange is when I performed the pgsql command to try to connect to the postgres database server, I got no response at all and the command did not even exit. Here is the error
WARN SqlExceptionHelper - SQL Error: 0, SQLState: 08001
ERROR SqlExceptionHelper - The connection attempt failed.
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:315)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
at org.postgresql.Driver.makeConnection(Driver.java:465)
at org.postgresql.Driver.connect(Driver.java:264)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180)
at org.hibernate.resource.transaction.backend.jdbc.internal.DdlTransactionIsolatorNonJtaImpl.getIsolatedConnection(DdlTransactionIsolatorNonJtaImpl.java:43)
... 122 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
at org.postgresql.core.PGStream.<init>(PGStream.java:95)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
... 136 common frames omitted
Solution
Finally I found the solution :
Original Post: First I need to use the ClusterIP service and not the headless service.
Solution to Update : It was the firewall blocking the requests between the pods inside my serves. That is why I was getting the error SQLState: 08001. Disabling the firewall solved the problem. Also I needed to specify the port in the application.properties. Because I got used to not specify the port 80 with HTTP connections, I thought that was the case for the postgres connection to the database, but I needed to specify the port :
spring.datasource.url=jdbc:postgresql://db-svc.default.svc.cluster.local:80/ecommerce?useSSL=false
It would be a better practise to change the service port to the default postgres server port: 5432
Answered By - joe1531
