Issue
The context is that I am working on a Kubernetes project, where we use a Geode cluster, and Spring Boot, also Spring Boot Data Geode (SBDG). We have developed an app with it, a ClientCache. Also we have a proprietary internal mechanism to generate cluster-internal certificates, this mechanism automatically renews certificates according to the best practices. We convert the PEM formatted certificate in our App code to JKS, and configured Spring with @EnableSSL annotation to take them.
So the issue is, that everything works wonderfully for the first cycle, when the connections were created with the JKS file the App initially started up with, however if the certificate is renewed, say hourly (in cloud this is best practice), Geode fails to be connected with a bunch of Exceptions, sometimes SSLException (readHandshakeRecord), many times with "Unable to connect to any locators in the list" (but I debugged, and it is also a HandshakeException, just wrapper in a connection-exception instead). The locators and servers are up and running (I checked with GFSH), just the App I think tries to connect with the old SSLContext and fails in the SSL handshake.
The only way so far I have found is to restart the App completely, but we would need this system to be automatic, and highly available, so this should not be the only way around this issue.
I think this problem is affecting a lot of Spring/Java projects as I have found this issue all around (Kafka, PGSQL, etc...).
Do any of you have any method to do this? Is there a way to:
- Recreate all the connections without me restarting the App?
- Invalidate the currently used connections somehow, and force the ClientCache to create new ones, re-reading the JKS file?
- Maybe let the Client App timeout the connections and destroy them, and create new ones, with refreshed SSLContext?
I did not find any possibilities for this.
EDIT: Let me add some code, to show how we do things, since we use Spring, it is dead simple:
@Configuration
@EnableGemfireRepositories(basePackages = "...")
@EnableEntityDefinedRegions(basePackages = "...")
@ClientCacheApplication
@EnableSsl(
truststore = "truststore.jks",
keystore = "keystore.jks",
truststorePassword = "pwd",
keystorePassword = "pwd"
)
public class GeodeTls {}
And that is it! We then use normal annotations for @Regions, and @Repositories, and we have our @RestControllers where we call the repository methods, most of them are just empty ones, or default ones as we use the OQL annotate method to do things with Spring. Since Geode has a property-based config, we never set KeyStores, TrustStores, I just happen to see them inside the code during debugging.
EDIT2: I have finally solved thanks to the below comments, it was this Geode ticket that helped a lot (thanks to Jen D for that): https://github.com/apache/geode/pull/2244, available since Geode 1.8.0. Also the below snippet was extremely useful about the Swappable KeyManager (thanks for Hakan54), I did a combined solution in the end. I had to be careful though, to set the default SSLContext only once as the subsequent sets were ineffective, and did not result in any failures. Now the App is stable it seems across certificate changes.
Solution
I came across your question yesterday and was working on a prototype. I think it might be possible in your case. However I just tried it out locally with a http client and a server which I was able to change the certificates at runtime without the need of restarting these applications or recreating the SSLContext.
Option 1
From your question I can understand that you are reading PEM files from somewhere and converting it to something else and at the end you are using a SSLContext. In that case I would assume you are creating a KeyManager and a TrustManager. If thats the case what you need to do is create a custom implementation of the KeyManager and TrustManager as a wrapper class to delegate the method calls to the actual KeyManager and TrustManager within the wrapper class. And also add a setter method to change the internal KeyManager and TrustManager when the certificates get updated.
In your case that would be a file-watcher which gets triggered when the PEM files have been changed. In that case you only need to regenerate the KeyManager and TrustManager with the new certificates and give it to the wrapped class by calling the setter method. Below is an example code snippet what you could use:
HotSwappableX509ExtendedKeyManager
import javax.net.ssl.SSLEngine;
import javax.net.ssl.X509ExtendedKeyManager;
import java.net.Socket;
import java.security.Principal;
import java.security.PrivateKey;
import java.security.cert.X509Certificate;
import java.util.Objects;
public final class HotSwappableX509ExtendedKeyManager extends X509ExtendedKeyManager {
private X509ExtendedKeyManager keyManager;
public HotSwappableX509ExtendedKeyManager(X509ExtendedKeyManager keyManager) {
this.keyManager = Objects.requireNonNull(keyManager);
}
@Override
public String chooseClientAlias(String[] keyType, Principal[] issuers, Socket socket) {
return keyManager.chooseClientAlias(keyType, issuers, socket);
}
@Override
public String chooseEngineClientAlias(String[] keyTypes, Principal[] issuers, SSLEngine sslEngine) {
return keyManager.chooseEngineClientAlias(keyTypes, issuers, sslEngine);
}
@Override
public String chooseServerAlias(String keyType, Principal[] issuers, Socket socket) {
return keyManager.chooseServerAlias(keyType, issuers, socket);
}
@Override
public String chooseEngineServerAlias(String keyType, Principal[] issuers, SSLEngine sslEngine) {
return keyManager.chooseEngineServerAlias(keyType, issuers, sslEngine);
}
@Override
public PrivateKey getPrivateKey(String alias) {
return keyManager.getPrivateKey(alias);
}
@Override
public X509Certificate[] getCertificateChain(String alias) {
return keyManager.getCertificateChain(alias);
}
@Override
public String[] getClientAliases(String keyType, Principal[] issuers) {
return keyManager.getClientAliases(keyType, issuers);
}
@Override
public String[] getServerAliases(String keyType, Principal[] issuers) {
return keyManager.getServerAliases(keyType, issuers);
}
public void setKeyManager(X509ExtendedKeyManager keyManager) {
this.keyManager = Objects.requireNonNull(keyManager);
}
}
HotSwappableX509ExtendedTrustManager
import javax.net.ssl.SSLEngine;
import javax.net.ssl.X509ExtendedTrustManager;
import java.net.Socket;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Arrays;
import java.util.Objects;
public class HotSwappableX509ExtendedTrustManager extends X509ExtendedTrustManager {
private X509ExtendedTrustManager trustManager;
public HotSwappableX509ExtendedTrustManager(X509ExtendedTrustManager trustManager) {
this.trustManager = Objects.requireNonNull(trustManager);
}
@Override
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
trustManager.checkClientTrusted(chain, authType);
}
@Override
public void checkClientTrusted(X509Certificate[] chain, String authType, Socket socket) throws CertificateException {
trustManager.checkClientTrusted(chain, authType, socket);
}
@Override
public void checkClientTrusted(X509Certificate[] chain, String authType, SSLEngine sslEngine) throws CertificateException {
trustManager.checkClientTrusted(chain, authType, sslEngine);
}
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
trustManager.checkServerTrusted(chain, authType);
}
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType, Socket socket) throws CertificateException {
trustManager.checkServerTrusted(chain, authType, socket);
}
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType, SSLEngine sslEngine) throws CertificateException {
trustManager.checkServerTrusted(chain, authType, sslEngine);
}
@Override
public X509Certificate[] getAcceptedIssuers() {
X509Certificate[] acceptedIssuers = trustManager.getAcceptedIssuers();
return Arrays.copyOf(acceptedIssuers, acceptedIssuers.length);
}
public void setTrustManager(X509ExtendedTrustManager trustManager) {
this.trustManager = Objects.requireNonNull(trustManager);
}
}
Usage
// Your key and trust manager created from the pem files
X509ExtendedKeyManager aKeyManager = ...
X509ExtendedTrustManager aTrustManager = ...
// Wrapping it into your hot swappable key and trust manager
HotSwappableX509ExtendedKeyManager swappableKeyManager = new HotSwappableX509ExtendedKeyManager(aKeyManager);
HotSwappableX509ExtendedTrustManager swappableTrustManager = new HotSwappableX509ExtendedTrustManager(aTrustManager);
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(new KeyManager[]{ swappableKeyManager }, new TrustManager[]{ swappableTrustManager })
// Give the sslContext instance to your server or client
// After some time change the KeyManager and TrustManager with the following snippet:
X509ExtendedKeyManager anotherKeyManager = ... // Created from the new pem files
X509ExtendedTrustManager anotherTrustManager = ... // Created from the new pem files
// Set your new key and trust manager into your swappable managers
swappableKeyManager.setKeyManager(anotherKeyManager)
swappableTrustManager.setTrustManager(anotherTrustManager)
So even when your SSLContext instance is cached in your server of client you can still swap in and out new keymanager and trustmanager.
The code snippets are available here:
Option 2
If you don't want to add the custom (HotSwappableKeyManager and HotSwappableTrustManager) code to your code base you can also use my library:
<dependency>
<groupId>io.github.hakky54</groupId>
<artifactId>sslcontext-kickstart</artifactId>
<version>7.4.5</version>
</dependency>
Usage
SSLFactory baseSslFactory = SSLFactory.builder()
.withDummyIdentityMaterial()
.withDummyTrustMaterial()
.withSwappableIdentityMaterial()
.withSwappableTrustMaterial()
.build();
SSLContext sslContext = sslFactory.getSslContext();
Runnable sslUpdater = () -> {
SSLFactory updatedSslFactory = SSLFactory.builder()
.withIdentityMaterial(Paths.get("/path/to/your/identity.jks"), "password".toCharArray())
.withTrustMaterial(Paths.get("/path/to/your/truststore.jks"), "password".toCharArray())
.build();
SSLFactoryUtils.reload(baseSslFactory, updatedSslFactory);
};
// initial update of ssl material to replace the dummies
sslUpdater.run();
// update ssl material every hour
Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(sslUpdater, 1, 1, TimeUnit.HOURS);
Update #1 - example with pem files In the comments someone requested an example with pem files, so below is an example of refreshing the ssl configuration with pem files:
First make sure you have the following library:
<dependency>
<groupId>io.github.hakky54</groupId>
<artifactId>sslcontext-kickstart-for-pem</artifactId>
<version>7.4.5</version>
</dependency>
And the code example:
SSLFactory baseSslFactory = SSLFactory.builder()
.withDummyIdentityMaterial()
.withDummyTrustMaterial()
.withSwappableIdentityMaterial()
.withSwappableTrustMaterial()
.build();
SSLContext sslContext = sslFactory.getSslContext();
Runnable sslUpdater = () -> {
X509ExtendedKeyManager keyManager = PemUtils.loadIdentityMaterial(Paths.get("/path/to/your/certificate-chain.pem"), Paths.get("/path/to/your/"private-key.pem"));
X509ExtendedTrustManager trustManager = PemUtils.loadTrustMaterial(Paths.get("/path/to/your/"some-trusted-certificate.pem"));
SSLFactory updatedSslFactory = SSLFactory.builder()
.withIdentityMaterial(keyManager)
.withTrustMaterial(trustManager)
.build();
SSLFactoryUtils.reload(baseSslFactory, updatedSslFactory);
};
// initial update of ssl material to replace the dummies
sslUpdater.run();
// update ssl material every hour
Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(sslUpdater, 1, 1, TimeUnit.HOURS);
Answered By - Hakan54
Answer Checked By - Robin (JavaFixing Admin)
