Performance considerations when using Apache Kafka with SSL/TLS

January 11, 2017
by Emanuel Muhr

Running a cluster of a distributed software such as Apache Kafka in a production environment will lead to operational concerns. One of these concerns is certainly security. Companies have varying guidelines and policies when it comes to security. This depends on various factors for example the type of application data that the system is processing or in which network infrastructure the system is running and what security concerns this zone implies on the servers that are running within.

A probable scenario is that such a cluster is run in an environment which dictates restrictiveness in regards to access control and protocol security. Access control means that identified clients communicate with the cluster only. When looking at protocol security the company or project might have a guideline that clients must use a secure protocol like SSL/TLS. Furthermore, communication among the cluster servers itself might also have to be secured.

Securing the communication with Apache Kafka is well documented (Confluent, 2016)

In short, a key store has to be generated for each machine in the cluster:

keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 –genkey

Generate a Certificate authority:

openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

Add the CA to the trust store:

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert

Export the certificate from the trust store:

keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file

Then sign it:

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days  
{validity} -CAcreateserial -passin pass:{ca-password}

Import the certificates:

keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert  
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed

Then for each broker the listener needs to be changed in the server.properties:

listeners=SSL://hostname:port

And the following configuration directives are necessary for enabling SSL in server.properties:

ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=test1234

The producers and consumers must use certificates accordingly.

Adding SSL/TLS to Kafka will have an impact on the system. The type and degree of impact differs by certain factors. Since encryption is done by the CPU of the servers this is of course one of them. Other factors include the type and implementation of the encryption algorithm used and the runtime environment.

In order to get a picture on the impact a test configuration was chosen which roughly corresponds to a setup which is run by a customer.

This setup is described here:

Running a performance test with this cluster without SSL yields the following results:

In comparison active SSL/TLS gives the following results:

The results show that in this specific environment activating SSL/TLS gives a throughput penalty of approx. 30%.  It is also clear that the maximum latency increases. This is not surprising given the fact that there are three servers and each message is replicated on all brokers. In this test the option for acknowledgements was set to all meaning that the broker will wait until all in-sync replicas are acknowledged. This is a setting which might be important in some scenarios.

While running the performance test the CPU was running at approx. 80% with SSL/TLS enabled. This could hint at the CPU as a limiting factor in this configuration and that by adding more cores the throughput could be increased.

If securing the Kafka network is a set requirement the implications on performance should be evaluated for each use case. Given these results it is clear that encryption has a negative effect on application scenarios where throughput is important.

An alternative for using SSL could be an infrastructure where the server instances are isolated in a separate network segment. In this case the access to this segment would be tightly controlled using for example firewalls.

Investigation showed that Kafka currently uses JDK's SSL engine and there is currently a pending ticket for Kafka to include OpenSSL ((Kafka, 2016)) which promises to be faster than the JDK implementation. Additionally, future JDKs might increase the performance further.

About the author: Emanuel Muhr
Comments
Join us
Are you creative and passionate about software development? Do you think unconventionally and act with initiative? Do you want to achieve great things within our team?