Run Apache Kafka on Kubernetes in Minutes with Strimzi
Strimzi is an open-source project that provides a simple way to run an Apache Kafka cluster on Kubernetes. With Strimzi, you can quickly set up a Kafka cluster and start producing and consuming messages, all within your local Kubernetes environment.
![Run Apache Kafka on Kubernetes in Minutes with Strimzi](/content/images/size/w600/2024/06/strimizi-kafka-kubernetes.png)
Key Features of Strimzi
- Secure by Default: Strimzi includes built-in security features like TLS, SCRAM-SHA, and OAuth authentication, as well as automated certificate management.
- Simple yet Configurable: Strimzi offers flexible deployment options, including NodePort, Load Balancer, and Ingress, as well as features like rack awareness for high availability.
- Kubernetes Native Experience: Strimzi uses a Kubernetes Operator to manage the entire Kafka lifecycle, allowing you to use familiar kubectl commands to deploy and manage your Kafka clusters.
What is Strimzi?
Strimzi is an open-source project that provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. It abstracts away the underlying infrastructure, allowing developers and operations teams to focus on the applications themselves rather than worrying about the intricacies of the hosting environment.
One of the key benefits of Strimzi is its ability to automate the deployment, scaling, and management of Kafka clusters. By using Kubernetes Custom Resources, Strimzi makes it easy to create, configure, and maintain Kafka clusters, topics, and users, all within your Kubernetes environment.
Strimzi also includes built-in security features, such as TLS encryption, SCRAM-SHA authentication, and OAuth support, ensuring that your Kafka cluster is secure by default. Additionally, Strimzi provides options for exposing your Kafka cluster outside of Kubernetes, including NodePort, Load Balancer, and Ingress, making it easy to integrate with other applications and services.
Whether you're running Kafka on a local Minikube cluster, a Kubernetes Kind environment, or a production-grade Kubernetes deployment, Strimzi makes it simple to get started and manage your Kafka infrastructure.
Getting Help
If you encounter any issues while using Strimzi, you can get help in the following ways:
- Send an email to the Strimzi Mailing list
- Ask your question on the Strimzi Slack Workspace
Quickstarts
Minikube
Minikube provides a local Kubernetes, designed to make it easy to learn and develop for Kubernetes. The Kubernetes cluster is started either inside a virtual machine, a container or on bare-metal, depending on the minikube driver you choose.
Installing the dependencies
This quickstart assumes that you have the latest version of the minikube
binary, which you can get from the minikube website.
Minikube requires a container or virtual machine manager. The Minikube documentation includes a list of suggested options in the getting started guide.
You'll also need the kubectl
binary, which you can get by following the kubectl
installation instructions from the Kubernetes website.
Once you have all the binaries installed, make sure everything works:
# Validate minikube
minikube version
# Validate kubectl
kubectl version
Starting the Kubernetes cluster
Start a local development cluster of Minikube that runs in a container or virtual machine manager.
minikube start --memory=4096 # 2GB default memory isn't always enough
Deploy Strimzi using installation files
Before deploying the Strimzi cluster operator, create a namespace called kafka
:
kubectl create namespace kafka
Apply the Strimzi install files, including ClusterRoles
, ClusterRoleBindings
and some Custom Resource Definitions (CRDs
). The CRDs define the schemas used for the custom resources (CRs, such as Kafka
, KafkaTopic
and so on) you will be using to manage Kafka clusters, topics and users.
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
The YAML files for ClusterRoles
and ClusterRoleBindings
downloaded from strimzi.io contain a default namespace of myproject
. The query parameter namespace=kafka
updates these files to use kafka
instead. By specifying -n kafka
when running kubectl create
, the definitions and configurations without a namespace reference are also installed in the kafka
namespace. If there is a mismatch between namespaces, then the Strimzi cluster operator will not have the necessary permissions to perform its operations.
Follow the deployment of the Strimzi cluster operator:
kubectl get pod -n kafka --watch
You can also follow the operator's log:
kubectl logs deployment/strimzi-cluster-operator -n kafka -f
Once the operator is running it will watch for new custom resources and create the Kafka cluster, topics or users that correspond to those custom resources.
Create an Apache Kafka cluster
Create a new Kafka custom resource to get a single node Apache Kafka cluster:
# Apply the `Kafka` Cluster CR file
kubectl apply -f https://strimzi.io/examples/latest/kafka/kraft/kafka-single-node.yaml -n kafka
Wait while Kubernetes starts the required pods, services, and so on:
kubectl wait kafka/my-cluster --for=condition=Ready --timeout=300s -n kafka
The above command might timeout if you're downloading images over a slow connection. If that happens you can always run it again.
Send and receive messages
With the cluster running, run a simple producer to send messages to a Kafka topic (the topic is automatically created):
kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.41.0-kafka-3.7.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic
Once everything is set up correctly, you'll see a prompt where you can type in your messages:
If you don't see a command prompt, try pressing enter.
>Hello Strimzi!
And to receive them in a different terminal, run:
kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.41.0-kafka-3.7.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning
If everything works as expected, you'll be able to see the message you produced in the previous step:
If you don't see a command prompt, try pressing enter.
>Hello Strimzi!
Enjoy your Apache Kafka cluster, running on Minikube!
Deleting your Apache Kafka cluster
When you are finished with your Apache Kafka cluster, you can delete it by running:
kubectl -n kafka delete $(kubectl get strimzi -o name -n kafka)
This will remove all Strimzi custom resources, including the Apache Kafka cluster and any KafkaTopic custom resources but leave the Strimzi cluster operator running so that it can respond to new Kafka custom resources.
Next, delete the Persistent Volume Claim (PVC) that was used by the cluster:
kubectl delete pvc -l strimzi.io/name=my-cluster-kafka -n kafka
Without deleting the PVC, the next Kafka cluster you might start will fail as it will try to use the volume that belonged to the previous Apache Kafka cluster.
Deleting the Strimzi cluster operator
When you want to fully remove the Strimzi cluster operator and associated definitions, you can run:
kubectl -n kafka delete -f 'https://strimzi.io/install/latest?namespace=kafka'
Deleting the kafka
namespace
Once it is not used, you can also delete the Kubernetes namespace:
kubectl delete namespace kafka
Where next?
- For an overview of the Strimzi components check out the overview guide.
- For alternative examples of the custom resource that defines the Kafka cluster have a look at these examples
Comment using your social account:
You will be asked to grant read-only access to your public profile and email address only to verify your identity. We will never post to your account. Select your preferred social account to get started.
|Service provided by Spectral Web Services.