Exposing Kafka from Rancher/K8S VM to Local Machine

Following on from my previous post on setting up Rancher/K8S on RancherOS in a VM in Windows for local development, a common task will be setting up container services within the cluster but then accessing those services from your local Windows machine (e.g. while developing in Visual Studio). In a lot of cases this is probably straightforward, either exposing ports directly using a service or using Ingress to route host headers to the correct internal service. However in the case of Kafka it's a bit more complex due to the way in which the brokers address themselves when the initial connection is received and the broker list is sent back. In a nutshell, the default Kafka setup from the Catalog Apps in Rancher binds the brokers to their POD IP, when the broker list is sent to Windows it cannot address these IPs (unless you want to set up some kind of natting). After some Googling and help from the following posts: https://rmoff.net/2018/08/02/kafka-listeners-explained/ https://github.com/helm/charts/issues/6670 I came up with the following instructions: STEP 1 (install Kafka in cluster): Install Kafka from the Rancher catalogue

1. your-dev-cluster > default > Catalog Apps > Launch
2. find and select "Kafka"
3. switch off the "Topics UI Layer 7 Loadbalancer" (near the bottom) - don't need it in dev.
4. click "Launch"
5. .. Wait until all the kafka services are running ..
6. You can now verify that the Landoop UI is running and seeing brokers by visiting the endpoint is has produced, e.g. http://rancherdev.yourdomain:30188 <-- random port, check what it says!!

Kafka is now available in the cluster, but not from Windows. Continue with step 2 --> STEP 2 (expose Kafka externally): Change the Kafka startup command for multiport listening

1. your-dev-cluster > default > workloads > kafka-kafka
2. Three dots, click "Edit"
3. Click "show advanced options"
4. Under Command > Entrypoint - paste the following:

   sh -exc 'export KAFKA_BROKER_ID=${HOSTNAME##*-} && \export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_IP}:9092,EXT://rancherdev.yourdomain.com:$((9093 + ${KAFKA_BROKER_ID})) && \export KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXT:PLAINTEXT && \export KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT && \exec /etc/confluent/docker/run'

5. Click "Upgrade"

Add service discovery for the new ports

1. your-dev-cluster > default > Service Discovery
2. Click "View/Edit YAML" on kafka-kafka..
3. Use the following lines for section "spec > ports" (assuming you have 3 instances of Kafka)

   ports:
     - name: broker
       port: 9092
       protocol: TCP
       targetPort: 9092
     - name: broker-ext0
       port: 9093
       protocol: TCP
       targetPort: 9093
     - name: broker-ext1
       port: 9094
       protocol: TCP
       targetPort: 9094
     - name: broker-ext2
       port: 9095
       protocol: TCP
       targetPort: 9095
   

Configure nginx to use TCP ConfigMap

1. your-dev-cluster > system > workloads > nginx-ingress-controller
2. Three dots > edit
3. Environment variables:
4. "Add from Source" > "Config Map" > "tcp-services"
5. Click "Upgrade"

Expose the port using Ingress TCP ConfigMap

1. your-dev-cluster > system > resources > config maps > ns: ingress-nginx > tcp-services
2. Three dots, click "Edit"
3. Add the following entries:

           - key = 9093
           - value = kafka/kafka-kafka:9093
           - key = 9094
           - value = kafka/kafka-kafka:9094
           - key = 9095
           - value = kafka/kafka-kafka:9095
   

Reboot the kafka services

1. your-dev-cluster > default > workloads > tick all and click 'redeploy'

Now from Windows try telnet to rancherdev.yourdomain.com 9093/9094/9095 or even better from WSL bash, install kafkacat and run: kafkacat -b rancherdev.yourdomain.com:9093 -L