Step by Step Recipe for Securing Kafka with Kerberos

Short Description:

Step by Step Recipe for Securing Kafka with Kerberos.

Article

I found it is a little tricky to get started with a Kerberos enabled Kafka cluster. I created this step by step recipe for securing Kafka with Kerberos, sending and receiving data on console. This is tested on HDP2.5.0 and Ambari 2.4.1.

• Enabled Kerberos using the Ambari Kerberos setup wizard under Admin -- Kerberos menu.
• On Ambari Kafka Config UI, change "listeners" property to "PLAINTEXTSASL://localhost:6667".
• Restart Kafka as requested by Ambari.
• Create a test topic in Kafka. Must use the kafka service user to do this.

　　

$ cd /usr/hdp/current/kafka-broker/bin
$ sudo su kafka
$ kinit -k -t /etc/security/keytabs/kafka.service.keytab kafka/ip----.ap-northeast-.compute.internal

$ ./kafka-topics.sh --zookeeper ip----.ap-northeast-.compute.internal: --create --topic foo --partitions  --replication-factor 

Created topic "bar".

• Grant permission to user. This can be done using Kafka native ACL mechanism or Apache Ranger. In the example, we use Kafka ACL. User bob needs to be existing in KDC.

  # Grant user bob as producer on topic foo
  ./kafka-acls.sh --authorizer-properties zookeeper.connect=ip----.ap-northeast-.compute.internal: \
    --add --allow-principal User:bob \
    --producer --topic foo
  
  Adding ACLs for resource `Topic:foo`:
    User:bob has Allow permission for operations: Describe from hosts: *
    User:bob has Allow permission for operations: Write from hosts: *
  
  Adding ACLs for resource `Cluster:kafka-cluster`:
    User:bob has Allow permission for operations: Create from hosts: *
  
  Current ACLs for resource `Topic:foo`:
    User:bob has Allow permission for operations: Describe from hosts: *
    User:bob has Allow permission for operations: Write from hosts: *
  
  # Grant user bob as consumer
  ./kafka-acls.sh --authorizer-properties zookeeper.connect=ip----.ap-northeast-.compute.internal: \
    --add --allow-principal User:bob \
    --consumer --topic foo --group *
  
  #--group后等跟等号，如--group=*。按照上面写法，只是赋给名称为connect-distributed.sh的groupid。因为kafka/bin下面第一个脚本是connect-distributed.sh

  
  Adding ACLs for resource `Topic:foo`:
    User:bob has Allow permission for operations: Read from hosts: *
    User:bob has Allow permission for operations: Describe from hosts: *
  
  Adding ACLs for resource `Group:connect-distributed.sh`:
    User:bob has Allow permission for operations: Read from hosts: *
  
  Current ACLs for resource `Topic:foo`:
    User:bob has Allow permission for operations: Read from hosts: *
    User:bob has Allow permission for operations: Describe from hosts: *
    User:bob has Allow permission for operations: Write from hosts: *
  
  Current ACLs for resource `Group:connect-distributed.sh`:
    User:bob has Allow permission for operations: Read from hosts: *

• Confirm the above works using the kafka console producer and consumer scripts.

  # Switch to bob user and log in to KDC.
  $ kinit bob
  
  # Start console producer
  $ ./kafka-console-producer.sh --broker-list ip----.ap-northeast-.compute.internal: --topic foo --security-protocol PLAINTEXTSASL
  
  # On another terminal, start console consumer
  ./kafka-console-consumer.sh --zookeeper ip----.ap-northeast-.compute.internal: --topic foo --security-protocol PLAINTEXTSASL 
  
  {metadata.broker.list=ip----.ap-northeast-.compute.internal:, request.timeout.ms=, client.id=console-consumer-, security.protocol=PLAINTEXTSASL}
  
  # Type something on the producer terminal, it should appears on the console terminal immediately.