Hi,

After enabled Kerberos using Ambari, I got problem creating topics in Kafka using the kafka-topics.sh script. The topic was created, but its status is wrong without leader. It seems the topic is created with PLAINTEXT, while there is only PLAINTEXTSASL broker in the cluster after enabled Kerberos. The only configuration change I made is to chagne broker listener from 'PLAINTEXT://localhost:6667' to 'PLAINTEXTSASL://localhost:6667'. As posted in this question, I also changed the kafka-topics.sh to make it work with Kerberos. I am using HDP2.5.3.

  1. $ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal --create --partitions 1 --replication-factor 1 --topic mytopic
  2. Created topic "mytopic".
  3. $ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal --describe --topic mytopic
  4. Topic:mytopic PartitionCount:1 ReplicationFactor:1 Configs:
  5. Topic: mytopic Partition: 0 Leader: none Replicas: 1001 Isr:
 
 
 0
最佳解答

个解答,截止mthiele  · 2017年01月23日 02:24

When you use a script, command, or API to create a topic, an entry is created under ZooKeeper. The only user with access to ZooKeeper is the service account running Kafka (by default, kafka). Therefore, the first step toward creating a Kafka topic on a secure cluster is to run kinit, specifying the Kafka service keytab. The second step is to create the topic.

  1. Run kinit, specifying the Kafka service keytab. For example:

    kinit -k -t /etc/security/keytabs/kafka.service.keytab kafka/c6401.ambari.apache.org@EXAMPLE.COM

  2. Next, create the topic. Run the kafka-topics.sh command-line tool with the following options:

    /bin/kafka-topics.sh --zookeeper <hostname>:<port> --create --topic <topic-name> --partitions <number-of-partitions> --replication-factor <number-of-replicating-servers>

    For example:

    1. /bin/kafka-topics.sh --zookeeper c6401.ambari.apache.org:2181 --create --topic test_topic --partitions 2 --replication-factor 2
    2. Created topic "test_topic".
 
 
 0  Show 2 · 分享
 

个解答,截止yjiang  · 2017年01月26日 10:23

@dbains@mthiele@Daniel Kozlowski,

Thank you. It works when created as kafka user.

To summarize all the steps -- from Ambari configs, to creating topic, granting permission and testing on kafka console producer/consumer scripts, I created this article Step by Step Recipe for Securing Kafka with Kerberos. Hope it saves others' time :-)

 
 
 1 · 分享
 

个解答,截止amankumbare  · 2017年03月07日 16:59

@yjiang

If you try to create a topic as a non kafka user, it creates a topic but with no Leader and ISR. This is a known issue. According to me, the reason behind this could be the zookeeper acl's. Once topic is created in zookeeper, its acl's will not allow kafka to read details about it.

If you want to create a topic as a non kafka user you need to workaround by following below steps :

If you are not using Ranger :

1. Make sure "auto.create.topic.enable = true"

2. Give acl's for the user from which you want to create a topic, for ex :

# bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic Test-topic

3. Do a kinit as a user from which you want to create topic.

4. Now try to produce messages to topic as that user :

# ./kafka-console-producer.sh --broker-list <hostname-broker>:6667 --topic Test-topic --security-protocol PLAINTEXTSASL

If you are using Ranger :

Instead of point 2 in above steps you will need to add a policy for the topic in ranger. Allow permissions for that user to produce, create, consumer. Restart kafka service. Then follow step 3 and 4 as mentioned above.

Hope this helps !!

 
 
 0 · 分享
 

个解答,截止dbains  · 2017年01月23日 18:21

@yjiang Yes, please try creating the topic as kafka user because when we use kafka-topics.sh to create a test topic, what this script does is to create a node in zookeeper path - /broker/topics/test

then brokers thread gets notified that a new node is created and broker then creates actual data for topic test that is the metadata and physical data. But notice that Brokers are kafka/host@REALM, so if a user other than kafka creates a topic it gets permission for example world:anyone:r sasl:xyz:crdwa

So the new node that is created in zookeeper path will have these permissions. Now when the broker gets alerted and tries to create metadata and physical data for this new topic, it wont be able to because broker principal is kafka but topic's is xyz

 
 
 0  Show 1 · 分享
 

个解答,截止Daniel Kozlowski  · 2017年01月23日 08:47

@yjiang

To answer your question -

---------------

When you use a script, command, or API to create a topic, an entry is created under ZooKeeper. The only user with access to ZooKeeper is the service account running Kafka (by default, kafka). Therefore, the first step toward creating a Kafka topic on a secure cluster is to runkinit, specifying the Kafka service keytab. The second step is to create the topic.

----------------

See more in here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_secure-kafka-ambari/content/ch_secure-kafka-create-topics.html

I hope this helps.

Problem of Creating Topics in Kafka with Kerberos的更多相关文章

  1. ERROR:"org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/topics/test" when creating or deleting Kafka operations authorized through the Ranger policies

    PROBLEM DESCRIPTION When creating or deleting topics in Kafka, they cannot be authorized through the ...

  2. Step by Step Recipe for Securing Kafka with Kerberos

    Short Description: Step by Step Recipe for Securing Kafka with Kerberos. Article I found it is a lit ...

  3. kafka Enabling Kerberos Authentication

    CDK 2.0 and higher Powered By Apache Kafka supports Kerberos authentication, but it is supported onl ...

  4. flume集成kafka(kafka开启kerberos)配置

    根据flume官网:当kafka涉及kerberos认证: 涉及两点配置,如下: 配置一:见下实例中红色部分 配置conf实例: [root@gz237-107 conf]# cat flume_sl ...

  5. 进行Spark,Kafka针对Kerberos相关配置

    1. 提交任务的命令 spark-submit \--class <classname> \--master yarn \--deploy-mode client \--executor- ...

  6. Kafka集成Kerberos之后如何使用生产者消费者命令

    1.生产者1.1.准备jaas.conf并添加到环境变量(使用以下方式的其中一种)1.1.1.使用Kinit方式前提是手动kinit 配置内容为: KafkaClient { com.sun.secu ...

  7. kafka实战kerberos

    more /etc/krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log a ...

  8. kafka 配置kerberos校验以及开启acl实践

    转载请注明原创地址:http://www.cnblogs.com/dongxiao-yang/p/7131626.html kafka从0.9版本以后引入了集群安全机制,由于最近需要新搭建一套kafk ...

  9. kafka Authentication using SASL/Kerberos

    Authentication using SASL/Kerberos Prerequisites KerberosIf your organization is already using a Ker ...

随机推荐

  1. July 07th. 2018, Week 27th. Saturday

    Soon is not as good as now. 别谈未来,现在就行动. From Seth Godin. I always told myself that I should finish w ...

  2. Spring Boot获取前端页面参数的几种方式总结

    Spring Boot的一个好处就是通过注解可以轻松获取前端页面的参数,之后可以将参数经过一系列处理传送到后台数据库. 获得的方式有很多种,这里稍微总结一下,大致分为以下几种: 1.指定前端url请求 ...

  3. k8s实战之从私有仓库拉取镜像 - kubernetes

    1.实战目的 从私有docker仓库拉取镜像,部署pod.上一篇中,我们搭建了私有的镜像仓库,这一篇我们将与k8s结合实战使用私有仓库. 2.登录docker 为了完成本次实战,需要登录docker, ...

  4. 使用BeanUtils类实现DTO之间的同名属性复制

    开发中经常碰到这样的场景,从数据库查询出来全部的字段,但是有些字段是不想给 客户端看到,这时就需要将属性从DAO复制到传给客户端的DTO对象,如果采用get/set, 那显得很麻烦.可使用反射实现. ...

  5. nginx在Centos7.5下源码安装和配置

    安装nginx 安装nginx依赖包 yum install -y pcre-devel zlib-devel openssl-devel wget gcc tree vim 进入目录/root/se ...

  6. vue2.0 配置环境总结(都是泪啊)

    最近有点空闲时间,终于把一直想学的vue提上了日程,以下是收集的一些帮助入门的链接 1:https://vuefe.cn/v2/guide/ vue2.0中文官网 2:https://router.v ...

  7. 读Cassandra源码之并发

    java 并发与线程池 java并发包使用Executor框架来进行线程的管理,Executor将任务的提交与执行过程分开,直接使用Runnable表示任务.future获取返回值.ExecutorS ...

  8. AlertConfirmDialog【基于AlertDialog的确认取消对话框】

    版权声明:本文为HaiyuKing原创文章,转载请注明出处! 前言 确认取消对话框,基于AlertDialog.不是基于DialogFragment. 按钮文本可以根据实际情况更换. 效果图 代码分析 ...

  9. [开发技巧]·Numpy广播机制的深入理解与应用

    [开发技巧]·Numpy广播机制的深入理解与应用 1.问题描述 我们在使用Numpy进行数据的处理时,经常会用到广播机制来简化操作,例如在所有元素都加上一个数,或者在某些纬度上作相同的操作.广播机制很 ...

  10. k8s应用机密信息与配置管理(九)--技术流ken

    secret 应用启动过程中可能需要一些敏感信息,比如访问数据库的用户名密码或者秘钥.将这些信息直接保存在容器镜像中显然不妥,Kubernetes 提供的解决方案是 Secret. Secret 会以 ...