最近启动 Hadoop, nodemanger 老挂,报kerberos 验证错误,各种查找原因,时间也同步,kint 也能登录到kerberos,一直找不到原因,最后发现是网关和远端的时间同步,但是不在同一个时区导致的问题

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for single/remote.bdsm.cmcc@BDSM.CMCC to 0.0.0.0/0.0.0.0:8031; Host Details : local host is: "remote.bdsm.cmcc/10.2.41.15"; destination host is: "0.0.0.0":8031;
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:264)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for single/remote.bdsm.cmcc@BDSM.CMCC to 0.0.0.0/0.0.0.0:8031; Host Details : local host is: "remote.bdsm.cmcc/10.2.41.15"; destination host is: "0.0.0.0":8031;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy28.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy29.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:257)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:191)
... 6 more
Caused by: java.io.IOException: Couldn't setup connection for single/remote.bdsm.cmcc@BDSM.CMCC to 0.0.0.0/0.0.0.0:8031
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:672)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 19 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 22 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 31 more
Caused by: KrbException: Clock skew too great (37) - PROCESS_TGS
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
... 34 more
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66)
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
... 40 more

  问题解决:

1. 查看两台当前时区

命令 : "date -R"

2. 修改设置Linux服务器时区

命令 : "tzselect"

3. 复制相应的时区文件,替换系统时区文件;或者创建链接文件

cp /usr/share/zoneinfo/$主时区/$次时区 /etc/localtime

例如:在设置中国时区使用亚洲/上海(+8)

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

然后重启 Hadoop,问题解决

启动 nodemanger 报错javax.security.sasl.SaslException: GSS initiate failed的更多相关文章

  1. 记CM+kerberos环境停电后无法启动报错An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism leve

    公司突然停电,然后cm环境无法重启,报错 An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslExc ...

  2. spark 卡在spark context,运行出现spark Exception encountered while connecting to the server : javax.security.sasl.SaslException

    原因: 使用root用户运行spark代码 解决方法:使用非管理员账户运行spark即可 [userone@localhost bin]$ ./add-user.sh What type of use ...

  3. tomcat7 启动项目报错 java.lang.NoSuchMethodError: javax.servlet.ServletContext.getSessionCookieConfig()

    JDK版本:jdk1.8.0_77 Tomcat 版本:apache-tomcat-7.0.47 异常重现步骤: 1.完成项目部署 2.启动Tomcat 异常头部信息:java.lang.NoSuch ...

  4. 启动elasticsearch报错

    could not find java; set JAVA_HOME or ensure java is in PATH 首先需要安装java 1.yum list installed |grep j ...

  5. 抓取https网页时,报错sun.security.validator.ValidatorException: PKIX path building failed 解决办法

    抓取https网页时,报错sun.security.validator.ValidatorException: PKIX path building failed 解决办法 原因是https证书问题, ...

  6. win7启动apache-activemq报错及解决办法

    win7启动apache-activemq报错,如下: jvm 1 |jvm 1 | Java Runtime: Oracle Corporation 1.8.0_74 C:\Program File ...

  7. IDEA启动Tomcat报错

    Maven编译成功, 可是启动Tomcat报错: Application Server was not connected before run configuration stop, reason: ...

  8. maven 项目启动tomcat报错 java.lang.ClassNotFoundException: org.springframework.web.context.ContextLoaderListener

    maven项目启动tomcat报错: java.lang.ClassNotFoundException: org.springframework.web.context.ContextLoaderLi ...

  9. 【转】Eclipse下启动tomcat报错:/bin/bootstrap.jar which is referenced by the classpath, does not exist.

    转载地址:http://blog.csdn.net/jnqqls/article/details/8946964 1.错误: 在Eclipse下启动tomcat的时候,报错为:Eclipse下启动to ...

随机推荐

  1. C++ 内存泄漏

    1. 内存泄漏: 在计算机科学中,内存泄漏指由于疏忽或错误造成程序未能释放已经不再使用的内存的情况. 内存泄漏并非指内存在物理上的消失,而是应用程序分配某段内存后,由于设计错误,导致在释放该段内存之前 ...

  2. nodejs 获取指定路径下所有的文件夹名

    示例:获取 ./components 下所有的文件夹名称 let components = [] const files = fs.readdirSync('./components') files. ...

  3. Linux C 中 open close read write 使用实例

    这里实现的是将文件cody.txt中的内容拷贝到to_cody.txt中去. 1 /* ======================================================== ...

  4. JavaScript之RegExp对象

    ECMAScript 通过 RegExp 类型来支持正则表达式.使用下面类似 Perl 的语法,就可以创建一个正则表达式 var expression = / pattern / flags ; 其中 ...

  5. 在CTreeCtrl控件点击事件中获取点击的项

    网上搜了一下,有两种方法: 1.使用GetSelectedItem() HTREEITEM hItem = m_treeCtrl.GetSelectedItem(); CString strText ...

  6. Unity3d地形刷入自定义树木

    Unity3d中新建地形后,可以在地形上刷草及树木等植物.那么接下来讲的就是如何刷入自定义树木. 我没有自定义的树木素材,所以就以导入的坦克模型为例,目标是将坦克模型的炮塔部分作为自定义“树木”刷入地 ...

  7. phpexcel图形图表(一)入门

    PHPExcel - Excel的PHP处理引擎 PHPExcel 提供了一系列的 PHP语言 类,让你可以轻松地读写操作以下格式的文件:.xls/.xlsx/.csv/.ods/Gnumeric/P ...

  8. Spring Java-based容器配置(二)

    组装Java-based的配置 使用@Import注解 跟在Spring XML文件里使用<import>元素加入模块化的配置相似,@Import注解同意你载入其它配置类中的@Bean定义 ...

  9. ES6 学习笔记 (1)

    笔记来源:廖雪峰老师的javascript全栈教程 ES6:JavaScript的标准,ECMAScript在不断发展,最新版ECMAScript 6标准(简称ES6)已经在2015年6月正式发布了, ...

  10. Erlang 和 Elixir 互相调用 (转)

    lixr设计目标之一就是要确保兼容性,可以兼容Erlang和其生态系统.Elixir和Erlang 都是运行同样的虚拟机平台(Erlang Virtual Machine).不管是在Erlang使用E ...