Flink Heartbeat of TaskManager和Heartbeat of ResourceManager timed out问题
最近上了个Flink任务,运行一段时间后就自动停止了,很是郁闷,查看最后一个chekpoint时间点,翻看时间日志
2019-12-13 07:25:24.566 flink [flink-akka.actor.default-dispatcher-41] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job PayOrder (88c9cc0c85875332cc5e4ed6418cd667) switched from state RUNNING to FAILING.java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_1566481621886_4397244_01_000004 timed out.
at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1656)
at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:339)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:154)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-12-13 07:25:24.519 flink [flink-akka.actor.default-dispatcher-20] INFO org.apache.flink.runtime.jobmaster.JobMaster - Close ResourceManager connection 9b7931812dbed76060b48a696d72a869: The heartbeat of ResourceManager with id 9b7931812dbed76060b48a696d72a869 timed out..
根据Heartbeat of TaskManager with id和The heartbeat of ResourceManager with id在源码中找出这样的代码
private class TaskManagerHeartbeatListener implements HeartbeatListener<AccumulatorReport, Void> { private final JobMasterGateway jobMasterGateway; private TaskManagerHeartbeatListener(JobMasterGateway jobMasterGateway) {
this.jobMasterGateway = Preconditions.checkNotNull(jobMasterGateway);
} @Override
public void notifyHeartbeatTimeout(ResourceID resourceID) {
jobMasterGateway.disconnectTaskManager(
resourceID,
new TimeoutException("Heartbeat of TaskManager with id " + resourceID + " timed out."));
} @Override
public void reportPayload(ResourceID resourceID, AccumulatorReport payload) {
for (AccumulatorSnapshot snapshot : payload.getAccumulatorSnapshots()) {
schedulerNG.updateAccumulators(snapshot);
}
} @Override
public CompletableFuture<Void> retrievePayload(ResourceID resourceID) {
return CompletableFuture.completedFuture(null);
}
} private class ResourceManagerHeartbeatListener implements HeartbeatListener<Void, Void> { @Override
public void notifyHeartbeatTimeout(final ResourceID resourceId) {
runAsync(() -> {
log.info("The heartbeat of ResourceManager with id {} timed out.", resourceId); if (establishedResourceManagerConnection != null && establishedResourceManagerConnection.getResourceManagerResourceID().equals(resourceId)) {
reconnectToResourceManager(
new JobMasterException(
String.format("The heartbeat of ResourceManager with id %s timed out.", resourceId)));
}
});
} @Override
public void reportPayload(ResourceID resourceID, Void payload) {
// nothing to do since the payload is of type Void
} @Override
public CompletableFuture<Void> retrievePayload(ResourceID resourceID) {
return CompletableFuture.completedFuture(null);
}
}
然后在这实例化
this.taskManagerHeartbeatManager = heartbeatServices.createHeartbeatManagerSender(resourceId,new TaskManagerHeartbeatListener(selfGateway),rpcService.getScheduledExecutor(),log);
顺着去heartbeatServices瞅瞅了
/**
* HeartbeatServices gives access to all services needed for heartbeating. This includes the
* creation of heartbeat receivers and heartbeat senders.
*/
public class HeartbeatServices { /** Heartbeat interval for the created services. */
protected final long heartbeatInterval; /** Heartbeat timeout for the created services. */
protected final long heartbeatTimeout; public HeartbeatServices(long heartbeatInterval, long heartbeatTimeout) {
Preconditions.checkArgument(0L < heartbeatInterval, "The heartbeat interval must be larger than 0.");
Preconditions.checkArgument(heartbeatInterval <= heartbeatTimeout, "The heartbeat timeout should be larger or equal than the heartbeat interval."); this.heartbeatInterval = heartbeatInterval;
this.heartbeatTimeout = heartbeatTimeout;
} /**
* Creates a heartbeat manager which does not actively send heartbeats.
*
* @param resourceId Resource Id which identifies the owner of the heartbeat manager
* @param heartbeatListener Listener which will be notified upon heartbeat timeouts for registered
* targets
* @param scheduledExecutor Scheduled executor to be used for scheduling heartbeat timeouts
* @param log Logger to be used for the logging
* @param <I> Type of the incoming payload
* @param <O> Type of the outgoing payload
* @return A new HeartbeatManager instance
*/
public <I, O> HeartbeatManager<I, O> createHeartbeatManager(
ResourceID resourceId,
HeartbeatListener<I, O> heartbeatListener,
ScheduledExecutor scheduledExecutor,
Logger log) { return new HeartbeatManagerImpl<>(
heartbeatTimeout,
resourceId,
heartbeatListener,
scheduledExecutor,
scheduledExecutor,
log);
} /**
* Creates a heartbeat manager which actively sends heartbeats to monitoring targets.
*
* @param resourceId Resource Id which identifies the owner of the heartbeat manager
* @param heartbeatListener Listener which will be notified upon heartbeat timeouts for registered
* targets
* @param scheduledExecutor Scheduled executor to be used for scheduling heartbeat timeouts
* @param log Logger to be used for the logging
* @param <I> Type of the incoming payload
* @param <O> Type of the outgoing payload
* @return A new HeartbeatManager instance which actively sends heartbeats
*/
public <I, O> HeartbeatManager<I, O> createHeartbeatManagerSender(
ResourceID resourceId,
HeartbeatListener<I, O> heartbeatListener,
ScheduledExecutor scheduledExecutor,
Logger log) { return new HeartbeatManagerSenderImpl<>(
heartbeatInterval,
heartbeatTimeout,
resourceId,
heartbeatListener,
scheduledExecutor,
scheduledExecutor,
log);
} /**
* Creates an HeartbeatServices instance from a {@link Configuration}.
*
* @param configuration Configuration to be used for the HeartbeatServices creation
* @return An HeartbeatServices instance created from the given configuration
*/
public static HeartbeatServices fromConfiguration(Configuration configuration) {
long heartbeatInterval = configuration.getLong(HeartbeatManagerOptions.HEARTBEAT_INTERVAL); long heartbeatTimeout = configuration.getLong(HeartbeatManagerOptions.HEARTBEAT_TIMEOUT); return new HeartbeatServices(heartbeatInterval, heartbeatTimeout);
}
}
没错超时时间就在HeartbeatManagerOptions.HEARTBEAT_TIMEOUT
/** Timeout for requesting and receiving heartbeat for both sender and receiver sides. */
public static final ConfigOption<Long> HEARTBEAT_TIMEOUT =
key("heartbeat.timeout")
.defaultValue(50000L)
.withDescription("Timeout for requesting and receiving heartbeat for both sender and receiver sides.");
引起心跳超时有可能是yarn压力比较大引起的,先暂时在conf/flink-conf.yaml将这个值调大一点,再观察。
#Timeout for requesting and receiving heartbeat for both sender and receiver sides.
heartbeat.timeout: 180000
Flink Heartbeat of TaskManager和Heartbeat of ResourceManager timed out问题的更多相关文章
- Flink JobManager 和 TaskManager 原理
转自:https://www.cnblogs.com/nicekk/p/11561836.html 一.概述 Flink 整个系统主要由两个组件组成,分别为 JobManager 和 TaskMana ...
- Linux-HA实战(1)— Heartbeat安装
接触Heartbeat主要是因为之前项目中使用了TFS,最近想给nameserver做HA,因为TFS官方用的Heartbeat,所以刚好了解下,参考了网络上很多内容,这里简单记录下. 内容 环境和软 ...
- 高可用集群heartbeat全攻略
heartbeat的概念 Linux-HA的全称是High-Availability Linux,它是一个开源项目,这个开源项目的目标是:通过社区开发者的共同努力,提供一个增强linux可靠性(r ...
- 使用Heartbeat实现双机热备
使用Heartbeat实现"双机热备"或者称为"双机互备"heartbeat的工作原理:heartbeat最核心的包含两个部分,心跳监測部分和资源接管部分,心跳 ...
- Heartbeat实现集群高可用热备
公司最近需要针对服务器实现热可用热备,这几天也一直在琢磨这个方面的东西,今天做了一些Heartbeat方面的工作,在此记录下来,给需要的人以参考. Heartbeat 项目是 Linux-HA 工程的 ...
- Heartbeat详解
转自:http://blog.sina.com.cn/s/blog_7b6fc4c901012om0.html 配置主节点的Heartbeat Heartbeat的主要配置文件有ha.cf.hares ...
- (转)Linux-HA实战(1)— Heartbeat安装
原文:http://blog.csdn.net/liaomin416100569/article/details/76087448-------centos7源代码编译安装heartbeat 原文:h ...
- Heartbeat使用梳理
在日常的集群系统架构中,一般用到Heartbeat的主要就2种:1)高可用(High Availability)HA集群, 使用Heartbeat实现,也称为"双机热备", &qu ...
- 1 NFS高可用解决方案之DRBD+heartbeat搭建
preface NFS作为业界常用的共享存储方案,被众多公司采用.我司也不列外,使用NFS作为共享存储,为前端WEB server提供服务,主要存储网页代码以及其他文件. 高可用方案 说道NFS,不得 ...
- HA(High available)--Heartbeat高可用性集群(双机热备)菜鸟入门级
HA(High available)--Heartbeat高可用性集群(双机热备) 1.理解:两台服务器A和B ,当A提供服务,B闲置待命,当A服务宕机,会自动切换至B机器继续提供服务.当主机恢复 ...
随机推荐
- Jenkins+Git+Gitlab+Ansible 持续集成和自动部署
- React 使用链表遍历组件树
React 为在有限的资源情况下,更好地控制UI的更新,提出了时间分片的概念.以达到三个目标: performing non-blocking rendering(无阻塞渲染):applying up ...
- el-tree 点击父级或者父级的checkbox,展开所有子集
<el-tree :data="authData" show-checkbox :check-strictly="true" :check-on-clic ...
- zabbix连接Mysql提示Can’t connect to local MySQL server through socket的解决方法
要解决此错误,有两种解决方法: 1.将连接参数"host"的值由"localhost"改成"127.0.0.1": 2.按下面方法,在&qu ...
- tp5中接口,需要跨页面传送id时,不使用地址栏传送id,使用session保存。
在Tp5中使用ajax点击某个按钮或连接时需要跳转页面并传送数据. 一般情况下都是绑定到跳转地址的后面以地址栏的方式传输,然后再去相应的页面使用js扒下来,比较麻烦,而且地址栏传值也不安全. 下面这个 ...
- (未完成)JAVAWEB学习——
一.Servlet开发 1.sun公司提供的一种动态web资源开发技术,本质上就要是一段Java小程序,可以将Servlet加入到容器中运行Servlet. *servlet容器 -- 能够运行ser ...
- IaaS--云虚拟机(二)(何恺铎《深入浅出云计算》笔记整理)
[如何挑选合适的虚拟机型号] 1.根据类型.云厂商会提供均衡型.计算密集型.内存优化型.图形计算型等常见的虚拟机类型.这些类型对应着硬件资源的某种合理配比或针对性强化,方便你在面向不同场景时,选择最合 ...
- 【Java学习day04】Java文件的创建和Java代码的执行
Java文件的创建和Java代码的执行 随便新建一个文件夹,存放代码 在新建的文件夹里新建一个java文件 新建一个文本文档 将新建的文本文档重命名为hello.java 注意了!后缀必须改为.jav ...
- KiCad,一款开源的PCB设计软件
Kicad,一款开源的PCB设计软件 1.偷偷摸摸久矣 还记得是大二的实践课,用Protel99se设计一款PCB,把电路图转印到铜板上并腐蚀出来,然后钻孔.焊接,那时候的软件是真难用,后面自学了Al ...
- CAD梦想画图中的“绘图工具——多线段”
CAD软件多线段命令是作为单个平面对象创建的相互连接的线段序列.可以创建直线段.圆弧段或两者的组合线段. 执行方式: (1)单击菜单栏"绘图→多线段".如图: (2)单击绘图工具栏 ...