three supported reliability levels: * End-to-end * Store on failure * Best effort
https://github.com/cloudera/flume/blob/master/flume-docs/src/docs/UserGuide/Introduction
=== Reliability | |
Reliability, the ability to continue delivering events in the face of | |
failures without losing data, is a vital feature of Flume. Large | |
distributed systems can and do suffer partial failures in many ways - | |
physical hardware can fail, resources such as network bandwidth or | |
memory can become scarce, or software can crash or run slowly. Flume | |
emphasizes fault-tolerance as a core design principle and keeps | |
running and collecting data even when many components have failed. | |
Flume can guarantee that all data received by an agent node will | |
eventually make it to the collector at the end of its flow as long as | |
the agent node keeps running. That is, data can be *reliably* | |
delivered to its eventual destination. | |
However, reliable delivery can be very resource intensive and is often | |
a stronger guarantee than some data sources require. Therefore, Flume | |
allows the user to specify, on a per-flow basis, the level of | |
reliability required. There are three supported reliability levels: | |
* End-to-end | |
* Store on failure | |
* Best effort | |
.A Note About Reliability | |
****************** | |
Although Flume is extremely tolerant to machine, network, and software | |
failures, there is never any such thing as '100% reliability'. If all | |
the machines in a Flume installation were irrevocably destroyed in | |
some terrible data center incident, all copies of Flume's data would | |
be lost and there would be no way to recover them. Therefore all of | |
Flume's reliability levels make guarantees about data delivery 'until | |
some maximum number of failures have occurred'. Flume's failure modes | |
- in terms of what can fail and what will keep running if they do - | |
are described in detail later in this guide. | |
****************** | |
The *end-to-end* reliability level guarantees that once Flume accepts | |
an event, that event will make it to the endpoint - as long as the | |
agent that accepted the event remains live long enough. The first | |
thing the agent does in this setting is write the event to disk in a | |
''write-ahead log'' (WAL) so that, if the agent crashes and restarts, | |
knowledge of the event is not lost. After the event has successfully | |
made its way to the end of its flow, an acknowledgment is sent back to | |
the originating agent so that it knows it no longer needs to store the | |
event on disk. This reliability level can withstand any number of | |
failures downstream of the initial agent. | |
The *store on failure* reliability level causes nodes to only require | |
an acknowledgement from the node one hop downstream. If the sending | |
node detects a failure, it will store data on its local disk until the | |
downstream node is repaired, or an alternate downstream destination | |
can be selected. While this is effective, data can be lost if a | |
compound or silent failure occurs. | |
The *best-effort* reliability level sends data to the next hop with no | |
attempts to confirm or retry delivery. If nodes fail, any data that | |
they were in the process of transmitting or receiving can be | |
lost. This is the weakest reliability level, but also the most | |
lightweight. |
=== Reliability | |
Reliability, the ability to continue delivering events in the face of | |
failures without losing data, is a vital feature of Flume. Large | |
distributed systems can and do suffer partial failures in many ways - | |
physical hardware can fail, resources such as network bandwidth or | |
memory can become scarce, or software can crash or run slowly. Flume | |
emphasizes fault-tolerance as a core design principle and keeps | |
running and collecting data even when many components have failed. | |
Flume can guarantee that all data received by an agent node will | |
eventually make it to the collector at the end of its flow as long as | |
the agent node keeps running. That is, data can be *reliably* | |
delivered to its eventual destination. | |
However, reliable delivery can be very resource intensive and is often | |
a stronger guarantee than some data sources require. Therefore, Flume | |
allows the user to specify, on a per-flow basis, the level of | |
reliability required. There are three supported reliability levels: | |
* End-to-end | |
* Store on failure | |
* Best effort | |
.A Note About Reliability | |
****************** | |
Although Flume is extremely tolerant to machine, network, and software | |
failures, there is never any such thing as '100% reliability'. If all | |
the machines in a Flume installation were irrevocably destroyed in | |
some terrible data center incident, all copies of Flume's data would | |
be lost and there would be no way to recover them. Therefore all of | |
Flume's reliability levels make guarantees about data delivery 'until | |
some maximum number of failures have occurred'. Flume's failure modes | |
- in terms of what can fail and what will keep running if they do - | |
are described in detail later in this guide. | |
****************** | |
The *end-to-end* reliability level guarantees that once Flume accepts | |
an event, that event will make it to the endpoint - as long as the | |
agent that accepted the event remains live long enough. The first | |
thing the agent does in this setting is write the event to disk in a | |
''write-ahead log'' (WAL) so that, if the agent crashes and restarts, | |
knowledge of the event is not lost. After the event has successfully | |
made its way to the end of its flow, an acknowledgment is sent back to | |
the originating agent so that it knows it no longer needs to store the | |
event on disk. This reliability level can withstand any number of | |
failures downstream of the initial agent. | |
The *store on failure* reliability level causes nodes to only require | |
an acknowledgement from the node one hop downstream. If the sending | |
node detects a failure, it will store data on its local disk until the | |
downstream node is repaired, or an alternate downstream destination | |
can be selected. While this is effective, data can be lost if a | |
compound or silent failure occurs. | |
The *best-effort* reliability level sends data to the next hop with no | |
attempts to confirm or retry delivery. If nodes fail, any data that | |
they were in the process of transmitting or receiving can be | |
lost. This is the weakest reliability level, but also the most | |
lightweight. |
three supported reliability levels: * End-to-end * Store on failure * Best effort的更多相关文章
- SignalR Supported Platforms -摘自网络
SignalR is supported under a variety of server and client configurations. In addition, each transpor ...
- store操作
store.remove(rs); store.sync({ success: function (e, opt) { this.store.commitChanges(); }, failure: ...
- extjs 解决使用store.sync()方法更新item有时不触发后台action的问题
问题描述: extjs 解决使用store.sync()方法更新item有时不触发后台action,不出发后台action的原因是item的字段值没有变化 解决方法: item.setDirty(tr ...
- PMP用语集
AC actual cost 实际成本 ACWP actual cost of work performed 已完工作实际成本 BAC budget at completion 完工预算 BCWP b ...
- Solaris10安装配置LDAP(iPlanet Directory Server )
Solaris10安装光盘自带了iPlanet Directory Server安装包,系统管理员可以利用iPlanet Directory Server在Solaris系统创建一个LDAP Serv ...
- memory ordering 内存排序
Memory ordering - Wikipedia https://en.wikipedia.org/wiki/Memory_ordering https://zh.wikipedia.org/w ...
- Web测试介绍2一 安全测试
安全测试是在IT软件产品的生命周期中,特别是产品开发基本完成到发布阶段,对产品进行检验以验证产品符合安全需求定义和产品质量标准的过程. 主要安全需求包括: (i) 认证 Authent ...
- Python 3.6.0的sqlite3模块无法执行VACUUM语句
Python 3.6.0的sqlite3模块存在一个bug(见issue 29003),无法执行VACUUM语句. 一执行就出现异常: Traceback (most recent call last ...
- Flume1.5.0的安装、部署、简单应用(含伪分布式、与hadoop2.2.0、hbase0.96的案例)
目录: 一.什么是Flume? 1)flume的特点 2)flume的可靠性 3)flume的可恢复性 4)flume 的 一些核心概念 二.flume的官方网站在哪里? 三.在哪里下载? 四.如何安 ...
随机推荐
- 转 Python爬虫入门四之Urllib库的高级用法
静觅 » Python爬虫入门四之Urllib库的高级用法 1.设置Headers 有些网站不会同意程序直接用上面的方式进行访问,如果识别有问题,那么站点根本不会响应,所以为了完全模拟浏览器的工作,我 ...
- 标准C程序设计七---42
Linux应用 编程深入 语言编程 标准C程序设计七---经典C11程序设计 以下内容为阅读: <标准C程序设计>(第7版) 作者 ...
- C++拷贝(复制)构造函数详解
原文:http://blog.csdn.net/lwbeyond/article/details/6202256/[侵删] 一. 什么是拷贝构造函数 首先对于普通类型的对象来说,它们之间的复制是很简单 ...
- C#中流写入类StreamWriter的介绍
C#中流写入类StreamWriter的介绍 (转) 应用FileStream类需要许多额外的数据类型转换工作,十分影响效率.使用StreamWriter类将提供更简单,更方便的操作方式. Str ...
- Python基础数据类型补充及深浅拷贝
本节主要内容:1. 基础数据类型补充2. set集合3. 深浅拷贝主要内容:一. 基础数据类型补充首先关于int和str在之前的学习中已经讲了80%以上了. 所以剩下的自己看一看就可以了.我们补充给一 ...
- VUE之命令行报错:Component template should contain exactly one root element. If you are using v-if on multiple elements, use v-else-if to chain them instead 解决办法
Failed to compile. ./node_modules/vue-loader/lib/template-compiler?{"id":"data-v-5992 ...
- 洛谷——P1078 文化之旅
P1078 文化之旅 题目描述 有一位使者要游历各国,他每到一个国家,都能学到一种文化,但他不愿意学习任何一种文化超过一次(即如果他学习了某种文化,则他就不能到达其他有这种文化的国家).不同的国家可能 ...
- HAXM 6.0.5显示不兼容Windows
HAXM 6.0.5显示不兼容Windows 最近更新Android后,用户会在Android Manager中发现,以前可以安装Intel x86模拟器现在不能安装了.提示错误信息如下:intel ...
- spring-mvc 的一些使用技巧(转)
APP 服务端的 Token 验证 通过拦截器对使用了@Authorization注解的方法进行请求拦截,从 http header 中取出 token 信息,验证其是否合法.非法直接返回 401 错 ...
- CEF General Usage(CEF3预览)
CEF General Usage(CEF3预览) 介绍 CEF全称Chromium Embedded Framework,是一个基于Google Chromium 的开源项目.Google Chro ...