Apache Tez Design
http://tez.incubator.apache.org/
http://dongxicheng.org/mapreduce-nextgen/apache-tez/
http://dongxicheng.org/mapreduce-nextgen/apache-tez-newest-progress/
Tez aims to be a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce.
In the short term the major focus is to support Hive and Pig, specifically to enable performance improvements to batch and ad-hoc interactive queries.
What services will Tez provide
Tez兼容传统的map-reduce jobs, 当然主要focus提供基于DAG的jobs和相应的API以及primitives.
Tez provides runtime components:
- An execution environment that can handle traditional map-reduce jobs
- An execution environment that handles DAG-based jobs comprising various built-in and extendable primitives
- Cluster-side determination of input pieces
- Runtime planning such as task cardinality determination and dynamic modification to the DAG structure
Tez provides APIs to access these services:
- Traditional map-reduce functionality is accessed via java classes written to the Job interface: org.apache.hadoop.mapred.Job and/or org.apache.hadoop.mapreduce.v2.app.job.Job;
and by specifying in yarn-site that the map-reduce framework should be Tez. - DAG-based execution is accessed via the new Tez DAG API: org.apache.tez.dag.api.*, org.apache.tez.engine.api.*.
Tez provides pre-made primitives for use with the DAG API (org.apache.tez.engine.common.*)
- Vertex Input
- Vertex Output
- Sorting
- Shuffling
- Merging
- Data transfer
Tez-YARN architecture
In the above figure Tez is represented by the red components: client-side API, an AppMaster, and multiple containers that execute child processes under the control of the AppMaster.
Three separate software stacks are involved in the execution of a Tez job, each using components from the clientapplication, Tez, and YARN:
DAG topologies and scenarios
The following terminology is used:
Job Vertex: A “stage” in the job plan. 逻辑顶点, 可以理解成stage
Job Edge: The logical connections between Job Vertices. 逻辑边, 关联
Vertex: A materialized stage at runtime comprising a certain number of materialized tasks. 物理顶点, 由并行的tasks节点组成
Edge: Represents actual data movement between tasks. 物理边, 代表实际数据流向
Task: A process performing computation within a YARN container. Task, 一个执行节点
Task cardinality: The number of materialized tasks in a Vertex. Task基数, Vertex的并发度
Static plan: Planning decisions fixed before job submission.
Dynamic plan: Planning decisions made at runtime in the AppMaster process.
Tez API
The Tez API comprises many services that support applications to run DAG-style jobs. An application that makes use of Tez will need to:
1. Create a job plan (the DAG) comprising vertices, edges, and data source references
2. Create task implementations that perform computations and interact with the DAG AppMaster
3. Configure Yarn and Tez appropriately
DAG definition API
抽象DAG的定义接口
public class DAG{
DAG();
void addVertex(Vertex);
void addEdge(Edge);
void addConfiguration(String, String);
void setName(String);
void verify();
DAGPlan createDaG();
} public class Vertex {
Vertex(String vertexName, String processorName, int parallelism);
void setTaskResource();
void setTaskLocationsHint(TaskLocationHint[]);
void setJavaOpts(String);
String getVertexName();
String getProcessorName();
int getParallelism();
Resource getTaskResource();
TaskLocationHint[] getTaskLocationsHint();
String getJavaOpts();
} public class Edge {
Edge(Vertex inputVertex, Vertex outputVertex, EdgeProperty edgeProperty);
String getInputVertex();
String getOutputVertex();
EdgeProperty getEdgeProperty();
String getId();
}
Execution APIs
Task作为Tez的执行者, 遵循input, output, processor的模式
public interface Master
//a context object for task execution. currently only stub public interface Input{
void initialize(Configuration conf, Master master)
boolean hasNext()
Object getNextKey()
Iterable<Object> getNextValues()
float getProgress()
void close()
} public interface Output{
void initialize(Configuration conf, Master master);
void write(Object key, Object value);
OutputContext getOutputContext();
void close();
} public interface Partitioner {
int getPartition(Object key, Object value, int numPartitions);
} public interface Processor {
void initialize(Configuration conf, Master master)
void process(Input[] in, Output[] out)
void close()
} public interface Task{
void initialize(Configuration conf, Master master)
Input[] getInputs();
Processor getProcessor();
Output[] getOutputs();
void run()
void close()
}
Apache Tez Design的更多相关文章
- CentOS 6.5 Maven 编译 Apache Tez 0.8.3 踩坑/报错解决记录
最近准备学习使用Tez,因此从官网下载了最新的Tez 0.8.3源码,按照安装教程编译使用.平时使用的集群环境是离线的,本打算这一次也进行离线编译,无奈一编译就开始报缺少jar包的错,即使手动下载ja ...
- Apache Tez 了解
你可能听说过Apache Tez,它是一个针对Hadoop数据处理应用程序的新分布式执行框架.但是它到底是什么呢?它的工作原理是什么?哪些人应该使用它,为什么?如果你有这些疑问,那么可以看一下Bika ...
- Apache Tez 0.7、0.83、 0.82 安装、调试笔记
———————————————————— 准备 Tez 编译环境 ———————————————————— 1 需要的支持 tez0.7 需要 Hadoop 2.60 以上 2 需要的 linux 相 ...
- Apache Tez on hive
———————————————————— 调配 Hadoop ———————————————————— 1 将 编译好的 TEZ .tar.gz 文件上传到 HDFS 中. hdfs fs -p ...
- Big Data资料汇总
整理和翻新一下自己看过和笔记过的Big Data相关的论文和Blog Streaming & Spark In-Stream Big Data Processing Discretized S ...
- apache开源项目 -- tez
为了更高效地运行存在依赖关系的作业(比如Pig和Hive产生的MapReduce作业),减少磁盘和网络IO,Hortonworks开发了DAG计 算框架Tez.Tez是从MapReduce计算框架演化 ...
- Hadoop2.0/YARN深入浅出(Hadoop2.0、Spark、Storm和Tez)
随着云计算.大数据迅速发展,亟需用hadoop解决大数据量高并发访问的瓶颈.谷歌.淘宝.百度.京东等底层都应用hadoop.越来越多的企 业急需引入hadoop技术人才.由于掌握Hadoop技术的开发 ...
- Apache 项目列表功能分类便于技术选型
big-data (49): Apache Accumulo Apache Airavata Apache Ambari Apache Apex Apache Avro Apache Be ...
- hive on tez配置
1.Tez简介 Tez是Hontonworks开源的支持DAG作业的计算框架,它可以将多个有依赖的作业转换为一个作业从而大幅提升MapReduce作业的性能.Tez并不直接面向最终用户--事实上它允许 ...
随机推荐
- Laravel框架学习(Response)
1.Response篇 1.1 基本响应 最基本的HTTP响应只需在路由闭包或控制器动作中返回一个简单字符串即可,但是具体业务逻辑中大部分响应都是在控制器动作中返回Response实例或者视图.Res ...
- apache与和mysql重启命令
修改linux服务器的http配置之后,必须重启Apache服务. 命令为: /etc/rc.d/init.d/httpd restart chown -R mysql:mysql 目录名 改变文件属 ...
- golang https server分析
https: HTTPS是http安全版本的实现,在http与tcp之间加了一层ssl/tls安全传输协议 为了防止请求被监听.篡改.冒充,在tls实现过程中引入了数字证书机制,数字证书由第三方权威机 ...
- The Definitive Guide To Django 2 学习笔记(六) 第四章 模板 (二)使用模板系统
模板系统不是django特有的,它是python的一个库,你可以在任何地方使用它. 使用方法: 1.使用 Template()方法创建Template对象.2.调用Template对象的render( ...
- cubieboard 资源
开发链接: http://pan.baidu.com/s/1o6r27V0 密码: 46h2 文档资料: http://pan.baidu.com/s/1kTFXMSZ 密码: 9pka 视频资料: ...
- Ubuntu Apache配置及开启mod_rewrite模块
刚刚将服务器系统从CentOS换成Ubuntu,将MySQL,Apache,PHP和Wordpress安装好后,发现打开主页是正常的,但是打开文章页面时出现错误.因为使用了自定义的固定链接设置,那自然 ...
- sign-up 签约注册
sign-up 搜索网络['sin,ʌp] n. 注册; 签约;
- 类加载器(ClassLoader)
静态库.动态连接库 程序编制一般需经编辑.编译.连接.加载和运行几个步骤.在我们的应用中,有一些公共代码是需要反复使用,就把这些代码编译为“库”文件:在连接步骤中,连接器将从库文件取得所需的代码,复制 ...
- 图像处理之拼接---图像拼接opencv
基于SURF特征的图像与视频拼接技术的研究和实现(一) 一直有计划研究实时图像拼接,但是直到最近拜读西电2013年张亚娟的<基于SURF特征的图像与视频拼接技术的研究和实现>,条 ...
- 使用StringTokenizer分解字符串
Java切割字符串.一般使用substring.split.StringTokenizer来处理,前两种是String对象的方法,使用字符串能够直接处理,本文介绍下StringTokenizer的使用 ...