Apache Tez Design
http://tez.incubator.apache.org/
http://dongxicheng.org/mapreduce-nextgen/apache-tez/
http://dongxicheng.org/mapreduce-nextgen/apache-tez-newest-progress/
Tez aims to be a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce.
In the short term the major focus is to support Hive and Pig, specifically to enable performance improvements to batch and ad-hoc interactive queries.
What services will Tez provide
Tez兼容传统的map-reduce jobs, 当然主要focus提供基于DAG的jobs和相应的API以及primitives.
Tez provides runtime components:
- An execution environment that can handle traditional map-reduce jobs
- An execution environment that handles DAG-based jobs comprising various built-in and extendable primitives
- Cluster-side determination of input pieces
- Runtime planning such as task cardinality determination and dynamic modification to the DAG structure
Tez provides APIs to access these services:
- Traditional map-reduce functionality is accessed via java classes written to the Job interface: org.apache.hadoop.mapred.Job and/or org.apache.hadoop.mapreduce.v2.app.job.Job;
and by specifying in yarn-site that the map-reduce framework should be Tez. - DAG-based execution is accessed via the new Tez DAG API: org.apache.tez.dag.api.*, org.apache.tez.engine.api.*.
Tez provides pre-made primitives for use with the DAG API (org.apache.tez.engine.common.*)
- Vertex Input
- Vertex Output
- Sorting
- Shuffling
- Merging
- Data transfer
Tez-YARN architecture
In the above figure Tez is represented by the red components: client-side API, an AppMaster, and multiple containers that execute child processes under the control of the AppMaster.
Three separate software stacks are involved in the execution of a Tez job, each using components from the clientapplication, Tez, and YARN:
DAG topologies and scenarios
The following terminology is used:
Job Vertex: A “stage” in the job plan. 逻辑顶点, 可以理解成stage
Job Edge: The logical connections between Job Vertices. 逻辑边, 关联
Vertex: A materialized stage at runtime comprising a certain number of materialized tasks. 物理顶点, 由并行的tasks节点组成
Edge: Represents actual data movement between tasks. 物理边, 代表实际数据流向
Task: A process performing computation within a YARN container. Task, 一个执行节点
Task cardinality: The number of materialized tasks in a Vertex. Task基数, Vertex的并发度
Static plan: Planning decisions fixed before job submission.
Dynamic plan: Planning decisions made at runtime in the AppMaster process.
Tez API
The Tez API comprises many services that support applications to run DAG-style jobs. An application that makes use of Tez will need to:
1. Create a job plan (the DAG) comprising vertices, edges, and data source references
2. Create task implementations that perform computations and interact with the DAG AppMaster
3. Configure Yarn and Tez appropriately
DAG definition API
抽象DAG的定义接口
public class DAG{
DAG();
void addVertex(Vertex);
void addEdge(Edge);
void addConfiguration(String, String);
void setName(String);
void verify();
DAGPlan createDaG();
} public class Vertex {
Vertex(String vertexName, String processorName, int parallelism);
void setTaskResource();
void setTaskLocationsHint(TaskLocationHint[]);
void setJavaOpts(String);
String getVertexName();
String getProcessorName();
int getParallelism();
Resource getTaskResource();
TaskLocationHint[] getTaskLocationsHint();
String getJavaOpts();
} public class Edge {
Edge(Vertex inputVertex, Vertex outputVertex, EdgeProperty edgeProperty);
String getInputVertex();
String getOutputVertex();
EdgeProperty getEdgeProperty();
String getId();
}
Execution APIs
Task作为Tez的执行者, 遵循input, output, processor的模式
public interface Master
//a context object for task execution. currently only stub public interface Input{
void initialize(Configuration conf, Master master)
boolean hasNext()
Object getNextKey()
Iterable<Object> getNextValues()
float getProgress()
void close()
} public interface Output{
void initialize(Configuration conf, Master master);
void write(Object key, Object value);
OutputContext getOutputContext();
void close();
} public interface Partitioner {
int getPartition(Object key, Object value, int numPartitions);
} public interface Processor {
void initialize(Configuration conf, Master master)
void process(Input[] in, Output[] out)
void close()
} public interface Task{
void initialize(Configuration conf, Master master)
Input[] getInputs();
Processor getProcessor();
Output[] getOutputs();
void run()
void close()
}
Apache Tez Design的更多相关文章
- CentOS 6.5 Maven 编译 Apache Tez 0.8.3 踩坑/报错解决记录
最近准备学习使用Tez,因此从官网下载了最新的Tez 0.8.3源码,按照安装教程编译使用.平时使用的集群环境是离线的,本打算这一次也进行离线编译,无奈一编译就开始报缺少jar包的错,即使手动下载ja ...
- Apache Tez 了解
你可能听说过Apache Tez,它是一个针对Hadoop数据处理应用程序的新分布式执行框架.但是它到底是什么呢?它的工作原理是什么?哪些人应该使用它,为什么?如果你有这些疑问,那么可以看一下Bika ...
- Apache Tez 0.7、0.83、 0.82 安装、调试笔记
———————————————————— 准备 Tez 编译环境 ———————————————————— 1 需要的支持 tez0.7 需要 Hadoop 2.60 以上 2 需要的 linux 相 ...
- Apache Tez on hive
———————————————————— 调配 Hadoop ———————————————————— 1 将 编译好的 TEZ .tar.gz 文件上传到 HDFS 中. hdfs fs -p ...
- Big Data资料汇总
整理和翻新一下自己看过和笔记过的Big Data相关的论文和Blog Streaming & Spark In-Stream Big Data Processing Discretized S ...
- apache开源项目 -- tez
为了更高效地运行存在依赖关系的作业(比如Pig和Hive产生的MapReduce作业),减少磁盘和网络IO,Hortonworks开发了DAG计 算框架Tez.Tez是从MapReduce计算框架演化 ...
- Hadoop2.0/YARN深入浅出(Hadoop2.0、Spark、Storm和Tez)
随着云计算.大数据迅速发展,亟需用hadoop解决大数据量高并发访问的瓶颈.谷歌.淘宝.百度.京东等底层都应用hadoop.越来越多的企 业急需引入hadoop技术人才.由于掌握Hadoop技术的开发 ...
- Apache 项目列表功能分类便于技术选型
big-data (49): Apache Accumulo Apache Airavata Apache Ambari Apache Apex Apache Avro Apache Be ...
- hive on tez配置
1.Tez简介 Tez是Hontonworks开源的支持DAG作业的计算框架,它可以将多个有依赖的作业转换为一个作业从而大幅提升MapReduce作业的性能.Tez并不直接面向最终用户--事实上它允许 ...
随机推荐
- 单双口RAM
// Quartus II Verilog Template// Simple Dual Port RAM with separate read/write addresses and// singl ...
- 解决Jenkins无法编译Egret5.0项目的问题
问题的原因可查看:https://blog.csdn.net/sujun10/article/details/75512929 解决 造成这个问题的原因是用户权限分配,你可以通过下面几步解决,而非改e ...
- JAVA Socket 底层是怎样基于TCP/IP 实现的???
首先必须明确:TCP/IP模型中有四层结构: 应用层(Application Layer).传输层(Transport Layer).网络层(Internet Layer ).链路层( ...
- 李洪强iOS开发OC[001]-NSLog函数的使用方法
- PHP特性整合(PHP5.X到PHP7.1.x)
Buid-in web server内置了一个简单的Web服务器 把当前目录作为Root Document只需要这条命令即可: php -S localhost:3300 也可以指定其它路径 php ...
- ORACLE 中 TRANSLATE的用法
--TRANSLATE(string,from_str,to_str) --to_str和from_str中的字符一一对应 --如果string里有,from_str字符集里没有的字符,将保留 --如 ...
- Ubuntu 12.04 server 如何安装 OpenERP 7(转)
不经意的一次看到OpenERP这个开源ERP,就被其丰富的功能,简洁的画面,熟悉的语言所吸引.迫不及待的多方查询资料,自己架设一个测试环境来进行了解.以下为测试安装时候的步骤说明,以备查询,并供有需要 ...
- android之ViewPager修改滑动速度
在android中,使用过viewpager的人都清楚,我们如果使用viewpager进行滑动时,如果通过手指滑动来进行的话,可以根据手指滑动的距离来实现,但是如果通过setCurrentItem函数 ...
- 【Mac + Appium + Python3.6学习(五)】之常用的Android自动化测试API总结
Github测试样例地址:https://github.com/appium-boneyard/sample-code/tree/master/sample-code/examples ①定位text ...
- 常见sql 写法总结
关于如何获取1对多数据中最大条数据的写法 例子: LEFT JOIN ( SELECT * FROM table AS n1 WHERE n1.ID IN ( SELECT MAX(id) FROM ...