准备工作  拷贝原来的模板

 mkdir oozie-apps
cd oozie-apps/
cp -r ../examples/apps/mar-reduce .
mv map-reduce mr-wordcount-wf

配置文件修改

workflow.xml :

 <!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:0.5" name="mr-wordcount-wf">
<start to="mr-node-wordcount"/>
<action name="mr-node-wordcount">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/${oozieDataRoot}/${outputDir}"/>
</prepare>
<configuration>
<property>
<name>mapred.mapper.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.reducer.new-api</name>
<value>true</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
<property>
<name>mapreduce.job.map.class</name>
<value>com.ibeifeng.hadoop.senior.mapreduce.WordCount$WordCountMapper</value>
</property>
<property>
<name>mapreduce.job.reduce.class</name>
<value>com.ibeifeng.hadoop.senior.mapreduce.WordCount$WordCountReducer</value>
</property> <property>
<name>mapreduce.map.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.map.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.job.output.key.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.job.output.value.class</name>
<value>org.apache.hadoop.io.IntWritable</value>
</property>
<property>
<name>mapreduce.input.fileinputformat.inputdir</name>
<value>${nameNode}/${oozieDataRoot}/${inputDir}</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>${nameNode}/${oozieDataRoot}/${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>

job.properties :

 #
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# nameNode=hdfs://hadoop:8020
jobTracker=hadoop:8032
queueName=default
oozieAppsRoot=user/root/oozie-apps
oozieDataRoot=user/root/oozie/datas oozie.wf.application.path=${nameNode}/${oozieAppsRoot}/mr-wordcount-wf/workflow.xml inputDir=mr-wordcount-wf/input
outputDir=mr-wordcount-wf/output

把自己写的MapReduce的jar 放上去

把改好的文件放入到hdfs

 /opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/bin/hadoop dfs -put oozie-apps/ oozie-apps

准备测试数据

在hdfs创建输入目录 并把刚创建的文件上传

 /opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/bin/hadoop dfs -mkdir -p oozie/datas/mr-wordcount-wf/input
/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/bin/hadoop dfs -put test-mapred.txt oozie/datas/mr-wordcount-wf/input

运行oozie

 export OOZIE_URL=http://localhost:11000/oozie
bin/oozie job -config oozie-apps/mr-wordcount-wf/job.properties -run

查看输出结果

原文件     统计后的文件

http://blog.csdn.net/wiborgite/article/details/78585689

oozie与mapreduce简单案例的更多相关文章

  1. 批处理引擎MapReduce应用案例

    批处理引擎MapReduce应用案例 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. MapReduce能够解决的问题有一个共同特点:任务可以被分解为多个子问题,且这些子问题相对独立 ...

  2. Servlet请求头response应用简单案例

    Servlet请求头response应用简单案例:访问AServlet重定向到BServlet,5秒后跳到CServlet,并显示图片: AServlet package cn.yzu; import ...

  3. winform 通过 html 与swf 交互 简单案例

    在上一篇 winform 与 html 交互 简单案例 中讲了winform与html之间的简单交互,接下来的内容是在winform中以html为中转站,实现将swf嵌入winform中并实现交互. ...

  4. [Design Pattern] Front Controller Pattern 简单案例

    Front Controller Pattern, 即前端控制器模式,用于集中化用户请求,使得所有请求都经过同一个前端控制器处理,处理内容有身份验证.权限验证.记录和追踪请求等,处理后再交由分发器把请 ...

  5. [Design Pattern] Observer Pattern 简单案例

    Observer Pattern,即观察者模式,当存在一对多关系,例如一个对象一有变动,就要自动通知被依赖的全部对象得场景,属于行为类的设计模式. 下面是一个观察者模式的简单案例. Observer ...

  6. [Design Pattern] Mediator Pattern 简单案例

    Meditor Pattern,即调解模式,用一个调解类类处理所有的沟通事件,使得降低多对象之间的沟通难度,属于行为类的设计模式.为了方便理解记忆,我也称其为,沟通模式. 下面是一个调解模式的简单案例 ...

  7. [Design Pattern] Iterator Pattern 简单案例

    Iterator Pattern,即迭代时模式,按照顺序依次遍历集合内的每一个元素,而不用了解集合的底层实现,属于行为类的设计模式.为了方便理解记忆,我也会称其为遍历模式. 下面是一个迭代器模式的简单 ...

  8. [Design Pattern] Command Pattern 简单案例

    Command Pattern, 即命令模式,把一个命令包裹在一个对象里面,将命令对象传递给命令的执行方,属于行为类的设计模式 下面是命令模式的一个简单案例. Stock 代表被操作的对象.Order ...

  9. [Design Pattern] Proxy Pattern 简单案例

    Proxy Pattern, 即代理模式,用一个类代表另一个类的功能,用于隐藏.解耦真正提供功能的类,属于结构类的设计模式. 下面是 代理模式的一个简单案例. Image 定义接口,RealImage ...

随机推荐

  1. PAT L1-032 Left-pad

    https://pintia.cn/problem-sets/994805046380707840/problems/994805100684361728 根据新浪微博上的消息,有一位开发者不满NPM ...

  2. Kettle 使用Json输入

    import java.math.BigDecimal; private static final String JD="jd"; private static final Str ...

  3. 有一个集合,判断集合里有没有“world”这个元素,如果有,添加“javaee”

    // 有一个集合,判断集合里有没有“world”这个元素,如果有,添加“javaee” List list = new ArrayList(); list.add("world") ...

  4. 【Linux学习笔记】Linux C中内联汇编的语法格式及使用方法(Inline Assembly in Linux C)

    http://blog.csdn.net/slvher/article/details/8864996 https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm. ...

  5. webgl学习笔记一-绘图单点

    写在前面   WebGl(全称:Web Graphics Library : web图形库) 是基于OpenGL ES 2.0的3D绘图协议.   WebGL完美地解决了现有的Web交互式三维动画的两 ...

  6. shit Rap & mock api

    shit Rap & mock api https://thx.github.io/RAP/study.html https://github.com/thx/RAP/wiki/quick_g ...

  7. utuntu下安装pip&pip3

    在utuntu下建议不要使用apt-get install 安装pip,会出现很多问题. 建议使用如下方式安装: wget https://bootstrap.pypa.io/get-pip.py - ...

  8. git commad

    repo forall -c 'echo $REPO_PATH;ssh trd.git.htc.com -p29419 -lowen_wen gerrit create-project --submi ...

  9. AngularJS中$watch

    $watch在digest执行时,如果watch观察的value与上一次执行时不一样时,就会被触发.angularjs内部的watch实现了页面随model的及时更新.$watch 方法在用的时候主要 ...

  10. (转)Redis使用详细教程

    转载至http://www.cnblogs.com/wangyuyu/p/3786236.html 一.Redis基础部分: 1.redis介绍与安装比mysql快10倍以上 ************ ...