软件安装路径

软件名称 版本 安装路径
jdk 1.6.0_12

/usr/java/jdk1.6.0_12
maven 3.1.0

/usr/local//apache-maven-3.1.0
Oozie 3.3.1

/usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1
hadoop 1.0.3

/usr/local/hadoop-1.0.3
mysql 5.5.7

/usr/local/mysql

环境变量.bash_profile

export HADOOP_HOME="/home/tnuser/hadoop"

export HADOOP_PREFIX=$HADOOP_HOME

export HBASE_HOME="/home/tnuser/hbase"

export HIVE_HOME="/home/tnuser/hive"

export OOZIE_HOME="/home/tnuser/oozie"

export JAVA_HOME="/usr/java/jdk1.6.0_12"

export JRE_HOME=$JAVA_HOME/jre

export MAVEN_HOME="/usr/local/apache-maven-3.1.0"

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib

export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$PIG_HOME/bin:$HIVE_HOME/bin:$MAVEN_HOME/bin:$OOZIE_HOME/bin:/usr/bin:/sbin:/usr/sbin/:$PATH

export PIG_CLASSPATH=/usr/java/jdk1..0_12/lib/tools.jar:$PIG_CLASSPATH

export HIVE_LIB=$HIVE_HOME/lib/hive-hbase-handler-0.10..jar,$HIVE_HOME/lib/hbase-0.92..jar,$HIVE_HOME/lib/zookeeper-3.4..jar,$HIVE_HOME/lib/guava-r09.jar

export "OOZIE_URL=http://localhost:11000/oozie"

家目录设置

$ ls -lh /home/tnuser/

total

lrwxrwxrwx  tnuser appuser  Jun  : hadoop -> /usr/local/hadoop-1.0.

lrwxrwxrwx  tnuser appuser  Jun  : hbase -> /usr/local/hbase-0.92.

lrwxrwxrwx  tnuser appuser  Jun  : hdfs -> /usr/local/data

drwxr-xr-x  tnuser appuser  Jun  : logs

lrwxrwxrwx  tnuser appuser  Jun  : oozie -> /usr/local/oozie-3.3./distro/target/oozie-3.3.-distro/oozie-3.3.

1. maven安装

$ wget https://archive.apache.org/dist/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.tar.gz

$ tar -zxf apache-maven-3.1.-bin.tar.gz

2.hadoop安装

略

3.mysql安装

略

4. 生成Oozie安装包

$ wget https://archive.apache.org/dist/oozie/3.3.1/oozie-3.3.1.tar.gz

$ tar -zxf oozie-3.3..tar.gz

$ cd /usr/local/contentplatform/oozie-3.3./bin

$ ./mkdistro.sh -DskipTests

报错及解决

报错

ERROR] Failed to execute goal on project oozie-sharelib-hive: Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hive:jar:3.3.1: Failed to collect dependencies at org.apache.hive:hive-cli:jar:0.9.0 -> org.apache.hive:hive-metastore:jar:0.9.0 -> javax.jdo:jdo2-api:jar:2.3-ec: Failed to read artifact descriptor for javax.jdo:jdo2-api:jar:2.3-ec: Could not transfer artifact javax.jdo:jdo2-api:pom:2.3-ec from/to Codehaus repository (http://repository.codehaus.org/): repository.codehaus.org: Unknown host repository.codehaus.org -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :oozie-sharelib-hive

ERROR, Oozie distro creation failed

解决

将安装根目录的pom.xml如下代码

        <repository>

            <id>Codehaus repository</id>

            <url>http://repository.codehaus.org/</url>

            <snapshots>

                <enabled>false</enabled>

            </snapshots>

        </repository>

替换为

        <repository>

            <id>Codehaus repository</id>

            <name>codehaus-mule-repo</name>

            <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>

            <layout>default</layout>

        </repository>

重新运行

./mkdistro.sh -DskipTests

出现以下结果说明生成安装包成功

Oozie distro created, DATE[2019.06.18-01:40:02GMT] VC-REV[unavailable], available at [/usr/local/oozie-3.3.1/distro/target]

5.安装Oozie

5.1 启用Oozie Web Console

wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

解压到$OOZIE_HOME根目录

5.2 复制$HADOOP_HOME以及$HADOOP_HOME/lib所有jar包到$OOZIE_HOME/libext

$ mkdir $OOZIE_HOME/libext

$ scp  $HADOOP_HOME/*.jar $OOZIE_HOME/libext

$ scp $HADOOP_HOME/lib/*.jar $OOZIE_HOME/libext

5.3 添加mysql数据库驱动到Oozie

$ wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz

$ mv mysql-connector-java-5.1..tar.gz $SQOOP_HOME/libext

5.4 安装Oozie

$ oozie-setup.sh -hadoop 0.20. $HADOOP_HOME -extjs ext-2.2

输出log

  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

INFO: Adding extension: /home/tnuser/oozie/libext/asm-3.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/aspectjrt-1.6.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/aspectjtools-1.6.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-1.7.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-core-1.8.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-cli-1.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-codec-1.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-collections-3.2.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-configuration-1.6.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-daemon-1.0.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-digester-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-el-1.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-httpclient-3.0.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-lang-2.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-1.1.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-api-1.0.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-math-2.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-net-1.4.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/core-3.1.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-ant-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-capacity-scheduler-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-client-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-core-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-examples-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-fairscheduler-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-minicluster-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-test-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-thriftfs-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-tools-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hsqldb-1.8.0.10.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jackson-core-asl-1.8.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jackson-mapper-asl-1.8.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jasper-compiler-5.5.12.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jasper-runtime-5.5.12.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jdeb-0.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-core-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-json-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-server-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jets3t-0.6.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jetty-6.1.26.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jetty-util-6.1.26.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jsch-0.1.42.jar

INFO: Adding extension: /home/tnuser/oozie/libext/junit-4.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/kfs-0.2.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/log4j-1.2.16.jar

INFO: Adding extension: /home/tnuser/oozie/libext/mockito-all-1.8.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/mysql-connector-java-5.1.24-bin.jar

INFO: Adding extension: /home/tnuser/oozie/libext/oro-2.0.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/servlet-api-2.5-20081211.jar

INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-api-1.4.3.jar

INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-log4j12-1.4.3.jar

INFO: Adding extension: /home/tnuser/oozie/libext/xmlenc-0.52.jar

Injecting following Hadoop JARs

/home/tnuser/hadoop/hadoop-core-1.0.3.jar

New Oozie WAR file with added 'Hadoop JARs, ExtJS library, JARs' at /home/tnuser/oozie/oozie-server/webapps/oozie.war

INFO: Oozie is ready to be started

5.5 生成Oozie数据库初始化文件

$ ooziedb.sh create -sqlfile oozie.sql

5.6 将初始化文件导入mysql数据库

CREATE DATABASE oozie_data DEFAULT CHARACTER SET uft8 ;

CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie';

GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'*'

use oozie;

source oozie.sql;

5.7 配置hadoop core-site.xml，添加以下参数，如之前未添加，需要重启hadoop

<property>

  <name>hadoop.proxyuser.tnuser.hosts</name>

  <value>*</value>

</property>

<property>

  <name>hadoop.proxyuser.tnuser.groups</name>

  <value>*</value>

</property>

5.8 启动Oozie

$ oozie-start.sh

$ netstat -antl | grep

tcp               0.0.0.0:           0.0.0.0:*               LISTEN

$ oozie admin -oozie http://localhost:11000/oozie -status

System mode: NORMAL

5.9 将Oozie共享库导入hadoop

$ cd $OOZIE_HOME

$ tar xvfz oozie-sharelib-3.3..tar.gz

$ HADOOP_HOME/bin/hadoop fs -put share share

$ hadoop fs -ls /user/tnuser/share/lib

Found  items

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/distcp

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/hive

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/mapreduce-streaming

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/oozie

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/pig

-rw-r--r--    tnuser supergroup        -- : /user/tnuser/share/lib/sharelib.properties

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/sqoop

6. 测试Oozie

6.1 修改example包文件对应属性值

$ cd $OOZIE_HOME

$ tar -zxf oozie-examples.tar.gz

$ ls -lh examples/apps/map-reduce/

total .0K

-rw-r--r--  tnuser appuser  Jun  : job.properties

drwxr-xr-x  tnuser appuser    Jun  : lib

-rw-r--r--  tnuser appuser .5K Jan    workflow.xml

将job.properties文件namenode和jobtracker地址修改为实际的地址

$ cat examples/apps/map-reduce/job.properties 

nameNode=hdfs://ec2d-newcntprocnn-01:9000

jobTracker=ec2d-newcntprocnn-:

queueName=default

examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce

outputDir=map-reduce

6.2 上传到hdfs

$ hadoop fs -put examples examples

$ hadoop fs -ls /user/tnuser

Found  items

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/examples

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share

6.3 测试Oozie

报错

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: User: tnuser is not allowed to impersonate tnuser]

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: Unauthorized connection for super-user: tnuser from IP 10.189.102.176]

经测试以下hdfs参数的value值为“*“会出现以上报错，需要修改hdfs参数为对应的真实值

<property>

  <name>hadoop.proxyuser.tnuser.hosts</name>

  <value>ec2d-newcntprocdn-04</value>

</property>

<property>

  <name>hadoop.proxyuser.tnuser.groups</name>

  <value>appuser</value>

</property>

正常运行结果

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

job: --oozie-tnus-W

$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W

Job ID : --oozie-tnus-W

------------------------------------------------------------------------------------------------------------------------------------

Workflow Name : map-reduce-wf

App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce

Status        : RUNNING

Run           :

User          : tnuser

Group         : -

Created       : -- : GMT

Started       : -- : GMT

Last Modified : -- : GMT

Ended         : -

CoordAction ID: -

Actions

------------------------------------------------------------------------------------------------------------------------------------

ID                                                                            Status    Ext ID                 Ext Status Err Code

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@:start:                                  OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@mr-node                                  START_RETRY-                      -          JA009

------------------------------------------------------------------------------------------------------------------------------------

$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W

Job ID : --oozie-tnus-W

------------------------------------------------------------------------------------------------------------------------------------

Workflow Name : map-reduce-wf

App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce

Status        : SUCCEEDED

Run           :

User          : tnuser

Group         : -

Created       : -- : GMT

Started       : -- : GMT

Last Modified : -- : GMT

Ended         : -- : GMT

CoordAction ID: -

Actions

------------------------------------------------------------------------------------------------------------------------------------

ID                                                                            Status    Ext ID                 Ext Status Err Code

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@:start:                                  OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@mr-node                                  OK        job_201906200714_0002  SUCCEEDED  -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@end                                      OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

Oozie 3.3.1安装的更多相关文章

  1. 2-10 就业课(2.0)-oozie：3、安装2

    第七步:修改oozie-site.xml cd /export/servers/oozie-4.1.0-cdh5.14.0/conf vim oozie-site.xml 如果没有这些属性,直接添加进 ...

  2. CDH5X 安装oozie报错To enable Oozie web console install the Ext JS library.

    最近在CDH5.X 安装oozie 服务,服务安装完毕,访问oozie server ui,报如下错误: 页面提示: Oozie web console is disabled.To enable O ...

  3. Oozie如何和安装部署

    1.Oozie的简单介绍: .Oozie是一个工作流引擎服务器,用于运行hadoop map/reduce和hive等任务工作流,同时Oozie还是一个Java web程序,运行在Java Servl ...

  4. 离线安装 Cloudera Manager 5 和 CDH5.10

    关于CDH和Cloudera Manager CDH (Cloudera's Distribution, including Apache Hadoop),是Hadoop众多分支中的一种,由Cloud ...

  5. 【Oozie学习之一】Oozie

    环境 虚拟机:VMware 10 Linux版本:CentOS-6.5-x86_64 客户端:Xshell4 FTP:Xftp4 jdk8 CM5.4 一.简介Oozie由Cloudera公司贡献给A ...

  6. 通过Cloudera Manager安装CDH 5.6

    CDH的简介 大家常常说CDH.其全称是:Cloudera's Distribution Including Apache Hadoop.简单的说是Cloudera公司的Hadoop平台,是在Apac ...

  7. Ambari下安装oozieUI界面无法访问问题

    前言: 其他集群管理工具安装oozie和手动编译安装oozie,关于oozie的UI界面无法访问的问题,大致和该方法类似. 找到下面4步骤里的视图显示的目录,有一个ext-2.2软链指向的路径 lrw ...

  8. 最新版大数据平台安装部署指南，HDP-2.6.5.0，ambari-2.6.2.0

    一.服务器环境配置 1 系统要求 名称 地址 操作系统 root密码 Master1 10.1.0.30 Centos 7.7 Root@bidsum1 Master2 10.1.0.105 Cent ...

  9. 《OD大数据实战》Hue环境搭建

    官网: http://archive.cloudera.com/cdh5/cdh/5/hue-3.7.0-cdh5.3.6/ 一.Hue环境搭建 1. 下载 http://archive.cloude ...

随机推荐

  1. entity framework delete table Error 11007:

    udate model from database 数据库表删除时,会出现“Error 11007:”的异常,此时在.edmx文件中找到此表的实体发现还存在,删除它就没有这个错误 了.

  2. string、wstring、CString 相互转换

    关于string wstring cstring的功能这里不详细叙述了 可参见这里:https://www.cnblogs.com/guolixiucai/p/4716521.html 关于转换这里只 ...

  3. TTTTTTTTTTTT CF 653D 送邮递员

    链接:给一张n个点m条带权边的有向图,有x个人从起点出发到终点,每个人带的都带相同重量的货物, 规定一条边最多能经过其上权的重量的货物,问最多能带多重的货物? 2 ≤ n ≤ 50, 1 ≤ m ≤  ...

  4. idea2018.3.6安装与破解教程（亲测可用、破解到2100年）

    最近,帮室友进行idea安装,之前自己安装借鉴的博客已404,在网上找了好几个都无效,想着总结一份备用. 此博客是又找了一台电脑,边安装边写的. 目录 (已安装好的,可以直接看idea2018.3.6 ...

  5. 「SDOI2017」硬币游戏

    题目链接 问题分析 首先一个显然的做法就是建出AC自动机,然后高斯消元.但是这样的复杂度是\(O(n^3m^3)\)的. 我们发现其实只需要求AC自动机上\(n\)个状态的概率,而其余的概率是没有用的 ...

  6. 隐藏表单域、URL重写、cookie、session

    隐藏表单域: 隐藏域是用来收集或发送信息的不可见元素,对于网页的访问者来说,隐藏域是看不见的.当表单被提交时,隐藏域就会将信息用你设置时定义的名称和值发送到服务器上. 代码格式:<input t ...

  7. Java容器——Set和顺序存储

    当Set使用自己创建的类型时,存储的顺序如何维护,在不同的Set实现中会有不同,而且它们对于在特定的Set中放置的元素类型也有不同的要求: Set(interface) 存入Set的每个元素都必须是唯 ...

  8. 分布式-信息方式-ActiveMQ的静态网络连接

                           ActiveMQ的静态网络连接 在一台服务器上启动多个Broker步骤如下:1:把整个conf文件夹复制一份,比如叫做conf22:修改里面的 activ ...

  9. html外部文件读取/写入

    1.文件的读取 外部文件读取控件: <input type="file" id="file_jquery" onchange="file_jqu ...

  10. spark 笔记 15: ShuffleManager，shuffle map两端的stage/task的桥梁

    无论是Hadoop还是spark,shuffle操作都是决定其性能的重要因素.在不能减少shuffle的情况下,使用一个好的shuffle管理器也是优化性能的重要手段. ShuffleManager的 ...