软件安装路径

软件名称 版本 安装路径
jdk 1.6.0_12

/usr/java/jdk1.6.0_12
maven 3.1.0

/usr/local//apache-maven-3.1.0
Oozie 3.3.1

/usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1
hadoop 1.0.3

/usr/local/hadoop-1.0.3
mysql 5.5.7

/usr/local/mysql

环境变量.bash_profile

export HADOOP_HOME="/home/tnuser/hadoop"

export HADOOP_PREFIX=$HADOOP_HOME

export HBASE_HOME="/home/tnuser/hbase"

export HIVE_HOME="/home/tnuser/hive"

export OOZIE_HOME="/home/tnuser/oozie"

export JAVA_HOME="/usr/java/jdk1.6.0_12"

export JRE_HOME=$JAVA_HOME/jre

export MAVEN_HOME="/usr/local/apache-maven-3.1.0"

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib

export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$PIG_HOME/bin:$HIVE_HOME/bin:$MAVEN_HOME/bin:$OOZIE_HOME/bin:/usr/bin:/sbin:/usr/sbin/:$PATH

export PIG_CLASSPATH=/usr/java/jdk1..0_12/lib/tools.jar:$PIG_CLASSPATH

export HIVE_LIB=$HIVE_HOME/lib/hive-hbase-handler-0.10..jar,$HIVE_HOME/lib/hbase-0.92..jar,$HIVE_HOME/lib/zookeeper-3.4..jar,$HIVE_HOME/lib/guava-r09.jar

export "OOZIE_URL=http://localhost:11000/oozie"

家目录设置

$ ls -lh /home/tnuser/

total

lrwxrwxrwx  tnuser appuser  Jun  : hadoop -> /usr/local/hadoop-1.0.

lrwxrwxrwx  tnuser appuser  Jun  : hbase -> /usr/local/hbase-0.92.

lrwxrwxrwx  tnuser appuser  Jun  : hdfs -> /usr/local/data

drwxr-xr-x  tnuser appuser  Jun  : logs

lrwxrwxrwx  tnuser appuser  Jun  : oozie -> /usr/local/oozie-3.3./distro/target/oozie-3.3.-distro/oozie-3.3.

1. maven安装

$ wget https://archive.apache.org/dist/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.tar.gz

$ tar -zxf apache-maven-3.1.-bin.tar.gz

2.hadoop安装

略

3.mysql安装

略

4. 生成Oozie安装包

$ wget https://archive.apache.org/dist/oozie/3.3.1/oozie-3.3.1.tar.gz

$ tar -zxf oozie-3.3..tar.gz

$ cd /usr/local/contentplatform/oozie-3.3./bin

$ ./mkdistro.sh -DskipTests

报错及解决

报错

ERROR] Failed to execute goal on project oozie-sharelib-hive: Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hive:jar:3.3.1: Failed to collect dependencies at org.apache.hive:hive-cli:jar:0.9.0 -> org.apache.hive:hive-metastore:jar:0.9.0 -> javax.jdo:jdo2-api:jar:2.3-ec: Failed to read artifact descriptor for javax.jdo:jdo2-api:jar:2.3-ec: Could not transfer artifact javax.jdo:jdo2-api:pom:2.3-ec from/to Codehaus repository (http://repository.codehaus.org/): repository.codehaus.org: Unknown host repository.codehaus.org -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :oozie-sharelib-hive

ERROR, Oozie distro creation failed

解决

将安装根目录的pom.xml如下代码

        <repository>

            <id>Codehaus repository</id>

            <url>http://repository.codehaus.org/</url>

            <snapshots>

                <enabled>false</enabled>

            </snapshots>

        </repository>

替换为

        <repository>

            <id>Codehaus repository</id>

            <name>codehaus-mule-repo</name>

            <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>

            <layout>default</layout>

        </repository>

重新运行

./mkdistro.sh -DskipTests

出现以下结果说明生成安装包成功

Oozie distro created, DATE[2019.06.18-01:40:02GMT] VC-REV[unavailable], available at [/usr/local/oozie-3.3.1/distro/target]

5.安装Oozie

5.1 启用Oozie Web Console

wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

解压到$OOZIE_HOME根目录

5.2 复制$HADOOP_HOME以及$HADOOP_HOME/lib所有jar包到$OOZIE_HOME/libext

$ mkdir $OOZIE_HOME/libext

$ scp  $HADOOP_HOME/*.jar $OOZIE_HOME/libext

$ scp $HADOOP_HOME/lib/*.jar $OOZIE_HOME/libext

5.3 添加mysql数据库驱动到Oozie

$ wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz

$ mv mysql-connector-java-5.1..tar.gz $SQOOP_HOME/libext

5.4 安装Oozie

$ oozie-setup.sh -hadoop 0.20. $HADOOP_HOME -extjs ext-2.2

输出log

  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

INFO: Adding extension: /home/tnuser/oozie/libext/asm-3.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/aspectjrt-1.6.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/aspectjtools-1.6.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-1.7.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-core-1.8.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-cli-1.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-codec-1.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-collections-3.2.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-configuration-1.6.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-daemon-1.0.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-digester-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-el-1.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-httpclient-3.0.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-lang-2.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-1.1.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-api-1.0.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-math-2.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-net-1.4.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/core-3.1.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-ant-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-capacity-scheduler-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-client-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-core-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-examples-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-fairscheduler-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-minicluster-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-test-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-thriftfs-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-tools-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hsqldb-1.8.0.10.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jackson-core-asl-1.8.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jackson-mapper-asl-1.8.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jasper-compiler-5.5.12.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jasper-runtime-5.5.12.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jdeb-0.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-core-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-json-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-server-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jets3t-0.6.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jetty-6.1.26.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jetty-util-6.1.26.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jsch-0.1.42.jar

INFO: Adding extension: /home/tnuser/oozie/libext/junit-4.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/kfs-0.2.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/log4j-1.2.16.jar

INFO: Adding extension: /home/tnuser/oozie/libext/mockito-all-1.8.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/mysql-connector-java-5.1.24-bin.jar

INFO: Adding extension: /home/tnuser/oozie/libext/oro-2.0.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/servlet-api-2.5-20081211.jar

INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-api-1.4.3.jar

INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-log4j12-1.4.3.jar

INFO: Adding extension: /home/tnuser/oozie/libext/xmlenc-0.52.jar

Injecting following Hadoop JARs

/home/tnuser/hadoop/hadoop-core-1.0.3.jar

New Oozie WAR file with added 'Hadoop JARs, ExtJS library, JARs' at /home/tnuser/oozie/oozie-server/webapps/oozie.war

INFO: Oozie is ready to be started

5.5 生成Oozie数据库初始化文件

$ ooziedb.sh create -sqlfile oozie.sql

5.6 将初始化文件导入mysql数据库

CREATE DATABASE oozie_data DEFAULT CHARACTER SET uft8 ;

CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie';

GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'*'

use oozie;

source oozie.sql;

5.7 配置hadoop core-site.xml，添加以下参数，如之前未添加，需要重启hadoop

<property>

  <name>hadoop.proxyuser.tnuser.hosts</name>

  <value>*</value>

</property>

<property>

  <name>hadoop.proxyuser.tnuser.groups</name>

  <value>*</value>

</property>

5.8 启动Oozie

$ oozie-start.sh

$ netstat -antl | grep

tcp               0.0.0.0:           0.0.0.0:*               LISTEN

$ oozie admin -oozie http://localhost:11000/oozie -status

System mode: NORMAL

5.9 将Oozie共享库导入hadoop

$ cd $OOZIE_HOME

$ tar xvfz oozie-sharelib-3.3..tar.gz

$ HADOOP_HOME/bin/hadoop fs -put share share

$ hadoop fs -ls /user/tnuser/share/lib

Found  items

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/distcp

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/hive

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/mapreduce-streaming

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/oozie

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/pig

-rw-r--r--    tnuser supergroup        -- : /user/tnuser/share/lib/sharelib.properties

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/sqoop

6. 测试Oozie

6.1 修改example包文件对应属性值

$ cd $OOZIE_HOME

$ tar -zxf oozie-examples.tar.gz

$ ls -lh examples/apps/map-reduce/

total .0K

-rw-r--r--  tnuser appuser  Jun  : job.properties

drwxr-xr-x  tnuser appuser    Jun  : lib

-rw-r--r--  tnuser appuser .5K Jan    workflow.xml

将job.properties文件namenode和jobtracker地址修改为实际的地址

$ cat examples/apps/map-reduce/job.properties 

nameNode=hdfs://ec2d-newcntprocnn-01:9000

jobTracker=ec2d-newcntprocnn-:

queueName=default

examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce

outputDir=map-reduce

6.2 上传到hdfs

$ hadoop fs -put examples examples

$ hadoop fs -ls /user/tnuser

Found  items

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/examples

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share

6.3 测试Oozie

报错

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: User: tnuser is not allowed to impersonate tnuser]

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: Unauthorized connection for super-user: tnuser from IP 10.189.102.176]

经测试以下hdfs参数的value值为“*“会出现以上报错，需要修改hdfs参数为对应的真实值

<property>

  <name>hadoop.proxyuser.tnuser.hosts</name>

  <value>ec2d-newcntprocdn-04</value>

</property>

<property>

  <name>hadoop.proxyuser.tnuser.groups</name>

  <value>appuser</value>

</property>

正常运行结果

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

job: --oozie-tnus-W

$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W

Job ID : --oozie-tnus-W

------------------------------------------------------------------------------------------------------------------------------------

Workflow Name : map-reduce-wf

App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce

Status        : RUNNING

Run           :

User          : tnuser

Group         : -

Created       : -- : GMT

Started       : -- : GMT

Last Modified : -- : GMT

Ended         : -

CoordAction ID: -

Actions

------------------------------------------------------------------------------------------------------------------------------------

ID                                                                            Status    Ext ID                 Ext Status Err Code

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@:start:                                  OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@mr-node                                  START_RETRY-                      -          JA009

------------------------------------------------------------------------------------------------------------------------------------

$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W

Job ID : --oozie-tnus-W

------------------------------------------------------------------------------------------------------------------------------------

Workflow Name : map-reduce-wf

App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce

Status        : SUCCEEDED

Run           :

User          : tnuser

Group         : -

Created       : -- : GMT

Started       : -- : GMT

Last Modified : -- : GMT

Ended         : -- : GMT

CoordAction ID: -

Actions

------------------------------------------------------------------------------------------------------------------------------------

ID                                                                            Status    Ext ID                 Ext Status Err Code

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@:start:                                  OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@mr-node                                  OK        job_201906200714_0002  SUCCEEDED  -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@end                                      OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

Oozie 3.3.1安装的更多相关文章

  1. 2-10 就业课(2.0)-oozie：3、安装2

    第七步:修改oozie-site.xml cd /export/servers/oozie-4.1.0-cdh5.14.0/conf vim oozie-site.xml 如果没有这些属性,直接添加进 ...

  2. CDH5X 安装oozie报错To enable Oozie web console install the Ext JS library.

    最近在CDH5.X 安装oozie 服务,服务安装完毕,访问oozie server ui,报如下错误: 页面提示: Oozie web console is disabled.To enable O ...

  3. Oozie如何和安装部署

    1.Oozie的简单介绍: .Oozie是一个工作流引擎服务器,用于运行hadoop map/reduce和hive等任务工作流,同时Oozie还是一个Java web程序,运行在Java Servl ...

  4. 离线安装 Cloudera Manager 5 和 CDH5.10

    关于CDH和Cloudera Manager CDH (Cloudera's Distribution, including Apache Hadoop),是Hadoop众多分支中的一种,由Cloud ...

  5. 【Oozie学习之一】Oozie

    环境 虚拟机:VMware 10 Linux版本:CentOS-6.5-x86_64 客户端:Xshell4 FTP:Xftp4 jdk8 CM5.4 一.简介Oozie由Cloudera公司贡献给A ...

  6. 通过Cloudera Manager安装CDH 5.6

    CDH的简介 大家常常说CDH.其全称是:Cloudera's Distribution Including Apache Hadoop.简单的说是Cloudera公司的Hadoop平台,是在Apac ...

  7. Ambari下安装oozieUI界面无法访问问题

    前言: 其他集群管理工具安装oozie和手动编译安装oozie,关于oozie的UI界面无法访问的问题,大致和该方法类似. 找到下面4步骤里的视图显示的目录,有一个ext-2.2软链指向的路径 lrw ...

  8. 最新版大数据平台安装部署指南，HDP-2.6.5.0，ambari-2.6.2.0

    一.服务器环境配置 1 系统要求 名称 地址 操作系统 root密码 Master1 10.1.0.30 Centos 7.7 Root@bidsum1 Master2 10.1.0.105 Cent ...

  9. 《OD大数据实战》Hue环境搭建

    官网: http://archive.cloudera.com/cdh5/cdh/5/hue-3.7.0-cdh5.3.6/ 一.Hue环境搭建 1. 下载 http://archive.cloude ...

随机推荐

  1. SpringMvc (注解)中的上传文件

    第一步:导入commons-fileupload-1.3.1.jar 和commons-io-2.2.jar 架包 第二步:在applicationContext.xml中 配置 <bean i ...

  2. python6---购物车程序

    #!/usr/bin/env python#_author: harry#date: 2019.11.12# -*- coding: utf-8 -*-product_list =[ ('mac',9 ...

  3. 24. ClustrixDB 持久性配置文件

    这些持久性选项不适用于内存中的表.有关更多信息,请参见内存表. ClustrixDB提供了一个选项,通过指定如何提交事务并使其持久,从而提高性能.当提交成功通知应用程序时,用户可以在提交过程中指定提交 ...

  4. Python天天学_04_基础四

    Python_day_04 金角大王: http://www.cnblogs.com/alex3714/articles/5765046.html ------Python是一个优雅的大姐姐 学习方式 ...

  5. CF873F Forbidden Indices 后缀自动机+水题

    刷刷水~ Code: #include <cstdio> #include <cstring> #include <algorithm> #define N 200 ...

  6. 翻译我去issues提问的回答内容

    我提问的原因主要是我要做.net core ABP的Token刷新功能,基本都根据网上的文章整合进ABP了,在如何存储的时候,总觉得系统的AbpUserTokens表为啥不利用进来,但是又找不到相关介 ...

  7. 工具类-ApplicationContextUtil

    package com.zhouyy.netBank.util; import org.springframework.beans.BeansException; import org.springf ...

  8. webpack 最新版

    之前说过老的版本,webpack@3.8.1 这个版本,现在我们来看看,新版本和老版本的区别 webpack 4 以上的版本 npm 全称 node package manager (node 包管理 ...

  9. debug1: Could not open authorized keys

    ssh登录的时候一直日志一直出现debug1: Could not open authorized keys登录不上,检查文件夹权限都正常用这条命令解决了 restorecon -FRvv /home ...

  10. Uep的ajaxform和ajaxgrid组件获取数据源

    对于ajaxform组件var record = ajaxform.getRecord();var storeId = record.get("storeId");var stor ...