wrapper x64 版本发布到centos
背景:
项目需要在spark任务提交服务器节点上自动提交任务到spark集群上。因此创建了一个固定时间监控任务项目,使用timer定时监控oracle数据库中是否有spark提交任务,如果有spark提交任务,就使用java.lang.ProcessBuilder对象调sh文件(spark-submit.sh),来提交任务。
部署项目、wrapper x64到CentOS服务器(spark任务提交服务器)
1、首先需要下载wrapper x64 linux版本上的安装包:http://pan.baidu.com/s/1nvc7JC1
2、将wrapper x64 linux服务器工具包下载到centos x64服务上,在centos x64上存储路径为:/app/tt/service/wrapper。
[tt@vh01 wrapper]$ ls
bin conf doc lib logs src
3、把使用timer实现的固定时间间隔执行监控提交spark任务的java项目发布成My_AutoExecutePlan.jar包,上传到/app/tt/service/wrapper/lib下,并把其依赖的包也上传到/app/tt/service/wrapper/lib下
[tt@vh01 wrapper]$ cd lib
[tt@vh01 lib]$ ls
commons-logging.jar fastjson-1.2.10.jar libwrapper.so ojdbc7.jar wrapper.jar
downjar.sh My_AutoExecutePlan.jar log4j.jar wrapperdemo.jar wrappertest.jar
备注:
- 其中标记为灰色的jar包为wrapper默认的自带的jar包,不可以删除;
- 其中标记为红色的jar包为固定时间监控的jar包和其所以来的jar包;
- 其中标记为蓝色的sh为下载文件的shell命令文件。
4、进入到/app/tt/service/wrapper/bin下,创建script文件夹,把需要spark-submit.sh文件放到/app/tt/service/wrapper/bin/script路径下;创建conf文件把log4.properties、jdbc.properties等文件存放到/app/tt/service/wrapper/bin/conf下;把spark-submit.sh所需要的spark任务包ParserLogOnSpark.jar包存放到/app/tt/service/wrapper/bin下:
[tt@vh01 wrapper]$ cd bin
[tt@vh01 bin]$ ls
conf scripts
demoapp ParserLogOnSpark.jar wrapper SparkSubmitService
SparkSubmitService文件为服务器启动文件,它是由demoapp文件复制而来:
# Application
APP_NAME="SparkSubmitService"
APP_LONG_NAME="Task_SparkSubmitService" # If uncommented (and set to false), APP_NAME and APP_LONG_NAME will no longer
# be passed to the wrapper. See documentation for details.
#APP_NAME_PASS_TO_WRAPPER=false # Wrapper
WRAPPER_CMD="./wrapper"
WRAPPER_CONF="../conf/wrapper.conf"
注意:
- 该文件是从demoapp文件复制而来,名称只需要有意义就可以;
- 文件内容中只需要修改"APP_NAME" & "APP_LONG_NAME"的名称为一个有意义的名称;
- 该文件告诉了另外一个重要信息就是真正wrapper配置文件是在/app/tt/service/wrapper/conf下的wrapper.conf
5、配置/app/tt/service/wrapper/conf下的wrapper.conf文件内容:
#encoding=UTF-8
# Configuration files must begin with a line specifying the encoding
# of the the file.
#
# NOTE - Please use src/conf/wrapper.conf.in as a template for your
# own application rather than the values used for the
# TestWrapper sample. #********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
# Professional and Standard Editions of the Wrapper require a valid
# License Key to start. Licenses can be purchased or a trial license
# requested on the following pages:
# http://wrapper.tanukisoftware.com/purchase
# http://wrapper.tanukisoftware.com/trial # Include file problems can be debugged by removing the first '#'
# from the following line:
##include.debug # The Wrapper will look for either of the following optional files for a
# valid License Key. License Key properties can optionally be included
# directly in this configuration file.
#include ../conf/wrapper-license.conf
#include ../conf/wrapper-license-%WRAPPER_HOST_NAME%.conf # The following property will output information about which License Key(s)
# are being found, and can aid in resolving any licensing problems.
#wrapper.license.debug=TRUE #********************************************************************
# Wrapper Localization
#********************************************************************
# Specify the locale which the Wrapper should use. By default the system
# locale is used.
#wrapper.lang=en_US # en_US or ja_JP # Specify the location of the Wrapper's language resources. If these are
# missing, the Wrapper will default to the en_US locale.
wrapper.lang.folder=../lang #********************************************************************
# Wrapper Java Properties
#********************************************************************
# Java Application
# Locate the java binary on the system PATH:
#wrapper.java.command=java
# Specify a specific java binary:
#set.JAVA_HOME=/java/path
wrapper.java.command=/usr/local/jdk1.8.0_77/bin/java # Tell the Wrapper to log the full generated Java command line.
#wrapper.java.command.loglevel=INFO # Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp # Java Classpath (include wrapper.jar) Add class path elements as
# needed starting from 1
wrapper.java.classpath.1=../lib/My_AutoExecutePlan.jar
wrapper.java.classpath.2=../lib/wrapper.jar
wrapper.java.classpath.3=../lib/commons-logging.jar
wrapper.java.classpath.4=../lib/fastjson-1.2.10.jar
wrapper.java.classpath.5=../lib/log4j.jar
wrapper.java.classpath.6=../lib/ojdbc7.jar
wrapper.java.classpath.7=/usr/local/jdk1.8.0_77/lib/dt.jar
wrapper.java.classpath.8=/usr/local/jdk1.8.0_77/lib/tools.jar # Java Library Path (location of Wrapper.DLL or libwrapper.so)
wrapper.java.library.path.1=../lib # Java Bits. On applicable platforms, tells the JVM to run in 32 or 64-bit mode.
wrapper.java.additional.auto_bits=TRUE # Java Additional Parameters
wrapper.java.additional.1= # Initial Java Heap Size (in MB)
#wrapper.java.initmemory=3 # Maximum Java Heap Size (in MB)
#wrapper.java.maxmemory=64 # Application parameters. Add parameters as needed starting from 1
wrapper.app.parameter.1=My Project Jar Main Class Name #********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Enables Debug output from the Wrapper.
# wrapper.debug=TRUE # Format of output for the console. (See docs for formats)
wrapper.console.format=PM # Log Level for console output. (See docs for log levels)
wrapper.console.loglevel=INFO # Log file to use for wrapper output logging.
wrapper.logfile=../logs/wrapper.log # Format of output for the log file. (See docs for formats)
wrapper.logfile.format=LPTM # Log Level for log file output. (See docs for log levels)
wrapper.logfile.loglevel=INFO # Maximum size that the log file will be allowed to grow to before
# the log is rolled. Size is specified in bytes. The default value
# of 0, disables log rolling. May abbreviate with the 'k' (kb) or
# 'm' (mb) suffix. For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=0 # Maximum number of rolled log files which will be allowed before old
# files are deleted. The default value of 0 implies no limit.
wrapper.logfile.maxfiles=0 # Log Level for sys/event log output. (See docs for log levels)
wrapper.syslog.loglevel=NONE #********************************************************************
# Wrapper General Properties
#********************************************************************
# Allow for the use of non-contiguous numbered properties
wrapper.ignore_sequence_gaps=TRUE # Do not start if the pid file already exists.
wrapper.pidfile.strict=TRUE # Title to use when running as a console
wrapper.console.title=SparkSubmitService Wrapper Application #********************************************************************
# Wrapper JVM Checks
#********************************************************************
# Detect DeadLocked Threads in the JVM. (Requires Standard Edition)
wrapper.check.deadlock=TRUE
wrapper.check.deadlock.interval=10
wrapper.check.deadlock.action=RESTART
wrapper.check.deadlock.output=FULL # Out Of Memory detection.
# (Ignore output from dumping the configuration to the console. This is only needed by the TestWrapper sample application.)
wrapper.filter.trigger.999=wrapper.filter.trigger.*java.lang.OutOfMemoryError
wrapper.filter.allow_wildcards.999=TRUE
wrapper.filter.action.999=NONE
# Ignore -verbose:class output to avoid false positives.
wrapper.filter.trigger.1000=[Loaded java.lang.OutOfMemoryError
wrapper.filter.action.1000=NONE
# (Simple match)
wrapper.filter.trigger.1001=java.lang.OutOfMemoryError
# (Only match text in stack traces if -XX:+PrintClassHistogram is being used.)
#wrapper.filter.trigger.1001=Exception in thread "*" java.lang.OutOfMemoryError
#wrapper.filter.allow_wildcards.1001=TRUE
wrapper.filter.action.1001=RESTART
wrapper.filter.message.1001=The JVM has run out of memory. #********************************************************************
# Wrapper Email Notifications. (Requires Professional Edition)
#********************************************************************
# Common Event Email settings.
#wrapper.event.default.email.debug=TRUE
#wrapper.event.default.email.smtp.host=<SMTP_Host>
#wrapper.event.default.email.smtp.port=25
#wrapper.event.default.email.subject=[%WRAPPER_HOSTNAME%:%WRAPPER_NAME%:%WRAPPER_EVENT_NAME%] Event Notification
#wrapper.event.default.email.sender=<Sender email>
#wrapper.event.default.email.recipient=<Recipient email> # Configure the log attached to event emails.
#wrapper.event.default.email.attach_log=TRUE
#wrapper.event.default.email.maillog.lines=50
#wrapper.event.default.email.maillog.format=LPTM
#wrapper.event.default.email.maillog.loglevel=INFO # Enable specific event emails.
#wrapper.event.wrapper_start.email=TRUE
#wrapper.event.jvm_prelaunch.email=TRUE
#wrapper.event.jvm_start.email=TRUE
#wrapper.event.jvm_started.email=TRUE
#wrapper.event.jvm_deadlock.email=TRUE
#wrapper.event.jvm_stop.email=TRUE
#wrapper.event.jvm_stopped.email=TRUE
#wrapper.event.jvm_restart.email=TRUE
#wrapper.event.jvm_failed_invocation.email=TRUE
#wrapper.event.jvm_max_failed_invocations.email=TRUE
#wrapper.event.jvm_kill.email=TRUE
#wrapper.event.jvm_killed.email=TRUE
#wrapper.event.jvm_unexpected_exit.email=TRUE
#wrapper.event.wrapper_stop.email=TRUE # Specify custom mail content
wrapper.event.jvm_restart.email.body=The JVM was restarted.\n\nPlease check on its status.\n #********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
# using this configuration file has been installed as a service.
# Please uninstall the service before modifying this section. The
# service can then be reinstalled. # Name of the service
wrapper.name=SparkSubmitService # Display name of the service
wrapper.displayname=Task_SparkSubmitService # Description of the service
wrapper.description=Submit Spark Job # Service dependencies. Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1= # Mode in which the service is installed. AUTO_START, DELAY_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START # Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false
备注:
其中放大加粗红色的配置项是需要需要为自己项目的配置向,其他项可以不修改。
启动服务器:
1、启动前需要先修改权限为777
chmod SparkSubmitService
2、Linux服务启动命令:
[tt@vh01 bin]$ ./SparkSubmitService start
Starting Task_SparkSubmitService...
Waiting for Task_SparkSubmitService.....
running: PID:16531
3、Linux服务停止命令:
[tt@vh01 bin]$ ./SparkSubmitService stop
Stopping Task_SparkSubmitService...
Stopped Task_SparkSubmitService.
Linux服务命令help列表:
[tt@vh01 bin]$ ./SparkSubmitService
Usage: ./SparkSubmitService [ console | start | stop | restart | condrestart | status | install | installstart | remove | dump ] Commands:
console Launch in the current console.
start Start in the background as a daemon process.
stop Stop if running as a daemon or in another console.
restart Stop if running and then start.
condrestart Restart only if already running.
status Query the current status.
install Install to start automatically when system boots.
installstart Install and start running as a daemon process.
remove Uninstall.
dump Request a Java thread dump if running.
4、Linux服务状态查看命令:
[tt@vh01 bin]$ ./SparkSubmitService status
Task_SparkSubmitService is not running.
5、CentOS下设置SparkSubmitService为开启启动项:
需要在wrapper的bin目录下执行以下命令:
cp SparkSubmitService /etc/init.d/SparkSubmitService
cd /etc/rc.d/init.d
chmod +x SparkSubmitService
/sbin/chkconfig --del SparkSubmitService
/sbin/chkconfig --add SparkSubmitService
wrapper x64 版本发布到centos的更多相关文章
- ASP.NET Core 2.0 : 九.从Windows发布到CentOS的跨平台部署
本文聊一下如何在Windows上用VS开发并发布, 然后将其部署到CentOS上.对于我们一些常在Windows上逛的来说,CentOS用起来还真有些麻烦.MSDN官方有篇文章大概讲了一下(链接),按 ...
- glibc 各版本发布时间以及内核默认glibc版本
最近有些软件要求glibc 2.14+,centos 6.x自带的版本是2.12的,特查了下glibc 各版本发布时间以及与对应的内核,如下: Complete glibc release histo ...
- Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践(二)
Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践(一) 接上一篇 3. Nginx配置反向代理 3.1 cnetos 安装nginx 首先,我们需要在服务器上安装N ...
- Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践(一)
环境 本地 win7 服务器:Virtual Box 上的Centos ssh工具: Xshell 文件传输: xftp 1.在本地创建asp.net core应用发布 1.1 使用Vs2017 新建 ...
- RabbitMQ 3.6.1 升级至 3.7.9 版本(Windows 升级至Centos)
随着公司业务量的增加,原本部署在Windows服务器的RabbitMQ集群(3.6.1)总是出现莫名其妙的问题,经查询官方Issue,确认是RabbitMQ 3.6.1 版本的bug.查看从3.6.1 ...
- .net core web发布到CentOS汇总
直到今天我的博客终于可以见世人了,中间懒了很长一段时间,什么也没干,也没怎么学习,前段时间也是各种折腾,无心学习.本篇主要汇总下从一开始到现在遇到的问题汇总,作为学习笔记.我的博客就是我的学习笔记,因 ...
- Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践
原文:Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践 环境 本地 win7 服务器:Virtual Box 上的Centos ssh工具: Xshell 文件传输 ...
- .Net Core 2.0发布到 CentOS
上一篇 在CentOS 7中 使用 Nginx 反代 .Net Core 中创建的项目是创建的默认项目,现在我们来将我们开发的项目发布到 CentOs 中,反代还是使用 Nginx 1.创建一个 .N ...
- Visual Lab Online —— Alpha版本发布声明
Visual Lab Online -- Alpha版本发布声明 项目 内容 班级:北航2020春软件工程 博客园班级博客 作业:Alpha阶段发布声明 发布声明 目录 Visual Lab Onli ...
随机推荐
- 边做边学入门微信小程序之仿豆瓣评分
微信小程序由于适用性强.逻辑简要.开发迅速的特性,叠加具有海量活跃用户的腾讯公司背景,逐渐成为了轻量级单一功能应用场景的较佳承载方式,诸如电影购票.外卖点餐.移动商城.生活服务等场景服务提供商迅速切入 ...
- 堆排序(Java数组实现)
堆排序:利用大根堆 数组全部入堆,再出堆从后向前插入回数组中,数组就从小到大有序了. public class MaxHeap<T extends Comparable<? super T ...
- 1-2 Spring 的基本使用
1.Spring容器的配置文件applicationContext.xml的引入 名称可以自己定义 <?xml version="1.0" encoding="UT ...
- 笔记:Spring Cloud Hystrix 异常处理、缓存和请求合并
异常处理 在 HystrixCommand 实现的run方法中抛出异常,除了 HystrixBadRequestException之外,其他异常均会被Hystrix 认为命令执行失败并触发服务降级处理 ...
- Spring Boot 引入自定义yml
喜欢yml配置文件格式的人性化,也喜欢properties配置文件管理方式的人性化, 那么下面我们就来看一下 yml 是如何配置和使用类似properties管理方式的人性化. 配置文件 设置Spri ...
- java必学的5种排序算法
第一种冒泡排序 第二种 选择排序 第三种.插入排序
- Be Better , Be Better
Be Better! 这不是一道题,只是我的flag.初三寒假,一个本应该对着计算机翻天覆雨的假期,我在鬼班撸高中课...其实感触是从初中课得来的.有些事,以前我说是我不懂,现在我不说不是我不懂.Ju ...
- memcache图形化管理工具MemAdmin
给大家介绍一款 memcache图形化管理工具: MemAdmin 下载地址: http://www.junopen.com/memadmin/ wget http://www.junopen.com ...
- Nginx配置文件nginx.conf中文详解(转)
#定义Nginx运行的用户和用户组user www www; #nginx进程数,建议设置为等于CPU总核心数.worker_processes 8; #全局错误日志定义类型,[ debug | in ...
- python内置函数 divmod()
先来看一下builtins.py中的代码: def divmod(x, y): # known case of builtins.divmod """ Return th ...