整合apache+tomcat+keepalived实现高可用tomcat集群

Apache是一个强大的Web服务器在处理静态页面、处理大量网络客户请求、支持服务的种类以及可配置方面都有优势，高速并且强壮。但是没有JSP/Servlet的解析能力。整合Apache和Tomcat可以看作是用Tomcat做Apache的jsp/servlet解析插件，将两者优势结合起来。不过Tomcat作为一个Web服务器，本身具备了基本的Web服务功能，在SUN的力推下，将来或许越来越强壮到不需要借助Apache优势的地步。

Tomcat是一个免费的开放源代码的Web 应用服务器，属于轻量级应用服务器，在中小型系统和并发访问用户不是很多的场合下被普遍使用，是开发和调试JSP 程序的首选。Tomcat 很受广大程序员的喜欢，因为它运行时占用的系统资源小，扩展性好，支持负载平衡与邮件服务等开发应用系统常用的功能；而且它还在不断的改进和完善中，任何一个感兴趣的程序员都可以更改它或在其中加入新的功能。

Apache与tomcat的异同：

apache支持静态页，tomcat支持动态的，比如servlet等。

一般使用apache+tomcat的话，apache只是作为一个转发，对jsp的处理是由tomcat来处理的。

apache可以支持php\cgi\perl,但是要使用java的话，你需要tomcat在apache后台支撑，将java请求由apache转发给tomcat处理。

apache是web服务器,Tomcat是应用（java）服务器，它只是一个servlet(jsp也翻译成servlet)容器，可以认为是apache的扩展，但是可以独立于apache运行。

Apache是专门用了提供HTTP服务的，以及相关配置的（例如虚拟主机、URL转发等等）。

Tomcat是Apache组织在符合J2EE的JSP、Servlet标准下开发的一个JSP服务器；

Apache与tomcat整合的好处：

如果客户端请求的是静态页面，则只需要Apache服务器响应请求。

如果客户端请求动态页面，则是Tomcat服务器响应请求。

因为jsp是服务器端解释代码的，这样整合就可以减少Tomcat的服务开销。

一.环境与功能说明

前端：apache配置负载均衡、反向代理（mod_jk模块）

后端：tomcat服务器

前端高可用：通过keepalived实现

系统版本：CentOS 6.4_x86_64（最小化）

apache版本：httpd-2.4.9

arp版本：apr-1.4.6

apr-util版本：apr-util-1.4.1

tomcat版本：apache-tomcat-7.0.53

Jdk版本：jdk-7u40-linux-x64

tomcat-commectors版本：tomcat-connectors-1.2.40

keepalived版本：keepalived-1.2.7-3.el6.x86_64(使用yum安装)

二.安装与配置apache服务器（172.16.10.72、172.16.10.75执行同样的安装与配置，这里只演示一次）

1.安装编译工具和依赖包

yum groupinstall "Development Tools" "Development Libraries" –y

2.下载软件包

yum -y install wget
wget http://mirror.esocc.com/apache//httpd/httpd-2.4.9.tar.gz
wget http://mirrors.axint.net/apache//apr/apr-1.4.6.tar.gz
wget http://mirrors.axint.net/apache//apr/apr-util-1.4.1.tar.gz

3.安装apache

3.1.安装apr和apr-tuil

httpd-2.4.9需要较新版本的apr和apr-util，因此需要事先对其进行升级。升级方式有两种，一种是通过源代码编译安装，一种是直接升级rpm包。以下是采用前者进行安装：

# tar xf apr-1.4..tar.gz
# tar xf apr-util-1.4..tar.gz
# cd apr-1.4.
# ./configure --prefix=/usr/local/apr
# make
# make install
# cd ..
 
# cd apr-util-1.4.
# ./configure --prefix=/usr/local/apr-util --with-apr=/usr/local/apr
# make
# make install
# cd ..

3.2.安装http-2.4.9

# ./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd24 --enable-so --enable-ssl --enable-cgi --enable-rewrite --with-zlib --with-pcre --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util/ --enable-modules=most --enable-mpms-shared=all
# make && make install

4.配置SysV服务脚本/etc/init.d/httpd

# cp build/rpm/httpd.init /etc/init.d/httpd
#修改如下内容：

# vi /etc/init.d/httpd
httpd=${HTTPD-/usr/local/apache/bin/httpd}
pidfile=${PIDFILE-/usr/local/apache/logs/${prog}.pid}
lockfile=${LOCKFILE-/var/lock/subsys/${prog}}
RETVAL=
# check for 1.3 configuration
check13 () {
        CONFFILE=/etc/httpd24/httpd.conf
 
#修改环境变量并把Httpd加入到系统服务

# echo "PATH=/usr/local/apache/bin:$PATH" >> /etc/profile.d/http.sh
  # . /etc/profile.d/http.sh
  # ln -s /usr/local/apache/include/ /usr/include/httpd
  # chkconfig --add httpd

5.安装tomcat-connectors

# wget http://apache.fayea.com/apache-mirror/tomcat/tomcat-connectors/jk/tomcat-connectors-1.2.40-src.tar.gz
# tar xf tomcat-connectors-1.2.-src.tar.gz
# cd tomcat-connectors-1.2.-src/native/
# ./configure --with-apxs=/usr/local/apache/bin/apxs
# make
# make install

安装完成后会生成如下文件

# ls /usr/local/apache/modules/|grep 'mod_jk'
mod_jk.so

启动httpd服务

# service httpd start

执行./configure --with-apxs=/usr/local/apache/bin/apxs时可能会遇到以下报错：

need to check for Perl first, apxs depends on it...
checking for perl... /usr/bin/perl
could not find /usr/local/apache/bin/apxs
configure: error: You must specify a valid --with-apxs path

解法办法：

vi /usr/local/apache/bin/apxs
把第一行改成：#!/usr/bin/perl -w

另外检查系统中是否已经安装过perl，如果没有安装，用yum安装即可。

三.安装与配置tomcat服务器（172.16.10.61、172.16.10.62执行同样的安装，这里只演示一次）

1.安装JDK

下载地址：http://download.oracle.com/

# rpm -ivh jdk-7u40-linux-x64.rpm
# vim /etc/profile.d/java.sh
export JAVA_HOME=/usr/java/latest
export PATH=$JAVA_HOME/bin:$PATH
# . /etc/profile.d/java.sh
# java -version
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) -Bit Server VM (build 24.0-b56, mixed mode)

2.安装Tomcat

下载地址：http://apache.fayea.com/apache-mirror/tomcat/tomcat-7/v7.0.53/bin/apache-tomcat-7.0.53.tar.gz

# tar xf apache-tomcat-7.0..tar.gz -C /usr/local/
# cd /usr/local/
# ln -s apache-tomcat-7.0. tomcat
# vim /etc/profile.d/tomcat.sh
export CATALINA_HOME=/usr/local/tomcat
export PATH=$CATALINA_HOME/bin:$PATH
# . /etc/profile.d/tomcat.sh
# /usr/local/tomcat/bin/catalina.sh version
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/java/latest
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Server version: Apache Tomcat/7.0.
Server built:   Mar   ::
Server number:  7.0.53.0
OS Name:        Linux
OS Version:     2.6.-.el6.x86_64
Architecture:   amd64
JVM Version:    1.7.0_40-b43
JVM Vendor:     Oracle Corporation

3.启动Tomcat服务并访问测试，默认访问端口为8080：

# /usr/local/tomcat/bin/catalina.sh start
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/java/latest
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
# ss -antpl | grep
                                 :::                         :::*      users:(("java",,))

4.开启Tomcat管理及状态页面

# vim /usr/local/tomcat/conf/tomcat-users.xml(在</tomcat-users>标签上一行添加如下两行)
  <role rolename="manager-gui"/>
  <user username="tomcat" password="tomcat" roles="manager-gui"/> #后台管理页面登录名及密码均为tomcat
# /usr/local/tomcat/bin/catalina.sh stop
# /usr/local/tomcat/bin/catalina.sh start

5.访问Tomcat后台管理

四.配置Apache使用mod_jk模块实现代理及负载均衡

1.编辑Apache主配置文件，实现代理功能（172.16.10.72）

echo "Include /etc/httpd24/extra/mod_jk.conf" >> /etc/httpd24/httpd.conf  #将此指令添加到文件末尾即可

2.创建mod_jk.conf及workers.properties文件

# cat >>/etc/httpd24/extra/mod_jk.conf <<EOF
LoadModule  jk_module  modules/mod_jk.so
JkWorkersFile  /etc/httpd24/extra/workers.properties
JkLogFile  logs/mod_jk.log
JkLogLevel  debug
JkMount  /*  TomcatA
JkMount  /status/  stat1
EOF
 
# cat >>/etc/httpd24/extra/workers.properties <<EOF
worker.list=TomcatA,stat1
worker.TomcatA.port=8009
worker.TomcatA.host=172.16.10.61
worker.TomcatA.type=ajp13
worker.TomcatA.lbfactor=1
worker.stat1.type = status
EOF

3.编辑httpd.conf修改ServerName并重启httpd服务

#vi /etc/httpd24/httpd.conf
ServerName 172.16.10.72:80
# service httpd restart

4.修改tomcat服务器（172.16.10.61），添加测试页面

# cd /usr/local/tomcat/conf/
# cp server.xml server.xml.bak
# vi server.xml
<Engine name="Catalina" defaultHost="172.16.10.72">
<Host name="172.16.10.72"  appBase="/web/eivll0m"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="eivll0m_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
        <Context path="" docBase="/web/eivll0m" />
      </Host>

创建网站存放目录并创建测试页

# mkdir -p /web/eivll0m
# vi /web/eivll0m/index.jsp
<%@ page language="java" %>
<html>
  <head><title>TomcatA</title></head>
  <body>
    <h1><font color="red">TomcatA </font></h1>
    <table align="centre" border="">
      <tr>
        <td>Session ID</td>
    <% session.setAttribute("abc","abc"); %>
        <td><%= session.getId() %></td>
      </tr>
      <tr>
        <td>Created on</td>
        <td><%= session.getCreationTime() %></td>
     </tr>
    </table>
  </body>
</html>

重启tomcat服务并查看是否启动成功

# catalina.sh stop
# catalina.sh configtest
# catalina.sh start
# ss -antpl | grep java
                     ::ffff:127.0.0.1:                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))

5.访问Apache主机(172.16.10.72)，验证是否代理成功

6.修改Apache配置文件(mod_jk.conf、workers.properties)，实现负载均衡

# cat >>/etc/httpd24/extra/mod_jk.conf <<EOF
LoadModule  jk_module  modules/mod_jk.so
JkWorkersFile  /etc/httpd24/extra/workers.properties
JkLogFile  logs/mod_jk.log
JkLogLevel  debug
JkMount  /*  lbcluster
JkMount  /status/  stat1
EOF

# cat >>/etc/httpd/extra/workers.properties <<EOF
worker.list=lbcluster,stat1
worker.TomcatA.port=8009
worker.TomcatA.host=172.16.10.61
worker.TomcatA.type=ajp13
worker.TomcatA.lbfactor=1
worker.TomcatB.port = 8009
worker.TomcatB.host=172.16.10.62
worker.TomcatB.type = ajp13
worker.TomcatB.lbfactor = 1
worker.lbcluster.type = lb
worker.lbcluster.sticky_session = 0  #取值为{1|0}1将用户session与后端服务器绑定，0为不绑定，如果支持session复制或session共享可以设置为0
worker.lbcluster.balance_workers = TomcatA, TomcatB
worker.stat1.type = status
EOF

7.修改tomcat服务器（172.16.10.62），添加测试页面

# cd /usr/local/tomcat/conf/
# cp server.xml server.xml.bak
# vi server.xml
<Engine name="Catalina" defaultHost="172.16.10.72">
<Host name="172.16.10.72"  appBase="/web/eivll0m"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="eivll0m_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
        <Context path="" docBase="/web/eivll0m" />
      </Host>

创建网站存放目录并创建测试页

# mkdir -p /web/eivll0m
# vi /web/eivll0m/index.jsp
<%@ page language="java" %>
<html>
  <head><title>TomcatB</title></head>
  <body>
    <h1><font color="blue">TomcatB </font></h1>
    <table align="centre" border="">
      <tr>
        <td>Session ID</td>
    <% session.setAttribute("abc","abc"); %>
        <td><%= session.getId() %></td>
      </tr>
      <tr>
        <td>Created on</td>
        <td><%= session.getCreationTime() %></td>
     </tr>
    </table>
  </body>
</html>

重启tomcat服务并查看是否启动成功

# catalina.sh stop
# catalina.sh configtest
# catalina.sh start
# ss -antpl | grep java
                     ::ffff:127.0.0.1:                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))

8.访问Apache服务器（172.16.10.72），测试是否实现负载均衡（如下表示测试成功）

五.安装并配置Keepalived实现高可用

1.在apache服务器（172.16.10.72、172.16.10.75）安装keepalived

yum -y install keepalived

2.配置Keepalived(172.16.10.72)

# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
 
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from admin@eivll0m.com
   smtp_server 172.16.10.0
   smtp_connect_timeout
   router_id LVS_DEVEL
}
 
vrrp_script chk_httpd {
    script "killall -0 httpd"
    interval
    weight -
}
 
vrrp_instance httpd_1 {
    state MASTER
    interface eth0
    virtual_router_id
    priority 100           #定义优先级
    advert_int
    authentication {
        auth_type PASS
        auth_pass
    }
    virtual_ipaddress {
    172.16.10.77           #vip
    }
    track_script {
    chk_httpd
    }
}
 
# service keepalived start  #启动keepalived服务
# chkconfig keepalived on

3.配置Keepalived(172.16.10.75)

# vi /etc/keepalived/keepalived.conf
 
! Configuration File for keepalived
 
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from admin@eivll0m.com
   smtp_server 172.17.10.0
   smtp_connect_timeout
   router_id LVS_DEVEL
}
 
vrrp_script chk_httpd {
    script "killall -0 httpd"
    interval
    weight -
}
 
vrrp_instance httpd_1 {
    state BACKUP
    interface eth0
    virtual_router_id
    priority 99            #定义优先级
    advert_int
    authentication {
        auth_type PASS
        auth_pass
    }
    virtual_ipaddress {
    172.16.10.77           #vip
    }
    track_script {
    chk_httpd
    }
}

# service keepalived start  
# chkconfig keepalived on

4.编辑httpd.conf(172.16.10.72)，并拷贝至172.16.10.75

# vi /etc/httpd24/httpd.conf
ServerName www.eivll0m.com:
# service httpd restart
# cd /etc/httpd24/
# scp httpd.conf 172.16.10.75:/etc/httpd24/
# cd extra/
# scp mod_jk.conf workers.properties 172.16.10.75:/etc/httpd24/extra/
# service httpd restart #两台apache服务器分别执行

5.编辑tomcat服务器(172.17.10.61和172.16.10.62)的server.xml

# vi server.xml
<Engine name="Catalina" defaultHost="www.eivll0m.com">
<Host name="www.eivll0m.com"  appBase="/web/eivll0m"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="eivll0m_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
        <Context path="" docBase="/web/eivll0m" />
      </Host>
# catalina.sh stop
# catalina.sh start

5.通过www.eivll0m.com(172.16.10.77)进行访问测试

注意：测试时关闭所有服务器iptables，如果没有做DNS解析，可在测试机上修改hosts文件来实现。

6.模拟一台apache故障，验证keepalived是否可以成功转移服务

# ip addr show eth0   #172.16..72上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP qlen
    link/ether :0c::e9:b8: brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.72/ brd 172.16.10.255 scope global eth0
    inet 172.16.10.77/ scope global eth0
    inet6 fe80::20c:29ff:fee9:b852/ scope link tentative dadfailed
       valid_lft forever preferred_lft forever
 
# ip addr show eth0 #172.16..75上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP qlen
    link/ether :0c::c8:1a: brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.75/ brd 172.16.10.255 scope global eth0
    inet 192.168.200.16/ scope global eth0
    inet 192.168.200.17/ scope global eth0
    inet 192.168.200.18/ scope global eth0
    inet6 fe80::20c:29ff:fec8:1a08/ scope link tentative dadfailed
       valid_lft forever preferred_lft forever
 
# service httpd stop  #停止172.16.10.72上的httpd服务
# ip addr show eth0   #172.16..72上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP qlen
    link/ether :0c::e9:b8: brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.72/ brd 172.16.10.255 scope global eth0
    inet6 fe80::20c:29ff:fee9:b852/ scope link tentative dadfailed
       valid_lft forever preferred_lft forever
 
# ip addr show eth0  #172.16.10.75上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP qlen
    link/ether :0c::c8:1a: brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.75/ brd 172.16.10.255 scope global eth0
    inet 192.168.200.16/ scope global eth0
    inet 192.168.200.17/ scope global eth0
    inet 192.168.200.18/ scope global eth0
    inet 172.16.10.77/ scope global eth0       #vip已经成功转移过来
    inet6 fe80::20c:29ff:fec8:1a08/ scope link tentative dadfailed
       valid_lft forever preferred_lft forever

通过上面的步骤我们可以看到，在172.16.10.72上停止apache服务，keepalived会检测到，另一台keepalived服务器会接管vip，web服务不会中断，从而实现了高可用性。

7.配置tomcat实现session共享

7.1.配置172.16.10.61这台tomcat服务器

# vi /usr/local/tomcat/conf/server.xml  #在<Engine>标签内（对所有主机生效）添加或修改如下内容，另外也可以添加在<Host>标签内，对当前主机生效。并去掉此段前后注释
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                 channelSendOptions="">
          <Manager className="org.apache.catalina.ha.session.DeltaManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"/>
          <Channel className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership className="org.apache.catalina.tribes.membership.McastService"
                        address="224.0.0.1"   #组播地址
                        port=""
                        frequency=""
                        dropTime=""/>
            <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="172.16.10.61"
                      port=""
                      autoBind=""
                      selectorTimeout=""
                      maxThreads=""/>
            <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
              <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
            </Sender>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
          </Channel>
          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                 filter=""/>
          <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
          <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                    tempDir="/tmp/war-temp/"
                    deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"
                    watchEnabled="false"/>
          <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
          <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
        </Cluster>

# mkdir /web/eivll0m/WEB-INF
# cp /usr/local/tomcat/conf/web.xml /web/eivll0m/WEB-INF/
# vim /web/eivll0m/WEB-INF/web.xml
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
                      http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
  version="3.0">
<distributable/>                      #添加此行内容

# catalina.sh stop
# catalina.sh start
# ss -tanlp | grep java
                     ::ffff:127.0.0.1:                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))
                 ::ffff:172.16.10.61:                         :::*      users:(("java",,))

7.2.配置172.16.10.62这台tomcat服务器

# scp /usr/local/tomcat/conf/server.xml 172.16.10.62:/usr/local/tomcat/conf/
# scp -r /web/eivll0m/WEB-INF 172.16.10.62:/web/eivll0m/

# vi /usr/local/tomcat/conf/server.xml
                      address="172.16.10.62"   #更改IP
                      port=""
                      autoBind=""
                      selectorTimeout=""
                      maxThreads=""/>
# catalina.sh stop
# catalina.sh start
# ss -antpl|grep java
                     ::ffff:127.0.0.1:                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))
                                 :::                         :::*      users:(("java",,))
                 ::ffff:172.16.10.62:                         :::*      users:(("java",,))

注：因为tomcat的session同步功能需要用到组播，windows默认情况下是开通组播服务的，但是linux默认情况下并没有开通，可以通过指令打开route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0，如果需要服务器启动时即开通组播需在/etc/sysconfig/static-routes文件内加入eht0 net 224.0.0.0 netmask 240.0.0.0。

7.3.验证session共享是否实现

通过以上截图可以看出session共享已经实现。