使用Mod_JK链接Apache和Tomcat

There are many potential problems associated with the default configuration of mod_jk.  Let's say it is perfectly adequate for a very low traffic website, but when pushing any moderate to high load to mod_jk, there will be connection problems.  This is not due to any bug in mod_jk whatsoever, however, it is because the default configuration makes no assumption about your existing hardware or potential load, so, therefore, it is not tuned accordingly.

Note that the configuration recommendations here are optimal as a base configuration to avoid many of the common problems users experience with mod_jk. There exist many other useful optimizations, but these depend on the environment and web application in use. See http://tomcat.apache.org/connectors-doc/reference/workers.html for details on all available mod_jk properties.

Let's take a look at a typical default configuration for Apache/Tomcat/mod_jk:

workers.properties

worker.list=loadbalancer,status

worker.node1.port=8009
worker.node1.host=node1.mydomain.com
worker.node1.type=ajp13
worker.node1.lbfactor=1

worker.node2.port=8009
worker.node2.host= node2.mydomain.com
worker.node2.type=ajp13
worker.node2.lbfactor=1

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2

worker.status.type=status

JBoss Web's (Tomcat) server.xml AJP snippet:

<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3"
         emptySessionPath="true" enableLookups="false" redirectPort="8443" ></Connector>

Apache's httpd.conf:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

The above configuration, under load, may cause mod_jk to be very slow and unresponsive, cause http errors, and cause half-closed connections.   These problems can arise because there are no connection timeouts specified to take care of orphaned connections, no error handling properties defined in workers.properties, and no connection limits set in Apache and Tomcat.

First off, lets take care of Tomcat:

Configuring server.xml:

The main concern with server.xml is setting the connectionTimeout which

sets the SO_TIMEOUT of the underlying socket.  So when a connection in

Tomcat hasn't had a request in the amount of time specified by

connectionTimeout, then the connection dies off.  This is necessary because if the connection hasn't been used for a certain period of

time then there is the chance that it is half-close on the mod_jk end.

If the connection isn't closed there will be an inflation of threads

which can over time hit the maxThreads count in Tomcat then Tomcat will

not be able to accept any new connections.  A connectionTimeout of 600000 (10 minutes) is a good number to start out with.  There may be a situation where the connections are not being recycled fast enough, in this instance the connectionTimeout could be lowered to 60000 or 1 minute.

When setting connectionTimeout in Tomcat, mod_jk should also have

connect_timeout/prepost_timeout set, which allows detection that the

Tomcat connection has been closed and preventing a retry request.

The recommended value of maxThreads is 200 per CPU, so here we assume the server is a single core machine.  If it has been quad core, we could push that value to 800, and more depending on RAM and other machine specs.

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="200"
           connectionTimeout="600000"></Connector>

Configuring workers.properties:

See comments inline.

worker.list=loadbalancer,status

worker.template.port=8009
worker.template.type=ajp13
worker.template.lbfactor=1
#ping_timeout was introduced in 1.2.27
worker.template.ping_timeout=1000
#ping_mode was introduced in 1.2.27, if not using 1.2.27 please specify connect_timeout=10000 and prepost_timeout=10000 as an alternative
worker.template.ping_mode=A
worker.template.socket_timeout=10
#It is not necessary to specify connection_pool_timeout if you are running the worker mpm 
worker.template.connection_pool_timeout=600

#Referencing the template worker properties makes the workers.properties shorter and more concise
worker.node1.reference=worker.template
worker.node1.host=192.168.1.2

worker.node2.reference=worker.template
worker.node2.host=192.168.1.3

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=True

worker.status.type=status

The key points in the above workers.properties is we've added limits for the connections mod_jk makes.  With the base configuration, socket timeouts default to infinite. The other important properties are ping_mode and ping_timeout which handle probing a connection for errors and connection_pool_timeout which must be set to equal server.xml's connectionTimeout when using the prefork mpm.  When these two values are the same, after a connection has been inactive for x amount of time, the connection in mod_jk and Tomcat will be closed at the same time, preventing a half-closed connection.

Configuring Apache

Make note that maxThreads for the AJP connection should coincide with

the MaxClients set in Apache's httpd.conf.  MaxClients needs to be set

in the correct module in Apache.

This can be determined by running httpd -V:

# httpd -V

Server version: Apache/2.2.3
Server built:   Sep 11 2006 09:43:05
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.8
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

Which tells me the Server MPM is Prefork.  This is not always 100% accurate so you should also view the output of /etc/sysconfig/httpd to see if the following line is there: HTTPD=/usr/sbin/httpd.worker.  If it is commented out you are running prefork, otherwise if uncommented worker.

httpd.conf:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
MaxClients       200
MaxRequestsPerChild  0
</IfModule>

Or if Apache is using worker, it is
<IfModule worker.c>
StartServers         2
MaxClients         200
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

MaxRequestsPerChild is 0, this is the recommended value when using

mod_jk as mod_jk keeps open persistent connections.  The key values in

the above configuration are MaxClients and MaxRequestsPerChild, the rest

of the values are left as default.  Note that MaxRequestsPerChild is

recommended to be 0 however the value may need to be greater than 0

depending on if Apache is used for other modules also, especially in the

case of resource leakage.

Advanced worker-mpm Configuration

To get the most out of your mod_jk setup you should be using Apache's worker mpm which provides a definite performance improvement over the prefork mpm.  The following section will detail how to configure Apache/mod_jk/Tomcat with the worker mpm and the math behind the configuration.

Let's start out with the worker mpm configuration

<IfModule mpm_worker_module>

ThreadLimit 100
StartServers 5
MaxClients 1000
MinSpareThreads 100
MaxSpareThreads 1000
ThreadsPerChild 100
MaxRequestsPerChild 0
</IfModule>

The optimal configuration completely depends on the hardware being used and the load requirements.  But a general rule of thumb, keep processes low and thread count high.  To determine the number of processes Apache will use simply divide MaxClients by ThreadPerChild.  So in this case MaxClients (1000) / ThreadsPerChild (100) = Processes (10), so Apache will allocate a maximum of 100 threads per each 10 child processes resulting in a total of 1000 possible clients.

Now to translate this to mod_jk, mod_jk maintains a connection pool for each worker defined in workers.properties.  By default with Apache mod_jk sets connection_pool_size to ThreadsPerChild, so in the above case that would translate to 100, giving 1000 possible connections to JBoss.  This may or may not be desired.

Let's take a common example, there will be 3 JBoss servers that combined needed to be able to handle 900 concurrent connections

worker.list=loadbalancer,status

worker.template.type=ajp13
worker.template.port=8009
worker.template.ping_mode=A
worker.template.connection_pool_size=30
worker.template.socket_timeout=10
worker.template.retries=20

worker.node1.reference=worker.template
worker.node1.host=192.168.0.101

worker.node2.reference=worker.template
worker.node2.host=192.168.0.102

worker.node3.reference=worker.template
worker.node3.host=192.168.0.103

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2, node3
worker.loadbalancer.sticky_session=True

worker.status.type=status

The above configuration tells mod_jk to multiplex 30 connections to the available Apache processes, which is 10 processes.  So that means 30 connections multiplexed over 10 processes gives 300 possible connections to each backend worker.  Furthermore, the total connections able to be used in this configuration from Apache is 900 which means  100 connections will be left over for static content or whatnot.

Next configure maxThreads in each ajp connector to match the above.

Node 1 ajp connector:

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="300"
           connectionTimeout="600000"></Connector>

Node 2 ajp connector:

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="300"
           connectionTimeout="600000"></Connector>

Node 3 ajp connector:

<Connector port="8009"
           address="${jboss.bind.address}"
           emptySessionPath="true"
           enableLookups="false"
           redirectPort="8443"
           protocol="AJP/1.3"
           maxThreads="300"
           connectionTimeout="600000"></Connector>

Remember when using connectionTimeout which is always recommended, prepost_timeout and connect_timeout also need to be set, which is done.  I'm not showing sticky session configuration, but that covered in the main mod_jk article in using mod_jk with JBoss.

conf\mod_jk.conf

# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so

# Where to find workers.properties
JkWorkersFile conf/workers.properties

# Where to put jk logs
JkLogFile logs/mod_jk2.log

# Set the jk log level [debug/error/info]
JkLogLevel error

# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# JkOptions indicates to send SSK KEY SIZE
#JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkOptions +ForwardKeySize +ForwardURICompatUnparsed -ForwardDirectories

# JkRequestLogFormat
JkRequestLogFormat "%R %r %T"

# Mount your applications
#JkMount /* loadbalancer

# You can use external file for mount points.
# It will be checked for updates each 60 seconds.
# The format of the file is: /url=worker
# /examples/*=loadbalancer

JkMountFile conf/uriworkermap.properties

# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm

# Add jkstatus for managing runtime data
<Location /jkstatus/>
        JkMount status
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1
</Location>

conf\uriworkermap.properties

# Simple worker configuration file Mount the Servlet context to the ajp13 worker

#/jmx-console=loadbalancer;reply_timeout=1800000
#/jmx-console/*=loadbalancer;reply_timeout=1800000

#/web-console=loadbalancer;reply_timeout=1800000
#/web-console/*=loadbalancer;reply_timeout=1800000

/getpaid=loadbalancer;reply_timeout=1800000
/getpaid/*=loadbalancer;reply_timeout=1800000

/courier=loadbalancer;reply_timeout=1800000
/courier/*=loadbalancer;reply_timeout=1800000

conf\workers.properties

# Define list of workers that will be used for mapping requests

worker.list=loadbalancer,status

worker.template.type=ajp13
worker.template.lbfactor=1
worker.template.ping_mode=A
worker.template.ping_timeout=20000
worker.template.socket_connect_timeout=2000
worker.template.cachesize=10

#Nod1 and Node2 for performance testing.

worker.node1.reference=worker.template
worker.node1.host=10.255.205.127
worker.node1.port=8109

worker.node2.reference=worker.template
worker.node2.host=10.255.205.127
worker.node2.port=8209

# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.sticky_session=1
worker.loadbalancer.balance_workers=node1,node2

# Status worker for managing load balancer
worker.status.type=status

# Define Node1
#worker.node1.reference=worker.template
#worker.node1.host=10.255.205.11
#worker.node1.port=8409

# section for dashboards is commented

# Define Dashboards
#worker.dashboards.reference=worker.template
#worker.dashboards.host=10.255.205.11
#worker.dashboards.port=8009

#worker.loadbalancerdb.type=lb
#worker.loadbalancerdb.sticky_session=1
#worker.loadbalancerdb.balance_workers=dashboards

conf\httpd.conf
# GP setup

# Include mod_jk's specific configuration file
Include conf/mod-jk.conf

Win32DisableAcceptEx

<IfModule mpm_winnt.c>
  ThreadsPerChild       10000
  MaxRequestsPerChild     0
</IfModule>

<Location />
        AllowOverride None
        Order deny,allow
        Deny  from all
        Allow from all
</Location>

AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml application/x-javascript text/javascript

apache +mod_jk+tomcat 高并发优化

apache +mod_jk+tomcat 高并发优化

本机部署了多个MYSQL 多个TOMCAT 并发大概5000

主要配置JK  和TOMCAT下的server.xml

以下是我的配置效果还算不错

workers.java_home=/usr/local/jdk
    ps=/
    worker.list=controller,tomcat,tomcat1,tomcat2,tomcat3,tomcat4,tomcat5
    worker.tomcat.port=8009
    worker.tomcat.host=localhost
    worker.tomcat.type=ajp13
    worker.tomcat.lbfactor=1

worker.tomcat5.port=11009
    worker.tomcat5.host=localhost
    worker.tomcat5.type=ajp13
    worker.tomcat5.socket_keepalive=1
    worker.tomcat5.lbfactor=1

worker.tomcat1.port=10003
    worker.tomcat1.host=localhost
    worker.tomcat1.type=ajp13
    worker.tomcat1.socket_keepalive=1
    worker.tomcat1.lbfactor=1

worker.tomcat2.port=10006
    worker.tomcat2.host=localhost
    worker.tomcat2.type=ajp13
    worker.tomcat2.lbfactor=1

worker.tomcat3.port=7003
     worker.tomcat3.host=localhost
     worker.tomcat3.type=ajp13
     worker.tomcat3.lbfactor=1

worker.tomcat4.port=9199
     worker.tomcat4.host=localhost
     worker.tomcat4.type=ajp13
     worker.tomcat4.lbfactor=1

worker.controller.type=lb
    worker.controller.balance_workers=tomcat,tomcat1,tomcat2,tomcat3,tomcat4,tomcat5
    worker.controller.sticky_session=1
    worker.status.type=status

worker.controller.sticky_session=1
    worker.status.type=status
    worker.controller.recovery_options=3
    worker.controller.socket_keepalive=True
    worker.controller.connection_pool_size=120
    worker.controller.connection_pool_minsize=0
    worker.controller.connection_pool_timeout=20000

以下2参数重要

worker.controller.recovery_options=3
worker.controller.connection_pool_minsize=0

下面是SERVER.XML的配置

<Connector port=”8080″ protocol=”HTTP/1.1″
               maxHttpHeaderSize=”8192″ useBodyEncodingForURI=”true”
               acceptCount=”500″
               maxThreads=”2000″
               minSpareThreads=”100″
               maxSpareThreads=”2000″
               tcpNoDelay=”true”
               socketBuffer=”4096″
               enableLookups=”false”
               redirectPort=”8443″
               connectionTimeout=”200000″  />
    <!– A “Connector” using the shared thread pool–>
    <!–
    <Connector executor=”tomcatThreadPool”
               port=”8080″ protocol=”HTTP/1.1″
               connectionTimeout=”20000″
               redirectPort=”8443″ />
    –>
    <!– Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the
         connector should be using the OpenSSL style configuration
         described in the APR documentation –>
    <!–
    <Connector port=”8443″ protocol=”HTTP/1.1″ SSLEnabled=”true”
               maxThreads=”150″ scheme=”https” secure=”true”
               clientAuth=”false” sslProtocol=”TLS” />
    –>

<!– Define an AJP 1.3 Connector on port 8009 –>
    <Connector port=”8009″ protocol=”AJP/1.3″ redirectPort=”8443″
    connectionTimeout=”200000″  maxThreads=”2000″ acceptCount=”500″
            />

注意 ：压力大了 会抱JK连不到TOMCAT的错误 我也找了好久 是JK仍给TOMCAT是通过AJP13端口去处理 所有优化AJP13

把参数加到 <Connector port=”8009″ protocol=”AJP/1.3″ redirectPort=”8443″
    connectionTimeout=”200000″  maxThreads=”2000″ acceptCount=”500″
            />

就没什么问题了 大家可以试下

Apache+Tomcat负载实战调优记录

一般来说，按照网上提供的一些方法，将Apache和多个Tomcat通过jk连接，进行集群负载是没有什么大的问题的，但当访问量增加得比较大而且峰值持续时间较长时，默认的一些配置就不够用了。

今天亲身经历了这样一个过程，apache的最大连接数调到了4500，但都还不够用，一会就跑满了。而Tomcat这边压力却不大，同时，发现mod_jk.log里面有很多这样的错误记录：

(tomcat) Tomcat is down or refused connection. No response has been sent to the client (yet) 或者connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)

google 一查，才知道是Apache和Tomcat的connectionTimeout参数设置得不同步造成的，默认的jk是不超时的，也就是说进行长连接，而Tomcat默认的时长是500ms，所以二者的超时时间不匹配，造成jk反复的重连Tomcat，全部都堵在了Tomcat这一端。

调优方法：在Tomcat的Connector节，增加如下配置：

connectionTimeout="200000" maxThreads="2000" acceptCount="500"

可解决该问题，Apache的连接数一下就降下来了

先记录到这里，有时间再具体来分析。