Kubernetes环境下如何运行Coherence缓存集群
Oracle官方出了一个如何在Docker环境下运行Coherence的技术文档，大家可以参考：
https://github.com/oracle/docker-images/tree/master/OracleCoherence
但是对于一个熟悉Coherence的老司机来说，简单搭建起来只是个初步方案，在客户的环境总是各种特性和定制化配置，所以本文研究的也是如何将已经客户化的Coherence架构构建在Kubernetes开源框架上。
背景架构说明
话不多说，找一个客户的典型的Coherence架构
架构说明：
- Coherence的节点有不同的角色，分成了存储节点，代理节点，管理节点和客户端
- 不同的角色需要配置不同的Cacheconfig文件
- 整个架构是属于Coherence Extend的方式，客户端通过tcp长连接的方式连入Coherence集群
- 存储节点存储数据，代理节点，管理节点和客户端都不存储数据，代理节点主要负责调度客户端请求到Coherence集群中。
Kubernetes环境下架构
区别在于:
- 客户端WebLogic Server基本都是基于Replication Controller部署成一个个的Pod
- 后端的CacheServer角色的节点以及负责代理的节点都是部署成两组不同的Replication Controller
- cacheserver中每个pod都通过flanneld绑定不同的ip,proxy server节点也是绑定不同的ip以及相同的端口9099
现在问题是proxy server的每个ip都是不固定的，这样在weblogic端就需要指定后端动态变化的ip连入集群。初步想法是让proxy pod绑定服务，然后通过servicename接入。
所以首先需要配置的是dns,作用是每个weblogic pod都能通过dns解析servicename,并转换到proxy server的具体地址。
构建Coherence Proxy Images
其实coherence cacheserver和proxyserver可以定制一个image,然后通过不同的参数来切换不同的配置文件，但这里采用最简单的办法，针对不同角色构建不同的images.
在 /home/weblogic/docker/OracleCoherence/dockerfiles/12.2.1.0.0 目录下加入一个文件proxy-cache-config.xml作为proxyserver启动的配置参数。
[root@k8s-node- 12.2.1.0.]# cat proxy-cache-config.xml
<?xml version="1.0"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>*</cache-name>
<scheme-name>distributed-scheme</scheme-name>
</cache-mapping>
</caching-scheme-mapping> <caching-schemes>
<!-- Distributed caching scheme. -->
<distributed-scheme>
<scheme-name>distributed-scheme</scheme-name>
<service-name>DistributedCache</service-name>
<thread-count></thread-count>
<backup-count></backup-count>
<backing-map-scheme>
<local-scheme>
<scheme-name>LocalSizeLimited</scheme-name>
</local-scheme> </backing-map-scheme>
<autostart>true</autostart>
<local-storage>false</local-storage>
</distributed-scheme>
<local-scheme>
<scheme-name>LocalSizeLimited</scheme-name>
<eviction-policy>LRU</eviction-policy>
<high-units></high-units>
<unit-calculator>BINARY</unit-calculator>
<unit-factor></unit-factor>
<expiry-delay>48h</expiry-delay>
</local-scheme> <proxy-scheme>
<service-name>ExtendTcpProxyService</service-name>
<thread-count></thread-count>
<acceptor-config>
<tcp-acceptor>
<local-address>
<address>0.0.0.0</address>
<port></port>
</local-address>
</tcp-acceptor>
</acceptor-config>
<autostart>true</autostart>
</proxy-scheme> </caching-schemes>
</cache-config>
注意这里的address,0.0.0.0意味着可以绑定任何生成的ip.
修改Dockerfile,最好建立一个新的Dockerfile.proxy
[root@k8s-node- 12.2.1.0.]# cat Dockerfile.proxy
# LICENSE CDDL 1.0 + GPL 2.0
#
# ORACLE DOCKERFILES PROJECT
# --------------------------
# This is the Dockerfile for Coherence 12.2. Standalone Distribution
#
# REQUIRED BASE IMAGE TO BUILD THIS IMAGE
# ---------------------------------------
# This Dockerfile requires the base image oracle/serverjre:
# (see https://github.com/oracle/docker-images/tree/master/OracleJava)
#
# REQUIRED FILES TO BUILD THIS IMAGE
# ----------------------------------
# () fmw_12.2.1..0_coherence_Disk1_1of1.zip
#
# Download the Standalone installer from http://www.oracle.com/technetwork/middleware/coherence/downloads/index.html
#
# HOW TO BUILD THIS IMAGE
# -----------------------
# Put all downloaded files in the same directory as this Dockerfile
# Run:
# $ sh buildDockerImage.sh -s
#
# or if your Docker client requires root access you can run:
# $ sudo sh buildDockerImage.sh -s
# # Pull base image
# ---------------
FROM oracle/serverjre: # Maintainer
# ----------
MAINTAINER Jonathan Knight # Environment variables required for this build (do NOT change)
ENV FMW_PKG=fmw_12.2.1..0_coherence_Disk1_1of1.zip \
FMW_JAR=fmw_12.2.1..0_coherence.jar \
ORACLE_HOME=/u01/oracle/oracle_home \
PATH=$PATH:/usr/java/default/bin:/u01/oracle/oracle_home/oracle_common/common/bin \
CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom" ENV COHERENCE_HOME=$ORACLE_HOME/coherence # Copy files required to build this image
COPY $FMW_PKG install.file oraInst.loc /u01/
COPY start.sh /start.sh
COPY proxy-cache-config.xml $COHERENCE_HOME/conf/proxy-cache-config.xml RUN useradd -b /u01 -m -s /bin/bash oracle && \
echo oracle:oracle | chpasswd && \
chmod +x /start.sh && \
chmod a+xr /u01 && \
chown -R oracle:oracle /u01 USER oracle # Install and configure Oracle JDK
# Setup required packages (unzip), filesystem, and oracle user
# ------------------------------------------------------------
RUN cd /u01 && $JAVA_HOME/bin/jar xf /u01/$FMW_PKG && cd - && \
$JAVA_HOME/bin/java -jar /u01/$FMW_JAR -silent -responseFile /u01/install.file -invPtrLoc /u01/oraInst.loc -jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation ORACLE_HOME=$ORACLE_HOME && \
rm /u01/$FMW_JAR /u01/$FMW_PKG /u01/oraInst.loc /u01/install.file ENTRYPOINT ["/start.sh"]
区别在于需要把刚才的文件copy到容器中去
然后修改start.sh
[root@k8s-node- 12.2.1.0.]# cat start.sh
#!/usr/bin/env sh #!/bin/sh -e -x -u trap "echo TRAPed signal" HUP INT QUIT KILL TERM main() { COMMAND=server
SCRIPT_NAME=$(basename "${0}")
MAIN_CLASS="com.tangosol.net.DefaultCacheServer" case "${1}" in
server) COMMAND=${}; shift ;;
console) COMMAND=${}; shift ;;
queryplus) COMMAND=queryPlus; shift ;;
help) COMMAND=${}; shift ;;
esac case ${COMMAND} in
server) server ;;
console) console ;;
queryPlus) queryPlus ;;
help) usage; exit ;;
*) server ;;
esac
} # ---------------------------------------------------------------------------
# Display the help text for this script
# ---------------------------------------------------------------------------
usage() {
echo "Usage: ${SCRIPT_NAME} [type] [args]"
echo ""
echo "type: - the type of process to run, must be one of:"
echo " server - runs a storage enabled DefaultCacheServer"
echo " (server is the default if type is omitted)"
echo " console - runs a storage disabled Coherence console"
echo " query - runs a storage disabled QueryPlus session"
echo " help - displays this usage text"
echo ""
echo "args: - any subsequent arguments are passed as program args to the main class"
echo ""
echo "Environment Variables: The following environment variables affect the script operation"
echo ""
echo "JAVA_OPTS - this environment variable adds Java options to the start command,"
echo " for example memory and other system properties"
echo ""
echo "COH_WKA - Sets the WKA address to use to discover a Coherence cluster."
echo ""
echo "COH_EXTEND_PORT - If set the Extend Proxy Service will listen on this port instead"
echo " of the default ephemeral port."
echo ""
echo "Any jar files added to the /lib folder will be pre-pended to the classpath."
echo "The /conf folder is on the classpath so any files in this folder can be loaded by the process."
echo ""
} server() {
PROPS=""
CLASSPATH=""
MAIN_CLASS="com.tangosol.net.DefaultCacheServer"
start
} console() {
PROPS="-Dcoherence.localstorage=false"
CLASSPATH=""
MAIN_CLASS="com.tangosol.net.CacheFactory"
start
} queryPlus() {
PROPS="-Dcoherence.localstorage=false"
CLASSPATH="${COHERENCE_HOME}/lib/jline.jar"
MAIN_CLASS="com.tangosol.coherence.dslquery.QueryPlus"
start
} start() { if [ "${COH_WKA}" != "" ]
then
PROPS="${PROPS} -Dcoherence.wka=${COH_WKA}"
fi if [ "${COH_EXTEND_PORT}" != "" ]
then
PROPS="${PROPS} -Dcoherence.cacheconfig=extend-cache-config.xml -Dcoherence.extend.port=${COH_EXTEND_PORT}"
fi CLASSPATH="/conf:/lib/*:${CLASSPATH}:${COHERENCE_HOME}/conf:${COHERENCE_HOME}/lib/coherence.jar" CMD="${JAVA_HOME}/bin/java -cp ${CLASSPATH} ${PROPS} -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=proxy-cache-config.xml ${JAVA_OPTS} ${MAIN_CLASS} ${COH_MAIN_ARGS}" echo "Starting Coherence ${COMMAND} using ${CMD}" exec ${CMD}
} main "$@"
主要是在最后的java -cp中的修改。修改相应的build文件
最后build image(-s是指用standardalong版本)
sh buildProxyServer.sh -v 12.2.1.0. -s
构建Coherence CacheServer Images
按照类似方法，先新建一个storage-cache-server.xml
[root@k8s-node- 12.2.1.0.]# cat storage-cache-config.xml
<?xml version="1.0"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config> <caching-scheme-mapping>
<cache-mapping>
<cache-name>*</cache-name>
<scheme-name>distributed-pof</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>distributed-pof</scheme-name>
<service-name>DistributedCache</service-name> <backing-map-scheme> <local-scheme/> </backing-map-scheme> <listener/>
<autostart>true</autostart>
<local-storage>true</local-storage>
</distributed-scheme>
</caching-schemes>
</cache-config>
然后建立一个Dockerfile.cacheserver
[root@k8s-node- 12.2.1.0.]# cat Dockerfile.cacheserver
# LICENSE CDDL 1.0 + GPL 2.0
#
# ORACLE DOCKERFILES PROJECT
# --------------------------
# This is the Dockerfile for Coherence 12.2. Standalone Distribution
#
# REQUIRED BASE IMAGE TO BUILD THIS IMAGE
# ---------------------------------------
# This Dockerfile requires the base image oracle/serverjre:
# (see https://github.com/oracle/docker-images/tree/master/OracleJava)
#
# REQUIRED FILES TO BUILD THIS IMAGE
# ----------------------------------
# () fmw_12.2.1..0_coherence_Disk1_1of1.zip
#
# Download the Standalone installer from http://www.oracle.com/technetwork/middleware/coherence/downloads/index.html
#
# HOW TO BUILD THIS IMAGE
# -----------------------
# Put all downloaded files in the same directory as this Dockerfile
# Run:
# $ sh buildDockerImage.sh -s
#
# or if your Docker client requires root access you can run:
# $ sudo sh buildDockerImage.sh -s
# # Pull base image
# ---------------
FROM oracle/serverjre: # Maintainer
# ----------
MAINTAINER Jonathan Knight # Environment variables required for this build (do NOT change)
ENV FMW_PKG=fmw_12.2.1..0_coherence_Disk1_1of1.zip \
FMW_JAR=fmw_12.2.1..0_coherence.jar \
ORACLE_HOME=/u01/oracle/oracle_home \
PATH=$PATH:/usr/java/default/bin:/u01/oracle/oracle_home/oracle_common/common/bin \
CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom" ENV COHERENCE_HOME=$ORACLE_HOME/coherence # Copy files required to build this image
COPY $FMW_PKG install.file oraInst.loc /u01/
COPY start.sh /start.sh
COPY storage-cache-config.xml $COHERENCE_HOME/conf/storage-cache-config.xml RUN useradd -b /u01 -m -s /bin/bash oracle && \
echo oracle:oracle | chpasswd && \
chmod +x /start.sh && \
chmod a+xr /u01 && \
chown -R oracle:oracle /u01 USER oracle # Install and configure Oracle JDK
# Setup required packages (unzip), filesystem, and oracle user
# ------------------------------------------------------------
RUN cd /u01 && $JAVA_HOME/bin/jar xf /u01/$FMW_PKG && cd - && \
$JAVA_HOME/bin/java -jar /u01/$FMW_JAR -silent -responseFile /u01/install.file -invPtrLoc /u01/oraInst.loc -jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation ORACLE_HOME=$ORACLE_HOME && \
rm /u01/$FMW_JAR /u01/$FMW_PKG /u01/oraInst.loc /u01/install.file ENTRYPOINT ["/start.sh"]
最后修改start.sh,关键语句是
CMD="${JAVA_HOME}/bin/java -cp ${CLASSPATH} ${PROPS} -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.cacheconfig=storage-cache-config.xml ${JAVA_OPTS} ${MAIN_CLASS} ${COH_MAIN_ARGS}"
最后build image
sh buildCacheServer.sh -v 12.2.1.0. -s
通过docker images看到
[root@k8s-node- 12.2.1.0.]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
-domain v2 326bf14bb29f About an hour ago 2.055 GB
oracle/coherence 12.2.1.0.-cacheserver 57a90e86e1d2 hours ago MB
oracle/coherence 12.2.1.0.-proxy 238c85d61468 hours ago MB
在master节点创建一系列ReplicationController
coherence-proxy.yaml
[root@k8s-master ~]# cat coherence-proxy.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: coherence-proxy
spec:
replicas:
template:
metadata:
labels:
coherencecluster: "proxy"
version: "0.1"
spec:
containers:
- name: coherenceproxy
image: oracle/coherence:12.2.1.0.-proxy
ports:
- containerPort:
---
apiVersion: v1
kind: Service
metadata:
name: coherenceproxysvc
labels:
coherencecluster: proxy
spec:
type: NodePort
ports:
- port:
protocol: TCP
targetPort:
nodePort:
selector:
coherencecluster: proxy
coherence-cacheserver.yaml
[root@k8s-master ~]# cat coherence-cacheserver.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: coherence-cacheserver
spec:
replicas:
template:
metadata:
labels:
coherencecluster: "mycluster"
version: "0.1"
spec:
containers:
- name: coherencecacheserver
image: oracle/coherence:12.2.1.0.-cacheserver
kubectl create -f coherence-proxy.yaml kubectl create -f coherence-cacheserver.yaml
然后看看pod是否启动成功
[root@k8s-master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
coherence-cacheserver-96kz7 / Running 1h 192.168.33.4 k8s-node-
coherence-cacheserver-z67ht / Running 1h 192.168.33.3 k8s-node-
coherence-proxy-j7r0w / Running 1h 192.168.33.5 k8s-node-
coherence-proxy-tg8n8 / Running 1h 192.168.33.6 k8s-node-
登录进去后确定coherence集群成员状态，看member成员的个数基本确定已经都加入集群了。
MasterMemberSet(
ThisMember=Member(Id=, Timestamp=-- ::04.244, Address=192.168.33.6:, MachineId=, Location=machine:coherence-proxy-tg8n8,process:, Role=CoherenceServer)
OldestMember=Member(Id=, Timestamp=-- ::16.941, Address=192.168.33.4:, MachineId=, Location=machine:coherence-cacheserver-96kz7,process:, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=
Member(Id=, Timestamp=-- ::16.941, Address=192.168.33.4:, MachineId=, Location=machine:coherence-cacheserver-96kz7,process:, Role=CoherenceServer)
Member(Id=, Timestamp=-- ::20.836, Address=192.168.33.3:, MachineId=, Location=machine:coherence-cacheserver-z67ht,process:, Role=CoherenceServer)
Member(Id=, Timestamp=-- ::02.144, Address=192.168.33.5:, MachineId=, Location=machine:coherence-proxy-j7r0w,process:, Role=CoherenceServer)
Member(Id=, Timestamp=-- ::04.244, Address=192.168.33.6:, MachineId=, Location=machine:coherence-proxy-tg8n8,process:, Role=CoherenceServer)
)
MemberId|ServiceJoined|MemberState
|-- ::16.941|JOINED,
|-- ::20.836|JOINED,
|-- ::02.144|JOINED,
|-- ::04.244|JOINED
RecycleMillis=
RecycleSet=MemberSet(Size=
Member(Id=, Timestamp=-- ::25.741, Address=192.168.33.6:, MachineId=)
Member(Id=, Timestamp=-- ::25.74, Address=192.168.33.5:, MachineId=)
Member(Id=, Timestamp=-- ::45.413, Address=192.168.33.5:, MachineId=)
Member(Id=, Timestamp=-- ::45.378, Address=192.168.33.6:, MachineId=)
)
) TcpRing{Connections=[]}
IpMonitor{Addresses=, Timeout=15s}
确定service状态
[root@k8s-master ~]# kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coherenceproxysvc 10.254.22.102 <nodes> :/TCP 1h
kubernetes 10.254.0.1 <none> /TCP 26d
coherenceproxysvc已经启动。
开始Coherence客户端WebLogic Pod的配置
因为需要修改setDomainEnv.sh文件，将客户端的coherence配置文件写入，所以转入weblogic目录
[root@k8s-node- -domain]# pwd
/home/weblogic/docker/OracleWebLogic/samples/-domain
新建一个proxy-client.xml
[root@k8s-node- -domain]# cat proxy-client.xml
<?xml version="1.0"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>*</cache-name>
<scheme-name>extend-dist</scheme-name>
</cache-mapping>
</caching-scheme-mapping> <caching-schemes>
<remote-cache-scheme>
<scheme-name>extend-dist</scheme-name>
<service-name>ExtendTcpCacheService</service-name>
<initiator-config>
<tcp-initiator>
<remote-addresses>
<socket-address>
<address>coherenceproxysvc</address>
<port></port>
</socket-address>
</remote-addresses>
<connect-timeout>10s</connect-timeout>
</tcp-initiator>
<outgoing-message-handler>
<request-timeout>5s</request-timeout>
</outgoing-message-handler>
</initiator-config>
</remote-cache-scheme>
</caching-schemes>
</cache-config>
需要注意的是address要指到service的名称，依靠dns去解析。
修改Dockerfile,核心是加入JAVA_OPTIONS和CLASSPATH.
FROM oracle/weblogic:12.1.-generic # Maintainer
# ----------
MAINTAINER Bruno Borges <bruno.borges@oracle.com> # WLS Configuration
# -------------------------------
ARG ADMIN_PASSWORD
ARG PRODUCTION_MODE ENV DOMAIN_NAME="base_domain" \
DOMAIN_HOME="/u01/oracle/user_projects/domains/base_domain" \
ADMIN_PORT="" \
ADMIN_HOST="wlsadmin" \
NM_PORT="" \
MS_PORT="" \
PRODUCTION_MODE="${PRODUCTION_MODE:-prod}" \
JAVA_OPTIONS="-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=/u01/oracle/proxy-client.xml" \
CLASSPATH="/u01/oracle/coherence.jar" \
PATH=$PATH:/u01/oracle/oracle_common/common/bin:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin:/u01/oracle # Add files required to build this image
USER oracle
COPY container-scripts/* /u01/oracle/
COPY coherence.jar /u01/oracle/
COPY proxy-client.xml /u01/oracle/ # Configuration of WLS Domain
WORKDIR /u01/oracle
RUN /u01/oracle/wlst /u01/oracle/create-wls-domain.py && \
mkdir -p /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security && \
echo "username=weblogic" > /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security/boot.properties && \
echo "password=$ADMIN_PASSWORD" >> /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security/boot.properties && \
echo ". /u01/oracle/user_projects/domains/base_domain/bin/setDomainEnv.sh" >> /u01/oracle/.bashrc && \
echo "export PATH=$PATH:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin" >> /u01/oracle/.bashrc && \
cp /u01/oracle/commEnv.sh /u01/oracle/wlserver/common/bin/commEnv.sh && \
rm /u01/oracle/create-wls-domain.py /u01/oracle/jaxrs2-template.jar # Expose Node Manager default port, and also default http/https ports for admin console
EXPOSE $NM_PORT $ADMIN_PORT $MS_PORT WORKDIR $DOMAIN_HOME # Define default command to start bash.
CMD ["startWebLogic.sh"]
然后build image
docker build -t -domain:v2 --build-arg ADMIN_PASSWORD=welcome1 .
构建weblogic pod作为一个客户端
[root@k8s-master ~]# cat weblogic-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: weblogic
spec:
containers:
- name: weblogic
image: -domain:v2
ports:
- containerPort:
启动以后，通过log确定weblogic启动时确实把我们客户化的参数加入
[root@k8s-master ~]# kubectl logs weblogic
.
.
JAVA Memory arguments: -Djava.security.egd=file:/dev/./urandom
.
CLASSPATH=/u01/oracle/wlserver/../oracle_common/modules/javax.persistence_2..jar:/u01/oracle/wlserver/../wlserver/modules/com.oracle.weblogic.jpa21support_1.0.0.0_2-.jar:/usr/java/jdk1..0_101/lib/tools.jar:/u01/oracle/wlserver/server/lib/weblogic_sp.jar:/u01/oracle/wlserver/server/lib/weblogic.jar:/u01/oracle/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/u01/oracle/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0..jar:/u01/oracle/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.1.0..jar:/u01/oracle/wlserver/common/derby/lib/derbyclient.jar:/u01/oracle/wlserver/common/derby/lib/derby.jar:/u01/oracle/wlserver/server/lib/xqrl.jar:/u01/oracle/coherence.jar
.
PATH=/u01/oracle/wlserver/server/bin:/u01/oracle/wlserver/../oracle_common/modules/org.apache.ant_1.9.2/bin:/usr/java/jdk1..0_101/jre/bin:/usr/java/jdk1..0_101/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin:/u01/oracle/oracle_common/common/bin:/u01/oracle/oracle_common/common/bin:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin:/u01/oracle
.
***************************************************
* To start WebLogic Server, use a username and *
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http://hostname:port/console *
***************************************************
starting weblogic with Java version:
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) -Bit Server VM (build 25.101-b13, mixed mode)
Starting WLS with line:
/usr/java/jdk1..0_101/bin/java -server -Djava.security.egd=file:/dev/./urandom -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/oracle/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=/u01/oracle/proxy-client.xml -Djava.endorsed.dirs=/usr/java/jdk1..0_101/jre/lib/endorsed:/u01/oracle/wlserver/../oracle_common/modules/endorsed -da -Dwls.home=/u01/oracle/wlserver/server -Dweblogic.home=/u01/oracle/wlserver/server -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server
部署一个HelloWorld.war文件，核心的index.jsp代码是:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<%@page import="java.util.*"%>
<%@page import="com.tangosol.net.*"%>
<%@ page contentType="text/html;charset=windows-1252"%> <html>
<body> This is a Helloworld test</body>
<h3> <%
String mysession;
NamedCache cache;
cache = CacheFactory.getCache("demoCache");
cache.put("eric","eric.nie@oracle.com"); %>
Get Eric Email:<%=cache.get("eric").toString()%>
</h3> </html>
[root@k8s-master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
coherence-cacheserver-96kz7 / Running 1h 192.168.33.4 k8s-node-
coherence-cacheserver-z67ht / Running 1h 192.168.33.3 k8s-node-
coherence-proxy-j7r0w / Running 1h 192.168.33.5 k8s-node-
coherence-proxy-tg8n8 / Running 1h 192.168.33.6 k8s-node-
weblogic / Running 1h 192.168.33.7 k8s-node-
部署后访问
查看后面的weblogic日志
<Jun , :: AM GMT> <Notice> <WebLogicServer> <BEA-> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>
<Jun , :: AM GMT> <Notice> <WebLogicServer> <BEA-> <The server started in RUNNING mode.>
<Jun , :: AM GMT> <Warning> <Server> <BEA-> <The hostname "localhost", maps to multiple IP addresses: 127.0.0.1, :::::::.>
<Jun , :: AM GMT> <Notice> <WebLogicServer> <BEA-> <Server state changed to RUNNING.>
-- ::29.193/280.892 Oracle Coherence 12.1.3.0. <Info> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational configuration from "jar:file:/u01/oracle/coherence/lib/coherence.jar!/tangosol-coherence.xml"
-- ::29.451/281.086 Oracle Coherence 12.1.3.0. <Info> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational overrides from "jar:file:/u01/oracle/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
-- ::29.479/281.114 Oracle Coherence 12.1.3.0. <D5> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
-- ::29.511/281.146 Oracle Coherence 12.1.3.0. <D5> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "cache-factory-config.xml" is not specified
-- ::29.525/281.159 Oracle Coherence 12.1.3.0. <D5> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "cache-factory-builder-config.xml" is not specified
-- ::29.526/281.161 Oracle Coherence 12.1.3.0. <D5> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified Oracle Coherence Version 12.1.3.0. Build
Grid Edition: Development mode
Copyright (c) , , Oracle and/or its affiliates. All rights reserved. -- ::29.672/281.306 Oracle Coherence GE 12.1.3.0. <Info> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded cache configuration from "file:/u01/oracle/proxy-client.xml"; this document does not refer to any schema definition and has not been validated.
-- ::30.225/281.860 Oracle Coherence GE 12.1.3.0. <Info> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Created cache factory com.tangosol.net.ExtensibleConfigurableCacheFactory
-- ::30.507/282.142 Oracle Coherence GE 12.1.3.0. <D5> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connecting Socket to 10.254.203.94:
-- ::30.534/282.169 Oracle Coherence GE 12.1.3.0. <Info> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error connecting Socket to 10.254.203.94:: java.net.ConnectException: Connection refused
-- ::11.056/862.693 Oracle Coherence GE 12.1.3.0. <Info> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Restarting Service: ExtendTcpCacheService
-- ::11.156/862.791 Oracle Coherence GE 12.1.3.0. <D5> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connecting Socket to 10.254.22.102:
-- ::11.188/862.837 Oracle Coherence GE 12.1.3.0. <Info> (thread=[ACTIVE] ExecuteThread: '' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connected Socket to 10.254.22.102:
问题和定位
关于DNS解析是否正确，可以通过下面命令
iptables -L -v -n -t nat
查看路由是否正确，主要是看coherenceproxysvc的路由是否到正确的pod和端口.
[root@k8s-node- -domain]# iptables -L -v -n -t nat
Chain PREROUTING (policy ACCEPT packets, bytes)
pkts bytes target prot opt in out source destination
1898K KUBE-SERVICES all -- * * 0.0.0.0/ 0.0.0.0/ /* kubernetes service portals */
DOCKER all -- * * 0.0.0.0/ 0.0.0.0/ ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT packets, bytes)
pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT packets, bytes)
pkts bytes target prot opt in out source destination
126K KUBE-SERVICES all -- * * 0.0.0.0/ 0.0.0.0/ /* kubernetes service portals */
DOCKER all -- * * 0.0.0.0/ !127.0.0.0/ ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT packets, bytes)
pkts bytes target prot opt in out source destination
283K MASQUERADE all -- * !docker0 192.168.33.0/ 0.0.0.0/
1716K KUBE-POSTROUTING all -- * * 0.0.0.0/ 0.0.0.0/ /* kubernetes postrouting rules */
RETURN all -- * * 192.168.122.0/ 224.0.0.0/
RETURN all -- * * 192.168.122.0/ 255.255.255.255
MASQUERADE tcp -- * * 192.168.122.0/ !192.168.122.0/ masq ports: -
MASQUERADE udp -- * * 192.168.122.0/ !192.168.122.0/ masq ports: -
MASQUERADE all -- * * 192.168.122.0/ !192.168.122.0/ Chain DOCKER ( references)
pkts bytes target prot opt in out source destination
RETURN all -- docker0 * 0.0.0.0/ 0.0.0.0/ Chain KUBE-MARK-DROP ( references)
pkts bytes target prot opt in out source destination
MARK all -- * * 0.0.0.0/ 0.0.0.0/ MARK or 0x8000 Chain KUBE-MARK-MASQ ( references)
pkts bytes target prot opt in out source destination
MARK all -- * * 0.0.0.0/ 0.0.0.0/ MARK or 0x4000 Chain KUBE-NODEPORTS ( references)
pkts bytes target prot opt in out source destination
KUBE-MARK-MASQ tcp -- * * 0.0.0.0/ 0.0.0.0/ /* default/coherenceproxysvc: */ tcp dpt:
KUBE-SVC-BQXHRGVXFCEH2BHH tcp -- * * 0.0.0.0/ 0.0.0.0/ /* default/coherenceproxysvc: */ tcp dpt: Chain KUBE-POSTROUTING ( references)
pkts bytes target prot opt in out source destination
MASQUERADE all -- * * 0.0.0.0/ 0.0.0.0/ /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000 Chain KUBE-SEP-67FRRWLKQK2OD4HZ ( references)
pkts bytes target prot opt in out source destination
KUBE-MARK-MASQ all -- * * 192.168.33.2 0.0.0.0/ /* kube-system/kube-dns:dns-tcp */
DNAT tcp -- * * 0.0.0.0/ 0.0.0.0/ /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.33.2: Chain KUBE-SEP-GIM2MHZZZBZJL55J ( references)
pkts bytes target prot opt in out source destination
KUBE-MARK-MASQ all -- * * 192.168.0.105 0.0.0.0/ /* default/kubernetes:https */
DNAT tcp -- * * 0.0.0.0/ 0.0.0.0/ /* default/kubernetes:https */ recent: SET name: KUBE-SEP-GIM2MHZZZBZJL55J side: source mask: 255.255.255.255 tcp to:192.168.0.105: Chain KUBE-SEP-IM4M52WKVEC4AZF3 ( references)
pkts bytes target prot opt in out source destination
KUBE-MARK-MASQ all -- * * 192.168.33.6 0.0.0.0/ /* default/coherenceproxysvc: */
DNAT tcp -- * * 0.0.0.0/ 0.0.0.0/ /* default/coherenceproxysvc: */ tcp to:192.168.33.6: Chain KUBE-SEP-LUF3R3GRCSK6KKRS ( references)
pkts bytes target prot opt in out source destination
KUBE-MARK-MASQ all -- * * 192.168.33.2 0.0.0.0/ /* kube-system/kube-dns:dns */
DNAT udp -- * * 0.0.0.0/ 0.0.0.0/ /* kube-system/kube-dns:dns */ udp to:192.168.33.2: Chain KUBE-SEP-ZZECWQBQCJPODCBC ( references)
pkts bytes target prot opt in out source destination
KUBE-MARK-MASQ all -- * * 192.168.33.5 0.0.0.0/ /* default/coherenceproxysvc: */
DNAT tcp -- * * 0.0.0.0/ 0.0.0.0/ /* default/coherenceproxysvc: */ tcp to:192.168.33.5: Chain KUBE-SERVICES ( references)
pkts bytes target prot opt in out source destination
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/ 10.254.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/ 10.254.254.254 /* kube-system/kube-dns:dns cluster IP */ udp dpt:
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/ 10.254.254.254 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:
KUBE-SVC-BQXHRGVXFCEH2BHH tcp -- * * 0.0.0.0/ 10.254.22.102 /* default/coherenceproxysvc: cluster IP */ tcp dpt:
KUBE-NODEPORTS all -- * * 0.0.0.0/ 0.0.0.0/ /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL Chain KUBE-SVC-BQXHRGVXFCEH2BHH ( references)
pkts bytes target prot opt in out source destination
KUBE-SEP-ZZECWQBQCJPODCBC all -- * * 0.0.0.0/ 0.0.0.0/ /* default/coherenceproxysvc: */ statistic mode random probability 0.50000000000
KUBE-SEP-IM4M52WKVEC4AZF3 all -- * * 0.0.0.0/ 0.0.0.0/ /* default/coherenceproxysvc: */ Chain KUBE-SVC-ERIFXISQEP7F7OF4 ( references)
pkts bytes target prot opt in out source destination
KUBE-SEP-67FRRWLKQK2OD4HZ all -- * * 0.0.0.0/ 0.0.0.0/ /* kube-system/kube-dns:dns-tcp */ Chain KUBE-SVC-NPX46M4PTMTKRN6Y ( references)
pkts bytes target prot opt in out source destination
KUBE-SEP-GIM2MHZZZBZJL55J all -- * * 0.0.0.0/ 0.0.0.0/ /* default/kubernetes:https */ recent: CHECK seconds: reap name: KUBE-SEP-GIM2MHZZZBZJL55J side: source mask: 255.255.255.255
KUBE-SEP-GIM2MHZZZBZJL55J all -- * * 0.0.0.0/ 0.0.0.0/ /* default/kubernetes:https */ Chain KUBE-SVC-TCOU7JCQXEZGVUNU ( references)
pkts bytes target prot opt in out source destination
KUBE-SEP-LUF3R3GRCSK6KKRS all -- * * 0.0.0.0/ 0.0.0.0/ /* kube-system/kube-dns:dns */
可优化之处：
- 构建image脚本沿用官方修改，很多不必要的地方可以删除
- cacheserver和proxy的image可以采用同一个，通过不同的参数调用不同的命令启动。
Kubernetes环境下如何运行Coherence缓存集群的更多相关文章
- ubuntu14.04环境下利用docker搭建solrCloud集群
在Ubuntu14.04操作系统的宿主机中,安装docker17.06.3,将宿主机的操作系统制作成docker基础镜像,之后使用自制的基础镜像在docker中启动3个容器,分配固定IP,再在3个容器 ...
- EHCache分布式缓存集群环境配置
EHCache分布式缓存集群环境配置 ehcache提供三种网络连接策略来实现集群,rmi,jgroup还有jms.同时ehcache可以可以实现多播的方式实现集群,也可以手动指定集群主机序列实现集群 ...
- Windows下ELK环境搭建（单机多节点集群部署）
1.背景 日志主要包括系统日志.应用程序日志和安全日志.系统运维和开发人员可以通过日志了解服务器软硬件信息.检查配置过程中的错误及错误发生的原因.经常分析日志可以了解服务器的负荷,性能安全性,从而及时 ...
- Kubernetes环境下的各种调试方法
作者:Jack47 转载请保留作者和原文出处 欢迎关注我的微信公众账号程序员杰克,两边的文章会同步,也可以添加我的RSS订阅源. 本文介绍在Kubernetes环境下的调试方法,希望对读者有用.如果关 ...
- 项目总结10：通过反射解决springboot环境下从redis取缓存进行转换时出现ClassCastException异常问题
通过反射解决springboot环境下从redis取缓存进行转换时出现ClassCastException异常问题 关键字 springboot热部署 ClassCastException异常 反射 ...
- EhCache 分布式缓存/缓存集群
开发环境: System:Windows JavaEE Server:tomcat5.0.2.8.tomcat6 JavaSDK: jdk6+ IDE:eclipse.MyEclipse 6.6 开发 ...
- EhCache 分布式缓存/缓存集群（转）
开发环境: System:Windows JavaEE Server:tomcat5.0.2.8.tomcat6 JavaSDK: jdk6+ IDE:eclipse.MyEclipse 6.6 开发 ...
- Window平台搭建Redis分布式缓存集群 （一）server搭建及性能測试
百度定义:Redis是一个key-value存储系统.和Memcached类似,它支持存储的value类型相对很多其它.包含string(字符串).list(链表).set(集合).zset(sort ...
- Linux系统下安装Redis和Redis集群配置
Linux系统下安装Redis和Redis集群配置 一. 下载.安装.配置环境: 1.1.>官网下载地址: https://redis.io/download (本人下载的是3.2.8版本:re ...
随机推荐
- 单选按钮 JradioButton 和复选框 JcheckBox 的使用
package first; import javax.swing.*; import java.awt.*; import java.awt.event.*; class BRTest extend ...
- 使用Redirector插件解决googleapis公共库加载的问题
最近访问一些面向国外的网站总是会出现ajax.googleaips.com无法加载的情况.以下为加载stackoverflow时的情境: 图1 -无法加载的google公共库 问题的原因是谷歌没有在国 ...
- python 微博评论获取的时候日期格式化
# -*- coding: utf-8 -*- # @Time : 2018/03/05 10:57 # @Author : cxa # @File : testDataTime.py # @Soft ...
- sql文格式替换
SELECT COUNT(1)FROM TAB1 TWHERE T.IS_DEL <> 0 将以上SQL格式替换为代码中的格式 sql.AppendLine(" SELECT & ...
- 【反演复习计划】【bzoj1011】zap-queries
快三个月没做反演题了吧…… 感觉高一上学期学的全忘了…… 所以还得从零开始学推式子. # bzoj1011 标签(空格分隔): 未分类 --- 原题意思是求以下式子:$Ans=\sum\limits_ ...
- 【POI2017||bzoj4726】Sabota?
上学期putsnan过了一次,这学期认真写了一遍…… #include<bits/stdc++.h> #define N 500010 using namespace std; ]; ,n ...
- JSOI2016酱油记
高一,第一次参加NOIP. 后悔初中没有报过名...唉,后悔也来不及了. 不知道自己一个暑假干了什么...算法没学多少,脑子倒是越来越不好使了. 过了初赛,数周后一脸茫(meng)然(bi)地去考场. ...
- java小工具——timer
定时器Timer 用途:做定时器用的. public static void main(String[] args) { Timer timer = new Timer(); timer.schedu ...
- 【转】Spring MVC 解读——<mvc:annotation-driven/>
转载自:http://my.oschina.net/HeliosFly/blog/205343 一.AnnotationDrivenBeanDefinitionParser 通常如果我们希望通过注解的 ...
- jQuery获取标签中的元素
获取双标签之间的内容 在JavaScript中,获取双标签之间的内容是这样的: <!DOCTYPE html> <html lang="en"> <h ...