lvs+keepalived+nginx实现高性能负载均衡集群
一、为什么要使用负载均衡技术？
1、系统高可用性
2、 系统可扩展性
3、 负载均衡能力
LVS+keepalived能很好的实现以上的要求，LVS提供负载均衡，keepalived提供健康检查，故障转移，提高系统的可用性！采用这样的架构以后很容易对现有系统进行扩展，只要在后端添加或者减少realserver，只要更改lvs的配置文件，并能实现无缝配置变更！
二、LVS+Keepalived介绍
1、 LVS
LVS是一个开源的软件，可以实现LINUX平台下的简单负载均衡。LVS是Linux Virtual Server的缩写，意思是Linux虚拟服务器。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；八种调度算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。
2、 keepalived
Keepalived 是运行在lvs 之上，它的主要功能是实现真实机的故障隔离及负载均衡器间的失败切换，提高系统的可用性
三、环境：
四台服务器，系统全为CentOS6.8：
192.168.2.203 master lvs+keepalived
192.168.2.202 backup lvs+keepalived
192.168.2.204 web1（nginx）
192.168.2.205 web2 （nginx）
vip：192.168.2.13
其中nginx已预装好，这里不再写搭建过程
四、搭建并配置
1、分别在backup lvs和master lvs上安装lvs
root@bogon src]# yum -y install ipvsadm
已加载插件：fastestmirror
设置安装进程
Determining fastest mirrors
epel/metalink | 5.4 kB :
* base: mirror.lzu.edu.cn
... ...
已安装:
ipvsadm.x86_64 :1.26-.el6 作为依赖被安装:
libnl.x86_64 :1.1.-.el6 完毕！
2、把ipvsadm模块加载进系统
[root@bogon src]# ipvsadm
IP Virtual Server version 1.2. (size=)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@bogon src]# lsmod | grep ip_vs
ip_vs
libcrc32c ip_vs
ipv6 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
3、分别在backup lvs和master lvs上安装keepalived（keepalived官网：http://www.keepalived.org/）
[root@bogon src]# tar zxf keepalived-1.2..tar.gz
[root@bogon src]# cd keepalived-1.2.
[root@bogon keepalived-1.2.]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.-642.3..el6.x86_64/
报错：
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
系统缺少openssl-devel包所致
安装openssl-devel
root@bogon keepalived-1.2.]# yum -y install openssl-devel
再次编辑安装
[root@bogon keepalived-1.2.]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.-642.3..el6.x86_64/
Keepalived configuration
------------------------
Keepalived version : 1.2.
Compiler : gcc
Preprocessor flags : -I/lib/modules/2.6.-642.3..el6.x86_64//include
Compiler flags : -Wall -Wunused -Wstrict-prototypes
Linker flags :
Extra Lib : -ldl -lssl -lcrypto
Use IPVS Framework : Yes
IPVS use libnl : No
IPVS syncd attributes : No
IPVS bit stats : No
fwmark socket support : Yes
Use VRRP Framework : Yes
Use VRRP VMAC : Yes
Use VRRP authentication : Yes
With ip rules/routes : Yes
SNMP keepalived support : No
SNMP checker support : No
SNMP RFCv2 support : No
SNMP RFCv3 support : No
SHA1 support : No
Use Debug flags : No
Stacktrace support : No
Memory alloc check : No
libnl version : None
Use IPv4 devconf : No
Use libiptc : No
Use libipset : No
Build genhash : Yes
Build documentation : No [root@bogon keepalived-1.2.]# make && make install
[root@bogon keepalived-1.2.]# ln -s /usr/local/sbin/keepalived /sbin/
[root@bogon keepalived-1.2.]# chkconfig --add keepalived
[root@bogon keepalived-1.2.]# chkconfig --level keepalived on
4、配置keepalived
lvs-master的配置文件如下
[root@bogon keepalived-1.2.]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { #全局配置部分
# notification_email { #email 通知，基本不用此处所以注释掉
# acassen@firewall.loc
# failover@firewall.loc
# sysadmin@firewall.loc
# }
# notification_email_from Alexandre.Cassen@firewall.loc
# smtp_server 192.168.200.1
# smtp_connect_timeout
router_id LVS_DEVEL # 设置lvs的id，在一个网络内应该是唯一的
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval
vrrp_gna_interval
} vrrp_instance VI_1 { #vrrp实例定义部分
state MASTER #设置lvs的状态，报错MASTER和BACKUP两种，必须大写
interface eth1 #设置对外服务的接口
virtual_router_id 60 #设置虚拟路由标示，这个标示是一个数字，同一个vrrp实例使用唯一标示
priority 100 #定义优先级，数字越大优先级越高，在一个vrrp——instance下，master的优先级必须大于backup
advert_int 1 #设定master与backup负载均衡器之间同步检查的时间间隔，单位是秒
authentication { #设置验证类型和密码
auth_type PASS #主要有PASS和AH两种
auth_pass 1111 #验证密码，同一个vrrp_instance下MASTER和BACKUP密码必须相同
}
virtual_ipaddress { #设置虚拟ip地址，可以设置多个，每行一个
192.168.2.13
}
} virtual_server 192.168.2.13 { #设置虚拟服务器，需要指定虚拟ip和服务端口
delay_loop 3 #健康检查时间间隔
lb_algo rr #负载均衡调度算法
lb_kind DR #负载均衡转发规则
persistence_timeout 50 #设置会话保持时间，对动态网页非常有用
protocol TCP #指定转发协议类型，有TCP和UDP两种 real_server 192.168.2.204 { #配置服务器节点1，需要指定real server的真实IP地址和端口
weight 1 #设置权重，数字越大权重越高
TCP_CHECK { #realserver的状态监测设置部分单位秒
connect_timeout 3 #超时时间
nb_get_retry 3 #重试次数
delay_before_retry 3 #重试间隔
connect_port #监测端口
}
}
real_server 192.168.2.205 {
weight
TCP_CHECK {
connect_timeout
nb_get_retry
delay_before_retry
connect_port
}
} }
LVS-backup的配置文件如下
[root@bogon keepalived-1.2.]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs {
# notification_email {
# acassen@firewall.loc
# failover@firewall.loc
# sysadmin@firewall.loc
# }
# notification_email_from Alexandre.Cassen@firewall.loc
# smtp_server 192.168.200.1
smtp_connect_timeout
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval
vrrp_gna_interval
} vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.2.13
}
} virtual_server 192.168.2.13 {
delay_loop
lb_algo rr
lb_kind DR
persistence_timeout
protocol TCP real_server 192.168.2.204 {
weight
TCP_CHECK {
connect_timeout
nb_get_retry
delay_before_retry
connect_port
}
}
real_server 192.168.2.205 {
weight
TCP_CHECK {
connect_timeout
nb_get_retry
delay_before_retry
connect_port
}
} }
5、realserver的配置
两台web服务器都要执行下面脚本
[root@bogon www]# cat /etc/rc.d/init.d/realserver.sh
#!/bin/bash
# description: Config realserver lo and apply noarp SNS_VIP=192.168.2.13 /etc/rc.d/init.d/functions case "$1" in
start)
ifconfig lo: $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
/sbin/route add -host $SNS_VIP dev lo:
echo "" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null >&
echo "RealServer Start OK" ;;
stop)
ifconfig lo: down
route del $SNS_VIP >/dev/null >&
echo "" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit
esac exit
[root@bogon www]# /etc/rc.d/init.d/realserver.sh start
/etc/rc.d/init.d/realserver.sh: line : /etc/rc.d/init.d/functions: 权限不够
RealServer Start OK
[root@bogon www]# ifconfig
eth0 Link encap:Ethernet HWaddr :0C::::DF
inet addr:192.168.12.129 Bcast:192.168.12.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe41:71df/ Scope:Link
UP BROADCAST RUNNING MULTICAST MTU: Metric:
RX packets: errors: dropped: overruns: frame:
TX packets: errors: dropped: overruns: carrier:
collisions: txqueuelen:
RX bytes: (134.0 KiB) TX bytes: (7.1 KiB) eth1 Link encap:Ethernet HWaddr :0C::::E9
inet addr:192.168.2.204 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe41:71e9/ Scope:Link
UP BROADCAST RUNNING MULTICAST MTU: Metric:
RX packets: errors: dropped: overruns: frame:
TX packets: errors: dropped: overruns: carrier:
collisions: txqueuelen:
RX bytes: (22.3 MiB) TX bytes: (2.0 MiB) lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::/ Scope:Host
UP LOOPBACK RUNNING MTU: Metric:
RX packets: errors: dropped: overruns: frame:
TX packets: errors: dropped: overruns: carrier:
collisions: txqueuelen:
RX bytes: (182.0 b) TX bytes: (182.0 b) lo: Link encap:Local Loopback
inet addr:192.168.2.13 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU: Metric:
6、启动keepalived并进行测试
[root@bogon keepalived-1.2.]# service keepalived start
正在启动 keepalived： [确定]
lvs-master
[root@bogon keepalived-1.2.]# tail -f /var/log/messages
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_healthcheckers[]: Netlink reflector reports IP 192.168.2.13 added
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
[root@bogon keepalived-1.2.]# ipvsadm -L -n
IP Virtual Server version 1.2. (size=)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.2.13: rr persistent
-> 192.168.2.204: Route
-> 192.168.2.205: Route
访问curl http://192.168.2.13/test.txt
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
关掉web2再次测试
[root@www etc]# curl http://192.168.2.13/test.txt
it is web1
[root@www etc]# curl http://192.168.2.13/test.txt
it is web1
查看lvs-master
[root@bogon keepalived-1.2.]# ipvsadm -L -n
IP Virtual Server version 1.2. (size=)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.2.13: rr persistent
-> 192.168.2.204: Route
[root@bogon keepalived-1.2.]# tail -f /var/log/messages
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_healthcheckers[]: TCP connection to [192.168.2.205]: failed.
Oct :: bogon Keepalived_healthcheckers[]: TCP connection to [192.168.2.205]: failed.
Oct :: bogon Keepalived_healthcheckers[]: Check on service [192.168.2.205]: failed after retry.
Oct :: bogon Keepalived_healthcheckers[]: Removing service [192.168.2.205]: from VS [192.168.2.13]:
Oct :: bogon Keepalived_healthcheckers[]: Remote SMTP server [192.168.200.1]: connected.
Oct :: bogon Keepalived_healthcheckers[]: Timeout reading data to remote SMTP server [192.168.200.1]:.
已经自动把web2剔除
打开web2再次查看
[root@bogon keepalived-1.2.]# ipvsadm -L -n
IP Virtual Server version 1.2. (size=)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.2.13: rr persistent
-> 192.168.2.204: Route
-> 192.168.2.205: Route
[root@bogon keepalived-1.2.]# tail -f /var/log/messages
Oct :: bogon Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: bogon Keepalived_healthcheckers[]: TCP connection to [192.168.2.205]: failed.
Oct :: bogon Keepalived_healthcheckers[]: TCP connection to [192.168.2.205]: failed.
Oct :: bogon Keepalived_healthcheckers[]: Check on service [192.168.2.205]: failed after retry.
Oct :: bogon Keepalived_healthcheckers[]: Removing service [192.168.2.205]: from VS [192.168.2.13]:
Oct :: bogon Keepalived_healthcheckers[]: Remote SMTP server [192.168.200.1]: connected.
Oct :: bogon Keepalived_healthcheckers[]: Timeout reading data to remote SMTP server [192.168.200.1]:.
Oct :: bogon Keepalived_healthcheckers[]: TCP connection to [192.168.2.205]: success.
Oct :: bogon Keepalived_healthcheckers[]: Adding service [192.168.2.205]: to VS [192.168.2.13]:
Oct :: bogon Keepalived_healthcheckers[]: Remote SMTP server [192.168.200.1]: connected.
恢复后已自动添加进来
关掉lvs master的keepalived
[root@bogon keepalived-1.2.]# service keepalived stop
停止 keepalived： [确定]
访问web并查看lvs backup
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@lys2 src]# tail -f /var/log/messages
Oct :: lys2 Keepalived_vrrp[]: VRRP_Instance(VI_1) Transition to MASTER STATE
Oct :: lys2 Keepalived_vrrp[]: VRRP_Instance(VI_1) Entering MASTER STATE
Oct :: lys2 Keepalived_vrrp[]: VRRP_Instance(VI_1) setting protocol VIPs.
Oct :: lys2 Keepalived_healthcheckers[]: Netlink reflector reports IP 192.168.2.13 added
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct :: lys2 Keepalived_vrrp[]: Sending gratuitous ARP on eth1 for 192.168.2.13
[root@lys2 src]# ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c:::0f:e3 brd ff:ff:ff:ff:ff:ff
inet 192.168.144.101/ brd 192.168.144.255 scope global eth0
inet6 fe80::20c:29ff:fe89:fe3/ scope link
valid_lft forever preferred_lft forever
: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c:::0f:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.2.202/ brd 192.168.2.255 scope global eth1
inet 192.168.2.13/ scope global eth1
inet6 fe80::20c:29ff:fe89:fed/ scope link
valid_lft forever preferred_lft forever
可以看到lvs backup已自动切换成master状态并自动绑定了vip
查看lvs master vip
[root@bogon keepalived-1.2.]# ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c:::4d:7a brd ff:ff:ff:ff:ff:ff
inet 192.168.12.128/ brd 192.168.12.255 scope global eth0
inet6 fe80::20c:29ff:fe55:4d7a/ scope link
valid_lft forever preferred_lft forever
: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c:::4d: brd ff:ff:ff:ff:ff:ff
inet 192.168.2.203/ brd 192.168.2.255 scope global eth1
inet6 fe80::20c:29ff:fe55:4d84/ scope link
valid_lft forever preferred_lft forever
已自动解除vip
到处全部结束
lvs+keepalived+nginx实现高性能负载均衡集群的更多相关文章
- lvs+keepalived+nginx实现高性能负载均衡集群【转】
转自 lvs+keepalived+nginx实现高性能负载均衡集群 - 青衫lys - 博客园http://www.cnblogs.com/liuyisai/p/5990645.html 一.为什么 ...
- 【Nginx】(主从热备)LVS+Keepalived+Nginx实现高性能负载均衡集群
一.LVS 1.1 概述 1.2 Nginx与LVS区别什么 二.Keepalived 2.1 概述 2.2 keepalived和其工作原理 三.LVS+Keepalived+Nginx 搭建双机主 ...
- LVS+Keepalived 实现高可用负载均衡集群
LVS+Keepalived 实现高可用负载均衡集群 随着网站业务量的增长,网站的服务器压力越来越大?需要负载均衡方案!商业的硬件如 F5 ,Array又太贵,你们又是创业型互联公司如何有效 ...
- Nginx+Tomcat高性能负载均衡集群搭建
转载请注明出处:http://www.cnblogs.com/Joanna-Yan/p/8745794.html Nginx是一个高性能的HTTP服务器/反向代理服务器及电子邮件(IMAP/POP3) ...
- Nginx+Tomcat+Memcached负载均衡集群服务搭建
操作系统:CentOS6.5 本文档主要讲解,如何在CentOS6.5下搭建Nginx+Tomcat+Memcached负载均衡集群服务器,Nginx负责负载均衡,Tomcat负责实际服务,Memc ...
- Nginx+Tomcat7+Mencached负载均衡集群部署笔记
Nginx+Tomcat+Memcached负载均衡集群服务搭建 操作系统:CentOS6.5 本文档主要解说,怎样在CentOS6.5下搭建Nginx+Tomcat+Memcached负载均衡集群s ...
- 简单的 Nginx+Tomcat 配置负载均衡集群
简单 Nginx+Tomcat 配置负载均衡集群 前期准备 解压两个tomcat,修改端口号 server1:8081 server:8082 同时启动 nginx官网下载解压版nginx 创建一个简 ...
- Nginx+Tomcat搭建高性能负载均衡集群
一. 工具 nginx-1.8.0 apache-tomcat-6.0.33 二. 目标 实现高性能负载均衡的Tomcat集群: 三. 步骤 1.首先下载Nginx ...
- Nginx+Tomcat搭建高性能负载均衡集群(转)
转载自:http://blog.csdn.net/wang379275614/article/details/47778201 一. 工具 nginx-1.8.0 apache-tomca ...
随机推荐
- 关于Python3爬虫抓取网页Unicode
import urllib.requestresponse = urllib.request.urlopen('http://www.baidu.com')html = response.read() ...
- 后台使用Spring MVC 4.15 版本 通过 ajaxFileUpload plugin插件上传文件相应时引起的一个小问题，Chrome、Firefox中出现SyntaxError:unexpected token <
html: 使用ajaxFileUpload插件做文件上传时,后台返回json格式的数据,js代码如下: 接下来,把结果错误信息打印出来: 先在网上找了下解决办法方案,stackoverflow上有说 ...
- Java 7 新特性
try( InputStream is = new FileInputStream(path); XSSFWorkbook xssfWorkbook = new XSSFWorkbook(is); ) ...
- git乱码问题解决
etc\gitconfig文件中增加以下内容: [gui] encoding = utf-8 [i18n] commitencoding = gbk [svn] pathnam ...
- NPM 相关
1. 官方网站 https://docs.npmjs.com/ 2. 显示Global Module安装过什么 p.p1 { margin: 0.0px 0.0px 0.0px 0.0px; font ...
- Eclipse 4.2 (Juno) 'Cannot create a server using the selected type' in Tomcat 7 - Stack Overflow.html
[太神奇了,真的可以呀] 原文:http://stackoverflow.com/questions/13423593/eclipse-4-2-juno-cannot-create-a-server- ...
- 解决Bringing up interface eth0: Device eth0 does not seem to be present, delaying initialization.
一.问题描述 OS:centos 原因是拷贝虚拟机造成的. 使用vmworkstation打开虚拟机的时候,要选择copy而非move. 二.解决描述 网络上解决步骤各异,其实就一句话.只要保证vmw ...
- favicon.ico应用与正则表达式验证邮箱（可自动删除前后的空格）
1.favicon.ico制作:favicon.ico可以ps制作;“shortcut icon”中间有一个空格 <head> <link rel="shortcut ic ...
- java -json()
json-lib和org.json的使用几乎是相同的,我总结出的区别有两点: 两种包 1. List集合转换成json方法 List list = new ArrayList(); list.add( ...
- RabbitMQ的work queue(1)
http://www.rabbitmq.com/tutorials/tutorial-two-java.html 在第一个教程中,我们通过一个命名队列来发送消息和接受消息.在这一节,我们将创建一个工作 ...