shell脚本常规技巧
邮件相关
发送邮件：
#!/usr/bin/python import sys;
import smtplib;
from email.MIMEText import MIMEText mail_host = sys.argv[]
mail_user = sys.argv[]
mail_pass = sys.argv[]
mail_from = sys.argv[]
mail_to = sys.argv[]
mail_to_list = mail_to.split(';') mail_subject = sys.argv[]
mail_content = sys.argv[] msg = MIMEText(mail_content, _subtype = 'html', _charset = 'utf-8')
msg['From'] = mail_from
msg['Subject'] = mail_subject
msg['To'] = mail_to
msg['MIME-Version'] = "1.0"
msg['Content-Type'] = "text/html" try:
client = smtplib.SMTP()
client.connect(mail_host)
client.login(mail_user, mail_pass)
client.sendmail(mail_from, mail_to_list, msg.as_string())
except:
print "something error..."
用法：python ./send_mail.py $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
带附件版本：
#!/usr/bin/python import os
import sys;
import smtplib;
from email.MIMEText import MIMEText
from email.MIMEMultipart import MIMEMultipart mail_host = sys.argv[1]
mail_user = sys.argv[2]
mail_pass = sys.argv[3]
mail_from = sys.argv[4]
mail_to = sys.argv[5]
mail_to_list = mail_to.split(';') mail_subject = sys.argv[6]
mail_content = sys.argv[7]
mail_attach = '' if len(sys.argv) > 8:
mail_attach = sys.argv[8]
#print mail_attach msg = MIMEMultipart()
#
body = MIMEText(mail_content, _subtype = 'html', _charset = 'utf-8')
msg.attach(body)
#
if os.path.isfile(mail_attach):
att = MIMEText(open(mail_attach,'rb').read(),'base64','gb2312')
att["Content-Type"] = 'application/octet-stream'
att["Content-Disposition"] = 'attachment;filename="' + os.path.basename(mail_attach) +'"'
msg.attach(att) msg['From'] = mail_from
msg['Subject'] = mail_subject
msg['To'] = mail_to
msg['MIME-Version'] = "1.0"
msg['Content-Type'] = "text/html" try:
client = smtplib.SMTP()
client.connect(mail_host)
client.login(mail_user, mail_pass)
client.sendmail(mail_from, mail_to_list, msg.as_string())
except:
print "something error..."
发送脚本：
#! /bin/bash export LC_ALL="en_US.UTF-8" if [ $# -lt ]
then
echo "Usage: $0 process_name"
exit -
fi cd $(dirname $)
LOCALIP=`/sbin/ifconfig eth0 | grep "inet addr" | cut -f -d ":" | cut -f -d " "`
HOSTNAME=`hostname`
gday=`date +"%x %X"` MAIL_HOST="smtp.exmail.qq.com"
MAIL_USER="op@test.com"
MAIL_PASS="optest"
MAIL_FROM="op@test.com"
MAIL_TO="chenzhj@test.com"
MAIL_SUBJECT="$HOSTNAME[$LOCALIP] $1 $gday" MAIL_CONTENT="" echo "$MAIL_SUBJECT" #echo "python ./send_mail.py" $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
python ./send_mail.py $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
生成表格邮件：
#!/bin/sh TIME=`date +%Y-%m-%d -d "1 days ago"` echo "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">"
echo "<html xmlns=\"http://www.w3.org/1999/xhtml\">"
echo "<head>"
echo "<meta http-equiv=\"Content-Type\" content=\"text/html;charset=gb2312\">"
#echo "<meta http-equiv=Content-Type content=\"text/html; charset=utf-8\">"
echo "</head>"
echo " <body>"
echo " <h2>今日半价mysql数据统计</h2>"
echo ' <table border="1">'
echo " <tr>"
echo " <th>DateTime</th>"
echo " <th>统计站点</th>"
echo " <th>总商品数</th>"
echo " <th>在架商品数</th>"
echo " <th>下架商品数</th>"
echo " <th>增量总数</th>"
echo " <th>在架的增量数</th>"
echo " </tr>"
YESTERDAY=`date +%Y-%m-%d -d'-1 days'`
cat ../data/static.$YESTERDAY | while read line
do
echo $line | awk '
{ print "\t\t\t<tr>\n\t\t\t\t<td>'$TIME'</td>"
for (i=1; i<=6; i++) { print "\t\t\t\t<td>"$i"</td>"
}
print "\t\t\t</tr>"
}'
done
echo " </table>"
echo " </body>"
echo " </html>"
awk
1、传递外部shell参数(-v)、同时处理多文件（getline） 示例：
#!/bin/bash if [ $# -ne ]
then
echo "Usage: $0 local_log bc_log"
exit
fi awk -v bc_log=$ 'BEGIN{
SRC=;
TAR=;
/*print "bc_log:",bc_log;*/
}{
SRC=$; close(bc_log);
while(getline TAR < bc_log )
{
if(TAR>SRC)
{
print SRC;
break;
}
if(TAR==SRC)
{
break;
}
}
}
字典过滤
#! /bin/gawk -f
BEGIN{
total=;
while( getline< "/home/hadoop/zhenjing/solr/2.data" >)
{
dict[$] = ;
total++;
}
#print total;
}
{
if( $ in dict ){
next
}
print $
}
END{
}
分析日志，计算平均时间
#! /bin/gawk -f
START {
sum_time["item.get"] = ;
sum_count["item.get"] = ;
}
{
tmp = substr($, index($, "method"));
split(tmp, arr, "\"");
method = arr[];
if(method == "") {
next;
}
tmp = substr($, index($, "time:"));
split(tmp, arr, ":");
time = arr[];
# print method, time;
if(time > ){
sum_time[method] = sum_time[method] + time;
sum_count[method] = sum_count[method] + ;
}
}
END {
for(method in sum_time){
if(sum_count[method] > ){
print method " = " sum_time[method] / sum_count[method];
}
}
}
awk统计tomcat android机型：
#! /bin/gawk -f
# tail /tmp/temai.txt | grep Android | cut -d" " -f - | cut -d')' -f1
BEGIN{
FS = "; ";
}
{
if( $ in dict_os ){
dict_os[$] = dict_os[$] + ;
} else {
dict_os[$] = ;
}
if( $ in dict_jixing ){
dict_jixing[$] = dict_jixing[$] + ;
} else {
dict_jixing[$] = ;
}
}
END {
for(os in dict_os){
print dict_os[os] " " os;
}
for(jx in dict_jixing){
print dict_jixing[jx] " " jx;
awk平均分拆文件：
awk '{hash=int(NR % 10 ); print $0 >> "outfile_tmp_"hash }' weiboid.txt
时间相关
基于时间的循环：date命令
typeset readonly START_DATE=$(date --date="$1" +%Y%m%d);
for ((CUR_DATE=START_DATE; CUR_DATE<=END_DATE; CUR_DATE=$(date --date="${CUR_DATE} 1 days" +%Y%m%d) ))
Linux时钟分为系统时钟（System Clock）和硬件（Real Time Clock，简称RTC）时钟。系统时钟是指当前Linux Kernel中的时钟，而硬件时钟则是主板上由电池供电的时钟，这个硬件时钟可以在BIOS中进行设置。当Linux启动时，硬件时钟会去读取系统时钟的设置，然后系统时钟就会独立于硬件运作。Linux中的所有命令（包括函数）都是采用的系统时钟设置。在Linux中，用于时钟查看和设置的命令主要有date、hwclock和clock。其中，clock和hwclock用法相近。
date 010318242008.30(月日时分年.秒)
date 010318242008(月日时分年)
date -s 20080103
date -s 18:24
clock -w
hwclock --set --date="07/07/06 10:19" （月/日/年 时:分:秒）
硬件时钟与系统时钟同步：# hwclock --hctosys（hc代表硬件时间，sys代表系统时间）或者# clock --hctosys
系统时钟和硬件时钟同步：# hwclock --systohc或者# clock --systohc
xargs
echo "FILE.txt" | grep -v '^$' \
| xargs -L 1 -I {} find . -maxdepth 1 -type f -name "{}" \
| sort | xargs -L 1 basename | xargs md5sum
somecommand | xargs -s 50000 echo | xargs -I '{}' -s 100000 rm '{}'
LOCALIP=`/sbin/ifconfig eth1 | grep "inet addr" | cut -f 2 -d ":" | cut -f 1 -d " "`
LOCALIP="$(/sbin/ifconfig eth1 | sed -n '/inet addr/s/.*inet addr:\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*/\1/gp')"
LOCAL_IP="$(/sbin/ifconfig | sed -n 's/^.*inet addr\:\([[:digit:]\.]\{7,14\}\).*$/\1/gp' | head -1)";
array 相关
#! /bin/bash
colors=(red green blue)
element_count=${#colors[@]}
# Special syntax to extract number of elements in array.
# element_count=${#colors[*]} works also.
#
# The "@" variable allows word splitting within quotes
#+ (extracts variables separated by whitespace).
#
# This corresponds to the behavior of "$@" and "$*"
#+ in positional parameters.
# array loop:
# Way :
index=
while [ "$index" -lt "$element_count" ]
do # List all the elements in the array.
echo ${colors[$index]}
# ${colors[index]} also works because it's within ${ ... } brackets.
let "index = $index + 1"
# Or:
# ((index++))
done
# Way :
# Each array element listed on a separate line.
# If this is not desired, use echo -n "${colors[$index]} "
# Doing it with a "for" loop instead:
for i in "${colors[@]}"
do
echo "$i"
done
echo
echo "For test:"
# Usage: ArrayCheckExist "$value" "${arr[@]}"
function ArrayCheckExist
{
value=$
i=
for str in "$@"
do
((i++))
if [ $i -eq ]; then # first parameter is checked value
continue
fi
if [[ "$value" == "$str" ]]; then
#echo "$value" "$str"
return # exist
fi
done
return
}
dirs=() # empty array
index=
while read ip dir info tomcat port
do
if [[ $ip == "#" || -z $port ]]
then
continue
fi
# index=`echo -n $tomcat |md5sum|cut -d ' ' -f1` # get string md5sum
ArrayCheckExist "$tomcat" "${dirs[@]}"
if [ $? -eq ]; then
dirs[ ${#dirs[@]} ]=$tomcat
fi
done < ./di.conf
for i in "${dirs[@]}"
do
echo "$i"
done
echo "Array function test:"
ArrayCheckExist "${dirs[@]}"
return_val=$?
if [[ $return_val -gt ]]; then
echo "${dirs[0]} Exist."
fi
ArrayCheckExist "test" "${dirs[@]}"
if [ $? -gt ]; then
echo "test Exist."
fi
ArrayCheckExist "test"
if [ $? -gt ]; then
echo "test Exist."
fi
ArrayCheckExist "/usr/local/webserver/di/tomcat-di2" "${dirs[@]}"
if [ $? -gt ]; then
echo "Exist."
fi
统计脚本
solr：
cat shards | awk -F "QTime=" '{sum+=$2; cnt++; if($2> 500) slow++;} END{print cnt; print slow++; print sum/cnt; print slow/cnt}'
dc：
cat dc.log | grep -o "spending time:[0-9]\+" | awk -F: '{sum+= $2; cnt++; if($2 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
rts：
cat dc.log | grep -o "search rts used: [0-9]\+" | awk -F: '{sum+= $2; cnt++; if($2 > 20) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
searchserver：
cat searchserver.log.2013-07-08.log | grep -o "totalTime=[0-9]\+" | awk -F= '{sum+= $2; cnt++; if($2 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
appserver:
cat userTime-appserver-2013-07-25_* | awk '{sum+= $5; cnt++; if($5 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
solr相关
solr core数据对比
#! /bin/bash if [ $# -lt ]
then
echo "Usage $0 query_file solr1 solr2 [out_file]"
exit -
fi OUT_FILE="/tmp/query"
if [ $# -gt ]
then
OUT_FILE=$
fi while read query
do
query=`echo $query | sed 's/wt=javabin/wt=json/g'` # KEY: set wt=json for solr request.
wget "http://$2/solr/C2C_TAOBAO_0/select/?$query" -q -O ${OUT_FILE}_0
wget "http://$3/solr/C2C_TAOBAO_0/select/?$query" -q -O ${OUT_FILE}_1 # del QTime from solr "json" response
sed -i 's/"QTime":[0-9]\+,//g' ${OUT_FILE}_0
sed -i 's/"QTime":[0-9]\+,//g' ${OUT_FILE}_1 cmp ${OUT_FILE}_0 ${OUT_FILE}_1 > /dev/null
if [ "$?" != ]
then
echo "$query"
fi done < $
solr查询参数提取
#! /bin/gawk -f
/select params=\{/,/\}/ {
if( match($, /\{[^\}]+\}/, str)){
pos = length(str[]) -
print substr(str[], , pos)
}
}
获取淘宝真实链接：
#! /bin/bash if [ $# -lt ]
then
echo "Usage: $0 url"
exit
fi export LC_ALL=en_US.UTF-8
OUT=log wget $ -o $OUT -O tmp req_arr=`grep "Location:" $OUT | grep "s.click.taobao.com" | cut -d" " -f ` taobao_req=`echo $req_arr | cut -d" " -f `
refer_req=`echo $req_arr | cut -d" " -f ` ext_key=`echo $refer_req | awk -F"et%3D" '{print $2}' | sed -e 's/%25/%/g'` #echo "$taobao_req&ref=&et=$ext_key"
#echo "curl \"$taobao_req&ref=&et=$ext_key\" -I -e \"$refer_req\""
#curl -s "$taobao_req&ref=&et=$ext_key" -I -e "$refer_req" curl -s "$taobao_req&ref=&et=$ext_key" -I -e "$refer_req" | grep "^Location" | cut -d" " -f
日常数据备份脚本
#! /bin/bash export LC_ALL=en_US.UTF-
echo `date`
cd `dirname $`
pwd echo `date +%s`
# back taday data
DATE=`date +%Y-%m-%d` SQL_FILE="jira_$DATE.sql"
/opt/lampp/bin/mysqldump --default-character-set=utf8 -h 10.1.22.40 -u jira -pkoudai123 jira > $SQL_FILE r1=`ls -ld $SQL_FILE | awk '{print int($5/1024)}'` DATA_FILE="jira_home_$DATE.tar.gz"
tar czf $DATA_FILE /data/server/jira-home r2=`ls -ld $DATA_FILE | awk '{print int($5/1024)}'` echo $r1
echo $r2 echo `date +%s` if [[ $r1 -gt && $r2 -gt ]]
then
# del old data
DELDAY=`date --date='-15 day' +%Y-%m-%d` echo "del data : $DELDAY" rm *$DELDAY* fi echo "`date +%s` FINISH !"
操作文件
#! /bin/bash if [ $# -lt ]
then
echo "Usage $0 in_file out_file num"
exit -
fi INPUT=$
OUTPUT=$ num=
if [ $# -gt ]
then
num=$
fi for i in `seq $num`
do
echo "$i" awk -F ^ -v prefix=$i '{ if(NF > 10) print prefix$0}' ${INPUT} >> ${OUTPUT}
done
shell脚本常规技巧的更多相关文章
- shell脚本常用技巧
shell脚本常用技巧 1.获取随机字符串或数字 ~]#echo $RANDOM | md5sum | cut -c 1-6 ~]#openssl rand -base64 4 | cut -c 1- ...
- shell脚本使用技巧3--调试
1.使用-x,开启shell脚本的跟踪调试功能 ex:bash -x script.sh or sh -x script.sh 2.使用set -x 和 set +x对脚本进行部分调试(输入中间的内容 ...
- shell脚本调试技巧
shell脚本调试之工具——bashdb http://www.cnblogs.com/itcomputer/p/5011845.html
- SHELL 脚本小技巧
脚本很简单,直接上功能介绍及脚本,可以做模板使用: 记录日志,记录脚本开始执行时间.结束时间 usage 函数,脚本需接参数执行,避免误执行,告诉用户,这个脚本的使用方法 加锁,创建锁文件,脚本不允许 ...
- shell脚本小技巧
输入参数错误时,退格会出现^H,这个时候只要在脚本顶部加一条语句:stty erase ^h就可以了 #!/bin/sh stty erase ^h
- shell脚本使用技巧5--字符分隔
#!/bin/bash #filename:ifs.sh data="name,sex,rollon,location" oldIFS=$IFS IFS=, for item in ...
- shell脚本使用技巧4--读取字符,重复执行
ls | cat -n > out.txt 给输出的信息加行号并导出到out.txt 利用shell生成一个独立的进程 pwd; (cd /bin; ls); pwd; 开启一个子shell,不 ...
- shell脚本使用技巧2
0--stdin标准输入 1--stdout标准输出 2--stderr标准错误 重定向 echo "this is a good idea " > temp.txt tem ...
- Shell脚本小技巧收集
1.使用python快速搭建一个web服务器 访问端口8000 python -m SimpleHTTPServer 2.获取文件大小 stat -c %s $file stat --printf=' ...
随机推荐
- AndroidLinker与SO加壳技术之下篇
点此查看上篇<AndroidLinker与SO加壳技术之上篇> 2.4 链接 链接过程由 soinfo_link_image 函数完成,主要可以分为四个主要步骤: 1. 定位 dynami ...
- [学习笔记]坚果云网盘，SVN异地代码管理
SVN的好处不必多说了.但是如果希望有一份自己的用来学习和储备的代码仓库,那么能够异地同步是必不可少的了. 参考作者Mike_QSJ的文章,但是实际上做了很大的改动.一方面使用更常见的windows系 ...
- tomcat连接池配置详解
<bean class="org.apache.tomcat.jdbc.pool.PoolProperties"> <property name="ur ...
- prism4 StockTrader RI 项目分析一些体会2
prism 对于逻辑复杂的页面,通过建立 controller实现逻辑管理 按着一般的做法就是,各模块的viewmodel import由各模块去实例化(理解有限),但是通过controller实现了 ...
- <转>浅析长度为0的数组
前面在看Xen的源码时,遇到了一段代码,如下所示: 注意上面最后一行的代码,这里定义了一个长度为的数组,这种用法可以吗?为什么可以使用长度为0 的数组?长度为的数组到底怎么使用?……这篇文章主要针对该 ...
- group_concat函数详解
来自: http://hchmsguo.iteye.com/blog/555543 MySQL中group_concat函数 完整的语法如下: group_concat([DISTINCT] 要连接的 ...
- Web调用FastReport的配置问题
1.修改配置webconfig文件 IIS6: <system.web> <httpHandlers> <add path="FastReport.Export ...
- .net之微信企业号开发(三) 回调模式的接口开发
一.前言 微信企业号应用中,有两种模式,一种是普通模式,这种模式只能进行简单网页链接,以及发送固定的消息.为了可以让企业号的用户更好的与应用交互,微信提供了回调模式,这种回调模式的可以将用户发送给微信 ...
- 【转】IE劫持原理 BHO
为什么"浏览器劫持"能够如此猖狂呢?放眼众多论坛的求助贴,我们不时可以看到诸如"我的IE被主页被改了,我用杀毒工具扫了一遍都没发现病毒,我把主页改回自己的地址,可是一重启 ...
- 第59讲：Scala中隐式转换初体验
今天学习了下隐式转换的内容.所谓隐式转换,就是说,一个实例拥用1 2 3方法,但是当它需要4方法的时候,它没有,但是却可以通过转换成另一种类型来调用4方法,而且这种转换是自动转换不需要人为干预的,这种 ...