shell脚本常规技巧
邮件相关
发送邮件：
#!/usr/bin/python import sys;
import smtplib;
from email.MIMEText import MIMEText mail_host = sys.argv[]
mail_user = sys.argv[]
mail_pass = sys.argv[]
mail_from = sys.argv[]
mail_to = sys.argv[]
mail_to_list = mail_to.split(';') mail_subject = sys.argv[]
mail_content = sys.argv[] msg = MIMEText(mail_content, _subtype = 'html', _charset = 'utf-8')
msg['From'] = mail_from
msg['Subject'] = mail_subject
msg['To'] = mail_to
msg['MIME-Version'] = "1.0"
msg['Content-Type'] = "text/html" try:
client = smtplib.SMTP()
client.connect(mail_host)
client.login(mail_user, mail_pass)
client.sendmail(mail_from, mail_to_list, msg.as_string())
except:
print "something error..."
用法：python ./send_mail.py $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
带附件版本：
#!/usr/bin/python import os
import sys;
import smtplib;
from email.MIMEText import MIMEText
from email.MIMEMultipart import MIMEMultipart mail_host = sys.argv[1]
mail_user = sys.argv[2]
mail_pass = sys.argv[3]
mail_from = sys.argv[4]
mail_to = sys.argv[5]
mail_to_list = mail_to.split(';') mail_subject = sys.argv[6]
mail_content = sys.argv[7]
mail_attach = '' if len(sys.argv) > 8:
mail_attach = sys.argv[8]
#print mail_attach msg = MIMEMultipart()
#
body = MIMEText(mail_content, _subtype = 'html', _charset = 'utf-8')
msg.attach(body)
#
if os.path.isfile(mail_attach):
att = MIMEText(open(mail_attach,'rb').read(),'base64','gb2312')
att["Content-Type"] = 'application/octet-stream'
att["Content-Disposition"] = 'attachment;filename="' + os.path.basename(mail_attach) +'"'
msg.attach(att) msg['From'] = mail_from
msg['Subject'] = mail_subject
msg['To'] = mail_to
msg['MIME-Version'] = "1.0"
msg['Content-Type'] = "text/html" try:
client = smtplib.SMTP()
client.connect(mail_host)
client.login(mail_user, mail_pass)
client.sendmail(mail_from, mail_to_list, msg.as_string())
except:
print "something error..."
发送脚本：
#! /bin/bash export LC_ALL="en_US.UTF-8" if [ $# -lt ]
then
echo "Usage: $0 process_name"
exit -
fi cd $(dirname $)
LOCALIP=`/sbin/ifconfig eth0 | grep "inet addr" | cut -f -d ":" | cut -f -d " "`
HOSTNAME=`hostname`
gday=`date +"%x %X"` MAIL_HOST="smtp.exmail.qq.com"
MAIL_USER="op@test.com"
MAIL_PASS="optest"
MAIL_FROM="op@test.com"
MAIL_TO="chenzhj@test.com"
MAIL_SUBJECT="$HOSTNAME[$LOCALIP] $1 $gday" MAIL_CONTENT="" echo "$MAIL_SUBJECT" #echo "python ./send_mail.py" $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
python ./send_mail.py $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
生成表格邮件：
#!/bin/sh TIME=`date +%Y-%m-%d -d "1 days ago"` echo "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">"
echo "<html xmlns=\"http://www.w3.org/1999/xhtml\">"
echo "<head>"
echo "<meta http-equiv=\"Content-Type\" content=\"text/html;charset=gb2312\">"
#echo "<meta http-equiv=Content-Type content=\"text/html; charset=utf-8\">"
echo "</head>"
echo " <body>"
echo " <h2>今日半价mysql数据统计</h2>"
echo ' <table border="1">'
echo " <tr>"
echo " <th>DateTime</th>"
echo " <th>统计站点</th>"
echo " <th>总商品数</th>"
echo " <th>在架商品数</th>"
echo " <th>下架商品数</th>"
echo " <th>增量总数</th>"
echo " <th>在架的增量数</th>"
echo " </tr>"
YESTERDAY=`date +%Y-%m-%d -d'-1 days'`
cat ../data/static.$YESTERDAY | while read line
do
echo $line | awk '
{ print "\t\t\t<tr>\n\t\t\t\t<td>'$TIME'</td>"
for (i=1; i<=6; i++) { print "\t\t\t\t<td>"$i"</td>"
}
print "\t\t\t</tr>"
}'
done
echo " </table>"
echo " </body>"
echo " </html>"
awk
1、传递外部shell参数(-v)、同时处理多文件（getline） 示例：
#!/bin/bash if [ $# -ne ]
then
echo "Usage: $0 local_log bc_log"
exit
fi awk -v bc_log=$ 'BEGIN{
SRC=;
TAR=;
/*print "bc_log:",bc_log;*/
}{
SRC=$; close(bc_log);
while(getline TAR < bc_log )
{
if(TAR>SRC)
{
print SRC;
break;
}
if(TAR==SRC)
{
break;
}
}
}
字典过滤
#! /bin/gawk -f
BEGIN{
total=;
while( getline< "/home/hadoop/zhenjing/solr/2.data" >)
{
dict[$] = ;
total++;
}
#print total;
}
{
if( $ in dict ){
next
}
print $
}
END{
}
分析日志，计算平均时间
#! /bin/gawk -f
START {
sum_time["item.get"] = ;
sum_count["item.get"] = ;
}
{
tmp = substr($, index($, "method"));
split(tmp, arr, "\"");
method = arr[];
if(method == "") {
next;
}
tmp = substr($, index($, "time:"));
split(tmp, arr, ":");
time = arr[];
# print method, time;
if(time > ){
sum_time[method] = sum_time[method] + time;
sum_count[method] = sum_count[method] + ;
}
}
END {
for(method in sum_time){
if(sum_count[method] > ){
print method " = " sum_time[method] / sum_count[method];
}
}
}
awk统计tomcat android机型：
#! /bin/gawk -f
# tail /tmp/temai.txt | grep Android | cut -d" " -f - | cut -d')' -f1
BEGIN{
FS = "; ";
}
{
if( $ in dict_os ){
dict_os[$] = dict_os[$] + ;
} else {
dict_os[$] = ;
}
if( $ in dict_jixing ){
dict_jixing[$] = dict_jixing[$] + ;
} else {
dict_jixing[$] = ;
}
}
END {
for(os in dict_os){
print dict_os[os] " " os;
}
for(jx in dict_jixing){
print dict_jixing[jx] " " jx;
awk平均分拆文件：
awk '{hash=int(NR % 10 ); print $0 >> "outfile_tmp_"hash }' weiboid.txt
时间相关
基于时间的循环：date命令
typeset readonly START_DATE=$(date --date="$1" +%Y%m%d);
for ((CUR_DATE=START_DATE; CUR_DATE<=END_DATE; CUR_DATE=$(date --date="${CUR_DATE} 1 days" +%Y%m%d) ))
Linux时钟分为系统时钟（System Clock）和硬件（Real Time Clock，简称RTC）时钟。系统时钟是指当前Linux Kernel中的时钟，而硬件时钟则是主板上由电池供电的时钟，这个硬件时钟可以在BIOS中进行设置。当Linux启动时，硬件时钟会去读取系统时钟的设置，然后系统时钟就会独立于硬件运作。Linux中的所有命令（包括函数）都是采用的系统时钟设置。在Linux中，用于时钟查看和设置的命令主要有date、hwclock和clock。其中，clock和hwclock用法相近。
date 010318242008.30(月日时分年.秒)
date 010318242008(月日时分年)
date -s 20080103
date -s 18:24
clock -w
hwclock --set --date="07/07/06 10:19" （月/日/年 时:分:秒）
硬件时钟与系统时钟同步：# hwclock --hctosys（hc代表硬件时间，sys代表系统时间）或者# clock --hctosys
系统时钟和硬件时钟同步：# hwclock --systohc或者# clock --systohc
xargs
echo "FILE.txt" | grep -v '^$' \
| xargs -L 1 -I {} find . -maxdepth 1 -type f -name "{}" \
| sort | xargs -L 1 basename | xargs md5sum
somecommand | xargs -s 50000 echo | xargs -I '{}' -s 100000 rm '{}'
LOCALIP=`/sbin/ifconfig eth1 | grep "inet addr" | cut -f 2 -d ":" | cut -f 1 -d " "`
LOCALIP="$(/sbin/ifconfig eth1 | sed -n '/inet addr/s/.*inet addr:\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*/\1/gp')"
LOCAL_IP="$(/sbin/ifconfig | sed -n 's/^.*inet addr\:\([[:digit:]\.]\{7,14\}\).*$/\1/gp' | head -1)";
array 相关
#! /bin/bash
colors=(red green blue)
element_count=${#colors[@]}
# Special syntax to extract number of elements in array.
# element_count=${#colors[*]} works also.
#
# The "@" variable allows word splitting within quotes
#+ (extracts variables separated by whitespace).
#
# This corresponds to the behavior of "$@" and "$*"
#+ in positional parameters.
# array loop:
# Way :
index=
while [ "$index" -lt "$element_count" ]
do # List all the elements in the array.
echo ${colors[$index]}
# ${colors[index]} also works because it's within ${ ... } brackets.
let "index = $index + 1"
# Or:
# ((index++))
done
# Way :
# Each array element listed on a separate line.
# If this is not desired, use echo -n "${colors[$index]} "
# Doing it with a "for" loop instead:
for i in "${colors[@]}"
do
echo "$i"
done
echo
echo "For test:"
# Usage: ArrayCheckExist "$value" "${arr[@]}"
function ArrayCheckExist
{
value=$
i=
for str in "$@"
do
((i++))
if [ $i -eq ]; then # first parameter is checked value
continue
fi
if [[ "$value" == "$str" ]]; then
#echo "$value" "$str"
return # exist
fi
done
return
}
dirs=() # empty array
index=
while read ip dir info tomcat port
do
if [[ $ip == "#" || -z $port ]]
then
continue
fi
# index=`echo -n $tomcat |md5sum|cut -d ' ' -f1` # get string md5sum
ArrayCheckExist "$tomcat" "${dirs[@]}"
if [ $? -eq ]; then
dirs[ ${#dirs[@]} ]=$tomcat
fi
done < ./di.conf
for i in "${dirs[@]}"
do
echo "$i"
done
echo "Array function test:"
ArrayCheckExist "${dirs[@]}"
return_val=$?
if [[ $return_val -gt ]]; then
echo "${dirs[0]} Exist."
fi
ArrayCheckExist "test" "${dirs[@]}"
if [ $? -gt ]; then
echo "test Exist."
fi
ArrayCheckExist "test"
if [ $? -gt ]; then
echo "test Exist."
fi
ArrayCheckExist "/usr/local/webserver/di/tomcat-di2" "${dirs[@]}"
if [ $? -gt ]; then
echo "Exist."
fi
统计脚本
solr：
cat shards | awk -F "QTime=" '{sum+=$2; cnt++; if($2> 500) slow++;} END{print cnt; print slow++; print sum/cnt; print slow/cnt}'
dc：
cat dc.log | grep -o "spending time:[0-9]\+" | awk -F: '{sum+= $2; cnt++; if($2 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
rts：
cat dc.log | grep -o "search rts used: [0-9]\+" | awk -F: '{sum+= $2; cnt++; if($2 > 20) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
searchserver：
cat searchserver.log.2013-07-08.log | grep -o "totalTime=[0-9]\+" | awk -F= '{sum+= $2; cnt++; if($2 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
appserver:
cat userTime-appserver-2013-07-25_* | awk '{sum+= $5; cnt++; if($5 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
solr相关
solr core数据对比
#! /bin/bash if [ $# -lt ]
then
echo "Usage $0 query_file solr1 solr2 [out_file]"
exit -
fi OUT_FILE="/tmp/query"
if [ $# -gt ]
then
OUT_FILE=$
fi while read query
do
query=`echo $query | sed 's/wt=javabin/wt=json/g'` # KEY: set wt=json for solr request.
wget "http://$2/solr/C2C_TAOBAO_0/select/?$query" -q -O ${OUT_FILE}_0
wget "http://$3/solr/C2C_TAOBAO_0/select/?$query" -q -O ${OUT_FILE}_1 # del QTime from solr "json" response
sed -i 's/"QTime":[0-9]\+,//g' ${OUT_FILE}_0
sed -i 's/"QTime":[0-9]\+,//g' ${OUT_FILE}_1 cmp ${OUT_FILE}_0 ${OUT_FILE}_1 > /dev/null
if [ "$?" != ]
then
echo "$query"
fi done < $
solr查询参数提取
#! /bin/gawk -f
/select params=\{/,/\}/ {
if( match($, /\{[^\}]+\}/, str)){
pos = length(str[]) -
print substr(str[], , pos)
}
}
获取淘宝真实链接：
#! /bin/bash if [ $# -lt ]
then
echo "Usage: $0 url"
exit
fi export LC_ALL=en_US.UTF-8
OUT=log wget $ -o $OUT -O tmp req_arr=`grep "Location:" $OUT | grep "s.click.taobao.com" | cut -d" " -f ` taobao_req=`echo $req_arr | cut -d" " -f `
refer_req=`echo $req_arr | cut -d" " -f ` ext_key=`echo $refer_req | awk -F"et%3D" '{print $2}' | sed -e 's/%25/%/g'` #echo "$taobao_req&ref=&et=$ext_key"
#echo "curl \"$taobao_req&ref=&et=$ext_key\" -I -e \"$refer_req\""
#curl -s "$taobao_req&ref=&et=$ext_key" -I -e "$refer_req" curl -s "$taobao_req&ref=&et=$ext_key" -I -e "$refer_req" | grep "^Location" | cut -d" " -f
日常数据备份脚本
#! /bin/bash export LC_ALL=en_US.UTF-
echo `date`
cd `dirname $`
pwd echo `date +%s`
# back taday data
DATE=`date +%Y-%m-%d` SQL_FILE="jira_$DATE.sql"
/opt/lampp/bin/mysqldump --default-character-set=utf8 -h 10.1.22.40 -u jira -pkoudai123 jira > $SQL_FILE r1=`ls -ld $SQL_FILE | awk '{print int($5/1024)}'` DATA_FILE="jira_home_$DATE.tar.gz"
tar czf $DATA_FILE /data/server/jira-home r2=`ls -ld $DATA_FILE | awk '{print int($5/1024)}'` echo $r1
echo $r2 echo `date +%s` if [[ $r1 -gt && $r2 -gt ]]
then
# del old data
DELDAY=`date --date='-15 day' +%Y-%m-%d` echo "del data : $DELDAY" rm *$DELDAY* fi echo "`date +%s` FINISH !"
操作文件
#! /bin/bash if [ $# -lt ]
then
echo "Usage $0 in_file out_file num"
exit -
fi INPUT=$
OUTPUT=$ num=
if [ $# -gt ]
then
num=$
fi for i in `seq $num`
do
echo "$i" awk -F ^ -v prefix=$i '{ if(NF > 10) print prefix$0}' ${INPUT} >> ${OUTPUT}
done
shell脚本常规技巧的更多相关文章
- shell脚本常用技巧
shell脚本常用技巧 1.获取随机字符串或数字 ~]#echo $RANDOM | md5sum | cut -c 1-6 ~]#openssl rand -base64 4 | cut -c 1- ...
- shell脚本使用技巧3--调试
1.使用-x,开启shell脚本的跟踪调试功能 ex:bash -x script.sh or sh -x script.sh 2.使用set -x 和 set +x对脚本进行部分调试(输入中间的内容 ...
- shell脚本调试技巧
shell脚本调试之工具——bashdb http://www.cnblogs.com/itcomputer/p/5011845.html
- SHELL 脚本小技巧
脚本很简单,直接上功能介绍及脚本,可以做模板使用: 记录日志,记录脚本开始执行时间.结束时间 usage 函数,脚本需接参数执行,避免误执行,告诉用户,这个脚本的使用方法 加锁,创建锁文件,脚本不允许 ...
- shell脚本小技巧
输入参数错误时,退格会出现^H,这个时候只要在脚本顶部加一条语句:stty erase ^h就可以了 #!/bin/sh stty erase ^h
- shell脚本使用技巧5--字符分隔
#!/bin/bash #filename:ifs.sh data="name,sex,rollon,location" oldIFS=$IFS IFS=, for item in ...
- shell脚本使用技巧4--读取字符,重复执行
ls | cat -n > out.txt 给输出的信息加行号并导出到out.txt 利用shell生成一个独立的进程 pwd; (cd /bin; ls); pwd; 开启一个子shell,不 ...
- shell脚本使用技巧2
0--stdin标准输入 1--stdout标准输出 2--stderr标准错误 重定向 echo "this is a good idea " > temp.txt tem ...
- Shell脚本小技巧收集
1.使用python快速搭建一个web服务器 访问端口8000 python -m SimpleHTTPServer 2.获取文件大小 stat -c %s $file stat --printf=' ...
随机推荐
- 解决PKIX(PKIX path building failed) 问题 unable to find valid certification path to requested target
最近在写java的一个服务,需要给远程服务器发送post请求,认证方式为Basic Authentication,在请求过程中出现了 PKIX path building failed: sun.se ...
- MongoDB基本管理命令
MongoDB是一个NoSQL数据库系统:一个数据库可以包含多个集合(Collection),每个集合对应于关系数据库中的表:而每个集合中 可以存储一组由列标识的记录,列是可以自由定义的,非常灵活,由 ...
- SpringMVC 结合HttpClient调用第三方接口实现
使用HttpClient 依赖jar包 1:commons-httpclient-3.0.jar 2:commons-logging-1.1.1.jar 3:commons-codec-1.6.jar ...
- AngularJS的小知识点
小知识点:$scope和$rootScope (1)每次使用ngController指令,都会调用控制器的创建函数,创建出一个控制器对象. (2)每次创建一个控制器对象,AngularJS都会创建一个 ...
- 初识C语言
C语言是开发iOS软件的基础. 一.C语言简介 1. 简史 1) C语言于1972年发明,首次使用是用于重写UINX操作系统(UNIX以前主要是用汇编语言写的,它奠定了操作 ...
- sqlserver2012 表分区
无论是新建数据库,还是现有的问题,都可以执行表分区的操作. 1.在数据库中点鼠标右键点击属性,在选择页,选中文件栏,在数据库文件列表中,可以看到现有的数据库文件逻辑名称.文件类型.初始大小.保存位置等 ...
- andorid frameanimation
Android中的逐帧动画 先来说说什么是逐帧动画,逐帧动画是一种常见的动画形式(Frame By Frame),其原理是在“连续的关键帧”中分解动画动作,也就是在时间轴的每帧上逐帧绘制不同的内容,使 ...
- Merge在Sqlserver使用例子说明
---文章 MatchInt的方式 Create table SourceTable([ID] int,[Desc] varchar(50)); Create table TargetTable([I ...
- [机器学习] 在茫茫人海中发现相似的你：实现局部敏感哈希(LSH)并应用于文档检索
简介 局部敏感哈希(Locality Sensitive Hasing)是一种近邻搜索模型,由斯坦福大学的Mose Charikar提出.我们用一种随机投影(Random Projection)的方式 ...
- 日常工作中的点滴:C# 根据字节长度截包含中文的字符串
方法中利用正则表达式判断某个字符是否是中文 public string SubStringB(string text,int length){ int target=0; int b=0; for(i ...