shell脚本常规技巧
邮件相关
发送邮件：
#!/usr/bin/python import sys;
import smtplib;
from email.MIMEText import MIMEText mail_host = sys.argv[]
mail_user = sys.argv[]
mail_pass = sys.argv[]
mail_from = sys.argv[]
mail_to = sys.argv[]
mail_to_list = mail_to.split(';') mail_subject = sys.argv[]
mail_content = sys.argv[] msg = MIMEText(mail_content, _subtype = 'html', _charset = 'utf-8')
msg['From'] = mail_from
msg['Subject'] = mail_subject
msg['To'] = mail_to
msg['MIME-Version'] = "1.0"
msg['Content-Type'] = "text/html" try:
client = smtplib.SMTP()
client.connect(mail_host)
client.login(mail_user, mail_pass)
client.sendmail(mail_from, mail_to_list, msg.as_string())
except:
print "something error..."
用法：python ./send_mail.py $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
带附件版本：
#!/usr/bin/python import os
import sys;
import smtplib;
from email.MIMEText import MIMEText
from email.MIMEMultipart import MIMEMultipart mail_host = sys.argv[1]
mail_user = sys.argv[2]
mail_pass = sys.argv[3]
mail_from = sys.argv[4]
mail_to = sys.argv[5]
mail_to_list = mail_to.split(';') mail_subject = sys.argv[6]
mail_content = sys.argv[7]
mail_attach = '' if len(sys.argv) > 8:
mail_attach = sys.argv[8]
#print mail_attach msg = MIMEMultipart()
#
body = MIMEText(mail_content, _subtype = 'html', _charset = 'utf-8')
msg.attach(body)
#
if os.path.isfile(mail_attach):
att = MIMEText(open(mail_attach,'rb').read(),'base64','gb2312')
att["Content-Type"] = 'application/octet-stream'
att["Content-Disposition"] = 'attachment;filename="' + os.path.basename(mail_attach) +'"'
msg.attach(att) msg['From'] = mail_from
msg['Subject'] = mail_subject
msg['To'] = mail_to
msg['MIME-Version'] = "1.0"
msg['Content-Type'] = "text/html" try:
client = smtplib.SMTP()
client.connect(mail_host)
client.login(mail_user, mail_pass)
client.sendmail(mail_from, mail_to_list, msg.as_string())
except:
print "something error..."
发送脚本：
#! /bin/bash export LC_ALL="en_US.UTF-8" if [ $# -lt ]
then
echo "Usage: $0 process_name"
exit -
fi cd $(dirname $)
LOCALIP=`/sbin/ifconfig eth0 | grep "inet addr" | cut -f -d ":" | cut -f -d " "`
HOSTNAME=`hostname`
gday=`date +"%x %X"` MAIL_HOST="smtp.exmail.qq.com"
MAIL_USER="op@test.com"
MAIL_PASS="optest"
MAIL_FROM="op@test.com"
MAIL_TO="chenzhj@test.com"
MAIL_SUBJECT="$HOSTNAME[$LOCALIP] $1 $gday" MAIL_CONTENT="" echo "$MAIL_SUBJECT" #echo "python ./send_mail.py" $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
python ./send_mail.py $MAIL_HOST $MAIL_USER $MAIL_PASS $MAIL_FROM $MAIL_TO "$MAIL_SUBJECT" "$MAIL_CONTENT"
生成表格邮件：
#!/bin/sh TIME=`date +%Y-%m-%d -d "1 days ago"` echo "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">"
echo "<html xmlns=\"http://www.w3.org/1999/xhtml\">"
echo "<head>"
echo "<meta http-equiv=\"Content-Type\" content=\"text/html;charset=gb2312\">"
#echo "<meta http-equiv=Content-Type content=\"text/html; charset=utf-8\">"
echo "</head>"
echo " <body>"
echo " <h2>今日半价mysql数据统计</h2>"
echo ' <table border="1">'
echo " <tr>"
echo " <th>DateTime</th>"
echo " <th>统计站点</th>"
echo " <th>总商品数</th>"
echo " <th>在架商品数</th>"
echo " <th>下架商品数</th>"
echo " <th>增量总数</th>"
echo " <th>在架的增量数</th>"
echo " </tr>"
YESTERDAY=`date +%Y-%m-%d -d'-1 days'`
cat ../data/static.$YESTERDAY | while read line
do
echo $line | awk '
{ print "\t\t\t<tr>\n\t\t\t\t<td>'$TIME'</td>"
for (i=1; i<=6; i++) { print "\t\t\t\t<td>"$i"</td>"
}
print "\t\t\t</tr>"
}'
done
echo " </table>"
echo " </body>"
echo " </html>"
awk
1、传递外部shell参数(-v)、同时处理多文件（getline） 示例：
#!/bin/bash if [ $# -ne ]
then
echo "Usage: $0 local_log bc_log"
exit
fi awk -v bc_log=$ 'BEGIN{
SRC=;
TAR=;
/*print "bc_log:",bc_log;*/
}{
SRC=$; close(bc_log);
while(getline TAR < bc_log )
{
if(TAR>SRC)
{
print SRC;
break;
}
if(TAR==SRC)
{
break;
}
}
}
字典过滤
#! /bin/gawk -f
BEGIN{
total=;
while( getline< "/home/hadoop/zhenjing/solr/2.data" >)
{
dict[$] = ;
total++;
}
#print total;
}
{
if( $ in dict ){
next
}
print $
}
END{
}
分析日志，计算平均时间
#! /bin/gawk -f
START {
sum_time["item.get"] = ;
sum_count["item.get"] = ;
}
{
tmp = substr($, index($, "method"));
split(tmp, arr, "\"");
method = arr[];
if(method == "") {
next;
}
tmp = substr($, index($, "time:"));
split(tmp, arr, ":");
time = arr[];
# print method, time;
if(time > ){
sum_time[method] = sum_time[method] + time;
sum_count[method] = sum_count[method] + ;
}
}
END {
for(method in sum_time){
if(sum_count[method] > ){
print method " = " sum_time[method] / sum_count[method];
}
}
}
awk统计tomcat android机型：
#! /bin/gawk -f
# tail /tmp/temai.txt | grep Android | cut -d" " -f - | cut -d')' -f1
BEGIN{
FS = "; ";
}
{
if( $ in dict_os ){
dict_os[$] = dict_os[$] + ;
} else {
dict_os[$] = ;
}
if( $ in dict_jixing ){
dict_jixing[$] = dict_jixing[$] + ;
} else {
dict_jixing[$] = ;
}
}
END {
for(os in dict_os){
print dict_os[os] " " os;
}
for(jx in dict_jixing){
print dict_jixing[jx] " " jx;
awk平均分拆文件：
awk '{hash=int(NR % 10 ); print $0 >> "outfile_tmp_"hash }' weiboid.txt
时间相关
基于时间的循环：date命令
typeset readonly START_DATE=$(date --date="$1" +%Y%m%d);
for ((CUR_DATE=START_DATE; CUR_DATE<=END_DATE; CUR_DATE=$(date --date="${CUR_DATE} 1 days" +%Y%m%d) ))
Linux时钟分为系统时钟（System Clock）和硬件（Real Time Clock，简称RTC）时钟。系统时钟是指当前Linux Kernel中的时钟，而硬件时钟则是主板上由电池供电的时钟，这个硬件时钟可以在BIOS中进行设置。当Linux启动时，硬件时钟会去读取系统时钟的设置，然后系统时钟就会独立于硬件运作。Linux中的所有命令（包括函数）都是采用的系统时钟设置。在Linux中，用于时钟查看和设置的命令主要有date、hwclock和clock。其中，clock和hwclock用法相近。
date 010318242008.30(月日时分年.秒)
date 010318242008(月日时分年)
date -s 20080103
date -s 18:24
clock -w
hwclock --set --date="07/07/06 10:19" （月/日/年 时:分:秒）
硬件时钟与系统时钟同步：# hwclock --hctosys（hc代表硬件时间，sys代表系统时间）或者# clock --hctosys
系统时钟和硬件时钟同步：# hwclock --systohc或者# clock --systohc
xargs
echo "FILE.txt" | grep -v '^$' \
| xargs -L 1 -I {} find . -maxdepth 1 -type f -name "{}" \
| sort | xargs -L 1 basename | xargs md5sum
somecommand | xargs -s 50000 echo | xargs -I '{}' -s 100000 rm '{}'
LOCALIP=`/sbin/ifconfig eth1 | grep "inet addr" | cut -f 2 -d ":" | cut -f 1 -d " "`
LOCALIP="$(/sbin/ifconfig eth1 | sed -n '/inet addr/s/.*inet addr:\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*/\1/gp')"
LOCAL_IP="$(/sbin/ifconfig | sed -n 's/^.*inet addr\:\([[:digit:]\.]\{7,14\}\).*$/\1/gp' | head -1)";
array 相关
#! /bin/bash
colors=(red green blue)
element_count=${#colors[@]}
# Special syntax to extract number of elements in array.
# element_count=${#colors[*]} works also.
#
# The "@" variable allows word splitting within quotes
#+ (extracts variables separated by whitespace).
#
# This corresponds to the behavior of "$@" and "$*"
#+ in positional parameters.
# array loop:
# Way :
index=
while [ "$index" -lt "$element_count" ]
do # List all the elements in the array.
echo ${colors[$index]}
# ${colors[index]} also works because it's within ${ ... } brackets.
let "index = $index + 1"
# Or:
# ((index++))
done
# Way :
# Each array element listed on a separate line.
# If this is not desired, use echo -n "${colors[$index]} "
# Doing it with a "for" loop instead:
for i in "${colors[@]}"
do
echo "$i"
done
echo
echo "For test:"
# Usage: ArrayCheckExist "$value" "${arr[@]}"
function ArrayCheckExist
{
value=$
i=
for str in "$@"
do
((i++))
if [ $i -eq ]; then # first parameter is checked value
continue
fi
if [[ "$value" == "$str" ]]; then
#echo "$value" "$str"
return # exist
fi
done
return
}
dirs=() # empty array
index=
while read ip dir info tomcat port
do
if [[ $ip == "#" || -z $port ]]
then
continue
fi
# index=`echo -n $tomcat |md5sum|cut -d ' ' -f1` # get string md5sum
ArrayCheckExist "$tomcat" "${dirs[@]}"
if [ $? -eq ]; then
dirs[ ${#dirs[@]} ]=$tomcat
fi
done < ./di.conf
for i in "${dirs[@]}"
do
echo "$i"
done
echo "Array function test:"
ArrayCheckExist "${dirs[@]}"
return_val=$?
if [[ $return_val -gt ]]; then
echo "${dirs[0]} Exist."
fi
ArrayCheckExist "test" "${dirs[@]}"
if [ $? -gt ]; then
echo "test Exist."
fi
ArrayCheckExist "test"
if [ $? -gt ]; then
echo "test Exist."
fi
ArrayCheckExist "/usr/local/webserver/di/tomcat-di2" "${dirs[@]}"
if [ $? -gt ]; then
echo "Exist."
fi
统计脚本
solr：
cat shards | awk -F "QTime=" '{sum+=$2; cnt++; if($2> 500) slow++;} END{print cnt; print slow++; print sum/cnt; print slow/cnt}'
dc：
cat dc.log | grep -o "spending time:[0-9]\+" | awk -F: '{sum+= $2; cnt++; if($2 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
rts：
cat dc.log | grep -o "search rts used: [0-9]\+" | awk -F: '{sum+= $2; cnt++; if($2 > 20) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
searchserver：
cat searchserver.log.2013-07-08.log | grep -o "totalTime=[0-9]\+" | awk -F= '{sum+= $2; cnt++; if($2 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
appserver:
cat userTime-appserver-2013-07-25_* | awk '{sum+= $5; cnt++; if($5 > 500) slow++ } END{print cnt; print slow; print sum/cnt; print slow/cnt}'
solr相关
solr core数据对比
#! /bin/bash if [ $# -lt ]
then
echo "Usage $0 query_file solr1 solr2 [out_file]"
exit -
fi OUT_FILE="/tmp/query"
if [ $# -gt ]
then
OUT_FILE=$
fi while read query
do
query=`echo $query | sed 's/wt=javabin/wt=json/g'` # KEY: set wt=json for solr request.
wget "http://$2/solr/C2C_TAOBAO_0/select/?$query" -q -O ${OUT_FILE}_0
wget "http://$3/solr/C2C_TAOBAO_0/select/?$query" -q -O ${OUT_FILE}_1 # del QTime from solr "json" response
sed -i 's/"QTime":[0-9]\+,//g' ${OUT_FILE}_0
sed -i 's/"QTime":[0-9]\+,//g' ${OUT_FILE}_1 cmp ${OUT_FILE}_0 ${OUT_FILE}_1 > /dev/null
if [ "$?" != ]
then
echo "$query"
fi done < $
solr查询参数提取
#! /bin/gawk -f
/select params=\{/,/\}/ {
if( match($, /\{[^\}]+\}/, str)){
pos = length(str[]) -
print substr(str[], , pos)
}
}
获取淘宝真实链接：
#! /bin/bash if [ $# -lt ]
then
echo "Usage: $0 url"
exit
fi export LC_ALL=en_US.UTF-8
OUT=log wget $ -o $OUT -O tmp req_arr=`grep "Location:" $OUT | grep "s.click.taobao.com" | cut -d" " -f ` taobao_req=`echo $req_arr | cut -d" " -f `
refer_req=`echo $req_arr | cut -d" " -f ` ext_key=`echo $refer_req | awk -F"et%3D" '{print $2}' | sed -e 's/%25/%/g'` #echo "$taobao_req&ref=&et=$ext_key"
#echo "curl \"$taobao_req&ref=&et=$ext_key\" -I -e \"$refer_req\""
#curl -s "$taobao_req&ref=&et=$ext_key" -I -e "$refer_req" curl -s "$taobao_req&ref=&et=$ext_key" -I -e "$refer_req" | grep "^Location" | cut -d" " -f
日常数据备份脚本
#! /bin/bash export LC_ALL=en_US.UTF-
echo `date`
cd `dirname $`
pwd echo `date +%s`
# back taday data
DATE=`date +%Y-%m-%d` SQL_FILE="jira_$DATE.sql"
/opt/lampp/bin/mysqldump --default-character-set=utf8 -h 10.1.22.40 -u jira -pkoudai123 jira > $SQL_FILE r1=`ls -ld $SQL_FILE | awk '{print int($5/1024)}'` DATA_FILE="jira_home_$DATE.tar.gz"
tar czf $DATA_FILE /data/server/jira-home r2=`ls -ld $DATA_FILE | awk '{print int($5/1024)}'` echo $r1
echo $r2 echo `date +%s` if [[ $r1 -gt && $r2 -gt ]]
then
# del old data
DELDAY=`date --date='-15 day' +%Y-%m-%d` echo "del data : $DELDAY" rm *$DELDAY* fi echo "`date +%s` FINISH !"
操作文件
#! /bin/bash if [ $# -lt ]
then
echo "Usage $0 in_file out_file num"
exit -
fi INPUT=$
OUTPUT=$ num=
if [ $# -gt ]
then
num=$
fi for i in `seq $num`
do
echo "$i" awk -F ^ -v prefix=$i '{ if(NF > 10) print prefix$0}' ${INPUT} >> ${OUTPUT}
done
shell脚本常规技巧的更多相关文章
- shell脚本常用技巧
shell脚本常用技巧 1.获取随机字符串或数字 ~]#echo $RANDOM | md5sum | cut -c 1-6 ~]#openssl rand -base64 4 | cut -c 1- ...
- shell脚本使用技巧3--调试
1.使用-x,开启shell脚本的跟踪调试功能 ex:bash -x script.sh or sh -x script.sh 2.使用set -x 和 set +x对脚本进行部分调试(输入中间的内容 ...
- shell脚本调试技巧
shell脚本调试之工具——bashdb http://www.cnblogs.com/itcomputer/p/5011845.html
- SHELL 脚本小技巧
脚本很简单,直接上功能介绍及脚本,可以做模板使用: 记录日志,记录脚本开始执行时间.结束时间 usage 函数,脚本需接参数执行,避免误执行,告诉用户,这个脚本的使用方法 加锁,创建锁文件,脚本不允许 ...
- shell脚本小技巧
输入参数错误时,退格会出现^H,这个时候只要在脚本顶部加一条语句:stty erase ^h就可以了 #!/bin/sh stty erase ^h
- shell脚本使用技巧5--字符分隔
#!/bin/bash #filename:ifs.sh data="name,sex,rollon,location" oldIFS=$IFS IFS=, for item in ...
- shell脚本使用技巧4--读取字符,重复执行
ls | cat -n > out.txt 给输出的信息加行号并导出到out.txt 利用shell生成一个独立的进程 pwd; (cd /bin; ls); pwd; 开启一个子shell,不 ...
- shell脚本使用技巧2
0--stdin标准输入 1--stdout标准输出 2--stderr标准错误 重定向 echo "this is a good idea " > temp.txt tem ...
- Shell脚本小技巧收集
1.使用python快速搭建一个web服务器 访问端口8000 python -m SimpleHTTPServer 2.获取文件大小 stat -c %s $file stat --printf=' ...
随机推荐
- git conifg
1. git config简介 我们知道config是配置的意思,那么git config命令就是对git进行一些配置.而配置一般都是写在配置文件里面,那么git的配置文件在哪里呢?互动一下,先问下大 ...
- SQL触发器实例讲解
SQL触发器实例1 定义: 何为触发器?在SQL Server里面也就是对某一个表的一定的操作,触发某种条件,从而执行的一段程序.触发器是一个特殊的存储过程. 常见的触发器有三种:分别应用于Inser ...
- 不同包中继承关系访问protected内部类问题
有两个包pack1和pack2,pack1中是父类,pack2中子类继承自pack1中的父类.这里主要探索一下子类访问父类中protected内部类的问题: 第一个类: package pack1; ...
- C# Susan边缘检测(Susan Edge Detection)
Susan边缘检测,方法简单,效率高,具体参照 The SUSAN Edge Detector in Detail, 修改dThreshold值,可以改变检测效果,用参照提供的重心法.力矩法可得到边缘 ...
- asp.net运行时错误：没有为扩展名".cshtml"注册的提供程序。
解决方法: 一. 在machine.config或web.config中的<compilation><buildProviders>节注册一个.请确保所注册的提供程序具有包含 ...
- ES6中的var let const应如何选择
javascript世界里面的每个人都在说有关ECMAScript 6 (ES6,也称作ES 2015)的话题,对象的巨大变化 ( 类 , super() , 等), 函数 (默认参数等), 以及模块 ...
- Python小练习四
# 使用给定的宽度打印格式化后的价格列表 width = (int)(input('Please enter width:')) price_width = 10 item_width = width ...
- Block循环引用问题研究
自从苹果在objc中添加Block功能支持以后已经过了很久.目前网上对于Block的使用有很多介绍.不过对于Block的内存管理问题,则是众说纷纭.再加上objc开始使用ARC以后,对于Block的内 ...
- js 变量提升和函数提升原理
关于js的变量,开始的时候我们都会被告知,变量声明应该在引用该变量之前.关于为什么要这样做呢,开始的时候本着会用就行的目的,也没去深究.不过后来经常会发现一些让人很费解的..姑且称为现象吧.先看一段代 ...
- springboot使用之三：springboot使用logback日志
springboot 默认使用的日志就是logback,所以使用logback不需要添加日志相关依赖了,执行 添加logback.xml配置文件,springboot这个聪明的框架便能识处理你的配置. ...