python BitTornado P2P分发大文件

P2P分发大文件思路

1、将软件包生成种子文件

2、通过saltstack将种子文件分发至每台服务器

3、每台服务器进行种子下载

推荐使用Twitter开源的murder。Twitter用它来分发大文件完成代码更新。

下载地址: https://github.com/lg/murder

本文介绍murder的dist，它是基于python开发的。

murder dist

• murder_tracker.py  可以理解它为调度中心
• murder_make_torrent.py  生成种子，并且注册在调度中心
• murder_client.py  播种和获取文件包（上传、下载）

举例部署结构：

192.168.1.11  调度中心

192.168.1.12  种子生成、第一个播种文件服务器

192.168.1.100-200  下载文件

1、启动调度中心

python murder_tracker.py --port 8998 --dfile data --logfile urder_tracker.log

• --port 监听的端口,默认是8998
• --dfile 存储近期下载信息的文件
• --logfile 日志文件，默认是标准输出

shell启动脚本

 #!/bin/bash
 #
 # Start/Stop murder-tracker
 #
 # description: murder-tracker
 # processname: murder-tracker
 
 if [ -f /etc/rc.d/init.d/functions ]; then
     . /etc/rc.d/init.d/functions
 fi
 
 name="murder-tracker"
 murder_tracker_bin="/usr/local/dist/murder_tracker.py"
 murder_tracker_log="/usr/local/dist/logs/murder_tracker.log"
 murder_tracker_data="/usr/local/dist/var/tracker_data"
 murder_user=murder
 
 find_tracker_process () {
     PID=`ps -ef | grep murder_tracker | grep python |grep -v $|grep -v grep|grep -v sh|grep -v root| awk '{ print $2 }'`
     echo $PID
 }
 
 start () {
     getent passwd $murder_user  >/dev/null || useradd -r  -s /sbin/nologin $murder_user
     LOG_DIR=`dirname ${murder_tracker_log}`
     DATA_DIR=`dirname ${murder_tracker_data}`
     if [ ! -d $LOG_DIR ]; then
       echo -e  "\e[35mLog dir ${LOG_DIR} doesn't exist. Creating\e[0m"
       mkdir -p $LOG_DIR
     fi
     if [ ! -d $DATA_DIR ]; then
       echo -e  "\e[35mLog dir ${DATA_DIR} doesn't exist. Creating\e[0m"
       mkdir -p $DATA_DIR
     fi
     chown -R $murder_user:$murder_user $DATA_DIR  $LOG_DIR
 
     find_tracker_process
     if [ "$PID" != "" ]; then
        echo -e  "\e[35m$name is already running!\e[0m"
     else
        daemon --user $murder_user  nohup  python $murder_tracker_bin --dfile $murder_tracker_data --logfile $murder_tracker_log > /dev/null >& &
        echo -e "\e[35mStarting $name Done\e[0m"
     fi
 }
 
 stop () {
     find_tracker_process
     if [ "$PID" != "" ]; then
         echo -e "\e[35mStopping $name\e[0m"
         kill $PID
     else
         echo -e "\e[35m$name is not running yet\e[0m"
     fi
 }
 
 case $ in
 start)
         start
         ;;
 stop)
         stop
         exit
         ;;
 restart)
         stop
         sleep
         start
         ;;
 status)
         find_tracker_process
         if [ "$PID" != "" ]; then
           echo -e "\e[35m$name is running: $PID\e[0m"
           exit
         else
           echo -e "\e[35m$name is not running\e[0m"
           exit
         fi
         ;;
 *)
         echo -e "\e[35mUsage: $0 {start|stop|restart|status}\e[0m"
         RETVAL=
 esac
 exit 

2、生成种子文件

python murder_make_torrent.py deploy.tar.gz 192.168.1.11:8998 deploy.torrent

3、创建第一个播种文件服务节点

python murder_client.py seed  deploy.torrent deploy.tar.gz 192.168.1.12

• 最后一个参数是本机的IP地址

4、通过saltstack将种子文件deploy.torrent发送到每台服务器上

5、所有节点192.168.1.100-200获取文件包，这些服务器下载文件，同时也承担播种节点

python murder_client.py peer deploy.torrent deploy.tar.gz 192.168.1.xxx

• 最后一个参数是本机的IP地址

备注：这些主机分发时防火墙需要放开，程序中默认使用10000-60000之间端口，BitTornado/download_bt1.py中定义

('minport', 10000, 'minimum port to listen on, counts up if unavailable'),
('maxport', 60000, 'maximum port to listen on'),
('random_port', 1, 'whether to choose randomly inside the port range ' +
        'instead of counting up linearly'),

参考博文：http://john88wang.blog.51cto.com/2165294/1793080