CentOS上安装FastDFS分布式文件系统

鱼大自己写的项目简介：http://bbs.chinaunix.net/thread-1920470-1-1.html

架构简介：http://www.programmer.com.cn/4380/

下载地址：http://sourceforge.net/projects/fastdfs/files/

下载后编译client

我们主要下载了一下安装包

nginx-1.6.0.tar.gz

fastdfs_client_java_v1.10.tar.gz

FastDFS_v5.01.tar.gz

fastdfs-nginx-module_v1.16.tar.gz

我们准备在30、33上装Tracker，31、32装Storage Grup1，34、35装Storage Group2

一、安装FastDFS

首先安装Tracker：

解压fastDFS，进入目录，make.sh，然后make.sh install

之后看见：

[root@bxcs_repository bin]# ll /usr/local/bin/
总用量
-rwxr-xr-x    root root   6月   : fdfs_appender_test
-rwxr-xr-x    root root   6月   : fdfs_appender_test1
-rwxr-xr-x    root root   6月   : fdfs_append_file
-rwxr-xr-x    root root   6月   : fdfs_crc32
-rwxr-xr-x    root root   6月   : fdfs_delete_file
-rwxr-xr-x    root root   6月   : fdfs_download_file
-rwxr-xr-x    root root   6月   : fdfs_file_info
-rwxr-xr-x    root root   6月   : fdfs_monitor
-rwxr-xr-x    root root     6月   : fdfs_restart.sh
-rwxr-xr-x    root root     6月   : fdfs_stop.sh
-rwxr-xr-x    root root  6月   : fdfs_storaged
-rwxr-xr-x    root root   6月   : fdfs_test
-rwxr-xr-x    root root   6月   : fdfs_test1
-rwxr-xr-x    root root   6月   : fdfs_trackerd
-rwxr-xr-x    root root   6月   : fdfs_upload_appender
-rwxr-xr-x    root root   6月   : fdfs_upload_file
-rwxr-xr-x  root root  6月   : git
-rwxr-xr-x    root root   6月   : git-cvsserver
-rwxr-xr-x    root root   6月   : gitk
-rwxr-xr-x  root root  6月   : git-receive-pack
-rwxr-xr-x    root root  6月   : git-shell
-rwxr-xr-x  root root  6月   : git-upload-archive
-rwxr-xr-x    root root  6月   : git-upload-pack

其中fdfs_restart.sh和fdfs_stop.sh以前叫restart.sh、stop.sh，为了避免混淆，重命名了。

mv /usr/local/bin/restart.sh /usr/local/bin/fdfs_restart.sh
mv /usr/local/bin/stop.sh /usr/local/bin/fdfs_stop.sh

配置文件：

[root@bxcs_repository FastDFS]# ll /etc/fdfs/
总用量
-rw-r--r--  root root   6月   : client.conf
-rw-r--r--  root root    6月   : http.conf
-rw-r--r--  root root  6月   : mime.types
-rw-r--r--  root root   6月   : storage.conf
-rw-r--r--  root root   6月   : tracker.conf

我们改一下tracker.conf

vim /etc/fdfs/tracker.conf

#一般只需改动以下几个参数即可：
disabled=false            #启用配置文件
port=                #设置tracker的端口号
base_path=/usr/local/fdfs/tracker   #设置tracker的数据文件和日志目录（需预先创建）
http.server_port=     #设置http端口号

上面的路径 /usr/local/fdfs/tracker需要提前创建好

运行tracker试试：

[root@bxcs_repository FastDFS]# fdfs_trackerd /etc/fdfs/tracker.conf restart
查看状态：[root@bxcs_repository FastDFS]# netstat -unltp | grep fdfs
tcp                              /fdfs_trackerd
[root@bxcs_repository FastDFS]# ps -ef |grep fdfs
root              : ?        :: fdfs_trackerd /etc/fdfs/tracker.conf restart

去tracker的日志看看有没有错：

[root@bxcs_repository FastDFS]# tail -f /usr/local/fdfs/tracker/logs/trackerd.log
[-- ::] INFO - FastDFS v5., base_path=/usr/local/fdfs/tracker, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=, bind_addr=, max_connections=, accept_threads=, work_threads=, store_lookup=, store_group=, store_server=, store_path=, reserved_storage_space=, allow_ip_count=-, sync_log_buff_interval=10s, check_active_interval=120s, thread_stack_size= KB, storage_ip_changed_auto_adjust=, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=, slot_min_size=, slot_max_size= MB, trunk_file_size= MB, trunk_create_file_advance=, trunk_create_file_time_base=:, trunk_create_file_interval=, trunk_create_file_space_threshold= GB, trunk_init_check_occupying=, trunk_init_reload_from_binlog=, trunk_compress_binlog_min_interval=, use_storage_id=, id_type_in_filename=ip, storage_id_count=, rotate_error_log=, error_log_rotate_time=:, rotate_error_log_size=, store_slave_file_use_link=, use_connection_pool=, g_connection_pool_max_idle_time=3600s
[-- ::] INFO - local_host_ip_count: ,  127.0.0.1  192.168.3.30

开机启动Tracker：

[root@bxcs_repository FastDFS]# vim /etc/rc.local
/usr/local/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart

OK,我们给另外一台机器192.168.3.33也安装上Tracker。

然后安装Storage：

同样，解压，然后./make.sh   ./make.sh install在/usr/local/bin和 /etc/fdfs/下有一堆命令和配置文件。

重命名一下shell脚本，不然容易混淆：

mv /usr/local/bin/restart.sh /usr/local/bin/fdfs_restart.sh
mv /usr/local/bin/stop.sh /usr/local/bin/fdfs_stop.sh

配置Storage节点：

多个tracker可以配置多次tracker_server，这样可以解决tracker_server的单点问题：

vim /etc/fdfs/storage.conf
#一般只需改动以下几个参数即可：
disabled=false                    #启用配置文件
group_name=group1                 #组名，根据实际情况修改
port=                        #设置storage的端口号
base_path=/fdfs/storage           #设置storage的日志目录（需预先创建）
store_path_count=                #存储路径个数，需要和store_path个数匹配
base_path=/usr/local/fdfs/storagestore_path0=/usr/local/fdfs/storage #存储路径,store_path#, based 0, if store_path0 not exists, it's value is base_pathtracker_server=: #tracker服务器的IP地址和端口号

tracker_server=192.168.3.33:22122 #tracker服务器的IP地址和端口号

http.server_port= #设置http端口号

必须先创建/usr/local/fdfs/storage目录，否则启动报错：

[-- ::] ERROR - , "/usr/local/fdfs/storage" can't be accessed, error info: No such file or directory

创建后启动：

/usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf restart

data path: /usr/local/fdfs/storage/data, mkdir sub dir...
 ...
 ...
 ...
 ...
 ...
 ...
 ...
 ...
 ...
 ...
mkdir data path: 0A ...
mkdir data path: 0B ...
....
....
....
mkdir data path: FA ...
mkdir data path: FB ...
mkdir data path: FC ...
mkdir data path: FD ...
mkdir data path: FE ...
mkdir data path: FF ...
data path: /usr/local/fdfs/storage/data, mkdir sub dir done.

然后查看网络：

[root@bxcsweb bin]# netstat -unltp | grep fdfs
tcp                              /fdfs_storaged

查看进程：

[root@bxcsweb bin]# ps -ef | grep fdfs
root              : ?        :: fdfs_storaged /etc/fdfs/storage.conf restart
root           : pts/    :: grep fdfs

查看启动日志：

tail -f /usr/local/fdfs/storage/logs/storaged.log

[-- ::] INFO - FastDFS v5., base_path=/usr/local/fdfs/storage, store_path_count=, subdir_count_per_path=, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=, bind_addr=, client_bind=, max_connections=, accept_threads=, work_threads=, disk_rw_separated=, disk_reader_threads=, disk_writer_threads=, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=:, sync_end_time=:, write_mark_file_freq=, allow_ip_count=-, file_distribute_path_mode=, file_distribute_rotate_count=, fsync_after_written_bytes=, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size= KB, upload_priority=, if_alias_prefix=, check_file_duplicate=, file_signature_method=hash, FDHT group count=, FDHT server count=, FDHT key_namespace=, FDHT keep_alive=, HTTP server port=, domain name=, use_access_log=, rotate_access_log=, access_log_rotate_time=:, rotate_error_log=, error_log_rotate_time=:, rotate_access_log_size=, rotate_error_log_size=, file_sync_skip_invalid_record=, use_connection_pool=, g_connection_pool_max_idle_time=3600s
[-- ::] INFO - , use_storage_id=, id_type_in_filename=ip, storage_ip_changed_auto_adjust=, store_path=, reserved_storage_space=, slot_min_size=, slot_max_size= MB, trunk_file_size= MB, trunk_create_file_advance=, trunk_create_file_time_base=:, trunk_create_file_interval=, trunk_create_file_space_threshold= GB, trunk_init_check_occupying=, trunk_init_reload_from_binlog=, trunk_compress_binlog_min_interval=, store_slave_file_use_link=
[-- ::] INFO - , tracker_client_ip:
[-- ::] ERROR - , server: , response status  !=
[-- ::] INFO - local_host_ip_count: ,  127.0.0.1  192.168.3.31
[-- ::] INFO - , successfully connect to tracker server , as a tracker client, my ip is 192.168.3.31
[-- ::] INFO - , successfully connect to tracker server , as a tracker client, my ip is 192.168.3.31
[-- ::] INFO - , tracker server , set tracker leader: 

看见成功启动了。

使用fdfs_monitor命令查看Storage是否已登记到Tracker：（注意标红那一行）

[root@bxcsweb bin]# fdfs_monitor /etc/fdfs/storage.conf
[-- ::] DEBUG - base_path=/usr/local/fdfs/storage, connect_timeout=, network_timeout=, tracker_server_count=, anti_steal_token=, anti_steal_secret_key length=, use_connection_pool=                                   , g_connection_pool_max_idle_time=3600s, use_storage_id=, storage server 

server_count=, server_index=

tracker server is 

group count: 

Group :
group name = group1
disk total space =  MB
disk  MB
trunk  MB
storage server count =
active server count =
storage server port =
storage HTTP port =
store path count =
subdir count per path =
current
current trunk 

        Storage :
                id = 192.168.3.31
                ip_addr = 192.168.3.31 (bxcsweb)  ACTIVE
                http domain =
                version = 5.01
                -- ::
                up -- ::
                total storage =  MB
                 MB
                upload priority =
                store_path_count =
                subdir_count_per_path =
                storage_port =
                storage_http_port =
                current_write_path =
                source storage id=
                if_trunk_server=
                total_upload_count =
                success_upload_count =
                total_append_count =
                success_append_count =
                total_modify_count =
                success_modify_count =
                total_truncate_count =
                success_truncate_count =
                total_set_meta_count =
                success_set_meta_count =
                total_delete_count =
                success_delete_count =
                total_download_count =
                success_download_count =
                total_get_meta_count =
                success_get_meta_count =
                total_create_link_count =
                success_create_link_count =
                total_delete_link_count =
                success_delete_link_count =
                total_upload_bytes =
                success_upload_bytes =
                total_append_bytes =
                success_append_bytes =
                total_modify_bytes =
                success_modify_bytes =
                stotal_download_bytes =
                success_download_bytes =
                total_sync_in_bytes =
                success_sync_in_bytes =
                total_sync_out_bytes =
                success_sync_out_bytes =
                total_file_open_count =
                success_file_open_count =
                total_file_read_count =
                success_file_read_count =
                total_file_write_count =
                success_file_write_count =
                last_heart_beat_time = -- ::
                last_source_update = -- ::
                last_sync_update = -- ::
                last_synced_timestamp = -- ::

如果一个group1内有多个storage机器，上面的监控应该显示

Group :
group name = group1
disk total space =  MB
disk  MB
trunk  MB
storage server count =
active server count =
storage server port =
storage HTTP port =
store path count =
subdir count per path =
current
current trunk 

        Storage :
                id = 192.168.3.31
                ip_addr = 192.168.3.31 (localhost)  ACTIVE
。。。。
。。。。

        Storage :
                id = 192.168.3.32
                ip_addr = 192.168.3.32 (bxcscache)  ACTIVE。。。。。。。。

设置开机启动：

[root@bxcsweb bin]# vim /etc/rc.local
#start FastDFS
/usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf restart

两台Tracker(30/33)、四台Storage【分为两个Group，分别命名为group1(31/32)、group2(34/35)】安装完之后的信息：

[root@bxcs_mongoSlave FastDFS]# fdfs_monitor /etc/fdfs/storage.conf
[-- ::] DEBUG - base_path=/usr/local/fdfs/storage, connect_timeout=, network_timeout=, tracker_server_count=, anti_steal_token=, anti_steal_secret_key length=, use_connection_pool=, g_connection_pool_max_idle_time=3600s, use_storage_id=, storage server 

server_count=, server_index=

tracker server is 

group count: 

Group :
group name = group1
disk total space =  MB
disk  MB
trunk  MB
storage server count =
active server count =
storage server port =
storage HTTP port =
store path count =
subdir count per path =
current
current trunk 

        Storage :
                id = 192.168.3.31
                ip_addr = 192.168.3.31 (localhost)  ACTIVE
                。。。
        Storage :
                id = 192.168.3.32
                ip_addr = 192.168.3.32 (localhost)  ACTIVE
                。。。

Group :
group name = group2
disk total space =  MB
disk  MB
trunk  MB
storage server count =
active server count =
storage server port =
storage HTTP port =
store path count =
subdir count per path =
current
current trunk 

        Storage :
                id = 192.168.3.34
                ip_addr = 192.168.3.34 (localhost)  ACTIVE
                ...
        Storage :
                id = 192.168.3.35
                ip_addr = 192.168.3.35 (bxcs_mongoSlave)  ACTIVE
                ...

storage节点安装nginx和fastdfs-nginx-module模块：

“在storage上安装的nginx主要为了提供http的访问服务，同时解决group中storage服务器的同步延迟问题。”

要求必须先安装zlib和pcre包。

yum -y install zlib zlib-devel openssl openssl--devel pcre pcre-devel

安装nginx可参见之前的博文http://www.cnblogs.com/radio/p/3782280.html，只是安装的时候注意config的时候添加一个模块：

./configure --prefix=/usr/local/nginx --add-module=/usr/local/fastdfs-nginx-module/src 

结果输出：

adding module in /usr/local/fastdfs-nginx-module/src/
 + ngx_http_fastdfs_module was configured
checking for PCRE library ... found
checking for PCRE JIT support ... not found
checking for md5 in system md library ... not found
checking for md5 in system md5 library ... not found
checking for md5 in system OpenSSL crypto library ... not found
checking for sha1 in system md library ... not found
checking for sha1 in system OpenSSL crypto library ... not found
checking for zlib library ... found
creating objs/Makefile

Configuration summary
  + using system PCRE library
  + OpenSSL library is not used
  + using builtin md5 code
  + sha1 library is not found
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

然后 make、make install就OK。

安装完之后，将插件的配置文件拷贝到FastDFS配置文件目录下：

[root@bxcs_mongoSlave conf]# cp /usr/local/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/

然后编辑这个配置文件：

[root@bxcs_mongoSlave conf]# vim /etc/fdfs/mod_fastdfs.conf

一般只需改动以下几个参数即可：
base_path=/usr/local/fdfs/storage #保存日志目录
tracker_server= #tracker服务器的IP地址以及端口号
tracker_server= #tracker服务器的IP地址以及端口号
storage_server_port=         #storage服务器的端口号
group_name=group2                 #当前服务器的group名
url_have_group_name = true        #文件url中是否有group名
store_path_count=                #存储路径个数，需要和store_path个数匹配
store_path0=/usr/local/fdfs/storage  #存储路径
http.need_find_content_type=true  #从文件扩展名查找文件类型（nginx时为true）没找到这个配置参数，可能新版移除了。
group_count =                    #设置组的个数

在末尾增加3个组的具体信息：
[group1]
group_name=group1
storage_server_port=
store_path_count=
store_path0=/usr/local/fdfs/storage

# group settings
# since v1.
# when support multi-group, uncomment following section as neccessary
[group2]
group_name=group2
storage_server_port=
store_path_count=
store_path0=/usr/local/fdfs/storage

建立软连接：（暂时不明白什么意思，不清楚为什么nginx和此处的M00有何对应，想必应该是Tracker会在URL后面加上M00）

[root@bxcs_mongoSlave sbin]# ln -s /usr/local/fdfs/storage/data/ /usr/local/fdfs/storage/data/M00

nginx配置：

    server {
        listen       ;
        server_name  localhost;

        location ~/group[-]/M00 {
            root /usr/local/fdfs/storage/data;
            ngx_fastdfs_module;
        }
    }

然后进入nginx的sbin，./nginx启动，看看错误日志：

[root@bxcs_mongoSlave sbin]# cat /usr/local/nginx/logs/error.log

ngx_http_fastdfs_process_init pid=
[-- ::] INFO - local_host_ip_count: ,  127.0.0.1  192.168.3.35
[-- ::] INFO - local_host_ip_count: ,  127.0.0.1  192.168.3.35
[-- ::] INFO - fastdfs apache / nginx module v1., response_mode=proxy, base_path=/tmp, url_have_group_name=, group_count=, connect_timeout=, network_timeout=, tracker_server_count=, if_alias_prefix=, local_host_ip_count=, anti_steal_token=, token_ttl=0s, anti_steal_secret_key length=, token_check_fail content_type=, token_check_fail buff length=, load_fdfs_parameters_from_tracker=, storage_sync_file_max_delay=86400s, use_storage_id=, storage server , flv_support=, flv_extension=flv
[-- ::] INFO - group . group_name=group1, storage_server_port=, path_count=, store_path0=/usr/local/fdfs/storage
[-- ::] INFO - group . group_name=group2, storage_server_port=, path_count=, store_path0=/usr/local/fdfs/storage
[-- ::] INFO - fastdfs apache / nginx module v1., response_mode=proxy, base_path=/tmp, url_have_group_name=, group_count=, connect_timeout=, network_timeout=, tracker_server_count=, if_alias_prefix=, local_host_ip_count=, anti_steal_token=, token_ttl=0s, anti_steal_secret_key length=, token_check_fail content_type=, token_check_fail buff length=, load_fdfs_parameters_from_tracker=, storage_sync_file_max_delay=86400s, use_storage_id=, storage server , flv_support=, flv_extension=flv
[-- ::] INFO - group . group_name=group1, storage_server_port=, path_count=, store_path0=/usr/local/fdfs/storage
[-- ::] INFO - group . group_name=group2, storage_server_port=, path_count=, store_path0=/usr/local/fdfs/storage

没有ERROR,然后看这台机器的9090端口可以打开nginx服务页面。

OK，接着在31、32、34的storage机器上也安装nginx。

Tracker机器安装Nginx

我们没有用缓存模块，直接常规安装就行，注意一下配置文件：

    #fastDFS group1
    upstream fastDFS_group1{
        #ip_hash
        server ;
        server ;
    }

    #fastDFS group2
    upstream fastDFS_group2{
        server ;
        server ;
    }

    server {
        #Current FastDFS Tracker Http Port at /etc/fdfs/tracker.conf
        listen       ;
        server_name  fastDFS_Tracker_Http;

        #proxy: Redirect Tracker Http Request to Storage Node nginx Server
        location /group1/M00 {
            proxy_pass http://fastDFS_group1;
            include proxy.conf;
        }

        location /group2/M00 {
            proxy_pass http://fastDFS_group2;
            include proxy.conf;
        }
    }

PS:如果要用缓存，需要给nginx安装缓存模块ngx_cache_purge，详见下方参考目录。

安装完之后启动起来。

测试Tracker的Http：

先编辑30Tracker的client.conf，然后我们上传一个图片。

vim /etc/fdfs/client.conf
修改以下参数：
base_path=/usr/local/fdfs/tracker                    #日志存放路径
tracker_server=          #tracker服务器IP地址和端口号
tracker_server=          #tracker服务器IP地址和端口号
http.tracker_server_port=              #tracker服务器的http端口号

编辑完之后上传两个图片：（使用了/usr/local/bin中的fdfs_upload_file命令）

[root@bxcs_repository sbin]# fdfs_upload_file /etc/fdfs/client.conf /usr/local/test.png
group2/M00///wKgDIlOrbiyAcv0PAAA3CQYsNEs507.png

[root@bxcs_repository sbin]# fdfs_upload_file /etc/fdfs/client.conf /usr/local/test.jpeg
group2/M00///wKgDI1Orb9aAeJnkAAEFKS6QV9I89.jpeg

FastDFS为我们返回了这个文件的路径：group2/M00/00/00/wKgDI1Orb9aAeJnkAAEFKS6QV9I89.jpeg

我们在浏览器中访问Tracker：http://192.168.3.30:9090/group2/M00/00/00/wKgDI1Orb9aAeJnkAAEFKS6QV9I89.jpeg

结果成功读取了这个文件！

停止和重启脚本：

停止
/usr/local/bin/fdfs_stop.sh /usr/local/bin/fdfs_trackerd /etc/fdfs/tracker.conf
/usr/local/bin/fdfs_stop.sh /usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf

重启:
/usr/local/bin/fdfs_restart.sh /usr/local/bin/fdfs_trackerd /etc/fdfs/tracker.conf
/usr/local/bin/fdfs_restart.sh /usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf

OK，至此FastDFS安装完毕。

接下来就是使用鱼大的javaDriver操作FastDFS了。

本文参考了：

http://bbs.chinaunix.net/thread-4133106-1-1.html （缓存配置，详见此文）

另外，可用资料：

常见问题：http://www.cnblogs.com/chinacloud/archive/2011/01/19/1938870.html

一个配置文件详解：http://blog.csdn.net/poechant/article/details/6996047