[转帖]goofys
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
Overview
Goofys allows you to mount an S3 bucket as a filey system.
It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have an on disk data cache (checkout catfs), and consistency model is close-to-open.
Installation
On Linux, install via pre-built binaries. You may also need to install fuse too if you want to mount it on startup.
On macOS, install via Homebrew:
$ brew cask install osxfuse
$ brew install goofys
- Or build from source with Go 1.10 or later:
$ export GOPATH=$HOME/work
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys
Usage
$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys <bucket> <mountpoint>
$ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix
Users can also configure credentials via the AWS CLI or the AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables.
To mount an S3 bucket on startup, make sure the credential is configured for root
, and can add this to /etc/fstab
:
goofys#bucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666,--dir-mode=0777 0 0
See also: Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2.
Got more questions? Check out questions other people asked
Benchmark
Using --stat-cache-ttl 1s --type-cache-ttl 1s
for goofys -ostat_cache_expire=1
for s3fs to simulate cold runs. Detail for the benchmark can be found in bench.sh. Raw data is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a connected to a bucket in us-west-2. Units are seconds.
To run the benchmark, configure EC2's instance role to be able to write to $TESTBUCKET
, and then do:
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench
# result will be written to $TESTBUCKET
See also: cached benchmark result and result on Azure.
License
Copyright (C) 2015 - 2019 Ka-Hing Cheung
Licensed under the Apache License, Version 2.0
Current Status
goofys has been tested under Linux and macOS.
List of non-POSIX behaviors/limitations:
- only sequential writes supported
- does not store file mode/owner/group
- use
--(dir|file)-mode
or--(uid|gid)
options
- use
- does not support symlink or hardlink
ctime
,atime
is always the same asmtime
- cannot
rename
directories with more than 1000 children unlink
returns success even if file is not presentfsync
is ignored, files are only flushed onclose
Compatibility with non-AWS S3
goofys has been tested with the following non-AWS S3 providers:
- Amplidata / WD ActiveScale
- Ceph (ex: Digital Ocean Spaces, DreamObjects, gridscale)
- EdgeFS
- EMC Atmos
- Google Cloud Storage
- Minio (limited)
- OpenStack Swift
- S3Proxy
- Scaleway
- Wasabi
Additionally, goofys also works with the following non-S3 object stores:
- Azure Blob Storage
- Azure Data Lake Gen1
- Azure Data Lake Gen2
References
- Data is stored on Amazon S3
- Amazon SDK for Go
- Other related fuse filesystems
- catfs: caching layer that can be used with goofys
- s3fs: another popular filesystem for S3
- gcsfuse: filesystem for Google Cloud Storage. Goofys borrowed some skeleton code from this project.
- S3Proxy is used for
go test
- fuse binding, also used by
gcsfuse
[转帖]goofys的更多相关文章
- nginx负载均衡基于ip_hash的session粘帖
nginx负载均衡基于ip_hash的session粘帖 nginx可以根据客户端IP进行负载均衡,在upstream里设置ip_hash,就可以针对同一个C类地址段中的客户端选择同一个后端服务器,除 ...
- [转帖]网络协议封封封之Panabit配置文档
原帖地址:http://myhat.blog.51cto.com/391263/322378
- [转帖]零投入用panabit享受万元流控设备——搭建篇
原帖地址:http://net.it168.com/a2009/0505/274/000000274918.shtml 你想合理高效的管理内网流量吗?你想针对各个非法网络应用与服务进行合理限制吗?你是 ...
- 3d数学总结帖
3d数学总结帖,以下是对3d学习过程中数学知识的简单总结 角度值和弧度制的互转 Deg2Rad 角度A1转弧度A2 => A2=A1*PI/180 Rad2Deg 弧度A2转换角度A1 => ...
- [转帖]The Lambda Calculus for Absolute Dummies (like myself)
Monday, May 7, 2012 The Lambda Calculus for Absolute Dummies (like myself) If there is one highly ...
- [转帖]FPGA开发工具汇总
原帖:http://blog.chinaaet.com/yocan/p/5100017074 ----------------------------------------------------- ...
- [Android分享] 【转帖】Android ListView的A-Z字母排序和过滤搜索功能
感谢eoe社区的分享 最近看关于Android实现ListView的功能问题,一直都是小伙伴们关心探讨的Android开发问题之一,今天看到有关ListView实现A-Z字母排序和过滤搜索功能 ...
- AxureRP7.0各类交互效果汇总帖(转)
了便于大家参考,我把这段时间发布分享的所有关于AxureRP7.0的原型做了整理. 以下资源均有对应的RP源文件可以下载. 当然 ,其中有部分是需要通过完成解密游戏[攻略]才能得到下载地址或者下载密码 ...
- 未能加载文件或程序集“Newtonsoft.Json, Version=4.0.0.0, Culture=neutral, PublicKeyToken=30a [问题点数:40分,结帖人u010259408]
未能加载文件或程序集“Newtonsoft.Json, Version=4.0.0.0, Culture=neutral, PublicKeyToken=30a [问题点数:40分,结帖人u01025 ...
- 转帖-[教程] Win7精简教程(简易中度)2016年8月-0day
[教程] Win7精简教程(简易中度)2016年8月 0day 发表于 2016-8-19 16:08:41 https://www.itsk.com/thread-370260-1-1.html ...
随机推荐
- 浅谈树形DP
树形DP是动态规划中最难也最常考的内容.具有DP和图论相结合的特点. 但从本质上来说,树形DP只不过是一种线性DP,只是将它与搜索结合起来了而已. 树形DP的基本步骤 读图 树形DP的题目中,通常会给 ...
- spring-cloud-alibaba项目打包
在父依赖中加入 <build> <plugins> <plugin> <groupId>org.springframework.boot</gro ...
- 华为云推出全自研数据库,GaussDB(openGauss)能否撑起一片天?
摘要:GaussDB(openGauss) 基于华为云底座,能够快速全球化部署,同时支持用户的本地化部署诉求,跟云上生态工具紧密结合让用户在迁移.开发.运维上省时省心. GaussDB(openGau ...
- 一文总结GaussDB通信原理知识
摘要:从发展历程到通信模型设计,到你了解一下GaussDB通信原理知识. MPPDB通信库发展历程 Postgres-XC 方法:采用libpq通信库实现CN和DN之间的连接,CN负责计算,DN仅进行 ...
- Python网络编程:ZeroMQ
大家好,我是老胡.最近在和小伙伴们一起搞事情,我是学统计出身,编程能力其实很差,有点拖后腿了.所以需要恶补基础,这个系列会更新几篇,感兴趣的同学可以一起学习交流. ZeroMQ概述 ZeroMQ(又名 ...
- ABAP步循环
一.在界面中循环输出行数据,屏幕直接画出行数,需要计算翻页,一旦界面行数变动,则需要更改代码,所以引入步循环 二.步循环 首先在界面上画出要展示的内容 注意,在步循环中,文本的名称和输入框的名称不能相 ...
- 在 HTML 页面中使用 React
该方案使用场景:在html页面中使用react,主js文件index.js和其它非react功能使用js模块化的方式开发,适合轻量级中小型应用 index.html代码: 引入react.react- ...
- 聊聊与前端工程师天然互补的 Serverless
作者:灵轮_(阿里云前端技术专家)_ 作为前端工程师,我们的使命是为用户提供良好的前端用户体验.随着云原生时代的到来,显而易见的,我们能做的更多了.Serverless 产品的特点是免运维.按量付费和 ...
- 4、SpringBoot连接数据库引入druid
系列导航 springBoot项目打jar包 1.springboot工程新建(单模块) 2.springboot创建多模块工程 3.springboot连接数据库 4.SpringBoot连接数据库 ...
- echart问题集合
legend与图标间隔 echarts自定义tooltip提示框内容 https://blog.csdn.net/dreamsup/article/details/56667330 echarts中自 ...