Supercharging your ETL with Airflow and Singer
Earlier this year we introduced Singer, an open source project that helps data teams build simple, composable ETL. Singer provides a standard way for anyone to pull data from and send data to any source.
For many companies, however, being able to push and pull data or move things from A to B is the only part of the problem. Data extraction is often part of a more complex workflow that involves scheduled tasks, complex dependencies, and the need for scalable, distributed architecture.
Enter Apache Airflow. Originally developed at Airbnb and now a part of the Apache Incubator, Airflow takes the simplicity of a cron scheduler and adds all the facets of a modern workflow tool: dependency graphs, detailed logging, automated notifications, scalable infrastructure, and a graphical user interface.
A dependency tree and history of task runs from Airflow’s UI
Imagine a company that relies on data from multiple data sources, including SaaS tools, databases, and flat files. Several times a day this company might want to ingest new data from these sources in parallel. The company might manipulate it in some way, then dump the output into a data warehouse.
Airflow and Singer can make all of that happen. With a few lines of code, you can use Airflow to easily schedule and run Singer tasks, which can then trigger the remainder of your workflow.
A real-world example
Let’s look at a real-world example developed by a member of the Singer community. In this scenario we’re going to be pulling in CSV files, but Singer can work with any data source.
Our user has a specific sequence of tasks they need to complete each morning:
- Download new compressed CSV files from an AWS S3 bucket
- Decompress those files
- Use a Singer CSV tap to push the data to a Singer Stitch target. In this example we’re dumping data into Amazon Redshift, but you could target Google BigQuery or Postgres, too.
- Delete the compressed and decompressed files
This entire workflow, including all scripts, logging, and the Airflow implementation itself, is accomplished in fewer than 160 lines of Python code in this repo. Let’s see how it’s done.
Firing up Airflow
First we get Airflow running as described on the project’s Quick Start page with four commands:
# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow
# install from pypi using pip
pip install airflow
# initialize the database
airflow initdb
# start the web server, default port is 8080
airflow webserver -p 8080
Upon running that last command, you should see some ASCII art, letting you know that the web server is online:
Now, point your browser to http://localhost:8080/ to see a screen that looks like this:
At this point, you’re ready to create your own Airflow DAG (Directed Acyclic Graph) to perform data workflow tasks. For our purposes, we can get a ready-made DAG by cloning the airflow-singer repo:
git clone git@github.com:robertjmoore/airflow-singer.git
Customizing the repo
For Airflow to find the DAG in this repo, you’ll need to tweak the dags_folder variable the ~/airflow/airflow.cfg file to point to the dags directory inside the repo:
You’ll also want to make a few tweaks to the singer.py file in the repo’s dags folder to reflect your contact info and the location of the repo on your local file system:
Restart the web server with the command airflow webserver -p 8080
, then refresh the Airflow UI in your browser. You should now see the DAG from our repo:
Clicking on it will show us the Graph View, which lays out the steps taken each morning when the DAG is run:
This dependency map is governed by a few lines of code inside the dags/singer.py file. Let’s unpack a little of what’s going on.
Exploring the DAG
This tiny file defines the whole graph. Each of these tasks is a step in the DAG, and the final four lines draw out the dependencies that exist between them.
You’ll notice that, in this file, each step is a BashOperator that calls a specific command-line task and waits for its successful completion. Airflow supports a number of other operators and allows you to build your own. This makes it easy for a DAG to include interactions with databases, email services, and chat tools like Slack.
Interacting with Singer
To get a better idea of how Singer is integrated, check out the individual files in the scripts/ directory. You'll find Python scripts that download data from an Amazon S3 bucket, extract that data, and delete the files on completion.
The most interesting step is the process of using Singer to extract the data from the CSV files and push it to a target – namely Stitch.
We should also note that the CSV tap requires a config file that tells it where to find the CSV files to push to Singer, so one step in our DAG is to generate that JSON config file and then point it to the files we just extracted. We do this by generating a few lines of JSON code. Note that we use the global Airflow variable execution_date across our various scripts to be sure we deposit and retrieve the files from the same path.
Once that config file has been generated, we call Singer to do all the work in a single command line:
tap-csv -c ~/config/csv-config.json | target-stitch -c ~/config/stitch_config.json
This doesn’t even require a special Python script — the entire instruction is laid out in a single line of the singer.py DAG file.
Conclusion
As you can see, incorporating Singer into your Airflow DAGs gives you a powerful way to move data automatically. Anyone can extract and load data with a one-line instruction, using a growing ecosystem of taps and targets.
Supercharging your ETL with Airflow and Singer的更多相关文章
- Airbnb架构要点分享——阅读心得
目前,Airbnb已经使用了大约5000个AWS EC2实例,其中大约1500个实例用于部署其应用程序中面向Web的部分,其余的3500个实例用于各种分析和机器学习算法.而且,随着Airbnb的发展, ...
- 《Airbnb架构要点分享》阅读笔记
Airbnb成立于2008年8月,总部位于加利福尼亚州旧金山市.Airbnb是一个值得信赖的社区型市场,在这里人们可以通过网站.手机或平板电脑发布.发掘和预订世界各地的独特房源,其业务已经覆盖190个 ...
- Singer 开源便捷的ETL 工具
singer 是一个强大,灵活的etl 工具,我们可以方便的提取web api,file,queue,基本上各种你可以想到的 数据源. singer 有一套自己的数据处理规范, taps, targe ...
- Singer 学习三 使用Singer进行mongodb 2 postgres 数据转换
Singer 可以方便的进行数据的etl 处理,我们可以处理的数据可以是api 接口,也可以是数据库数据,或者 是文件 备注: 测试使用docker-compose 运行&&提供数据库 ...
- Singer 学习二 使用Singer进行gitlab 2 postgres 数据转换
Singer 可以方便的进行数据的etl 处理,我们可以处理的数据可以是api 接口,也可以是数据库数据,或者 是文件 备注: 测试使用docker-compose 运行&&提供数据库 ...
- 3.Airflow使用
1. airflow简介2. 相关概念2.1 服务进程2.1.1. web server2.1.2. scheduler2.1.3. worker2.1.4. celery flower2.2 相关概 ...
- 4.airflow测试
1.测试sqoop任务1.1 测试全量抽取1.1.1.直接执行命令1.1.2.以shell文件方式执行sqoop或hive任务1.2 测试增量抽取2.测试hive任务3.总结 当前生产上的任务主要分为 ...
- 【airflow实战系列】 基于 python 的调度和监控工作流的平台
简介 airflow 是一个使用python语言编写的data pipeline调度和监控工作流的平台.Airflow被Airbnb内部用来创建.监控和调整数据管道.任何工作流都可以在这个使用Pyth ...
- 调度系统Airflow的第一个DAG
Airflow的第一个DAG 考虑了很久,要不要记录airflow相关的东西, 应该怎么记录. 官方文档已经有比较详细的介绍了,还有各种博客,我需要有一份自己的笔记吗? 答案就从本文开始了. 本文将从 ...
随机推荐
- .net core 2.0的认证和授权
在asp.net core中,微软提供了基于认证(Authentication)和授权(Authorization)的方式,来实现权限管理的,本篇博文,介绍基于固定角色的权限管理和自定义角色权限管理, ...
- [shell] while read line 与for循环的区别
[shell] while read line 与for循环的区别 while read line 与for循环的区别---转载整理 while read line 是一次性将文件信息读入并赋值给变量 ...
- JS获取当前时间戳及时间戳的转换
获取现在的Unix时间戳(Unix timestamp) Math.round(new Date().getTime()/1000) //getTime()返回数值的单位是毫秒 Unix时间戳(Un ...
- (原创)对比组态软件,使用C#开发的服务器和客户端软件的优势
在当前经济形势和市场环境下,中小企业面对萧条的消费市场,恶化的外部贸易环境,刚性支出高成本人工和生产要素,通货膨胀,隐性的腐化支出等各种因素的作用导致企业生存艰难,企业需要在各方面削减支出,拓展市场寻 ...
- JavaScript入门(一)
JavaScript入门篇—开篇 Document对象 1Document对象表示当前页面,HTML在浏览器中是以DOM形式表示为树形结构.Document是DOM树的根节点.(因此需要查找DOM树中 ...
- springboot整合docker部署
环境安装 首先,需要安装Docker(例如:docker for windows) 下载地址:https://download.docker.com/win/stable/Docker%20for%2 ...
- JavaWeb第二天--CSS
CSS CSS简述 CSS是什么?有什么作用? CSS(Cascading Style Sheets):层叠样式表. CSS通常称为CSS样式或层叠样式表.主要用于设置HTML页面中的文本内容(字体. ...
- Spark高级函数应用【combineByKey、transform】
一.combineByKey算子简介 功能:实现分组自定义求和及计数. 特点:用于处理(key,value)类型的数据. 实现步骤: 1.对要处理的数据进行初始化,以及一些转化操作 2.检测key是否 ...
- 【Docker】Docker容器中安装vim命令
1)先执行 apt-get update 2) 再执行 apt-get install vim
- Linux-crontab定时执行脚本配置
crontab是一个可以根据自己配置的命令定时执行脚本的服务 安装crontab(centos) yum install Vixie-cron yum install crontabs vixie-c ...