转自:https://www.stitchdata.com/blog/supercharging-etl-with-airflow-and-singer/ singer 团队关于singer 与airflow 集成的文章

Earlier this year we introduced Singer, an open source project that helps data teams build simple, composable ETL. Singer provides a standard way for anyone to pull data from and send data to any source.

For many companies, however, being able to push and pull data or move things from A to B is the only part of the problem. Data extraction is often part of a more complex workflow that involves scheduled tasks, complex dependencies, and the need for scalable, distributed architecture.

Enter Apache Airflow. Originally developed at Airbnb and now a part of the Apache Incubator, Airflow takes the simplicity of a cron scheduler and adds all the facets of a modern workflow tool: dependency graphs, detailed logging, automated notifications, scalable infrastructure, and a graphical user interface.

A dependency tree and history of task runs from Airflow’s UI

Imagine a company that relies on data from multiple data sources, including SaaS tools, databases, and flat files. Several times a day this company might want to ingest new data from these sources in parallel. The company might manipulate it in some way, then dump the output into a data warehouse.

Airflow and Singer can make all of that happen. With a few lines of code, you can use Airflow to easily schedule and run Singer tasks, which can then trigger the remainder of your workflow.

A real-world example

Let’s look at a real-world example developed by a member of the Singer community. In this scenario we’re going to be pulling in CSV files, but Singer can work with any data source.

Our user has a specific sequence of tasks they need to complete each morning:

  • Download new compressed CSV files from an AWS S3 bucket
  • Decompress those files
  • Use a Singer CSV tap to push the data to a Singer Stitch target. In this example we’re dumping data into Amazon Redshift, but you could target Google BigQuery or Postgres, too.
  • Delete the compressed and decompressed files

This entire workflow, including all scripts, logging, and the Airflow implementation itself, is accomplished in fewer than 160 lines of Python code in this repo. Let’s see how it’s done.

Firing up Airflow

First we get Airflow running as described on the project’s Quick Start page with four commands:

# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow # install from pypi using pip
pip install airflow # initialize the database
airflow initdb # start the web server, default port is 8080
airflow webserver -p 8080

Upon running that last command, you should see some ASCII art, letting you know that the web server is online:

Now, point your browser to http://localhost:8080/ to see a screen that looks like this:

At this point, you’re ready to create your own Airflow DAG (Directed Acyclic Graph) to perform data workflow tasks. For our purposes, we can get a ready-made DAG by cloning the airflow-singer repo:

git clone git@github.com:robertjmoore/airflow-singer.git

Customizing the repo

For Airflow to find the DAG in this repo, you’ll need to tweak the dags_folder variable the ~/airflow/airflow.cfg file to point to the dags directory inside the repo:

You’ll also want to make a few tweaks to the singer.py file in the repo’s dags folder to reflect your contact info and the location of the repo on your local file system:

Restart the web server with the command airflow webserver -p 8080, then refresh the Airflow UI in your browser. You should now see the DAG from our repo:

Clicking on it will show us the Graph View, which lays out the steps taken each morning when the DAG is run:

This dependency map is governed by a few lines of code inside the dags/singer.py file. Let’s unpack a little of what’s going on.

Exploring the DAG

This tiny file defines the whole graph. Each of these tasks is a step in the DAG, and the final four lines draw out the dependencies that exist between them.

You’ll notice that, in this file, each step is a BashOperator that calls a specific command-line task and waits for its successful completion. Airflow supports a number of other operators and allows you to build your own. This makes it easy for a DAG to include interactions with databases, email services, and chat tools like Slack.

Interacting with Singer

To get a better idea of how Singer is integrated, check out the individual files in the scripts/ directory. You'll find Python scripts that download data from an Amazon S3 bucket, extract that data, and delete the files on completion.

The most interesting step is the process of using Singer to extract the data from the CSV files and push it to a target – namely Stitch.

We should also note that the CSV tap requires a config file that tells it where to find the CSV files to push to Singer, so one step in our DAG is to generate that JSON config file and then point it to the files we just extracted. We do this by generating a few lines of JSON code. Note that we use the global Airflow variable execution_date across our various scripts to be sure we deposit and retrieve the files from the same path.

Once that config file has been generated, we call Singer to do all the work in a single command line:

tap-csv -c ~/config/csv-config.json | target-stitch -c ~/config/stitch_config.json

This doesn’t even require a special Python script — the entire instruction is laid out in a single line of the singer.py DAG file.

Conclusion

As you can see, incorporating Singer into your Airflow DAGs gives you a powerful way to move data automatically. Anyone can extract and load data with a one-line instruction, using a growing ecosystem of taps and targets.

Supercharging your ETL with Airflow and Singer的更多相关文章

  1. Airbnb架构要点分享——阅读心得

    目前,Airbnb已经使用了大约5000个AWS EC2实例,其中大约1500个实例用于部署其应用程序中面向Web的部分,其余的3500个实例用于各种分析和机器学习算法.而且,随着Airbnb的发展, ...

  2. 《Airbnb架构要点分享》阅读笔记

    Airbnb成立于2008年8月,总部位于加利福尼亚州旧金山市.Airbnb是一个值得信赖的社区型市场,在这里人们可以通过网站.手机或平板电脑发布.发掘和预订世界各地的独特房源,其业务已经覆盖190个 ...

  3. Singer 开源便捷的ETL 工具

    singer 是一个强大,灵活的etl 工具,我们可以方便的提取web api,file,queue,基本上各种你可以想到的 数据源. singer 有一套自己的数据处理规范, taps, targe ...

  4. Singer 学习三 使用Singer进行mongodb 2 postgres 数据转换

    Singer 可以方便的进行数据的etl 处理,我们可以处理的数据可以是api 接口,也可以是数据库数据,或者 是文件 备注: 测试使用docker-compose 运行&&提供数据库 ...

  5. Singer 学习二 使用Singer进行gitlab 2 postgres 数据转换

    Singer 可以方便的进行数据的etl 处理,我们可以处理的数据可以是api 接口,也可以是数据库数据,或者 是文件 备注: 测试使用docker-compose 运行&&提供数据库 ...

  6. 3.Airflow使用

    1. airflow简介2. 相关概念2.1 服务进程2.1.1. web server2.1.2. scheduler2.1.3. worker2.1.4. celery flower2.2 相关概 ...

  7. 4.airflow测试

    1.测试sqoop任务1.1 测试全量抽取1.1.1.直接执行命令1.1.2.以shell文件方式执行sqoop或hive任务1.2 测试增量抽取2.测试hive任务3.总结 当前生产上的任务主要分为 ...

  8. 【airflow实战系列】 基于 python 的调度和监控工作流的平台

    简介 airflow 是一个使用python语言编写的data pipeline调度和监控工作流的平台.Airflow被Airbnb内部用来创建.监控和调整数据管道.任何工作流都可以在这个使用Pyth ...

  9. 调度系统Airflow的第一个DAG

    Airflow的第一个DAG 考虑了很久,要不要记录airflow相关的东西, 应该怎么记录. 官方文档已经有比较详细的介绍了,还有各种博客,我需要有一份自己的笔记吗? 答案就从本文开始了. 本文将从 ...

随机推荐

  1. SQL 2008R2问题:用户、组或角色'XXX'在当前数据库中已存在?

    为一个数据库添加一个用户或者映射数据库时,提示以下错误信息: 用户.组或角色 '*****' 在当前数据库中已存在. (Microsoft SQLServer, 错误 : 15023) 问题原因:在还 ...

  2. C# 使用SuperSocket

    一.需求场景 (1)使用SuperSocket进行网络通信 二.所需软件 (1)SocketTool 三.实现步骤   (1)使用Nuget管理器安装SuperSocket.Engine 该插件已经包 ...

  3. Python中的单例设计

    01. 单例设计模式 设计模式 设计模式 是 前人工作的总结和提炼,通常,被人们广泛流传的设计模式都是针对 某一特定问题 的成熟的解决方案 使用 设计模式 是为了可重用代码.让代码更容易被他人理解.保 ...

  4. Windows Server2008服务器ping不通问题解决

    https://blog.csdn.net/x541211190/article/details/78221949

  5. 工具sublime安装

    默认安装后是英文版 view-show console 安装packagecontrol https://packagecontrol.io/installation ctrl+`打开控制台,输入代码 ...

  6. 《JavaScript高级程序设计》笔记:新兴的API

    requestAnimationFrame() 大多数电脑显示器的刷新频率60HZ,大概相当于每秒钟重绘60次.因此,最平滑动画的最佳循环间隔是1000ms/60,约等于17ms. mozReques ...

  7. psexec与wmi在内网渗透的使用

    psexec是一个很好的管理工具,在内网渗透中也被广泛使用. 但太“出名”也往往会遭来各种麻烦. 在有安全监听.防护的内网中使用psexec会容易触发告警. 1.psexec用法(前提:对方要开启ad ...

  8. 如何实现android和服务器长连接

    转载 这种功能实际上就是数据同步,同时要考虑手机本身.电量.网络流量等等限制因素,所以通常在移动端上有一下两个解决方案: 1.一种是定时去server查询数据,通常是使用HTTP协议来访问web服务器 ...

  9. python3常用的内置函数

    数学相关 abs(a) : 求取绝对值.abs(-1) max(list) : 求取list最大值.max([1,2,3]) min(list) : 求取list最小值.min([1,2,3]) su ...

  10. iOS静态库转Framework动态库

    参考文章: iOS静态库(.a 和framework)  XCode6制作动态及静态Framework  说说iOS中静态库的开发  dyld: Library not loaded: @rpath/ ...