Docker 安装 Apache Airflow

参考资料

安装依赖

  1. Docker Engine
  2. Docker Composite

快速运行 Apache Airflow 2.2.4

在 Docker 使用 CeleryExecutor(一种统计 worker 数量的途径) 快速运行 Apache Airflow

1. 下载 docker-compose.yaml

命令:


  1. # 创建一个目录
  2. mkdir -p /home/public/Soft/airflow
  3. cd /home/public/Soft/airflow
  4. # 下载
  5. curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.2.4/docker-compose.yaml'

这个文件包含了多个服务的定义:

  • airflow-scheduler - The scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete.
  • airflow-webserver - The webserver is available at http://localhost:8080.
  • airflow-worker - The worker that executes the tasks given by the scheduler.
  • airflow-init - The initialization service.
  • flower - The flower app for monitoring the environment. It is available at http://localhost:5555.
  • postgres - The database.
  • redis - The redis - broker that forwards messages from scheduler to worker.

docker-compose.yaml 文件内容如下:

  1. # Licensed to the Apache Software Foundation (ASF) under one
  2. # or more contributor license agreements. See the NOTICE file
  3. # distributed with this work for additional information
  4. # regarding copyright ownership. The ASF licenses this file
  5. # to you under the Apache License, Version 2.0 (the
  6. # "License"); you may not use this file except in compliance
  7. # with the License. You may obtain a copy of the License at
  8. #
  9. # http://www.apache.org/licenses/LICENSE-2.0
  10. #
  11. # Unless required by applicable law or agreed to in writing,
  12. # software distributed under the License is distributed on an
  13. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  14. # KIND, either express or implied. See the License for the
  15. # specific language governing permissions and limitations
  16. # under the License.
  17. #
  18. # Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
  19. #
  20. # WARNING: This configuration is for local development. Do not use it in a production deployment.
  21. #
  22. # This configuration supports basic configuration using environment variables or an .env file
  23. # The following variables are supported:
  24. #
  25. # AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
  26. # Default: apache/airflow:2.2.4
  27. # AIRFLOW_UID - User ID in Airflow containers
  28. # Default: 50000
  29. # Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
  30. #
  31. # _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
  32. # Default: airflow
  33. # _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
  34. # Default: airflow
  35. # _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
  36. # Default: ''
  37. #
  38. # Feel free to modify this file to suit your needs.
  39. ---
  40. version: '3'
  41. x-airflow-common:
  42. &airflow-common
  43. # In order to add custom dependencies or upgrade provider packages you can use your extended image.
  44. # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  45. # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  46. image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.4}
  47. # build: .
  48. environment:
  49. &airflow-common-env
  50. AIRFLOW__CORE__EXECUTOR: CeleryExecutor
  51. AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
  52. AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
  53. AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
  54. AIRFLOW__CORE__FERNET_KEY: ''
  55. AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
  56. AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
  57. AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
  58. _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  59. volumes:
  60. - ./dags:/opt/airflow/dags
  61. - ./logs:/opt/airflow/logs
  62. - ./plugins:/opt/airflow/plugins
  63. user: "${AIRFLOW_UID:-50000}:0"
  64. depends_on:
  65. &airflow-common-depends-on
  66. redis:
  67. condition: service_healthy
  68. postgres:
  69. condition: service_healthy
  70. services:
  71. postgres:
  72. image: postgres:13
  73. environment:
  74. POSTGRES_USER: airflow
  75. POSTGRES_PASSWORD: airflow
  76. POSTGRES_DB: airflow
  77. volumes:
  78. - postgres-db-volume:/var/lib/postgresql/data
  79. healthcheck:
  80. test: ["CMD", "pg_isready", "-U", "airflow"]
  81. interval: 5s
  82. retries: 5
  83. restart: always
  84. redis:
  85. image: redis:latest
  86. expose:
  87. - 6379
  88. healthcheck:
  89. test: ["CMD", "redis-cli", "ping"]
  90. interval: 5s
  91. timeout: 30s
  92. retries: 50
  93. restart: always
  94. airflow-webserver:
  95. <<: *airflow-common
  96. command: webserver
  97. ports:
  98. - 8080:8080
  99. healthcheck:
  100. test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
  101. interval: 10s
  102. timeout: 10s
  103. retries: 5
  104. restart: always
  105. depends_on:
  106. <<: *airflow-common-depends-on
  107. airflow-init:
  108. condition: service_completed_successfully
  109. airflow-scheduler:
  110. <<: *airflow-common
  111. command: scheduler
  112. healthcheck:
  113. test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
  114. interval: 10s
  115. timeout: 10s
  116. retries: 5
  117. restart: always
  118. depends_on:
  119. <<: *airflow-common-depends-on
  120. airflow-init:
  121. condition: service_completed_successfully
  122. airflow-worker:
  123. <<: *airflow-common
  124. command: celery worker
  125. healthcheck:
  126. test:
  127. - "CMD-SHELL"
  128. - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
  129. interval: 10s
  130. timeout: 10s
  131. retries: 5
  132. environment:
  133. <<: *airflow-common-env
  134. # Required to handle warm shutdown of the celery workers properly
  135. # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
  136. DUMB_INIT_SETSID: "0"
  137. restart: always
  138. depends_on:
  139. <<: *airflow-common-depends-on
  140. airflow-init:
  141. condition: service_completed_successfully
  142. airflow-triggerer:
  143. <<: *airflow-common
  144. command: triggerer
  145. healthcheck:
  146. test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
  147. interval: 10s
  148. timeout: 10s
  149. retries: 5
  150. restart: always
  151. depends_on:
  152. <<: *airflow-common-depends-on
  153. airflow-init:
  154. condition: service_completed_successfully
  155. airflow-init:
  156. <<: *airflow-common
  157. entrypoint: /bin/bash
  158. # yamllint disable rule:line-length
  159. command:
  160. - -c
  161. - |
  162. function ver() {
  163. printf "%04d%04d%04d%04d" $${1//./ }
  164. }
  165. airflow_version=$$(gosu airflow airflow version)
  166. airflow_version_comparable=$$(ver $${airflow_version})
  167. min_airflow_version=2.2.0
  168. min_airflow_version_comparable=$$(ver $${min_airflow_version})
  169. if (( airflow_version_comparable < min_airflow_version_comparable )); then
  170. echo
  171. echo -e "\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m"
  172. echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
  173. echo
  174. exit 1
  175. fi
  176. if [[ -z "${AIRFLOW_UID}" ]]; then
  177. echo
  178. echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
  179. echo "If you are on Linux, you SHOULD follow the instructions below to set "
  180. echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
  181. echo "For other operating systems you can get rid of the warning with manually created .env file:"
  182. echo " See: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user"
  183. echo
  184. fi
  185. one_meg=1048576
  186. mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
  187. cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
  188. disk_available=$$(df / | tail -1 | awk '{print $$4}')
  189. warning_resources="false"
  190. if (( mem_available < 4000 )) ; then
  191. echo
  192. echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
  193. echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
  194. echo
  195. warning_resources="true"
  196. fi
  197. if (( cpus_available < 2 )); then
  198. echo
  199. echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
  200. echo "At least 2 CPUs recommended. You have $${cpus_available}"
  201. echo
  202. warning_resources="true"
  203. fi
  204. if (( disk_available < one_meg * 10 )); then
  205. echo
  206. echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
  207. echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
  208. echo
  209. warning_resources="true"
  210. fi
  211. if [[ $${warning_resources} == "true" ]]; then
  212. echo
  213. echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
  214. echo "Please follow the instructions to increase amount of resources available:"
  215. echo " https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin"
  216. echo
  217. fi
  218. mkdir -p /sources/logs /sources/dags /sources/plugins
  219. chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
  220. exec /entrypoint airflow version
  221. # yamllint enable rule:line-length
  222. environment:
  223. <<: *airflow-common-env
  224. _AIRFLOW_DB_UPGRADE: 'true'
  225. _AIRFLOW_WWW_USER_CREATE: 'true'
  226. _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
  227. _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
  228. user: "0:0"
  229. volumes:
  230. - .:/sources
  231. airflow-cli:
  232. <<: *airflow-common
  233. profiles:
  234. - debug
  235. environment:
  236. <<: *airflow-common-env
  237. CONNECTION_CHECK_MAX_COUNT: "0"
  238. # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
  239. command:
  240. - bash
  241. - -c
  242. - airflow
  243. flower:
  244. <<: *airflow-common
  245. command: celery flower
  246. ports:
  247. - 5555:5555
  248. healthcheck:
  249. test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
  250. interval: 10s
  251. timeout: 10s
  252. retries: 5
  253. restart: always
  254. depends_on:
  255. <<: *airflow-common-depends-on
  256. airflow-init:
  257. condition: service_completed_successfully
  258. volumes:
  259. postgres-db-volume:

2. 在 docker-compose.yaml 同级目录下创建文件夹

在 docker-compose.yaml 同级目录下,创建 dags logs plugins文件夹

  1. cd /home/public/Soft/airflow
  2. mkdir -p ./dags
  3. mkdir -p ./logs
  4. mkdir -p ./plugins

dags logs plugins文件夹 作用:

  • ./dags - you can put your DAG files here.
  • ./logs - contains logs from task execution and scheduler.
  • ./plugins - you can put your custom plugins here.

3. 初始化环境

初始化环境,就是添加几个文件夹。

3.1 设置正确的用户

命令:

  1. cd /home/public/Soft/airflow
  2. echo -e "AIRFLOW_UID=$(id -u)" > .env

其中,AIRFLOW_UID 是 Docker Compose 环境变量,具体请看(https://airflow.apache.org/docs/apache-airflow/2.2.4/start/docker.html#docker-compose-env-variables )。

生成的 .env 文件内容可能如下:

  1. AIRFLOW_UID=50000

3.2 初始化数据库

  1. cd /home/public/Soft/airflow
  2. docker-compose up airflow-init

控制台可能打印如下内容:

  1. airflow-init_1 | Upgrades done
  2. airflow-init_1 | Admin user airflow created
  3. airflow-init_1 | 2.2.4
  4. start_airflow-init_1 exited with code 0

初始化,默认的 Airflow 的登陆用户和密码 : airflow airflow

4. 运行 airflow

  1. cd /home/public/Soft/airflow
  2. docker-compose up

5. 访问环境

有3中方式访问环境:命令行,浏览器访问,REST API。

5.1 命令行

下载 airflow.sh

  1. curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.2.4/airflow.sh'
  2. chmod +x airflow.sh

airflow.sh 脚本内容如下:

  1. #!/usr/bin/env bash
  2. # Licensed to the Apache Software Foundation (ASF) under one
  3. # or more contributor license agreements. See the NOTICE file
  4. # distributed with this work for additional information
  5. # regarding copyright ownership. The ASF licenses this file
  6. # to you under the Apache License, Version 2.0 (the
  7. # "License"); you may not use this file except in compliance
  8. # with the License. You may obtain a copy of the License at
  9. #
  10. # http://www.apache.org/licenses/LICENSE-2.0
  11. #
  12. # Unless required by applicable law or agreed to in writing,
  13. # software distributed under the License is distributed on an
  14. # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  15. # KIND, either express or implied. See the License for the
  16. # specific language governing permissions and limitations
  17. # under the License.
  18. #
  19. # Run airflow command in container
  20. #
  21. PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
  22. set -euo pipefail
  23. export COMPOSE_FILE="${PROJECT_DIR}/docker-compose.yaml"
  24. if [ $# -gt 0 ]; then
  25. exec docker-compose run --rm airflow-cli "${@}"
  26. else
  27. exec docker-compose run --rm airflow-cli
  28. fi

使用 airflow.sh 可以快速执行命令,例如:

  1. arflow.sh info

5.2 浏览器访问

浏览器访问 http://localhost:8080
默认登录名和密码: airflow airflow

5.3 给 REST API 发请求

使用 curl 发请求:

  1. ENDPOINT_URL="http://localhost:8080/"
  2. curl -X GET \
  3. --user "airflow:airflow" \
  4. "${ENDPOINT_URL}/api/v1/pools"

清除容器

清除容器,卷等,命令如下:

  1. docker-compose down --volumes --rmi all

清除环境信息

以上是快速启动配置,如果需要定制化配置,则可以先清除环境信息

  1. 停止容器
  1. cd /home/public/Soft/airflow
  2. docker-compose down --volumes --remove-orphans
  1. 删除下载目录和 docker-compose.yaml
  1. cd /home/public/Soft/airflow
  2. rm -rf *
  1. 重新下载 docker-compose.yaml
  1. curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.2.4/docker-compose.yaml'
  1. 从开头重新执行指令

在 Docker 上快速运行 Apache Airflow 2.2.4的更多相关文章

  1. Windows OS上安装运行Apache Kafka教程

    Windows OS上安装运行Apache Kafka教程 下面是分步指南,教你如何在Windows OS上安装运行Apache Zookeeper和Apache Kafka. 简介 本文讲述了如何在 ...

  2. 在docker上安装运行mysql实例

    ps:实验环境是:CentOS Linux release 7.3  64位1.获取mysql镜像从docker hub的仓库中拉取mysql镜像docker pull mysql查看镜像docker ...

  3. Kafka-Docker:使用Docker运行Apache Kafka的步骤

    1.目标 在这个Kafka教程中,我们将学习Kafka-Docker的概念.此外,我们将在Kafka中看到Docker的卸载过程.这包括使用Docker 运行Apache Kafka的所有步骤  .除 ...

  4. Apache PredictionIO在Docker上的搭建及使用

    1.Apache PredictionIO介绍 Apache PredictionIO 是一个孵化中的机器学习服务器,它可以为为开发人员和数据科学家创建任何机器学习任务的预测引擎.官方原文: Apac ...

  5. 在 Docker 上运行一个 RESTful 风格的微服务

    tags: Microservice Restful Docker Author: Andy Ai Weibo:NinetyH GitHub: https://github.com/aiyanbo/d ...

  6. .NET Core 3.0 部署在docker上运行

    自从.NET Core3.0发布之后,写了几篇关于.NET Core 3.0的文章,有助于你快速入门.NET Core3.0. 本篇文章主要讲解如何一步步创建一个mvc项目,然后发布并部署在Docke ...

  7. 在OSX和Windows版本Docker上运行GUI程序

    看到很多人在Docker问题区讨论:如何在OS X和Windows的Docker上运行GUI程序, 随手记录几个参考资料: https://github.com/docker/docker/issue ...

  8. 理一下docker在各平台上的运行机制

    理一下docker在各平台上的运行机制 首先,从内核共享与否 docker在linux上共享内核,无需虚拟化,完全支持native功能(https://docs.docker.com/engine/i ...

  9. ELK 性能(3) — 在 Docker 上运行高性能容错的 Elasticsearch 集群

    ELK 性能(3) - 在 Docker 上运行高性能容错的 Elasticsearch 集群 介绍 在 Docker 上运行高性能容错的 Elasticsearch 集群 内容 通常熟悉的开发流程是 ...

随机推荐

  1. Python基础练习之购物车

    #前置知识点 # enumerate(LIST) # 输出一个元组,第一个为下标,第二个为元素 # a = [1, 2, 3, 4] # for i in enumerate(a): # print( ...

  2. (acwing蓝桥杯c++AB组)2.1 二分

    二分与前缀和 文章目录 二分与前缀和 二分 整数二分核心思想 整数二分模板 整数二分步骤总结: 题目链接 实数二分核心思想: 题目链接 三分法思想: 二分 难点:二分的边界问题 整数二分核心思想 确定 ...

  3. Rafy 框架:领域控制器

    本文简要说明如何使用 Rafy 框架中的领域控制器. 简介 领域控制器是 Rafy 框架中用于封装领域逻辑的主要方式. 在控制器中,开发者可以封装大量的业务逻辑,并向外暴露业务接口.内部的逻辑在实现时 ...

  4. 羽夏看Win系统内核—— x64 番外篇

    写在前面   此系列是本人一个字一个字码出来的,包括示例和实验截图.由于系统内核的复杂性,故可能有错误或者不全面的地方,如有错误,欢迎批评指正,本教程将会长期更新. 如有好的建议,欢迎反馈.码字不易, ...

  5. eureka自我保护机制是什么?

    当Eureka Server 节点在短时间内丢失了过多实例的连接时(比如网络故障或频繁启动关闭客户端)节点会进入自我保护模式,保护注册信息,不再删除注册数据,故障恢复时,自动退出自我保护模式.

  6. Zookeeper 对节点的 watch监听通知是永久的吗?为什么 不是永久的?

    不是.官方声明:一个 Watch 事件是一个一次性的触发器,当被设置了 Watch 的数据发生了改变的时候,则服务器将这个改变发送给设置了 Watch 的客户端, 以便通知它们. 为什么不是永久的,举 ...

  7. 发现程序美----while+for冒泡实现的

    思想记录: 每一轮回的冒泡都将产生一个最大值,其后每次循环次数都将少一次,因为每次都会确定一个最大值. private void method(){ int[] list = {10,7,8,4,7, ...

  8. List和 Map区别?

    一个是存储单列数据的集合,另一个是存储键和值这样的双列数据的集合,List中存储的数据是有顺序,并且允许重复:Map中存储的数据是没有顺序的,其键是不能重复的,它的值是可以有重复的.

  9. Statement 和 PreparedStatement 有什么区别?哪个性 能更好?

    与 Statement 相比,①PreparedStatement 接口代表预编译的语句,它主要的优 势在于可以减少 SQL 的编译错误并增加 SQL 的安全性(减少 SQL 注射攻击的可 能性):② ...

  10. java-web中的Filter&Listener

    Filter过滤器 当访问服务器资源的时候,过滤器可以将i气你个球拦截下来,完成一些特殊的功能 过滤器的作用: 一般用于完成通用的操作,如验证登陆,统一的编码处理,敏感字符过滤.就是打游戏骂人,会出现 ...