Singer 可以方便的进行数据的etl 处理,我们可以处理的数据可以是api 接口,也可以是数据库数据,或者
是文件
备注: 测试使用docker-compose 运行&&提供数据库内容,使用virtualenv && python 3.5 以及以上

环境准备

  • docker-compose 文件
 
version: "3"
services:
  gogs-service:
    image: gogs/gogs
    ports:
      - "10022:22"
      - "10080:3000"
  mysql:
    image: mysql:5.7.16
    ports:
      - 3306:3306
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    environment:
      MYSQL_ROOT_PASSWORD: dalongrong
      MYSQL_DATABASE: gogs
      MYSQL_USER: gogs
      MYSQL_PASSWORD: dalongrong
      TZ: Asia/Shanghai
  postgres:
    image: postgres:9.6.11
    ports:
    - "5432:5432"
    environment:
    - "POSTGRES_PASSWORD:dalong"
 
 
  • postgres target 配置
    target.json
 
{
    "host": "localhost",
    "port": 5432,
    "dbname": "postgres",
    "user": "postgres",
    "password": "postgres",
    "schema": "public"
}
 
 
  • 创建gitlab virtualenv
virtualenv gitlab  
source ./gitlab/bin/activate
pip install tap-gitlab
 
  • 创建access_token
    从gitlab 官方网站创建即可
  • gitlab tap 配置文件
    格式如下,因为隐私没有暴露:
 
{
 "api_url": "https://gitlab.com/api/v4",
 "private_token": "xxxxxxx",
 "groups": "",
 "projects": "", 
 "start_date":"2010-01-01T00:00:00Z"
}
 
 

运行&&效果

  • 运行
./gitlab/bin/tap-gitlab -c gitlab.json | ./postgres/bin/target-postgres -c target.json
 
  • 效果
INFO Starting sync
INFO GET https://gitlab.com/api/v4/projects/dalongrong%2Fdemoapp?private_token=XXXXXXXXX
INFO Table 'projects' does not exist. Creating... CREATE TABLE public.projects ("approvals_before_merge" bigint, "archived" boolean, "avatar
_url" character varying, "builds_enabled" boolean, "container_registry_enabled" boolean, "created_at" timestamp without time zone, "creator_
id" bigint, "default_branch" character varying, "description" character varying, "forks_count" bigint, "http_url_to_repo" character varying,
 "id" bigint, "issues_enabled" boolean, "last_activity_at" timestamp without time zone, "lfs_enabled" boolean, "merge_requests_enabled" bool
ean, "name" character varying, "name_with_namespace" character varying, "namespace__id" bigint, "namespace__kind" character varying, "namespace__name" character varying, "namespace__path" character varying, "only_allow_merge_if_all_discussions_are_resolved" boolean, "only_allow_merge_if_build_succeeds" boolean, "open_issues_count" bigint, "owner_id" bigint, "path" character varying, "path_with_namespace" character varying, "public" boolean, "public_builds" boolean, "request_access_enabled" boolean, "shared_runners_enabled" boolean, "shared_with_groups" jsonb, "snippets_enabled" boolean, "ssh_url_to_repo" character varying, "star_count" bigint, "tag_list" jsonb, "visibility_level" bigint, "web_url" character varying, "wiki_enabled" boolean, PRIMARY KEY ("id"))
INFO Table 'branches' does not exist. Creating... CREATE TABLE public.branches ("commit_id" character varying, "developers_can_merge" boolean, "developers_can_push" boolean, "merged" boolean, "name" character varying, "project_id" bigint, "protected" boolean, PRIMARY KEY ("project_id", "name"))
INFO Table 'commits' does not exist. Creating... CREATE TABLE public.commits ("allow_failure" boolean, "author_email" character varying, "author_name" character varying, "committer_email" character varying, "committer_name" character varying, "created_at" timestamp without time zone, "id" character varying, "message" character varying, "project_id" bigint, "short_id" character varying, "title" character varying, PRIMARY KEY ("id"))
INFO Table 'issues' does not exist. Creating... CREATE TABLE public.issues ("assignee_id" bigint, "author_id" bigint, "confidential" boolean, "created_at" timestamp without time zone, "description" character varying, "due_date" character varying, "id" bigint, "iid" bigint, "labels" jsonb, "milestone_id" bigint, "project_id" bigint, "state" character varying, "subscribed" boolean, "title" character varying, "updated_at" timestamp wi
 
 

说明

使用类似的方法,我们也可以转换github 的以及jira 等基于api 开发的模型

参考资料

https://github.com/singer-io/tap-gitlab
https://github.com/rongfengliang/singer-mysql2postges-demo

Singer 学习二 使用Singer进行gitlab 2 postgres 数据转换的更多相关文章

  1. Singer 学习三 使用Singer进行mongodb 2 postgres 数据转换

    Singer 可以方便的进行数据的etl 处理,我们可以处理的数据可以是api 接口,也可以是数据库数据,或者 是文件 备注: 测试使用docker-compose 运行&&提供数据库 ...

  2. Singer 学习一 使用Singer进行mysql 2 postgres 数据转换

    Singer 因为版本的问题,推荐的运行方式是使用virtualenv,对于taps&& target 的运行都是 推荐使用此方式,不然包兼容的问题太费事了 备注: 使用docker- ...

  3. Singer 学习七 运行&&开发taps、targets (二 targets 运行说明)

    接上文: Singer 学习六 运行&&开发taps.targets (一 taps 运行说明) 说明target 需要tap 进行配合运行,所以需要了解tap 的使用 运行targe ...

  4. emberjs学习二(ember-data和localstorage_adapter)

    emberjs学习二(ember-data和localstorage_adapter) 准备工作 首先我们加入ember-data和ember-localstorage-adapter两个依赖项,使用 ...

  5. ReactJS入门学习二

    ReactJS入门学习二 阅读目录 React的背景和基本原理 理解React.render() 什么是JSX? 为什么要使用JSX? JSX的语法 如何在JSX中如何使用事件 如何在JSX中如何使用 ...

  6. TweenMax动画库学习(二)

    目录            TweenMax动画库学习(一)            TweenMax动画库学习(二)            TweenMax动画库学习(三)            Tw ...

  7. Hbase深入学习(二) 安装hbase

    Hbase深入学习(二) 安装hbase This guidedescribes setup of a standalone hbase instance that uses the local fi ...

  8. Struts2框架学习(二) Action

    Struts2框架学习(二) Action Struts2框架中的Action类是一个单独的javabean对象.不像Struts1中还要去继承HttpServlet,耦合度减小了. 1,流程 拦截器 ...

  9. Python学习二:词典基础详解

    作者:NiceCui 本文谢绝转载,如需转载需征得作者本人同意,谢谢. 本文链接:http://www.cnblogs.com/NiceCui/p/7862377.html 邮箱:moyi@moyib ...

随机推荐

  1. 【转载四】Grafana系列教程–Grafana基本概念

    在上面几篇文章中,我们介绍了Grafana的安装配置以及运行的方法,本篇文章我们就来介绍下Grafana的基本概念. 有问题欢迎加群讨论,InfluxDB&Grafana技术交流群:58048 ...

  2. awk使用教程

    gawk - pattern scanning and processing language 基本用法:gawk [options] 'program' FILE ... program:PATTE ...

  3. [Linux]Linux下yaf发送IPV4和IPV6的IPfix

    一.IPFIX与NetFlow 一.IPFIX 全称IP Flow Information Export,即IP流信息输出,是网络流量监测的国际标准.IPFIX是IETF的一个工作组,它的主要工作就是 ...

  4. בוא--来吧--IPA--希伯来语

    灰常好听的希伯来语歌曲, Rita唱得真好.

  5. [Leetcode 392]判断子序列 Is Subsequence

    [思路] 判断s是否为t的子串,所以length(s)<=length(t).于是两个指针,一次循环. 将s.t转换为数组p1.p2. i为过程中s的匹配长度. i=0空串,单独讨论返回true ...

  6. 1.5 socket服务器传输文件

    socket服务器代码 # -*- coding: utf-8 -*-import sys,os,time,_thread from socket import * host = 'localhost ...

  7. MyEclipse教程:使用UML创建模块库——第二部分(一)

    MyEclipse 在线订购年终抄底促销!火爆开抢>> [MyEclipse最新版下载] UML2建模文件存储在建模存储库中,建模可用于生成Java代码,或者可以从代码生成模型. 本教程介 ...

  8. linux 常用重启

    sudo reboot 为最常用重启 Linux centos关机与重启命令详解与实战 Linux centos重启命令: 1.reboot 普通重启 2.shutdown -r now 立刻重启(r ...

  9. JDK的下载及配置

    下载地址,为了兼容最好安装1.8版本 jdk1.8:http://jbox.jd.com/quickshare/d2wheyazjtdccshou2dbo24roejdk1.7:http://jbox ...

  10. idea 一些插件配置

    接触maven快2年了吧,对maven还是一知半解其实.得到了一些教训,就是少转牛角尖,多把握实际需要的东西,一口一口吃饭. 插件化很常见了.这里记录idea使用的jetty插件 和tomcat插件和 ...