Introducing Outflux: a smart way out of InfluxDB
转自:https://blog.timescale.com/migrate-outflux-a-smart-way-out-of-influxdb/
Migrate your workload from InfluxDB to TimescaleDB with just a single command
Users are often asking us how to migrate off of InfluxDB because they want the reliability and flexibility of TimescaleDB. To make it easier for them, we are introducing a new migration tool called Outflux.
Designed to help users seamlessly migrate from InfluxDB to TimescaleDB, Outflux is built in a modular fashion, enabling users to either migrate schema or data or both directly into TimescaleDB. It’s easy to use, configurable, and most importantly, it’s fast.
In this post, we will first cover the motivations behind creating Outflux, then dive deeper into how we built it, how it works, and how to get started.
[Or jump straight to the tutorial.]
Why we created Outflux
We give props to InfluxData for being an early entry in the time-series database market, but as with any technology, one-size does not fit all. Aside from receiving feedback from our users, we also saw a visible market gap between the services InfluxDB provides, and what users actually need (which led to the birth of TimescaleDB).
Early on we recognized that working with IoT data was a whole new ballgame than the DevOps workloads InfluxDB is designed for. The ability to combine your time-series data with your relational data (and often, geo-spatial data) is becoming increasingly important for organizations. TimescaleDB is optimized for these types of workloads and gives organizations the option to scale from the cloud to the edge.
We’d also be remiss not to touch on the fact that TimescaleDB is built on the SQL open-standard. This allows organizations to unify under a single query language that is well known among developers across the globe. InfluxData initially decided to create their own SQL-like query language called InfluxQL, but that had quirks and gotchas that tripped up developers. Then they created a new language, Flux, to solve the issues associated with InfluxQL. Now their users are faced with learning a whole new language which creates a split ecosystem and even more confusion. Not ideal. (More on SQL vs. Flux here.)
We’ve benchmarked TimescaleDB against InfluxDB and encourage you to review the results. You might just find that InfluxDB is satisfactory for your needs, or you might decide that it’s time for an upgrade and switch to TimescaleDB.
How we built Outflux & how it works
After some careful evaluation and research, we discovered that several API clients exist for InfluxDB for a variety of programming languages including Python and Go. The Go client is maintained by InfluxData themselves, and using this API client allowed us to tailor the tool to our needs.
We found that we could control:
- The amount of data being selected from the InfluxDB server
- Which data (configurably) is exported
- The input format for TimescaleDB (and additional transformations before inserting)
- The concurrency level
From here, we built outflux
as a series of libraries connected in one CLI that selects data from InfluxDB using their HTTP API, discovers and transforms the data schema, and imports it into TimescaleDB concurrently.
Underneath the covers, Outflux implements an Extraction Pipeline that has three stages:
- Extractor: where Outflux queries the input database (InfluxDB) by selecting data specified by the user and converting it into an intermediate format
- Transformation Chain: where configurable changes can be made to the data (i.e. castings, filters, column generation) while preserving the intermediate format
- Injector: where the specific code for the receiving database resides (Outflux knows how to transform the intermediate/neutral format and insert it)
Each of the components work independently of each other in a separate coroutine/thread. Multiple pipelines can be used to export each measurement in a concurrent fashion. The Extraction Pipelines are spawned, joined and managed by the main Outflux component. The figure below illustrates the data flow once the Extraction Pipeline is in place.
Getting started
Now that you know more about Outflux’s internal framework, you are ready to get started! Outflux is an open-source tool and the code is available on GitHub in a public repository.
Using the tool is easy and involves just a simple command: `migrate`. Outflux manages the schema discovery, validation, and creation. It also handles exporting and importing data from an input database to an output database.
Before you begin setting up Outflux you need to ensure you have 1) a running instance of InfluxDB at a known location and a means to connect to it and 2) TimescaleDB installed and a means to connect to it.
If all the pre-requirements are met, you can begin installing Outflux by...
- Visiting the releases section of the repository
- Downloading the latest compressed tarball for your platform
- Extracting it to a preferred location
If you navigate to where you extracted the archive and execute:
$ ./outflux --help
Outflux offers the capabilities to migrate an InfluxDB database, or specific measurements to TimescaleDB. It can also allow a user to transfer only the schema of a database or measurement to TimescaleDB
Usage:
outflux [command]
Available Commands:
help Help about any command
migrate Migrate the schema and data from InfluxDB measurements into TimescaleDB hypertables
schema-transfer Discover the schema of measurements and validate or prepare a TimescaleDB hyper-table with the discovered schema
You will see the help output for Outflux, a brief explanation of what it can do, the usage, and available commands.
For instructions on how to set up Outflux from source, you can visit the README. For step-by-step instructions on how to get started, please read our tutorial.
Note: Outflux currently only supports bulk migrations. Live, continuous migrations from InfluxDB will be supported with other upcoming solutions.
Next steps
Are you ready to migrate off of InfluxDB and upgrade to TimescaleDB? Yes? We thought so! As mentioned above, you can follow the tutorial for in-depth instructions. Be sure to check out the Outflux page for more information and ways to contact us.
If you are new to TimescaleDB, follow these installation instructions. (Note: If you are just getting started, we encourage you to check out our features matrix to see which version of TimescaleDB is best for you.)
FAQs
How much does Outflux cost?
Outflux is free to use! TimescaleDB is also free to use, but if you are looking to upgrade to TimescaleDB Enterprise, we offer different pricing options based on your needs.
Can you tell me more about the differences between TimescaleDB vs InfluxDB?
Of course! We’ve benchmarked TimescaleDB vs InfluxDB and you can read the results on our whitepaper.
What is the difference between SQL and Flux?
We are glad you asked! Read this post to learn all about the two query languages.
Does Outflux do migrations in a live fashion?
No, Outflux currently only supports bulk migrations, so active inserts into InfluxDB after Outflux is used will not migrate over. Continuous migrations from InfluxDB will be supported with other upcoming solutions.
Introducing Outflux: a smart way out of InfluxDB的更多相关文章
- 使用outflux 导入influxdb 的数据到timescaledb
influxdb 以及timescaledb 都是不错的时序数据库,timescaledb 团队提供了直接从influxdb 导入 环境准备 docker-compose 文件 version: &q ...
- Introducing Makisu: Uber’s Fast, Reliable Docker Image Builder for Apache Mesos and Kubernetes
转自:https://eng.uber.com/makisu/?amp To ensure the stable, scalable growth of our diverse tech stack, ...
- Smart/400开发上手1:入门
1.介绍 Smart/400是在AS/400之上的开发平台,管理开发.运维的全生命周期. 2.设计基础 Introducing Fields Smart通过字段字典Field Dictionary来存 ...
- Method and apparatus for establishing IEEE 1588 clock synchronization across a network element comprising first and second cooperating smart interface converters wrapping the network element
Apparatus for making legacy network elements transparent to IEEE 1588 Precision Time Protocol operat ...
- Building Applications with Force.com and VisualForce (DEV401)(五):Application Essential: Introducing Business Logic
Dev 401-005 Application Essential: Introducing Business Logic Module Agenda1.Custom object Queues2.W ...
- Smart Tag——DevExpress WPF初探
Smart Tag是一个设计时扩展,所有标准控件均自带这个功能,当然也包括 DevExpress WPF Controls .可以快速设置控件的值或者绑定最重要的属性.它还可以帮助你完成一些重复的工作 ...
- 【容器云】十分钟快速构建 Influxdb+cadvisor+grafana 监控
本文作者:七牛云布道师@陈爱珍,DBAPlus社群联合发起人.前新炬技术专家.多年企业级系统的应用运维及分布式系统实战经验.现专注于容器.微服务及DevOps落地的研究与实践. 安装过程 三个都直接下 ...
- International Conference for Smart Health 2015 Call for Papers
Advancing Informatics for healthcare and healthcare applications has become an international researc ...
- Call for Papers International Conference for Smart Health (ICSH) 2014
Call for PapersInternational Conference for Smart Health (ICSH) 2014 Beijing, China July 10-11, 2014 ...
随机推荐
- windows2012的服务器远程桌面提示内部错误的问题解决方法
一.问题表象 我们在OpenStack安装了windows server2012r2版本的虚拟机,在本地通过远程桌面连接时,输入账号密码后,提示连接断开或者内部错误的问题 二.解决办法 1)windo ...
- P2886 [USACO07NOV]牛继电器Cow Relays
题目描述 For their physical fitness program, N (2 ≤ N ≤ 1,000,000) cows have decided to run a relay race ...
- 纯css实现评分
用到的知识点: E:checked:单选或复选框被选中 E ~ F: 选择后面的兄弟节点们:选择后面的兄弟节点 E::after,E::before: 伪元素选择器 在匹配E的元素后面(前面)插入内容 ...
- c++学习过程
作者本人也是一名初学者,我的QQ:2522929921,可以一起交流啊! 希望广大初学者能够一起进步: 1.掌握编程思维真的很重要!!!!***. 2.不能刻意记忆语法规则. 3.在循序渐进的项目实战 ...
- js循环出相同name,不同id的按钮,对其进行点击回复操作
function getseat(){ var option= "<button class='btn'style='margin:5px;' onclick='onclickSeat ...
- vue+koa实现简单的图书小程序(2)
记录一下实现我们图书的扫码功能: https://developers.weixin.qq.com/miniprogram/dev/api/scancode.html要多读文档 scanBook () ...
- 元组tuple 可迭代对象
1. 字符串.元组和列表相互转化 s = 'abcdef' li = ['a','b','c','d'] tu = (1,2,4,9) v = tuple(s) print('v = ',v) w = ...
- yii2下载
我发现yii2下载贼困难,不过折腾了很久终于搞下来了. 一,设置镜像 安装composer , 下载 composer.phar 添加中国镜像 composer config -g repo. ...
- 转 Mac 下自带的中文输入法不显示汉字提示问题
原文 https://blog.csdn.net/moxi_wang/article/details/50721326 当时聊天的时候不知道那个手指头按错了什么键 导致Mac自带的中文输入法不能提示显 ...
- 田螺便利店—filezilla实现Linux和windows通信(二)
filezilla,FileZilla是一个免费开源的FTP软件,分为客户端版本和服务器版本,具备所有的FTP软件功能.可控性.有条理的界面和管理多站点的简化方式使得Filezilla客户端版成为一个 ...