influxDB应用及TICK stack

InfluxData平台用于处理度量和事件的时间序列平台，常被称为TICK stack，包含4个组件：Telegraf，influxDB，Chronograf和Kapacitor，分别负责时间序列数据的：data collection，data storage，data visualization，和data processing and alerting。

一．  基础

1．     基础概念

influxdb是时间序列数据库，与常用的关系数据库模型不同。以下列采样数据为例介绍。

time：influxdb是时间序列数据库，所有都与time相关，time列存储timestamp，显示日期和时间，遵循RFC3339 UTC格式。

database：通用概念，管理一系列相关数据的集合，包含users，数据等，针对influxdb，包含users，retention policies，continuous queries和time series data。

measurement：一类数据的集合，与关系数据库的table等同。measurement的名字是strings。采样数据的measurement是census。

point：一条数据，Each point is uniquely identified by its series and timestamp。每条influxdb数据写规则遵循line protocal：

<measurement>[,<tag-key>=<tag-value>...] <field-key>=<field-value>[,<field2-key>=<field2-value>...] [unix-nano-timestamp]

与采样数据的每行相对应。

tag：用于存储metadata，用于标记数据。tags是键值对，由tag keys和tag values组成。采用数据中tag keys是location和scientist，tag key location的tag values是1和2，tag key scientist的tag values是langstroth和perpetua。tags是可索引的（indexed），可根据tags进行快速查询。tags是可选的，可以不含有tags。

tag set：不同tag键值对的组合，采样数据中有四个tag set:

l  location = 1, scientist = langstroth

l  location = 2, scientist = langstroth

l  location = 1, scientist = perpetua

l  location = 2, scientist = perpetua

field：存储数据，键值对，由field keys和field values组成，采样数据中field key是butterflies和honeybees，field values记录数量。field keys是string，field values可为strings，floats，integers或者Booleans。对influxdb每条数据，fields是必有的。fields不可索引，这意味着以field过滤数据时必须扫描measurement所有数据，效率不高。一般情况下，field不应包含元数据。

field set：field-key和field-value的集合。每条记录（point）的field-key和field-value是一条field set。

RP，retention policy：保存策略。描述数据保存多久（duration）和数据保存几份（replication）。一般为autogen retention policy：an infinite duration and a replication factor set to one。

series：数据的集合，共享同一retention policy、measurement、tag set。采样数据中包含4组series：

l  series 1    autogen census  location = 1,scientist = langstroth

l  series 2    autogen census  location = 2,scientist = langstroth

l  series 3    autogen census  location = 1,scientist = perpetua

l  series 4    autogen census  location = 2,scientist = perpetua

2．     扩展概念

batch：批处理，与通用概念相同。A collection of points in line protocol format, separated by newlines (0x0A). A batch of points may be submitted to the database using a single HTTP request to the write endpoint. influxdb推荐batch大小为5000-10000 points。

CQ：continuous query，An InfluxQL query that runs automatically and periodically within a database. 在一个数据库中自动周期运行的数据查询。

now()：The local server’s nanosecond timestamp.

schema：How the data are organized in InfluxDB. The fundamentals of the InfluxDB schema are databases, retention policies, series, measurements, tag keys, tag values, and field keys.

wal：write ahead log，The temporary cache for recently written points. To reduce the frequency with which the permanent storage files are accessed, InfluxDB caches new points in the WAL until their total size or age triggers a flush to more permanent storage. This allows for efficient batching of the writes into the TSM.

tsm：Time Structured Merge tree，The purpose-built data storage format for InfluxDB. TSM allows for greater compaction and higher write and read throughput than existing B+ or LSM tree implementations.

3．     InfluxQL

InfluxQL是一种类似SQL的查询语言，用于与influxdb中的数据进行交互。详细语法见https://docs.influxdata.com/influxdb/v1.7/query_language/。

1．select

SELECT <field_key>[,<field_key>,<tag_key>] FROM <measurement_name>[,<measurement_name>]

l  当包含tag_key时必须指定一个field_key。

l  <database_name>.<retention_policy_name>.<measurement_name>  限定measurement

l  <database_name>..<measurement_name> 采用默认（default）retention policy

l  推荐用双引号将field/tag/measurement等引用。

l  "<field_key>"::field,"<tag_key>"::tag 当field和tag有相同名字时，用于指定名称是field还是tag

SELECT_clause FROM_clause WHERE <conditional_expression> [(AND|OR) <conditional_expression> [...]]

2． where

where基于fields/tags/timestamps过滤数据，the default time range is between 1677-09-21 00:12:43.145224194 and 2262-04-11T23:47:16.854775806Z UTC. For SELECT statements with a GROUP BY time() clause, the default time range is between 1677-09-21 00:12:43.145224194 UTC and now().

l  SELECT * FROM "h2o_feet" WHERE time > now() - 7d

l  SELECT * FROM "h2o_feet" WHERE "water_level" + 2 > 11.9

3． group by

Group by支持tags或时间间隔分组

SELECT_clause FROM_clause [WHERE_clause] GROUP BY [* | <tag_key>[,<tag_key]]

SELECT <function>(<field_key>) FROM_clause WHERE <time_range> GROUP BY time(<time_interval>),[tag_key] [fill(<fill_option>)]

SELECT <function>(<field_key>) FROM_clause WHERE <time_range> GROUP BY time(<time_interval>,<offset_interval>),[tag_key] [fill(<fill_option>)]

4．         info

INFO写查询结果到指定measurement

SELECT_clause INTO <measurement_name> FROM_clause [WHERE_clause] [GROUP_BY_clause]

l  INTO <database_name>.<retention_policy_name>.<measurement_name>

l  INTO <database_name>.<retention_policy_name>.:MEASUREMENT FROM /<regular_expression>/

Writes data to all measurements in the user-specified database and retention policy that match the regular expressionin the FROM clause. :MEASUREMENT is a backreference to each measurement matched in the FROM clause.

5．         order by

ORDER BY time DESC反序显示数据

SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] ORDER BY time DESC

6．         limit

LIMIT/SLIMIT限定points和series数量

SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] LIMIT <N>
SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] GROUP BY *[,time(<time_interval>)] [ORDER_BY_clause] LIMIT <N1> SLIMIT <N2>

7．         tz

The tz() clause returns the UTC offset for the specified timezone.

SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] tz('<time_zone>')

l  SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:18:00Z' tz('America/Chicago')

8．         数据类型指定

SELECT_clause <field_key>::<type> FROM_clause

type可以为float，integer，string，或boolean。

9．         数据类型转换

SELECT_clause <field_key>::<type> FROM_clause

type可为float或integer。

10．         多语句查询用分号（；）分割。

11．         子查询

SELECT_clause FROM ( SELECT_statement ) [...]
SELECT_clause FROM ( SELECT_clause FROM ( SELECT_statement ) [...] ) [...]

二．  influxdata平台安装

两种体验方式：本地安装或docker安装。

1．     本地安装

在Ubuntu16.04上可直接从influxdata仓库中https://repos.influxdata.com/ubuntu/pool/stable/下载编译好的deb包通过如下命令安装：

sudo dpkg -i <package_name>

软件包版本号及端口如下：

name	version	port	config file
telegraf	1.11.0		/etc/telegraf/telegraf.conf
influxdb	1.7.6	8086 HTTP API 8088 RPC 8083 web	/etc/influxdb/influxdb.conf
chronograf	1.7.12	8888 web
kapacitor	1.5.2	9092 HTTP API	/etc/kapacitor/kapacitor.conf

2．     docker安装

参考https://docs.influxdata.com/kapacitor/v1.5/introduction/install-docker/部署，https://docs.influxdata.com/downloads/tik-docker-tutorial.tar.gz下载docker-compose相关文件。

本部署方案中包括三个组件influxdb、telegraf和kapacitor，不包含chronograf。

三．  influxdb使用

有三种方式可访问influxdb：命令行（command line interface），库（client library）和HTTP API。

命令行

influxDB安装包默认包含cli工具influx，可直接输入命令启动influx：

$influx
$ influx -precision rfc3339
Connected to http://localhost:8086 version 1.7.6
InfluxDB shell version: 1.7.
Enter an InfluxQL query
> help
Usage:
        connect <host:port>   connects to another node specified by host:port
        auth                  prompts for username and password
        pretty                toggles pretty print for the json format
        chunked               turns on chunked responses from server
        chunk size <size>     sets the size of the chunked responses.  Set to  to reset to the default chunked size
        use <db_name>         sets current database
        format <format>       specifies the format of the server responses: json, csv, or column
        precision <format>    specifies the format of the timestamp: rfc3339, h, m, s, ms, u or ns
        consistency <level>   sets write consistency level: any, one, quorum, or all
        history               displays command history
        settings              outputs the current settings for the shell
        clear                 clears settings such as database or retention policy.  run 'clear' for help
        exit/quit/ctrl+d      quits the influx shell

        show databases        show database names
        show series           show series information
        show measurements     show measurement information
        show tag keys         show tag key information
        show field keys       show field key information

        A full list of influxql commands can be found at:
        https://docs.influxdata.com/influxdb/latest/query_language/spec/
> show databases
name: databases
name
----
telegraf
_internal
> use telegraf
Using database telegraf
> show measurements
name: measurements
name
----
cpu
disk
diskio
kernel
mem
processes
swap
system
> show series
key
---
cpu,cpu=cpu-total,host=ubuntu-wang
cpu,cpu=cpu0,host=ubuntu-wang
cpu,cpu=cpu1,host=ubuntu-wang
disk,device=sda1,fstype=ext4,host=ubuntu-wang,mode=rw,path=/
disk,device=sda1,fstype=ext4,host=ubuntu-wang,mode=rw,path=/var/lib/kubelet
disk,device=sda1,fstype=ext4,host=ubuntu-wang,mode=rw,path=/var/lib/rancher/volumes
diskio,host=ubuntu-wang,name=loop0
diskio,host=ubuntu-wang,name=sda
diskio,host=ubuntu-wang,name=sda1
diskio,host=ubuntu-wang,name=sda2
diskio,host=ubuntu-wang,name=sda5
kernel,host=ubuntu-wang
mem,host=ubuntu-wang
processes,host=ubuntu-wang
swap,host=ubuntu-wang
system,host=ubuntu-wang
> select * from cpu limit
name: cpu
time                 cpu       host        usage_guest usage_guest_nice usage_idle        usage_iowait        usage_irq usage_nice usage_softirq       usage_steal usage_system       usage_user
----                 ---       ----        ----------- ---------------- ----------        ------------        --------- ---------- -------------       ----------- ------------       ----------
--24T07::20Z cpu-total ubuntu-wang                            95.42682926828536 0.25406504065046054                    0.20325203252035398            1.8800813008130035 2.235772357723244
--24T07::20Z cpu0      ubuntu-wang                            95.24291497978086 0.20242914979758425                    0.3036437246964483             1.8218623481786898 2.429149797571011

常用命令：

help：显示帮助信息及命令

show databases：显示所有数据库

use <db_name>：设置当前数据库

show measurements：显示数据库中所有表（measurements）

insert：插入数据

select：查询数据

exit：退出命令行

注：数据插入查询可使用SQL语句完成，可参考https://docs.influxdata.com/influxdb/v1.7/query_language/data_exploration/

参考：

1． https://docs.influxdata.com/   https://github.com/influxdata/docs.influxdata.com

2． https://docs.influxdata.com/influxdb/v1.7/concepts/key_concepts/ 基础概念

3． https://docs.influxdata.com/influxdb/v1.7/tools/api/   HTTP API

4． https://docs.influxdata.com/influxdb/v1.7/query_language/data_exploration/  SELECT

5． https://jasper-zhang1.gitbooks.io/influxdb/content/  中文文档

6． https://repos.influxdata.com/  不同系统安装包下载

7． Using InfluxDB in Grafana