基于Kibana的可视化监控报警插件sentinl入门

sentinl是什么

Kibi/Kibana Alert & Reporting App
Watching your data, 24/7/365

sentinl是一个免费的kibana预警与报告插件，与付费软件X-Pack功能类似。

Some Examples for illustration:

HIT COUNT PER HOUR
QUESTION: How many hits does index X receive hourly?
WATCHER: query index and return count of hits in last hour
ACTION: Notify with number of Hits per hour
METRIC THRESHOLDS
QUESTION: Is any of my monitored metrics surpassing a certain value?
WATCHER: query index and type for specific values, aggregated by an arbitrary field.
ACTION: Notify with aggs bucket details every time a threshold is surpassed or spike anomaly detected.
BLACKLISTS HITS
QUESTION: Is any of my users trying to reach blacklisted destinations?
WATCHER: query firewall logs comparing destination IPs to a blacklist.
ACTION: Notify admin via email if any IP >= 10 matches returned
FAILED LOGINS
QUESTION: Are there recurring failure attempts authenticating users on my network?
WATCHER: query active directory logs for login failures in last hour and compare to user index. .
ACTION: Notify admin via webhook if >= 10 matches returned
LEAK DETECTION (chain)
QUESTION: Are there any public leaks about my data I was not aware of?
WATCHER: query for user emails included in published leaks ingested from third parties.
ACTION: Save hits in secondary result Index. Notify via email if leak was not known in a secondary Watcher

安装

./kibana-plugin install file:./sentinl-v6.0.1.zip

安装完成后，要重启kibana

fuser -n tcp 5601

ps -ef | grep node

kill -9 pid

./kibana &

使用步骤

使用包括5个步骤

Step 1: New Watcher

give our Watcher a name and choose an execution frequency

Step 2: Input Query

es的搜索与聚合

Step 3: Condition

validate if the results received back are worth processing

语法与x-pack script condition语法类似

相当于过滤条件

"condition": {

      "script": {

        "script": "payload.hits.total>=1"  //当报警条件为***出现的次数大于1

      }

    }  

"condition": {

      "script": {

        "script": "payload.hits.hits[0]._source.responsetime > 0.01" // 检索条件 响应时间大于 0.01秒

      }

    }

Step 4: Transform

Our data might need adjustments or post processing. Process our payload using a javascript expression/script

事后处理

Step 5: Actions

Let's form a notification using the mustache templating language。

可以采用多种方式发送通知。

transform

How to Adapt or Post-Process data

Post Process事后的处理。

The transform script is the wild member of the family and can be used to inject simple or complex logic into the pipeline before delivery to actions using pure javascript.

From converting format types, through generating brand new payload keys and interpolating data, transform is the way up. The script expects a boolean condition to trigger actions. A false condition can be forced to stop the execution. BONUS: Transforms can be saved and used across Watchers! "transform": { "script": { "script": "payload.newvar = payload.aggs.some.values['95.0']" } }

action举例之邮件发送

Troubleshooting

kibana.yml

logging.verbose: true

sentinl:

  settings:

    email:

      active: true

      host: smtp.exmail.qq.com

      ssl: false

    report:

      active: true

      tmp_path: /tmp/

上面是官网的，下面是实践已OK

sentinl:

  settings:

    email:

      active: true

      user: tanyk@huawangtech.com

      password: Dd@2016

      host: smtp.exmail.qq.com

      ssl: true

      timeout: 10000

    report:

      active: true

      tmp_path: /tmp/

先测试

mailx -S smtp=<smtp-server-address> -r <from-address> -s <subject> -v <to-address> < body.txt

yum -y install sendmail

yum install -y sendmail-cf

/etc/init.d/sendmail start

chkconfig sendmail on

yum install -y mailx

vim /etc/mail.rc（optional）

set from=tanyk@mail.com

set smtp=smtp.exmail.qq.com

set smtp-auth-user=tanyk@mail.com

set smtp-auth-password=******

set smtp-auth=login

set nss-config-dir="/etc/pki/nssdb/"

test

echo "This is the message body and contains the message" | mailx -v -r "tanyk@mail.com" -s "This is the subject" -S smtp="smtp.exmail.qq.com" -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user="tanyk@mail.com" -S smtp-auth-password="******" -S ssl-verify=ignore -S nss-config-dir="/etc/pki/nssdb/" tanyk@163.com

参考文献