ELK监控系统nginx / mysql慢日志

ELK监控系统nginx / mysql慢日志

elasticsearch

logstash

kibana

ELK监控系统nginx日志

1.环境准备

centos6.8_64 mini

IP:192.168.10.78

tar包：

logstash-2.4.0.tar.gz

elasticsearch-2.4.0.tar.gz

kibana-4.6.1-linux-x86_64.tar.gz

JDK环境

elasticsearch	logstach	kibana	java
V2.4	V2.4	V4.6.1	V1.8.0_111

nginx采用yum安装即可，安装后启动此服务。

2.logstash简单配置编写

logstash目录下新建测试文件test.conf 其内动如下

1. input { 
   

   ---

2. file { 
   

   ---

3. type => "syslog" 
   

   ---

4. tags => ["log"] 
   

   ---

5. path => ["/var/log/messages","/log/*.log"] 
   

   ---

6. start_position => beginning 
   

   ---

7. ignore_older => 0 
   

   ---

8. } 
   

   ---

9. file { 
   

   ---

10. type => "nginx_log" 
    

    ---

11. tags => ["nginx"] 
    

    ---

12. path => ["/var/log/nginx/access.log"] 
    

    ---

13. start_position => beginning 
    

    ---

14. ignore_older => 0 
    

    ---

15. } 
    

    ---

16. } 
    

    ---

17. ---

18. ---

19. output 
    

    ---

20. { 
    

    ---

21. ---

22. elasticsearch { 
    

    ---

23. hosts => [ "192.168.10.78:9200" ] 
    

    ---

24. } 
    

    ---

25. ---

26. } 
    

    ---

编写好后测试运行：

1. [root@localhost logstash]# ./bin/logstash -f test.conf  
   

   ---

2. Settings: Default pipeline workers: 2 
   

   ---

3. Pipeline main started 
   

   ---

此时logstash已经运行，我们通过浏览器访问web：192.168.10.68

同时在终端下重启网卡服务，用以查看系统日志是否有输出至kibana

打开kibana查看日志输出

测试kibana.png

可以看见有网卡日志和web访问日志。

接下来监控mysql日志

yum install mysql mysql-server -y

安装完直接在kibana就看见安装mysql的日志输出结果了

Time    type    tags    message
November 10th 2016, 10:18:13.193    syslog  log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:13.057    syslog  log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.187    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.185    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64
November 10th 2016, 10:18:10.183    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64
November 10th 2016, 10:18:10.051    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.050    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64
November 10th 2016, 10:18:10.048    syslog  log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64

接下来修改test.conf配置文件，添加对mysql满日志的监控

1.yum install mysql mysql-server

2.service network restart

3.mysql慢日志开启:

(此处参考http://blog.itpub.net/29500582/viewspace-1432985/)

Mysql 启动慢查询日志 （不用重启）

1. 查看mysql系统参数

mysql> show variables like "%slow%";

+---------------------------+-------------------------------+

| Variable_name | Value |

+---------------------------+-------------------------------+

| log_slow_admin_statements | OFF |

| log_slow_slave_statements | OFF |

| slow_launch_time | 2 |

| slow_query_log | OFF |

| slow_query_log_file | /mysql/data/nagiosdb-slow.log |

+---------------------------+-------------------------------+

5 rows in set (0.00 sec)

slow_query_log： off关闭状态 on开启状态

slow_launch_time 默认超过2s为慢查询

slow_query_log_file 慢查询日志存放地点

这三个参数，在不同的mysql版本中，不太一样，不过都可以通过 show variables like "%slow%" 查看出来

1. 运行如下命令即可运行慢查询日志

mysql> set global slow_query_log=ON;

Query OK, 0 rows affected (0.03 sec)

mysql> set global slow_launch_time=5;

Query OK, 0 rows affected (0.00 sec)

mysql> show variables like "%slow%";

+---------------------------+-------------------------------+

| Variable_name | Value |

+---------------------------+-------------------------------+

| log_slow_admin_statements | OFF |

| log_slow_slave_statements | OFF |

| slow_launch_time | 5 |

| slow_query_log | ON |

| slow_query_log_file | /mysql/data/nagiosdb-slow.log |

+---------------------------+-------------------------------+

5 rows in set (0.00 sec)

mysql 5.1.6版本起，slow_query_log 和 slow_launch_time 支持写文件或写数据库表两种方式，并且日志的开启，输出方式的修改，都可以在global级别动态修改。

只需简单通过set global slow_query_log=ON;即可开启慢查询，而不需要重启数据库！

1. 可以直接写到配置文件中 my.cnf

slow_query_log_file=/mysql/log/nagiosdb-slow.log

slow_launch_time=5

可以完成配置！！

！！！根据上述配置开启慢日志后，查询本机日志名称及目录，不要搞错。

在test.conf配置文件添加如下：

1. # this is mysql log 
   

   ---

2. file { 
   

   ---

3. type => "mysql_log" 
   

   ---

4. tags => ["mysql"] 
   

   ---

5. path => ["/var/log/mysql/mysqld.log"] 
   

   ---

6. start_position => beginning 
   

   ---

7. ignore_older => 0 
   

   ---

8. } 
   

   ---

9. # this is mysql-slow log 
   

   ---

10. file { 
    

    ---

11. type => "mysql_slow" 
    

    ---

12. tags => ["mysql-slow"] 
    

    ---

13. path => ["/var/run/mysqld/mysqld-slow.log"] 
    

    ---

14. start_position => beginning 
    

    ---

15. ignore_older => 0 
    

    ---

16. } 
    

    ---

配置完后启动logstash

[root@localhost logstash]# ./bin/logstash -f test.conf

确认慢查询已开启

mysql> show variables like "%slow%";
+---------------------+---------------------------------+
| Variable_name       | Value                           |
+---------------------+---------------------------------+
| log_slow_queries    | ON                              |
| slow_launch_time    | 2                               |
| slow_query_log      | ON                              |
| slow_query_log_file | /var/run/mysqld/mysqld-slow.log |
+---------------------+---------------------------------+
4 rows in set (0.00 sec)

mysql>

输入测试命令，并查看kibana是否有mysql-low数据输出；

mysql> select sleep(6);
+----------+
| sleep(6) |
+----------+
|        0 |
+----------+
1 row in set (5.99 sec)
mysql>

慢数据输出正常：

!!!目前所有日志数据均未作规范化输出处理，只是简单测试了elk的基本功能，后续将升入研究学习elk.后面会继续做好笔记供自己参考学习。