Suzf Blog

前言 我想使用 RHEL/CentOS 系统自带的日志工具Rsyslog, 根据程序日志类别动态存储到一个以程序名命名的文件。

前言

日志中心化给我们带来了好多优势， 那么我们如何收集 Nginx 日志到日志服务器呢？ Nginx 自 1.7.1 版本之后开始支持 syslog.

Originally posted on Centralized logging for fun and profit

Setting up a centralized log server using syslog isn't as hard as many may believe. Whether it's logs from Apache, nginx, email services, or even from your own Python applications having a central log server gives you many benefits:

Benefits to a centralized logs

• Reduces disk space usage and disk I/O on core servers that should be busy doing something else. This is especially true if you want to log all queries to your database. Doing this on the same disk as your actual database creates a write for every read and an extra write for every write.
• Removes logs from the server in the event of an intrusion or system failure. By having the logs elsewhere you at least have a chance of finding something useful about what happened.
• All of your logs are in one place, duh! This makes things like grepping through say Apache error logs across multiple webservers easier than bouncing around between boxes. Any log processing and log rotation can also be centralized which may delay your sysadmin from finally snapping and killing everyone.

前言 本文主要简述的如何使用 Rsyslog 实现日志的中心化，并利用 ELK Stack 实现图形搜索与数据统计。

基本日志记录教程

日志是跟踪一些软件运行时发生的事件的手段。软件的开发人员添加日志调用到他们的代码中，以指示已发生的某些事件。一个事件是通过一个描述性消息可任选地含有可变数据（即是该事件的每次发生潜在不同的数据）中。事件是很重要的，开发者通常通过事件追踪问题， 重要性也可称为水平或严重程度。

什么时候使用 logging

日志提供了简单的日志使用一组方便的功能。这里有 debug()，info()，warning()，error()和 critical()。要确定何时使用日志记录，请参阅下表，其中规定，对于一组常见任务，使用最好的工具。

ELK(Elasticsearch + Logstash + Kibana) 是一套开源的日志管理方案
Elasticsearch：负责日志检索和分析
Logstash：负责日志的收集，处理和储存
Kibana：负责日志的可视化

Logstash: The server component of Logstash that processes incoming logs
Elasticsearch: Stores all of the logs
Kibana 4: Web interface for searching and visualizing logs, which will be proxied through Nginx
Logstash Forwarder: Installed on servers that will send their logs to Logstash, Logstash Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to communicate with Logstash

Reference：
JDK - http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch - https://www.elastic.co/downloads/elasticsearch
Logstash - https://www.elastic.co/downloads/logstash
Kibana - https://www.elastic.co/downloads/kibana
redis - http://redis.io/download

数据流流向如下
Logstash-forwarder--->Logstash--->Elasticsearch--->kibana--->nginx--->客户浏览器