Fork me on GitHub
Suzf  Blog

Tag elasticsearch

How-to use rsyslog parse log to json format and then store in ES

前言

本文的主要目的是将 Apache 的访问日志转换成 JSON 格式,然后存储到 ES, 供 ELK Stack Platform 进行数据分析与统计。 尽管现在许多发行版本都已经默认安装上了 rsyslog, 但是还是推荐从 rsyslog repositories 获取最新的稳定版本。这样你将从中获益。你将需要下面的软件包:

  • rsyslog-mmnormalize. This gives you mmnormalize, a module that will do the parsing of common Apache logs to JSON
  • rsyslog-elasticsearch, for the Elasticsearch output

让我现在就开始配置吧。需要执行以下操作:

  1. 加载所需模块
  2. 将 Apache log 传送到中心日志服务器
  3. 配置主队列缓冲您的消息。 这也是定义工作线程数和批量大小的地方(也可以是 Elasticsearch 块的大小)
  4. 将 Apache log 转换成 JSON
  5. 定义一个模板,您可以在其中指定 JSON 信息的格式。 您可以使用此模板通过Elasticsearch输出将日志发送到 Elasticsearch / Logstash

[译] How-to Send Rsyslog directly to Elasticsearch

Originally posted on the server as codeRsyslog to Elasticsearch

长时间以来,系统管理部门已经知道,集中日志很重要,无论是故障排除还是出于安全考虑。 在我看来,我不仅要集中日志,还要让它们可以搜索。 (在集中式日志文件上的 grep 不是很强大,但这不是我正在寻找的解决方案。)

Elasticsearch

... is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

它提供的几个基础的功能有:存储/索引/搜索 数据。它是典型 ELK Stack(Elasticsearch, Logstash, and Kibana )的一部分。 作为系统管理员,我认为还需要注意的是,当用作集群时,Elasticsearch是分布式系统。 不应该轻率地添加另一个分布式系统。 作为一个例子,我现在有了生产分布式系统的MySQL Galera,RabbitMQ,Nomad和Elasticsearch。

Structured Logging with Rsyslog and Elasticsearch

Article: Structured Logging with Rsyslog and Elasticsearch Author: Radu Gheorghe

As more and more organizations are starting to use our Performance Monitoring and Search Analytics services, we have more and more logs from various components that make up these applications.  So what do we do?  Do we just keep logging everything to files, rotating them, and grepping them when we need to troubleshoot something?  There must be something better we can do!  And indeed, there is – so much so, that we’ll soon be launching Logsene – a Log Analytics service to complement SPM.  When your applications generate a lot of logs, you’d probably want to make some sense of them by searching and/or statistics. Here’s when structured logging comes in handy, and I would like to share some thoughts and configuration examples of how you could use a popular syslog daemon like rsyslog to handle both structured and unstructured logs. Then I’m going to look at how you can take those logs, format them in JSON, and index them with Elasticsearch – for some fast and easy searching and statistics.  If you are going to Berlin Buzzwords this year and you are into logging, Logstash, ElasticSearch, or Kibana, I’ll be talking about them in my JSON logging with ElasticSearch presentation.

How-to Install Web Admin Tool On Elasticsearch 5

Head Plugin

Running as a plugin of Elasticsearch (deprecated)

for Elasticsearch 5.x: site plugins are not supported. Run as a standalone server for Elasticsearch 2.x: sudo elasticsearch/bin/plugin install mobz/elasticsearch-head open http://localhost:9200/_plugin/head/

How-to Install Elasticsearch 5.x Cluster On CentOS7

前言

ES 群集的部署可谓是“傻瓜式”的,需要自定义的地方就是 `/etc/elasticsearch/elasticsearch.yml` 里的 cluster.name。然后,在 `Disicovery` 可达的范围内,所有的 elasticsearch node 会自动寻找和自己相同的cluster.name 的兄弟, 然后按照最朴素的先来后到的规则确定master。至此集群创建完成。

节点类型

当我们启动Elasticsearch的实例,就会启动至少一个节点。相同集群名的多个节点的连接就组成了一个集群,在默认情况下,集群中的每个节点都可以处理http请求和集群节点间的数据传输,集群中所有的节点都知道集群中其他所有的节点,可以将客户端请求转发到适当的节点。节点有以下类型:

  • 主(master)节点:在一个节点上当node.master设置为True(默认)的时候,它有资格被选作为主节点,控制整个集群。
  • 数据(data)节点:在一个节点上node.data设置为True(默认)的时候。该节点保存数据和执行数据相关的操作,如增删改查,搜索,和聚合。
  • 客户端节点:当一个节点的node.master和node.data都设置为false的时候,它既不能保持数据也不能成为主节点,该节点可以作为客户端节点,可以响应用户的情况,并把相关操作发送到其他节点。
  • 部落节点: 当一个节点配置tribe.*的时候,它是一个特殊的客户端,它可以连接多个集群,在所有连接的集群上执行搜索和其他操作。

默认情况下,节点配置是一个主节点和一个数据节点。这是非常方便的小集群,但随着集群的发展,分离主节点和数据节点将变得很重要。

How-to Install elasticsearch with RPM on CentOS7

The RPM for Elasticsearch can be downloaded from our website or from our RPM repository. It can be used to install Elasticsearch on any RPM-based system such as OpenSuSE, SLES,
Centos, Red Hat, and Oracle Enterprise.
The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.
RPM install is not supported on distributions with old versions of RPM, such as SLES 11 and CentOS 5. Please see Install Elasticsearch with .zip or .tar.gz instead. Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Install Java

# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm"
# rpm -Uvh jdk-8u65-linux-x64.rpm
# java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

Import the Elasticsearch PGP Key

Download and install the public signing key:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Installing from the RPM repository

cat > /etc/yum.repos.d/elasticsearch.repo << EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum install elasticsearch

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
Elasticsearch can be started and stopped as follows:
sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service
When systemd logging is enabled, the logging information are available using the journalctl commands: To tail the journal:
sudo journalctl -f
To list journal entries for the elasticsearch service:
sudo journalctl --unit elasticsearch
To list journal entries for the elasticsearch service starting from a given time:
sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"
Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:
# ss -ntlp | grep 9200
LISTEN     0      128       ::ffff:127.0.0.1:9200                    :::*                   users:(("java",pid=1287,fd=130))
LISTEN     0      128        ::1:9200                    :::*                   users:(("java",pid=1287,fd=128))
# curl -XGET '127.0.0.1:9200/?pretty'
{
  "name" : "ZlbJlUp",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "pm-y-OxIT2qFutEF4cxRUQ",
  "version" : {
    "number" : "5.3.0",
    "build_hash" : "3adb13b",
    "build_date" : "2017-03-23T03:31:50.652Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}
Link: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html

Elasticsearch 节点类型介绍

摘要
在Elasticsearch中节点可以分为主(master)节点,数据(data)节点,客户端节点和部落节点,每种类型的节点有不同的使用方法,对于一个大的集群中,合理的配置这些属性,对集群的健壮性和性能有很大的帮助。
节点类型
当我们启动Elasticsearch的实例,就会启动至少一个节点。相同集群名的多个节点的连接就组成了一个集群,在默认情况下,集群中的每个节点都可以处理http请求和集群节点间的数据传输,集群中所有的节点都知道集群中其他所有的节点,可以将客户端请求转发到适当的节点。节点有以下类型:
  • 主(master)节点:在一个节点上当node.master设置为True(默认)的时候,它有资格被选作为主节点,控制整个集群。
  • 数据(data)节点:在一个节点上node.data设置为True(默认)的时候。该节点保存数据和执行数据相关的操作,如增删改查,搜索,和聚合。
  • 客户端节点:当一个节点的node.master和node.data都设置为false的时候,它既不能保持数据也不能成为主节点,该节点可以作为客户端节点,可以响应用户的情况,并把相关操作发送到其他节点。
  • 部落节点: 当一个节点配置tribe.*的时候,它是一个特殊的客户端,它可以连接多个集群,在所有连接的集群上执行搜索和其他操作。
默认情况下,节点配置是一个主节点和一个数据节点。这是非常方便的小集群,但随着集群的发展,分离主节点和数据节点将变得很重要。

How-to: install elasticsearch plugin

elasticsearch 插件

由于公司内部访问权限控制严格,自己搭建的虚拟机只能通过搭建代理上网

因为某种限制第一种安装未成功, 所以有了后面的方法。
自动安装
[11:38:08][root@ocean-lab elasticsearch]$ ./bin/plugin -install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip...
Failed to install mobz/elasticsearch-head, reason: failed to download out of all possible locations..., use --verbose to get detailed information

[11:35:35][root@ocean-lab elasticsearch]$ wget https://github.com/mobz/elasticsearch-head/archive/master.zip
--2015-11-26 11:35:56--  https://github.com/mobz/elasticsearch-head/archive/master.zip
Connecting to x.x.9.158:3128... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://codeload.github.com/mobz/elasticsearch-head/zip/master [following]
--2015-11-26 11:36:05--  https://codeload.github.com/mobz/elasticsearch-head/zip/master
Connecting to x.x.9.158:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 899159 (878K) [application/zip]
Saving to: “master.zip”

100%[==================================================
2015-11-26 11:36:10 (292 KB/s) - “master.zip” saved [899159/899159]

手动安装
[16:12:12][root@ocean-lab elasticsearch]$ ./bin/plugin --install elasticsearch-head --url file:///usr/share/elasticsearch/plugins/master.zip
-> Installing elasticsearch-head...
Trying file:/usr/share/elasticsearch/plugins/master.zip...
Downloading .........DONE
Installed elasticsearch-head into /usr/share/elasticsearch/plugins/head

Reference  elasticsearch-modules-plugins-install