Fork me on GitHub
Suzf  Blog

How-to Installation and configuration ELK Stack 5

前言

本文主要讲述的是 ELK Stack 5 的简易安装和配置。 Elasticsearch - 分布式的文档(document)存储引擎。它可以实时存储并检索复杂数据结构。 Kibana - 在 Elasticsearch 中分析和搜索数据信息的 bashboard Filebeat - 为 Elasticsearch 传送日志数据的轻量级的组件(基于  Logstash-Forwarder)。 关于 Elasticsearch 5 的安装 --> How-to Install elasticsearch with RPM on CentOS7

How-to Install Elasticsearch 5.x Cluster On CentOS7

前言

ES 群集的部署可谓是“傻瓜式”的,需要自定义的地方就是 `/etc/elasticsearch/elasticsearch.yml` 里的 cluster.name。然后,在 `Disicovery` 可达的范围内,所有的 elasticsearch node 会自动寻找和自己相同的cluster.name 的兄弟, 然后按照最朴素的先来后到的规则确定master。至此集群创建完成。

节点类型

当我们启动Elasticsearch的实例,就会启动至少一个节点。相同集群名的多个节点的连接就组成了一个集群,在默认情况下,集群中的每个节点都可以处理http请求和集群节点间的数据传输,集群中所有的节点都知道集群中其他所有的节点,可以将客户端请求转发到适当的节点。节点有以下类型:

  • 主(master)节点:在一个节点上当node.master设置为True(默认)的时候,它有资格被选作为主节点,控制整个集群。
  • 数据(data)节点:在一个节点上node.data设置为True(默认)的时候。该节点保存数据和执行数据相关的操作,如增删改查,搜索,和聚合。
  • 客户端节点:当一个节点的node.master和node.data都设置为false的时候,它既不能保持数据也不能成为主节点,该节点可以作为客户端节点,可以响应用户的情况,并把相关操作发送到其他节点。
  • 部落节点: 当一个节点配置tribe.*的时候,它是一个特殊的客户端,它可以连接多个集群,在所有连接的集群上执行搜索和其他操作。

默认情况下,节点配置是一个主节点和一个数据节点。这是非常方便的小集群,但随着集群的发展,分离主节点和数据节点将变得很重要。

How-to Install elasticsearch with RPM on CentOS7

The RPM for Elasticsearch can be downloaded from our website or from our RPM repository. It can be used to install Elasticsearch on any RPM-based system such as OpenSuSE, SLES,
Centos, Red Hat, and Oracle Enterprise.
The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.
RPM install is not supported on distributions with old versions of RPM, such as SLES 11 and CentOS 5. Please see Install Elasticsearch with .zip or .tar.gz instead. Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Install Java

# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm"
# rpm -Uvh jdk-8u65-linux-x64.rpm
# java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

Import the Elasticsearch PGP Key

Download and install the public signing key:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Installing from the RPM repository

cat > /etc/yum.repos.d/elasticsearch.repo << EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum install elasticsearch

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
Elasticsearch can be started and stopped as follows:
sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service
When systemd logging is enabled, the logging information are available using the journalctl commands: To tail the journal:
sudo journalctl -f
To list journal entries for the elasticsearch service:
sudo journalctl --unit elasticsearch
To list journal entries for the elasticsearch service starting from a given time:
sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"
Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:
# ss -ntlp | grep 9200
LISTEN     0      128       ::ffff:127.0.0.1:9200                    :::*                   users:(("java",pid=1287,fd=130))
LISTEN     0      128        ::1:9200                    :::*                   users:(("java",pid=1287,fd=128))
# curl -XGET '127.0.0.1:9200/?pretty'
{
  "name" : "ZlbJlUp",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "pm-y-OxIT2qFutEF4cxRUQ",
  "version" : {
    "number" : "5.3.0",
    "build_hash" : "3adb13b",
    "build_date" : "2017-03-23T03:31:50.652Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.1"
  },
  "tagline" : "You Know, for Search"
}
Link: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html

How-to set up and use graphite on Ubuntu

Graphite 是一个Python写的web应用,采用django框架,Graphite用来进行收集服务器所有的即时状态,用户请求信息,Memcached命中率,RabbitMQ消息服务器的状态,Unix操作系统的负载状态,Graphite服务器大约每分钟需要有4800次更新操作,Graphite采用简单的文本协议和绘图功能可以方便地使用在任何操作系统上。 Graphite 仅是一个画图工具,不主动地收集数据,而是将接收到的数据以图形的方式展现出来。 因为Graphite不收集数据,所以需要将指定格式的数据发送给Graphite,其在将受到的数据,绘制成图像。 Graphite的功能

  • 存储时间序列数据
  • 根据需要呈现数据的图形

Graphite 三大组件:

  1. Graphite-Web, a Django-based web application that renders graphs and dashboards
  2. The Carbon metric processing daemons
  3. The Whisper time-series database library

Graphite Overview

What Graphite is and is not

Graphite does two things:
  1. Store numeric time-series data
  2. Render graphs of this data on demand
What Graphite does not do is collect data for you, however there are some tools out there that know how to send data to graphite. Even though it often requires a little code, sending data to Graphite is very simple.

About the project

Graphite is an enterprise-scale monitoring tool that runs well on cheap hardware. It was originally designed and written by Chris Davis at Orbitz in 2006 as side project that ultimately grew to be a foundational monitoring tool. In 2008, Orbitz allowed Graphite to be released under the open source Apache 2.0 license. Since then Chris has continued to work on Graphite and has deployed it at other companies including Sears, where it serves as a pillar of the e-commerce monitoring system. Today many large companies use it.

The architecture in a nutshell

Graphite consists of 3 software components:
  1. carbon - a Twisted daemon that listens for time-series data
  2. whisper - a simple database library for storing time-series data (similar in design to RRD)
  3. graphite webapp - A Django webapp that renders graphs on-demand using Cairo
Feeding in your data is pretty easy, typically most of the effort is in collecting the data to begin with. As you send datapoints to Carbon, they become immediately available for graphing in the webapp. The webapp offers several ways to create and display graphs including a simple URL API for rendering that makes it easy to embed graphs in other webpages.
Source http://graphite.readthedocs.io/en/latest/overview.html

How-to Deploy Django Apps with uWSGI and Nginx on Ubuntu

Django 的部署可以有很多方式,采用 nginx + uwsgi 的方式是其中比较常见的一种方式。

准备工作

安装所需软件

pip install Django
apt-get install nginx

pip install uwsgi
# 注: 在 Debian/Ubuntu 系统中需要安装 python-dev

基本测试

创建一个 test.py 的文件

# test.py
def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return [b"Hello World"] # python3
    #return ["Hello World"] # python2

运行 uWSGI

uwsgi --http :8000 --wsgi-file test.py

参数含义: http :8000: 使用http协议 8000端口 wsgi-file test.py: 加载指定文件 test.py 执行 Curl 命令在终端上返回 Hello world

$curl http://172.16.9.110:8000
Hello World

如果返回的是正确的内容,说明下面工作流程的组件是工作的:

the web client <-> uWSGI <-> Python

测试你的 Django 项目 确保你的项目是可以顺利运行的

python manage.py runserver 0.0.0.0:8000

如果没有报错,使用 uWSGI 的方式运行它

uwsgi --http :8000 --wsgi-file mysite.wsgi

通过浏览器访问正常, 说明下面工作流程的组件是工作的:

the web client <-> uWSGI <-> Django

配置 Nginx

# mysite_nginx.conf

# the upstream component nginx needs to connect to
upstream django {
    server unix:///path/to/your/mysite/mysite.sock; # for a file socket
    # server 127.0.0.1:8001; # for a web port socket (we'll use this first)
}

# configuration of the server
server {
    # the port your site will be served on
    listen      8000;
    # the domain name it will serve for
    server_name .example.com; # substitute your machine's IP address or FQDN
    charset     utf-8;

    # max upload size
    client_max_body_size 75M;   # adjust to taste

    # Django media
    location /media  {
        alias /path/to/your/mysite/media;  # your Django project's media files - amend as required
    }

    location /static {
        alias /path/to/your/mysite/static; # your Django project's static files - amend as required
    }

    # Finally, send all non-media requests to the Django server.
    location / {
        uwsgi_pass  django;
        include     /path/to/your/mysite/uwsgi_params; # the uwsgi_params file you installed
    }
}

Running the Django application with uswgi and nginx

如果上面一切都显示正常,则下面命令可以拉起django application

uwsgi --socket mysite.sock --wsgi-file mysite.wsgi --chmod-socket=664

Configuring uWSGI to run with a .ini file

每次都运行上面命令拉起django application实在麻烦,使用.ini文件能简化工作,方法如下: 在application目录下创建文件mysite_uwsgi.ini,填入并修改下面内容:

# mysite_uwsgi.ini file
[uwsgi]

# Django-related settings
# the base directory (full path)
chdir           = /path/to/your/project
# Django's wsgi file
module          = project.wsgi
# the virtualenv (full path)
home            = /path/to/virtualenv

# process-related settings
# master
master          = true
# maximum number of worker processes
processes       = 10
# the socket (use the full path to be safe
socket          = /path/to/your/project/mysite.sock
# ... with appropriate permissions - may be needed
# chmod-socket    = 664
# clear environment on exit
vacuum          = true

之后使用这个文件运行 uwsgi

uwsgi --ini mysite_uwsgi.ini # the --ini option is used to specify a file

Once again, test that the Django site works as expected.

参考链接

[0] http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html

[1] http://www.jianshu.com/p/e6ff4a28ab5a

How-to Configure Consul in a Production Environment on Ubuntu

什么是 Consul? Consul有多个组件,但作为一个整体,它是用于发现和配置基础架构中的服务的工具。它提供几个关键特性: 服务发现:Consul的客户端可以提供一个服务,如api或mysql,其他客户端可以使用Consul来发现给定服务的提供者。使用DNS或HTTP,应用程序可以轻松找到它们所依赖的服务。 健康检查:Consul 客户端可以提供与给定服务(“web服务器返回200 OK”)或本地节点(“内存利用率低于90%”)相关联的任何数量的运行状况检查。操作员可以使用此信息来监视群集运行状况,并且服务发现组件使用此信息来将流量路由到不正常的主机。 键/值存储:应用程序可以使用Consul的分层键/值存储用于任何用途,包括动态配置,功能标记,协调,领导选举等。简单的HTTP API使其易于使用。 多数据中心:Consul支持多个数据中心开箱即用。这意味着Consul的用户不必担心构建额外的抽象层以扩展到多个区域。 Consul对DevOps社区和应用程序开发人员设计的更为友好,使其成为完美的现代化,弹性基础设施。

How-to enable Maven on jenkins

Apache Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information.

Jenkins ver. 2.32.1

Step 1: Install JDK

官方站点 http://openjdk.java.net/install/

-- The openjdk-8-jre package contains just the Java Runtime Environment.

-- If you want to develop Java programs then please install the openjdk-8-jdk package.

# install jdk
sudo apt-get install openjdk-8-jdk -y
 
# make sure the jdk install success
java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
 
# now jdk installed, Let's add JAVA_HOME Env variable
cat > /etc/profile.d/java-8-openjdk.sh << EOF
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
PATH=\$JAVA_HOME/bin:\$PATH
export JAVA_HOME PATH
EOF
 
source /etc/profile.d/java-8-openjdk.sh

Step 2: Downloading and Setting Up Maven

maven 的官方下载站点 Apache Maven Download. 在我写这篇文章的时候,Maven 的最新版本是 3.3.9.

cd /usr/local/src
wget http://mirror.its.dal.ca/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
tar xf apache-maven-3.3.9-bin.tar.gz
mv apache-maven-3.3.9 /usr/local/


cat > /etc/profile.d/apache-maven.sh << EOF
MAVEN_HOME=/usr/local/apache-maven-3.3.9
export MAVEN_HOME
export PATH=\${PATH}:\${MAVEN_HOME}/bin
EOF
 
source /etc/profile.d/apache-maven.sh

# 查看 maven 版本
# mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T05:41:47-11:00)
Maven home: /usr/local/apache-maven-3.3.9
Java version: 1.8.0_111, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-57-generic", arch: "amd64", family: "unix"

Step 3: Setting up Jenkins  JDK and Maven

登陆 Jenkins dashboard,点击 系统管理  -->  Global Tool Configuration -- > JDK 安装 jenkins_jdk --> Maven安装 jenkins_maven 最后保存配置

Step 4: Install Maven Plugin

参见 How-to install jenkins plugin via manual installation

cd /var/lib/jenkins/plugins
wget http://updates.jenkins-ci.org/download/plugins/javadoc/1.4/javadoc.hpi
wget http://updates.jenkins-ci.org/download/plugins/maven-plugin/2.14/maven-plugin.hpi
/etc/init.d/jenkins  restart

Now You can create Maven Project

hello_jenkins_maven ~~~ EOF ~~~