Fork me on GitHub
Suzf Blog

Nginx+Redis+Tomcat实现session共享的集群

Jeffrey Linux Nginx tomcat redis

License: Attribution-NonCommercial-ShareAlike 4.0 International

本文出自 Suzf Blog。 如未注明，均为 SUZF.NET 原创。

转载请注明：http://suzf.net/post/632

Nginx 作为目前最流行的开源反向代理HTTP Server，用于实现资源缓存、web server负载均衡等功能，由于其轻量级、高性能、高可靠等特点在互联网项目中有着非常普遍的应用，相关概念网上有丰富的介绍。分布式web server集群部署后需要实现session共享，针对 tomcat 服务器的实现方案多种多样，比如 tomcat cluster session 广播、nginx IP hash策略、nginx sticky module等方案，本文主要介绍了使用 redis 服务器进行 session 统一存储管理的共享方案。

使 用Nginx作为Tomcat的负载平衡器，Tomcat的会话Session数据存储在Redis，能够实现0当机的7×24运营效果。因为将会话存储 在Redis中，因此Nginx就不必配置成stick粘粘某个Tomcat方式，这样才能真正实现后台多个Tomcat负载平衡，用户请求能够发往任何 一个tomcat主机，当我们需要部署新应用代码时，只要停止任何一台tomcat，所有当前在线用户都会导向到运行中的tomcat实例，因为会话数据 被序列化到Redis，在线用户不会受到影响，一旦停掉的tomcat实例上线，另外其他重复部署过程。

frame-20160121153429_41249
实验环境

      IP            Roles               port

node1 192.168.9.10  nginx,redis,tomcat  80,1111,8080

node2 192.168.9.70  tomcat              8080

软件工具包
apache-tomcat-7.0.68.tar
redis-3.0.7.tar
jdk-7u79-linux-x64.rpm
nginx-1.0.15-12.el6.x86_64.rpm

tomcat需要的jar包
commons-logging-1.1.3.jar
commons-pool2-2.2.jar
jedis-2.5.2.jar
tomcat-juli.jar
tomcat-redis-session-manage-tomcat7.jar
安装redis
1. Redis 安装
== Redis 源码安装
— 参照源码 README
— 默认端口 6379

*** 安装redis之前先要确认系统已经安装了GCC和libc库 ***

wget http://download.redis.io/releases/redis-3.0.7.tar.gz
tar xf redis-3.0.7.tar.gz
cd redis-3.0.7
make
# make test
make PREFIX=/usr/local/redis install

 

== 可执行命令

redis-server    Redis服务系统
redis-cli       Redis一个客户端管理工具
redis-benchmark 用来检测redis性能
redis-check-aof & redis-check-dump 用来处理损坏的数据文件
redis-sentinel -> redis-server   < soft link >

cd utils
./install_server.sh

^_^[15:50:34][root@master01 utils]#bash install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 1111
Please select the redis config file name [/etc/redis/1111.conf]
Selected default – /etc/redis/1111.conf
Please select the redis log file name [/var/log/redis_1111.log]
Selected default – /var/log/redis_1111.log
Please select the data directory for this instance [/var/lib/redis/1111]
Selected default – /var/lib/redis/1111
Please select the redis executable path [/usr/sbin/redis-server] /usr/local/redis/bin/redis-server
Selected config:
Port           : 1111
Config file    : /etc/redis/1111.conf
Log file       : /var/log/redis_1111.log
Data dir       : /var/lib/redis/1111
Executable     : /usr/local/redis/bin/redis-server
Cli Executable : /usr/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/1111.conf => /etc/init.d/redis_1111
Installing service…
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server…
Installation successful!

 

Redis 系统配置

echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
sysctl vm.overcommit_memory=1
echo 511 > /proc/sys/net/core/somaxconn

启动服务

/etc/init.d/redis_1111 start

配置文件可以用默认的配置，从服务器上配置文文件打开slavefo 加上主服务器的ip和端口，还有其他的参数如日志和dump.rdb保存路径可以适当修改

FAQ:

make时如果有类似如下报错：
Selecting a non-default memory allocator when building Redis is done by setting
the `MALLOC` environment variable. Redis is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.

make MALLOC=libc
安装 JDK and Tomcat

# wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" `curl -s http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html | awk -F'"' '/jdk\-7.*\-linux\-x64.rpm/{print $12;exit}'`

# rpm -ivh jdk-7u79-linux-x64.rpm

# cat > /etc/profile.d/java.sh << EOF

JAVA_HOME=/usr/java/latest

PATH=\$JAVA_HOME/bin:\$PATH

export JAVA_HOME PATH

EOF

# wget -c `curl -s "https://tomcat.apache.org/download-70.cgi" | awk -F'"' '/apache\-tomcat\-7.*.tar.gz/{print $2;exit}'`

# tar xf apache-tomcat-7.*.tar.gz -C /usr/local/

# ln -sv /usr/local/apache-tomcat-7.* /usr/local/tomcat

# cat > /etc/profile.d/tomcat.sh << EOF

export CATALINA_HOME=/usr/local/tomcat

export PATH=\$CATALINA_HOME/bin:\$PATH

EOF

# source /etc/profile.d/java.sh

# source /etc/profile.d/tomcat.sh

# catalina.sh version

Using CATALINA_BASE:   /usr/local/tomcat

Using CATALINA_HOME:   /usr/local/tomcat

Using CATALINA_TMPDIR: /usr/local/tomcat/temp

Using JRE_HOME:        /usr/java/latest

Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar

Server version: Apache Tomcat/7.0.68

Server built:   Feb 8 2016 20:25:54 UTC

Server number:  7.0.68.0

OS Name:        Linux

OS Version:     2.6.32-573.8.1.el6.x86_64

Architecture:   amd64

JVM Version:    1.7.0_79-b15

JVM Vendor:     Oracle Corporation

 

*** 将jar包全都拷进tomcat lib目录下 ***

^_^[14:43:41][root@master01 lib]#pwd
/usr/local/tomcat/lib
^_^[14:44:47][root@master01 lib]#ls -lth | tail -5
-rw-r--r-- 1 root root  61K Nov 13 15:18 commons-logging-1.1.3.jar
-rw-r--r-- 1 root root 106K Nov 13 15:18 commons-pool2-2.2.jar
-rw-r--r-- 1 root root 308K Nov 13 15:18 jedis-2.5.2.jar
-rw-r--r-- 1 root root  74K Nov 13 15:18 tomcat-juli.jar
-rw-r--r-- 1 root root  20K Nov 13 15:18 tomcat-redis-session-manage-tomcat7.jar

修改conf目录下context.xml配置文件，session共享保存设置
在<Context>标签中

<Valve className="com.orangefunction.tomcat.redissessions.RedisSessionHandlerValve" />
<Manager className="com.orangefunction.tomcat.redissessions.RedisSessionManager" 
host="192.168.9.10" 
port="1111" 
database="0" 
maxInactiveInterval="60"/>

添加测试页

vim webapps/ROOT/test.jsp

<%@ page language="java" %>

<html>

  <head><title>TomcatI</title></head>

  <body>

    <h1><font color="green">TomcatI.suzf.net</font></h1>

    <table align="centre" border="1">

      <tr>

        <td>Session ID</td>

    <% session.setAttribute("tomcat.suzf.net","tomcat.suzf.net"); %>

        <td><%= session.getId() %></td>

      </tr>

      <tr>

        <td>Created on</td>

        <td><%= session.getCreationTime() %></td>

     </tr>

    </table>

  </body>

</html>

将tomcat目录拷贝到其他tomcat服务器
将测试页代码 Titile 和 color 修改下，以作区分。

Nginx安装

yum install nginx -y 

Nginx 相关配置
# grep -v "^$\|^.*#"  /etc/nginx/nginx.conf

user              nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {

    worker_connections  1024;

}

http {

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for"';

    upstream suzf-net-tomcat {

       server 192.168.9.10:8080;
       server 192.168.9.70:8080;

    }

    server {

        listen 80;
        server_name tomcat.suzf.net;

        location / {

             index   index.jsp index.html index.htm;
             proxy_pass http://suzf-net-tomcat;
             proxy_redirect off;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        }

    }

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    server_tokens   off;
    keepalive_timeout  65;

    include /etc/nginx/conf.d/*.conf;

}

 

检测nginx 配置文件 & 重载配置文件

#/etc/init.d/nginx configtest 

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful

#/etc/init.d/nginx  reload

Reloading nginx:                                           [  OK  ]

浏览器验证：

tom1-20160318130409

tom2-20160318130451

 

 查看Redis 数据

redis-20160318130539

 

通过 tomcat-redis-session-manager  自动获取tomcat 所需 jar 包 

*** 未测 *** 

tomcat-redis-session-manager 是一个用来将 Tomcat 的 Session 数据存储在 Redis 库中的项目。

官网：https://github.com/jcoleman/tomcat-redis-session-manager

*** 由于源码构建基于 gradle，请先配置 gradle 环境。***

wget https://downloads.gradle.org/distributions/gradle-2.10-all.zip

unzip -q gradle-2.10-all.zip -d /usr/local/

ln -sv /usr/local/gradle-2.10 /usr/local/gradle

#echo "export PATH=/usr/local/gradle/bin:\$PATH" > /etc/profile.d/gradle2.10.sh

#source /etc/profile.d/gradle2.10.sh

 

安装 tomcat-redis-session-manager

git clone https://github.com/jcoleman/tomcat-redis-session-manager.git

cd tomcat-redis-session-manager/

mv build.gradle build.gradle.bak

vim build.gradle

#diff -ruN build.gradle*

--- build.gradle 2016-01-04 07:07:59.000000000 +0800

+++ build.gradle.old 2016-03-18 10:48:15.620704187 +0800

@@ -44,13 +44,8 @@

   archives sourcesJar

 }


-//signing {

-//  sign configurations.archives

-//}

-

-task copyJars(type: Copy) {

-  from configurations.runtime

-  into 'dist'  

+signing {

+  sign configurations.archives

 }


 uploadArchives {

@@ -58,9 +53,9 @@

     mavenDeployer {

       beforeDeployment { MavenDeployment deployment -> signing.signPom(deployment) }


-      //repository(url: "https://oss.sonatype.org/service/local/staging/deploy/maven2/") {

-      //  authentication(userName: sonatypeUsername, password: sonatypePassword)

-      //}

+      repository(url: "https://oss.sonatype.org/service/local/staging/deploy/maven2/") {

+        authentication(userName: sonatypeUsername, password: sonatypePassword)

+      }

       //repository(url: "https://oss.sonatype.org/content/repositories/snapshots") {

       //  authentication(userName: sonatypeUsername, password: sonatypePassword)

       //}

@@ -101,4 +96,4 @@

       }

     }

   }

-}

\ No newline at end of file

+}



# gradle build -x test copyJars
# mkdir /tmp/jar
# cp -a build/libs/*.jar /tmp/jar/ && cp -a dist/*.jar /tmp/jar/

最后将 jar 包 分发到tomcat 项目中

 

「一键投喂 软糖/蛋糕/布丁/牛奶/冰阔乐！」

Suzf Blog

(๑＞ڡ＜)☆ 谢谢 ~

使用微信扫描二维码完成支付