資料工程 – 架設高可用性RabbitMQ叢集

0
85

架設RabbitMQ(以下簡稱RMQ)叢集的時候最麻煩的是要下指令去做join node的動作,無法方便的做CD,DevOps非常頭大。

如果你是用docker swarm架設的話,docker swarm的load balance是走round robin的方式,有沒有更有效率的load balance可以用呢?

這篇文章內容會介紹如何用Consul+Haproxy解決上面的兩個問題,並使用Docker Swarm來架設RMQ叢集。

架構說明

RMQ Cluster+Haproxy+Consul整體架構

如上圖所示,我們會用Haproxy替換掉Swarm的load balance,並且使用服務發現系統Consul來做RMQ叢集的自動串起。

範例會有3台的RMQ node service,1個Haproxy service,3個Consul service在RMQ node上。

預期的狀況會是Swarm下達部署指令後,Consul會自動將3台RMQ串起來,並且透過Haproxy達到load balance的效果,我們所需要做的就只是下一次Swarm指令和觀察狀況即可。

Step 1. 檢查並設置Docker環境

Check docker swarm cluster

Set docker node label

$ docker node update --label-add rabbitmq1=true --label-add consul=true demo1
$ docker node update --label-add rabbitmq2=true demo2
$ docker node update --label-add rabbitmq3=true demo3

Step 2. Haproxy設定

Haproxy settings

haproxy/conf/haproxy.cfg

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    maxconn 4096

defaults
    log     global
    option  tcplog
    option  dontlognull
    timeout connect 6s
    timeout client 120s
    timeout server 120s

listen  stats
    bind *:1936
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /

listen rabbitmq
    bind   *:5672
    mode   tcp
    server rabbitmq-01 rabbitmq-01:5672 check
    server rabbitmq-02 rabbitmq-02:5672 check
    server rabbitmq-03 rabbitmq-03:5672 check

listen rabbitmq-ui
    bind   *:15672
    mode   http
    server rabbitmq-01 rabbitmq-01:15672 check
    server rabbitmq-02 rabbitmq-02:15672 check
    server rabbitmq-03 rabbitmq-03:15672 check

Haproxy Dockerfile

haproxy/Dockerfile

FROM haproxy:1.9.8
COPY conf/haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

Step 3. Consul設定

docker-compose-consul.yml

docker-compose-consul.yml

version: "3.2"

services:

  consul:
    image: consul:1.6.0
    hostname: "{{.Node.Hostname}}"
    networks:
      - consul
      - demo-net
    ports:
      - 8400:8400
      - 8500:8500
      - 8600:53
    deploy:
      mode: global
      placement:
        constraints: [node.labels.consul == true]
    command: [ "agent", "-server", "-bootstrap-expect=3", "-retry-max=3", "-retry-interval=10s", "-datacenter=demo-net", "-join=consul", "-retry-join=consul", "-bind={{ GetInterfaceIP \"eth0\" }}", "-client=0.0.0.0", "-ui"]

networks:
  consul:
  demo-net:
    external: true

Step 4. RMQ設定

RabbitMQ settings

rabitmq/conf/rabbitmq.conf

loopback_users.admin = false
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_consul
cluster_formation.consul.host = consul
cluster_formation.node_cleanup.only_log_warning = true
cluster_formation.consul.svc_addr_auto = true
cluster_partition_handling = autoheal
#Flow Control is triggered if memory usage above %80.
vm_memory_high_watermark.relative = 0.8
#Flow Control is triggered if free disk size below 5GB.
disk_free_limit.absolute = 5GB

RabbitMQ Dockerfile

rabitmq/Dockerfile

FROM rabbitmq:3.7.14-management

ADD conf/ /etc/rabbitmq/

docker-compose.yml

docker-compose.yml

version: "3.2"

services:

  rabbitmq-01:
    build:
      context: ./rabitmq
      dockerfile: Dockerfile
    image: 10.140.15.238:5000/rabbitmq
    hostname: rabbitmq-01
    environment:
      - RABBITMQ_DEFAULT_USER=demo
      - RABBITMQ_DEFAULT_PASS=demo
      - RABBITMQ_ERLANG_COOKIE="MY-SECRET-KEY-123"
      - RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.8
      - RABBITMQ_NODENAME=rabbitmq
      - RABBITMQ_HIPE_COMPILE=1
    networks:
      - demo-net
    logging:
        driver: "json-file"
        options:
          max-size: "10m"
    deploy:
      mode: global
      placement:
        constraints:
          - node.labels.rabbitmq1 == true

  rabbitmq-02:
    build:
      context: ./rabitmq
      dockerfile: Dockerfile
    image: 10.140.15.238:5000/rabbitmq
    hostname: rabbitmq-02
    environment:
      - RABBITMQ_DEFAULT_USER=demo
      - RABBITMQ_DEFAULT_PASS=demo
      - RABBITMQ_ERLANG_COOKIE="MY-SECRET-KEY-123"
      - RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.8
      - RABBITMQ_NODENAME=rabbitmq
      - RABBITMQ_HIPE_COMPILE=1
    networks:
      - demo-net
    logging:
        driver: "json-file"
        options:
          max-size: "10m"
    deploy:
      mode: global
      placement:
        constraints:
          - node.labels.rabbitmq2 == true

  rabbitmq-03:
    build:
      context: ./rabitmq
      dockerfile: Dockerfile
    image: 10.140.15.238:5000/rabbitmq
    hostname: rabbitmq-03
    environment:
      - RABBITMQ_DEFAULT_USER=demo
      - RABBITMQ_DEFAULT_PASS=demo
      - RABBITMQ_ERLANG_COOKIE="MY-SECRET-KEY-123"
      - RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.8
      - RABBITMQ_NODENAME=rabbitmq
      - RABBITMQ_HIPE_COMPILE=1
    networks:
      - demo-net
    logging:
        driver: "json-file"
        options:
          max-size: "10m"
    deploy:
      mode: global
      placement:
        constraints:
          - node.labels.rabbitmq3 == true

  haproxy:
    build:
      context: ./haproxy
      dockerfile: Dockerfile
    image: 10.140.15.238:5000/rmq-haproxy
    ports:
      - 15672:15672
      - 5672:5672
      - 1936:1936
    logging:
        driver: "json-file"
        options:
          max-size: "10m"
    networks:
      - demo-net
    deploy:
      mode: global
      placement:
        constraints: [node.labels.rabbitmq1 == true]

networks:
  demo-net:
    external: true

Step 5. 啟動指令與打包

將Haproxy、Consul與RMQ設定完成後,接下來就是啟動Service,由於是部署在Docker Swarm,會需要建立image後上傳到私人或公開的registry server,在以下Makefile的build的部分就是在做這件事情。

部署Service的run部分,我們會先啟動一台Consul node,等待幾秒再啟動其他的Consul node,這個動作是降低Consul選舉Leader node時可能會發生的無法競選出線的問題。

run的最後部分啟動RMQ和Haproxy。

Makefile

Makefile

build:
    # clear network setting
    docker network rm demo-net
    # docker build img
    docker network create --attachable -d overlay demo-net
    docker-compose build
    docker-compose push

run:
    # clear labels
    docker node update --label-rm rabbitmq1 --label-rm consul --label-rm demo1 || true
    docker node update --label-rm rabbitmq2 --label-rm consul --label-rm demo2 || true
    docker node update --label-rm rabbitmq3 --label-rm consul --label-rm demo3 || true

    # add rabbitmq label
    docker node update --label-add rabbitmq1=true --label-add consul=true demo1 || true
    docker node update --label-add rabbitmq2=true demo2 || true
    docker node update --label-add rabbitmq3=true demo3 || true

    # start demo1 consul
    docker stack deploy -c docker-compose-consul.yml consul
    sleep 10

    # start demo2 demo3 consul
    docker node update --label-add consul=true demo2
    docker node update --label-add consul=true demo3
    sleep 60

    # start rabbitmq and haproxy
    docker stack deploy -c docker-compose.yml rmq

執行 demo project

建立好Makefile把指令都包起來後,要部署project就變得更輕鬆了,只要下兩行指令就行。

# 建立環境與docker image
$ make build

# 部署專案
$ make run

執行成功畫面

Consul

http://localhost:8500/ui/demo-net/services/rabbitmq

Haproxy

http://localhost:1936/

RMQ

http://localhost:15672

Demo Project GitHub

https://github.com/mmmaaaxxx77/20PC_DockerSwarmRabbitMQCluster

References

Leave a Reply

avatar
  Subscribe  
Notify of