956237586 – 第 6 页 – HYL Studio

20230204尝试搭建k8s青春版

视频版

https://www.bilibili.com/video/BV1qj411T7d5/

背景

之前一直想搭建k8s，但按多年前的印象这玩意很复杂。今天先尝试个青春版k3s

安装准备

按文档说是离线在线都行，离线的需要个私有的镜像中心，但镜像中心最简便的搭建方式是跑个容器。。。死循环了啊，所以先试在线的。

家里的资源不多，尝试做个乞丐版，2~3节点就行了，先弄单机后面再加worker

环境要求

硬件按文档建议是最少1c1g，我就给2c2g吧，机器数量是2

系统要求Ubuntu是22.04

可以使用 --with-node-id 给每个节点后面增加一个随机的后缀。或者使用命令行参数 --node-name 或环境变量 $K3S_NODE_NAME 给集群的每个节点传递唯一的名字

网络每个节点需要6443 可以被访问

Flannel VXLAN 节点间通过 UDP 8472 通信

Flannel Wireguard backend 节点间通过 UDP 51820(ipv4) 51821(ipv6)

安装过程


k3s1@k3s1:~$ curl -sfL https://get.k3s.io | sh -
[sudo] password for k3s1:
[INFO]  Finding release for channel stable
[INFO]  Using v1.25.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

k3s1@k3s1:~$ curl -sfL https://get.k3s.io | sh -

[sudo] password for k3s1:

[INFO] Finding release for channel stable

[INFO] Using v1.25.6+k3s1 as release

[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt

[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s

[INFO] Verifying binary download

[INFO] Installing k3s to /usr/local/bin/k3s

[INFO] Skipping installation of SELinux RPM

[INFO] Creating /usr/local/bin/kubectl symlink to k3s

[INFO] Creating /usr/local/bin/crictl symlink to k3s

[INFO] Creating /usr/local/bin/ctr symlink to k3s

[INFO] Creating killall script /usr/local/bin/k3s-killall.sh

[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh

[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env

[INFO] systemd: Creating service file /etc/systemd/system/k3s.service

[INFO] systemd: Enabling k3s unit

Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

主节点安装过程如上，疑似成功。。。

查看进程如下，可以看到/run/k3s/containerd/containerd.sock说明containerd已经在运行了


k3s1@k3s1:~$ ps aux|grep k3s
k3s1        1757  0.0  0.4  19152  9796 ?        Ss   11:25   0:00 /lib/systemd/systemd --user
k3s1        1759  0.0  0.2 105292  4420 ?        S    11:25   0:00 (sd-pam)
k3s1        1769  0.0  0.2   8264  5192 tty1     S+   11:25   0:00 -bash
root        2406  0.0  0.4  13788  9016 ?        Ss   11:29   0:00 sshd: k3s1 [priv]
k3s1        2478  0.0  0.3  13924  6000 ?        R    11:29   0:00 sshd: k3s1@pts/0
k3s1        2479  0.0  0.2   8276  5288 pts/0    Ss   11:29   0:00 -bash
root        2727 33.2 22.3 1194116 447444 ?      Ssl  11:31   1:01 /usr/local/bin/k3s server
root        2772 11.0  3.7 771044 75764 ?        Sl   11:31   0:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
root        3467  0.1  0.5 720748 10652 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9b07e1b9470f0e657d9a00667c44bb955e3d5c7f049036adfd38d086d73026e1 -address /run/k3s/containerd/containerd.sock
root        3511  0.1  0.5 721004 10740 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 65fba4a19627e12286bbdb2056f27f1c063c060255510b9b997e291a44e1c736 -address /run/k3s/containerd/containerd.sock
root        3521  0.0  0.4 720748  9852 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3193ac9aa5db3faec494a7fa431c2bbed4c9a124c67042ef956ba1f1b08783b0 -address /run/k3s/containerd/containerd.sock
root        3538  0.0  0.5 720748 10972 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id bed5712d5e71c22a2b4863c61df5402ed3cd2e468b89b3699f3b2db48ea2f522 -address /run/k3s/containerd/containerd.sock
root        3563  0.0  0.5 720492 10208 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id b76d14ccfaa1a5743f4fa77aef74c210b89badd130c75244e8561dffa643cdba -address /run/k3s/containerd/containerd.sock
k3s1        3944  6.1  2.3 751748 47708 ?        Ssl  11:32   0:04 /metrics-server --cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s
k3s1        4181  0.0  0.1   9128  3480 pts/0    R+   11:34   0:00 ps aux
k3s1        4182  0.0  0.0   6432   720 pts/0    S+   11:34   0:00 grep --color=auto k3s

k3s1@k3s1:~$ ps aux|grep k3s

k3s1 1757 0.0 0.4 19152 9796 ? Ss 11:25 0:00 /lib/systemd/systemd --user

k3s1 1759 0.0 0.2 105292 4420 ? S 11:25 0:00 (sd-pam)

k3s1 1769 0.0 0.2 8264 5192 tty1 S+ 11:25 0:00 -bash

root 2406 0.0 0.4 13788 9016 ? Ss 11:29 0:00 sshd: k3s1 [priv]

k3s1 2478 0.0 0.3 13924 6000 ? R 11:29 0:00 sshd: k3s1@pts/0

k3s1 2479 0.0 0.2 8276 5288 pts/0 Ss 11:29 0:00 -bash

root 2727 33.2 22.3 1194116 447444 ? Ssl 11:31 1:01 /usr/local/bin/k3s server

root 2772 11.0 3.7 771044 75764 ? Sl 11:31 0:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd

root 3467 0.1 0.5 720748 10652 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9b07e1b9470f0e657d9a00667c44bb955e3d5c7f049036adfd38d086d73026e1 -address /run/k3s/containerd/containerd.sock

root 3511 0.1 0.5 721004 10740 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 65fba4a19627e12286bbdb2056f27f1c063c060255510b9b997e291a44e1c736 -address /run/k3s/containerd/containerd.sock

root 3521 0.0 0.4 720748 9852 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3193ac9aa5db3faec494a7fa431c2bbed4c9a124c67042ef956ba1f1b08783b0 -address /run/k3s/containerd/containerd.sock

root 3538 0.0 0.5 720748 10972 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id bed5712d5e71c22a2b4863c61df5402ed3cd2e468b89b3699f3b2db48ea2f522 -address /run/k3s/containerd/containerd.sock

root 3563 0.0 0.5 720492 10208 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id b76d14ccfaa1a5743f4fa77aef74c210b89badd130c75244e8561dffa643cdba -address /run/k3s/containerd/containerd.sock

k3s1 3944 6.1 2.3 751748 47708 ? Ssl 11:32 0:04 /metrics-server --cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s

k3s1 4181 0.0 0.1 9128 3480 pts/0 R+ 11:34 0:00 ps aux

k3s1 4182 0.0 0.0 6432 720 pts/0 S+ 11:34 0:00 grep --color=auto k3s

metrics-server不知道是干什么的，后面再翻翻文档。

网络监听情况如下，可以看到大部分都是本地监听，只有10250 6443是有可能对外开放的

按文档描述是个TCP的端口监听，看起来和metrics server进程有关，好像是kubelet在节点之间通讯使用的。官方原文：If you wish to utilize the metrics server, all nodes must be accessible to each other on port 10250.

6443是TCP的监听，流量方向是从Agent向Server的，应该是worker节点向master注册数据用的。官方描述为 supervisor and Kubernetes API Server


k3s1@k3s1:~$ netstat -anp|grep LISTEN
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:10010         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10258         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::10250                :::*                    LISTEN      -
tcp6       0      0 :::6443                 :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

k3s1@k3s1:~$ netstat -anp|grep LISTEN

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp 0 0 127.0.0.1:10010 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:6444 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10256 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10258 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -

tcp6 0 0 :::10250 :::* LISTEN -

tcp6 0 0 :::6443 :::* LISTEN -

tcp6 0 0 :::22 :::* LISTEN -

按文档所说配置文件内容如下


k3s1@k3s1:~$ sudo cat /etc/rancher/k3s/k3s.yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3SGhjTk1qTXdNakEwTVRFek1URTVXaGNOTXpNd01qQXhNVEV6TVRFNQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTS0lXTmNvbWZleEhPcndCVjUwdWx5ZjkwQlo2a0sxOStGUW9GN0pNUloKYi9UdndyZW9vNXhwWHNNekpEYlVUWkFzUXdlOXBDK3BoTjRaQ2ZFNVloRlVvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVURPelE3YkFldXh2a3JZN1hnSDI1CkNDOGpRcFF3Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnYW5rMlprZncrSHZCdHBvNnBuYmJiRDRuSExHSzlRZ3YKZFRLZTY3dm8zWVVDSUNtK0pJdisxSzBBcUpmUXA5L1JkdDRoMkF6Slc1WW5CenVIYzdLckhiRHgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRlZ0F3SUJBZ0lJVmVMRW9qM1NqUUV3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamMxTlRFd01qYzVNQjRYRFRJek1ESXdOREV4TXpFeE9Wb1hEVEkwTURJdwpOREV4TXpFeE9Wb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJBSld6ZVpTK0h6bHZCMkgKMkVJUUh5bEZQL3pDMmpZTktoZGtLTFF0NjhtTENWNnpKRWhxVDhYby9EUXdoc20zL2tmQWp1QlJ2R0tRNmsvRgpONDRIU0oyalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUUdCNlk4LzB5MjhSbmJnemdsejRxcXNURE5QekFLQmdncWhrak9QUVFEQWdOSkFEQkcKQWlFQTcybUw5b1dKeDM0SzZOZlVsWGp2dEQ5WXRJQXZuMWFuUUJkeHlXSUhOYThDSVFEUFllK3RZemtFR0RNbAppcnlkVTJpTHM0dWdVdXpwQThqM1Q2bEd4MU1nYUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZURDQ0FSMmdBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwClpXNTBMV05oUURFMk56VTFNVEF5Tnprd0hoY05Nak13TWpBME1URXpNVEU1V2hjTk16TXdNakF4TVRFek1URTUKV2pBak1TRXdId1lEVlFRRERCaHJNM010WTJ4cFpXNTBMV05oUURFMk56VTFNVEF5Tnprd1dUQVRCZ2NxaGtqTwpQUUlCQmdncWhrak9QUU1CQndOQ0FBUjB1cWV6cUh6S2owRUt2dFdNZ1YrQ09mblk0aGU3UFVLdnZlMWQ4d0VJCjFhb2laYjdzM05lK3ZvMzBQRkY5ZzdVZksrdGVJVjYxem80UytqVm9YU0k0bzBJd1FEQU9CZ05WSFE4QkFmOEUKQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVCZ2VtUFA5TXR2RVoyNE00SmMrSwpxckV3elQ4d0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFNVUlXUEExeXZYdEJsUUpFRER2aUtyM1hYbVJWb2hyClZGK3YzK0h4VXVGSUFpRUF5cmFhZlRiRG4vaTBwUmNHWnVEOHIzaTZQWHlJbUo0M1FpQnpLbHVjcnBFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUJzaUxpNU1NaG5wbUpXclJtSkxETUZobDVBSVcxTEQrS0tPRzBFRTlzc2RvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFQWxiTjVsTDRmT1c4SFlmWVFoQWZLVVUvL01MYU5nMHFGMlFvdEMzcnlZc0pYck1rU0dwUAp4ZWo4TkRDR3liZitSOENPNEZHOFlwRHFUOFUzamdkSW5RPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

k3s1@k3s1:~$ sudo cat /etc/rancher/k3s/k3s.yaml

apiVersion: v1

clusters:

- cluster:

certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3SGhjTk1qTXdNakEwTVRFek1URTVXaGNOTXpNd01qQXhNVEV6TVRFNQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTS0lXTmNvbWZleEhPcndCVjUwdWx5ZjkwQlo2a0sxOStGUW9GN0pNUloKYi9UdndyZW9vNXhwWHNNekpEYlVUWkFzUXdlOXBDK3BoTjRaQ2ZFNVloRlVvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVURPelE3YkFldXh2a3JZN1hnSDI1CkNDOGpRcFF3Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnYW5rMlprZncrSHZCdHBvNnBuYmJiRDRuSExHSzlRZ3YKZFRLZTY3dm8zWVVDSUNtK0pJdisxSzBBcUpmUXA5L1JkdDRoMkF6Slc1WW5CenVIYzdLckhiRHgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

server: https://127.0.0.1:6443

contexts:

- context:

cluster: default

user: default

current-context: default

kind: Config

preferences: {}

users:

- name: default

user:

client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRlZ0F3SUJBZ0lJVmVMRW9qM1NqUUV3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamMxTlRFd01qYzVNQjRYRFRJek1ESXdOREV4TXpFeE9Wb1hEVEkwTURJdwpOREV4TXpFeE9Wb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJBSld6ZVpTK0h6bHZCMkgKMkVJUUh5bEZQL3pDMmpZTktoZGtLTFF0NjhtTENWNnpKRWhxVDhYby9EUXdoc20zL2tmQWp1QlJ2R0tRNmsvRgpONDRIU0oyalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUUdCNlk4LzB5MjhSbmJnemdsejRxcXNURE5QekFLQmdncWhrak9QUVFEQWdOSkFEQkcKQWlFQTcybUw5b1dKeDM0SzZOZlVsWGp2dEQ5WXRJQXZuMWFuUUJkeHlXSUhOYThDSVFEUFllK3RZemtFR0RNbAppcnlkVTJpTHM0dWdVdXpwQThqM1Q2bEd4MU1nYUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZURDQ0FSMmdBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwClpXNTBMV05oUURFMk56VTFNVEF5Tnprd0hoY05Nak13TWpBME1URXpNVEU1V2hjTk16TXdNakF4TVRFek1URTUKV2pBak1TRXdId1lEVlFRRERCaHJNM010WTJ4cFpXNTBMV05oUURFMk56VTFNVEF5Tnprd1dUQVRCZ2NxaGtqTwpQUUlCQmdncWhrak9QUU1CQndOQ0FBUjB1cWV6cUh6S2owRUt2dFdNZ1YrQ09mblk0aGU3UFVLdnZlMWQ4d0VJCjFhb2laYjdzM05lK3ZvMzBQRkY5ZzdVZksrdGVJVjYxem80UytqVm9YU0k0bzBJd1FEQU9CZ05WSFE4QkFmOEUKQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVCZ2VtUFA5TXR2RVoyNE00SmMrSwpxckV3elQ4d0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFNVUlXUEExeXZYdEJsUUpFRER2aUtyM1hYbVJWb2hyClZGK3YzK0h4VXVGSUFpRUF5cmFhZlRiRG4vaTBwUmNHWnVEOHIzaTZQWHlJbUo0M1FpQnpLbHVjcnBFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUJzaUxpNU1NaG5wbUpXclJtSkxETUZobDVBSVcxTEQrS0tPRzBFRTlzc2RvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFQWxiTjVsTDRmT1c4SFlmWVFoQWZLVVUvL01MYU5nMHFGMlFvdEMzcnlZc0pYck1rU0dwUAp4ZWo4TkRDR3liZitSOENPNEZHOFlwRHFUOFUzamdkSW5RPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

按文档描述会生成这几个相关命令kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh。具体位置如下，均在/usr/local/bin下，所以可以从任意目录执行。

说实话k3s-killall.sh和k3s-uninstall.sh感觉并不适合放进这个位置，不知道是不是官方怕你找不到故意也放bin里了，这俩实际生产场景下完全没有全局任意目录运行的必要。万一误执行了就GG


k3s1@k3s1:~$ which kubectl
/usr/local/bin/kubectl
k3s1@k3s1:~$ which crictl
/usr/local/bin/crictl
k3s1@k3s1:~$ which k3s-killall.sh
/usr/local/bin/k3s-killall.sh
k3s1@k3s1:~$ which k3s-uninstall.sh
/usr/local/bin/k3s-uninstall.sh

k3s1@k3s1:~$ which kubectl

/usr/local/bin/kubectl

k3s1@k3s1:~$ which crictl

/usr/local/bin/crictl

k3s1@k3s1:~$ which k3s-killall.sh

/usr/local/bin/k3s-killall.sh

k3s1@k3s1:~$ which k3s-uninstall.sh

/usr/local/bin/k3s-uninstall.sh

尝试获取当前集群的节点，虽然知道只有一个，但需要看看命令是不是能正常执行，别和之前有的服务一样看着没事但实际没正常运行。。。。


k3s1@k3s1:~$ sudo kubectl get nodes
NAME   STATUS   ROLES                  AGE   VERSION
k3s1   Ready    control-plane,master   14m   v1.25.6+k3s1

k3s1@k3s1:~$ sudo kubectl get nodes

NAME STATUS ROLES AGE VERSION

k3s1 Ready control-plane,master 14m v1.25.6+k3s1

这里有个需要关注的东西就是ROLES都有啥，需要看文档了。目前的control-plane顾名思义是控制面, master是主节点。不知道k8s的角色分配是不是和CDH一样能灵活转移。。。

查看TOKEN，准备安装第一个worker节点，按文档描述在这里/var/lib/rancher/k3s/server/node-token

注意，我在一个隔离环境测试，所以/var/lib/rancher/k3s/server/node-token和/etc/rancher/k3s/k3s.yaml的内容暴露了无所谓，待会会直接删除虚机，生产的千万不要轻易暴露。


sudo cat /var/lib/rancher/k3s/server/node-token
K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9

sudo cat /var/lib/rancher/k3s/server/node-token

K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9

按要求拼接worker的安装命令如下，到第二个机器上执行


curl -sfL https://get.k3s.io | K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -

0 1	curl -sfL https://get.k3s.io \| K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -

执行到systemd卡住了（后面已确认只是单纯的慢）


k3s2@k3s2:~$ curl -sfL https://get.k3s.io | K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -
[sudo] password for k3s2:
[INFO]  Finding release for channel stable
[INFO]  Using v1.25.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

k3s2@k3s2:~$ curl -sfL https://get.k3s.io | K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -

[sudo] password for k3s2:

[INFO] Finding release for channel stable

[INFO] Using v1.25.6+k3s1 as release

[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt

[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s

[INFO] Verifying binary download

[INFO] Installing k3s to /usr/local/bin/k3s

[INFO] Skipping installation of SELinux RPM

[INFO] Creating /usr/local/bin/kubectl symlink to k3s

[INFO] Creating /usr/local/bin/crictl symlink to k3s

[INFO] Creating /usr/local/bin/ctr symlink to k3s

[INFO] Creating killall script /usr/local/bin/k3s-killall.sh

[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh

[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env

[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service

[INFO] systemd: Enabling k3s-agent unit

Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.

[INFO] systemd: Starting k3s-agent

再开一个shell查看进程如下


k3s2@k3s2:~$ ps aux|grep k3s
k3s2        1794  0.0  0.4  19040  9712 ?        Ss   11:25   0:00 /lib/systemd/systemd --user
k3s2        1796  0.0  0.2 105288  4416 ?        S    11:25   0:00 (sd-pam)
k3s2        1801  0.0  0.2   8264  5280 tty1     S+   11:25   0:00 -bash
root        2368  0.0  0.4  13796  8968 ?        Ss   11:29   0:00 sshd: k3s2 [priv]
k3s2        2442  0.0  0.3  13932  6124 ?        S    11:29   0:00 sshd: k3s2@pts/0
k3s2        2443  0.0  0.2   8276  5172 pts/0    Ss   11:29   0:00 -bash
k3s2        3063  0.0  0.0   2608  1904 pts/0    S+   11:53   0:00 sh -
root        3241  0.0  0.2   9264  4656 pts/0    S+   11:54   0:00 sudo systemctl restart k3s-agent
root        3242  0.0  0.1   9276  3760 pts/0    S+   11:54   0:00 systemctl restart k3s-agent
root        3263  6.8  4.3 842952 87268 ?        Ssl  11:54   0:08 /usr/local/bin/k3s agent
root        3328  0.5  0.4  13796  9000 ?        Ss   11:56   0:00 sshd: k3s2 [priv]
k3s2        3416  0.0  0.3  13932  6140 ?        S    11:56   0:00 sshd: k3s2@pts/1
k3s2        3417  1.6  0.2   8276  5100 pts/1    Ss   11:56   0:00 -bash
k3s2        3428  0.0  0.1   8888  3380 pts/1    R+   11:56   0:00 ps aux
k3s2        3429  0.0  0.0   6432   724 pts/1    S+   11:56   0:00 grep --color=auto k3s

k3s2@k3s2:~$ ps aux|grep k3s

k3s2 1794 0.0 0.4 19040 9712 ? Ss 11:25 0:00 /lib/systemd/systemd --user

k3s2 1796 0.0 0.2 105288 4416 ? S 11:25 0:00 (sd-pam)

k3s2 1801 0.0 0.2 8264 5280 tty1 S+ 11:25 0:00 -bash

root 2368 0.0 0.4 13796 8968 ? Ss 11:29 0:00 sshd: k3s2 [priv]

k3s2 2442 0.0 0.3 13932 6124 ? S 11:29 0:00 sshd: k3s2@pts/0

k3s2 2443 0.0 0.2 8276 5172 pts/0 Ss 11:29 0:00 -bash

k3s2 3063 0.0 0.0 2608 1904 pts/0 S+ 11:53 0:00 sh -

root 3241 0.0 0.2 9264 4656 pts/0 S+ 11:54 0:00 sudo systemctl restart k3s-agent

root 3242 0.0 0.1 9276 3760 pts/0 S+ 11:54 0:00 systemctl restart k3s-agent

root 3263 6.8 4.3 842952 87268 ? Ssl 11:54 0:08 /usr/local/bin/k3s agent

root 3328 0.5 0.4 13796 9000 ? Ss 11:56 0:00 sshd: k3s2 [priv]

k3s2 3416 0.0 0.3 13932 6140 ? S 11:56 0:00 sshd: k3s2@pts/1

k3s2 3417 1.6 0.2 8276 5100 pts/1 Ss 11:56 0:00 -bash

k3s2 3428 0.0 0.1 8888 3380 pts/1 R+ 11:56 0:00 ps aux

k3s2 3429 0.0 0.0 6432 724 pts/1 S+ 11:56 0:00 grep --color=auto k3s

根据systemd服务注册的命令/usr/local/bin/k3s agent，可以看到大概率是执行systemctl restart k3s-agent卡住的


k3s2@k3s2:~$ cat /etc/systemd/system/k3s-agent.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    agent \

k3s2@k3s2:~$ cat /etc/systemd/system/k3s-agent.service

[Unit]

Description=Lightweight Kubernetes

Documentation=https://k3s.io

Wants=network-online.target

After=network-online.target

[Install]

WantedBy=multi-user.target

[Service]

Type=notify

EnvironmentFile=-/etc/default/%N

EnvironmentFile=-/etc/sysconfig/%N

EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env

KillMode=process

Delegate=yes

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=1048576

LimitNPROC=infinity

LimitCORE=infinity

TasksMax=infinity

TimeoutStartSec=0

Restart=always

RestartSec=5s

ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'

ExecStartPre=-/sbin/modprobe br_netfilter

ExecStartPre=-/sbin/modprobe overlay

ExecStart=/usr/local/bin/k3s \

agent \

~~token本身错的可能性不大，所以大概率是K3S_URL的问题，因为我用了本地的DNS，所以写的是域名。尝试配置下HOSTS试试~~

保险起见我去 /var/lib/rancher/k3s/agent/containerd/containerd.log看了下，居然还在执行东西。。。难道只是单纯的慢？这时候主节点已经能看到agent了，但ROLES里是none让我比较纠结是不是正常


k3s1@k3s1:~$ sudo kubectl get nodes
NAME   STATUS   ROLES                  AGE    VERSION
k3s1   Ready    control-plane,master   37m    v1.25.6+k3s1
k3s2   Ready    <none>                 5m8s   v1.25.6+k3s1

k3s1@k3s1:~$ sudo kubectl get nodes

NAME STATUS ROLES AGE VERSION

k3s1 Ready control-plane,master 37m v1.25.6+k3s1

k3s2 Ready <none> 5m8s v1.25.6+k3s1

根据命令行参数打印更加详细的节点信息如下，这样看起来worker工作是正常的


k3s1@k3s1:~$ sudo kubectl get nodes -o wide  --show-labels=true
NAME   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME          LABELS
k3s1   Ready    control-plane,master   44m   v1.25.6+k3s1   192.168.2.227   <none>        Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.15-k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
k3s2   Ready    <none>                 12m   v1.25.6+k3s1   192.168.2.176   <none>        Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.15-k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s2,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s

k3s1@k3s1:~$ sudo kubectl get nodes -o wide --show-labels=true

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME LABELS

k3s1 Ready control-plane,master 44m v1.25.6+k3s1 192.168.2.227 <none> Ubuntu 20.04.5 LTS 5.4.0-125-generic containerd://1.6.15-k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s

k3s2 Ready <none> 12m v1.25.6+k3s1 192.168.2.176 <none> Ubuntu 20.04.5 LTS 5.4.0-125-generic containerd://1.6.15-k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s2,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s

测试

尝试构建一个服务是否能正常运行，从k8s官网看看能不能扒一个现成的yaml测试


k3s1@k3s1:~$ sudo  kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

k3s1@k3s1:~$ sudo kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Kustomize Version: v4.5.7

Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

尝试创建DEMO


k3s1@k3s1:~$ sudo kubectl get deployments
No resources found in default namespace.
k3s1@k3s1:~$ sudo kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1
deployment.apps/kubernetes-bootcamp created
k3s1@k3s1:~$ sudo kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   0/1     1            0           4s

k3s1@k3s1:~$ sudo kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   1/1     1            1           44s

k3s1@k3s1:~$ sudo kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-75c5d958ff-dzdkw   1/1     Running   0          3m23s

k3s1@k3s1:~$ sudo kubectl describe pods
Name:             kubernetes-bootcamp-75c5d958ff-dzdkw
Namespace:        default
Priority:         0
Service Account:  default
Node:             k3s2/192.168.2.176
Start Time:       Sat, 04 Feb 2023 12:23:03 +0000
Labels:           app=kubernetes-bootcamp
                  pod-template-hash=75c5d958ff
Annotations:      <none>
Status:           Running
IP:               10.42.1.3
IPs:
  IP:           10.42.1.3
Controlled By:  ReplicaSet/kubernetes-bootcamp-75c5d958ff
Containers:
  kubernetes-bootcamp:
    Container ID:   containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9
    Image:          gcr.io/google-samples/kubernetes-bootcamp:v1
    Image ID:       gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 04 Feb 2023 12:23:46 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-r2mwg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  4m23s  default-scheduler  Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2
  Normal  Pulling    4m22s  kubelet            Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"
  Normal  Pulled     3m40s  kubelet            Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)
  Normal  Created    3m40s  kubelet            Created container kubernetes-bootcamp
  Normal  Started    3m40s  kubelet            Started container kubernetes-bootcamp


k3s1@k3s1:~$ sudo kubectl describe pods kubernetes-bootcamp-75c5d958ff-dzdkw
Name:             kubernetes-bootcamp-75c5d958ff-dzdkw
Namespace:        default
Priority:         0
Service Account:  default
Node:             k3s2/192.168.2.176
Start Time:       Sat, 04 Feb 2023 12:23:03 +0000
Labels:           app=kubernetes-bootcamp
                  pod-template-hash=75c5d958ff
Annotations:      <none>
Status:           Running
IP:               10.42.1.3
IPs:
  IP:           10.42.1.3
Controlled By:  ReplicaSet/kubernetes-bootcamp-75c5d958ff
Containers:
  kubernetes-bootcamp:
    Container ID:   containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9
    Image:          gcr.io/google-samples/kubernetes-bootcamp:v1
    Image ID:       gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 04 Feb 2023 12:23:46 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-r2mwg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  5m28s  default-scheduler  Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2
  Normal  Pulling    5m28s  kubelet            Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"
  Normal  Pulled     4m46s  kubelet            Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)
  Normal  Created    4m46s  kubelet            Created container kubernetes-bootcamp
  Normal  Started    4m46s  kubelet            Started container kubernetes-bootcamp

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

k3s1@k3s1:~$ sudo kubectl get deployments

No resources found in default namespace.

k3s1@k3s1:~$ sudo kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1

deployment.apps/kubernetes-bootcamp created

k3s1@k3s1:~$ sudo kubectl get deployments

NAME READY UP-TO-DATE AVAILABLE AGE

kubernetes-bootcamp 0/1 1 0 4s

k3s1@k3s1:~$ sudo kubectl get deployments

NAME READY UP-TO-DATE AVAILABLE AGE

kubernetes-bootcamp 1/1 1 1 44s

k3s1@k3s1:~$ sudo kubectl get pods

NAME READY STATUS RESTARTS AGE

kubernetes-bootcamp-75c5d958ff-dzdkw 1/1 Running 0 3m23s

k3s1@k3s1:~$ sudo kubectl describe pods

Name: kubernetes-bootcamp-75c5d958ff-dzdkw

Namespace: default

Priority: 0

Service Account: default

Node: k3s2/192.168.2.176

Start Time: Sat, 04 Feb 2023 12:23:03 +0000

Labels: app=kubernetes-bootcamp

pod-template-hash=75c5d958ff

Annotations: <none>

Status: Running

IP: 10.42.1.3

IPs:

IP: 10.42.1.3

Controlled By: ReplicaSet/kubernetes-bootcamp-75c5d958ff

Containers:

kubernetes-bootcamp:

Container ID: containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9

Image: gcr.io/google-samples/kubernetes-bootcamp:v1

Image ID: gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af

Port: <none>

Host Port: <none>

State: Running

Started: Sat, 04 Feb 2023 12:23:46 +0000

Ready: True

Restart Count: 0

Environment: <none>

Mounts:

/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)

Conditions:

Type Status

Initialized True

Ready True

ContainersReady True

PodScheduled True

Volumes:

kube-api-access-r2mwg:

Type: Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds: 3607

ConfigMapName: kube-root-ca.crt

ConfigMapOptional: <nil>

DownwardAPI: true

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 4m23s default-scheduler Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2

Normal Pulling 4m22s kubelet Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"

Normal Pulled 3m40s kubelet Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)

Normal Created 3m40s kubelet Created container kubernetes-bootcamp

Normal Started 3m40s kubelet Started container kubernetes-bootcamp

k3s1@k3s1:~$ sudo kubectl describe pods kubernetes-bootcamp-75c5d958ff-dzdkw

Name: kubernetes-bootcamp-75c5d958ff-dzdkw

Namespace: default

Priority: 0

Service Account: default

Node: k3s2/192.168.2.176

Start Time: Sat, 04 Feb 2023 12:23:03 +0000

Labels: app=kubernetes-bootcamp

pod-template-hash=75c5d958ff

Annotations: <none>

Status: Running

IP: 10.42.1.3

IPs:

IP: 10.42.1.3

Controlled By: ReplicaSet/kubernetes-bootcamp-75c5d958ff

Containers:

kubernetes-bootcamp:

Container ID: containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9

Image: gcr.io/google-samples/kubernetes-bootcamp:v1

Image ID: gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af

Port: <none>

Host Port: <none>

State: Running

Started: Sat, 04 Feb 2023 12:23:46 +0000

Ready: True

Restart Count: 0

Environment: <none>

Mounts:

/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)

Conditions:

Type Status

Initialized True

Ready True

ContainersReady True

PodScheduled True

Volumes:

kube-api-access-r2mwg:

Type: Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds: 3607

ConfigMapName: kube-root-ca.crt

ConfigMapOptional: <nil>

DownwardAPI: true

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 5m28s default-scheduler Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2

Normal Pulling 5m28s kubelet Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"

Normal Pulled 4m46s kubelet Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)

Normal Created 4m46s kubelet Created container kubernetes-bootcamp

Normal Started 4m46s kubelet Started container kubernetes-bootcamp

尝试使用 kubectl exec进入容器


k3s1@k3s1:~$ sudo kubectl exec -it kubernetes-bootcamp-75c5d958ff-dzdkw bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

root@kubernetes-bootcamp-75c5d958ff-dzdkw:/# cat server.js
var http = require('http');
var requests=0;
var podname= process.env.HOSTNAME;
var startTime;
var host;
var handleRequest = function(request, response) {
  response.setHeader('Content-Type', 'text/plain');
  response.writeHead(200);
  response.write("Hello Kubernetes bootcamp! | Running on: ");
  response.write(host);
  response.end(" | v=1\n");
  console.log("Running On:" ,host, "| Total Requests:", ++requests,"| App Uptime:", (new Date() - startTime)/1000 , "seconds", "| Log Time:",new Date());
}
var www = http.createServer(handleRequest);
www.listen(8080,function () {
    startTime = new Date();;
    host = process.env.HOSTNAME;
    console.log ("Kubernetes Bootcamp App Started At:",startTime, "| Running On: " ,host, "\n" );
});

k3s1@k3s1:~$ sudo kubectl exec -it kubernetes-bootcamp-75c5d958ff-dzdkw bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

root@kubernetes-bootcamp-75c5d958ff-dzdkw:/# cat server.js

var http = require('http');

var requests=0;

var podname= process.env.HOSTNAME;

var startTime;

var host;

var handleRequest = function(request, response) {

response.setHeader('Content-Type', 'text/plain');

response.writeHead(200);

response.write("Hello Kubernetes bootcamp! | Running on: ");

response.write(host);

response.end(" | v=1\n");

console.log("Running On:" ,host, "| Total Requests:", ++requests,"| App Uptime:", (new Date() - startTime)/1000 , "seconds", "| Log Time:",new Date());

}

var www = http.createServer(handleRequest);

www.listen(8080,function () {

startTime = new Date();;

host = process.env.HOSTNAME;

console.log ("Kubernetes Bootcamp App Started At:",startTime, "| Running On: " ,host, "\n" );

});

总结

相比多年前的印象，现在的k8s可以借助k3s这个青春版快速的启动一个集群，是时候更新下这个错误认知了，现在启动一个k8s测试用是相当简便并且成本很低的。下次尝试原版k8s，成功后尝试kubekey这个类似ansible一样的自动化安装工具

参考

https://docs.k3s.io/architecture

https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/rancher-v2-7-1/

资源分配和实际占用统计 https://docs.k3s.io/reference/resource-profiling

自动化安装 https://kubesphere.io/zh/docs/v3.3/installing-on-linux/introduction/kubekey/

ROLES https://blog.csdn.net/Lingoesforstudy/article/details/116484624

https://kubernetes.io/zh-cn/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro/

https://blog.csdn.net/qq_44246980/article/details/120143353

20221217mongodb集群搭建

视频版

https://www.bilibili.com/video/BV1Lv4y1Q7xe/

环境准备

按https://www.mongodb.com/docs/manual/administration/production-notes/#std-label-prod-notes-supported-platforms 所说6.0支持20.04和18.04

没说22.04支持，保险起见按文档准备系统版本

至少3个以上的奇数节点，5.0以上必须使用域名配置，节点间端口27017网络要求互通


IP1 mongodb0.example.net
IP2 mongodb1.example.net
IP3 mongodb2.example.net
IP1 test.w.mongo.db
IP2 test.r.mongo.db
IP3 test.r.mongo.db

IP1 mongodb0.example.net

IP2 mongodb1.example.net

IP3 mongodb2.example.net

IP1 test.w.mongo.db

IP2 test.r.mongo.db

IP3 test.r.mongo.db

1.安装


wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -
sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -

sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

sudo apt-get update

sudo apt-get install -y mongodb-org

常见目录


数据 /var/lib/mongodb 
日志 /var/log/mongodb
配置 /etc/mongod.conf

数据 /var/lib/mongodb

日志 /var/log/mongodb

配置 /etc/mongod.conf

常用命令


sudo systemctl start mongod
sudo systemctl stop mongod
sudo systemctl restart mongod
sudo systemctl status mongod
mongosh

sudo systemctl start mongod

sudo systemctl stop mongod

sudo systemctl restart mongod

sudo systemctl status mongod

mongosh

2.集群配置

MongoDB 的复制集是由一组保持相同数据集的 mongod 进程组成的。复制集提供了冗余和高可用的能力，同时它是所有生产环境部署的基础

在特定情况下可以提高读取性能，因为客户端可以把读取请求发送给不同的服务器。被复制多份数据可以用于灾备或审计

复制集包含几个 data bearing nodes和可选的一个arbiter node。在data bearing nodes上有且仅有一个成员是主节点，主节点接收所有写请求操作，一个主节点会使用 { w: "majority" } 来标识

设置bindIp启用远程访问，生产环境注意提前设置密码和防火墙

https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-net.bindIp

配置集群名

https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-replication.replSetName


mongosh
#初始化集群配置
rs.initiate( {
   _id : "rs0",
   members: [
      { _id: 0, host: "mongodb0.example.net:27017" },
      { _id: 1, host: "mongodb1.example.net:27017" },
      { _id: 2, host: "mongodb2.example.net:27017" }
   ]
})

test> rs.initiate( {
...    _id : "rs0",
...    members: [
...       { _id: 0, host: "test1.mongo.db:27017" },
...       { _id: 1, host: "test2.mongo.db:27017" },
...       { _id: 2, host: "test3.mongo.db:27017" }
...    ]
... })
{ ok: 1 }
rs0 [direct: other] test>

#查看集群配置
rs.conf()

rs0 [direct: other] test> rs.conf()
{
  _id: 'rs0',
  version: 1,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'test1.mongo.db:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 1,
      host: 'test2.mongo.db:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 2,
      host: 'test3.mongo.db:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    }
  ],
  protocolVersion: Long("1"),
  writeConcernMajorityJournalDefault: true,
  settings: {
    chainingAllowed: true,
    heartbeatIntervalMillis: 2000,
    heartbeatTimeoutSecs: 10,
    electionTimeoutMillis: 10000,
    catchUpTimeoutMillis: -1,
    catchUpTakeoverDelayMillis: 30000,
    getLastErrorModes: {},
    getLastErrorDefaults: { w: 1, wtimeout: 0 },
    replicaSetId: ObjectId("639db0b39d9a0c1c17cfaf5a")
  }
}
rs0 [direct: primary] test>


#查看主节点
rs.status()


rs0 [direct: primary] test> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:06:57.005Z"),
  myState: 1,
  term: Long("1"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    lastCommittedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    appliedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    durableOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671278771, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate("2022-12-17T12:06:22.806Z"),
    electionTerm: Long("1"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:06:22.864Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:06:23.645Z")
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 142,
      optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-17T12:06:52.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671278782, i: 1 }),
      electionDate: ISODate("2022-12-17T12:06:22.000Z"),
      configVersion: 1,
      configTerm: 1,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 45,
      optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-17T12:06:52.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.864Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test1.mongo.db:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 1
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 45,
      optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-17T12:06:52.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.860Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test1.mongo.db:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 1
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671278812, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671278812, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

mongosh

#初始化集群配置

rs.initiate( {

_id : "rs0",

members: [

{ _id: 0, host: "mongodb0.example.net:27017" },

{ _id: 1, host: "mongodb1.example.net:27017" },

{ _id: 2, host: "mongodb2.example.net:27017" }

]

})

test> rs.initiate( {

... _id : "rs0",

... members: [

... { _id: 0, host: "test1.mongo.db:27017" },

... { _id: 1, host: "test2.mongo.db:27017" },

... { _id: 2, host: "test3.mongo.db:27017" }

... ]

... })

{ ok: 1 }

rs0 [direct: other] test>

#查看集群配置

rs.conf()

rs0 [direct: other] test> rs.conf()

{

_id: 'rs0',

version: 1,

term: 1,

members: [

{

_id: 0,

host: 'test1.mongo.db:27017',

arbiterOnly: false,

buildIndexes: true,

hidden: false,

priority: 1,

tags: {},

secondaryDelaySecs: Long("0"),

votes: 1

{

_id: 1,

host: 'test2.mongo.db:27017',

arbiterOnly: false,

buildIndexes: true,

hidden: false,

priority: 1,

tags: {},

secondaryDelaySecs: Long("0"),

votes: 1

{

_id: 2,

host: 'test3.mongo.db:27017',

arbiterOnly: false,

buildIndexes: true,

hidden: false,

priority: 1,

tags: {},

secondaryDelaySecs: Long("0"),

votes: 1

}

protocolVersion: Long("1"),

writeConcernMajorityJournalDefault: true,

settings: {

chainingAllowed: true,

heartbeatIntervalMillis: 2000,

heartbeatTimeoutSecs: 10,

electionTimeoutMillis: 10000,

catchUpTimeoutMillis: -1,

catchUpTakeoverDelayMillis: 30000,

getLastErrorModes: {},

getLastErrorDefaults: { w: 1, wtimeout: 0 },

replicaSetId: ObjectId("639db0b39d9a0c1c17cfaf5a")

}

rs0 [direct: primary] test>

#查看主节点

rs.status()

rs0 [direct: primary] test> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:06:57.005Z"),

myState: 1,

term: Long("1"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

lastCommittedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

appliedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

durableOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671278771, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'electionTimeout',

lastElectionDate: ISODate("2022-12-17T12:06:22.806Z"),

electionTerm: Long("1"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:06:22.864Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:06:23.645Z")

members: [

{

_id: 0,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 142,

optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDate: ISODate("2022-12-17T12:06:52.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671278782, i: 1 }),

electionDate: ISODate("2022-12-17T12:06:22.000Z"),

configVersion: 1,

configTerm: 1,

self: true,

lastHeartbeatMessage: ''

{

_id: 1,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 45,

optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDate: ISODate("2022-12-17T12:06:52.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.864Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test1.mongo.db:27017',

syncSourceId: 0,

infoMessage: '',

configVersion: 1,

configTerm: 1

{

_id: 2,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 45,

optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDate: ISODate("2022-12-17T12:06:52.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.860Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test1.mongo.db:27017',

syncSourceId: 0,

infoMessage: '',

configVersion: 1,

configTerm: 1

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671278812, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671278812, i: 1 })

}

3.灾备测试

挂掉1之后到2 上查看集群状态如下，可以发现2自动变成了主


rs0 [direct: primary] test> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:18:35.005Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671279498, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'stepUpRequestSkipDryRun',
    lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 0,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")
  },
  electionParticipantMetrics: {
    votedForCandidate: true,
    electionTerm: Long("1"),
    lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),
    electionCandidateMemberId: 0,
    voteReason: '',
    lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    priorityAtElection: 1
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 0,
      state: 8,
      stateStr: '(not reachable/healthy)',
      uptime: 0,
      optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDate: ISODate("1970-01-01T00:00:00.000Z"),
      optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:17:38.860Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:17:38.860Z"),
      lastHeartbeat: ISODate("2022-12-17T12:18:33.050Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:17:41.884Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: 'Error connecting to test1.mongo.db:27017 (192.168.2.224:27017) :: caused by :: Connection refused',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 839,
      optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:18:28.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671279448, i: 1 }),
      electionDate: ISODate("2022-12-17T12:17:28.000Z"),
      configVersion: 1,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 742,
      optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:18:28.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:18:28.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      lastHeartbeat: ISODate("2022-12-17T12:18:34.909Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:18:33.989Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test2.mongo.db:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671279508, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671279508, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

rs0 [direct: primary] test> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:18:35.005Z"),

myState: 1,

term: Long("2"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

lastCommittedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

appliedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

durableOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671279498, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'stepUpRequestSkipDryRun',

lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),

electionTerm: Long("2"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

priorPrimaryMemberId: 0,

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")

electionParticipantMetrics: {

votedForCandidate: true,

electionTerm: Long("1"),

lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),

electionCandidateMemberId: 0,

voteReason: '',

lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

priorityAtElection: 1

members: [

{

_id: 0,

health: 0,

state: 8,

stateStr: '(not reachable/healthy)',

uptime: 0,

optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDate: ISODate("1970-01-01T00:00:00.000Z"),

optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:17:38.860Z"),

lastDurableWallTime: ISODate("2022-12-17T12:17:38.860Z"),

lastHeartbeat: ISODate("2022-12-17T12:18:33.050Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:17:41.884Z"),

pingMs: Long("0"),

lastHeartbeatMessage: 'Error connecting to test1.mongo.db:27017 (192.168.2.224:27017) :: caused by :: Connection refused',

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

configVersion: 1,

configTerm: 2

{

_id: 1,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 839,

optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:18:28.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671279448, i: 1 }),

electionDate: ISODate("2022-12-17T12:17:28.000Z"),

configVersion: 1,

configTerm: 2,

self: true,

lastHeartbeatMessage: ''

{

_id: 2,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 742,

optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:18:28.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:18:28.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastHeartbeat: ISODate("2022-12-17T12:18:34.909Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:18:33.989Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test2.mongo.db:27017',

syncSourceId: 1,

infoMessage: '',

configVersion: 1,

configTerm: 2

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671279508, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671279508, i: 1 })

}

重启1之后，再次查看，可以发现1重新加入了集群，但身份是从节点


rs0 [direct: primary] test> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:19:42.878Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671279558, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'stepUpRequestSkipDryRun',
    lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 0,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")
  },
  electionParticipantMetrics: {
    votedForCandidate: true,
    electionTerm: Long("1"),
    lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),
    electionCandidateMemberId: 0,
    voteReason: '',
    lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    priorityAtElection: 1
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 5,
      optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:19:39.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastHeartbeat: ISODate("2022-12-17T12:19:41.318Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:19:41.768Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test3.mongo.db:27017',
      syncSourceId: 2,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 907,
      optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:19:39.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671279448, i: 1 }),
      electionDate: ISODate("2022-12-17T12:17:28.000Z"),
      configVersion: 1,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 810,
      optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:19:39.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastHeartbeat: ISODate("2022-12-17T12:19:40.962Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:19:42.044Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test2.mongo.db:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671279579, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671279579, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

rs0 [direct: primary] test> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:19:42.878Z"),

myState: 1,

term: Long("2"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

lastCommittedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

appliedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

durableOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671279558, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'stepUpRequestSkipDryRun',

lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),

electionTerm: Long("2"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

priorPrimaryMemberId: 0,

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")

electionParticipantMetrics: {

votedForCandidate: true,

electionTerm: Long("1"),

lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),

electionCandidateMemberId: 0,

voteReason: '',

lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

priorityAtElection: 1

members: [

{

_id: 0,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 5,

optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:19:39.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastHeartbeat: ISODate("2022-12-17T12:19:41.318Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:19:41.768Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test3.mongo.db:27017',

syncSourceId: 2,

infoMessage: '',

configVersion: 1,

configTerm: 2

{

_id: 1,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 907,

optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:19:39.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671279448, i: 1 }),

electionDate: ISODate("2022-12-17T12:17:28.000Z"),

configVersion: 1,

configTerm: 2,

self: true,

lastHeartbeatMessage: ''

{

_id: 2,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 810,

optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:19:39.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastHeartbeat: ISODate("2022-12-17T12:19:40.962Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:19:42.044Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test2.mongo.db:27017',

syncSourceId: 1,

infoMessage: '',

configVersion: 1,

configTerm: 2

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671279579, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671279579, i: 1 })

}

从2写入点新数据，到1查询测试，可以发现是正常的


rs0 [direct: secondary] test> use sample_mflix
switched to db sample_mflix
rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite1" } )
[
  {
    _id: ObjectId("639db429d29c478bd6ec0507"),
    title: 'The Favourite1',
    genres: [ 'Drama', 'History' ],
    runtime: 121,
    rated: 'R',
    year: 2018,
    directors: [ 'Yorgos Lanthimos' ],
    cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],
    type: 'movie'
  }
]
rs0 [direct: secondary] sample_mflix>

rs0 [direct: secondary] test> use sample_mflix

switched to db sample_mflix

rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite1" } )

[

{

_id: ObjectId("639db429d29c478bd6ec0507"),

title: 'The Favourite1',

genres: [ 'Drama', 'History' ],

runtime: 121,

rated: 'R',

year: 2018,

directors: [ 'Yorgos Lanthimos' ],

cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],

type: 'movie'

}

]

rs0 [direct: secondary] sample_mflix>

挂掉现在的从节点3，写入新数据后再重启3


rs0 [direct: primary] sample_mflix> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:22:41.079Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671279738, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'stepUpRequestSkipDryRun',
    lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 0,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")
  },
  electionParticipantMetrics: {
    votedForCandidate: true,
    electionTerm: Long("1"),
    lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),
    electionCandidateMemberId: 0,
    voteReason: '',
    lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    priorityAtElection: 1
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 183,
      optime: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:22:28.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:22:28.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      lastHeartbeat: ISODate("2022-12-17T12:22:39.460Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:22:39.967Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 1086,
      optime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:22:38.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671279448, i: 1 }),
      electionDate: ISODate("2022-12-17T12:17:28.000Z"),
      configVersion: 1,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 0,
      state: 8,
      stateStr: '(not reachable/healthy)',
      uptime: 0,
      optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDate: ISODate("1970-01-01T00:00:00.000Z"),
      optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:22:28.878Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:22:28.878Z"),
      lastHeartbeat: ISODate("2022-12-17T12:22:39.106Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:22:34.175Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: 'Error connecting to test3.mongo.db:27017 (192.168.2.178:27017) :: caused by :: Connection refused',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671279758, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671279758, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

rs0 [direct: primary] sample_mflix> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:22:41.079Z"),

myState: 1,

term: Long("2"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

lastCommittedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

appliedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

durableOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671279738, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'stepUpRequestSkipDryRun',

lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),

electionTerm: Long("2"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

priorPrimaryMemberId: 0,

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")

electionParticipantMetrics: {

votedForCandidate: true,

electionTerm: Long("1"),

lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),

electionCandidateMemberId: 0,

voteReason: '',

lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

priorityAtElection: 1

members: [

{

_id: 0,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 183,

optime: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:22:28.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:22:28.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastHeartbeat: ISODate("2022-12-17T12:22:39.460Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:22:39.967Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

configVersion: 1,

configTerm: 2

{

_id: 1,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 1086,

optime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:22:38.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671279448, i: 1 }),

electionDate: ISODate("2022-12-17T12:17:28.000Z"),

configVersion: 1,

configTerm: 2,

self: true,

lastHeartbeatMessage: ''

{

_id: 2,

health: 0,

state: 8,

stateStr: '(not reachable/healthy)',

uptime: 0,

optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDate: ISODate("1970-01-01T00:00:00.000Z"),

optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:22:28.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:28.878Z"),

lastHeartbeat: ISODate("2022-12-17T12:22:39.106Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:22:34.175Z"),

pingMs: Long("0"),

lastHeartbeatMessage: 'Error connecting to test3.mongo.db:27017 (192.168.2.178:27017) :: caused by :: Connection refused',

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

configVersion: 1,

configTerm: 2

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671279758, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671279758, i: 1 })

}

可以发现新写入的数据依然可以正常查询


rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite2" } )
[
  {
    _id: ObjectId("639db4b7d29c478bd6ec0508"),
    title: 'The Favourite2',
    genres: [ 'Drama', 'History' ],
    runtime: 121,
    rated: 'R',
    year: 2018,
    directors: [ 'Yorgos Lanthimos' ],
    cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],
    type: 'movie'
  }
]

rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite2" } )

[

{

_id: ObjectId("639db4b7d29c478bd6ec0508"),

title: 'The Favourite2',

genres: [ 'Drama', 'History' ],

runtime: 121,

rated: 'R',

year: 2018,

directors: [ 'Yorgos Lanthimos' ],

cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],

type: 'movie'

}

]

附录


mongo

https://mirrors.tuna.tsinghua.edu.cn

#分配DNS
192.168.2.224 test1.mongo.db
192.168.2.147 test2.mongo.db
192.168.2.178 test3.mongo.db

#远程ssh+免密+禁用密码登录
ssh mongo1@test1.mongo.db
ssh mongo2@test2.mongo.db
ssh mongo3@test3.mongo.db

#安装mongo
wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -
sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

#单机测试
mongosh
端口 27017
db
use test

#集群配置
修改每个机器的bindIp
sudo vim /etc/mongod.conf
sudo service mongod restart

#尝试远程连接
mongosh test1.mongo.db
mongosh test2.mongo.db
mongosh test3.mongo.db

#配置集群名
  replSetName: "rs0"

sudo vim /etc/mongod.conf
sudo service mongod restart

#初始化集群
rs.initiate( {
   _id : "rs0",
   members: [
      { _id: 0, host: "test1.mongo.db:27017" },
      { _id: 1, host: "test2.mongo.db:27017" },
      { _id: 2, host: "test3.mongo.db:27017" }
   ]
})

rs.conf()

rs.status()

#集群测试
use sample_mflix

db.movies.insertOne(
  {
    title: "The Favourite",
    genres: [ "Drama", "History" ],
    runtime: 121,
    rated: "R",
    year: 2018,
    directors: [ "Yorgos Lanthimos" ],
    cast: [ "Olivia Colman", "Emma Stone", "Rachel Weisz" ],
    type: "movie"
  }
)

db.getMongo().setReadPref('secondary')
db.movies.find( { title: "The Favourite" } )

mongo

https://mirrors.tuna.tsinghua.edu.cn

#分配DNS

192.168.2.224 test1.mongo.db

192.168.2.147 test2.mongo.db

192.168.2.178 test3.mongo.db

#远程ssh+免密+禁用密码登录

ssh mongo1@test1.mongo.db

ssh mongo2@test2.mongo.db

ssh mongo3@test3.mongo.db

#安装mongo

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -

sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

sudo apt-get update

sudo apt-get install -y mongodb-org

#单机测试

mongosh

端口 27017

use test

#集群配置

修改每个机器的bindIp

sudo vim /etc/mongod.conf

sudo service mongod restart

#尝试远程连接

mongosh test1.mongo.db

mongosh test2.mongo.db

mongosh test3.mongo.db

#配置集群名

replSetName: "rs0"

sudo vim /etc/mongod.conf

sudo service mongod restart

#初始化集群

rs.initiate( {

_id : "rs0",

members: [

{ _id: 0, host: "test1.mongo.db:27017" },

{ _id: 1, host: "test2.mongo.db:27017" },

{ _id: 2, host: "test3.mongo.db:27017" }

]

})

rs.conf()

rs.status()

#集群测试

use sample_mflix

db.movies.insertOne(

{

title: "The Favourite",

genres: [ "Drama", "History" ],

runtime: 121,

rated: "R",

year: 2018,

directors: [ "Yorgos Lanthimos" ],

cast: [ "Olivia Colman", "Emma Stone", "Rachel Weisz" ],

type: "movie"

}

)

db.getMongo().setReadPref('secondary')

db.movies.find( { title: "The Favourite" } )

参考

https://www.mongodb.com/docs/manual/

https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition

https://www.mongodb.com/docs/manual/replication/

https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set/

https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set-with-keyfile-access-control/#std-label-deploy-repl-set-with-auth

20221127redis-sentinel集群搭建

视频版

https://www.bilibili.com/video/BV1u84y1y7Mu/

1.安装redis

环境信息，按官方文档准备三个节点


redis1
192.168.2.104
ssh redis1@192.168.2.104

redis2
192.168.2.166
ssh redis2@192.168.2.166

redis3
192.168.2.139
ssh redis3@192.168.2.139

redis1

192.168.2.104

ssh redis1@192.168.2.104

redis2

192.168.2.166

ssh redis2@192.168.2.166

redis3

192.168.2.139

ssh redis3@192.168.2.139

执行安装过程


curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list

sudo apt-get update
sudo apt-get install redis redis-sentinel

curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list

sudo apt-get update

sudo apt-get install redis redis-sentinel

2.配置集群复制配置

复制的配置，假设第一个是主节点。这里测试没做安全防护，生产记得加密码和网段限制


sudo vim /etc/redis/redis.conf
#config all nodes
bind * -::1
protected-mode no

#only in redis2 and redis3
replicaof 192.168.2.104 6379

sudo vim /etc/redis/redis.conf

#config all nodes

bind * -::1

protected-mode no

#only in redis2 and redis3

replicaof 192.168.2.104 6379

记得重启，之后测试集群复制情况，正常后继续配置sentinel


redis1@redis1:~$ redis-cli -p 6379
127.0.0.1:6379> get test
"2"
127.0.0.1:6379> set test 1
OK
127.0.0.1:6379> get test
"1"
127.0.0.1:6379> exit
redis1@redis1:~$ redis-cli -h 192.168.2.166 -p 6379
192.168.2.166:6379> get test
"1"
192.168.2.166:6379>
redis1@redis1:~$ ^C
redis1@redis1:~$ redis-cli -h 192.168.2.139 -p 6379
192.168.2.139:6379> get test
"1"
192.168.2.139:6379>

redis1@redis1:~$ redis-cli -p 6379

127.0.0.1:6379> get test

"2"

127.0.0.1:6379> set test 1

127.0.0.1:6379> get test

"1"

127.0.0.1:6379> exit

redis1@redis1:~$ redis-cli -h 192.168.2.166 -p 6379

192.168.2.166:6379> get test

"1"

192.168.2.166:6379>

redis1@redis1:~$ ^C

redis1@redis1:~$ redis-cli -h 192.168.2.139 -p 6379

192.168.2.139:6379> get test

"1"

192.168.2.139:6379>

3.配置sentinel

新建配置文件sentinel.conf 这里测试的是自己新建的，记得关掉系统的service防止干扰。实际应该修改/etc/redis下的配置


port 5000
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

#with ip or port mapping(docker or nat)
sentinel announce-ip <ip>
sentinel announce-port <port>

port 5000

sentinel monitor mymaster 127.0.0.1 6379 2

sentinel down-after-milliseconds mymaster 5000

sentinel failover-timeout mymaster 60000

sentinel parallel-syncs mymaster 1

#with ip or port mapping(docker or nat)

sentinel announce-ip <ip>

sentinel announce-port <port>

启动成功效果


redis1@redis1:~$ redis-sentinel sentinel.conf
3385:X 27 Nov 2022 12:11:35.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3385:X 27 Nov 2022 12:11:35.380 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=3385, just started
3385:X 27 Nov 2022 12:11:35.380 # Configuration loaded
3385:X 27 Nov 2022 12:11:35.381 * Increased maximum number of open files to 10032 (it was originally set to 1024).
3385:X 27 Nov 2022 12:11:35.381 * monotonic clock: POSIX clock_gettime
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 7.0.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 5000
 |    `-._   `._    /     _.-'    |     PID: 3385
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

3385:X 27 Nov 2022 12:11:35.387 * Sentinel new configuration saved on disk
3385:X 27 Nov 2022 12:11:35.387 # Sentinel ID is 9a7a4e3269530706fd1002c3018a3f0c525c25d6
3385:X 27 Nov 2022 12:11:35.387 # +monitor master mymaster 192.168.2.104 6379 quorum 2
3385:X 27 Nov 2022 12:11:52.666 * +sentinel sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379
3385:X 27 Nov 2022 12:11:52.670 * Sentinel new configuration saved on disk
3385:X 27 Nov 2022 12:11:59.182 * +sentinel sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379
3385:X 27 Nov 2022 12:11:59.186 * Sentinel new configuration saved on disk

redis1@redis1:~$ redis-sentinel sentinel.conf

3385:X 27 Nov 2022 12:11:35.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

3385:X 27 Nov 2022 12:11:35.380 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=3385, just started

3385:X 27 Nov 2022 12:11:35.380 # Configuration loaded

3385:X 27 Nov 2022 12:11:35.381 * Increased maximum number of open files to 10032 (it was originally set to 1024).

3385:X 27 Nov 2022 12:11:35.381 * monotonic clock: POSIX clock_gettime

_._

_.-``__ ''-._

_.-`` `. `_. ''-._ Redis 7.0.5 (00000000/0) 64 bit

.-`` .-```. ```\/ _.,_ ''-._

( ' , .-` | `, ) Running in sentinel mode

|`-._`-...-` __...-.``-._|'` _.-'| Port: 5000

| `-._ `._ / _.-' | PID: 3385

`-._ `-._ `-./ _.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' | https://redis.io

`-._ `-._`-.__.-'_.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' |

`-._ `-._`-.__.-'_.-' _.-'

`-._ `-.__.-' _.-'

`-._ _.-'

`-.__.-'

3385:X 27 Nov 2022 12:11:35.387 * Sentinel new configuration saved on disk

3385:X 27 Nov 2022 12:11:35.387 # Sentinel ID is 9a7a4e3269530706fd1002c3018a3f0c525c25d6

3385:X 27 Nov 2022 12:11:35.387 # +monitor master mymaster 192.168.2.104 6379 quorum 2

3385:X 27 Nov 2022 12:11:52.666 * +sentinel sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379

3385:X 27 Nov 2022 12:11:52.670 * Sentinel new configuration saved on disk

3385:X 27 Nov 2022 12:11:59.182 * +sentinel sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379

3385:X 27 Nov 2022 12:11:59.186 * Sentinel new configuration saved on disk

观察端口监听5000


redis1@redis1:~$ netstat -anp|grep 5000
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      3385/redis-sentinel
tcp        0      0 192.168.2.104:55560     192.168.2.139:5000      ESTABLISHED 3385/redis-sentinel
tcp        0      0 192.168.2.104:44506     192.168.2.166:5000      ESTABLISHED 3385/redis-sentinel
tcp        0      0 192.168.2.104:5000      192.168.2.166:36802     ESTABLISHED 3385/redis-sentinel
tcp        0      0 192.168.2.104:5000      192.168.2.139:33910     ESTABLISHED 3385/redis-sentinel
tcp6       0      0 :::5000                 :::*                    LISTEN      3385/redis-sentinel

redis1@redis1:~$ netstat -anp|grep 5000

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 3385/redis-sentinel

tcp 0 0 192.168.2.104:55560 192.168.2.139:5000 ESTABLISHED 3385/redis-sentinel

tcp 0 0 192.168.2.104:44506 192.168.2.166:5000 ESTABLISHED 3385/redis-sentinel

tcp 0 0 192.168.2.104:5000 192.168.2.166:36802 ESTABLISHED 3385/redis-sentinel

tcp 0 0 192.168.2.104:5000 192.168.2.139:33910 ESTABLISHED 3385/redis-sentinel

tcp6 0 0 :::5000 :::* LISTEN 3385/redis-sentinel

查看集群状态


sentinel master mymaster
SENTINEL get-master-addr-by-name mymaster
SENTINEL replicas mymaster
SENTINEL sentinels mymaster

sentinel master mymaster

SENTINEL get-master-addr-by-name mymaster

SENTINEL replicas mymaster

SENTINEL sentinels mymaster

redis1


redis1@redis1:~$ redis-cli -p 5000
127.0.0.1:5000> sentinel master mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) "192.168.2.104"
 5) "port"
 6) "6379"
 7) "runid"
 8) "4097e9a96cdb74dd4239aede54843e33104e6913"
 9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "244"
19) "last-ping-reply"
20) "244"
21) "down-after-milliseconds"
22) "5000"
23) "info-refresh"
24) "3457"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "103880"
29) "config-epoch"
30) "0"
31) "num-slaves"
32) "0"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "60000"
39) "parallel-syncs"
40) "1"

127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster
1) "192.168.2.104"
2) "6379"

127.0.0.1:5000> SENTINEL sentinels mymaster
1)  1) "name"
    2) "ea6cbec52f9c04c6c325414a05562a69b503129c"
    3) "ip"
    4) "192.168.2.166"
    5) "port"
    6) "5000"
    7) "runid"
    8) "ea6cbec52f9c04c6c325414a05562a69b503129c"
    9) "flags"
   10) "sentinel"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "241"
   19) "last-ping-reply"
   20) "241"
   21) "down-after-milliseconds"
   22) "5000"
   23) "last-hello-message"
   24) "763"
   25) "voted-leader"
   26) "?"
   27) "voted-leader-epoch"
   28) "0"
2)  1) "name"
    2) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"
    3) "ip"
    4) "192.168.2.139"
    5) "port"
    6) "5000"
    7) "runid"
    8) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"
    9) "flags"
   10) "sentinel"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "241"
   19) "last-ping-reply"
   20) "241"
   21) "down-after-milliseconds"
   22) "5000"
   23) "last-hello-message"
   24) "425"
   25) "voted-leader"
   26) "?"
   27) "voted-leader-epoch"
   28) "0"

100

101

102

103

104

redis1@redis1:~$ redis-cli -p 5000

127.0.0.1:5000> sentinel master mymaster

1) "name"

2) "mymaster"

3) "ip"

4) "192.168.2.104"

5) "port"

6) "6379"

7) "runid"

8) "4097e9a96cdb74dd4239aede54843e33104e6913"

9) "flags"

10) "master"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "244"

19) "last-ping-reply"

20) "244"

21) "down-after-milliseconds"

22) "5000"

23) "info-refresh"

24) "3457"

25) "role-reported"

26) "master"

27) "role-reported-time"

28) "103880"

29) "config-epoch"

30) "0"

31) "num-slaves"

32) "0"

33) "num-other-sentinels"

34) "2"

35) "quorum"

36) "2"

37) "failover-timeout"

38) "60000"

39) "parallel-syncs"

40) "1"

127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster

1) "192.168.2.104"

2) "6379"

127.0.0.1:5000> SENTINEL sentinels mymaster

1) 1) "name"

2) "ea6cbec52f9c04c6c325414a05562a69b503129c"

3) "ip"

4) "192.168.2.166"

5) "port"

6) "5000"

7) "runid"

8) "ea6cbec52f9c04c6c325414a05562a69b503129c"

9) "flags"

10) "sentinel"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "241"

19) "last-ping-reply"

20) "241"

21) "down-after-milliseconds"

22) "5000"

23) "last-hello-message"

24) "763"

25) "voted-leader"

26) "?"

27) "voted-leader-epoch"

28) "0"

2) 1) "name"

2) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"

3) "ip"

4) "192.168.2.139"

5) "port"

6) "5000"

7) "runid"

8) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"

9) "flags"

10) "sentinel"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "241"

19) "last-ping-reply"

20) "241"

21) "down-after-milliseconds"

22) "5000"

23) "last-hello-message"

24) "425"

25) "voted-leader"

26) "?"

27) "voted-leader-epoch"

28) "0"

redis2


redis1@redis1:~$ redis-cli -h 192.168.2.166 -p 5000
192.168.2.166:5000> sentinel master mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) "192.168.2.104"
 5) "port"
 6) "6379"
 7) "runid"
 8) "4097e9a96cdb74dd4239aede54843e33104e6913"
 9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "979"
19) "last-ping-reply"
20) "979"
21) "down-after-milliseconds"
22) "5000"
23) "info-refresh"
24) "8165"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "188616"
29) "config-epoch"
30) "0"
31) "num-slaves"
32) "0"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "60000"
39) "parallel-syncs"
40) "1"
192.168.2.166:5000> SENTINEL sentinels mymaster
1)  1) "name"
    2) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"
    3) "ip"
    4) "192.168.2.139"
    5) "port"
    6) "5000"
    7) "runid"
    8) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"
    9) "flags"
   10) "sentinel"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "855"
   19) "last-ping-reply"
   20) "855"
   21) "down-after-milliseconds"
   22) "5000"
   23) "last-hello-message"
   24) "166"
   25) "voted-leader"
   26) "?"
   27) "voted-leader-epoch"
   28) "0"
2)  1) "name"
    2) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"
    3) "ip"
    4) "192.168.2.104"
    5) "port"
    6) "5000"
    7) "runid"
    8) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"
    9) "flags"
   10) "sentinel"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "855"
   19) "last-ping-reply"
   20) "855"
   21) "down-after-milliseconds"
   22) "5000"
   23) "last-hello-message"
   24) "64"
   25) "voted-leader"
   26) "?"
   27) "voted-leader-epoch"
   28) "0"
192.168.2.166:5000>

100

redis1@redis1:~$ redis-cli -h 192.168.2.166 -p 5000

192.168.2.166:5000> sentinel master mymaster

1) "name"

2) "mymaster"

3) "ip"

4) "192.168.2.104"

5) "port"

6) "6379"

7) "runid"

8) "4097e9a96cdb74dd4239aede54843e33104e6913"

9) "flags"

10) "master"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "979"

19) "last-ping-reply"

20) "979"

21) "down-after-milliseconds"

22) "5000"

23) "info-refresh"

24) "8165"

25) "role-reported"

26) "master"

27) "role-reported-time"

28) "188616"

29) "config-epoch"

30) "0"

31) "num-slaves"

32) "0"

33) "num-other-sentinels"

34) "2"

35) "quorum"

36) "2"

37) "failover-timeout"

38) "60000"

39) "parallel-syncs"

40) "1"

192.168.2.166:5000> SENTINEL sentinels mymaster

1) 1) "name"

2) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"

3) "ip"

4) "192.168.2.139"

5) "port"

6) "5000"

7) "runid"

8) "095fc33d13e733d7f974faf0c2b2e6aba015fc3f"

9) "flags"

10) "sentinel"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "855"

19) "last-ping-reply"

20) "855"

21) "down-after-milliseconds"

22) "5000"

23) "last-hello-message"

24) "166"

25) "voted-leader"

26) "?"

27) "voted-leader-epoch"

28) "0"

2) 1) "name"

2) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"

3) "ip"

4) "192.168.2.104"

5) "port"

6) "5000"

7) "runid"

8) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"

9) "flags"

10) "sentinel"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "855"

19) "last-ping-reply"

20) "855"

21) "down-after-milliseconds"

22) "5000"

23) "last-hello-message"

24) "64"

25) "voted-leader"

26) "?"

27) "voted-leader-epoch"

28) "0"

192.168.2.166:5000>

redis3


redis1@redis1:~$ redis-cli -h 192.168.2.139 -p 5000
192.168.2.139:5000> sentinel master mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) "192.168.2.104"
 5) "port"
 6) "6379"
 7) "runid"
 8) "4097e9a96cdb74dd4239aede54843e33104e6913"
 9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "571"
19) "last-ping-reply"
20) "571"
21) "down-after-milliseconds"
22) "5000"
23) "info-refresh"
24) "8363"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "239355"
29) "config-epoch"
30) "0"
31) "num-slaves"
32) "0"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "60000"
39) "parallel-syncs"
40) "1"
192.168.2.139:5000> SENTINEL sentinels mymaster
1)  1) "name"
    2) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"
    3) "ip"
    4) "192.168.2.104"
    5) "port"
    6) "5000"
    7) "runid"
    8) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"
    9) "flags"
   10) "sentinel"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "232"
   19) "last-ping-reply"
   20) "232"
   21) "down-after-milliseconds"
   22) "5000"
   23) "last-hello-message"
   24) "753"
   25) "voted-leader"
   26) "?"
   27) "voted-leader-epoch"
   28) "0"
2)  1) "name"
    2) "ea6cbec52f9c04c6c325414a05562a69b503129c"
    3) "ip"
    4) "192.168.2.166"
    5) "port"
    6) "5000"
    7) "runid"
    8) "ea6cbec52f9c04c6c325414a05562a69b503129c"
    9) "flags"
   10) "sentinel"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "232"
   19) "last-ping-reply"
   20) "232"
   21) "down-after-milliseconds"
   22) "5000"
   23) "last-hello-message"
   24) "1740"
   25) "voted-leader"
   26) "?"
   27) "voted-leader-epoch"
   28) "0"

redis1@redis1:~$ redis-cli -h 192.168.2.139 -p 5000

192.168.2.139:5000> sentinel master mymaster

1) "name"

2) "mymaster"

3) "ip"

4) "192.168.2.104"

5) "port"

6) "6379"

7) "runid"

8) "4097e9a96cdb74dd4239aede54843e33104e6913"

9) "flags"

10) "master"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "571"

19) "last-ping-reply"

20) "571"

21) "down-after-milliseconds"

22) "5000"

23) "info-refresh"

24) "8363"

25) "role-reported"

26) "master"

27) "role-reported-time"

28) "239355"

29) "config-epoch"

30) "0"

31) "num-slaves"

32) "0"

33) "num-other-sentinels"

34) "2"

35) "quorum"

36) "2"

37) "failover-timeout"

38) "60000"

39) "parallel-syncs"

40) "1"

192.168.2.139:5000> SENTINEL sentinels mymaster

1) 1) "name"

2) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"

3) "ip"

4) "192.168.2.104"

5) "port"

6) "5000"

7) "runid"

8) "9a7a4e3269530706fd1002c3018a3f0c525c25d6"

9) "flags"

10) "sentinel"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "232"

19) "last-ping-reply"

20) "232"

21) "down-after-milliseconds"

22) "5000"

23) "last-hello-message"

24) "753"

25) "voted-leader"

26) "?"

27) "voted-leader-epoch"

28) "0"

2) 1) "name"

2) "ea6cbec52f9c04c6c325414a05562a69b503129c"

3) "ip"

4) "192.168.2.166"

5) "port"

6) "5000"

7) "runid"

8) "ea6cbec52f9c04c6c325414a05562a69b503129c"

9) "flags"

10) "sentinel"

11) "link-pending-commands"

12) "0"

13) "link-refcount"

14) "1"

15) "last-ping-sent"

16) "0"

17) "last-ok-ping-reply"

18) "232"

19) "last-ping-reply"

20) "232"

21) "down-after-milliseconds"

22) "5000"

23) "last-hello-message"

24) "1740"

25) "voted-leader"

26) "?"

27) "voted-leader-epoch"

28) "0"

3.测试故障迁移

模拟主节点挂掉30s或者直接关掉服务,可以看到自动选主，并且修改slave的指向(redis2日志)

见加粗部分


sudo service redis-server stop
redis-cli -p 6379 DEBUG sleep 30

sudo service redis-server stop

redis-cli -p 6379 DEBUG sleep 30


redis1@redis1:~$ redis-sentinel sentinel.conf
3450:X 27 Nov 2022 12:23:13.821 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3450:X 27 Nov 2022 12:23:13.821 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=3450, just started
3450:X 27 Nov 2022 12:23:13.821 # Configuration loaded
3450:X 27 Nov 2022 12:23:13.822 * Increased maximum number of open files to 10032 (it was originally set to 1024).
3450:X 27 Nov 2022 12:23:13.823 * monotonic clock: POSIX clock_gettime
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 7.0.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 5000
 |    `-._   `._    /     _.-'    |     PID: 3450
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

3450:X 27 Nov 2022 12:23:13.826 # Sentinel ID is 9a7a4e3269530706fd1002c3018a3f0c525c25d6
3450:X 27 Nov 2022 12:23:13.826 # +monitor master mymaster 192.168.2.104 6379 quorum 2
3450:X 27 Nov 2022 12:23:13.829 * +slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:23:13.832 * Sentinel new configuration saved on disk
3450:X 27 Nov 2022 12:23:13.832 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:23:13.836 * Sentinel new configuration saved on disk
3450:X 27 Nov 2022 12:23:18.844 # +sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:23:18.844 # +sdown sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:23:25.263 # -sdown sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:23:37.694 # -sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:26:23.114 # +sdown master mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:26:23.191 # +odown master mymaster 192.168.2.104 6379 #quorum 3/2
3450:X 27 Nov 2022 12:26:23.192 # +new-epoch 4
<strong>3450:X 27 Nov 2022 12:26:23.192 # +try-failover master mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:26:23.197 * Sentinel new configuration saved on disk
3450:X 27 Nov 2022 12:26:23.197 # +vote-for-leader 9a7a4e3269530706fd1002c3018a3f0c525c25d6 4
3450:X 27 Nov 2022 12:26:23.200 # ea6cbec52f9c04c6c325414a05562a69b503129c voted for ea6cbec52f9c04c6c325414a05562a69b503129c 4
3450:X 27 Nov 2022 12:26:23.204 # 095fc33d13e733d7f974faf0c2b2e6aba015fc3f voted for ea6cbec52f9c04c6c325414a05562a69b503129c 4
3450:X 27 Nov 2022 12:26:24.346 # +config-update-from sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379
3450:X 27 Nov 2022 12:26:24.346 # +switch-master mymaster 192.168.2.104 6379 192.168.2.139 6379
3450:X 27 Nov 2022 12:26:24.348 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.139 6379
3450:X 27 Nov 2022 12:26:24.349 * +slave slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379
3450:X 27 Nov 2022 12:26:24.357 * Sentinel new configuration saved on disk
3450:X 27 Nov 2022 12:26:29.371 # +sdown slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379</strong>

redis1@redis1:~$ redis-sentinel sentinel.conf

3450:X 27 Nov 2022 12:23:13.821 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

3450:X 27 Nov 2022 12:23:13.821 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=3450, just started

3450:X 27 Nov 2022 12:23:13.821 # Configuration loaded

3450:X 27 Nov 2022 12:23:13.822 * Increased maximum number of open files to 10032 (it was originally set to 1024).

3450:X 27 Nov 2022 12:23:13.823 * monotonic clock: POSIX clock_gettime

_._

_.-``__ ''-._

_.-`` `. `_. ''-._ Redis 7.0.5 (00000000/0) 64 bit

.-`` .-```. ```\/ _.,_ ''-._

( ' , .-` | `, ) Running in sentinel mode

|`-._`-...-` __...-.``-._|'` _.-'| Port: 5000

| `-._ `._ / _.-' | PID: 3450

`-._ `-._ `-./ _.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' | https://redis.io

`-._ `-._`-.__.-'_.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' |

`-._ `-._`-.__.-'_.-' _.-'

`-._ `-.__.-' _.-'

`-._ _.-'

`-.__.-'

3450:X 27 Nov 2022 12:23:13.826 # Sentinel ID is 9a7a4e3269530706fd1002c3018a3f0c525c25d6

3450:X 27 Nov 2022 12:23:13.826 # +monitor master mymaster 192.168.2.104 6379 quorum 2

3450:X 27 Nov 2022 12:23:13.829 * +slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:23:13.832 * Sentinel new configuration saved on disk

3450:X 27 Nov 2022 12:23:13.832 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:23:13.836 * Sentinel new configuration saved on disk

3450:X 27 Nov 2022 12:23:18.844 # +sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:23:18.844 # +sdown sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:23:25.263 # -sdown sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:23:37.694 # -sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:26:23.114 # +sdown master mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:26:23.191 # +odown master mymaster 192.168.2.104 6379 #quorum 3/2

3450:X 27 Nov 2022 12:26:23.192 # +new-epoch 4

<strong>3450:X 27 Nov 2022 12:26:23.192 # +try-failover master mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:26:23.197 * Sentinel new configuration saved on disk

3450:X 27 Nov 2022 12:26:23.197 # +vote-for-leader 9a7a4e3269530706fd1002c3018a3f0c525c25d6 4

3450:X 27 Nov 2022 12:26:23.200 # ea6cbec52f9c04c6c325414a05562a69b503129c voted for ea6cbec52f9c04c6c325414a05562a69b503129c 4

3450:X 27 Nov 2022 12:26:23.204 # 095fc33d13e733d7f974faf0c2b2e6aba015fc3f voted for ea6cbec52f9c04c6c325414a05562a69b503129c 4

3450:X 27 Nov 2022 12:26:24.346 # +config-update-from sentinel ea6cbec52f9c04c6c325414a05562a69b503129c 192.168.2.166 5000 @ mymaster 192.168.2.104 6379

3450:X 27 Nov 2022 12:26:24.346 # +switch-master mymaster 192.168.2.104 6379 192.168.2.139 6379

3450:X 27 Nov 2022 12:26:24.348 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.139 6379

3450:X 27 Nov 2022 12:26:24.349 * +slave slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379

3450:X 27 Nov 2022 12:26:24.357 * Sentinel new configuration saved on disk

3450:X 27 Nov 2022 12:26:29.371 # +sdown slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379</strong>


redis2@redis2:~$ redis-sentinel sentinel.conf
3098:X 27 Nov 2022 12:23:24.963 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3098:X 27 Nov 2022 12:23:24.963 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=3098, just started
3098:X 27 Nov 2022 12:23:24.963 # Configuration loaded
3098:X 27 Nov 2022 12:23:24.965 * Increased maximum number of open files to 10032 (it was originally set to 1024).
3098:X 27 Nov 2022 12:23:24.965 * monotonic clock: POSIX clock_gettime
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 7.0.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 5000
 |    `-._   `._    /     _.-'    |     PID: 3098
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

3098:X 27 Nov 2022 12:23:24.968 # Sentinel ID is ea6cbec52f9c04c6c325414a05562a69b503129c
3098:X 27 Nov 2022 12:23:24.968 # +monitor master mymaster 192.168.2.104 6379 quorum 2
3098:X 27 Nov 2022 12:23:24.972 * +slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:23:24.975 * Sentinel new configuration saved on disk
3098:X 27 Nov 2022 12:23:24.975 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:23:24.979 * Sentinel new configuration saved on disk
3098:X 27 Nov 2022 12:23:29.998 # +sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:23:37.449 # -sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:23.136 # +sdown master mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:23.237 # +odown master mymaster 192.168.2.104 6379 #quorum 2/2
3098:X 27 Nov 2022 12:26:23.237 # +new-epoch 4
3098:X 27 Nov 2022 12:26:23.237 # +try-failover master mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:23.242 * Sentinel new configuration saved on disk
3098:X 27 Nov 2022 12:26:23.242 # +vote-for-leader ea6cbec52f9c04c6c325414a05562a69b503129c 4
3098:X 27 Nov 2022 12:26:23.248 # 9a7a4e3269530706fd1002c3018a3f0c525c25d6 voted for 9a7a4e3269530706fd1002c3018a3f0c525c25d6 4
3098:X 27 Nov 2022 12:26:23.253 # 095fc33d13e733d7f974faf0c2b2e6aba015fc3f voted for ea6cbec52f9c04c6c325414a05562a69b503129c 4
3098:X 27 Nov 2022 12:26:23.333 # +elected-leader master mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:23.333 # +failover-state-select-slave master mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:23.417 # +selected-slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:23.417 * +failover-state-send-slaveof-noone slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:23.483 * +failover-state-wait-promotion slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:24.307 * Sentinel new configuration saved on disk
<strong>3098:X 27 Nov 2022 12:26:24.307 # +promoted-slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:24.307 # +failover-state-reconf-slaves master mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:24.394 * +slave-reconf-sent slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:25.322 * +slave-reconf-inprog slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:25.323 * +slave-reconf-done slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:25.399 # -odown master mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:25.399 # +failover-end master mymaster 192.168.2.104 6379
3098:X 27 Nov 2022 12:26:25.399 # +switch-master mymaster 192.168.2.104 6379 192.168.2.139 6379
3098:X 27 Nov 2022 12:26:25.400 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.139 6379
3098:X 27 Nov 2022 12:26:25.400 * +slave slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379
3098:X 27 Nov 2022 12:26:25.403 * Sentinel new configuration saved on disk
3098:X 27 Nov 2022 12:26:30.447 # +sdown slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379
3098:X 27 Nov 2022 12:29:48.267 # -sdown slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379
3098:X 27 Nov 2022 12:29:58.242 * +convert-to-slave slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379</strong>

redis2@redis2:~$ redis-sentinel sentinel.conf

3098:X 27 Nov 2022 12:23:24.963 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

3098:X 27 Nov 2022 12:23:24.963 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=3098, just started

3098:X 27 Nov 2022 12:23:24.963 # Configuration loaded

3098:X 27 Nov 2022 12:23:24.965 * Increased maximum number of open files to 10032 (it was originally set to 1024).

3098:X 27 Nov 2022 12:23:24.965 * monotonic clock: POSIX clock_gettime

_._

_.-``__ ''-._

_.-`` `. `_. ''-._ Redis 7.0.5 (00000000/0) 64 bit

.-`` .-```. ```\/ _.,_ ''-._

( ' , .-` | `, ) Running in sentinel mode

|`-._`-...-` __...-.``-._|'` _.-'| Port: 5000

| `-._ `._ / _.-' | PID: 3098

`-._ `-._ `-./ _.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' | https://redis.io

`-._ `-._`-.__.-'_.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' |

`-._ `-._`-.__.-'_.-' _.-'

`-._ `-.__.-' _.-'

`-._ _.-'

`-.__.-'

3098:X 27 Nov 2022 12:23:24.968 # Sentinel ID is ea6cbec52f9c04c6c325414a05562a69b503129c

3098:X 27 Nov 2022 12:23:24.968 # +monitor master mymaster 192.168.2.104 6379 quorum 2

3098:X 27 Nov 2022 12:23:24.972 * +slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:23:24.975 * Sentinel new configuration saved on disk

3098:X 27 Nov 2022 12:23:24.975 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:23:24.979 * Sentinel new configuration saved on disk

3098:X 27 Nov 2022 12:23:29.998 # +sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:23:37.449 # -sdown sentinel 095fc33d13e733d7f974faf0c2b2e6aba015fc3f 192.168.2.139 5000 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:23.136 # +sdown master mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:23.237 # +odown master mymaster 192.168.2.104 6379 #quorum 2/2

3098:X 27 Nov 2022 12:26:23.237 # +new-epoch 4

3098:X 27 Nov 2022 12:26:23.237 # +try-failover master mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:23.242 * Sentinel new configuration saved on disk

3098:X 27 Nov 2022 12:26:23.242 # +vote-for-leader ea6cbec52f9c04c6c325414a05562a69b503129c 4

3098:X 27 Nov 2022 12:26:23.248 # 9a7a4e3269530706fd1002c3018a3f0c525c25d6 voted for 9a7a4e3269530706fd1002c3018a3f0c525c25d6 4

3098:X 27 Nov 2022 12:26:23.253 # 095fc33d13e733d7f974faf0c2b2e6aba015fc3f voted for ea6cbec52f9c04c6c325414a05562a69b503129c 4

3098:X 27 Nov 2022 12:26:23.333 # +elected-leader master mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:23.333 # +failover-state-select-slave master mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:23.417 # +selected-slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:23.417 * +failover-state-send-slaveof-noone slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:23.483 * +failover-state-wait-promotion slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:24.307 * Sentinel new configuration saved on disk

<strong>3098:X 27 Nov 2022 12:26:24.307 # +promoted-slave slave 192.168.2.139:6379 192.168.2.139 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:24.307 # +failover-state-reconf-slaves master mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:24.394 * +slave-reconf-sent slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:25.322 * +slave-reconf-inprog slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:25.323 * +slave-reconf-done slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:25.399 # -odown master mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:25.399 # +failover-end master mymaster 192.168.2.104 6379

3098:X 27 Nov 2022 12:26:25.399 # +switch-master mymaster 192.168.2.104 6379 192.168.2.139 6379

3098:X 27 Nov 2022 12:26:25.400 * +slave slave 192.168.2.166:6379 192.168.2.166 6379 @ mymaster 192.168.2.139 6379

3098:X 27 Nov 2022 12:26:25.400 * +slave slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379

3098:X 27 Nov 2022 12:26:25.403 * Sentinel new configuration saved on disk

3098:X 27 Nov 2022 12:26:30.447 # +sdown slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379

3098:X 27 Nov 2022 12:29:48.267 # -sdown slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379

3098:X 27 Nov 2022 12:29:58.242 * +convert-to-slave slave 192.168.2.104:6379 192.168.2.104 6379 @ mymaster 192.168.2.139 6379</strong>

这时候再看master已经变成了redis3


redis1@redis1:~$ redis-cli -p 5000
127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster
1) "192.168.2.139"
2) "6379"

redis1@redis1:~$ redis-cli -p 5000

127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster

1) "192.168.2.139"

2) "6379"

去redis3改点数据，去redis2看变不变


redis1@redis1:~$ redis-cli -h 192.168.2.139 -p 6379
192.168.2.139:6379> get test
"1"
192.168.2.139:6379> set test 2
OK
192.168.2.139:6379> get test
"2"
192.168.2.139:6379>
redis1@redis1:~$ redis-cli -h 192.168.2.166 -p 6379
192.168.2.166:6379> get test
"2"
192.168.2.166:6379>

redis1@redis1:~$ redis-cli -h 192.168.2.139 -p 6379

192.168.2.139:6379> get test

"1"

192.168.2.139:6379> set test 2

192.168.2.139:6379> get test

"2"

192.168.2.139:6379>

redis1@redis1:~$ redis-cli -h 192.168.2.166 -p 6379

192.168.2.166:6379> get test

"2"

192.168.2.166:6379>

显然redis3改的值可以从redis2读到，说明复制正常，继续尝试恢复redis1看看能不能读到test=2


redis1@redis1:~$ redis-cli -p 6379
127.0.0.1:6379> get test
"2"
127.0.0.1:6379>

redis1@redis1:~$ redis-cli -p 6379

127.0.0.1:6379> get test

"2"

127.0.0.1:6379>

也可以读到，高可用集群搭建完成

附用到的命令


https://mirrors.tuna.tsinghua.edu.cn

redis1
192.168.2.104
ssh redis1@192.168.2.104

redis2
192.168.2.166
ssh redis2@192.168.2.166

redis3
192.168.2.139
ssh redis3@192.168.2.139

6379

sudo vim /etc/redis/redis.conf
#config all nodes
bind * -::1
protected-mode no

#only in redis2 and redis3
replicaof 192.168.2.104 6379

sudo service redis-server restart


vim sentinel.conf
port 5000 
sentinel monitor mymaster 192.168.2.104 6379 2 
sentinel down-after-milliseconds mymaster 5000 
sentinel failover-timeout mymaster 60000 
sentinel parallel-syncs mymaster 1

redis-sentinel sentinel.conf
redis-cli -p 5000
redis-cli -h 192.168.2.166 -p 5000
redis-cli -h 192.168.2.139 -p 5000

SENTINEL master mymaster 
SENTINEL get-master-addr-by-name mymaster 
SENTINEL replicas mymaster 
SENTINEL sentinels mymaster

redis-cli -p 6379
redis-cli -h 192.168.2.166 -p 6379
redis-cli -h 192.168.2.139 -p 6379

https://mirrors.tuna.tsinghua.edu.cn

redis1

192.168.2.104

ssh redis1@192.168.2.104

redis2

192.168.2.166

ssh redis2@192.168.2.166

redis3

192.168.2.139

ssh redis3@192.168.2.139

6379

sudo vim /etc/redis/redis.conf

#config all nodes

bind * -::1

protected-mode no

#only in redis2 and redis3

replicaof 192.168.2.104 6379

sudo service redis-server restart

vim sentinel.conf

port 5000

sentinel monitor mymaster 192.168.2.104 6379 2

sentinel down-after-milliseconds mymaster 5000

sentinel failover-timeout mymaster 60000

sentinel parallel-syncs mymaster 1

redis-sentinel sentinel.conf

redis-cli -p 5000

redis-cli -h 192.168.2.166 -p 5000

redis-cli -h 192.168.2.139 -p 5000

SENTINEL master mymaster

SENTINEL get-master-addr-by-name mymaster

SENTINEL replicas mymaster

SENTINEL sentinels mymaster

redis-cli -p 6379

redis-cli -h 192.168.2.166 -p 6379

redis-cli -h 192.168.2.139 -p 6379

参考

https://redis.io/docs/

https://redis.io/docs/getting-started/installation/install-redis-on-linux/

https://redis.io/docs/management/replication/

https://redis.io/docs/management/sentinel/

20221114mysql-shell部署mysql集群

视频版

https://www.bilibili.com/video/BV1H24y117uh/

1.安装mysql-apt-config并准备基础环境

channel选mysql8, apt update后就能安装 mysql-shell了

获取debian-sys-maint的密码备用

使用/etc/hosts尝试模拟域名或者自行配置DNS

mysqld下的配置增加report_host=mysql1.db,


#/etc/mysql/mysql.conf.d/
report_host=mysql1.db 
#/etc/hosts 192.168.2.190 mysql1.db

#/etc/mysql/mysql.conf.d/

report_host=mysql1.db

#/etc/hosts 192.168.2.190 mysql1.db

2.创建账户

2.1.一个InnoDB Cluster server configuration账户用于配置集群的实例

使用dba.configureInstance()创建

2.2.一个或多个InnoDB Cluster administrator账户可以在集群安装后被用来管理服务器实例

在所有节点加入集群后，使用cluster.setupAdminAccount(),这个命令会创建指定的账号并赋予所需的权限，创建这个账号的事务会写入binlog并发送给集群的其他服务器

2.3一个或多个MySQL Router账户可以被MySQL Router用来连接集群

以上每个账户必须在所有InnoDB Cluster节点同时存在, 并使用相同的用户名和相同的密码

mysqlsh进入终端，在三个机器重复这个过程


mysqlsh>
dba.configureInstance('debian-sys-maint@localhost:3306',{clusterAdmin:"'icadmin'@'localhost'"})
dba.configureInstance('debian-sys-maint@localhost:3306',{clusterAdmin:"'icadmin'@'192.168.2.%'"})

mysqlsh>

dba.configureInstance('debian-sys-maint@localhost:3306',{clusterAdmin:"'icadmin'@'localhost'"})

dba.configureInstance('debian-sys-maint@localhost:3306',{clusterAdmin:"'icadmin'@'192.168.2.%'"})


 MySQL  JS > dba.configureInstance('root@localhost:3306',{clusterAdmin:"'icadmin'@'localhost'"})
Configuring local MySQL instance listening at port 3306 for use in an InnoDB cluster...

This instance reports its own address as mysql1.db:3306
Password for new account: ********
Confirm password: ********

applierWorkerThreads will be set to the default value of 4.

NOTE: Some configuration options need to be fixed:
+----------------------------------------+---------------+----------------+--------------------------------------------------+
| Variable                               | Current Value | Required Value | Note
                 |
+----------------------------------------+---------------+----------------+--------------------------------------------------+
| binlog_transaction_dependency_tracking | COMMIT_ORDER  | WRITESET       | Update the server variable                       |
| enforce_gtid_consistency               | OFF           | ON             | Update read-only variable and restart the server |
| gtid_mode                              | OFF           | ON             | Update read-only variable and restart the server |
| server_id                              | 1             | <unique ID>    | Update read-only variable and restart the server |
+----------------------------------------+---------------+----------------+--------------------------------------------------+

Some variables need to be changed, but cannot be done dynamically on the server.
Do you want to perform the required configuration changes? [y/n]: y
Do you want to restart the instance after configuring it? [y/n]: y

Cluster admin user 'icadmin'@'localhost' created.
Configuring instance...
The instance 'mysql1.db:3306' was configured to be used in an InnoDB cluster.
Restarting MySQL...
NOTE: MySQL server at mysql1.db:3306 was restarted.

MySQL JS > dba.configureInstance('root@localhost:3306',{clusterAdmin:"'icadmin'@'localhost'"})

Configuring local MySQL instance listening at port 3306 for use in an InnoDB cluster...

This instance reports its own address as mysql1.db:3306

Password for new account: ********

Confirm password: ********

applierWorkerThreads will be set to the default value of 4.

NOTE: Some configuration options need to be fixed:

+----------------------------------------+---------------+----------------+--------------------------------------------------+

| Variable | Current Value | Required Value | Note

+----------------------------------------+---------------+----------------+--------------------------------------------------+

| enforce_gtid_consistency | OFF | ON | Update read-only variable and restart the server |

| gtid_mode | OFF | ON | Update read-only variable and restart the server |

+----------------------------------------+---------------+----------------+--------------------------------------------------+

Some variables need to be changed, but cannot be done dynamically on the server.

Do you want to perform the required configuration changes? [y/n]: y

Do you want to restart the instance after configuring it? [y/n]: y

Cluster admin user 'icadmin'@'localhost' created.

Configuring instance...

The instance 'mysql1.db:3306' was configured to be used in an InnoDB cluster.

Restarting MySQL...

NOTE: MySQL server at mysql1.db:3306 was restarted.

3.检查机器

使用dba.checkInstanceConfiguration(instance) 可以检查节点是否满足 Section 7.1, “InnoDB Cluster Requirements” 提到的前置条件，详见 Checking Instance State

使用dba.configureInstance() 来自动配置节点

连接服务器所用的账户必须有这里要求的权限 Configuring InnoDB Cluster Administrator Accounts Manually

三个机器分别检查自己本地和对方


mysqlsh>
dba.checkInstanceConfiguration('icadmin@localhost:3306')
dba.checkInstanceConfiguration('icadmin@mysql1.db:3306')
dba.checkInstanceConfiguration('icadmin@mysql2.db:3306')
dba.checkInstanceConfiguration('icadmin@mysql3.db:3306')
换官方源安装mysql就不报错了，ubuntu的源会有错。。。。

mysqlsh>

dba.checkInstanceConfiguration('icadmin@localhost:3306')

dba.checkInstanceConfiguration('icadmin@mysql1.db:3306')

dba.checkInstanceConfiguration('icadmin@mysql2.db:3306')

dba.checkInstanceConfiguration('icadmin@mysql3.db:3306')

换官方源安装mysql就不报错了，ubuntu的源会有错。。。。

成功会输出这样的


 MySQL  JS > dba.checkInstanceConfiguration('icadmin@mysql3.db:3306')
Please provide the password for 'icadmin@mysql3.db:3306': ********
Save password for 'icadmin@mysql3.db:3306'? [Y]es/[N]o/Ne[v]er (default No): Y
Validating MySQL instance at mysql3.db:3306 for use in an InnoDB cluster...

This instance reports its own address as mysql3.db:3306

Checking whether existing tables comply with Group Replication requirements...
No incompatible tables detected

Checking instance configuration...
Instance configuration is compatible with InnoDB cluster

The instance 'mysql3.db:3306' is valid to be used in an InnoDB cluster.

{
    "status": "ok"
}

MySQL JS > dba.checkInstanceConfiguration('icadmin@mysql3.db:3306')

Please provide the password for 'icadmin@mysql3.db:3306': ********

Save password for 'icadmin@mysql3.db:3306'? [Y]es/[N]o/Ne[v]er (default No): Y

Validating MySQL instance at mysql3.db:3306 for use in an InnoDB cluster...

This instance reports its own address as mysql3.db:3306

Checking whether existing tables comply with Group Replication requirements...

No incompatible tables detected

Checking instance configuration...

Instance configuration is compatible with InnoDB cluster

The instance 'mysql3.db:3306' is valid to be used in an InnoDB cluster.

{

"status": "ok"

}

4.创建集群并添加节点

使用\connect 连接主节点


MySQL  JS > \connect icadmin@mysql1.db:3306
Creating a session to 'icadmin@mysql1.db:3306'
Fetching schema names for auto-completion... Press ^C to stop.
Your MySQL connection id is 18
Server version: 8.0.31 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.
MySQL  mysql1.db:3306 ssl  JS >

MySQL JS > \connect icadmin@mysql1.db:3306

Creating a session to 'icadmin@mysql1.db:3306'

Fetching schema names for auto-completion... Press ^C to stop.

Your MySQL connection id is 18

Server version: 8.0.31 MySQL Community Server - GPL

No default schema selected; type \use <schema> to set one.

MySQL mysql1.db:3306 ssl JS >

使用dba.createCluster() 使用当前 MySQL Shell 连接的节点作为集群的种子


 MySQL  mysql1.db:3306 ssl  JS > dba.createCluster('testCluster', {replicationAllowedHost:'192.168.2.0/24'})

A new InnoDB Cluster will be created on instance 'mysql1.db:3306'.

Validating instance configuration at mysql1.db:3306...

This instance reports its own address as mysql1.db:3306

Instance configuration is suitable.
NOTE: Group Replication will communicate with other members using 'mysql1.db:3306'. Use the localAddress option to override.

Creating InnoDB Cluster 'testCluster' on 'mysql1.db:3306'...

Adding Seed Instance...
Cluster successfully created. Use Cluster.addInstance() to add MySQL instances.
At least 3 instances are needed for the cluster to be able to withstand up to
one server failure.

<Cluster:testCluster>

MySQL mysql1.db:3306 ssl JS > dba.createCluster('testCluster', {replicationAllowedHost:'192.168.2.0/24'})

A new InnoDB Cluster will be created on instance 'mysql1.db:3306'.

Validating instance configuration at mysql1.db:3306...

This instance reports its own address as mysql1.db:3306

Instance configuration is suitable.

NOTE: Group Replication will communicate with other members using 'mysql1.db:3306'. Use the localAddress option to override.

Creating InnoDB Cluster 'testCluster' on 'mysql1.db:3306'...

Adding Seed Instance...

Cluster successfully created. Use Cluster.addInstance() to add MySQL instances.

At least 3 instances are needed for the cluster to be able to withstand up to

one server failure.

<Cluster:testCluster>

使用setupAdminAccount广播管理账号,生产环境的话要注意下安全，这里测试没限定网段


 MySQL  mysql1.db:3306 ssl  JS > var cluster = dba.getCluster()
 MySQL  mysql1.db:3306 ssl  JS > cluster
<Cluster:testCluster>
 MySQL  mysql1.db:3306 ssl  JS > cluster.setupAdminAccount("icadmin")

Missing the password for new account icadmin@%. Please provide one.
Password for new account: ********
Confirm password: ********

Creating user icadmin@%.
Account icadmin@% was successfully created.

 MySQL  mysql1.db:3306 ssl  JS >

MySQL mysql1.db:3306 ssl JS > var cluster = dba.getCluster()

MySQL mysql1.db:3306 ssl JS > cluster

<Cluster:testCluster>

MySQL mysql1.db:3306 ssl JS > cluster.setupAdminAccount("icadmin")

Missing the password for new account icadmin@%. Please provide one.

Password for new account: ********

Confirm password: ********

Creating user icadmin@%.

Account icadmin@% was successfully created.

MySQL mysql1.db:3306 ssl JS >

使用cluster.addInstance()向当前集群内增加节点,C选择完全量复制，I选择增量复制，默认是全量


MySQL  mysql1.db:3306 ssl  JS > cluster.addInstance('icadmin@mysql2.db:3306')

NOTE: The target instance 'mysql2.db:3306' has not been pre-provisioned (GTID set is empty). The Shell is unable to decide whether incremental state recovery can correctly provision it.
The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of 'mysql2.db:3306' with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.

The incremental state recovery may be safely used if you are sure all updates ever executed in the cluster were done with GTIDs enabled, there are no purged transactions and the new instance contains the same GTID set as the cluster or a subset of it. To use this method by default, set the 'recoveryMethod' option to 'incremental'.


Please select a recovery method [C]lone/[I]ncremental recovery/[A]bort (default Clone): C
Validating instance configuration at mysql2.db:3306...

This instance reports its own address as mysql2.db:3306

Instance configuration is suitable.
NOTE: Group Replication will communicate with other members using 'mysql2.db:3306'. Use the localAddress option to override.

A new instance will be added to the InnoDB cluster. Depending on the amount of
data on the cluster this might take from a few seconds to several hours.

Adding instance to the cluster...

Monitoring recovery process of the new cluster member. Press ^C to stop monitoring and let it continue in background.
Clone based state recovery is now in progress.

NOTE: A server restart is expected to happen as part of the clone process. If the
server does not support the RESTART command or does not come back after a
while, you may need to manually start it back.

* Waiting for clone to finish...
NOTE: mysql2.db:3306 is being cloned from mysql1.db:3306
** Stage DROP DATA: Completed
** Clone Transfer
    FILE COPY  ############################################################  100%  Completed
    PAGE COPY  ############################################################  100%  Completed
    REDO COPY  ############################################################  100%  Completed

NOTE: mysql2.db:3306 is shutting down...

* Waiting for server restart... ready
* mysql2.db:3306 has restarted, waiting for clone to finish...
** Stage RESTART: Completed
* Clone process has finished: 73.65 MB transferred in 2 sec (36.83 MB/s)

State recovery already finished for 'mysql2.db:3306'

The instance 'mysql2.db:3306' was successfully added to the cluster.

MySQL mysql1.db:3306 ssl JS > cluster.addInstance('icadmin@mysql2.db:3306')

NOTE: The target instance 'mysql2.db:3306' has not been pre-provisioned (GTID set is empty). The Shell is unable to decide whether incremental state recovery can correctly provision it.

The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of 'mysql2.db:3306' with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.

The incremental state recovery may be safely used if you are sure all updates ever executed in the cluster were done with GTIDs enabled, there are no purged transactions and the new instance contains the same GTID set as the cluster or a subset of it. To use this method by default, set the 'recoveryMethod' option to 'incremental'.

Please select a recovery method [C]lone/[I]ncremental recovery/[A]bort (default Clone): C

Validating instance configuration at mysql2.db:3306...

This instance reports its own address as mysql2.db:3306

Instance configuration is suitable.

NOTE: Group Replication will communicate with other members using 'mysql2.db:3306'. Use the localAddress option to override.

A new instance will be added to the InnoDB cluster. Depending on the amount of

data on the cluster this might take from a few seconds to several hours.

Adding instance to the cluster...

Monitoring recovery process of the new cluster member. Press ^C to stop monitoring and let it continue in background.

Clone based state recovery is now in progress.

NOTE: A server restart is expected to happen as part of the clone process. If the

server does not support the RESTART command or does not come back after a

while, you may need to manually start it back.

* Waiting for clone to finish...

NOTE: mysql2.db:3306 is being cloned from mysql1.db:3306

** Stage DROP DATA: Completed

** Clone Transfer

FILE COPY ############################################################ 100% Completed

PAGE COPY ############################################################ 100% Completed

REDO COPY ############################################################ 100% Completed

NOTE: mysql2.db:3306 is shutting down...

* Waiting for server restart... ready

* mysql2.db:3306 has restarted, waiting for clone to finish...

** Stage RESTART: Completed

* Clone process has finished: 73.65 MB transferred in 2 sec (36.83 MB/s)

State recovery already finished for 'mysql2.db:3306'

The instance 'mysql2.db:3306' was successfully added to the cluster.

使用cluster.status()查看当前集群状态


 MySQL  mysql1.db:3306 ssl  JS > cluster.status()
{
    "clusterName": "testCluster",
    "defaultReplicaSet": {
        "name": "default",
        "primary": "mysql1.db:3306",
        "ssl": "REQUIRED",
        "status": "OK_NO_TOLERANCE",
        "statusText": "Cluster is NOT tolerant to any failures.",
        "topology": {
            "mysql1.db:3306": {
                "address": "mysql1.db:3306",
                "memberRole": "PRIMARY",
                "mode": "R/W",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.0.31"
            },
            "mysql2.db:3306": {
                "address": "mysql2.db:3306",
                "memberRole": "SECONDARY",
                "mode": "R/O",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.0.31"
            }
        },
        "topologyMode": "Single-Primary"
    },
    "groupInformationSourceMember": "mysql1.db:3306"
}

MySQL mysql1.db:3306 ssl JS > cluster.status()

{

"clusterName": "testCluster",

"defaultReplicaSet": {

"name": "default",

"primary": "mysql1.db:3306",

"ssl": "REQUIRED",

"status": "OK_NO_TOLERANCE",

"statusText": "Cluster is NOT tolerant to any failures.",

"topology": {

"mysql1.db:3306": {

"address": "mysql1.db:3306",

"memberRole": "PRIMARY",

"mode": "R/W",

"readReplicas": {},

"replicationLag": "applier_queue_applied",

"role": "HA",

"status": "ONLINE",

"version": "8.0.31"

"mysql2.db:3306": {

"address": "mysql2.db:3306",

"memberRole": "SECONDARY",

"mode": "R/O",

"readReplicas": {},

"replicationLag": "applier_queue_applied",

"role": "HA",

"status": "ONLINE",

"version": "8.0.31"

}

"topologyMode": "Single-Primary"

"groupInformationSourceMember": "mysql1.db:3306"

}

重复这个过程，添加mysql3


 MySQL  mysql1.db:3306 ssl  JS > cluster.addInstance('icadmin@mysql3.db:3306')

NOTE: The target instance 'mysql3.db:3306' has not been pre-provisioned (GTID set is empty). The Shell is unable to decide whether incremental state recovery can correctly provision it.
The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of 'mysql3.db:3306' with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.

The incremental state recovery may be safely used if you are sure all updates ever executed in the cluster were done with GTIDs enabled, there are no purged transactions and the new instance contains the same GTID set as the cluster or a subset of it. To use this method by default, set the 'recoveryMethod' option to 'incremental'.


Please select a recovery method [C]lone/[I]ncremental recovery/[A]bort (default Clone):
Validating instance configuration at mysql3.db:3306...

This instance reports its own address as mysql3.db:3306

Instance configuration is suitable.
NOTE: Group Replication will communicate with other members using 'mysql3.db:3306'. Use the localAddress option to override.

A new instance will be added to the InnoDB cluster. Depending on the amount of
data on the cluster this might take from a few seconds to several hours.

Adding instance to the cluster...

Monitoring recovery process of the new cluster member. Press ^C to stop monitoring and let it continue in background.
Clone based state recovery is now in progress.

NOTE: A server restart is expected to happen as part of the clone process. If the
server does not support the RESTART command or does not come back after a
while, you may need to manually start it back.

* Waiting for clone to finish...
NOTE: mysql3.db:3306 is being cloned from mysql1.db:3306
** Stage DROP DATA: Completed
** Clone Transfer
    FILE COPY  ############################################################  100%  Completed
    PAGE COPY  ############################################################  100%  Completed
    REDO COPY  ############################################################  100%  Completed

NOTE: mysql3.db:3306 is shutting down...

* Waiting for server restart... ready
* mysql3.db:3306 has restarted, waiting for clone to finish...
** Stage RESTART: Completed
* Clone process has finished: 73.65 MB transferred in 2 sec (36.83 MB/s)

State recovery already finished for 'mysql3.db:3306'

The instance 'mysql3.db:3306' was successfully added to the cluster.

MySQL mysql1.db:3306 ssl JS > cluster.addInstance('icadmin@mysql3.db:3306')

NOTE: The target instance 'mysql3.db:3306' has not been pre-provisioned (GTID set is empty). The Shell is unable to decide whether incremental state recovery can correctly provision it.

The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of 'mysql3.db:3306' with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.

Please select a recovery method [C]lone/[I]ncremental recovery/[A]bort (default Clone):

Validating instance configuration at mysql3.db:3306...

This instance reports its own address as mysql3.db:3306

Instance configuration is suitable.

NOTE: Group Replication will communicate with other members using 'mysql3.db:3306'. Use the localAddress option to override.

A new instance will be added to the InnoDB cluster. Depending on the amount of

data on the cluster this might take from a few seconds to several hours.

Adding instance to the cluster...

Monitoring recovery process of the new cluster member. Press ^C to stop monitoring and let it continue in background.

Clone based state recovery is now in progress.

NOTE: A server restart is expected to happen as part of the clone process. If the

server does not support the RESTART command or does not come back after a

while, you may need to manually start it back.

* Waiting for clone to finish...

NOTE: mysql3.db:3306 is being cloned from mysql1.db:3306

** Stage DROP DATA: Completed

** Clone Transfer

FILE COPY ############################################################ 100% Completed

PAGE COPY ############################################################ 100% Completed

REDO COPY ############################################################ 100% Completed

NOTE: mysql3.db:3306 is shutting down...

* Waiting for server restart... ready

* mysql3.db:3306 has restarted, waiting for clone to finish...

** Stage RESTART: Completed

* Clone process has finished: 73.65 MB transferred in 2 sec (36.83 MB/s)

State recovery already finished for 'mysql3.db:3306'

The instance 'mysql3.db:3306' was successfully added to the cluster.

再次查看集群状态，可以看到mysql1是主，mysql2和3是从


 MySQL  mysql1.db:3306 ssl  JS > cluster.status()
{
    "clusterName": "testCluster",
    "defaultReplicaSet": {
        "name": "default",
        "primary": "mysql1.db:3306",
        "ssl": "REQUIRED",
        "status": "OK",
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
        "topology": {
            "mysql1.db:3306": {
                "address": "mysql1.db:3306",
                "memberRole": "PRIMARY",
                "mode": "R/W",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.0.31"
            },
            "mysql2.db:3306": {
                "address": "mysql2.db:3306",
                "memberRole": "SECONDARY",
                "mode": "R/O",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.0.31"
            },
            "mysql3.db:3306": {
                "address": "mysql3.db:3306",
                "memberRole": "SECONDARY",
                "mode": "R/O",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.0.31"
            }
        },
        "topologyMode": "Single-Primary"
    },
    "groupInformationSourceMember": "mysql1.db:3306"
}

MySQL mysql1.db:3306 ssl JS > cluster.status()

{

"clusterName": "testCluster",

"defaultReplicaSet": {

"name": "default",

"primary": "mysql1.db:3306",

"ssl": "REQUIRED",

"status": "OK",

"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",

"topology": {

"mysql1.db:3306": {

"address": "mysql1.db:3306",

"memberRole": "PRIMARY",

"mode": "R/W",

"readReplicas": {},

"replicationLag": "applier_queue_applied",

"role": "HA",

"status": "ONLINE",

"version": "8.0.31"

"mysql2.db:3306": {

"address": "mysql2.db:3306",

"memberRole": "SECONDARY",

"mode": "R/O",

"readReplicas": {},

"replicationLag": "applier_queue_applied",

"role": "HA",

"status": "ONLINE",

"version": "8.0.31"

"mysql3.db:3306": {

"address": "mysql3.db:3306",

"memberRole": "SECONDARY",

"mode": "R/O",

"readReplicas": {},

"replicationLag": "applier_queue_applied",

"role": "HA",

"status": "ONLINE",

"version": "8.0.31"

}

"topologyMode": "Single-Primary"

"groupInformationSourceMember": "mysql1.db:3306"

}

使用 cluster.checkInstanceState() 检查集群内节点状态


 MySQL  mysql1.db:3306 ssl  JS > cluster.checkInstanceState('icadmin@mysql1.db:3306')
Cluster.checkInstanceState: The instance 'mysql1.db:3306' already belongs to the cluster: 'testCluster'. (RuntimeError)
 MySQL  mysql1.db:3306 ssl  JS > cluster.checkInstanceState('icadmin@mysql2.db:3306')
Cluster.checkInstanceState: The instance 'mysql2.db:3306' already belongs to the cluster: 'testCluster'. (RuntimeError)
 MySQL  mysql1.db:3306 ssl  JS > cluster.checkInstanceState('icadmin@mysql3.db:3306')
Cluster.checkInstanceState: The instance 'mysql3.db:3306' already belongs to the cluster: 'testCluster'. (RuntimeError)

MySQL mysql1.db:3306 ssl JS > cluster.checkInstanceState('icadmin@mysql1.db:3306')

Cluster.checkInstanceState: The instance 'mysql1.db:3306' already belongs to the cluster: 'testCluster'. (RuntimeError)

MySQL mysql1.db:3306 ssl JS > cluster.checkInstanceState('icadmin@mysql2.db:3306')

Cluster.checkInstanceState: The instance 'mysql2.db:3306' already belongs to the cluster: 'testCluster'. (RuntimeError)

MySQL mysql1.db:3306 ssl JS > cluster.checkInstanceState('icadmin@mysql3.db:3306')

Cluster.checkInstanceState: The instance 'mysql3.db:3306' already belongs to the cluster: 'testCluster'. (RuntimeError)

5.集群测试

5.1.写入测试

连上mysql1，往mysql1里写下数据测试下，在mysql2和3上查看


mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 11036483-680c-11ed-932f-000c2956aa71 | mysql1.db   |        3306 | ONLINE       | PRIMARY     | 8.0.31         | MySQL                      |
| group_replication_applier | 12c67576-680c-11ed-b216-000c2924d927 | mysql2.db   |        3306 | ONLINE       | SECONDARY   | 8.0.31         | MySQL                      |
| group_replication_applier | 12e569b4-680c-11ed-b942-000c29409b22 | mysql3.db   |        3306 | ONLINE       | SECONDARY   | 8.0.31         | MySQL                      |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

3 rows in set (0.00 sec)


mysql> CREATE DATABASE test;
E test;
CREATE TABLE t1 (c1Query OK, 1 row affected (0.02 sec)

mysql> USE test;
 INT PRIMARY KEY, c2 TEXT NOTDatabase changed
mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY, c2 TEXT NOT NULL);
NSERT INTO t1 VALUES (1, 'Luis');Query OK, 0 rows affected (0.04 sec)

mysql> INSERT INTO t1 VALUES (1, 'Luis');
Query OK, 1 row affected (0.01 sec)

mysql> CREATE DATABASE test;

E test;

CREATE TABLE t1 (c1Query OK, 1 row affected (0.02 sec)

mysql> USE test;

INT PRIMARY KEY, c2 TEXT NOTDatabase changed

mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY, c2 TEXT NOT NULL);

NSERT INTO t1 VALUES (1, 'Luis');Query OK, 0 rows affected (0.04 sec)

mysql> INSERT INTO t1 VALUES (1, 'Luis');

Query OK, 1 row affected (0.01 sec)


mysql> select * from test.t1;
+----+------+
| c1 | c2   |
+----+------+
|  1 | Luis |
+----+------+
1 row in set (0.00 sec)

mysql> select * from test.t1;

+----+------+

| c1 | c2 |

+----+------+

| 1 | Luis |

+----+------+

1 row in set (0.00 sec)

5.2 灾备测试

我们尝试直接停掉mysql1的服务来模拟主库挂掉，在两外两个机器上查看集群状态，可以看到mysql2自动成为了PRIMARY


mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 12c67576-680c-11ed-b216-000c2924d927 | mysql2.db   |        3306 | ONLINE       | PRIMARY     | 8.0.31         | MySQL                      |
| group_replication_applier | 12e569b4-680c-11ed-b942-000c29409b22 | mysql3.db   |        3306 | ONLINE       | SECONDARY   | 8.0.31         | MySQL                      |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
2 rows in set (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

2 rows in set (0.00 sec)

向mysql2写入数据测试


mysql> INSERT INTO test.t1 VALUES (2, 'HylTest');
Query OK, 1 row affected (0.00 sec)

mysql>

mysql> INSERT INTO test.t1 VALUES (2, 'HylTest');

Query OK, 1 row affected (0.00 sec)

mysql>

到mysql3查看，可以看到数据正常


mysql> select * from test.t1;
+----+---------+
| c1 | c2      |
+----+---------+
|  1 | Luis    |
|  2 | HylTest |
+----+---------+
2 rows in set (0.00 sec)

mysql> select * from test.t1;

+----+---------+

| c1 | c2 |

+----+---------+

| 1 | Luis |

| 2 | HylTest |

+----+---------+

2 rows in set (0.00 sec)

重新把mysql1启动，在mysql2也就是当前的主节点查看集群，可以看到mysql1自动成为了SECONDARY


mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 11036483-680c-11ed-932f-000c2956aa71 | mysql1.db   |        3306 | ONLINE       | SECONDARY   | 8.0.31         | MySQL                      |
| group_replication_applier | 12c67576-680c-11ed-b216-000c2924d927 | mysql2.db   |        3306 | ONLINE       | PRIMARY     | 8.0.31         | MySQL                      |
| group_replication_applier | 12e569b4-680c-11ed-b942-000c29409b22 | mysql3.db   |        3306 | ONLINE       | SECONDARY   | 8.0.31         | MySQL                      |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

3 rows in set (0.00 sec)

到mysql1检查下c1=2也就是宕机期间的数据还在不在，可以发现mysql1重新加入集群后会自动恢复宕机期间的数据


mysql> select * from test.t1;
+----+---------+
| c1 | c2      |
+----+---------+
|  1 | Luis    |
|  2 | HylTest |
+----+---------+
2 rows in set (0.00 sec)

mysql> select * from test.t1;

+----+---------+

| c1 | c2 |

+----+---------+

| 1 | Luis |

| 2 | HylTest |

+----+---------+

2 rows in set (0.00 sec)

附录：用到的命令


mysql1
192.168.2.192

mysql2
192.168.2.171

mysql3
192.168.2.219

scp .\mysql-apt-config_0.8.24-1_all.deb mysql1@192.168.2.192:/home/mysql1
scp .\mysql-apt-config_0.8.24-1_all.deb mysql2@192.168.2.171:/home/mysql2
scp .\mysql-apt-config_0.8.24-1_all.deb mysql3@192.168.2.219:/home/mysql3
sudo dpkg -i mysql-apt-config_0.8.24-1_all.deb
sudo apt install -y net-tools mysql-apt-config
#https://mirrors.tuna.tsinghua.edu.cn
sudo sed -i "s@http://.*archive.ubuntu.com@https://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list
sudo sed -i "s@http://.*security.ubuntu.com@https://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list
sudo apt update
sudo apt install -y mysql-server mysql-shell

root
bqRnG3iP
mysql -u root -p 

MySQL 8 uses a new authentication based on improved SHA256-based password methods. It is recommended 
that all new MySQL Server installations use this method going forward. This new authentication       
plugin requires new versions of connectors and clients, with support for this new authentication     
method (caching_sha2_password). Currently MySQL 8 Connectors and community drivers built with        
libmysqlclient21 support this new method. Clients built with older versions of libmysqlclient may    
not be able to connect to the new server.                                                            
                                                                                                     
To retain compatibility with older client software, the default authentication plugin can be set to  
the legacy value (mysql_native_password) This should only be done if required third-party software   
has not been updated to work with the new authentication method. The change will be written to the   
file /etc/mysql/mysql.conf.d/default-auth-override.cnf                                               
                                                                                                     
After installation, the default can be changed by setting the default_authentication_plugin server   
setting.                                                                                             

#for ubuntu default mysql source
sudo cat /etc/mysql/debian.cnf
debian-sys-maint

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf
report_host=mysql1.db
report_host=mysql2.db
report_host=mysql3.db

sudo service mysql restart

#/etc/hosts
192.168.2.192 mysql1.db
192.168.2.171 mysql2.db
192.168.2.219 mysql3.db

icadmin
bqRnG3iP

dba.configureInstance('root@localhost:3306',{clusterAdmin:"'icadmin'@'localhost'"})
dba.configureInstance('root@localhost:3306',{clusterAdmin:"'icadmin'@'192.168.2.%'"})

dba.checkInstanceConfiguration('icadmin@localhost:3306') 
dba.checkInstanceConfiguration('icadmin@mysql1.db:3306')
dba.checkInstanceConfiguration('icadmin@mysql2.db:3306')
dba.checkInstanceConfiguration('icadmin@mysql3.db:3306')

\connect icadmin@mysql1.db:3306
dba.createCluster('testCluster', {replicationAllowedHost:'192.168.2.0/24'})

var cluster = dba.getCluster()
cluster.setupAdminAccount("icadmin")

cluster.addInstance('icadmin@mysql2.db:3306')
cluster.addInstance('icadmin@mysql3.db:3306')

SELECT * FROM performance_schema.replication_group_members;
 
CREATE DATABASE test;
USE test;
CREATE TABLE t1 (c1 INT PRIMARY KEY, c2 TEXT NOT NULL);
INSERT INTO t1 VALUES (1, 'Luis');
INSERT INTO t1 VALUES (2, 'HylTest');
INSERT INTO t1 VALUES (3, 'HylTest3');
INSERT INTO t1 VALUES (4, 'HylTest4');
 
select * from test.t1;

mysql1

192.168.2.192

mysql2

192.168.2.171

mysql3

192.168.2.219

scp .\mysql-apt-config_0.8.24-1_all.deb mysql1@192.168.2.192:/home/mysql1

scp .\mysql-apt-config_0.8.24-1_all.deb mysql2@192.168.2.171:/home/mysql2

scp .\mysql-apt-config_0.8.24-1_all.deb mysql3@192.168.2.219:/home/mysql3

sudo dpkg -i mysql-apt-config_0.8.24-1_all.deb

sudo apt install -y net-tools mysql-apt-config

#https://mirrors.tuna.tsinghua.edu.cn

sudo sed -i "s@http://.*archive.ubuntu.com@https://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list

sudo sed -i "s@http://.*security.ubuntu.com@https://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list

sudo apt update

sudo apt install -y mysql-server mysql-shell

root

bqRnG3iP

mysql -u root -p

MySQL 8 uses a new authentication based on improved SHA256-based password methods. It is recommended

that all new MySQL Server installations use this method going forward. This new authentication

plugin requires new versions of connectors and clients, with support for this new authentication

method (caching_sha2_password). Currently MySQL 8 Connectors and community drivers built with

libmysqlclient21 support this new method. Clients built with older versions of libmysqlclient may

not be able to connect to the new server.

To retain compatibility with older client software, the default authentication plugin can be set to

the legacy value (mysql_native_password) This should only be done if required third-party software

has not been updated to work with the new authentication method. The change will be written to the

file /etc/mysql/mysql.conf.d/default-auth-override.cnf

After installation, the default can be changed by setting the default_authentication_plugin server

setting.

#for ubuntu default mysql source

sudo cat /etc/mysql/debian.cnf

debian-sys-maint

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

report_host=mysql1.db

report_host=mysql2.db

report_host=mysql3.db

sudo service mysql restart

#/etc/hosts

192.168.2.192 mysql1.db

192.168.2.171 mysql2.db

192.168.2.219 mysql3.db

icadmin

bqRnG3iP

dba.configureInstance('root@localhost:3306',{clusterAdmin:"'icadmin'@'localhost'"})

dba.configureInstance('root@localhost:3306',{clusterAdmin:"'icadmin'@'192.168.2.%'"})

dba.checkInstanceConfiguration('icadmin@localhost:3306')

dba.checkInstanceConfiguration('icadmin@mysql1.db:3306')

dba.checkInstanceConfiguration('icadmin@mysql2.db:3306')

dba.checkInstanceConfiguration('icadmin@mysql3.db:3306')

\connect icadmin@mysql1.db:3306

dba.createCluster('testCluster', {replicationAllowedHost:'192.168.2.0/24'})

var cluster = dba.getCluster()

cluster.setupAdminAccount("icadmin")

cluster.addInstance('icadmin@mysql2.db:3306')

cluster.addInstance('icadmin@mysql3.db:3306')

SELECT * FROM performance_schema.replication_group_members;

CREATE DATABASE test;

USE test;

CREATE TABLE t1 (c1 INT PRIMARY KEY, c2 TEXT NOT NULL);

INSERT INTO t1 VALUES (1, 'Luis');

INSERT INTO t1 VALUES (2, 'HylTest');

INSERT INTO t1 VALUES (3, 'HylTest3');

INSERT INTO t1 VALUES (4, 'HylTest4');

select * from test.t1;

ubuntu安装速度测试

安系统时候在均使用清华源情况下居然发现有错,顺便测下怎么给资源能装的快点

#https://mirrors.tuna.tsinghua.edu.cn

都是40G预分配硬盘

是否联网|核数内存|单个CPU核数*CPU个数|测试结果虚拟磁盘名

联网|8c1g| 1c*8 不开启VT-x| 反复安装失败从未见过，可能是CPU内存给少了

联网|8c2g| 1c*8 不开启VT-x| 将近25min，怀疑是CPU给少了 mysql1.vmdk

联网|8c2g| 4c*2 不开启VT-x| 18min，大概10min装完，但是自动升级花了8min mysql1-0.vmdk

断网|8c2g| 4c*2 不开启VT-x| 6min，果然还是断网大法最好，后面都断网装系统好了。这样首次启动联网即可有DHCP下发的IP不影响测试 mysql1-1.vmdk

继续尝试增加核心数看看能否更快

断网|8c2g| 8c*1 不开启VT-x| 5min45s，基本持平，所以CPU数量并不影响这个速度，尝试削减核数 mysql1-2.vmdk

断网|4c2g| 4c*1 不开启VT-x|5min51s，基本持平，所以CPU核数并不影响这个速度，怀疑是单线程，因此试试1c*1 mysql1-3.vmdk

断网|1c2g| 1c*1 不开启VT-x|7min23s，明显下滑，说明不是单线程，至少是2个线程，不然不会这么长时间 mysql1-4.vmdk

断网|2c2g| 2c*1 不开启VT-x|6min22s，回归6min上下了，说明安装的过程确实是2个线程 mysql1-5.vmdk

断网|2c2g| 2c*1 开启VT-x|TODO 有闲心再测估计不会再快了

参考文档

https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-install-linux-quick.html

https://dev.mysql.com/doc/mysql-shell/8.0/en/configuring-router-user.html

https://dev.mysql.com/doc/mysql-shell/8.0/en/innodb-cluster-user-accounts.html#admin-api-configuring-users

https://dev.mysql.com/doc/mysql-shell/8.0/en/check-instance-configuration.html

https://dev.mysql.com/doc/mysql-shell/8.0/en/creating-user-accounts-for-admin-api.html

20221110nas扩容

综述

按之前所说，保留群晖系统，但文件不存群晖而放到HP服务器的硬件raid磁盘阵列

常见方案无非smb nfs sftp webdav等，但我之前想直接搞出个iSCSI挂到群晖只是理论上行，从esxi没能找到这种做法

思考了下，原理是esxi是第一层虚拟化，之后所有的东西已经不再直接和硬件打交道，所以就算能搞出iSCSI协议也有性能损失，还不如继续用更上层协议

现有的openwrt已经是一个linux内核了，理论上装插件就能支持这些。然而不幸的是openwrt原来是未路由器开发的嵌入式系统，但我并没有嵌入式系统的开发和debug经验，在尝试过安装插件各种蜜汁bug后自己编译，依然不行。更坑爹的是编译需要科学上网，编译环境的科学上网依赖openwrt内部插件分流，死循环

详细尝试记录在后面，持续更新中

openwrt尝试

暂时创建thin模式的虚拟磁盘4T，挂到opwnwrt的/data下

和普通服务器操作一样，cfdisk可以方便的创建分区，选GPT即可

mkfs.ext4 /dev/sdb可以格式化磁盘

给nas新建一个用户vim /etc/passwd

echo “nas:*:1000:1000:nas:/tmp:/bin/ash” >>/etc/passwd

passwd nas设置密码

/etc/group 新增组

echo “nas:x:1000:” >> /etc/group

smbpasswd -a nas添加smbpasswd用户

之后进入openwrt的挂载点下新建挂载点后选择自定义挂载位置到/data即可

然后开启openwrt的文件共享，这里使用samba，选择允许用户nas，保存应用即可

之后网页保存

注意，这里如果要扩容的是overlay分区，官方的x86-64镜像无法扩容overlay是已知bug，见https://github.com/openwrt/openwrt/issues/7352
如镜像版本 openwrt-22.03.2-x86-64-generic-squashfs-combined-efi
直接在/etc/rc.local 插入”mount /dev/sda3 /overlay” 就能绕过去，不需要重新编译

lean的版本修过了所以没问题不要浪费时间找乱七八糟的教程了，浪费时间
https://github.com/coolsnowwolf/lede/blob/master/package/system/fstools/patches/0001-fstools-support-extroot-for-non-MTD-rootfs_data.patch

这样操作完发现没权限写入，给加下权限 chmod o+w /data && chmod g+w /data

测速发现只有300Mbps的速度，感觉不对劲看了下openwrt的监控发现是CPU100%,尝试增加CPU核数到2继续测试

2core

尝试升级samba版本到4后发现luci-app-samba4装不上，暂时手动配置server /etc/samba/smb.config.template

[openwrtstorage]
path = /data
valid users = nas
writable = yes
read only = no
create mask = 0666
director mask = 0755
inherit owner = yes

这样虽然速度不太稳定，但勉强能跑满千兆网了，后面再研究为啥

nfs尝试

安装服务 sudo apt install nfs-kernel-server

添加共享 echo “/data 192.168.2.1(rw,sync,no_hide)”>>/etc/exports

修改端口 http://bryanw.tk/2012/specify-nfs-ports-ubuntu-linux/ 注意kernal-server的配置要用–port

rpcinfo -p查看端口

跨防火墙失败，因为ubuntu下的start-stop-daemon加了参数后就无法传递给mountd了，导致端口无法修改

https://wiki.debian.org/SecuringNFS

http://bryanw.tk/2012/specify-nfs-ports-ubuntu-linux/

直接放和nas一个网段，速度依旧跑不满千兆。。。。迷

openmediavault尝试

https://forum.openmediavault.org/

下载iso后新建虚机安装，这次数据盘直接用100G预分配的测速IO

在test下安装启动后没啥问题，修改到管理网的交换机下和nas放一起继续测速

整个界面简介明了，常见协议都支持

写入速度排查

这种情况下测试，客户端本地磁盘原地复制读写速度均大于1Gbps峰值，下载到同网段windows可以跑满1G带宽，可以排除客户端磁盘和网络瓶颈以及服务器网络瓶颈

scp linux->linux同文件的速度是30MBps，约240Mbps，因此怀疑服务器磁盘连续速度有瓶颈

dd测试 dd if=/dev/zero of=./test.data bs=1G count=1 oflag=direct发现结果如下

记录了1+0 的读入
记录了1+0 的写出
1073741824字节（1.1 GB，1.0 GiB）已复制，36.2007 s，29.7 MB/s

果然啊。。。折腾半天居然是这个问题

接下来继续排查是esxi的虚拟化导致，还是硬件故障

在esxi命令行下执行 time dd if=/dev/zero of=test.data bs=100M count=50
50+0 records in
50+0 records out
real 0m 24.03s
user 0m 16.95s
sys 0m 0.00s

平均速度是208MB/s，说明也不是HPraid卡和硬件的瓶颈，硬件本身至少有这个写入速度的能力

那就只能继续怀疑esxi虚拟化带来的问题了。。。。TMD上次win10启动缓慢查了半天就是esxi的锅，这次磁盘写入缓慢没准又是esxi的问题

不过也可能是我哪里没选对？

仔细观察发现esxi提醒的虚拟机操作系统的类型不对，我默认选的都是linux64，它建议我选Debian 10 64。选择后重启系统再试

dd if=/dev/zero of=./test.data bs=1G count=1 oflag=direct
记录了1+0 的读入
记录了1+0 的写出
1073741824字节（1.1 GB，1.0 GiB）已复制，17.7218 s，60.6 MB/s

确实有提升，真的是因为这个选项导致的吗？

swapoff -a 禁掉swap后再试，etc/fstab也注释掉swap分区

dd if=/dev/zero of=./test.data bs=1G count=1 oflag=direct
dd: 1073741824 字节的输入缓冲导致内存耗尽（1.0 GiB）

果然dd的运行依赖内存，更改成10次100M继续测速

dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，25.1924 s，41.6 MB/s

这个速度和windows上网卡320Mbps左右带宽监控的速度能对上

奇怪的是相同的虚拟机磁盘，win10下能有130MBps左右的写入速度，还是在同时读的情况下

所以综合排除应该是linux系统或者是ext4文件系统的问题，在旧NAS上也是单盘ext4相同命令测速如下

dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 7.46678 s, 140 MB/s

uname -a输出如下，尝试从这上面寻找点线索？

Linux openmediavault 5.16.0-0.bpo.4-amd64 #1 SMP PREEMPT Debian 5.16.12-1~bpo11+1 (2022-03-08) x86_64 GNU/Linux

尝试将磁盘改成独立，不受esxi快照影响继续测试如下

root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，4.37703 s，240 MB/s
root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，4.24505 s，247 MB/s
root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，4.25252 s，247 MB/s

root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，3.99055 s，263 MB/s

再次测速如下图，终于能跑满了，但奇怪的是，为啥非独立的NTFS磁盘没这个问题呢？debian linux下挂载的ext4才会这样

thin+独立模式测试

预分配100G测试ok后，继续尝试thin模式，因为我还想省点地方拿来测集群搭建用

root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，21.3584 s，49.1 MB/s
root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，21.3513 s，49.1 MB/s
root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，20.5089 s，51.1 MB/s
root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，17.6997 s，59.2 MB/s
root@openmediavault:~/data# dd if=/dev/zero of=./test.data bs=100M count=10 oflag=direct
记录了10+0 的读入
记录了10+0 的写出
1048576000字节（1.0 GB，1000 MiB）已复制，22.037 s，47.6 MB/s

这样差不多480Mbps的带宽，按这个效果来看的话，后面需要整理下文件做分类，然后弄两块虚拟磁盘都选独立。

thin模式用作长期存储，所以写入慢点也无所谓，同时是按实际大小消耗磁盘阵列，把需要备份的文件存到这里，有硬件raid的保护可以满足安全性提升的目的

预分配的磁盘用作下载热数据+日常使用，或者干脆接着用原来的nas和本地盘也行，反正都是下的电影和番剧，极端情况丢了就丢了，把种子放到HP服务器存一份就行

另外后面mysql集群使用的虚拟硬盘，必须勾选这个独立+预分配，否则会对后续分布式的性能测试带来难以排查的性能问题

参考

https://blog.csdn.net/kk380446/article/details/114213014

https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html

20221029解决esxi启动win10缓慢的bug

无情开评的小伙伴给装的esxi第二次启动win10时间巨长，翻阅论坛后发现是esxi的bug，升级就行，至少是vSphere 6.7 Update 3

触发条件也是完美符合我的场景。。。vmfs6+win10新版+有快照

讨论贴

https://communities.vmware.com/t5/ESXi-Discussions/windows-10-1809-slow/m-p/1424925

官方说明

https://kb.vmware.com/s/article/67426

ESXi670-202210001.zip

国内好人的镜像https://www.dinghui.org/vmware-iso-download.html

vmware -vl
VMware ESXi 6.7.0 build-8169922
VMware ESXi 6.7.0 GA

进ssh执行后升级重启服务器即可

esxcli software vib install -d=/vmfs/volumes/datastore1/patchs/ESXi670-202210001.zip

升级后检查版本

vmware -vl

VMware ESXi 6.7.0 build-20497097
VMware ESXi 6.7.0 Update 3

升级输出如下


Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMW_bootbank_bnxtnet_20.6.101.7-24vmw.670.3.73.14320388, VMW_bootbank_bnxtroce_20.6.101.0-20vmw.670.1.28.10302608, VMW_bootbank_brcmfcoe_11.4.1078.26-14vmw.670.3.159.18828794, VMW_bootbank_elxnet_11.4.1097.0-5vmw.670.3.73.14320388, VMW_bootbank_i40en_1.8.1.9-2vmw.670.3.73.14320388, VMW_bootbank_igbn_0.1.1.0-5vmw.670.3.73.14320388, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608, VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608, VMW_bootbank_ixgben_1.7.1.16-2vmw.670.3.104.16075168, VMW_bootbank_lpfc_11.4.33.26-14vmw.670.3.104.16075168, VMW_bootbank_lsi-mr3_7.708.07.00-3vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt35_09.00.00.00-5vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt3_17.00.02.00-1vmw.670.3.73.14320388, VMW_bootbank_misc-drivers_6.7.0-2.48.13006603, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.670.1.28.10302608, VMW_bootbank_ne1000_0.8.4-2vmw.670.2.48.13006603, VMW_bootbank_nenic_1.0.29.0-1vmw.670.3.73.14320388, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.3.112.16701467, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.3.104.16075168, VMW_bootbank_nfnic_4.0.0.44-0vmw.670.3.104.16075168, VMW_bootbank_nhpsa_2.0.22-3vmw.670.1.28.10302608, VMW_bootbank_nmlx4-core_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-en_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-rdma_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx5-core_4.17.13.1-1vmw.670.3.73.14320388, VMW_bootbank_nmlx5-rdma_4.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_ntg3_4.1.8.0-4vmw.670.3.189.20497097, VMW_bootbank_nvme_1.2.2.28-5vmw.670.3.159.18828794, VMW_bootbank_nvmxnet3_2.0.0.29-1vmw.670.1.28.10302608, VMW_bootbank_qedentv_2.0.6.4-10vmw.670.1.28.10302608, VMW_bootbank_qfle3f_1.0.25.0.2-15vmw.670.3.104.16075168, VMW_bootbank_sfvmk_1.0.0.1003-7vmw.670.3.104.16075168, VMW_bootbank_smartpqi_1.0.1.553-28vmw.670.3.73.14320388, VMW_bootbank_vmkfcoe_1.0.0.1-1vmw.670.1.28.10302608, VMW_bootbank_vmkusb_0.1-4vmw.670.3.159.18828794, VMW_bootbank_vmw-ahci_2.0.7-2vmw.670.3.143.17700523, VMW_bootbank_xhci-xhci_1.0-3vmw.670.3.185.20491463, VMware_bootbank_cpu-microcode_6.7.0-3.170.19898894, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.2-3.89.15160138, VMware_bootbank_esx-base_6.7.0-3.189.20497097, VMware_bootbank_esx-ui_1.43.10-20199807, VMware_bootbank_esx-update_6.7.0-3.189.20497097, VMware_bootbank_esx-xserver_6.7.0-3.189.20497097, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-16vmw.670.1.28.10302608, VMware_bootbank_lsu-intel-vmd-plugin_1.0.0-2vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.670.2.48.13006603, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-13vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-9vmw.670.2.48.13006603, VMware_bootbank_lsu-smartpqi-plugin_1.0.0-4vmw.670.3.159.18828794, VMware_bootbank_native-misc-drivers_6.7.0-3.89.15160138, VMware_bootbank_qlnativefc_3.1.8.0-5vmw.670.3.73.14320388, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2vmw.670.3.116.16713306, VMware_bootbank_vsan_6.7.0-3.189.20235859, VMware_bootbank_vsanhealth_6.7.0-3.189.20235860, VMware_locker_tools-light_12.0.6.20104755-20491463
   VIBs Removed: VMW_bootbank_bnxtnet_20.6.101.7-11vmw.670.0.0.8169922, VMW_bootbank_brcmfcoe_11.4.1078.0-8vmw.670.0.0.8169922, VMW_bootbank_elxnet_11.4.1094.0-5vmw.670.0.0.8169922, VMW_bootbank_i40en_1.3.1-18vmw.670.0.0.8169922, VMW_bootbank_igbn_0.1.0.0-15vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.670.0.0.8169922, VMW_bootbank_iser_1.0.0.0-1vmw.670.0.0.8169922, VMW_bootbank_ixgben_1.4.1-11vmw.670.0.0.8169922, VMW_bootbank_lpfc_11.4.33.1-6vmw.670.0.0.8169922, VMW_bootbank_lsi-mr3_7.702.13.00-4vmw.670.0.0.8169922, VMW_bootbank_lsi-msgpt2_20.00.04.00-4vmw.670.0.0.8169922, VMW_bootbank_lsi-msgpt35_03.00.01.00-10vmw.670.0.0.8169922, VMW_bootbank_lsi-msgpt3_16.00.01.00-1vmw.670.0.0.8169922, VMW_bootbank_misc-drivers_6.7.0-0.0.8169922, VMW_bootbank_mtip32xx-native_3.9.6-1vmw.670.0.0.8169922, VMW_bootbank_ne1000_0.8.3-4vmw.670.0.0.8169922, VMW_bootbank_nenic_1.0.11.0-1vmw.670.0.0.8169922, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.0.0.8169922, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.0.0.8169922, VMW_bootbank_nhpsa_2.0.22-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-core_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-en_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-rdma_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx5-core_4.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx5-rdma_4.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_ntg3_4.1.3.0-1vmw.670.0.0.8169922, VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3_2.0.0.27-1vmw.670.0.0.8169922, VMW_bootbank_qedentv_2.0.6.4-8vmw.670.0.0.8169922, VMW_bootbank_qfle3f_1.0.25.0.2-14vmw.670.0.0.8169922, VMW_bootbank_smartpqi_1.0.1.553-10vmw.670.0.0.8169922, VMW_bootbank_vmkfcoe_1.0.0.0-1vmw.670.0.0.8169922, VMW_bootbank_vmkusb_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmw-ahci_1.2.0-6vmw.670.0.0.8169922, VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922, VMware_bootbank_cpu-microcode_6.7.0-0.0.8169922, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.0-0.0.8169922, VMware_bootbank_esx-base_6.7.0-0.0.8169922, VMware_bootbank_esx-ui_1.25.0-7872652, VMware_bootbank_esx-xserver_6.7.0-0.0.8169922, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-13vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-12vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-8vmw.670.0.0.8169922, VMware_bootbank_native-misc-drivers_6.7.0-0.0.8169922, VMware_bootbank_qlnativefc_3.0.1.0-5vmw.670.0.0.8169922, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922, VMware_bootbank_vsan_6.7.0-0.0.8169922, VMware_bootbank_vsanhealth_6.7.0-0.0.8169922, VMware_locker_tools-light_10.2.0.7253323-8169922
   VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.670.0.0.8169922, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.670.0.0.8169922, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-via_0.3.3-2vmw.670.0.0.8169922, VMW_bootbank_block-cciss_3.6.14-10vmw.670.0.0.8169922, VMW_bootbank_char-random_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.670.0.0.8169922, VMW_bootbank_elxiscsi_11.4.1174.0-2vmw.670.0.0.8169922, VMW_bootbank_hid-hid_1.0-3vmw.670.0.0.8169922, VMW_bootbank_iavmd_1.2.0.1011-2vmw.670.0.0.8169922, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.670.0.0.8169922, VMW_bootbank_lpnic_11.4.59.0-1vmw.670.0.0.8169922, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.670.0.0.8169922, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.670.0.0.8169922, VMW_bootbank_net-bnx2x_1.78.80.v60.12-2vmw.670.0.0.8169922, VMW_bootbank_net-cdc-ether_1.0-3vmw.670.0.0.8169922, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.670.0.0.8169922, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.670.0.0.8169922, VMW_bootbank_net-enic_2.1.2.38-2vmw.670.0.0.8169922, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.670.0.0.8169922, VMW_bootbank_net-forcedeth_0.61-2vmw.670.0.0.8169922, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.670.0.0.8169922, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.670.0.0.8169922, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-nx-nic_5.0.621-5vmw.670.0.0.8169922, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.670.0.0.8169922, VMW_bootbank_net-usbnet_1.0-3vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3-ens_2.0.0.21-1vmw.670.0.0.8169922, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_pvscsi_0.1-2vmw.670.0.0.8169922, VMW_bootbank_qcnic_1.0.2.0.4-1vmw.670.0.0.8169922, VMW_bootbank_qfle3_1.0.50.11-9vmw.670.0.0.8169922, VMW_bootbank_qfle3i_1.0.2.3.9-3vmw.670.0.0.8169922, VMW_bootbank_qflge_1.1.0.11-1vmw.670.0.0.8169922, VMW_bootbank_sata-ahci_3.0-26vmw.670.0.0.8169922, VMW_bootbank_sata-ata-piix_2.12-10vmw.670.0.0.8169922, VMW_bootbank_sata-sata-nv_3.5-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-promise_2.12-3vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil24_1.1-1vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil_2.3-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-svw_2.3-3vmw.670.0.0.8169922, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.670.0.0.8169922, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.670.0.0.8169922, VMW_bootbank_scsi-aic79xx_3.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.670.0.0.8169922, VMW_bootbank_scsi-hpsa_6.0.0.84-3vmw.670.0.0.8169922, VMW_bootbank_scsi-ips_7.12.05-4vmw.670.0.0.8169922, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.670.0.0.8169922, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.670.0.0.8169922, VMW_bootbank_scsi-mpt2sas_19.00.00.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.670.0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922, VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922, VMware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-0.0.8169922, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922, VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: VMW_bootbank_bnxtnet_20.6.101.7-24vmw.670.3.73.14320388, VMW_bootbank_bnxtroce_20.6.101.0-20vmw.670.1.28.10302608, VMW_bootbank_brcmfcoe_11.4.1078.26-14vmw.670.3.159.18828794, VMW_bootbank_elxnet_11.4.1097.0-5vmw.670.3.73.14320388, VMW_bootbank_i40en_1.8.1.9-2vmw.670.3.73.14320388, VMW_bootbank_igbn_0.1.1.0-5vmw.670.3.73.14320388, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608, VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608, VMW_bootbank_ixgben_1.7.1.16-2vmw.670.3.104.16075168, VMW_bootbank_lpfc_11.4.33.26-14vmw.670.3.104.16075168, VMW_bootbank_lsi-mr3_7.708.07.00-3vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt35_09.00.00.00-5vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt3_17.00.02.00-1vmw.670.3.73.14320388, VMW_bootbank_misc-drivers_6.7.0-2.48.13006603, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.670.1.28.10302608, VMW_bootbank_ne1000_0.8.4-2vmw.670.2.48.13006603, VMW_bootbank_nenic_1.0.29.0-1vmw.670.3.73.14320388, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.3.112.16701467, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.3.104.16075168, VMW_bootbank_nfnic_4.0.0.44-0vmw.670.3.104.16075168, VMW_bootbank_nhpsa_2.0.22-3vmw.670.1.28.10302608, VMW_bootbank_nmlx4-core_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-en_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-rdma_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx5-core_4.17.13.1-1vmw.670.3.73.14320388, VMW_bootbank_nmlx5-rdma_4.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_ntg3_4.1.8.0-4vmw.670.3.189.20497097, VMW_bootbank_nvme_1.2.2.28-5vmw.670.3.159.18828794, VMW_bootbank_nvmxnet3_2.0.0.29-1vmw.670.1.28.10302608, VMW_bootbank_qedentv_2.0.6.4-10vmw.670.1.28.10302608, VMW_bootbank_qfle3f_1.0.25.0.2-15vmw.670.3.104.16075168, VMW_bootbank_sfvmk_1.0.0.1003-7vmw.670.3.104.16075168, VMW_bootbank_smartpqi_1.0.1.553-28vmw.670.3.73.14320388, VMW_bootbank_vmkfcoe_1.0.0.1-1vmw.670.1.28.10302608, VMW_bootbank_vmkusb_0.1-4vmw.670.3.159.18828794, VMW_bootbank_vmw-ahci_2.0.7-2vmw.670.3.143.17700523, VMW_bootbank_xhci-xhci_1.0-3vmw.670.3.185.20491463, VMware_bootbank_cpu-microcode_6.7.0-3.170.19898894, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.2-3.89.15160138, VMware_bootbank_esx-base_6.7.0-3.189.20497097, VMware_bootbank_esx-ui_1.43.10-20199807, VMware_bootbank_esx-update_6.7.0-3.189.20497097, VMware_bootbank_esx-xserver_6.7.0-3.189.20497097, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-16vmw.670.1.28.10302608, VMware_bootbank_lsu-intel-vmd-plugin_1.0.0-2vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.670.2.48.13006603, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-13vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-9vmw.670.2.48.13006603, VMware_bootbank_lsu-smartpqi-plugin_1.0.0-4vmw.670.3.159.18828794, VMware_bootbank_native-misc-drivers_6.7.0-3.89.15160138, VMware_bootbank_qlnativefc_3.1.8.0-5vmw.670.3.73.14320388, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2vmw.670.3.116.16713306, VMware_bootbank_vsan_6.7.0-3.189.20235859, VMware_bootbank_vsanhealth_6.7.0-3.189.20235860, VMware_locker_tools-light_12.0.6.20104755-20491463

VIBs Removed: VMW_bootbank_bnxtnet_20.6.101.7-11vmw.670.0.0.8169922, VMW_bootbank_brcmfcoe_11.4.1078.0-8vmw.670.0.0.8169922, VMW_bootbank_elxnet_11.4.1094.0-5vmw.670.0.0.8169922, VMW_bootbank_i40en_1.3.1-18vmw.670.0.0.8169922, VMW_bootbank_igbn_0.1.0.0-15vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.670.0.0.8169922, VMW_bootbank_iser_1.0.0.0-1vmw.670.0.0.8169922, VMW_bootbank_ixgben_1.4.1-11vmw.670.0.0.8169922, VMW_bootbank_lpfc_11.4.33.1-6vmw.670.0.0.8169922, VMW_bootbank_lsi-mr3_7.702.13.00-4vmw.670.0.0.8169922, VMW_bootbank_lsi-msgpt2_20.00.04.00-4vmw.670.0.0.8169922, VMW_bootbank_lsi-msgpt35_03.00.01.00-10vmw.670.0.0.8169922, VMW_bootbank_lsi-msgpt3_16.00.01.00-1vmw.670.0.0.8169922, VMW_bootbank_misc-drivers_6.7.0-0.0.8169922, VMW_bootbank_mtip32xx-native_3.9.6-1vmw.670.0.0.8169922, VMW_bootbank_ne1000_0.8.3-4vmw.670.0.0.8169922, VMW_bootbank_nenic_1.0.11.0-1vmw.670.0.0.8169922, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.0.0.8169922, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.0.0.8169922, VMW_bootbank_nhpsa_2.0.22-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-core_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-en_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-rdma_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx5-core_4.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx5-rdma_4.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_ntg3_4.1.3.0-1vmw.670.0.0.8169922, VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3_2.0.0.27-1vmw.670.0.0.8169922, VMW_bootbank_qedentv_2.0.6.4-8vmw.670.0.0.8169922, VMW_bootbank_qfle3f_1.0.25.0.2-14vmw.670.0.0.8169922, VMW_bootbank_smartpqi_1.0.1.553-10vmw.670.0.0.8169922, VMW_bootbank_vmkfcoe_1.0.0.0-1vmw.670.0.0.8169922, VMW_bootbank_vmkusb_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmw-ahci_1.2.0-6vmw.670.0.0.8169922, VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922, VMware_bootbank_cpu-microcode_6.7.0-0.0.8169922, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.0-0.0.8169922, VMware_bootbank_esx-base_6.7.0-0.0.8169922, VMware_bootbank_esx-ui_1.25.0-7872652, VMware_bootbank_esx-xserver_6.7.0-0.0.8169922, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-13vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-12vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-8vmw.670.0.0.8169922, VMware_bootbank_native-misc-drivers_6.7.0-0.0.8169922, VMware_bootbank_qlnativefc_3.0.1.0-5vmw.670.0.0.8169922, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922, VMware_bootbank_vsan_6.7.0-0.0.8169922, VMware_bootbank_vsanhealth_6.7.0-0.0.8169922, VMware_locker_tools-light_10.2.0.7253323-8169922

VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.670.0.0.8169922, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.670.0.0.8169922, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-via_0.3.3-2vmw.670.0.0.8169922, VMW_bootbank_block-cciss_3.6.14-10vmw.670.0.0.8169922, VMW_bootbank_char-random_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.670.0.0.8169922, VMW_bootbank_elxiscsi_11.4.1174.0-2vmw.670.0.0.8169922, VMW_bootbank_hid-hid_1.0-3vmw.670.0.0.8169922, VMW_bootbank_iavmd_1.2.0.1011-2vmw.670.0.0.8169922, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.670.0.0.8169922, VMW_bootbank_lpnic_11.4.59.0-1vmw.670.0.0.8169922, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.670.0.0.8169922, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.670.0.0.8169922, VMW_bootbank_net-bnx2x_1.78.80.v60.12-2vmw.670.0.0.8169922, VMW_bootbank_net-cdc-ether_1.0-3vmw.670.0.0.8169922, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.670.0.0.8169922, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.670.0.0.8169922, VMW_bootbank_net-enic_2.1.2.38-2vmw.670.0.0.8169922, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.670.0.0.8169922, VMW_bootbank_net-forcedeth_0.61-2vmw.670.0.0.8169922, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.670.0.0.8169922, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.670.0.0.8169922, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-nx-nic_5.0.621-5vmw.670.0.0.8169922, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.670.0.0.8169922, VMW_bootbank_net-usbnet_1.0-3vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3-ens_2.0.0.21-1vmw.670.0.0.8169922, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_pvscsi_0.1-2vmw.670.0.0.8169922, VMW_bootbank_qcnic_1.0.2.0.4-1vmw.670.0.0.8169922, VMW_bootbank_qfle3_1.0.50.11-9vmw.670.0.0.8169922, VMW_bootbank_qfle3i_1.0.2.3.9-3vmw.670.0.0.8169922, VMW_bootbank_qflge_1.1.0.11-1vmw.670.0.0.8169922, VMW_bootbank_sata-ahci_3.0-26vmw.670.0.0.8169922, VMW_bootbank_sata-ata-piix_2.12-10vmw.670.0.0.8169922, VMW_bootbank_sata-sata-nv_3.5-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-promise_2.12-3vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil24_1.1-1vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil_2.3-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-svw_2.3-3vmw.670.0.0.8169922, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.670.0.0.8169922, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.670.0.0.8169922, VMW_bootbank_scsi-aic79xx_3.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.670.0.0.8169922, VMW_bootbank_scsi-hpsa_6.0.0.84-3vmw.670.0.0.8169922, VMW_bootbank_scsi-ips_7.12.05-4vmw.670.0.0.8169922, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.670.0.0.8169922, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.670.0.0.8169922, VMW_bootbank_scsi-mpt2sas_19.00.00.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.670.0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922, VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922, VMware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-0.0.8169922, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922, VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922


尝试解决风扇转速问题

esxcli software vib install -d /vmfs/volumes/datastore1/patchs/ams-esxi6.7-bundle-11.2.0-6.zip
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: HPE_bootbank_amsd_670.11.2.0-12.7535516, HPE_bootbank_cru_670.6.7.10.14-1OEM.670.0.0.7535516, HPE_bootbank_fc-enablement_670.3.20.12-7535516, HPE_bootbank_ilo_670.10.1.0.16-1OEM.670.0.0.7535516, HPE_bootbank_smx-limited_670.03.13.00.3-7535516
   VIBs Removed:
   VIBs Skipped:

尝试解决风扇转速问题

esxcli software vib install -d /vmfs/volumes/datastore1/patchs/ams-esxi6.7-bundle-11.2.0-6.zip

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: HPE_bootbank_amsd_670.11.2.0-12.7535516, HPE_bootbank_cru_670.6.7.10.14-1OEM.670.0.0.7535516, HPE_bootbank_fc-enablement_670.3.20.12-7535516, HPE_bootbank_ilo_670.10.1.0.16-1OEM.670.0.0.7535516, HPE_bootbank_smx-limited_670.03.13.00.3-7535516

VIBs Removed:

VIBs Skipped:

安装后从ilo能看到

Name	Description	Version
amsd	Agentless Management Service for ESXi 6.7	670.11.2.0-12.7535516
cru	CRU Kernel Module for vSphere ESXi 6.7	670.6.7.10.14-1OEM.670.0.0.7535516
fc-enablement	Fiber Channel Enablement Package for ESXi 6.7.0	670.3.20.12-7535516
ilo	Channel Interface Kernel Module for vSphere ESXi 6.7	670.10.1.0.16-1OEM.670.0.0.7535516
lsu-smartpqi-plugin	SMARTPQI LSU Management Plugin	1.0.0-4vmw.670.3.159.18828794
smartpqi	VMware native driver module for Microsemi SmartPqi controllers	1.0.1.553-28vmw.670.3.73.14320388
smx-limited	WBEM Providers Limited for ESXi 6.7	670.03.13.00.3-7535516

20221024mysql集群搭建

背景

有了基础的虚拟化环境，打算从头搭建一个分布式应用环境，学习学习各种集群的搭建过程。在公司这些都是别的部门弄好了直接用的，包括高可用、灾备等特性都有值得我研究学习的地方。

集群方式选择

mysql的集群方法

传统方法是基于源节点二进制日志重放事件，并且要求日志文件和位置在源和副本之间是同步的

更新一点的方法是基于 global transaction identifiers (GTIDs)，GTIDs是事务性的，不需要依赖日志文件和位置就能工作，这极大的简化了许多常见的副本任务.使用GTIDs的方法保证了源和副本之间的一致性，所有的事务只要在源上提交了就一定会被应用到副本上。

基于日志的方法

源节点个每个副本都应该被设置唯一的ID (using the server_id system variable)，顺带一提，每个副本必须设置源节点的host name, log file文件名，文件内的偏移量

这些细节可以通过mysql的session语句控制。MySQL 8.0.23开始是 CHANGE REPLICATION SOURCE TO ，之前是 CHANGE MASTER TO . 这些细节被存储在副本连接的元数据存储中 (see Section 17.2.4, “Relay Log and Replication Metadata Repositories”).

这里有多种不同的方法可以设置副本, 使用某个具体的方法取决于你如何设置以及你是否在源节点已经有想要复制的数据

想要安装多实例的 MySQL, 你可以使用 InnoDB Cluster ，它可以让你使用 MySQL Shell 轻易的管理mysql服务器集群. InnoDB Cluster 用程序化的方式提供了环境包装了MySQL Group Replication，它可以让你轻易的部署一个高可用的Mysql集群。另外 InnoDB Cluster的接口可以无缝集成 MySQL Router, 它可以让你的程序无需编写降级逻辑的情况下连接集群. 对于不需要高可用的相似场景可以使用 InnoDB ReplicaSet. 这里是安装过程here.

这里是一些通用任务来改概括所有步骤:

1.在源节点上开启binlog并配置unique server ID Section 17.1.2.1, “Setting the Replication Source Configuration”.

可以考虑使用 –init-file 执行SET GLOBAL server_id = 2;


SERVER_ID=2
echo "SET GLOBAL server_id = $SERVER_ID;" > /etc/mysql/init.sql
echo "init_file=/etc/mysql/init.sql" >> /etc/mysql/mysql.conf.d/mysqld.cnf
service mysql restart

SERVER_ID=2

echo "SET GLOBAL server_id = $SERVER_ID;" > /etc/mysql/init.sql

echo "init_file=/etc/mysql/init.sql" >> /etc/mysql/mysql.conf.d/mysqld.cnf

service mysql restart

查看这个值


show global variables like 'server_id';

0 1	show global variables like 'server_id';

2.在每个副本上配置a unique server ID. Section 17.1.2.2, “Setting the Replica Configuration”.

重复1

3.可选，为每个副本创建不同的用户用于鉴权 Section 17.1.2.3, “Creating a User for Replication”.


CREATE USER 'repl'@'192.168.2.%' IDENTIFIED BY 'password';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.2.%'

CREATE USER 'repl'@'192.168.2.%' IDENTIFIED BY 'password';

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.2.%'

4.在创建快照或重启副本进程之前，在记录源节点上的binlog当前位置. 当配置你副本节点时你需要这个信息 Section 17.1.2.4, “Obtaining the Replication Source Binary Log Coordinates

在源节点上执行


FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

FLUSH TABLES WITH READ LOCK;

SHOW MASTER STATUS;

记录File | Position

5.如果你在源节点上已经有数据了并且想要同步到副本上，你需要创建数据的快照来复制数据到副本. 你所使用的存储引擎会决定你如何创建快照

当你使用 MyISAM时,你必须在源上停止处理语句来获得read-lock, 在允许源节点继续执行语句之前获取当前binlog的坐标并导出数据。 Section 17.1.2.4, “Obtaining the Replication Source Binary Log Coordinates”.

当你使用 InnoDB, 你不需要获取read-lock，一个够长的事务足够可以传输数据的快照（这么翻译感觉怪怪的，原文： you do not need a read-lock and a transaction that is long enough to transfer the data snapshot is sufficient.） Section 15.19, “InnoDB and MySQL Replication”.

6.在副本上设置连接的源节点，例如host name,、登录凭证、binlog文件名和位置 Section 17.1.2.7, “Setting the Source Configuration on the Replica”.

在副本节点上执行


CHANGE REPLICATION SOURCE TO
SOURCE_HOST='source_host_name',
SOURCE_USER='replication_user_name',
SOURCE_PASSWORD='replication_password',
SOURCE_LOG_FILE='recorded_log_file_name',
SOURCE_LOG_POS=recorded_log_position;

CHANGE REPLICATION SOURCE TO

SOURCE_HOST='source_host_name',

SOURCE_USER='replication_user_name',

SOURCE_PASSWORD='replication_password',

SOURCE_LOG_FILE='recorded_log_file_name',

SOURCE_LOG_POS=recorded_log_position;

7.对源和副本实施特定的安全措施 Section 17.3, “Replication Security”.

基础配置完成后，选择你的场景：

添加一个新的副本并同步空的源节点 Section 17.1.2.6.1, “Setting Up Replication with New Source and Replicas”.
向一个已有的mysql服务器设置新的源Section 17.1.2.6.2, “Setting Up Replication with Existing Data”.
向已有的分布式集群添加副本Section 17.1.2.8, “Adding Replicas to a Replication Environment”.

在管理MySQL集群之前，阅读完整的章节并尝试这里所有的语句，同时熟悉副本的启动选项

Section 13.4.1, “SQL Statements for Controlling Source Servers,

Section 13.4.2, “SQL Statements for Controlling Replica Servers.

Section 17.1.6, “Replication and Binary Logging Options and Variables.

基于GTIDs的方法

TODO

MRG搭建


ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCk4ZxaNgw6+2u6uQrNik11NZ30FMd7eMB1xGAx2T5VB2651fM8/WVF22iwc1A4BwvWVIPv5lkDIGMlYZ41qsnYZY0pioYkZfq2PnE2WrL/R5XHG7Cnuvt61aqy7Zrdwt/p2WEE/dQHN7U4nYCwGTHLaFz6c0YGtUNHzTuRJqjgQ1yMzqQJj+0oWxyNvuhH3n5ejeJuyWc5cP+O2cYDfCStYpA0KlmDjyaAHaJ2M/n+GYOX9q+4s1mIDsjUoCTWJ5OpBflKiKTPUNCEFH3waFsew2N/WF53gyycYCIc11Bdib7XFiSoydLOGh4JP9NZVNeEl5YMSCoHFbZcWEMSKTpwoFFISnkJPnUnYg0V1NeuA8fhx3b58+/TbdPPMCZIO5TCruCBhJK0UD2D64z167wLm4ZEOh4PLwQIYL6n719XpHdlZBKVv6UOmt+Z3nFH4W5jUNAde/+5r6LC3xg6MJeXTmBEO/omO37HCDCYq+4Wq5xru5EsR1t4rXb0lnZ6zX0= hyl-jumpbox\hyl-jumpbox@HYL-JUMPBOX


sudo apt install net-tools mysql-server -y

ssh mysql1@192.168.2.190
ssh mysql2@192.168.2.169
ssh mysql3@192.168.2.216

sudo cat /etc/mysql/debian.cnf

mysql1
192.168.2.190
user     = debian-sys-maint
password = UYbTJYUnTKzEnerQ

mysql2
192.168.2.169
user     = debian-sys-maint
password = NQTc7vw0eAzqrABq

mysql3
192.168.2.216
user     = debian-sys-maint
password = 0y2FH9aksKPTNqbu

192.168.2.190 mysql1
192.168.2.169 mysql2
192.168.2.216 mysql3

mysql -u debian-sys-maint -p
CREATE USER 'mysql'@'localhost' IDENTIFIED BY 'password';
GRANT ALL ON *.* TO 'mysql'@'localhost';

CREATE USER 'mysql'@'192.168.2.%' IDENTIFIED BY 'password';
GRANT ALL ON *.* TO 'mysql'@'192.168.2.%';
flush privileges;

mysql -u mysql -p
password

mysql -u mysql -h 192.168.2.190 -p
mysql -u mysql -h 192.168.2.169 -p
mysql -u mysql -h 192.168.2.216 -p
password

lisenport从127.0.0.1改为0.0.0.0监听所有网卡，或指定IP远程才可以连接3306


#setenforce=0
#打开/etc/selinux/config文件,修改SELINUX=DISABLED（需重启服务器才能生效）


disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"
server_id=1
gtid_mode=ON
enforce_gtid_consistency=ON
plugin_load_add='group_replication.so'
plugin_load_add='mysql_clone.so'
group_replication_group_name="52d4b7bf-56db-11ed-b0fa-000c29409b22"
group_replication_start_on_boot=off
group_replication_local_address="192.168.2.190:33061"
group_replication_group_seeds="192.168.2.190:33061,192.168.2.169:33061,192.168.2.216:33061"
group_replication_bootstrap_group=off

disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"
server_id=2
gtid_mode=ON
enforce_gtid_consistency=ON
plugin_load_add='group_replication.so'
plugin_load_add='mysql_clone.so'
group_replication_group_name="52d4b7bf-56db-11ed-b0fa-000c29409b22"
group_replication_start_on_boot=off
group_replication_local_address="192.168.2.169:33061"
group_replication_group_seeds="192.168.2.190:33061,192.168.2.169:33061,192.168.2.216:33061"
group_replication_bootstrap_group=off

disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"
server_id=3
gtid_mode=ON
enforce_gtid_consistency=ON
plugin_load_add='group_replication.so'
plugin_load_add='mysql_clone.so'
group_replication_group_name="52d4b7bf-56db-11ed-b0fa-000c29409b22"
group_replication_start_on_boot=off
group_replication_local_address="192.168.2.216:33061"
group_replication_group_seeds="192.168.2.190:33061,192.168.2.169:33061,192.168.2.216:33061"
group_replication_bootstrap_group=off

CREATE USER rpl_user@'%' IDENTIFIED WITH mysql_native_password BY 'password';
SET SQL_LOG_BIN=0;
ALTER USER rpl_user@'%' IDENTIFIED WITH mysql_native_password BY 'password';
GRANT REPLICATION SLAVE ON *.* TO rpl_user@'%';
GRANT CONNECTION_ADMIN ON *.* TO rpl_user@'%';
GRANT BACKUP_ADMIN ON *.* TO rpl_user@'%';
GRANT GROUP_REPLICATION_STREAM ON *.* TO rpl_user@'%';
FLUSH PRIVILEGES;
SET SQL_LOG_BIN=1;

SHOW PLUGINS;

master上启动同步后会开始监听33061端口，注意如果有防火墙需要打开，网上的文章不靠谱，没检查监听就去关防火墙是高危操作！
SET GLOBAL group_replication_bootstrap_group=ON;
START GROUP_REPLICATION USER='rpl_user', PASSWORD='password';
SET GLOBAL group_replication_bootstrap_group=OFF;

SELECT * FROM performance_schema.replication_group_members;

CREATE DATABASE test;
USE test;
CREATE TABLE t1 (c1 INT PRIMARY KEY, c2 TEXT NOT NULL);
INSERT INTO t1 VALUES (1, 'Luis');
INSERT INTO t1 VALUES (2, 'HylTest');
INSERT INTO t1 VALUES (3, 'HylTest3');
INSERT INTO t1 VALUES (4, 'HylTest4');

select * from test.t1;
SHOW BINLOG EVENTS;
truncate table test.t1;

slave上加入group
CHANGE REPLICATION SOURCE TO SOURCE_USER='rpl_user', SOURCE_PASSWORD='password' FOR CHANNEL 'group_replication_recovery';
START GROUP_REPLICATION USER='rpl_user', PASSWORD='password';

#停止同步
STOP GROUP_REPLICATION;

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCk4ZxaNgw6+2u6uQrNik11NZ30FMd7eMB1xGAx2T5VB2651fM8/WVF22iwc1A4BwvWVIPv5lkDIGMlYZ41qsnYZY0pioYkZfq2PnE2WrL/R5XHG7Cnuvt61aqy7Zrdwt/p2WEE/dQHN7U4nYCwGTHLaFz6c0YGtUNHzTuRJqjgQ1yMzqQJj+0oWxyNvuhH3n5ejeJuyWc5cP+O2cYDfCStYpA0KlmDjyaAHaJ2M/n+GYOX9q+4s1mIDsjUoCTWJ5OpBflKiKTPUNCEFH3waFsew2N/WF53gyycYCIc11Bdib7XFiSoydLOGh4JP9NZVNeEl5YMSCoHFbZcWEMSKTpwoFFISnkJPnUnYg0V1NeuA8fhx3b58+/TbdPPMCZIO5TCruCBhJK0UD2D64z167wLm4ZEOh4PLwQIYL6n719XpHdlZBKVv6UOmt+Z3nFH4W5jUNAde/+5r6LC3xg6MJeXTmBEO/omO37HCDCYq+4Wq5xru5EsR1t4rXb0lnZ6zX0= hyl-jumpbox\hyl-jumpbox@HYL-JUMPBOX

sudo apt install net-tools mysql-server -y

ssh mysql1@192.168.2.190

ssh mysql2@192.168.2.169

ssh mysql3@192.168.2.216

sudo cat /etc/mysql/debian.cnf

mysql1

192.168.2.190

user = debian-sys-maint

password = UYbTJYUnTKzEnerQ

mysql2

192.168.2.169

user = debian-sys-maint

password = NQTc7vw0eAzqrABq

mysql3

192.168.2.216

user = debian-sys-maint

password = 0y2FH9aksKPTNqbu

192.168.2.190 mysql1

192.168.2.169 mysql2

192.168.2.216 mysql3

mysql -u debian-sys-maint -p

CREATE USER 'mysql'@'localhost' IDENTIFIED BY 'password';

GRANT ALL ON *.* TO 'mysql'@'localhost';

CREATE USER 'mysql'@'192.168.2.%' IDENTIFIED BY 'password';

GRANT ALL ON *.* TO 'mysql'@'192.168.2.%';

flush privileges;

mysql -u mysql -p

password

mysql -u mysql -h 192.168.2.190 -p

mysql -u mysql -h 192.168.2.169 -p

mysql -u mysql -h 192.168.2.216 -p

password

lisenport从127.0.0.1改为0.0.0.0监听所有网卡，或指定IP远程才可以连接3306

#setenforce=0

#打开/etc/selinux/config文件,修改SELINUX=DISABLED（需重启服务器才能生效）

disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"

server_id=1

gtid_mode=ON

enforce_gtid_consistency=ON

plugin_load_add='group_replication.so'

plugin_load_add='mysql_clone.so'

group_replication_group_name="52d4b7bf-56db-11ed-b0fa-000c29409b22"

group_replication_start_on_boot=off

group_replication_local_address="192.168.2.190:33061"

group_replication_group_seeds="192.168.2.190:33061,192.168.2.169:33061,192.168.2.216:33061"

group_replication_bootstrap_group=off

disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"

server_id=2

gtid_mode=ON

enforce_gtid_consistency=ON

plugin_load_add='group_replication.so'

plugin_load_add='mysql_clone.so'

group_replication_group_name="52d4b7bf-56db-11ed-b0fa-000c29409b22"

group_replication_start_on_boot=off

group_replication_local_address="192.168.2.169:33061"

group_replication_group_seeds="192.168.2.190:33061,192.168.2.169:33061,192.168.2.216:33061"

group_replication_bootstrap_group=off

disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"

server_id=3

gtid_mode=ON

enforce_gtid_consistency=ON

plugin_load_add='group_replication.so'

plugin_load_add='mysql_clone.so'

group_replication_group_name="52d4b7bf-56db-11ed-b0fa-000c29409b22"

group_replication_start_on_boot=off

group_replication_local_address="192.168.2.216:33061"

group_replication_group_seeds="192.168.2.190:33061,192.168.2.169:33061,192.168.2.216:33061"

group_replication_bootstrap_group=off

CREATE USER rpl_user@'%' IDENTIFIED WITH mysql_native_password BY 'password';

SET SQL_LOG_BIN=0;

ALTER USER rpl_user@'%' IDENTIFIED WITH mysql_native_password BY 'password';

GRANT REPLICATION SLAVE ON *.* TO rpl_user@'%';

GRANT CONNECTION_ADMIN ON *.* TO rpl_user@'%';

GRANT BACKUP_ADMIN ON *.* TO rpl_user@'%';

GRANT GROUP_REPLICATION_STREAM ON *.* TO rpl_user@'%';

FLUSH PRIVILEGES;

SET SQL_LOG_BIN=1;

SHOW PLUGINS;

master上启动同步后会开始监听33061端口，注意如果有防火墙需要打开，网上的文章不靠谱，没检查监听就去关防火墙是高危操作！

SET GLOBAL group_replication_bootstrap_group=ON;

START GROUP_REPLICATION USER='rpl_user', PASSWORD='password';

SET GLOBAL group_replication_bootstrap_group=OFF;

SELECT * FROM performance_schema.replication_group_members;

CREATE DATABASE test;

USE test;

CREATE TABLE t1 (c1 INT PRIMARY KEY, c2 TEXT NOT NULL);

INSERT INTO t1 VALUES (1, 'Luis');

INSERT INTO t1 VALUES (2, 'HylTest');

INSERT INTO t1 VALUES (3, 'HylTest3');

INSERT INTO t1 VALUES (4, 'HylTest4');

select * from test.t1;

SHOW BINLOG EVENTS;

truncate table test.t1;

slave上加入group

CHANGE REPLICATION SOURCE TO SOURCE_USER='rpl_user', SOURCE_PASSWORD='password' FOR CHANNEL 'group_replication_recovery';

START GROUP_REPLICATION USER='rpl_user', PASSWORD='password';

#停止同步

STOP GROUP_REPLICATION;

注意，任意节点脱离group后需要手动执行start group_replication重新加入，执行前的数据保留在离开group时的状态

例如mysql2重新加入group变化如下


mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 5b1e6080-578f-11ed-bd92-000c2956aa71 | mysql1      |        3306 | ONLINE       | PRIMARY     | 8.0.31         | XCom                       |
| group_replication_applier | 5c6b9aaa-578f-11ed-946f-000c2924d927 | mysql2      |        3306 | RECOVERING   | SECONDARY   | 8.0.31         | XCom                       |
| group_replication_applier | 8153354e-578f-11ed-87f7-000c29409b22 | mysql3      |        3306 | ONLINE       | SECONDARY   | 8.0.31         | XCom                       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 5b1e6080-578f-11ed-bd92-000c2956aa71 | mysql1      |        3306 | ONLINE       | PRIMARY     | 8.0.31         | XCom                       |
| group_replication_applier | 5c6b9aaa-578f-11ed-946f-000c2924d927 | mysql2      |        3306 | ONLINE       | SECONDARY   | 8.0.31         | XCom                       |
| group_replication_applier | 8153354e-578f-11ed-87f7-000c29409b22 | mysql3      |        3306 | ONLINE       | SECONDARY   | 8.0.31         | XCom                       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.01 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

3 rows in set (0.00 sec)

mysql> SELECT * FROM performance_schema.replication_group_members;

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+

3 rows in set (0.01 sec)

尝试mysql shell自动配置

https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-innodb-cluster.html

参考文档

https://dev.mysql.com/doc/refman/8.0/en/replication.html

https://dev.mysql.com/doc/refman/8.0/en/group-replication-deploying-in-single-primary-mode.html

https://ethendev.github.io/2018/12/17/JPA-MySQL-read-write-separation/

https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-innodb-cluster.html