956237586 – 第 5 页 – HYL Studio

20230407kubekey搭建k8s

视频版

https://www.bilibili.com/video/BV1gL411Y75L

背景

20230204尝试搭建k8s青春版

上次尝试了k3s，本来计划是尝试原版k8s，但看了下文档感觉略麻烦，我决定放弃。。。。

按上次eric的建议这次尝试玩下kubekey

安装准备

环境要求

Ubuntu 16.04，18.04，20.04CPU：2 核，内存：4 G，硬盘：40 G

机器节点2c4g * 3

所有节点必须都能通过 SSH 访问。
所有节点时间同步。
所有节点都应使用 sudo/curl/openssl/tar

域名分配

k8s1.k8s.local

k8s2.k8s.local

k8s3.k8s.local

kubesphere.k8s.local

生成config


k8s1@k8s1:~$ export KKZONE=cn
k8s1@k8s1:~$ curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.7 sh -
Downloading kubekey v3.0.7 from https://kubernetes.pek3b.qingstor.com/kubekey/releases/download/v3.0.7/kubekey-v3.0.7-linux-amd64.tar.gz ...
Kubekey v3.0.7 Download Complete!

k8s1@k8s1:~$ chmod +x kk
k8s1@k8s1:~$ ./kk create config --with-kubesphere --with-kubesphere
Generate KubeKey config file successfully
k8s1@k8s1:~$ ls
config-sample.yaml  kk  kubekey-v3.0.7-linux-amd64.tar.gz
k8s1@k8s1:~$

k8s1@k8s1:~$ export KKZONE=cn

k8s1@k8s1:~$ curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.7 sh -

Downloading kubekey v3.0.7 from https://kubernetes.pek3b.qingstor.com/kubekey/releases/download/v3.0.7/kubekey-v3.0.7-linux-amd64.tar.gz ...

Kubekey v3.0.7 Download Complete!

k8s1@k8s1:~$ chmod +x kk

k8s1@k8s1:~$ ./kk create config --with-kubesphere --with-kubesphere

Generate KubeKey config file successfully

k8s1@k8s1:~$ ls

config-sample.yaml kk kubekey-v3.0.7-linux-amd64.tar.gz

k8s1@k8s1:~$

修改config


k8s1@k8s1:~$ vim config-sample.yaml
k8s1@k8s1:~$ cat config-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: master, address: 192.168.2.206, internalAddress: 192.168.2.206, user: k8s1, password: 123456}
  - {name: node1, address: 192.168.2.177, internalAddress: 192.168.2.177, user: k8s2, password: 123456}
  - {name: node2, address: 192.168.2.203, internalAddress: 192.168.2.203, user: k8s3, password: 123456}
  roleGroups:
    etcd:
    - master
    control-plane:
    - master
    worker:
    - node1
    - node2
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers
    # internalLoadbalancer: haproxy

    domain: kubesphere.k8s.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.23.10
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: docker
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []



---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.3.2
spec:
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  zone: ""
  local_registry: ""
  namespace_override: ""
  # dev_tag: ""
  etcd:
    monitoring: false
    endpointIps: localhost
    port: 2379
    tlsEnable: true
  common:
    core:
      console:
        enableMultiLogin: true
        port: 30880
        type: NodePort
    # apiserver:
    #  resources: {}
    # controllerManager:
    #  resources: {}
    redis:
      enabled: false
      volumeSize: 2Gi
    openldap:
      enabled: false
      volumeSize: 2Gi
    minio:
      volumeSize: 20Gi
    monitoring:
      # type: external
      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
      GPUMonitoring:
        enabled: false
    gpu:
      kinds:
      - resourceName: "nvidia.com/gpu"
        resourceType: "GPU"
        default: true
    es:
      # master:
      #   volumeSize: 4Gi
      #   replicas: 1
      #   resources: {}
      # data:
      #   volumeSize: 20Gi
      #   replicas: 1
      #   resources: {}
      logMaxAge: 7
      elkPrefix: logstash
      basicAuth:
        enabled: false
        username: ""
        password: ""
      externalElasticsearchHost: ""
      externalElasticsearchPort: ""
  alerting:
    enabled: false
    # thanosruler:
    #   replicas: 1
    #   resources: {}
  auditing:
    enabled: false
    # operator:
    #   resources: {}
    # webhook:
    #   resources: {}
  devops:
    enabled: false
    # resources: {}
    jenkinsMemoryLim: 8Gi
    jenkinsMemoryReq: 4Gi
    jenkinsVolumeSize: 8Gi
  events:
    enabled: false
    # operator:
    #   resources: {}
    # exporter:
    #   resources: {}
    # ruler:
    #   enabled: true
    #   replicas: 2
    #   resources: {}
  logging:
    enabled: false
    logsidecar:
      enabled: true
      replicas: 2
      # resources: {}
  metrics_server:
    enabled: false
  monitoring:
    storageClass: ""
    node_exporter:
      port: 9100
      # resources: {}
    # kube_rbac_proxy:
    #   resources: {}
    # kube_state_metrics:
    #   resources: {}
    # prometheus:
    #   replicas: 1
    #   volumeSize: 20Gi
    #   resources: {}
    #   operator:
    #     resources: {}
    # alertmanager:
    #   replicas: 1
    #   resources: {}
    # notification_manager:
    #   resources: {}
    #   operator:
    #     resources: {}
    #   proxy:
    #     resources: {}
    gpu:
      nvidia_dcgm_exporter:
        enabled: false
        # resources: {}
  multicluster:
    clusterRole: none
  network:
    networkpolicy:
      enabled: false
    ippool:
      type: none
    topology:
      type: none
  openpitrix:
    store:
      enabled: false
  servicemesh:
    enabled: false
    istio:
      components:
        ingressGateways:
        - name: istio-ingressgateway
          enabled: false
        cni:
          enabled: false
  edgeruntime:
    enabled: false
    kubeedge:
      enabled: false
      cloudCore:
        cloudHub:
          advertiseAddress:
            - ""
        service:
          cloudhubNodePort: "30000"
          cloudhubQuicNodePort: "30001"
          cloudhubHttpsNodePort: "30002"
          cloudstreamNodePort: "30003"
          tunnelNodePort: "30004"
        # resources: {}
        # hostNetWork: false
      iptables-manager:
        enabled: true
        mode: "external"
        # resources: {}
      # edgeService:
      #   resources: {}
  terminal:
    timeout: 600

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

k8s1@k8s1:~$ vim config-sample.yaml

k8s1@k8s1:~$ cat config-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2

kind: Cluster

metadata:

spec:

hosts:

- {name: master, address: 192.168.2.206, internalAddress: 192.168.2.206, user: k8s1, password: 123456}

- {name: node1, address: 192.168.2.177, internalAddress: 192.168.2.177, user: k8s2, password: 123456}

- {name: node2, address: 192.168.2.203, internalAddress: 192.168.2.203, user: k8s3, password: 123456}

roleGroups:

etcd:

- master

control-plane:

- master

worker:

- node1

- node2

controlPlaneEndpoint:

## Internal loadbalancer for apiservers

# internalLoadbalancer: haproxy

domain: kubesphere.k8s.local

address: ""

port: 6443

kubernetes:

version: v1.23.10

clusterName: cluster.local

autoRenewCerts: true

containerManager: docker

etcd:

type: kubekey

network:

plugin: calico

kubePodsCIDR: 10.233.64.0/18

kubeServiceCIDR: 10.233.0.0/18

## multus support. https://github.com/k8snetworkplumbingwg/multus-cni

multusCNI:

enabled: false

registry:

privateRegistry: ""

namespaceOverride: ""

registryMirrors: []

insecureRegistries: []

addons: []

---

apiVersion: installer.kubesphere.io/v1alpha1

kind: ClusterConfiguration

metadata:

namespace: kubesphere-system

labels:

version: v3.3.2

spec:

persistence:

storageClass: ""

authentication:

jwtSecret: ""

zone: ""

local_registry: ""

namespace_override: ""

# dev_tag: ""

etcd:

monitoring: false

endpointIps: localhost

port: 2379

tlsEnable: true

common:

core:

console:

enableMultiLogin: true

port: 30880

type: NodePort

# apiserver:

# resources: {}

# controllerManager:

# resources: {}

redis:

enabled: false

volumeSize: 2Gi

openldap:

enabled: false

volumeSize: 2Gi

minio:

volumeSize: 20Gi

monitoring:

# type: external

endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090

GPUMonitoring:

enabled: false

gpu:

kinds:

- resourceName: "nvidia.com/gpu"

resourceType: "GPU"

default: true

es:

# master:

# volumeSize: 4Gi

# replicas: 1

# resources: {}

# data:

# volumeSize: 20Gi

# replicas: 1

# resources: {}

logMaxAge: 7

elkPrefix: logstash

basicAuth:

enabled: false

username: ""

password: ""

externalElasticsearchHost: ""

externalElasticsearchPort: ""

alerting:

enabled: false

# thanosruler:

# replicas: 1

# resources: {}

auditing:

enabled: false

# operator:

# resources: {}

# webhook:

# resources: {}

devops:

enabled: false

# resources: {}

jenkinsMemoryLim: 8Gi

jenkinsMemoryReq: 4Gi

jenkinsVolumeSize: 8Gi

events:

enabled: false

# operator:

# resources: {}

# exporter:

# resources: {}

# ruler:

# enabled: true

# replicas: 2

# resources: {}

logging:

enabled: false

logsidecar:

enabled: true

replicas: 2

# resources: {}

metrics_server:

enabled: false

monitoring:

storageClass: ""

node_exporter:

port: 9100

# resources: {}

# kube_rbac_proxy:

# resources: {}

# kube_state_metrics:

# resources: {}

# prometheus:

# replicas: 1

# volumeSize: 20Gi

# resources: {}

# operator:

# resources: {}

# alertmanager:

# replicas: 1

# resources: {}

# notification_manager:

# resources: {}

# operator:

# resources: {}

# proxy:

# resources: {}

gpu:

nvidia_dcgm_exporter:

enabled: false

# resources: {}

multicluster:

clusterRole: none

network:

networkpolicy:

enabled: false

ippool:

type: none

topology:

type: none

openpitrix:

store:

enabled: false

servicemesh:

enabled: false

istio:

components:

ingressGateways:

- name: istio-ingressgateway

enabled: false

cni:

enabled: false

edgeruntime:

enabled: false

kubeedge:

enabled: false

cloudCore:

cloudHub:

advertiseAddress:

- ""

service:

cloudhubNodePort: "30000"

cloudhubQuicNodePort: "30001"

cloudhubHttpsNodePort: "30002"

cloudstreamNodePort: "30003"

tunnelNodePort: "30004"

# resources: {}

# hostNetWork: false

iptables-manager:

enabled: true

mode: "external"

# resources: {}

# edgeService:

# resources: {}

terminal:

timeout: 600

每个节点上安装依赖项


sudo apt install -y socat conntrack ebtables ipset

0 1	sudo apt install -y socat conntrack ebtables ipset

任务机上执行这个安装集群


./kk create cluster -f config-sample.yaml

0 1	./kk create cluster -f config-sample.yaml

安装过程


k8s1@k8s1:~$ ./kk create cluster -f config-sample.yaml


 _   __      _          _   __
| | / /     | |        | | / /
| |/ / _   _| |__   ___| |/ /  ___ _   _
|    \| | | | '_ \ / _ \    \ / _ \ | | |
| |\  \ |_| | |_) |  __/ |\  \  __/ |_| |
\_| \_/\__,_|_.__/ \___\_| \_/\___|\__, |
                                    __/ |
                                   |___/

13:11:55 UTC [GreetingsModule] Greetings
13:11:57 UTC message: [master]
Greetings, KubeKey!
13:11:58 UTC message: [node2]
Greetings, KubeKey!
13:11:59 UTC message: [node1]
Greetings, KubeKey!
13:11:59 UTC success: [master]
13:11:59 UTC success: [node2]
13:11:59 UTC success: [node1]
13:11:59 UTC [NodePreCheckModule] A pre-check on nodes
13:12:00 UTC success: [node2]
13:12:00 UTC success: [node1]
13:12:00 UTC success: [master]
13:12:00 UTC [ConfirmModule] Display confirmation form
+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| name   | sudo | curl | openssl | ebtables | socat | ipset | ipvsadm | conntrack | chrony | docker | containerd | nfs client | ceph client | glusterfs client | time         |
+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| master | y    | y    | y       | y        | y     | y     |         | y         |        |        |            |            |             |                  | UTC 13:12:00 |
| node1  | y    | y    | y       | y        | y     | y     |         | y         |        |        |            |            |             |                  | UTC 13:12:00 |
| node2  | y    | y    | y       | y        | y     | y     |         | y         |        |        |            |            |             |                  | UTC 13:12:00 |
+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

This is a simple check of your environment.
Before installation, ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations

Continue this installation? [yes/no]: yes

13:12:35 UTC success: [LocalHost]
13:12:35 UTC [NodeBinariesModule] Download installation binaries
13:12:35 UTC message: [localhost]
downloading amd64 kubeadm v1.23.10 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43.1M  100 43.1M    0     0  1012k      0  0:00:43  0:00:43 --:--:--  983k
13:13:20 UTC message: [localhost]
downloading amd64 kubelet v1.23.10 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  118M  100  118M    0     0  1022k      0  0:01:58  0:01:58 --:--:-- 1072k
13:15:21 UTC message: [localhost]
downloading amd64 kubectl v1.23.10 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 44.4M  100 44.4M    0     0  1031k      0  0:00:44  0:00:44 --:--:-- 1185k
13:16:07 UTC message: [localhost]
downloading amd64 helm v3.9.0 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 44.0M  100 44.0M    0     0  1021k      0  0:00:44  0:00:44 --:--:-- 1028k
13:16:52 UTC message: [localhost]
downloading amd64 kubecni v0.9.1 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 37.9M  100 37.9M    0     0  1041k      0  0:00:37  0:00:37 --:--:-- 1194k
13:17:30 UTC message: [localhost]
downloading amd64 crictl v1.24.0 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.8M  100 13.8M    0     0  1080k      0  0:00:13  0:00:13 --:--:-- 1218k
13:17:43 UTC message: [localhost]
downloading amd64 etcd v3.4.13 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 16.5M  100 16.5M    0     0  1034k      0  0:00:16  0:00:16 --:--:-- 1085k
13:18:00 UTC message: [localhost]
downloading amd64 docker 20.10.8 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 58.1M  100 58.1M    0     0  6954k      0  0:00:08  0:00:08 --:--:-- 7063k
13:18:10 UTC success: [LocalHost]
13:18:10 UTC [ConfigureOSModule] Get OS release
13:18:10 UTC success: [node1]
13:18:10 UTC success: [master]
13:18:10 UTC success: [node2]
13:18:10 UTC [ConfigureOSModule] Prepare to init OS
13:18:11 UTC success: [node1]
13:18:11 UTC success: [node2]
13:18:11 UTC success: [master]
13:18:11 UTC [ConfigureOSModule] Generate init os script
13:18:12 UTC success: [node1]
13:18:12 UTC success: [node2]
13:18:12 UTC success: [master]
13:18:12 UTC [ConfigureOSModule] Exec init os script
13:18:15 UTC stdout: [node1]
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
13:18:15 UTC stdout: [node2]
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
13:18:16 UTC stdout: [master]
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
13:18:16 UTC success: [node1]
13:18:16 UTC success: [node2]
13:18:16 UTC success: [master]
13:18:16 UTC [ConfigureOSModule] configure the ntp server for each node
13:18:16 UTC skipped: [node2]
13:18:16 UTC skipped: [master]
13:18:16 UTC skipped: [node1]
13:18:16 UTC [KubernetesStatusModule] Get kubernetes cluster status
13:18:17 UTC success: [master]
13:18:17 UTC [InstallContainerModule] Sync docker binaries
13:18:30 UTC success: [node1]
13:18:30 UTC success: [node2]
13:18:30 UTC success: [master]
13:18:30 UTC [InstallContainerModule] Generate docker service
13:18:30 UTC success: [node1]
13:18:30 UTC success: [master]
13:18:30 UTC success: [node2]
13:18:30 UTC [InstallContainerModule] Generate docker config
13:18:30 UTC success: [node1]
13:18:30 UTC success: [node2]
13:18:30 UTC success: [master]
13:18:30 UTC [InstallContainerModule] Enable docker
13:18:49 UTC success: [node1]
13:18:49 UTC success: [node2]
13:18:49 UTC success: [master]
13:18:49 UTC [InstallContainerModule] Add auths to container runtime
13:18:49 UTC skipped: [node1]
13:18:49 UTC skipped: [master]
13:18:49 UTC skipped: [node2]
13:18:49 UTC [PullModule] Start to pull images on all nodes
13:18:49 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6
13:18:49 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6
13:18:49 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6
13:18:50 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.23.10
13:18:50 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10
13:18:50 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10
13:19:00 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.23.10
13:19:00 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6
13:19:01 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6
13:19:05 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
13:19:05 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
13:19:09 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.23.10
13:19:14 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10
13:19:15 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2
13:19:16 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2
13:19:34 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6
13:19:37 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2
13:19:37 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2
13:19:40 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
13:19:50 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2
13:20:07 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2
13:20:07 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2
13:20:11 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2
13:20:29 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2
13:20:29 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2
13:20:35 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2
13:20:55 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2
13:20:58 UTC success: [node1]
13:20:58 UTC success: [node2]
13:20:58 UTC success: [master]
13:20:58 UTC [ETCDPreCheckModule] Get etcd status
13:20:58 UTC success: [master]
13:20:58 UTC [CertsModule] Fetch etcd certs
13:20:58 UTC success: [master]
13:20:58 UTC [CertsModule] Generate etcd Certs
[certs] Generating "ca" certificate and key
[certs] admin-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]
[certs] member-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]
[certs] node-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]
13:21:01 UTC success: [LocalHost]
13:21:01 UTC [CertsModule] Synchronize certs file
13:21:02 UTC success: [master]
13:21:02 UTC [CertsModule] Synchronize certs file to master
13:21:02 UTC skipped: [master]
13:21:02 UTC [InstallETCDBinaryModule] Install etcd using binary
13:21:04 UTC success: [master]
13:21:04 UTC [InstallETCDBinaryModule] Generate etcd service
13:21:04 UTC success: [master]
13:21:04 UTC [InstallETCDBinaryModule] Generate access address
13:21:04 UTC success: [master]
13:21:04 UTC [ETCDConfigureModule] Health check on exist etcd
13:21:04 UTC skipped: [master]
13:21:04 UTC [ETCDConfigureModule] Generate etcd.env config on new etcd
13:21:04 UTC success: [master]
13:21:04 UTC [ETCDConfigureModule] Refresh etcd.env config on all etcd
13:21:05 UTC success: [master]
13:21:05 UTC [ETCDConfigureModule] Restart etcd
13:21:09 UTC stdout: [master]
Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /etc/systemd/system/etcd.service.
13:21:09 UTC success: [master]
13:21:09 UTC [ETCDConfigureModule] Health check on all etcd
13:21:09 UTC success: [master]
13:21:09 UTC [ETCDConfigureModule] Refresh etcd.env config to exist mode on all etcd
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDConfigureModule] Health check on all etcd
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Backup etcd data regularly
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Generate backup ETCD service
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Generate backup ETCD timer
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Enable backup etcd service
13:21:11 UTC success: [master]
13:21:11 UTC [InstallKubeBinariesModule] Synchronize kubernetes binaries
13:21:48 UTC success: [node1]
13:21:48 UTC success: [master]
13:21:48 UTC success: [node2]
13:21:48 UTC [InstallKubeBinariesModule] Synchronize kubelet
13:21:48 UTC success: [node1]
13:21:48 UTC success: [node2]
13:21:48 UTC success: [master]
13:21:48 UTC [InstallKubeBinariesModule] Generate kubelet service
13:21:49 UTC success: [node1]
13:21:49 UTC success: [node2]
13:21:49 UTC success: [master]
13:21:49 UTC [InstallKubeBinariesModule] Enable kubelet service
13:21:52 UTC success: [node2]
13:21:52 UTC success: [node1]
13:21:52 UTC success: [master]
13:21:52 UTC [InstallKubeBinariesModule] Generate kubelet env
13:21:52 UTC success: [node1]
13:21:52 UTC success: [master]
13:21:52 UTC success: [node2]
13:21:52 UTC [InitKubernetesModule] Generate kubeadm config
13:21:53 UTC success: [master]
13:21:53 UTC [InitKubernetesModule] Init cluster using kubeadm
13:22:30 UTC stdout: [master]
W0407 13:21:53.232803    5651 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
[init] Using Kubernetes version: v1.23.10
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local kubesphere.k8s.local localhost master master.cluster.local node1 node1.cluster.local node2 node2.cluster.local] and IPs [10.233.0.1 192.168.2.206 127.0.0.1 192.168.2.177 192.168.2.203]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.505779 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 71j95c.vwt766fyex29car8
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join kubesphere.k8s.local:6443 --token 71j95c.vwt766fyex29car8 \
        --discovery-token-ca-cert-hash sha256:57fac9b5408266f709d667f092e5fb7e140e56480e9d2d10796007af241202a8 \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join kubesphere.k8s.local:6443 --token 71j95c.vwt766fyex29car8 \
        --discovery-token-ca-cert-hash sha256:57fac9b5408266f709d667f092e5fb7e140e56480e9d2d10796007af241202a8
13:22:30 UTC success: [master]
13:22:30 UTC [InitKubernetesModule] Copy admin.conf to ~/.kube/config
13:22:30 UTC success: [master]
13:22:30 UTC [InitKubernetesModule] Remove master taint
13:22:30 UTC skipped: [master]
13:22:30 UTC [InitKubernetesModule] Add worker label
13:22:30 UTC skipped: [master]
13:22:30 UTC [ClusterDNSModule] Generate coredns service
13:22:31 UTC success: [master]
13:22:31 UTC [ClusterDNSModule] Override coredns service
13:22:32 UTC stdout: [master]
service "kube-dns" deleted
13:22:35 UTC stdout: [master]
service/coredns created
Warning: resource clusterroles/system:coredns is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrole.rbac.authorization.k8s.io/system:coredns configured
13:22:35 UTC success: [master]
13:22:35 UTC [ClusterDNSModule] Generate nodelocaldns
13:22:35 UTC success: [master]
13:22:35 UTC [ClusterDNSModule] Deploy nodelocaldns
13:22:35 UTC stdout: [master]
serviceaccount/nodelocaldns created
daemonset.apps/nodelocaldns created
13:22:35 UTC success: [master]
13:22:35 UTC [ClusterDNSModule] Generate nodelocaldns configmap
13:22:36 UTC success: [master]
13:22:36 UTC [ClusterDNSModule] Apply nodelocaldns configmap
13:22:37 UTC stdout: [master]
configmap/nodelocaldns created
13:22:37 UTC success: [master]
13:22:37 UTC [KubernetesStatusModule] Get kubernetes cluster status
13:22:37 UTC stdout: [master]
v1.23.10
13:22:37 UTC stdout: [master]
master   v1.23.10   [map[address:192.168.2.206 type:InternalIP] map[address:master type:Hostname]]
13:22:44 UTC stdout: [master]
I0407 13:22:41.146907    6828 version.go:255] remote version is much newer: v1.26.3; falling back to: stable-1.23
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
2f289571fc7a08495750bd040c609d138754bdb3b691e110dfc62d9e9a983a33
13:22:44 UTC stdout: [master]
secret/kubeadm-certs patched
13:22:45 UTC stdout: [master]
secret/kubeadm-certs patched
13:22:45 UTC stdout: [master]
secret/kubeadm-certs patched
13:22:45 UTC stdout: [master]
vbbjzr.kxrrp9i7ttjop1q5
13:22:45 UTC success: [master]
13:22:45 UTC [JoinNodesModule] Generate kubeadm config
13:22:46 UTC skipped: [master]
13:22:46 UTC success: [node1]
13:22:46 UTC success: [node2]
13:22:46 UTC [JoinNodesModule] Join control-plane node
13:22:46 UTC skipped: [master]
13:22:46 UTC [JoinNodesModule] Join worker node
13:22:54 UTC stdout: [node2]
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0407 13:22:47.815262    4922 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0407 13:22:47.827002    4922 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
13:22:54 UTC stdout: [node1]
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0407 13:22:47.919379    4966 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0407 13:22:47.929063    4966 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
13:22:54 UTC success: [node2]
13:22:54 UTC success: [node1]
13:22:54 UTC [JoinNodesModule] Copy admin.conf to ~/.kube/config
13:22:54 UTC skipped: [master]
13:22:54 UTC [JoinNodesModule] Remove master taint
13:22:54 UTC skipped: [master]
13:22:54 UTC [JoinNodesModule] Add worker label to master
13:22:54 UTC skipped: [master]
13:22:54 UTC [JoinNodesModule] Synchronize kube config to worker
13:22:55 UTC success: [node1]
13:22:55 UTC success: [node2]
13:22:55 UTC [JoinNodesModule] Add worker label to worker
13:22:55 UTC stdout: [node2]
node/node2 labeled
13:22:55 UTC stdout: [node1]
node/node1 labeled
13:22:55 UTC success: [node2]
13:22:55 UTC success: [node1]
13:22:55 UTC [DeployNetworkPluginModule] Generate calico
13:22:56 UTC success: [master]
13:22:56 UTC [DeployNetworkPluginModule] Deploy calico
13:22:58 UTC stdout: [master]
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
13:22:58 UTC success: [master]
13:22:58 UTC [ConfigureKubernetesModule] Configure kubernetes
13:22:58 UTC success: [node2]
13:22:58 UTC success: [master]
13:22:58 UTC success: [node1]
13:22:58 UTC [ChownModule] Chown user $HOME/.kube dir
13:22:59 UTC success: [node1]
13:22:59 UTC success: [node2]
13:22:59 UTC success: [master]
13:22:59 UTC [AutoRenewCertsModule] Generate k8s certs renew script
13:23:00 UTC success: [master]
13:23:00 UTC [AutoRenewCertsModule] Generate k8s certs renew service
13:23:01 UTC success: [master]
13:23:01 UTC [AutoRenewCertsModule] Generate k8s certs renew timer
13:23:02 UTC success: [master]
13:23:02 UTC [AutoRenewCertsModule] Enable k8s certs renew service
13:23:03 UTC success: [master]
13:23:03 UTC [SaveKubeConfigModule] Save kube config as a configmap
13:23:05 UTC success: [LocalHost]
13:23:05 UTC [AddonsModule] Install addons
13:23:05 UTC success: [LocalHost]
13:23:05 UTC [DeployStorageClassModule] Generate OpenEBS manifest
13:23:05 UTC success: [master]
13:23:05 UTC [DeployStorageClassModule] Deploy OpenEBS as cluster default StorageClass
13:23:19 UTC success: [master]
13:23:19 UTC [DeployKubeSphereModule] Generate KubeSphere ks-installer crd manifests
13:23:20 UTC success: [master]
13:23:20 UTC [DeployKubeSphereModule] Apply ks-installer
13:23:28 UTC stdout: [master]
namespace/kubesphere-system created
serviceaccount/ks-installer created
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io created
clusterrole.rbac.authorization.k8s.io/ks-installer created
clusterrolebinding.rbac.authorization.k8s.io/ks-installer created
deployment.apps/ks-installer created
13:23:28 UTC success: [master]
13:23:28 UTC [DeployKubeSphereModule] Add config to ks-installer manifests
13:23:28 UTC success: [master]
13:23:28 UTC [DeployKubeSphereModule] Create the kubesphere namespace
13:23:30 UTC success: [master]
13:23:30 UTC [DeployKubeSphereModule] Setup ks-installer config
13:23:31 UTC stdout: [master]
secret/kube-etcd-client-certs created
13:23:31 UTC success: [master]
13:23:31 UTC [DeployKubeSphereModule] Apply ks-installer
13:23:34 UTC stdout: [master]
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
13:23:34 UTC success: [master]
Please wait for the installation to complete:    >>--->

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

k8s1@k8s1:~$ ./kk create cluster -f config-sample.yaml

_ __ _ _ __

| | / / | | | | / /

| |/ / _ _| |__ ___| |/ / ___ _ _

| \| | | | '_ \ / _ \ \ / _ \ | | |

| |\ \ |_| | |_) | __/ |\ \ __/ |_| |

\_| \_/\__,_|_.__/ \___\_| \_/\___|\__, |

__/ |

|___/

13:11:55 UTC [GreetingsModule] Greetings

13:11:57 UTC message: [master]

Greetings, KubeKey!

13:11:58 UTC message: [node2]

Greetings, KubeKey!

13:11:59 UTC message: [node1]

Greetings, KubeKey!

13:11:59 UTC success: [master]

13:11:59 UTC success: [node2]

13:11:59 UTC success: [node1]

13:11:59 UTC [NodePreCheckModule] A pre-check on nodes

13:12:00 UTC success: [node2]

13:12:00 UTC success: [node1]

13:12:00 UTC success: [master]

13:12:00 UTC [ConfirmModule] Display confirmation form

+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

| master | y | y | y | y | y | y | | y | | | | | | | UTC 13:12:00 |

| node1 | y | y | y | y | y | y | | y | | | | | | | UTC 13:12:00 |

| node2 | y | y | y | y | y | y | | y | | | | | | | UTC 13:12:00 |

+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

This is a simple check of your environment.

Before installation, ensure that your machines meet all requirements specified at

https://github.com/kubesphere/kubekey#requirements-and-recommendations

Continue this installation? [yes/no]: yes

13:12:35 UTC success: [LocalHost]

13:12:35 UTC [NodeBinariesModule] Download installation binaries

13:12:35 UTC message: [localhost]

downloading amd64 kubeadm v1.23.10 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 43.1M 100 43.1M 0 0 1012k 0 0:00:43 0:00:43 --:--:-- 983k

13:13:20 UTC message: [localhost]

downloading amd64 kubelet v1.23.10 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 118M 100 118M 0 0 1022k 0 0:01:58 0:01:58 --:--:-- 1072k

13:15:21 UTC message: [localhost]

downloading amd64 kubectl v1.23.10 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 44.4M 100 44.4M 0 0 1031k 0 0:00:44 0:00:44 --:--:-- 1185k

13:16:07 UTC message: [localhost]

downloading amd64 helm v3.9.0 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 44.0M 100 44.0M 0 0 1021k 0 0:00:44 0:00:44 --:--:-- 1028k

13:16:52 UTC message: [localhost]

downloading amd64 kubecni v0.9.1 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 37.9M 100 37.9M 0 0 1041k 0 0:00:37 0:00:37 --:--:-- 1194k

13:17:30 UTC message: [localhost]

downloading amd64 crictl v1.24.0 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 13.8M 100 13.8M 0 0 1080k 0 0:00:13 0:00:13 --:--:-- 1218k

13:17:43 UTC message: [localhost]

downloading amd64 etcd v3.4.13 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 16.5M 100 16.5M 0 0 1034k 0 0:00:16 0:00:16 --:--:-- 1085k

13:18:00 UTC message: [localhost]

downloading amd64 docker 20.10.8 ...

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 58.1M 100 58.1M 0 0 6954k 0 0:00:08 0:00:08 --:--:-- 7063k

13:18:10 UTC success: [LocalHost]

13:18:10 UTC [ConfigureOSModule] Get OS release

13:18:10 UTC success: [node1]

13:18:10 UTC success: [master]

13:18:10 UTC success: [node2]

13:18:10 UTC [ConfigureOSModule] Prepare to init OS

13:18:11 UTC success: [node1]

13:18:11 UTC success: [node2]

13:18:11 UTC success: [master]

13:18:11 UTC [ConfigureOSModule] Generate init os script

13:18:12 UTC success: [node1]

13:18:12 UTC success: [node2]

13:18:12 UTC success: [master]

13:18:12 UTC [ConfigureOSModule] Exec init os script

13:18:15 UTC stdout: [node1]

net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-arptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_local_reserved_ports = 30000-32767

vm.max_map_count = 262144

vm.swappiness = 1

fs.inotify.max_user_instances = 524288

kernel.pid_max = 65535

13:18:15 UTC stdout: [node2]

net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-arptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_local_reserved_ports = 30000-32767

vm.max_map_count = 262144

vm.swappiness = 1

fs.inotify.max_user_instances = 524288

kernel.pid_max = 65535

13:18:16 UTC stdout: [master]

net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-arptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_local_reserved_ports = 30000-32767

vm.max_map_count = 262144

vm.swappiness = 1

fs.inotify.max_user_instances = 524288

kernel.pid_max = 65535

13:18:16 UTC success: [node1]

13:18:16 UTC success: [node2]

13:18:16 UTC success: [master]

13:18:16 UTC [ConfigureOSModule] configure the ntp server for each node

13:18:16 UTC skipped: [node2]

13:18:16 UTC skipped: [master]

13:18:16 UTC skipped: [node1]

13:18:16 UTC [KubernetesStatusModule] Get kubernetes cluster status

13:18:17 UTC success: [master]

13:18:17 UTC [InstallContainerModule] Sync docker binaries

13:18:30 UTC success: [node1]

13:18:30 UTC success: [node2]

13:18:30 UTC success: [master]

13:18:30 UTC [InstallContainerModule] Generate docker service

13:18:30 UTC success: [node1]

13:18:30 UTC success: [master]

13:18:30 UTC success: [node2]

13:18:30 UTC [InstallContainerModule] Generate docker config

13:18:30 UTC success: [node1]

13:18:30 UTC success: [node2]

13:18:30 UTC success: [master]

13:18:30 UTC [InstallContainerModule] Enable docker

13:18:49 UTC success: [node1]

13:18:49 UTC success: [node2]

13:18:49 UTC success: [master]

13:18:49 UTC [InstallContainerModule] Add auths to container runtime

13:18:49 UTC skipped: [node1]

13:18:49 UTC skipped: [master]

13:18:49 UTC skipped: [node2]

13:18:49 UTC [PullModule] Start to pull images on all nodes

13:18:49 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6

13:18:49 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6

13:18:49 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6

13:18:50 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.23.10

13:18:50 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10

13:18:50 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10

13:19:00 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.23.10

13:19:00 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6

13:19:01 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6

13:19:05 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12

13:19:05 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12

13:19:09 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.23.10

13:19:14 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10

13:19:15 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2

13:19:16 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2

13:19:34 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6

13:19:37 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2

13:19:37 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2

13:19:40 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12

13:19:50 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2

13:20:07 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2

13:20:07 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2

13:20:11 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2

13:20:29 UTC message: [node1]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2

13:20:29 UTC message: [node2]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2

13:20:35 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2

13:20:55 UTC message: [master]

downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2

13:20:58 UTC success: [node1]

13:20:58 UTC success: [node2]

13:20:58 UTC success: [master]

13:20:58 UTC [ETCDPreCheckModule] Get etcd status

13:20:58 UTC success: [master]

13:20:58 UTC [CertsModule] Fetch etcd certs

13:20:58 UTC success: [master]

13:20:58 UTC [CertsModule] Generate etcd Certs

[certs] Generating "ca" certificate and key

[certs] admin-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]

[certs] member-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]

[certs] node-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]

13:21:01 UTC success: [LocalHost]

13:21:01 UTC [CertsModule] Synchronize certs file

13:21:02 UTC success: [master]

13:21:02 UTC [CertsModule] Synchronize certs file to master

13:21:02 UTC skipped: [master]

13:21:02 UTC [InstallETCDBinaryModule] Install etcd using binary

13:21:04 UTC success: [master]

13:21:04 UTC [InstallETCDBinaryModule] Generate etcd service

13:21:04 UTC success: [master]

13:21:04 UTC [InstallETCDBinaryModule] Generate access address

13:21:04 UTC success: [master]

13:21:04 UTC [ETCDConfigureModule] Health check on exist etcd

13:21:04 UTC skipped: [master]

13:21:04 UTC [ETCDConfigureModule] Generate etcd.env config on new etcd

13:21:04 UTC success: [master]

13:21:04 UTC [ETCDConfigureModule] Refresh etcd.env config on all etcd

13:21:05 UTC success: [master]

13:21:05 UTC [ETCDConfigureModule] Restart etcd

13:21:09 UTC stdout: [master]

Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /etc/systemd/system/etcd.service.

13:21:09 UTC success: [master]

13:21:09 UTC [ETCDConfigureModule] Health check on all etcd

13:21:09 UTC success: [master]

13:21:09 UTC [ETCDConfigureModule] Refresh etcd.env config to exist mode on all etcd

13:21:10 UTC success: [master]

13:21:10 UTC [ETCDConfigureModule] Health check on all etcd

13:21:10 UTC success: [master]

13:21:10 UTC [ETCDBackupModule] Backup etcd data regularly

13:21:10 UTC success: [master]

13:21:10 UTC [ETCDBackupModule] Generate backup ETCD service

13:21:10 UTC success: [master]

13:21:10 UTC [ETCDBackupModule] Generate backup ETCD timer

13:21:10 UTC success: [master]

13:21:10 UTC [ETCDBackupModule] Enable backup etcd service

13:21:11 UTC success: [master]

13:21:11 UTC [InstallKubeBinariesModule] Synchronize kubernetes binaries

13:21:48 UTC success: [node1]

13:21:48 UTC success: [master]

13:21:48 UTC success: [node2]

13:21:48 UTC [InstallKubeBinariesModule] Synchronize kubelet

13:21:48 UTC success: [node1]

13:21:48 UTC success: [node2]

13:21:48 UTC success: [master]

13:21:48 UTC [InstallKubeBinariesModule] Generate kubelet service

13:21:49 UTC success: [node1]

13:21:49 UTC success: [node2]

13:21:49 UTC success: [master]

13:21:49 UTC [InstallKubeBinariesModule] Enable kubelet service

13:21:52 UTC success: [node2]

13:21:52 UTC success: [node1]

13:21:52 UTC success: [master]

13:21:52 UTC [InstallKubeBinariesModule] Generate kubelet env

13:21:52 UTC success: [node1]

13:21:52 UTC success: [master]

13:21:52 UTC success: [node2]

13:21:52 UTC [InitKubernetesModule] Generate kubeadm config

13:21:53 UTC success: [master]

13:21:53 UTC [InitKubernetesModule] Init cluster using kubeadm

13:22:30 UTC stdout: [master]

W0407 13:21:53.232803 5651 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]

[init] Using Kubernetes version: v1.23.10

[preflight] Running pre-flight checks

[preflight] Pulling images required for setting up a Kubernetes cluster

[preflight] This might take a minute or two, depending on the speed of your internet connection

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

[certs] Using certificateDir folder "/etc/kubernetes/pki"

[certs] Generating "ca" certificate and key

[certs] Generating "apiserver" certificate and key

[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local kubesphere.k8s.local localhost master master.cluster.local node1 node1.cluster.local node2 node2.cluster.local] and IPs [10.233.0.1 192.168.2.206 127.0.0.1 192.168.2.177 192.168.2.203]

[certs] Generating "apiserver-kubelet-client" certificate and key

[certs] Generating "front-proxy-ca" certificate and key

[certs] Generating "front-proxy-client" certificate and key

[certs] External etcd mode: Skipping etcd/ca certificate authority generation

[certs] External etcd mode: Skipping etcd/server certificate generation

[certs] External etcd mode: Skipping etcd/peer certificate generation

[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation

[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation

[certs] Generating "sa" key and public key

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"

[kubeconfig] Writing "admin.conf" kubeconfig file

[kubeconfig] Writing "kubelet.conf" kubeconfig file

[kubeconfig] Writing "controller-manager.conf" kubeconfig file

[kubeconfig] Writing "scheduler.conf" kubeconfig file

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Starting the kubelet

[control-plane] Using manifest folder "/etc/kubernetes/manifests"

[control-plane] Creating static Pod manifest for "kube-apiserver"

[control-plane] Creating static Pod manifest for "kube-controller-manager"

[control-plane] Creating static Pod manifest for "kube-scheduler"

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

[apiclient] All control plane components are healthy after 23.505779 seconds

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster

NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.

[upload-certs] Skipping phase. Please see --upload-certs

[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]

[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

[bootstrap-token] Using token: 71j95c.vwt766fyex29car8

[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles

[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes

[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key

[addons] Applied essential addon: CoreDNS

[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities

and service account keys on each node and then running the following as root:

kubeadm join kubesphere.k8s.local:6443 --token 71j95c.vwt766fyex29car8 \

--discovery-token-ca-cert-hash sha256:57fac9b5408266f709d667f092e5fb7e140e56480e9d2d10796007af241202a8 \

--control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join kubesphere.k8s.local:6443 --token 71j95c.vwt766fyex29car8 \

--discovery-token-ca-cert-hash sha256:57fac9b5408266f709d667f092e5fb7e140e56480e9d2d10796007af241202a8

13:22:30 UTC success: [master]

13:22:30 UTC [InitKubernetesModule] Copy admin.conf to ~/.kube/config

13:22:30 UTC success: [master]

13:22:30 UTC [InitKubernetesModule] Remove master taint

13:22:30 UTC skipped: [master]

13:22:30 UTC [InitKubernetesModule] Add worker label

13:22:30 UTC skipped: [master]

13:22:30 UTC [ClusterDNSModule] Generate coredns service

13:22:31 UTC success: [master]

13:22:31 UTC [ClusterDNSModule] Override coredns service

13:22:32 UTC stdout: [master]

service "kube-dns" deleted

13:22:35 UTC stdout: [master]

service/coredns created

Warning: resource clusterroles/system:coredns is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.

clusterrole.rbac.authorization.k8s.io/system:coredns configured

13:22:35 UTC success: [master]

13:22:35 UTC [ClusterDNSModule] Generate nodelocaldns

13:22:35 UTC success: [master]

13:22:35 UTC [ClusterDNSModule] Deploy nodelocaldns

13:22:35 UTC stdout: [master]

serviceaccount/nodelocaldns created

daemonset.apps/nodelocaldns created

13:22:35 UTC success: [master]

13:22:35 UTC [ClusterDNSModule] Generate nodelocaldns configmap

13:22:36 UTC success: [master]

13:22:36 UTC [ClusterDNSModule] Apply nodelocaldns configmap

13:22:37 UTC stdout: [master]

configmap/nodelocaldns created

13:22:37 UTC success: [master]

13:22:37 UTC [KubernetesStatusModule] Get kubernetes cluster status

13:22:37 UTC stdout: [master]

v1.23.10

13:22:37 UTC stdout: [master]

master v1.23.10 [map[address:192.168.2.206 type:InternalIP] map[address:master type:Hostname]]

13:22:44 UTC stdout: [master]

I0407 13:22:41.146907 6828 version.go:255] remote version is much newer: v1.26.3; falling back to: stable-1.23

[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace

[upload-certs] Using certificate key:

2f289571fc7a08495750bd040c609d138754bdb3b691e110dfc62d9e9a983a33

13:22:44 UTC stdout: [master]

secret/kubeadm-certs patched

13:22:45 UTC stdout: [master]

secret/kubeadm-certs patched

13:22:45 UTC stdout: [master]

secret/kubeadm-certs patched

13:22:45 UTC stdout: [master]

vbbjzr.kxrrp9i7ttjop1q5

13:22:45 UTC success: [master]

13:22:45 UTC [JoinNodesModule] Generate kubeadm config

13:22:46 UTC skipped: [master]

13:22:46 UTC success: [node1]

13:22:46 UTC success: [node2]

13:22:46 UTC [JoinNodesModule] Join control-plane node

13:22:46 UTC skipped: [master]

13:22:46 UTC [JoinNodesModule] Join worker node

13:22:54 UTC stdout: [node2]

[preflight] Running pre-flight checks

[preflight] Reading configuration from the cluster...

[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

W0407 13:22:47.815262 4922 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]

W0407 13:22:47.827002 4922 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Starting the kubelet

[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:

* Certificate signing request was sent to apiserver and a response was received.

* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

13:22:54 UTC stdout: [node1]

[preflight] Running pre-flight checks

[preflight] Reading configuration from the cluster...

[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

W0407 13:22:47.919379 4966 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]

W0407 13:22:47.929063 4966 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Starting the kubelet

[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:

* Certificate signing request was sent to apiserver and a response was received.

* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

13:22:54 UTC success: [node2]

13:22:54 UTC success: [node1]

13:22:54 UTC [JoinNodesModule] Copy admin.conf to ~/.kube/config

13:22:54 UTC skipped: [master]

13:22:54 UTC [JoinNodesModule] Remove master taint

13:22:54 UTC skipped: [master]

13:22:54 UTC [JoinNodesModule] Add worker label to master

13:22:54 UTC skipped: [master]

13:22:54 UTC [JoinNodesModule] Synchronize kube config to worker

13:22:55 UTC success: [node1]

13:22:55 UTC success: [node2]

13:22:55 UTC [JoinNodesModule] Add worker label to worker

13:22:55 UTC stdout: [node2]

node/node2 labeled

13:22:55 UTC stdout: [node1]

node/node1 labeled

13:22:55 UTC success: [node2]

13:22:55 UTC success: [node1]

13:22:55 UTC [DeployNetworkPluginModule] Generate calico

13:22:56 UTC success: [master]

13:22:56 UTC [DeployNetworkPluginModule] Deploy calico

13:22:58 UTC stdout: [master]

configmap/calico-config created

customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created

clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrole.rbac.authorization.k8s.io/calico-node created

clusterrolebinding.rbac.authorization.k8s.io/calico-node created

daemonset.apps/calico-node created

serviceaccount/calico-node created

deployment.apps/calico-kube-controllers created

serviceaccount/calico-kube-controllers created

poddisruptionbudget.policy/calico-kube-controllers created

13:22:58 UTC success: [master]

13:22:58 UTC [ConfigureKubernetesModule] Configure kubernetes

13:22:58 UTC success: [node2]

13:22:58 UTC success: [master]

13:22:58 UTC success: [node1]

13:22:58 UTC [ChownModule] Chown user $HOME/.kube dir

13:22:59 UTC success: [node1]

13:22:59 UTC success: [node2]

13:22:59 UTC success: [master]

13:22:59 UTC [AutoRenewCertsModule] Generate k8s certs renew script

13:23:00 UTC success: [master]

13:23:00 UTC [AutoRenewCertsModule] Generate k8s certs renew service

13:23:01 UTC success: [master]

13:23:01 UTC [AutoRenewCertsModule] Generate k8s certs renew timer

13:23:02 UTC success: [master]

13:23:02 UTC [AutoRenewCertsModule] Enable k8s certs renew service

13:23:03 UTC success: [master]

13:23:03 UTC [SaveKubeConfigModule] Save kube config as a configmap

13:23:05 UTC success: [LocalHost]

13:23:05 UTC [AddonsModule] Install addons

13:23:05 UTC success: [LocalHost]

13:23:05 UTC [DeployStorageClassModule] Generate OpenEBS manifest

13:23:05 UTC success: [master]

13:23:05 UTC [DeployStorageClassModule] Deploy OpenEBS as cluster default StorageClass

13:23:19 UTC success: [master]

13:23:19 UTC [DeployKubeSphereModule] Generate KubeSphere ks-installer crd manifests

13:23:20 UTC success: [master]

13:23:20 UTC [DeployKubeSphereModule] Apply ks-installer

13:23:28 UTC stdout: [master]

namespace/kubesphere-system created

serviceaccount/ks-installer created

customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io created

clusterrole.rbac.authorization.k8s.io/ks-installer created

clusterrolebinding.rbac.authorization.k8s.io/ks-installer created

deployment.apps/ks-installer created

13:23:28 UTC success: [master]

13:23:28 UTC [DeployKubeSphereModule] Add config to ks-installer manifests

13:23:28 UTC success: [master]

13:23:28 UTC [DeployKubeSphereModule] Create the kubesphere namespace

13:23:30 UTC success: [master]

13:23:30 UTC [DeployKubeSphereModule] Setup ks-installer config

13:23:31 UTC stdout: [master]

secret/kube-etcd-client-certs created

13:23:31 UTC success: [master]

13:23:31 UTC [DeployKubeSphereModule] Apply ks-installer

13:23:34 UTC stdout: [master]

namespace/kubesphere-system unchanged

serviceaccount/ks-installer unchanged

customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged

clusterrole.rbac.authorization.k8s.io/ks-installer unchanged

clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged

deployment.apps/ks-installer unchanged

clusterconfiguration.installer.kubesphere.io/ks-installer created

13:23:34 UTC success: [master]

Please wait for the installation to complete: >>--->

这时候可以进到命令行查看安装过程，ks-installer-86ddb55c5b-gwlhh就是安装的pod


k8s3@k8s3:~$ kubectl get pods -A
NAMESPACE                      NAME                                             READY   STATUS              RESTARTS        AGE
kube-system                    calico-kube-controllers-676c86494f-q9d7f         1/1     Running             0               22m
kube-system                    calico-node-hqj7k                                1/1     Running             0               22m
kube-system                    calico-node-pr98x                                1/1     Running             0               22m
kube-system                    calico-node-xf7bt                                1/1     Running             0               22m
kube-system                    coredns-757cd945b-lx6mw                          1/1     Running             0               22m
kube-system                    coredns-757cd945b-ns8hj                          1/1     Running             0               22m
kube-system                    kube-apiserver-master                            1/1     Running             0               23m
kube-system                    kube-controller-manager-master                   1/1     Running             0               23m
kube-system                    kube-proxy-6bdxm                                 1/1     Running             0               22m
kube-system                    kube-proxy-ttf52                                 1/1     Running             0               22m
kube-system                    kube-proxy-xfhx2                                 1/1     Running             0               22m
kube-system                    kube-scheduler-master                            1/1     Running             0               23m
kube-system                    nodelocaldns-22gqz                               1/1     Running             0               22m
kube-system                    nodelocaldns-bx7nv                               1/1     Running             0               22m
kube-system                    nodelocaldns-t97rq                               1/1     Running             0               22m
kube-system                    openebs-localpv-provisioner-7974b86588-gg9nb     1/1     Running             1 (2m36s ago)   22m
kube-system                    snapshot-controller-0                            1/1     Running             0               18m
kubesphere-controls-system     default-http-backend-659cc67b6b-j8jl5            1/1     Running             0               12m
kubesphere-monitoring-system   alertmanager-main-0                              2/2     Running             0               3m54s
kubesphere-monitoring-system   alertmanager-main-1                              2/2     Running             0               3m52s
kubesphere-monitoring-system   alertmanager-main-2                              2/2     Running             0               3m51s
kubesphere-monitoring-system   kube-state-metrics-69f4fbb5d6-khf2c              3/3     Running             0               4m34s
kubesphere-monitoring-system   node-exporter-57sgh                              2/2     Running             0               4m42s
kubesphere-monitoring-system   node-exporter-k4dph                              2/2     Running             0               4m44s
kubesphere-monitoring-system   node-exporter-lfbkq                              2/2     Running             0               4m42s
kubesphere-monitoring-system   notification-manager-deployment-cdd656fd-nj6s8   2/2     Running             0               56s
kubesphere-monitoring-system   notification-manager-deployment-cdd656fd-qzvx8   2/2     Running             0               55s
kubesphere-monitoring-system   notification-manager-operator-7f7c564948-vj89q   2/2     Running             0               103s
kubesphere-monitoring-system   prometheus-k8s-0                                 2/2     Running             0               4m23s
kubesphere-monitoring-system   prometheus-k8s-1                                 2/2     Running             0               4m21s
kubesphere-monitoring-system   prometheus-operator-684988fc5c-n26kz             2/2     Running             0               4m53s
kubesphere-system              ks-apiserver-78bc58d684-76226                    0/1     ContainerCreating   0               12m
kubesphere-system              ks-console-799f77fb7d-w29wf                      1/1     Running             0               12m
kubesphere-system              ks-controller-manager-8569fb495c-b9qwv           0/1     ContainerCreating   0               12m
kubesphere-system              ks-installer-86ddb55c5b-gwlhh                    1/1     Running             0               22m

k8s3@k8s3:~$ kubectl get pods -A

NAMESPACE NAME READY STATUS RESTARTS AGE

kube-system calico-kube-controllers-676c86494f-q9d7f 1/1 Running 0 22m

kube-system calico-node-hqj7k 1/1 Running 0 22m

kube-system calico-node-pr98x 1/1 Running 0 22m

kube-system calico-node-xf7bt 1/1 Running 0 22m

kube-system coredns-757cd945b-lx6mw 1/1 Running 0 22m

kube-system coredns-757cd945b-ns8hj 1/1 Running 0 22m

kube-system kube-apiserver-master 1/1 Running 0 23m

kube-system kube-controller-manager-master 1/1 Running 0 23m

kube-system kube-proxy-6bdxm 1/1 Running 0 22m

kube-system kube-proxy-ttf52 1/1 Running 0 22m

kube-system kube-proxy-xfhx2 1/1 Running 0 22m

kube-system kube-scheduler-master 1/1 Running 0 23m

kube-system nodelocaldns-22gqz 1/1 Running 0 22m

kube-system nodelocaldns-bx7nv 1/1 Running 0 22m

kube-system nodelocaldns-t97rq 1/1 Running 0 22m

kube-system openebs-localpv-provisioner-7974b86588-gg9nb 1/1 Running 1 (2m36s ago) 22m

kube-system snapshot-controller-0 1/1 Running 0 18m

kubesphere-controls-system default-http-backend-659cc67b6b-j8jl5 1/1 Running 0 12m

kubesphere-monitoring-system alertmanager-main-0 2/2 Running 0 3m54s

kubesphere-monitoring-system alertmanager-main-1 2/2 Running 0 3m52s

kubesphere-monitoring-system alertmanager-main-2 2/2 Running 0 3m51s

kubesphere-monitoring-system kube-state-metrics-69f4fbb5d6-khf2c 3/3 Running 0 4m34s

kubesphere-monitoring-system node-exporter-57sgh 2/2 Running 0 4m42s

kubesphere-monitoring-system node-exporter-k4dph 2/2 Running 0 4m44s

kubesphere-monitoring-system node-exporter-lfbkq 2/2 Running 0 4m42s

kubesphere-monitoring-system notification-manager-deployment-cdd656fd-nj6s8 2/2 Running 0 56s

kubesphere-monitoring-system notification-manager-deployment-cdd656fd-qzvx8 2/2 Running 0 55s

kubesphere-monitoring-system notification-manager-operator-7f7c564948-vj89q 2/2 Running 0 103s

kubesphere-monitoring-system prometheus-k8s-0 2/2 Running 0 4m23s

kubesphere-monitoring-system prometheus-k8s-1 2/2 Running 0 4m21s

kubesphere-monitoring-system prometheus-operator-684988fc5c-n26kz 2/2 Running 0 4m53s

kubesphere-system ks-apiserver-78bc58d684-76226 0/1 ContainerCreating 0 12m

kubesphere-system ks-console-799f77fb7d-w29wf 1/1 Running 0 12m

kubesphere-system ks-controller-manager-8569fb495c-b9qwv 0/1 ContainerCreating 0 12m

kubesphere-system ks-installer-86ddb55c5b-gwlhh 1/1 Running 0 22m

可以进入pod容器查看目录


kubectl exec --namespace kubesphere-system ks-installer-86ddb55c5b-gwlhh -it -- /bin/bash

0 1	kubectl exec --namespace kubesphere-system ks-installer-86ddb55c5b-gwlhh -it -- /bin/bash

也可以直接用logs查看日志


kubectl logs --namespace kubesphere-system ks-installer-86ddb55c5b-gwlhh -f
前面省略了，最后长这样


PLAY RECAP *********************************************************************
localhost                  : ok=31   changed=22   unreachable=0    failed=0    skipped=16   rescued=0    ignored=0
Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
**************************************************
Waiting for all tasks to be completed ...
**************************************************
Waiting for all tasks to be completed ...
task network status is successful  (1/4)
task openpitrix status is successful  (2/4)
task multicluster status is successful  (3/4)
task monitoring status is successful  (4/4)
**************************************************
Collecting installation results ...

kubectl logs --namespace kubesphere-system ks-installer-86ddb55c5b-gwlhh -f

前面省略了，最后长这样

PLAY RECAP *********************************************************************

localhost : ok=31 changed=22 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0

Start installing monitoring

Start installing multicluster

Start installing openpitrix

Start installing network

**************************************************

Waiting for all tasks to be completed ...

**************************************************

Waiting for all tasks to be completed ...

task network status is successful (1/4)

task openpitrix status is successful (2/4)

task multicluster status is successful (3/4)

task monitoring status is successful (4/4)

**************************************************

Collecting installation results ...

观察到这两个pod一直起不来


kubesphere-system              ks-apiserver-78bc58d684-76226                    0/1     ContainerCreating   0               13m
kubesphere-system              ks-controller-manager-8569fb495c-b9qwv           0/1     ContainerCreating   0               13m

kubectl describe pod --namespace kubesphere-system ks-apiserver-78bc58d684-76226

Name:           ks-apiserver-78bc58d684-76226
Namespace:      kubesphere-system
Priority:       0
Node:           master/192.168.2.206
Start Time:     Fri, 07 Apr 2023 13:33:20 +0000
Labels:         app=ks-apiserver
                pod-template-hash=78bc58d684
                tier=backend
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/ks-apiserver-78bc58d684
Containers:
  ks-apiserver:
    Container ID:
    Image:         registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.2
    Image ID:
    Port:          9090/TCP
    Host Port:     0/TCP
    Command:
      ks-apiserver
      --logtostderr=true
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:        20m
      memory:     100Mi
    Liveness:     http-get http://:9090/kapis/version delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/kubesphere/ from kubesphere-config (rw)
      /etc/kubesphere/ingress-controller from ks-router-config (rw)
      /etc/localtime from host-time (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xp572 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  ks-router-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ks-router-config
    Optional:  false
  kubesphere-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubesphere-config
    Optional:  false
  host-time:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:
  kube-api-access-xp572:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    14m                   default-scheduler  Successfully assigned kubesphere-system/ks-apiserver-78bc58d684-76226 to master
  Warning  FailedMount  7m24s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kubesphere-config host-time kube-api-access-xp572 ks-router-config]: timed out waiting for the condition
  Warning  FailedMount  2m51s (x2 over 5m7s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kube-api-access-xp572 ks-router-config kubesphere-config host-time]: timed out waiting for the condition
  Warning  FailedMount  103s (x14 over 14m)   kubelet            MountVolume.SetUp failed for volume "kubesphere-config" : configmap "kubesphere-config" not found
  Warning  FailedMount  36s (x3 over 11m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[ks-router-config kubesphere-config host-time kube-api-access-xp572]: timed out waiting for the condition


kubectl describe pod --namespace kubesphere-system ks-controller-manager-8569fb495c-b9qwv
Name:           ks-controller-manager-8569fb495c-b9qwv
Namespace:      kubesphere-system
Priority:       0
Node:           master/192.168.2.206
Start Time:     Fri, 07 Apr 2023 13:33:19 +0000
Labels:         app=ks-controller-manager
                pod-template-hash=8569fb495c
                tier=backend
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/ks-controller-manager-8569fb495c
Containers:
  ks-controller-manager:
    Container ID:
    Image:         registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2
    Image ID:
    Ports:         8080/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      controller-manager
      --logtostderr=true
      --leader-elect=true
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1000Mi
    Requests:
      cpu:        30m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /etc/kubesphere/ from kubesphere-config (rw)
      /etc/localtime from host-time (ro)
      /tmp/k8s-webhook-server/serving-certs from webhook-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nfk8r (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kubesphere-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubesphere-config
    Optional:  false
  webhook-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ks-controller-manager-webhook-cert
    Optional:    false
  host-time:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:
  kube-api-access-nfk8r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    14m                  default-scheduler  Successfully assigned kubesphere-system/ks-controller-manager-8569fb495c-b9qwv to master
  Warning  FailedMount  14m                  kubelet            MountVolume.SetUp failed for volume "webhook-secret" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  7m43s (x3 over 12m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kubesphere-config webhook-secret host-time kube-api-access-nfk8r]: timed out waiting for the condition
  Warning  FailedMount  3m12s                kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kube-api-access-nfk8r kubesphere-config webhook-secret host-time]: timed out waiting for the condition
  Warning  FailedMount  2m (x14 over 14m)    kubelet            MountVolume.SetUp failed for volume "kubesphere-config" : configmap "kubesphere-config" not found
  Warning  FailedMount  55s (x2 over 5m28s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[host-time kube-api-access-nfk8r kubesphere-config webhook-secret]: timed out waiting for the condition

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

kubesphere-system ks-apiserver-78bc58d684-76226 0/1 ContainerCreating 0 13m

kubesphere-system ks-controller-manager-8569fb495c-b9qwv 0/1 ContainerCreating 0 13m

kubectl describe pod --namespace kubesphere-system ks-apiserver-78bc58d684-76226

Name: ks-apiserver-78bc58d684-76226

Namespace: kubesphere-system

Priority: 0

Node: master/192.168.2.206

Start Time: Fri, 07 Apr 2023 13:33:20 +0000

Labels: app=ks-apiserver

pod-template-hash=78bc58d684

tier=backend

Annotations: <none>

Status: Pending

IP:

IPs: <none>

Controlled By: ReplicaSet/ks-apiserver-78bc58d684

Containers:

ks-apiserver:

Container ID:

Image: registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.2

Image ID:

Port: 9090/TCP

Host Port: 0/TCP

Command:

ks-apiserver

--logtostderr=true

State: Waiting

Reason: ContainerCreating

Ready: False

Restart Count: 0

Limits:

cpu: 1

memory: 1Gi

Requests:

cpu: 20m

memory: 100Mi

Liveness: http-get http://:9090/kapis/version delay=15s timeout=15s period=10s #success=1 #failure=8

Environment: <none>

Mounts:

/etc/kubesphere/ from kubesphere-config (rw)

/etc/kubesphere/ingress-controller from ks-router-config (rw)

/etc/localtime from host-time (ro)

/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xp572 (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

ks-router-config:

Type: ConfigMap (a volume populated by a ConfigMap)

Name: ks-router-config

Optional: false

kubesphere-config:

Type: ConfigMap (a volume populated by a ConfigMap)

Name: kubesphere-config

Optional: false

host-time:

Type: HostPath (bare host directory volume)

Path: /etc/localtime

HostPathType:

kube-api-access-xp572:

Type: Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds: 3607

ConfigMapName: kube-root-ca.crt

ConfigMapOptional: <nil>

DownwardAPI: true

QoS Class: Burstable

Node-Selectors: <none>

Tolerations: CriticalAddonsOnly op=Exists

node-role.kubernetes.io/master:NoSchedule

node.kubernetes.io/not-ready:NoExecute op=Exists for 60s

node.kubernetes.io/unreachable:NoExecute op=Exists for 60s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 14m default-scheduler Successfully assigned kubesphere-system/ks-apiserver-78bc58d684-76226 to master

Warning FailedMount 7m24s kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kubesphere-config host-time kube-api-access-xp572 ks-router-config]: timed out waiting for the condition

Warning FailedMount 2m51s (x2 over 5m7s) kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kube-api-access-xp572 ks-router-config kubesphere-config host-time]: timed out waiting for the condition

Warning FailedMount 103s (x14 over 14m) kubelet MountVolume.SetUp failed for volume "kubesphere-config" : configmap "kubesphere-config" not found

Warning FailedMount 36s (x3 over 11m) kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[ks-router-config kubesphere-config host-time kube-api-access-xp572]: timed out waiting for the condition

kubectl describe pod --namespace kubesphere-system ks-controller-manager-8569fb495c-b9qwv

Name: ks-controller-manager-8569fb495c-b9qwv

Namespace: kubesphere-system

Priority: 0

Node: master/192.168.2.206

Start Time: Fri, 07 Apr 2023 13:33:19 +0000

Labels: app=ks-controller-manager

pod-template-hash=8569fb495c

tier=backend

Annotations: <none>

Status: Pending

IP:

IPs: <none>

Controlled By: ReplicaSet/ks-controller-manager-8569fb495c

Containers:

ks-controller-manager:

Container ID:

Image: registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2

Image ID:

Ports: 8080/TCP, 8443/TCP

Host Ports: 0/TCP, 0/TCP

Command:

controller-manager

--logtostderr=true

--leader-elect=true

State: Waiting

Reason: ContainerCreating

Ready: False

Restart Count: 0

Limits:

cpu: 1

memory: 1000Mi

Requests:

cpu: 30m

memory: 50Mi

Environment: <none>

Mounts:

/etc/kubesphere/ from kubesphere-config (rw)

/etc/localtime from host-time (ro)

/tmp/k8s-webhook-server/serving-certs from webhook-secret (rw)

/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nfk8r (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

kubesphere-config:

Type: ConfigMap (a volume populated by a ConfigMap)

Name: kubesphere-config

Optional: false

webhook-secret:

Type: Secret (a volume populated by a Secret)

SecretName: ks-controller-manager-webhook-cert

Optional: false

host-time:

Type: HostPath (bare host directory volume)

Path: /etc/localtime

HostPathType:

kube-api-access-nfk8r:

Type: Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds: 3607

ConfigMapName: kube-root-ca.crt

ConfigMapOptional: <nil>

DownwardAPI: true

QoS Class: Burstable

Node-Selectors: <none>

Tolerations: CriticalAddonsOnly op=Exists

node-role.kubernetes.io/master:NoSchedule

node.kubernetes.io/not-ready:NoExecute op=Exists for 60s

node.kubernetes.io/unreachable:NoExecute op=Exists for 60s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 14m default-scheduler Successfully assigned kubesphere-system/ks-controller-manager-8569fb495c-b9qwv to master

Warning FailedMount 14m kubelet MountVolume.SetUp failed for volume "webhook-secret" : failed to sync secret cache: timed out waiting for the condition

Warning FailedMount 7m43s (x3 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kubesphere-config webhook-secret host-time kube-api-access-nfk8r]: timed out waiting for the condition

Warning FailedMount 3m12s kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kube-api-access-nfk8r kubesphere-config webhook-secret host-time]: timed out waiting for the condition

Warning FailedMount 2m (x14 over 14m) kubelet MountVolume.SetUp failed for volume "kubesphere-config" : configmap "kubesphere-config" not found

Warning FailedMount 55s (x2 over 5m28s) kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[host-time kube-api-access-nfk8r kubesphere-config webhook-secret]: timed out waiting for the condition

都是源于configmap kubesphere-config不存在

多等了一会发现第二个pod ks-controller-manager-8569fb495c-b9qwv自己成功了，可能有降级？(来自20230701的更新：只是单纯的慢，实际是monitor还没装完，所以不动它也能自己好）


k8s3@k8s3:~$ kubectl describe pod --namespace kubesphere-system ks-controller-manager-8569fb495c-b9qwv
Name:         ks-controller-manager-8569fb495c-b9qwv
Namespace:    kubesphere-system
Priority:     0
Node:         master/192.168.2.206
Start Time:   Fri, 07 Apr 2023 13:33:19 +0000
Labels:       app=ks-controller-manager
              pod-template-hash=8569fb495c
              tier=backend
Annotations:  cni.projectcalico.org/containerID: 23d0c4603543c2e6bb05897f1801e37dc9475985d463721d3728553e85d4011c
              cni.projectcalico.org/podIP: 10.233.70.2/32
              cni.projectcalico.org/podIPs: 10.233.70.2/32
Status:       Running
IP:           10.233.70.2
IPs:
  IP:           10.233.70.2
Controlled By:  ReplicaSet/ks-controller-manager-8569fb495c
Containers:
  ks-controller-manager:
    Container ID:  docker://37abb2be10787e5ca931ad515edf70099aed3d5516ec24772aaff0903f915c74
    Image:         registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2
    Image ID:      docker-pullable://registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager@sha256:dce79d8513fa19105516e312d5b3b9828171d660a78aa7ebd866e42f6a51db25
    Ports:         8080/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      controller-manager
      --logtostderr=true
      --leader-elect=true
    State:          Running
      Started:      Fri, 07 Apr 2023 13:48:38 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1000Mi
    Requests:
      cpu:        30m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /etc/kubesphere/ from kubesphere-config (rw)
      /etc/localtime from host-time (ro)
      /tmp/k8s-webhook-server/serving-certs from webhook-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nfk8r (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kubesphere-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubesphere-config
    Optional:  false
  webhook-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ks-controller-manager-webhook-cert
    Optional:    false
  host-time:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:
  kube-api-access-nfk8r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 60s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 60s
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    17m                   default-scheduler  Successfully assigned kubesphere-system/ks-controller-manager-8569fb495c-b9qwv to master
  Warning  FailedMount  17m                   kubelet            MountVolume.SetUp failed for volume "webhook-secret" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  10m (x3 over 15m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kubesphere-config webhook-secret host-time kube-api-access-nfk8r]: timed out waiting for the condition
  Warning  FailedMount  6m25s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kube-api-access-nfk8r kubesphere-config webhook-secret host-time]: timed out waiting for the condition
  Warning  FailedMount  5m13s (x14 over 17m)  kubelet            MountVolume.SetUp failed for volume "kubesphere-config" : configmap "kubesphere-config" not found
  Warning  FailedMount  4m8s (x2 over 8m41s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[host-time kube-api-access-nfk8r kubesphere-config webhook-secret]: timed out waiting for the condition
  Normal   Pulling      2m59s                 kubelet            Pulling image "registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2"
  Normal   Pulled       2m17s                 kubelet            Successfully pulled image "registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2" in 42.28238507s
  Normal   Created      2m16s                 kubelet            Created container ks-controller-manager
  Normal   Started      2m14s                 kubelet            Started container ks-controller-manager

k8s3@k8s3:~$ kubectl describe pod --namespace kubesphere-system ks-controller-manager-8569fb495c-b9qwv

Name: ks-controller-manager-8569fb495c-b9qwv

Namespace: kubesphere-system

Priority: 0

Node: master/192.168.2.206

Start Time: Fri, 07 Apr 2023 13:33:19 +0000

Labels: app=ks-controller-manager

pod-template-hash=8569fb495c

tier=backend

Annotations: cni.projectcalico.org/containerID: 23d0c4603543c2e6bb05897f1801e37dc9475985d463721d3728553e85d4011c

cni.projectcalico.org/podIP: 10.233.70.2/32

cni.projectcalico.org/podIPs: 10.233.70.2/32

Status: Running

IP: 10.233.70.2

IPs:

IP: 10.233.70.2

Controlled By: ReplicaSet/ks-controller-manager-8569fb495c

Containers:

ks-controller-manager:

Container ID: docker://37abb2be10787e5ca931ad515edf70099aed3d5516ec24772aaff0903f915c74

Image: registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2

Image ID: docker-pullable://registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager@sha256:dce79d8513fa19105516e312d5b3b9828171d660a78aa7ebd866e42f6a51db25

Ports: 8080/TCP, 8443/TCP

Host Ports: 0/TCP, 0/TCP

Command:

controller-manager

--logtostderr=true

--leader-elect=true

State: Running

Started: Fri, 07 Apr 2023 13:48:38 +0000

Ready: True

Restart Count: 0

Limits:

cpu: 1

memory: 1000Mi

Requests:

cpu: 30m

memory: 50Mi

Environment: <none>

Mounts:

/etc/kubesphere/ from kubesphere-config (rw)

/etc/localtime from host-time (ro)

/tmp/k8s-webhook-server/serving-certs from webhook-secret (rw)

/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nfk8r (ro)

Conditions:

Type Status

Initialized True

Ready True

ContainersReady True

PodScheduled True

Volumes:

kubesphere-config:

Type: ConfigMap (a volume populated by a ConfigMap)

Name: kubesphere-config

Optional: false

webhook-secret:

Type: Secret (a volume populated by a Secret)

SecretName: ks-controller-manager-webhook-cert

Optional: false

host-time:

Type: HostPath (bare host directory volume)

Path: /etc/localtime

HostPathType:

kube-api-access-nfk8r:

Type: Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds: 3607

ConfigMapName: kube-root-ca.crt

ConfigMapOptional: <nil>

DownwardAPI: true

QoS Class: Burstable

Node-Selectors: <none>

Tolerations: CriticalAddonsOnly op=Exists

node-role.kubernetes.io/master:NoSchedule

node.kubernetes.io/not-ready:NoExecute op=Exists for 60s

node.kubernetes.io/unreachable:NoExecute op=Exists for 60s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 17m default-scheduler Successfully assigned kubesphere-system/ks-controller-manager-8569fb495c-b9qwv to master

Warning FailedMount 17m kubelet MountVolume.SetUp failed for volume "webhook-secret" : failed to sync secret cache: timed out waiting for the condition

Warning FailedMount 10m (x3 over 15m) kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kubesphere-config webhook-secret host-time kube-api-access-nfk8r]: timed out waiting for the condition

Warning FailedMount 6m25s kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[kube-api-access-nfk8r kubesphere-config webhook-secret host-time]: timed out waiting for the condition

Warning FailedMount 5m13s (x14 over 17m) kubelet MountVolume.SetUp failed for volume "kubesphere-config" : configmap "kubesphere-config" not found

Warning FailedMount 4m8s (x2 over 8m41s) kubelet Unable to attach or mount volumes: unmounted volumes=[kubesphere-config], unattached volumes=[host-time kube-api-access-nfk8r kubesphere-config webhook-secret]: timed out waiting for the condition

Normal Pulling 2m59s kubelet Pulling image "registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2"

Normal Pulled 2m17s kubelet Successfully pulled image "registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.2" in 42.28238507s

Normal Created 2m16s kubelet Created container ks-controller-manager

Normal Started 2m14s kubelet Started container ks-controller-manager

尝试手动修改kubesphere-config 里的jwtToken

kubectl get cm kubesphere-config -n kubesphere-system -o yaml

删除pod ks-apiserver后不再报错


#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://192.168.2.206:30880
Account: admin
Password: 123456
NOTES：
  1. After you log into the console, please check the
     monitoring status of service components in
     "Cluster Management". If any service is not
     ready, please wait patiently until all components
     are up and running.
  2. Please change the default password after login.

#####################################################
https://kubesphere.io             2023-04-07 14:27:35
#####################################################
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

#####################################################

### Welcome to KubeSphere! ###

#####################################################

Console: http://192.168.2.206:30880

Account: admin

Password: 123456

NOTES：

1. After you log into the console, please check the

monitoring status of service components in

"Cluster Management". If any service is not

ready, please wait patiently until all components

are up and running.

2. Please change the default password after login.

#####################################################

https://kubesphere.io 2023-04-07 14:27:35

#####################################################

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

参考

https://kubesphere.io/zh/docs/v3.3/installing-on-linux/introduction/multioverview/

https://github.com/kubesphere/kubesphere/issues/4729

http://www.minio.org.cn/docs/minio/kubernetes/upstream/

https://docs.ceph.com/en/quincy/cephfs/index.html

20230311neo4j测试

0.背景

因json文件、mysql、mongo存储都不能方便的满足我的要求，计划后续使用图数据库来存储复杂的的图状关系

1.安装

看了一眼dockerhub有现成的镜像，直接选了最新的


docker run -d \
    --publish=7474:7474 --publish=7687:7687 \
    --volume=/data/neo4j/data:/data \
    neo4j:5.5.0-community

docker run -d \

--publish=7474:7474 --publish=7687:7687 \

--volume=/data/neo4j/data:/data \

neo4j:5.5.0-community

7474是web端浏览器的端口，7687是数据库本身的端口。启动后默认用户名密码都是neo4j，记得修改密码。这里测试，因此用弱密码12345678

2.使用

2.1.基本语法

node节点


()                  //匿名的节点，不包含标签和变量，可以代表数据库的任意节点
(p:Person)          //使用变量p和标签Person
(:Technology)       //没有变量，标签Technology
(work:Company)      //使用变量work和标签Company

() //匿名的节点，不包含标签和变量，可以代表数据库的任意节点

(p:Person) //使用变量p和标签Person

(:Technology) //没有变量，标签Technology

(work:Company) //使用变量work和标签Company

relationship关系


//按这个方向存储数据
CREATE (p:Person)-[:LIKES]->(t:Technology)

//反向查询关系不会返回任何结果
MATCH (p:Person)<-[:LIKES]-(t:Technology)

//如果不确定方向，可以不执行关系的方向来查询
MATCH (p:Person)-[:LIKES]-(t:Technology)

//关系也可以用变量
-[rel]->
-[rel:LIKES]->

//按这个方向存储数据

CREATE (p:Person)-[:LIKES]->(t:Technology)

//反向查询关系不会返回任何结果

MATCH (p:Person)<-[:LIKES]-(t:Technology)

//如果不确定方向，可以不执行关系的方向来查询

MATCH (p:Person)-[:LIKES]-(t:Technology)

//关系也可以用变量

-[rel]->

-[rel:LIKES]->

properties属性


Node的属性(p:Person {name: 'Sally'})

Relationship的属性: -[rel:IS_FRIENDS_WITH {since: 2018}]->

Node的属性(p:Person {name: 'Sally'})

Relationship的属性: -[rel:IS_FRIENDS_WITH {since: 2018}]->

好了，你已经学会基本的语法了，是时候设计一个复杂的项目了（

3.集成到java中

尝试 spring-boot 3 + neo4j ORM

3.1.修改pom.xml


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-neo4j</artifactId>
        </dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-data-neo4j</artifactId>

</dependency>

3.2.entity类


import lombok.Data;
import org.springframework.data.annotation.CreatedDate;
import org.springframework.data.annotation.LastModifiedDate;
import org.springframework.data.neo4j.core.schema.*;
import org.springframework.data.neo4j.core.schema.Relationship.Direction;
import org.springframework.data.neo4j.core.support.UUIDStringGenerator;

import java.util.ArrayList;
import java.util.List;

@Data
@Node("LabelName")
public class SomeNodeEntity {

    @Id
    @GeneratedValue(generatorClass = UUIDStringGenerator.class)
    private String id;
    @Property("title")
    private String title;
    @Property("description")
    private String description;
    @CreatedDate //need @EnableNeo4jAuditing
    private Long createdAt;
    @LastModifiedDate //need @EnableNeo4jAuditing
    private Long updatedAt;
    @Relationship(type = "DIRECTED", direction = Direction.INCOMING)
    private List<OtherNodeEntity> directors = new ArrayList<>();

}

import lombok.Data;

import org.springframework.data.annotation.CreatedDate;

import org.springframework.data.annotation.LastModifiedDate;

import org.springframework.data.neo4j.core.schema.*;

import org.springframework.data.neo4j.core.schema.Relationship.Direction;

import org.springframework.data.neo4j.core.support.UUIDStringGenerator;

import java.util.ArrayList;

import java.util.List;

@Data

@Node("LabelName")

public class SomeNodeEntity {

@Id

@GeneratedValue(generatorClass = UUIDStringGenerator.class)

private String id;

@Property("title")

private String title;

@Property("description")

private String description;

@CreatedDate //need @EnableNeo4jAuditing

private Long createdAt;

@LastModifiedDate //need @EnableNeo4jAuditing

private Long updatedAt;

@Relationship(type = "DIRECTED", direction = Direction.INCOMING)

private List<OtherNodeEntity> directors = new ArrayList<>();

}

3.3 repo类


package cn.hylstudio.skykoma.data.collector.repo.neo4j;

import cn.hylstudio.skykoma.data.collector.entity.neo4j.MovieEntity;
import org.springframework.data.neo4j.repository.Neo4jRepository;

public interface MovieEntityRepo extends Neo4jRepository<MovieEntity, String> {
    MovieEntity findByTitle(String title);
}

package cn.hylstudio.skykoma.data.collector.repo.neo4j;

import cn.hylstudio.skykoma.data.collector.entity.neo4j.MovieEntity;

import org.springframework.data.neo4j.repository.Neo4jRepository;

public interface MovieEntityRepo extends Neo4jRepository<MovieEntity, String> {

MovieEntity findByTitle(String title);

}

注意：关系字段中，如果实体不存在则会自动创建Node，但如果修改实体导致数据减少，OGM不会删除关联的实体，只会删除关系

参考

https://docs.spring.io/spring-data/neo4j/docs/current/reference/html/#preface

https://neo4j.com/docs/getting-started/current/get-started-with-neo4j/

https://neo4j.com/docs/browser-manual/current/

https://www.w3cschool.cn/neo4j/neo4j_cql_introduction.html

20230203IDEA插件开发-Action的实现

背景

开发的插件点击后UI卡住，怀疑是阻塞了UI线程。。。。仔细看下文档

文档翻译-Action的实现

自定义的action类是派生自抽象类 AnAction的子类。当用户操作菜单项或工具栏按钮时，IDEA会调用action的方法

基于 AnAction 的类不能包含任何属性，因为 AnAction 的实例会存在 application整个生命周期。如果 AnAction 类使用了属性存储了短声明周期的数据但没有合理的清除，会造成内存泄漏。例如如果你把Project对象存到了这里，当用户关闭了这个Project之后就会发生内存泄漏

方法Override覆盖原则

每个action可以覆盖 AnAction.update()，必须覆盖AnAction.actionPerformed().

AnAction.update() 用来被IDEA调用更新action的状态，action是否可用/是否可见取决于action在当前UI上是否可用.AnActionEvent 对象会传递给这个方法并且包含当前action上下文的信息。可以通过修改和事件上下文关联的Presentation对象状态来改变action是否可用。像Overriding the AnAction.update() Method说的一样， update() 方法的执行速度非常重要

2022.3+版本后AnAction.getActionUpdateThread() 返回 ActionUpdateThread 线程决定了 update() 方法是否在 background thread (BGT) or the event-dispatching thread (EDT)上被调用。首选的方式是在BGT上执行update方法, 这样可以保证application侧读取 PSI, the virtual file system (VFS), 或 project models的有利条件。在BGT上运行update的Actions不应该直接访问Swing 组件的结构。反之EDT上运行的update的Action必须禁止访问PSI, VFS, 或 project的数据，但可以访问Swing组件和其他UI模型。所有由DataContext 提供的可访问数据在这里有说明 Determining the Action Context. 当必须从BGT切换回EDT时，需要使用 AnActionEvent.getUpdateSession()方法获得 UpdateSession 并调用 UpdateSession.compute() 在EDT上运行方法。从2022.3开始，插件的DevKit将会有检查 Plugin DevKit | Code | ActionUpdateThread 是否丢失，来提醒插件作者实现这个方法

AnAction.actionPerformed() 将会在可用之后当用户选择action时被IDEA调用。这个方法是实现action最麻烦的任务，它包含了当action被调用时要执行的代码。 actionPerformed()方法也接收一个 AnActionEvent作为参数，它用来访问任何上下文相关的信息例如 projects, files, selection。。。参考Overriding the AnAction.actionPerformed() Method 获取更多信息

这里还有其他的方法可以在 AnAction 类中被覆盖，例如改变action默认的 Presentation。还有一个使用场景可以通过覆盖构造器来达成，当动态注册到分组的时候demo可以在这里看到Grouping Actions

AnAction.update

IDEA会周期性的调用AnAction.update() 方法来响应用户的操作。 update() 方法用于动态判断action在当前上下文是否应该禁用或启用。方法的实现必须确保可用性状态和可见性状态，否则指定的action将会”卡住”。（译者注：这里的意思应该是从用户看来是卡住的现象，实际就是没响应任何动作）

AnAction.update()方法可能在UI线程上频繁的调用。所以这个方法必须非常快的执行，没有必须执行的实际动作。例如检查选中的对象是否可用，但不做任何文件系统相关的动作。如果新的状态无法被快速的确定是否可用，判定过程应该在AnAction.actionPerformed()中执行，并通知notify用户这个动作在当前上下文不能被执行

线程

//TODO

参考

https://plugins.jetbrains.com/docs/intellij/basic-action-system.html

https://plugins.jetbrains.com/docs/intellij/general-threading-rules.html#modality-and-invokelater

20230224dev-assistant idea plugin开发

既然ChatGPT能理解自然语言，为什么不让它来帮我写代码完成这个开发任务呢？

基于这个猜想，我尝试让它编写IDEA插件逻辑，以一个LOGGER插入为例。

我给的命令是

“

利用IntelliJ Platform Plugin SDK获取当前光标所在类的成员变量，检测是否存在LOGGER，如果不存在，在类声明的第一行插入LOGGER的声明，并import对应的slf4j依赖。用java实现

”

它给出的代码如下


import com.intellij.openapi.actionSystem.*;
import com.intellij.openapi.project.Project;
import com.intellij.psi.*;
import com.intellij.psi.search.GlobalSearchScope;
import com.intellij.psi.search.searches.ClassInheritorsSearch;
import com.intellij.psi.search.searches.ReferencesSearch;
import com.intellij.psi.util.PsiUtil;
import com.intellij.psi.util.PsiUtilCore;
import com.intellij.psi.util.PsiUtilCore;
import com.intellij.util.Query;
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;

import java.util.List;
import java.util.Objects;

public class AddLoggerToClassAction extends AnAction {

    private static final String LOGGER_DECLARATION = "private static final org.slf4j.Logger LOGGER = org.slf4j.LoggerFactory.getLogger(%s.class);";

    private static final String SLF4J_IMPORT = "import org.slf4j.LoggerFactory;";

    @Override
    public void actionPerformed(@NotNull AnActionEvent e) {
        // 获取当前项目和光标所在元素
        PsiFile psiFile = e.getData(CommonDataKeys.PSI_FILE);
        Editor editor = e.getData(CommonDataKeys.EDITOR);
        Project project = e.getProject();

        if (psiFile == null || editor == null || project == null) {
            return;
        }

        // 获取当前光标所在元素
        PsiElement psiElement = psiFile.findElementAt(editor.getCaretModel().getOffset());
        PsiClass psiClass = PsiUtil.getTopLevelClass(psiElement);

        if (psiClass == null) {
            return;
        }

        // 检查类中是否已经存在 LOGGER
        if (hasLogger(psiClass)) {
            return;
        }

        // 插入 LOGGER 声明并 import slf4j 依赖
        PsiElementFactory elementFactory = JavaPsiFacade.getElementFactory(project);
        PsiImportList importList = psiFile.getImportList();

        if (importList != null && importList.findOnDemandImportStatement("org.slf4j") == null) {
            importList.add(elementFactory.createImportStatementOnDemand("org.slf4j"));
        }

        PsiField loggerField = elementFactory.createFieldFromText(String.format(LOGGER_DECLARATION, psiClass.getName()), psiClass);

        PsiElement firstElement = PsiUtilCore.findFirstNonWhitespaceElement(psiClass);
        psiClass.addBefore(loggerField, firstElement);
    }

    private static boolean hasLogger(PsiClass psiClass) {
        for (PsiField field : psiClass.getFields()) {
            if (Objects.equals(field.getType().getCanonicalText(), "org.slf4j.Logger") &&
                    Objects.equals(field.getName(), "LOGGER") &&
                    field.getModifierList().hasModifierProperty(PsiModifier.STATIC) &&
                    field.getModifierList().hasModifierProperty(PsiModifier.FINAL)) {
                return true;
            }
        }
        return false;
    }
}

import com.intellij.openapi.actionSystem.*;

import com.intellij.openapi.project.Project;

import com.intellij.psi.*;

import com.intellij.psi.search.GlobalSearchScope;

import com.intellij.psi.search.searches.ClassInheritorsSearch;

import com.intellij.psi.search.searches.ReferencesSearch;

import com.intellij.psi.util.PsiUtil;

import com.intellij.psi.util.PsiUtilCore;

import com.intellij.util.Query;

import org.jetbrains.annotations.NotNull;

import org.jetbrains.annotations.Nullable;

import java.util.List;

import java.util.Objects;

public class AddLoggerToClassAction extends AnAction {

private static final String LOGGER_DECLARATION = "private static final org.slf4j.Logger LOGGER = org.slf4j.LoggerFactory.getLogger(%s.class);";

private static final String SLF4J_IMPORT = "import org.slf4j.LoggerFactory;";

@Override

public void actionPerformed(@NotNull AnActionEvent e) {

// 获取当前项目和光标所在元素

PsiFile psiFile = e.getData(CommonDataKeys.PSI_FILE);

Editor editor = e.getData(CommonDataKeys.EDITOR);

Project project = e.getProject();

if (psiFile == null || editor == null || project == null) {

return;

}

// 获取当前光标所在元素

PsiElement psiElement = psiFile.findElementAt(editor.getCaretModel().getOffset());

PsiClass psiClass = PsiUtil.getTopLevelClass(psiElement);

if (psiClass == null) {

return;

}

// 检查类中是否已经存在 LOGGER

if (hasLogger(psiClass)) {

return;

}

// 插入 LOGGER 声明并 import slf4j 依赖

PsiElementFactory elementFactory = JavaPsiFacade.getElementFactory(project);

PsiImportList importList = psiFile.getImportList();

if (importList != null && importList.findOnDemandImportStatement("org.slf4j") == null) {

importList.add(elementFactory.createImportStatementOnDemand("org.slf4j"));

}

PsiField loggerField = elementFactory.createFieldFromText(String.format(LOGGER_DECLARATION, psiClass.getName()), psiClass);

PsiElement firstElement = PsiUtilCore.findFirstNonWhitespaceElement(psiClass);

psiClass.addBefore(loggerField, firstElement);

}

private static boolean hasLogger(PsiClass psiClass) {

for (PsiField field : psiClass.getFields()) {

if (Objects.equals(field.getType().getCanonicalText(), "org.slf4j.Logger") &&

Objects.equals(field.getName(), "LOGGER") &&

field.getModifierList().hasModifierProperty(PsiModifier.STATIC) &&

field.getModifierList().hasModifierProperty(PsiModifier.FINAL)) {

return true;

}

return false;

}

对不起，是我高估了AI的实力，它部分代码在胡写。。。。得自己验证是否可行，它的代码看着没毛病，但实际很多地方都有问题= =

还是自己动手写吧，参考 https://www.bilibili.com/video/BV15D4y137Sp/

20230218长期计划：dev-assistant

背景

最近试用ChatGPT给我带来了新的想法，如果你觉得我后面写的比较离谱，那确实有可能是我错误的高估了现有AI的能力，就当我是在做梦吧hhhh。总的来说这种对话的方式极大的降低了人机交互的门槛。相比于传统的交互方式，以自然语言和计算机进行有效的交互是听上去理所当然，但实际上一直都不太成熟的方式。而ChatGPT给我们带来了新的希望，我认为之后这个技术足够成熟之后将会改变很多现有的交互逻辑。可以设想，假如以后AI能正确理解绝大部分的自然语言，那么是不是很多任务都不需要用鼠标和键盘来操作了？而根据网友给出的参考资料，有人正在尝试将自然语言转化为一系列的API调用（参考最后的附录），这意味着AI生成的内容将具有可执行的特性，不同的API完全可以代替现有的人工操作，完成一场人机交互的变革。

继上一次我根据AST上下文来做可控的LiveTemplate之后，我继续尝试了提取代码的AST节点并结构化其中的语义信息配合可视化来提高日常工作效率。但短期内无论从安全角度还是对成果负责的角度来说，程序生成的结果，或者说AI的动作都必须经过人工的确认以避免AI失控造成不好的后果。也就是说在相当长的一段时间内，AI的操作必须要可观测、可干预。以此为前提让我有了一个科幻的想法：我对电脑用自然语言下达任务后，电脑在我的监视下逐步完成任务。选择离我最近的场景来说就是日常的编码工作在有详细设计的情况下是否有希望完全自动化？

可行性分析

现有的代码生成AI大多是基于工程内已有的文件、注释、当前文件光标的上下文来猜测并生成代码，但这里面缺少了一个重要的信息就是目的。AI无法准确猜测人的意图，比如完成某个API的编码通常会分成接入层、业务层、数据层，多层次+多个方法的组合。AI每次只能预测我选定光标位置的代码内容，而无法连续跳跃多个层次来完成这个任务，依然需要人工大量操作IDE。如果能以自然语言提供给AI这个目的，是否有可能让AI自行完成这一系列的动作呢？由此会带来的一系列问题。开发人员对IDE的操作是否能通过文本精确的描述？开发人员操作IDE时依赖的信息有没有可能被收集起来并打上标记？如果将我日常操作IDE的记录和操作IDE时候的信息完整的记录下来并打上标记，能否训练出一个AI来理解并代替我进行IDE的操作？

以我目前的AI水平还停留在手写数字识别的入门级别，虽然有计划学习相关知识，不过并不是抱很大的希望能独立搞定模型的设计、训练之类如此专业的任务。但以我目前的能力，处理点数据、集成已有的功能还是不成什么问题的。因此我有一个长期计划的想法，设计并收集开发操作IDE的上下文信息，包括但不限于工程类型、文件组织结构、版本控制信息、产物信息、光标上下文信息等。依赖这些详细的信息，自动或手动的分析任务场景打上标记作为下一步AI训练的基础数据。抽取并实现一组API来获取、操作IDE内的动作并实时的反馈到屏幕上供人工干预这个过程。至于这个AI怎么训练，那可以将这个项目开源交给AI的专业人士搞定，我可以力所能及的收集数据或组合已有功能。退一步来说，就算AI搞不出来，收集的这些信息可以继续试用传统方式来处理，也能推动现一点点xxxAsCode的进程。

方案设计

项目名代号暂定为skykoma，分别取自skynet和tachikoma的一半，如果有好的名字欢迎反馈给我。整个项目拆分成若干模块分别独立实现，如果有生之年每个部分最后都能完全实现，则组合到一起。

信息收集模块：收集当前工程结构、版本控制、产物、光标上下文、IDE等信息，这些信息等价于开发员看到和了解到的信息。收集开发人员对IDE的操作序列数据。

信息处理模块：对收集到的信息做预处理，标记子序列对应的场景并用自然语言描述，在必要的位置插入人工干预的节点，以此作为训练的基础数据。

指令生成模块：接受自然语言描述的任务，生成一系列结构化指令

指令调度模块：接收一系列结构化指令并转化成具体的动作，动作的序列必须满足人工可观测、可干预、可理解，符合正常人操作的习惯，且动作序列有对应的API。理论上训练数据如果足够好可以达到部分效果，但为了可控和安全还是需要一个调度模块，为的是有机会通过程序强制干预AI的结果，加一道保险。

动作执行模块：给定具体动作并执行的同时收集结果、可视化反馈

以一个常见的场景为例：

自然语言描述：查找ClassA的所有实现类。

指令生成模块：打开搜索功能，输入关键字ClassA，从结果中定位ClassA，打开ClassA对应的文件编辑器，光标定位到ClassA的声明位置，搜索实现类(Ctrl+Alt+B)

指令调度模块插入必要指令：打开搜索功能，输入关键字ClassA，从结果中定位ClassA，判断是否能定位到ClassA（满足人工干预的要求），尝试打开ClassA对应的文件编辑器，光标定位到ClassA的声明位置，搜索实现类(Ctrl+Alt+B)，收集搜索结果同时展示到屏幕上（满足可观测的要求）

指令调度模块转化指令到具体的API：打开搜索功能，输入关键字ClassA，从结果中定位ClassA，判断是否能定位到ClassA（满足人工干预的要求），尝试打开ClassA对应的文件编辑器，搜索当前文件下所有的class声明（指令调度需要生成API可支持的操作），寻找ClassA的声明位置（指令调度需要生成API可支持的操作），搜索实现类(Ctrl+Alt+B)，收集搜索结果同时展示到屏幕上（满足可观测的要求）

执行模块：依赖指定调度的API结果控制IDE依次执行对应的指令

以下的ChatGPT给出的示例代码


import com.intellij.openapi.actionSystem.AnAction;
import com.intellij.openapi.actionSystem.AnActionEvent;
import com.intellij.openapi.project.Project;
import com.intellij.openapi.util.text.StringUtil;
import com.intellij.psi.*;
import org.jetbrains.annotations.NotNull;

import java.util.ArrayList;
import java.util.List;

public class FindClassImplementationsAction extends AnAction {

    @Override
    public void actionPerformed(@NotNull AnActionEvent e) {
        // 获取当前项目
        Project project = e.getProject();
        if (project == null) {
            return;
        }

        // 获取当前光标所在的类
        PsiElement psiElement = e.getData(CommonDataKeys.PSI_ELEMENT);
        if (psiElement instanceof PsiClass) {
            PsiClass psiClass = (PsiClass) psiElement;

            // 查找所有实现该类的子类
            List<PsiClass> implementations = getImplementations(psiClass);

            // 输出结果
            StringBuilder sb = new StringBuilder();
            sb.append("Implementations of ").append(psiClass.getName()).append(":\n\n");
            for (PsiClass implementation : implementations) {
                sb.append(implementation.getQualifiedName()).append("\n");
            }
            System.out.println(sb.toString());
        }
    }

    private List<PsiClass> getImplementations(PsiClass psiClass) {
        List<PsiClass> implementations = new ArrayList<>();
        Project project = psiClass.getProject();
        GlobalSearchScope scope = GlobalSearchScope.projectScope(project);
        String classQualifiedName = psiClass.getQualifiedName();

        // 查找实现该类或者接口的所有类
        Query<PsiClass> query = ClassInheritorsSearch.search(psiClass, scope, true);
        for (PsiClass result : query) {
            if (result != null && StringUtil.isNotEmpty(result.getQualifiedName())) {
                if (!result.getQualifiedName().equals(classQualifiedName)) {
                    implementations.add(result);
                }
            }
        }
        return implementations;
    }
}

import com.intellij.openapi.actionSystem.AnAction;

import com.intellij.openapi.actionSystem.AnActionEvent;

import com.intellij.openapi.project.Project;

import com.intellij.openapi.util.text.StringUtil;

import com.intellij.psi.*;

import org.jetbrains.annotations.NotNull;

import java.util.ArrayList;

import java.util.List;

public class FindClassImplementationsAction extends AnAction {

@Override

public void actionPerformed(@NotNull AnActionEvent e) {

// 获取当前项目

Project project = e.getProject();

if (project == null) {

return;

}

// 获取当前光标所在的类

PsiElement psiElement = e.getData(CommonDataKeys.PSI_ELEMENT);

if (psiElement instanceof PsiClass) {

PsiClass psiClass = (PsiClass) psiElement;

// 查找所有实现该类的子类

List<PsiClass> implementations = getImplementations(psiClass);

// 输出结果

StringBuilder sb = new StringBuilder();

sb.append("Implementations of ").append(psiClass.getName()).append(":\n\n");

for (PsiClass implementation : implementations) {

sb.append(implementation.getQualifiedName()).append("\n");

}

System.out.println(sb.toString());

}

private List<PsiClass> getImplementations(PsiClass psiClass) {

List<PsiClass> implementations = new ArrayList<>();

Project project = psiClass.getProject();

GlobalSearchScope scope = GlobalSearchScope.projectScope(project);

String classQualifiedName = psiClass.getQualifiedName();

// 查找实现该类或者接口的所有类

Query<PsiClass> query = ClassInheritorsSearch.search(psiClass, scope, true);

for (PsiClass result : query) {

if (result != null && StringUtil.isNotEmpty(result.getQualifiedName())) {

if (!result.getQualifiedName().equals(classQualifiedName)) {

implementations.add(result);

}

return implementations;

}

概要设计

以上模块将会划分成多个开源项目，为了便于实现部分逻辑上的模块会放到同一个实际的项目中

交互层 skykoma-plugin-idea(1), skykoma-plugin-vscode(2), skykoma-chatbot(3) , skykoma-agent(待定), skykoma-workspace(待定) 顾名思义交互层定位于用户的输入和输出，同一管理最外层的交互

数据层 skykoma-data-collector(4), skykoma-data-handler(5), 收集和处理所有相关的数据并存储、上报

逻辑层 skykoma-core(6), skykoma-cmd-generator(7), skykoma-api-transformer(8), skykoma-api-worker(9), core就是核心的模型负责自然语言处理并生成第一版指令序列，cmd-generator负责指令的调度、插值、过滤以符合前文所说的各种条件，api-transformer负责把指令序列做翻译成现存工具的API调用, worker负责调用具体的API

执行层 skykoma-api: skykoma-api-idea,skykoma-api-vscode,skykoma-api-git(10),skykoma-api-mvn(11),skykoma-api-gitlab(12),skykoma-api-github(13), 顾名思义，实现整合各个现有工具的功能，以localhost HTTP、cmdline、rpc等方式对外提供服务

里程碑规划

v0.1 搭框架

手动完成realworld或者其他什么的工程编写作为测试项目，要求满足常见的企业级开发特性。

初始化idea插件工程（https://github.com/dev-assistant/skykoma-plugin-idea），支持私有化部署，收集idea当前的项目信息。

初始化数据收集工程（https://github.com/dev-assistant/skykoma-data-collector），支持私有化部署，数据存储位置可自行选择。

skykoma-plugin-idea概要设计

架构上因为不想出现多个idea插件，因此设计一个skykoma-api-idea负责和idea的api具体交互，kykoma-plugin-idea依赖skykoma-api-idea来获得对IDE完整的感知和控制能力。plugin支持常用的LiveTemplate，设计生成规则，在固定场景下支持人工触发生成。skykoma-api-idea收集idea的状态和操作事件并上报给skykoma-data-collector达成数据收集目的。同时也可以通过websocket、http等协议监听本地端口或从外部的控制模块获取指令。

模板代码自动生成功能

模板代码生成主要依赖于LiveTemplate实现，分为静态生成和动态生成，静态生成是上下文无关的，动态生成可感知代码上下文。

静态生成

如@Autowire @Column 之类的，试用idea自带API完成命名风格的转换。

动态生成

日志：寻找上下文中可用的LOGGER，若LOGGER不存在则尝试先生成LOGGER字段，并根据当前上下文插入LOGGER语句。

SQL：寻找上下文中的ORM注解信息，根据字段类型生成默认的建表语句。

数据复制：根据上下文获取指定类的所有属性，复制到当前上下文中或指定的对象中，代替动态的反射copy避免运行时异常。

Service：增强自带的创建类过程，根据类名猜测正确的包名，一键生成带注解的Service类和接口并实现对应接口满足主流项目的风格要求，减少人工操作。

代码见 https://github.com/dev-assistant/skykoma-plugin-idea

数据上报功能

TODO 设计数据结构表示上下文，包括当前窗口信息、窗口标题、项目信息、git信息、全量AST信息等

代码见 https://github.com/dev-assistant/skykoma-data-collector

skykoma-data-collector概要设计

负责skykoma需要的数据收集工作，用多种合适的存储方案持久化用户每个操作的上下文信息和事件。语义图暂定neo4j图数据库，其他结构化信息继续使用mysql。

设计数据结构存储相关信息

有了全量AST和语义，才有可能做到
1.给定任意项目任意API、自动提取可运行的最小依赖沙盒环境
2.任意复杂业务逻辑分析，不重不漏
3.沙盒环境的修改可回填到原始项目：可debug、做测试用例、分析逻辑、修改
4.因沙河内依赖最小化了，理论上可以用自然语言总结所有的上下文和代码意图信息、为炼丹做准备。退一步即使炼丹失败，也能快速抽象出大量模板代码来使用
5.反向给定自然语言生成沙盒环境、可完成对任意项目的patch


1.ProjectEntity
(project:ProjectEntity {
    key:key//项目key
    name:name//项目名称
})
id	1 项目id
key:key//项目key
name:name//项目名称
name realworld  项目名称

CREATE INDEX idx_project_key IF NOT EXISTS FOR (m:ProjectEntity) ON (m.key) ;
2.ScanRecordEntity
(scanRecord:ScanRecordEntity){
    scanId:scanId//扫描id
}
CREATE INDEX idx_scan_record_scanId IF NOT EXISTS FOR (m:ScanRecordEntity) ON (m.scanId) ;
//扫描信息所属关系 1:n
(:ProjectEntity)-[:HAS_SCAN_RECORD]->(:ScanRecordEntity)

3.VCSEntity
(vcsEntity:VCSEntity{
    name:name,//仓库名称
    path:path,//仓库本地绝对路径
    vcsType:vcsType//仓库类型 git
})
VCSEntity
id	7aaf76a6-385a-480e-9377-335d798bb516
name	realworld-mdd
path	D:\code\realworld-mdd
vcsType	git
createdAt	1678551239850
updatedAt	1678551239850
//版本控制信息关系 1:1
(:ProjectEntity)-[:VCS_BY]->(:VCSEntity)

4.ModuleEntity
(module:ModuleEntity {
    name:name//模块名称
})
ModuleEntity
id	8a078585-5c06-43be-93b0-dd551244c64c
name root 模块名称
createdAt	1678551238729
updatedAt	1678551239249

//模块包含关系 1:n
(:ScanRecordEntity)-[:CONTAINS]->(:ModuleEntity)

5.FileEntity
(file:FileEntity{
    name:name,//文件名
    type:type,//文件类型 file/folder
    relativePath:relativePath//相对路径
})
FileEntity
id	d9cbe0a4-3f60-4704-a419-7489ee530fc8  文件id
name	user 
type	folder 文件类型 file/folder
relativePath	src-gen\main\java\cn\hylstudio\mdse\demo\realworld\model\response\user 相对路径
createdAt	1678551233924 
updatedAt	1678551239353
CREATE INDEX idx_file_id IF NOT EXISTS FOR (m:FileEntity) ON (m.id) ;
CREATE INDEX idx_file_relativePath IF NOT EXISTS FOR (m:FileEntity) ON (m.relativePath) ;
//符号连接关系 n:1
(:FileEntity)-[:SYMBOL_LINK]->(:FileEntity)
//项目根目录 1:1
(:ScanRecordEntity)-[:ROOT_AT]->(:FileEntity)
//模块根目录 1:n
(:ModuleEntity)-[:MODULE_ROOT{
    type: type//src testSrc resources testResources
}]->(:FileEntity)

6.PsiElementEntity
(:PsiElementEntity{//语法树
    className：className//psiElement实现类
    startOffset	//文本开始位置
    endOffset//文本结束位置
    originText//原始文本
})
PsiElementEntity
id	f30db139-4a90-4c10-b740-ac85514ab787
className	com.intellij.psi.impl.source.PsiImportListImpl
startOffset	59
endOffset	857
originText	'java code'
createdAt	1680274076903
updatedAt	1680274076903

6.1.类
com.intellij.psi.impl.source.PsiClassImpl
$.qualifiedName str //全限定类名 com.a.b.c.ClassA

6.2.注解
com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl
$.qualifiedName str //全限定类名 com.a.b.c.AnnotationA
$.attributes arr //属性数组
$.attributes[i] obj
$.attributes[i].name str //属性名
$.attributes[i].values arr //属性值
$.attributes[i].values[i] str //属性值

//文件包含语法树根节点 1:n
(:FileEntity)-[:HAS_PSI_ELEMENTS]->(:PsiElementEntity)
//语法树包含关系 1:n
(:PsiElementEntity)-[:CONTAINS]->(:PsiElementEntity)

7.AnnotationAttrEntity
(:AnnotationAttrEntity{//注解属性
    name
    values
})

CREATE INDEX idx_psi_element_id IF NOT EXISTS FOR (m:PsiElementEntity) ON (m.id) ;
CREATE INDEX idx_psi_className IF NOT EXISTS FOR (m:PsiElementEntity) ON (m.className) ;
CREATE INDEX idx_psi_qualifiedName IF NOT EXISTS FOR (m:PsiElementEntity) ON (m.qualifiedName) ;
//注解关系 1:n
(:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})-[:HAS_ANNOTATION]->(:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
(:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})-[:HAS_ANNOTATION]->(:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
//注解和注解属性的关系1:n
(:PsiElementEntity{className:"com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl"})-[:HAS_ATTR]->(:AnnotationAttrEntity)

8.ApiEndpointEntity
(:ApiEndpointEntity{
    scanId: scanId,
    method: httpMethod//RequestMethod.GET POST ....
    path: "/1/2/3" // /api/public/1/2/3
})

CREATE INDEX idx_api_endpoint_scanId IF NOT EXISTS FOR (m:ApiEndpointEntity) ON (m.scanId) ;
CREATE INDEX idx_api_endpoint_path IF NOT EXISTS FOR (m:ApiEndpointEntity) ON (m.path) ;
//api入口绑定关系 1:n
(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})
-[:HAS_ANNOTATION]->
(methodAnnotation:PsiElementEntity{
    className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',
    qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'
})
(method:PsiElementEntity)-[:HAS_ENDPOINT]->(:ApiEndPointEntity)


//查看IDEA日志
Get-Content  .\build\idea-sandbox\system\log\idea.log -Tail 100 -Wait

//列出所有项目
MATCH (project:ProjectEntity)
RETURN project.key

//列出指定项目下的扫描记录
MATCH (project:ProjectEntity)-[:HAS_SCAN_RECORD]->(scanRecord:ScanRecordEntity)
WHERE project.key = "realworld-mdd"
RETURN scanRecord.scanId, scanRecord.createdAt
ORDER BY scanRecord.createdAt DESC

//查看某次扫描记录
MATCH (project:ProjectEntity)
-[:HAS_SCAN_RECORD]->
(scanRecord:ScanRecordEntity)
-[*]-
(related)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db"
RETURN project, scanRecord, related

// 查找要连接ModuleRoot的FileEntity和FileEntity
MATCH (scanRecord:ScanRecordEntity)-[rr:CONTAINS]->(module:ModuleEntity)-[rrr:MODULE_ROOT]->(moduleRoot:FileEntity)
MATCH (scanRecord)-[:ROOT_AT]->(:FileEntity)-[:CONTAINS*1..]->(srcRoot:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db" AND
moduleRoot.relativePath = srcRoot.relativePath AND
moduleRoot <> srcRoot
RETURN scanRecord, rr, module, rrr, moduleRoot, srcRoot

// 连接ModuleRoot的FileEntity和FileEntity
MATCH (scanRecord:ScanRecordEntity)-[rr:CONTAINS]->(module:ModuleEntity)-[rrr:MODULE_ROOT]->(moduleRoot:FileEntity)
MATCH (scanRecord)-[:ROOT_AT]->(:FileEntity)-[:CONTAINS*1..]->(srcRoot:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db" AND
moduleRoot.relativePath = srcRoot.relativePath AND
moduleRoot <> srcRoot
MERGE (moduleRoot)-[rrrr:SYMBOL_LINK]->(srcRoot)
RETURN scanRecord, rr, module, rrr, moduleRoot, rrrr, srcRoot

// 断开连接ModuleRoot的FileEntity和FileEntity
MATCH (scanRecord:ScanRecordEntity)-[rr:CONTAINS]->(module:ModuleEntity)-[rrr:MODULE_ROOT]->(moduleRoot:FileEntity)
MATCH (scanRecord)-[:ROOT_AT]->(:FileEntity)-[:CONTAINS*1..]->(srcRoot:FileEntity)
MATCH (moduleRoot)-[link:SYMBOL_LINK]->(srcRoot:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db" AND
moduleRoot.relativePath = srcRoot.relativePath AND
moduleRoot <> srcRoot
DELETE link
RETURN scanRecord, rr, module, rrr, moduleRoot, srcRoot

// -[:TYPE*minHops..maxHops]->
//使用这个连接查询后面的文件树
MATCH (project:ProjectEntity)
-[:HAS_SCAN_RECORD]->(scanRecord:ScanRecordEntity)
-[:CONTAINS]->(module:ModuleEntity)
-[:MODULE_ROOT]->(moduleRoot:FileEntity)
-[:SYMBOL_LINK]->(srcRoot:FileEntity)
-[:CONTAINS*1..]->(files)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db" 
RETURN project, scanRecord, module, moduleRoot, srcRoot, files

//查找指定扫描记录里的单个文件
MATCH
(scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)
WHERE
scanRecord.scanId = "643e424c1b99436f82551420981669db"
AND
file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"
RETURN file

//查看指定文件的语法树
MATCH
(scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)
-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->(elements:PsiElementEntity)
WHERE
scanRecord.scanId = "643e424c1b99436f82551420981669db"
AND
file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"
RETURN file,elements
//获取当前文件所有的语法树节点类型
// RETURN  distinct(psiElements.className)
"com.intellij.psi.impl.source.tree.java.PsiPackageStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiKeywordImpl"
"com.intellij.psi.impl.source.PsiJavaCodeReferenceElementImpl"
"com.intellij.psi.impl.source.tree.java.PsiJavaTokenImpl"
"com.intellij.psi.impl.source.tree.java.PsiReferenceParameterListImpl"
"com.intellij.psi.impl.source.tree.java.PsiIdentifierImpl"
"com.intellij.psi.impl.source.tree.PsiWhiteSpaceImpl"
"com.intellij.psi.impl.source.PsiImportListImpl"
"com.intellij.psi.impl.source.PsiImportStatementImpl"
"com.intellij.psi.impl.source.PsiClassImpl"
"com.intellij.psi.impl.source.PsiMethodImpl"
"com.intellij.psi.impl.source.PsiModifierListImpl"
"com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl"
"com.intellij.psi.impl.source.tree.java.PsiAnnotationParamListImpl"
"com.intellij.psi.impl.source.tree.java.PsiNameValuePairImpl"
"com.intellij.psi.impl.source.tree.java.PsiReferenceExpressionImpl"
"com.intellij.psi.impl.source.tree.java.PsiLiteralExpressionImpl"
"com.intellij.psi.impl.source.PsiTypeElementImpl"
"com.intellij.psi.impl.source.PsiParameterListImpl"
"com.intellij.psi.impl.source.PsiParameterImpl"
"com.intellij.psi.impl.source.tree.java.PsiCodeBlockImpl"
"com.intellij.psi.impl.source.tree.java.PsiReturnStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiMethodCallExpressionImpl"
"com.intellij.psi.impl.source.tree.java.PsiExpressionListImpl"
"com.intellij.psi.impl.source.tree.java.PsiExpressionStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiDeclarationStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiLocalVariableImpl"
"com.intellij.psi.impl.source.tree.java.PsiIfStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiBlockStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiBinaryExpressionImpl"
"com.intellij.psi.impl.source.tree.java.PsiTypeParameterListImpl"
"com.intellij.psi.impl.source.PsiReferenceListImpl"
"com.intellij.psi.impl.source.tree.PsiCommentImpl"
"com.intellij.psi.impl.source.tree.java.PsiPrefixExpressionImpl"
"com.intellij.psi.impl.source.tree.java.PsiSwitchStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiSwitchLabelStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiCaseLabelElementListImpl"
"com.intellij.psi.impl.source.tree.java.PsiNewExpressionImpl"
"com.intellij.psi.impl.source.PsiDiamondTypeElementImpl"
"com.intellij.psi.impl.source.tree.java.PsiForeachStatementImpl"
"com.intellij.psi.impl.source.tree.java.PsiClassObjectAccessExpressionImpl"
"com.intellij.psi.impl.source.PsiFieldImpl"
"com.intellij.psi.impl.source.tree.java.PsiArrayInitializerMemberValueImpl"
"com.intellij.psi.impl.source.javadoc.PsiDocCommentImpl"
"com.intellij.psi.impl.source.javadoc.PsiDocTokenImpl"

//获取一个文件中的类、类的注解、类中的方法、方法注解，不使用HAS_ANNOTATION
MATCH (scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db"
AND
file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"
OPTIONAL MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->
//match class info
(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})
OPTIONAL MATCH (class)-[:CONTAINS]->
(classModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})
OPTIONAL MATCH (classModifierList)-[:CONTAINS]->
(classAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
OPTIONAL MATCH (classAnnotation)-[:HAS_ATTR]->
(classAnnotationAttr:AnnotationAttrEntity)
//match method info
OPTIONAL MATCH (class)-[:CONTAINS]->
(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})
OPTIONAL MATCH (method)-[:CONTAINS]->
(mehotdModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})
OPTIONAL MATCH (mehotdModifierList)-[:CONTAINS]->
(methodAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
OPTIONAL MATCH (methodAnnotation)-[:HAS_ATTR]->
(methodAnnotationAttr:AnnotationAttrEntity)
return 
class,classModifierList,classAnnotation,classAnnotationAttr,
method,mehotdModifierList,methodAnnotation,methodAnnotationAttr

//连接所有class和method上的注解
MATCH (scanRecord:ScanRecordEntity)-[:MODULE_ROOT|SYMBOL_LINK|CONTAINS*1..]->(file:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db"
AND
file.type = "file"
MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->
//match class info
(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})
MATCH (class)-[:CONTAINS]->
(classModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})
MATCH (classModifierList)-[:CONTAINS]->
(classAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
MATCH (classAnnotation)-[:HAS_ATTR]->
(classAnnotationAttr:AnnotationAttrEntity)
//match method info
MATCH (class)-[:CONTAINS]->
(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})
MATCH (method)-[:CONTAINS]->
(methodModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})
MATCH (methodModifierList)-[:CONTAINS]->
(methodAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
MATCH (methodAnnotation)-[:HAS_ATTR]->
(methodAnnotationAttr:AnnotationAttrEntity)
MERGE (class)-[:HAS_ANNOTATION]->(classAnnotation)
MERGE (method)-[:HAS_ANNOTATION]->(methodAnnotation)

//获取一个文件中的类、类的注解、类中的方法、方法注解，使用HAS_ANNOTATION
MATCH (scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db"
AND
file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"
OPTIONAL MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->
//match class info
(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})
OPTIONAL MATCH (class)-[:HAS_ANNOTATION]->
(classAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
OPTIONAL MATCH (classAnnotation)-[:HAS_ATTR]->
(classAnnotationAttr:AnnotationAttrEntity)
//match method info
OPTIONAL MATCH (class)-[:CONTAINS]->
(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})
OPTIONAL MATCH (method)-[:HAS_ANNOTATION]->
(methodAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})
OPTIONAL MATCH (methodAnnotation)-[:HAS_ATTR]->
(methodAnnotationAttr:AnnotationAttrEntity)
return 
class,classAnnotation,classAnnotationAttr,
method,methodAnnotation,methodAnnotationAttr

//单个文件，根据注解查询Api入口，并关联到方法上
MATCH (scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db"
AND
file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"
MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->
//match class info
(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})
OPTIONAL MATCH (class)-[:HAS_ANNOTATION]->
(classAnnotation:PsiElementEntity{
    className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',
    qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'
})
MATCH (classAnnotation)-[:HAS_ATTR]->
(classAnnotationAttr:AnnotationAttrEntity{name:"value"})
//match method info
MATCH (class)-[:CONTAINS]->
(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})
MATCH (method)-[:HAS_ANNOTATION]->
(methodAnnotation:PsiElementEntity{
    className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',
    qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'
})
MATCH (methodAnnotation)-[:HAS_ATTR]->
(apiPathsOnMethod:AnnotationAttrEntity{name:"value"})
MATCH (methodAnnotation)-[:HAS_ATTR]->
(apiMethods:AnnotationAttrEntity{name:"method"})
UNWIND classAnnotationAttr.values AS prefix
UNWIND apiMethods.values AS httpMethod
UNWIND apiPathsOnMethod.values AS path
MERGE (method)-[:HAS_ENDPOINT]->(apiEndpoint:ApiEndpointEntity{method: httpMethod, path: prefix + path})
RETURN method, apiEndpoint

//全量文件，根据注解生成Api入口，并关联到方法和扫描记录上
MATCH (scanRecord:ScanRecordEntity)-[:MODULE_ROOT|SYMBOL_LINK|CONTAINS*1..]->(file:FileEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db"
AND
file.type = "file"
MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->
//match class info
(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})
OPTIONAL MATCH (class)-[:HAS_ANNOTATION]->
(classAnnotation:PsiElementEntity{
    className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',
    qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'
})
MATCH (classAnnotation)-[:HAS_ATTR]->
(classAnnotationAttr:AnnotationAttrEntity{name:"value"})
//match method info
MATCH (class)-[:CONTAINS]->
(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})
MATCH (method)-[:HAS_ANNOTATION]->
(methodAnnotation:PsiElementEntity{
    className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',
    qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'
})
MATCH (methodAnnotation)-[:HAS_ATTR]->
(apiPathsOnMethod:AnnotationAttrEntity{name:"value"})
MATCH (methodAnnotation)-[:HAS_ATTR]->
(apiMethods:AnnotationAttrEntity{name:"method"})
UNWIND classAnnotationAttr.values AS prefix
UNWIND apiMethods.values AS httpMethod
UNWIND apiPathsOnMethod.values AS path
MERGE (apiEndpoint:ApiEndpointEntity {scanId: scanRecord.scanId, method: httpMethod, path: prefix + path})
MERGE (method)-[:HAS_ENDPOINT]->(apiEndpoint)
MERGE (scanRecord)-[:HAS_ENDPOINT]->(apiEndpoint)
RETURN method, apiEndpoint

//查找所有ApiEndpoint
MATCH (scanRecord:ScanRecordEntity)-[:HAS_ENDPOINT]->(apiEndpoint:ApiEndpointEntity)
WHERE 
scanRecord.scanId = "643e424c1b99436f82551420981669db"
RETURN apiEndpoint

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

1.ProjectEntity

(project:ProjectEntity {

key:key//项目key

name:name//项目名称

})

id 1 项目id

key:key//项目key

name:name//项目名称

name realworld 项目名称

CREATE INDEX idx_project_key IF NOT EXISTS FOR (m:ProjectEntity) ON (m.key) ;

2.ScanRecordEntity

(scanRecord:ScanRecordEntity){

scanId:scanId//扫描id

}

CREATE INDEX idx_scan_record_scanId IF NOT EXISTS FOR (m:ScanRecordEntity) ON (m.scanId) ;

//扫描信息所属关系 1:n

(:ProjectEntity)-[:HAS_SCAN_RECORD]->(:ScanRecordEntity)

3.VCSEntity

(vcsEntity:VCSEntity{

name:name,//仓库名称

path:path,//仓库本地绝对路径

vcsType:vcsType//仓库类型 git

})

VCSEntity

id 7aaf76a6-385a-480e-9377-335d798bb516

name realworld-mdd

path D:\code\realworld-mdd

vcsType git

createdAt 1678551239850

updatedAt 1678551239850

//版本控制信息关系 1:1

(:ProjectEntity)-[:VCS_BY]->(:VCSEntity)

4.ModuleEntity

(module:ModuleEntity {

name:name//模块名称

})

ModuleEntity

id 8a078585-5c06-43be-93b0-dd551244c64c

name root 模块名称

createdAt 1678551238729

updatedAt 1678551239249

//模块包含关系 1:n

(:ScanRecordEntity)-[:CONTAINS]->(:ModuleEntity)

5.FileEntity

(file:FileEntity{

name:name,//文件名

type:type,//文件类型 file/folder

relativePath:relativePath//相对路径

})

FileEntity

id d9cbe0a4-3f60-4704-a419-7489ee530fc8 文件id

name user

type folder 文件类型 file/folder

relativePath src-gen\main\java\cn\hylstudio\mdse\demo\realworld\model\response\user 相对路径

createdAt 1678551233924

updatedAt 1678551239353

CREATE INDEX idx_file_id IF NOT EXISTS FOR (m:FileEntity) ON (m.id) ;

CREATE INDEX idx_file_relativePath IF NOT EXISTS FOR (m:FileEntity) ON (m.relativePath) ;

//符号连接关系 n:1

(:FileEntity)-[:SYMBOL_LINK]->(:FileEntity)

//项目根目录 1:1

(:ScanRecordEntity)-[:ROOT_AT]->(:FileEntity)

//模块根目录 1:n

(:ModuleEntity)-[:MODULE_ROOT{

type: type//src testSrc resources testResources

}]->(:FileEntity)

6.PsiElementEntity

(:PsiElementEntity{//语法树

className：className//psiElement实现类

startOffset //文本开始位置

endOffset//文本结束位置

originText//原始文本

})

PsiElementEntity

id f30db139-4a90-4c10-b740-ac85514ab787

className com.intellij.psi.impl.source.PsiImportListImpl

startOffset 59

endOffset 857

originText 'java code'

createdAt 1680274076903

updatedAt 1680274076903

6.1.类

com.intellij.psi.impl.source.PsiClassImpl

$.qualifiedName str //全限定类名 com.a.b.c.ClassA

6.2.注解

com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl

$.qualifiedName str //全限定类名 com.a.b.c.AnnotationA

$.attributes arr //属性数组

$.attributes[i] obj

$.attributes[i].name str //属性名

$.attributes[i].values arr //属性值

$.attributes[i].values[i] str //属性值

//文件包含语法树根节点 1:n

(:FileEntity)-[:HAS_PSI_ELEMENTS]->(:PsiElementEntity)

//语法树包含关系 1:n

(:PsiElementEntity)-[:CONTAINS]->(:PsiElementEntity)

7.AnnotationAttrEntity

(:AnnotationAttrEntity{//注解属性

name

values

})

CREATE INDEX idx_psi_element_id IF NOT EXISTS FOR (m:PsiElementEntity) ON (m.id) ;

CREATE INDEX idx_psi_className IF NOT EXISTS FOR (m:PsiElementEntity) ON (m.className) ;

CREATE INDEX idx_psi_qualifiedName IF NOT EXISTS FOR (m:PsiElementEntity) ON (m.qualifiedName) ;

//注解关系 1:n

(:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})-[:HAS_ANNOTATION]->(:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

(:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})-[:HAS_ANNOTATION]->(:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

//注解和注解属性的关系1:n

(:PsiElementEntity{className:"com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl"})-[:HAS_ATTR]->(:AnnotationAttrEntity)

8.ApiEndpointEntity

(:ApiEndpointEntity{

scanId: scanId,

method: httpMethod//RequestMethod.GET POST ....

path: "/1/2/3" // /api/public/1/2/3

})

CREATE INDEX idx_api_endpoint_scanId IF NOT EXISTS FOR (m:ApiEndpointEntity) ON (m.scanId) ;

CREATE INDEX idx_api_endpoint_path IF NOT EXISTS FOR (m:ApiEndpointEntity) ON (m.path) ;

//api入口绑定关系 1:n

(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})

-[:HAS_ANNOTATION]->

(methodAnnotation:PsiElementEntity{

className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',

qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'

})

(method:PsiElementEntity)-[:HAS_ENDPOINT]->(:ApiEndPointEntity)

//查看IDEA日志

Get-Content .\build\idea-sandbox\system\log\idea.log -Tail 100 -Wait

//列出所有项目

MATCH (project:ProjectEntity)

RETURN project.key

//列出指定项目下的扫描记录

MATCH (project:ProjectEntity)-[:HAS_SCAN_RECORD]->(scanRecord:ScanRecordEntity)

WHERE project.key = "realworld-mdd"

RETURN scanRecord.scanId, scanRecord.createdAt

ORDER BY scanRecord.createdAt DESC

//查看某次扫描记录

MATCH (project:ProjectEntity)

-[:HAS_SCAN_RECORD]->

(scanRecord:ScanRecordEntity)

-[*]-

(related)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

RETURN project, scanRecord, related

// 查找要连接ModuleRoot的FileEntity和FileEntity

MATCH (scanRecord:ScanRecordEntity)-[rr:CONTAINS]->(module:ModuleEntity)-[rrr:MODULE_ROOT]->(moduleRoot:FileEntity)

MATCH (scanRecord)-[:ROOT_AT]->(:FileEntity)-[:CONTAINS*1..]->(srcRoot:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db" AND

moduleRoot.relativePath = srcRoot.relativePath AND

moduleRoot <> srcRoot

RETURN scanRecord, rr, module, rrr, moduleRoot, srcRoot

// 连接ModuleRoot的FileEntity和FileEntity

MATCH (scanRecord:ScanRecordEntity)-[rr:CONTAINS]->(module:ModuleEntity)-[rrr:MODULE_ROOT]->(moduleRoot:FileEntity)

MATCH (scanRecord)-[:ROOT_AT]->(:FileEntity)-[:CONTAINS*1..]->(srcRoot:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db" AND

moduleRoot.relativePath = srcRoot.relativePath AND

moduleRoot <> srcRoot

MERGE (moduleRoot)-[rrrr:SYMBOL_LINK]->(srcRoot)

RETURN scanRecord, rr, module, rrr, moduleRoot, rrrr, srcRoot

// 断开连接ModuleRoot的FileEntity和FileEntity

MATCH (scanRecord:ScanRecordEntity)-[rr:CONTAINS]->(module:ModuleEntity)-[rrr:MODULE_ROOT]->(moduleRoot:FileEntity)

MATCH (scanRecord)-[:ROOT_AT]->(:FileEntity)-[:CONTAINS*1..]->(srcRoot:FileEntity)

MATCH (moduleRoot)-[link:SYMBOL_LINK]->(srcRoot:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db" AND

moduleRoot.relativePath = srcRoot.relativePath AND

moduleRoot <> srcRoot

DELETE link

RETURN scanRecord, rr, module, rrr, moduleRoot, srcRoot

// -[:TYPE*minHops..maxHops]->

//使用这个连接查询后面的文件树

MATCH (project:ProjectEntity)

-[:HAS_SCAN_RECORD]->(scanRecord:ScanRecordEntity)

-[:CONTAINS]->(module:ModuleEntity)

-[:MODULE_ROOT]->(moduleRoot:FileEntity)

-[:SYMBOL_LINK]->(srcRoot:FileEntity)

-[:CONTAINS*1..]->(files)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

RETURN project, scanRecord, module, moduleRoot, srcRoot, files

//查找指定扫描记录里的单个文件

MATCH

(scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

AND

file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"

RETURN file

//查看指定文件的语法树

MATCH

(scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)

-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->(elements:PsiElementEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

AND

file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"

RETURN file,elements

//获取当前文件所有的语法树节点类型

// RETURN distinct(psiElements.className)

"com.intellij.psi.impl.source.tree.java.PsiPackageStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiKeywordImpl"

"com.intellij.psi.impl.source.PsiJavaCodeReferenceElementImpl"

"com.intellij.psi.impl.source.tree.java.PsiJavaTokenImpl"

"com.intellij.psi.impl.source.tree.java.PsiReferenceParameterListImpl"

"com.intellij.psi.impl.source.tree.java.PsiIdentifierImpl"

"com.intellij.psi.impl.source.tree.PsiWhiteSpaceImpl"

"com.intellij.psi.impl.source.PsiImportListImpl"

"com.intellij.psi.impl.source.PsiImportStatementImpl"

"com.intellij.psi.impl.source.PsiClassImpl"

"com.intellij.psi.impl.source.PsiMethodImpl"

"com.intellij.psi.impl.source.PsiModifierListImpl"

"com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl"

"com.intellij.psi.impl.source.tree.java.PsiAnnotationParamListImpl"

"com.intellij.psi.impl.source.tree.java.PsiNameValuePairImpl"

"com.intellij.psi.impl.source.tree.java.PsiReferenceExpressionImpl"

"com.intellij.psi.impl.source.tree.java.PsiLiteralExpressionImpl"

"com.intellij.psi.impl.source.PsiTypeElementImpl"

"com.intellij.psi.impl.source.PsiParameterListImpl"

"com.intellij.psi.impl.source.PsiParameterImpl"

"com.intellij.psi.impl.source.tree.java.PsiCodeBlockImpl"

"com.intellij.psi.impl.source.tree.java.PsiReturnStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiMethodCallExpressionImpl"

"com.intellij.psi.impl.source.tree.java.PsiExpressionListImpl"

"com.intellij.psi.impl.source.tree.java.PsiExpressionStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiDeclarationStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiLocalVariableImpl"

"com.intellij.psi.impl.source.tree.java.PsiIfStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiBlockStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiBinaryExpressionImpl"

"com.intellij.psi.impl.source.tree.java.PsiTypeParameterListImpl"

"com.intellij.psi.impl.source.PsiReferenceListImpl"

"com.intellij.psi.impl.source.tree.PsiCommentImpl"

"com.intellij.psi.impl.source.tree.java.PsiPrefixExpressionImpl"

"com.intellij.psi.impl.source.tree.java.PsiSwitchStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiSwitchLabelStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiCaseLabelElementListImpl"

"com.intellij.psi.impl.source.tree.java.PsiNewExpressionImpl"

"com.intellij.psi.impl.source.PsiDiamondTypeElementImpl"

"com.intellij.psi.impl.source.tree.java.PsiForeachStatementImpl"

"com.intellij.psi.impl.source.tree.java.PsiClassObjectAccessExpressionImpl"

"com.intellij.psi.impl.source.PsiFieldImpl"

"com.intellij.psi.impl.source.tree.java.PsiArrayInitializerMemberValueImpl"

"com.intellij.psi.impl.source.javadoc.PsiDocCommentImpl"

"com.intellij.psi.impl.source.javadoc.PsiDocTokenImpl"

//获取一个文件中的类、类的注解、类中的方法、方法注解，不使用HAS_ANNOTATION

MATCH (scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

AND

file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"

OPTIONAL MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->

//match class info

(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})

OPTIONAL MATCH (class)-[:CONTAINS]->

(classModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})

OPTIONAL MATCH (classModifierList)-[:CONTAINS]->

(classAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

OPTIONAL MATCH (classAnnotation)-[:HAS_ATTR]->

(classAnnotationAttr:AnnotationAttrEntity)

//match method info

OPTIONAL MATCH (class)-[:CONTAINS]->

(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})

OPTIONAL MATCH (method)-[:CONTAINS]->

(mehotdModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})

OPTIONAL MATCH (mehotdModifierList)-[:CONTAINS]->

(methodAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

OPTIONAL MATCH (methodAnnotation)-[:HAS_ATTR]->

(methodAnnotationAttr:AnnotationAttrEntity)

return

class,classModifierList,classAnnotation,classAnnotationAttr,

method,mehotdModifierList,methodAnnotation,methodAnnotationAttr

//连接所有class和method上的注解

MATCH (scanRecord:ScanRecordEntity)-[:MODULE_ROOT|SYMBOL_LINK|CONTAINS*1..]->(file:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

AND

file.type = "file"

MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->

//match class info

(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})

MATCH (class)-[:CONTAINS]->

(classModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})

MATCH (classModifierList)-[:CONTAINS]->

(classAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

MATCH (classAnnotation)-[:HAS_ATTR]->

(classAnnotationAttr:AnnotationAttrEntity)

//match method info

MATCH (class)-[:CONTAINS]->

(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})

MATCH (method)-[:CONTAINS]->

(methodModifierList:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiModifierListImpl'})

MATCH (methodModifierList)-[:CONTAINS]->

(methodAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

MATCH (methodAnnotation)-[:HAS_ATTR]->

(methodAnnotationAttr:AnnotationAttrEntity)

MERGE (class)-[:HAS_ANNOTATION]->(classAnnotation)

MERGE (method)-[:HAS_ANNOTATION]->(methodAnnotation)

//获取一个文件中的类、类的注解、类中的方法、方法注解，使用HAS_ANNOTATION

MATCH (scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

AND

file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"

OPTIONAL MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->

//match class info

(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})

OPTIONAL MATCH (class)-[:HAS_ANNOTATION]->

(classAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

OPTIONAL MATCH (classAnnotation)-[:HAS_ATTR]->

(classAnnotationAttr:AnnotationAttrEntity)

//match method info

OPTIONAL MATCH (class)-[:CONTAINS]->

(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})

OPTIONAL MATCH (method)-[:HAS_ANNOTATION]->

(methodAnnotation:PsiElementEntity{className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl'})

OPTIONAL MATCH (methodAnnotation)-[:HAS_ATTR]->

(methodAnnotationAttr:AnnotationAttrEntity)

return

class,classAnnotation,classAnnotationAttr,

method,methodAnnotation,methodAnnotationAttr

//单个文件，根据注解查询Api入口，并关联到方法上

MATCH (scanRecord:ScanRecordEntity)-[:ROOT_AT|CONTAINS*1..]->(file:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

AND

file.relativePath = "src\\main\\java\\cn\\hylstudio\\mdse\\demo\\realworld\\controller\\user\\UserController.java"

MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->

//match class info

(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})

OPTIONAL MATCH (class)-[:HAS_ANNOTATION]->

(classAnnotation:PsiElementEntity{

className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',

qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'

})

MATCH (classAnnotation)-[:HAS_ATTR]->

(classAnnotationAttr:AnnotationAttrEntity{name:"value"})

//match method info

MATCH (class)-[:CONTAINS]->

(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})

MATCH (method)-[:HAS_ANNOTATION]->

(methodAnnotation:PsiElementEntity{

className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',

qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'

})

MATCH (methodAnnotation)-[:HAS_ATTR]->

(apiPathsOnMethod:AnnotationAttrEntity{name:"value"})

MATCH (methodAnnotation)-[:HAS_ATTR]->

(apiMethods:AnnotationAttrEntity{name:"method"})

UNWIND classAnnotationAttr.values AS prefix

UNWIND apiMethods.values AS httpMethod

UNWIND apiPathsOnMethod.values AS path

MERGE (method)-[:HAS_ENDPOINT]->(apiEndpoint:ApiEndpointEntity{method: httpMethod, path: prefix + path})

RETURN method, apiEndpoint

//全量文件，根据注解生成Api入口，并关联到方法和扫描记录上

MATCH (scanRecord:ScanRecordEntity)-[:MODULE_ROOT|SYMBOL_LINK|CONTAINS*1..]->(file:FileEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

AND

file.type = "file"

MATCH (file)-[:HAS_PSI_ELEMENTS|CONTAINS*1..]->

//match class info

(class:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiClassImpl'})

OPTIONAL MATCH (class)-[:HAS_ANNOTATION]->

(classAnnotation:PsiElementEntity{

className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',

qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'

})

MATCH (classAnnotation)-[:HAS_ATTR]->

(classAnnotationAttr:AnnotationAttrEntity{name:"value"})

//match method info

MATCH (class)-[:CONTAINS]->

(method:PsiElementEntity{className:'com.intellij.psi.impl.source.PsiMethodImpl'})

MATCH (method)-[:HAS_ANNOTATION]->

(methodAnnotation:PsiElementEntity{

className:'com.intellij.psi.impl.source.tree.java.PsiAnnotationImpl',

qualifiedName: 'org.springframework.web.bind.annotation.RequestMapping'

})

MATCH (methodAnnotation)-[:HAS_ATTR]->

(apiPathsOnMethod:AnnotationAttrEntity{name:"value"})

MATCH (methodAnnotation)-[:HAS_ATTR]->

(apiMethods:AnnotationAttrEntity{name:"method"})

UNWIND classAnnotationAttr.values AS prefix

UNWIND apiMethods.values AS httpMethod

UNWIND apiPathsOnMethod.values AS path

MERGE (apiEndpoint:ApiEndpointEntity {scanId: scanRecord.scanId, method: httpMethod, path: prefix + path})

MERGE (method)-[:HAS_ENDPOINT]->(apiEndpoint)

MERGE (scanRecord)-[:HAS_ENDPOINT]->(apiEndpoint)

RETURN method, apiEndpoint

//查找所有ApiEndpoint

MATCH (scanRecord:ScanRecordEntity)-[:HAS_ENDPOINT]->(apiEndpoint:ApiEndpointEntity)

WHERE

scanRecord.scanId = "643e424c1b99436f82551420981669db"

RETURN apiEndpoint

参考

https://arxiv.org/pdf/2302.04761.pdf

https://arxiv.org/pdf/2211.10435.pdf

20230204尝试搭建k8s青春版

视频版

https://www.bilibili.com/video/BV1qj411T7d5/

背景

之前一直想搭建k8s，但按多年前的印象这玩意很复杂。今天先尝试个青春版k3s

安装准备

按文档说是离线在线都行，离线的需要个私有的镜像中心，但镜像中心最简便的搭建方式是跑个容器。。。死循环了啊，所以先试在线的。

家里的资源不多，尝试做个乞丐版，2~3节点就行了，先弄单机后面再加worker

环境要求

硬件按文档建议是最少1c1g，我就给2c2g吧，机器数量是2

系统要求Ubuntu是22.04

可以使用 --with-node-id 给每个节点后面增加一个随机的后缀。或者使用命令行参数 --node-name 或环境变量 $K3S_NODE_NAME 给集群的每个节点传递唯一的名字

网络每个节点需要6443 可以被访问

Flannel VXLAN 节点间通过 UDP 8472 通信

Flannel Wireguard backend 节点间通过 UDP 51820(ipv4) 51821(ipv6)

安装过程


k3s1@k3s1:~$ curl -sfL https://get.k3s.io | sh -
[sudo] password for k3s1:
[INFO]  Finding release for channel stable
[INFO]  Using v1.25.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

k3s1@k3s1:~$ curl -sfL https://get.k3s.io | sh -

[sudo] password for k3s1:

[INFO] Finding release for channel stable

[INFO] Using v1.25.6+k3s1 as release

[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt

[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s

[INFO] Verifying binary download

[INFO] Installing k3s to /usr/local/bin/k3s

[INFO] Skipping installation of SELinux RPM

[INFO] Creating /usr/local/bin/kubectl symlink to k3s

[INFO] Creating /usr/local/bin/crictl symlink to k3s

[INFO] Creating /usr/local/bin/ctr symlink to k3s

[INFO] Creating killall script /usr/local/bin/k3s-killall.sh

[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh

[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env

[INFO] systemd: Creating service file /etc/systemd/system/k3s.service

[INFO] systemd: Enabling k3s unit

Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

主节点安装过程如上，疑似成功。。。

查看进程如下，可以看到/run/k3s/containerd/containerd.sock说明containerd已经在运行了


k3s1@k3s1:~$ ps aux|grep k3s
k3s1        1757  0.0  0.4  19152  9796 ?        Ss   11:25   0:00 /lib/systemd/systemd --user
k3s1        1759  0.0  0.2 105292  4420 ?        S    11:25   0:00 (sd-pam)
k3s1        1769  0.0  0.2   8264  5192 tty1     S+   11:25   0:00 -bash
root        2406  0.0  0.4  13788  9016 ?        Ss   11:29   0:00 sshd: k3s1 [priv]
k3s1        2478  0.0  0.3  13924  6000 ?        R    11:29   0:00 sshd: k3s1@pts/0
k3s1        2479  0.0  0.2   8276  5288 pts/0    Ss   11:29   0:00 -bash
root        2727 33.2 22.3 1194116 447444 ?      Ssl  11:31   1:01 /usr/local/bin/k3s server
root        2772 11.0  3.7 771044 75764 ?        Sl   11:31   0:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
root        3467  0.1  0.5 720748 10652 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9b07e1b9470f0e657d9a00667c44bb955e3d5c7f049036adfd38d086d73026e1 -address /run/k3s/containerd/containerd.sock
root        3511  0.1  0.5 721004 10740 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 65fba4a19627e12286bbdb2056f27f1c063c060255510b9b997e291a44e1c736 -address /run/k3s/containerd/containerd.sock
root        3521  0.0  0.4 720748  9852 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3193ac9aa5db3faec494a7fa431c2bbed4c9a124c67042ef956ba1f1b08783b0 -address /run/k3s/containerd/containerd.sock
root        3538  0.0  0.5 720748 10972 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id bed5712d5e71c22a2b4863c61df5402ed3cd2e468b89b3699f3b2db48ea2f522 -address /run/k3s/containerd/containerd.sock
root        3563  0.0  0.5 720492 10208 ?        Sl   11:32   0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id b76d14ccfaa1a5743f4fa77aef74c210b89badd130c75244e8561dffa643cdba -address /run/k3s/containerd/containerd.sock
k3s1        3944  6.1  2.3 751748 47708 ?        Ssl  11:32   0:04 /metrics-server --cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s
k3s1        4181  0.0  0.1   9128  3480 pts/0    R+   11:34   0:00 ps aux
k3s1        4182  0.0  0.0   6432   720 pts/0    S+   11:34   0:00 grep --color=auto k3s

k3s1@k3s1:~$ ps aux|grep k3s

k3s1 1757 0.0 0.4 19152 9796 ? Ss 11:25 0:00 /lib/systemd/systemd --user

k3s1 1759 0.0 0.2 105292 4420 ? S 11:25 0:00 (sd-pam)

k3s1 1769 0.0 0.2 8264 5192 tty1 S+ 11:25 0:00 -bash

root 2406 0.0 0.4 13788 9016 ? Ss 11:29 0:00 sshd: k3s1 [priv]

k3s1 2478 0.0 0.3 13924 6000 ? R 11:29 0:00 sshd: k3s1@pts/0

k3s1 2479 0.0 0.2 8276 5288 pts/0 Ss 11:29 0:00 -bash

root 2727 33.2 22.3 1194116 447444 ? Ssl 11:31 1:01 /usr/local/bin/k3s server

root 2772 11.0 3.7 771044 75764 ? Sl 11:31 0:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd

root 3467 0.1 0.5 720748 10652 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9b07e1b9470f0e657d9a00667c44bb955e3d5c7f049036adfd38d086d73026e1 -address /run/k3s/containerd/containerd.sock

root 3511 0.1 0.5 721004 10740 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 65fba4a19627e12286bbdb2056f27f1c063c060255510b9b997e291a44e1c736 -address /run/k3s/containerd/containerd.sock

root 3521 0.0 0.4 720748 9852 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3193ac9aa5db3faec494a7fa431c2bbed4c9a124c67042ef956ba1f1b08783b0 -address /run/k3s/containerd/containerd.sock

root 3538 0.0 0.5 720748 10972 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id bed5712d5e71c22a2b4863c61df5402ed3cd2e468b89b3699f3b2db48ea2f522 -address /run/k3s/containerd/containerd.sock

root 3563 0.0 0.5 720492 10208 ? Sl 11:32 0:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id b76d14ccfaa1a5743f4fa77aef74c210b89badd130c75244e8561dffa643cdba -address /run/k3s/containerd/containerd.sock

k3s1 3944 6.1 2.3 751748 47708 ? Ssl 11:32 0:04 /metrics-server --cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s

k3s1 4181 0.0 0.1 9128 3480 pts/0 R+ 11:34 0:00 ps aux

k3s1 4182 0.0 0.0 6432 720 pts/0 S+ 11:34 0:00 grep --color=auto k3s

metrics-server不知道是干什么的，后面再翻翻文档。

网络监听情况如下，可以看到大部分都是本地监听，只有10250 6443是有可能对外开放的

按文档描述是个TCP的端口监听，看起来和metrics server进程有关，好像是kubelet在节点之间通讯使用的。官方原文：If you wish to utilize the metrics server, all nodes must be accessible to each other on port 10250.

6443是TCP的监听，流量方向是从Agent向Server的，应该是worker节点向master注册数据用的。官方描述为 supervisor and Kubernetes API Server


k3s1@k3s1:~$ netstat -anp|grep LISTEN
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:10010         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10258         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::10250                :::*                    LISTEN      -
tcp6       0      0 :::6443                 :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

k3s1@k3s1:~$ netstat -anp|grep LISTEN

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp 0 0 127.0.0.1:10010 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:6444 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10256 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10258 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN -

tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -

tcp6 0 0 :::10250 :::* LISTEN -

tcp6 0 0 :::6443 :::* LISTEN -

tcp6 0 0 :::22 :::* LISTEN -

按文档所说配置文件内容如下


k3s1@k3s1:~$ sudo cat /etc/rancher/k3s/k3s.yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3SGhjTk1qTXdNakEwTVRFek1URTVXaGNOTXpNd01qQXhNVEV6TVRFNQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTS0lXTmNvbWZleEhPcndCVjUwdWx5ZjkwQlo2a0sxOStGUW9GN0pNUloKYi9UdndyZW9vNXhwWHNNekpEYlVUWkFzUXdlOXBDK3BoTjRaQ2ZFNVloRlVvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVURPelE3YkFldXh2a3JZN1hnSDI1CkNDOGpRcFF3Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnYW5rMlprZncrSHZCdHBvNnBuYmJiRDRuSExHSzlRZ3YKZFRLZTY3dm8zWVVDSUNtK0pJdisxSzBBcUpmUXA5L1JkdDRoMkF6Slc1WW5CenVIYzdLckhiRHgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRlZ0F3SUJBZ0lJVmVMRW9qM1NqUUV3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamMxTlRFd01qYzVNQjRYRFRJek1ESXdOREV4TXpFeE9Wb1hEVEkwTURJdwpOREV4TXpFeE9Wb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJBSld6ZVpTK0h6bHZCMkgKMkVJUUh5bEZQL3pDMmpZTktoZGtLTFF0NjhtTENWNnpKRWhxVDhYby9EUXdoc20zL2tmQWp1QlJ2R0tRNmsvRgpONDRIU0oyalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUUdCNlk4LzB5MjhSbmJnemdsejRxcXNURE5QekFLQmdncWhrak9QUVFEQWdOSkFEQkcKQWlFQTcybUw5b1dKeDM0SzZOZlVsWGp2dEQ5WXRJQXZuMWFuUUJkeHlXSUhOYThDSVFEUFllK3RZemtFR0RNbAppcnlkVTJpTHM0dWdVdXpwQThqM1Q2bEd4MU1nYUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZURDQ0FSMmdBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwClpXNTBMV05oUURFMk56VTFNVEF5Tnprd0hoY05Nak13TWpBME1URXpNVEU1V2hjTk16TXdNakF4TVRFek1URTUKV2pBak1TRXdId1lEVlFRRERCaHJNM010WTJ4cFpXNTBMV05oUURFMk56VTFNVEF5Tnprd1dUQVRCZ2NxaGtqTwpQUUlCQmdncWhrak9QUU1CQndOQ0FBUjB1cWV6cUh6S2owRUt2dFdNZ1YrQ09mblk0aGU3UFVLdnZlMWQ4d0VJCjFhb2laYjdzM05lK3ZvMzBQRkY5ZzdVZksrdGVJVjYxem80UytqVm9YU0k0bzBJd1FEQU9CZ05WSFE4QkFmOEUKQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVCZ2VtUFA5TXR2RVoyNE00SmMrSwpxckV3elQ4d0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFNVUlXUEExeXZYdEJsUUpFRER2aUtyM1hYbVJWb2hyClZGK3YzK0h4VXVGSUFpRUF5cmFhZlRiRG4vaTBwUmNHWnVEOHIzaTZQWHlJbUo0M1FpQnpLbHVjcnBFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUJzaUxpNU1NaG5wbUpXclJtSkxETUZobDVBSVcxTEQrS0tPRzBFRTlzc2RvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFQWxiTjVsTDRmT1c4SFlmWVFoQWZLVVUvL01MYU5nMHFGMlFvdEMzcnlZc0pYck1rU0dwUAp4ZWo4TkRDR3liZitSOENPNEZHOFlwRHFUOFUzamdkSW5RPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

k3s1@k3s1:~$ sudo cat /etc/rancher/k3s/k3s.yaml

apiVersion: v1

clusters:

- cluster:

certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3SGhjTk1qTXdNakEwTVRFek1URTVXaGNOTXpNd01qQXhNVEV6TVRFNQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTnpVMU1UQXlOemt3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTS0lXTmNvbWZleEhPcndCVjUwdWx5ZjkwQlo2a0sxOStGUW9GN0pNUloKYi9UdndyZW9vNXhwWHNNekpEYlVUWkFzUXdlOXBDK3BoTjRaQ2ZFNVloRlVvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVURPelE3YkFldXh2a3JZN1hnSDI1CkNDOGpRcFF3Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnYW5rMlprZncrSHZCdHBvNnBuYmJiRDRuSExHSzlRZ3YKZFRLZTY3dm8zWVVDSUNtK0pJdisxSzBBcUpmUXA5L1JkdDRoMkF6Slc1WW5CenVIYzdLckhiRHgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

server: https://127.0.0.1:6443

contexts:

- context:

cluster: default

user: default

current-context: default

kind: Config

preferences: {}

users:

- name: default

user:

client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRlZ0F3SUJBZ0lJVmVMRW9qM1NqUUV3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamMxTlRFd01qYzVNQjRYRFRJek1ESXdOREV4TXpFeE9Wb1hEVEkwTURJdwpOREV4TXpFeE9Wb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJBSld6ZVpTK0h6bHZCMkgKMkVJUUh5bEZQL3pDMmpZTktoZGtLTFF0NjhtTENWNnpKRWhxVDhYby9EUXdoc20zL2tmQWp1QlJ2R0tRNmsvRgpONDRIU0oyalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUUdCNlk4LzB5MjhSbmJnemdsejRxcXNURE5QekFLQmdncWhrak9QUVFEQWdOSkFEQkcKQWlFQTcybUw5b1dKeDM0SzZOZlVsWGp2dEQ5WXRJQXZuMWFuUUJkeHlXSUhOYThDSVFEUFllK3RZemtFR0RNbAppcnlkVTJpTHM0dWdVdXpwQThqM1Q2bEd4MU1nYUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZURDQ0FSMmdBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwClpXNTBMV05oUURFMk56VTFNVEF5Tnprd0hoY05Nak13TWpBME1URXpNVEU1V2hjTk16TXdNakF4TVRFek1URTUKV2pBak1TRXdId1lEVlFRRERCaHJNM010WTJ4cFpXNTBMV05oUURFMk56VTFNVEF5Tnprd1dUQVRCZ2NxaGtqTwpQUUlCQmdncWhrak9QUU1CQndOQ0FBUjB1cWV6cUh6S2owRUt2dFdNZ1YrQ09mblk0aGU3UFVLdnZlMWQ4d0VJCjFhb2laYjdzM05lK3ZvMzBQRkY5ZzdVZksrdGVJVjYxem80UytqVm9YU0k0bzBJd1FEQU9CZ05WSFE4QkFmOEUKQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVCZ2VtUFA5TXR2RVoyNE00SmMrSwpxckV3elQ4d0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFNVUlXUEExeXZYdEJsUUpFRER2aUtyM1hYbVJWb2hyClZGK3YzK0h4VXVGSUFpRUF5cmFhZlRiRG4vaTBwUmNHWnVEOHIzaTZQWHlJbUo0M1FpQnpLbHVjcnBFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUJzaUxpNU1NaG5wbUpXclJtSkxETUZobDVBSVcxTEQrS0tPRzBFRTlzc2RvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFQWxiTjVsTDRmT1c4SFlmWVFoQWZLVVUvL01MYU5nMHFGMlFvdEMzcnlZc0pYck1rU0dwUAp4ZWo4TkRDR3liZitSOENPNEZHOFlwRHFUOFUzamdkSW5RPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

按文档描述会生成这几个相关命令kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh。具体位置如下，均在/usr/local/bin下，所以可以从任意目录执行。

说实话k3s-killall.sh和k3s-uninstall.sh感觉并不适合放进这个位置，不知道是不是官方怕你找不到故意也放bin里了，这俩实际生产场景下完全没有全局任意目录运行的必要。万一误执行了就GG


k3s1@k3s1:~$ which kubectl
/usr/local/bin/kubectl
k3s1@k3s1:~$ which crictl
/usr/local/bin/crictl
k3s1@k3s1:~$ which k3s-killall.sh
/usr/local/bin/k3s-killall.sh
k3s1@k3s1:~$ which k3s-uninstall.sh
/usr/local/bin/k3s-uninstall.sh

k3s1@k3s1:~$ which kubectl

/usr/local/bin/kubectl

k3s1@k3s1:~$ which crictl

/usr/local/bin/crictl

k3s1@k3s1:~$ which k3s-killall.sh

/usr/local/bin/k3s-killall.sh

k3s1@k3s1:~$ which k3s-uninstall.sh

/usr/local/bin/k3s-uninstall.sh

尝试获取当前集群的节点，虽然知道只有一个，但需要看看命令是不是能正常执行，别和之前有的服务一样看着没事但实际没正常运行。。。。


k3s1@k3s1:~$ sudo kubectl get nodes
NAME   STATUS   ROLES                  AGE   VERSION
k3s1   Ready    control-plane,master   14m   v1.25.6+k3s1

k3s1@k3s1:~$ sudo kubectl get nodes

NAME STATUS ROLES AGE VERSION

k3s1 Ready control-plane,master 14m v1.25.6+k3s1

这里有个需要关注的东西就是ROLES都有啥，需要看文档了。目前的control-plane顾名思义是控制面, master是主节点。不知道k8s的角色分配是不是和CDH一样能灵活转移。。。

查看TOKEN，准备安装第一个worker节点，按文档描述在这里/var/lib/rancher/k3s/server/node-token

注意，我在一个隔离环境测试，所以/var/lib/rancher/k3s/server/node-token和/etc/rancher/k3s/k3s.yaml的内容暴露了无所谓，待会会直接删除虚机，生产的千万不要轻易暴露。


sudo cat /var/lib/rancher/k3s/server/node-token
K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9

sudo cat /var/lib/rancher/k3s/server/node-token

K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9

按要求拼接worker的安装命令如下，到第二个机器上执行


curl -sfL https://get.k3s.io | K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -

0 1	curl -sfL https://get.k3s.io \| K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -

执行到systemd卡住了（后面已确认只是单纯的慢）


k3s2@k3s2:~$ curl -sfL https://get.k3s.io | K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -
[sudo] password for k3s2:
[INFO]  Finding release for channel stable
[INFO]  Using v1.25.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

k3s2@k3s2:~$ curl -sfL https://get.k3s.io | K3S_URL=https://k3s1.k3s.local:6443 K3S_TOKEN=K100772d3a421be961da31afce5338280032a42ad8a152c683d22a0d2d72040af71::server:5840b68a95f662baf09269b2eceb7ce9 sh -

[sudo] password for k3s2:

[INFO] Finding release for channel stable

[INFO] Using v1.25.6+k3s1 as release

[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/sha256sum-amd64.txt

[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.6+k3s1/k3s

[INFO] Verifying binary download

[INFO] Installing k3s to /usr/local/bin/k3s

[INFO] Skipping installation of SELinux RPM

[INFO] Creating /usr/local/bin/kubectl symlink to k3s

[INFO] Creating /usr/local/bin/crictl symlink to k3s

[INFO] Creating /usr/local/bin/ctr symlink to k3s

[INFO] Creating killall script /usr/local/bin/k3s-killall.sh

[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh

[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env

[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service

[INFO] systemd: Enabling k3s-agent unit

Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.

[INFO] systemd: Starting k3s-agent

再开一个shell查看进程如下


k3s2@k3s2:~$ ps aux|grep k3s
k3s2        1794  0.0  0.4  19040  9712 ?        Ss   11:25   0:00 /lib/systemd/systemd --user
k3s2        1796  0.0  0.2 105288  4416 ?        S    11:25   0:00 (sd-pam)
k3s2        1801  0.0  0.2   8264  5280 tty1     S+   11:25   0:00 -bash
root        2368  0.0  0.4  13796  8968 ?        Ss   11:29   0:00 sshd: k3s2 [priv]
k3s2        2442  0.0  0.3  13932  6124 ?        S    11:29   0:00 sshd: k3s2@pts/0
k3s2        2443  0.0  0.2   8276  5172 pts/0    Ss   11:29   0:00 -bash
k3s2        3063  0.0  0.0   2608  1904 pts/0    S+   11:53   0:00 sh -
root        3241  0.0  0.2   9264  4656 pts/0    S+   11:54   0:00 sudo systemctl restart k3s-agent
root        3242  0.0  0.1   9276  3760 pts/0    S+   11:54   0:00 systemctl restart k3s-agent
root        3263  6.8  4.3 842952 87268 ?        Ssl  11:54   0:08 /usr/local/bin/k3s agent
root        3328  0.5  0.4  13796  9000 ?        Ss   11:56   0:00 sshd: k3s2 [priv]
k3s2        3416  0.0  0.3  13932  6140 ?        S    11:56   0:00 sshd: k3s2@pts/1
k3s2        3417  1.6  0.2   8276  5100 pts/1    Ss   11:56   0:00 -bash
k3s2        3428  0.0  0.1   8888  3380 pts/1    R+   11:56   0:00 ps aux
k3s2        3429  0.0  0.0   6432   724 pts/1    S+   11:56   0:00 grep --color=auto k3s

k3s2@k3s2:~$ ps aux|grep k3s

k3s2 1794 0.0 0.4 19040 9712 ? Ss 11:25 0:00 /lib/systemd/systemd --user

k3s2 1796 0.0 0.2 105288 4416 ? S 11:25 0:00 (sd-pam)

k3s2 1801 0.0 0.2 8264 5280 tty1 S+ 11:25 0:00 -bash

root 2368 0.0 0.4 13796 8968 ? Ss 11:29 0:00 sshd: k3s2 [priv]

k3s2 2442 0.0 0.3 13932 6124 ? S 11:29 0:00 sshd: k3s2@pts/0

k3s2 2443 0.0 0.2 8276 5172 pts/0 Ss 11:29 0:00 -bash

k3s2 3063 0.0 0.0 2608 1904 pts/0 S+ 11:53 0:00 sh -

root 3241 0.0 0.2 9264 4656 pts/0 S+ 11:54 0:00 sudo systemctl restart k3s-agent

root 3242 0.0 0.1 9276 3760 pts/0 S+ 11:54 0:00 systemctl restart k3s-agent

root 3263 6.8 4.3 842952 87268 ? Ssl 11:54 0:08 /usr/local/bin/k3s agent

root 3328 0.5 0.4 13796 9000 ? Ss 11:56 0:00 sshd: k3s2 [priv]

k3s2 3416 0.0 0.3 13932 6140 ? S 11:56 0:00 sshd: k3s2@pts/1

k3s2 3417 1.6 0.2 8276 5100 pts/1 Ss 11:56 0:00 -bash

k3s2 3428 0.0 0.1 8888 3380 pts/1 R+ 11:56 0:00 ps aux

k3s2 3429 0.0 0.0 6432 724 pts/1 S+ 11:56 0:00 grep --color=auto k3s

根据systemd服务注册的命令/usr/local/bin/k3s agent，可以看到大概率是执行systemctl restart k3s-agent卡住的


k3s2@k3s2:~$ cat /etc/systemd/system/k3s-agent.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    agent \

k3s2@k3s2:~$ cat /etc/systemd/system/k3s-agent.service

[Unit]

Description=Lightweight Kubernetes

Documentation=https://k3s.io

Wants=network-online.target

After=network-online.target

[Install]

WantedBy=multi-user.target

[Service]

Type=notify

EnvironmentFile=-/etc/default/%N

EnvironmentFile=-/etc/sysconfig/%N

EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env

KillMode=process

Delegate=yes

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=1048576

LimitNPROC=infinity

LimitCORE=infinity

TasksMax=infinity

TimeoutStartSec=0

Restart=always

RestartSec=5s

ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'

ExecStartPre=-/sbin/modprobe br_netfilter

ExecStartPre=-/sbin/modprobe overlay

ExecStart=/usr/local/bin/k3s \

agent \

~~token本身错的可能性不大，所以大概率是K3S_URL的问题，因为我用了本地的DNS，所以写的是域名。尝试配置下HOSTS试试~~

保险起见我去 /var/lib/rancher/k3s/agent/containerd/containerd.log看了下，居然还在执行东西。。。难道只是单纯的慢？这时候主节点已经能看到agent了，但ROLES里是none让我比较纠结是不是正常


k3s1@k3s1:~$ sudo kubectl get nodes
NAME   STATUS   ROLES                  AGE    VERSION
k3s1   Ready    control-plane,master   37m    v1.25.6+k3s1
k3s2   Ready    <none>                 5m8s   v1.25.6+k3s1

k3s1@k3s1:~$ sudo kubectl get nodes

NAME STATUS ROLES AGE VERSION

k3s1 Ready control-plane,master 37m v1.25.6+k3s1

k3s2 Ready <none> 5m8s v1.25.6+k3s1

根据命令行参数打印更加详细的节点信息如下，这样看起来worker工作是正常的


k3s1@k3s1:~$ sudo kubectl get nodes -o wide  --show-labels=true
NAME   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME          LABELS
k3s1   Ready    control-plane,master   44m   v1.25.6+k3s1   192.168.2.227   <none>        Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.15-k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
k3s2   Ready    <none>                 12m   v1.25.6+k3s1   192.168.2.176   <none>        Ubuntu 20.04.5 LTS   5.4.0-125-generic   containerd://1.6.15-k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s2,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s

k3s1@k3s1:~$ sudo kubectl get nodes -o wide --show-labels=true

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME LABELS

k3s1 Ready control-plane,master 44m v1.25.6+k3s1 192.168.2.227 <none> Ubuntu 20.04.5 LTS 5.4.0-125-generic containerd://1.6.15-k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s

k3s2 Ready <none> 12m v1.25.6+k3s1 192.168.2.176 <none> Ubuntu 20.04.5 LTS 5.4.0-125-generic containerd://1.6.15-k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,egress.k3s.io/cluster=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s2,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s

测试

尝试构建一个服务是否能正常运行，从k8s官网看看能不能扒一个现成的yaml测试


k3s1@k3s1:~$ sudo  kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

k3s1@k3s1:~$ sudo kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Kustomize Version: v4.5.7

Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6+k3s1", GitCommit:"9176e03c5788e467420376d10a1da2b6de6ff31f", GitTreeState:"clean", BuildDate:"2023-01-26T00:47:47Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

尝试创建DEMO


k3s1@k3s1:~$ sudo kubectl get deployments
No resources found in default namespace.
k3s1@k3s1:~$ sudo kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1
deployment.apps/kubernetes-bootcamp created
k3s1@k3s1:~$ sudo kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   0/1     1            0           4s

k3s1@k3s1:~$ sudo kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   1/1     1            1           44s

k3s1@k3s1:~$ sudo kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-75c5d958ff-dzdkw   1/1     Running   0          3m23s

k3s1@k3s1:~$ sudo kubectl describe pods
Name:             kubernetes-bootcamp-75c5d958ff-dzdkw
Namespace:        default
Priority:         0
Service Account:  default
Node:             k3s2/192.168.2.176
Start Time:       Sat, 04 Feb 2023 12:23:03 +0000
Labels:           app=kubernetes-bootcamp
                  pod-template-hash=75c5d958ff
Annotations:      <none>
Status:           Running
IP:               10.42.1.3
IPs:
  IP:           10.42.1.3
Controlled By:  ReplicaSet/kubernetes-bootcamp-75c5d958ff
Containers:
  kubernetes-bootcamp:
    Container ID:   containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9
    Image:          gcr.io/google-samples/kubernetes-bootcamp:v1
    Image ID:       gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 04 Feb 2023 12:23:46 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-r2mwg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  4m23s  default-scheduler  Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2
  Normal  Pulling    4m22s  kubelet            Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"
  Normal  Pulled     3m40s  kubelet            Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)
  Normal  Created    3m40s  kubelet            Created container kubernetes-bootcamp
  Normal  Started    3m40s  kubelet            Started container kubernetes-bootcamp


k3s1@k3s1:~$ sudo kubectl describe pods kubernetes-bootcamp-75c5d958ff-dzdkw
Name:             kubernetes-bootcamp-75c5d958ff-dzdkw
Namespace:        default
Priority:         0
Service Account:  default
Node:             k3s2/192.168.2.176
Start Time:       Sat, 04 Feb 2023 12:23:03 +0000
Labels:           app=kubernetes-bootcamp
                  pod-template-hash=75c5d958ff
Annotations:      <none>
Status:           Running
IP:               10.42.1.3
IPs:
  IP:           10.42.1.3
Controlled By:  ReplicaSet/kubernetes-bootcamp-75c5d958ff
Containers:
  kubernetes-bootcamp:
    Container ID:   containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9
    Image:          gcr.io/google-samples/kubernetes-bootcamp:v1
    Image ID:       gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 04 Feb 2023 12:23:46 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-r2mwg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  5m28s  default-scheduler  Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2
  Normal  Pulling    5m28s  kubelet            Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"
  Normal  Pulled     4m46s  kubelet            Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)
  Normal  Created    4m46s  kubelet            Created container kubernetes-bootcamp
  Normal  Started    4m46s  kubelet            Started container kubernetes-bootcamp

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

k3s1@k3s1:~$ sudo kubectl get deployments

No resources found in default namespace.

k3s1@k3s1:~$ sudo kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1

deployment.apps/kubernetes-bootcamp created

k3s1@k3s1:~$ sudo kubectl get deployments

NAME READY UP-TO-DATE AVAILABLE AGE

kubernetes-bootcamp 0/1 1 0 4s

k3s1@k3s1:~$ sudo kubectl get deployments

NAME READY UP-TO-DATE AVAILABLE AGE

kubernetes-bootcamp 1/1 1 1 44s

k3s1@k3s1:~$ sudo kubectl get pods

NAME READY STATUS RESTARTS AGE

kubernetes-bootcamp-75c5d958ff-dzdkw 1/1 Running 0 3m23s

k3s1@k3s1:~$ sudo kubectl describe pods

Name: kubernetes-bootcamp-75c5d958ff-dzdkw

Namespace: default

Priority: 0

Service Account: default

Node: k3s2/192.168.2.176

Start Time: Sat, 04 Feb 2023 12:23:03 +0000

Labels: app=kubernetes-bootcamp

pod-template-hash=75c5d958ff

Annotations: <none>

Status: Running

IP: 10.42.1.3

IPs:

IP: 10.42.1.3

Controlled By: ReplicaSet/kubernetes-bootcamp-75c5d958ff

Containers:

kubernetes-bootcamp:

Container ID: containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9

Image: gcr.io/google-samples/kubernetes-bootcamp:v1

Image ID: gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af

Port: <none>

Host Port: <none>

State: Running

Started: Sat, 04 Feb 2023 12:23:46 +0000

Ready: True

Restart Count: 0

Environment: <none>

Mounts:

/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)

Conditions:

Type Status

Initialized True

Ready True

ContainersReady True

PodScheduled True

Volumes:

kube-api-access-r2mwg:

Type: Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds: 3607

ConfigMapName: kube-root-ca.crt

ConfigMapOptional: <nil>

DownwardAPI: true

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 4m23s default-scheduler Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2

Normal Pulling 4m22s kubelet Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"

Normal Pulled 3m40s kubelet Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)

Normal Created 3m40s kubelet Created container kubernetes-bootcamp

Normal Started 3m40s kubelet Started container kubernetes-bootcamp

k3s1@k3s1:~$ sudo kubectl describe pods kubernetes-bootcamp-75c5d958ff-dzdkw

Name: kubernetes-bootcamp-75c5d958ff-dzdkw

Namespace: default

Priority: 0

Service Account: default

Node: k3s2/192.168.2.176

Start Time: Sat, 04 Feb 2023 12:23:03 +0000

Labels: app=kubernetes-bootcamp

pod-template-hash=75c5d958ff

Annotations: <none>

Status: Running

IP: 10.42.1.3

IPs:

IP: 10.42.1.3

Controlled By: ReplicaSet/kubernetes-bootcamp-75c5d958ff

Containers:

kubernetes-bootcamp:

Container ID: containerd://871938aef1c0923d784e32020801861b78b2dfb7d7a317423965e002826b4af9

Image: gcr.io/google-samples/kubernetes-bootcamp:v1

Image ID: gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af

Port: <none>

Host Port: <none>

State: Running

Started: Sat, 04 Feb 2023 12:23:46 +0000

Ready: True

Restart Count: 0

Environment: <none>

Mounts:

/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2mwg (ro)

Conditions:

Type Status

Initialized True

Ready True

ContainersReady True

PodScheduled True

Volumes:

kube-api-access-r2mwg:

Type: Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds: 3607

ConfigMapName: kube-root-ca.crt

ConfigMapOptional: <nil>

DownwardAPI: true

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 5m28s default-scheduler Successfully assigned default/kubernetes-bootcamp-75c5d958ff-dzdkw to k3s2

Normal Pulling 5m28s kubelet Pulling image "gcr.io/google-samples/kubernetes-bootcamp:v1"

Normal Pulled 4m46s kubelet Successfully pulled image "gcr.io/google-samples/kubernetes-bootcamp:v1" in 41.916860932s (41.916911292s including waiting)

Normal Created 4m46s kubelet Created container kubernetes-bootcamp

Normal Started 4m46s kubelet Started container kubernetes-bootcamp

尝试使用 kubectl exec进入容器


k3s1@k3s1:~$ sudo kubectl exec -it kubernetes-bootcamp-75c5d958ff-dzdkw bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

root@kubernetes-bootcamp-75c5d958ff-dzdkw:/# cat server.js
var http = require('http');
var requests=0;
var podname= process.env.HOSTNAME;
var startTime;
var host;
var handleRequest = function(request, response) {
  response.setHeader('Content-Type', 'text/plain');
  response.writeHead(200);
  response.write("Hello Kubernetes bootcamp! | Running on: ");
  response.write(host);
  response.end(" | v=1\n");
  console.log("Running On:" ,host, "| Total Requests:", ++requests,"| App Uptime:", (new Date() - startTime)/1000 , "seconds", "| Log Time:",new Date());
}
var www = http.createServer(handleRequest);
www.listen(8080,function () {
    startTime = new Date();;
    host = process.env.HOSTNAME;
    console.log ("Kubernetes Bootcamp App Started At:",startTime, "| Running On: " ,host, "\n" );
});

k3s1@k3s1:~$ sudo kubectl exec -it kubernetes-bootcamp-75c5d958ff-dzdkw bash

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

root@kubernetes-bootcamp-75c5d958ff-dzdkw:/# cat server.js

var http = require('http');

var requests=0;

var podname= process.env.HOSTNAME;

var startTime;

var host;

var handleRequest = function(request, response) {

response.setHeader('Content-Type', 'text/plain');

response.writeHead(200);

response.write("Hello Kubernetes bootcamp! | Running on: ");

response.write(host);

response.end(" | v=1\n");

console.log("Running On:" ,host, "| Total Requests:", ++requests,"| App Uptime:", (new Date() - startTime)/1000 , "seconds", "| Log Time:",new Date());

}

var www = http.createServer(handleRequest);

www.listen(8080,function () {

startTime = new Date();;

host = process.env.HOSTNAME;

console.log ("Kubernetes Bootcamp App Started At:",startTime, "| Running On: " ,host, "\n" );

});

总结

相比多年前的印象，现在的k8s可以借助k3s这个青春版快速的启动一个集群，是时候更新下这个错误认知了，现在启动一个k8s测试用是相当简便并且成本很低的。下次尝试原版k8s，成功后尝试kubekey这个类似ansible一样的自动化安装工具

参考

https://docs.k3s.io/architecture

https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/rancher-v2-7-1/

资源分配和实际占用统计 https://docs.k3s.io/reference/resource-profiling

自动化安装 https://kubesphere.io/zh/docs/v3.3/installing-on-linux/introduction/kubekey/

ROLES https://blog.csdn.net/Lingoesforstudy/article/details/116484624

https://kubernetes.io/zh-cn/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro/

https://blog.csdn.net/qq_44246980/article/details/120143353

20221217mongodb集群搭建

视频版

https://www.bilibili.com/video/BV1Lv4y1Q7xe/

环境准备

按https://www.mongodb.com/docs/manual/administration/production-notes/#std-label-prod-notes-supported-platforms 所说6.0支持20.04和18.04

没说22.04支持，保险起见按文档准备系统版本

至少3个以上的奇数节点，5.0以上必须使用域名配置，节点间端口27017网络要求互通


IP1 mongodb0.example.net
IP2 mongodb1.example.net
IP3 mongodb2.example.net
IP1 test.w.mongo.db
IP2 test.r.mongo.db
IP3 test.r.mongo.db

IP1 mongodb0.example.net

IP2 mongodb1.example.net

IP3 mongodb2.example.net

IP1 test.w.mongo.db

IP2 test.r.mongo.db

IP3 test.r.mongo.db

1.安装


wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -
sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -

sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

sudo apt-get update

sudo apt-get install -y mongodb-org

常见目录


数据 /var/lib/mongodb 
日志 /var/log/mongodb
配置 /etc/mongod.conf

数据 /var/lib/mongodb

日志 /var/log/mongodb

配置 /etc/mongod.conf

常用命令


sudo systemctl start mongod
sudo systemctl stop mongod
sudo systemctl restart mongod
sudo systemctl status mongod
mongosh

sudo systemctl start mongod

sudo systemctl stop mongod

sudo systemctl restart mongod

sudo systemctl status mongod

mongosh

2.集群配置

MongoDB 的复制集是由一组保持相同数据集的 mongod 进程组成的。复制集提供了冗余和高可用的能力，同时它是所有生产环境部署的基础

在特定情况下可以提高读取性能，因为客户端可以把读取请求发送给不同的服务器。被复制多份数据可以用于灾备或审计

复制集包含几个 data bearing nodes和可选的一个arbiter node。在data bearing nodes上有且仅有一个成员是主节点，主节点接收所有写请求操作，一个主节点会使用 { w: "majority" } 来标识

设置bindIp启用远程访问，生产环境注意提前设置密码和防火墙

https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-net.bindIp

配置集群名

https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-replication.replSetName


mongosh
#初始化集群配置
rs.initiate( {
   _id : "rs0",
   members: [
      { _id: 0, host: "mongodb0.example.net:27017" },
      { _id: 1, host: "mongodb1.example.net:27017" },
      { _id: 2, host: "mongodb2.example.net:27017" }
   ]
})

test> rs.initiate( {
...    _id : "rs0",
...    members: [
...       { _id: 0, host: "test1.mongo.db:27017" },
...       { _id: 1, host: "test2.mongo.db:27017" },
...       { _id: 2, host: "test3.mongo.db:27017" }
...    ]
... })
{ ok: 1 }
rs0 [direct: other] test>

#查看集群配置
rs.conf()

rs0 [direct: other] test> rs.conf()
{
  _id: 'rs0',
  version: 1,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'test1.mongo.db:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 1,
      host: 'test2.mongo.db:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 2,
      host: 'test3.mongo.db:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
      votes: 1
    }
  ],
  protocolVersion: Long("1"),
  writeConcernMajorityJournalDefault: true,
  settings: {
    chainingAllowed: true,
    heartbeatIntervalMillis: 2000,
    heartbeatTimeoutSecs: 10,
    electionTimeoutMillis: 10000,
    catchUpTimeoutMillis: -1,
    catchUpTakeoverDelayMillis: 30000,
    getLastErrorModes: {},
    getLastErrorDefaults: { w: 1, wtimeout: 0 },
    replicaSetId: ObjectId("639db0b39d9a0c1c17cfaf5a")
  }
}
rs0 [direct: primary] test>


#查看主节点
rs.status()


rs0 [direct: primary] test> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:06:57.005Z"),
  myState: 1,
  term: Long("1"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    lastCommittedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    appliedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    durableOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
    lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671278771, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate("2022-12-17T12:06:22.806Z"),
    electionTerm: Long("1"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:06:22.864Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:06:23.645Z")
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 142,
      optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-17T12:06:52.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671278782, i: 1 }),
      electionDate: ISODate("2022-12-17T12:06:22.000Z"),
      configVersion: 1,
      configTerm: 1,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 45,
      optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-17T12:06:52.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.864Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test1.mongo.db:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 1
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 45,
      optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },
      optimeDate: ISODate("2022-12-17T12:06:52.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),
      lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.860Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test1.mongo.db:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 1
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671278812, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671278812, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

mongosh

#初始化集群配置

rs.initiate( {

_id : "rs0",

members: [

{ _id: 0, host: "mongodb0.example.net:27017" },

{ _id: 1, host: "mongodb1.example.net:27017" },

{ _id: 2, host: "mongodb2.example.net:27017" }

]

})

test> rs.initiate( {

... _id : "rs0",

... members: [

... { _id: 0, host: "test1.mongo.db:27017" },

... { _id: 1, host: "test2.mongo.db:27017" },

... { _id: 2, host: "test3.mongo.db:27017" }

... ]

... })

{ ok: 1 }

rs0 [direct: other] test>

#查看集群配置

rs.conf()

rs0 [direct: other] test> rs.conf()

{

_id: 'rs0',

version: 1,

term: 1,

members: [

{

_id: 0,

host: 'test1.mongo.db:27017',

arbiterOnly: false,

buildIndexes: true,

hidden: false,

priority: 1,

tags: {},

secondaryDelaySecs: Long("0"),

votes: 1

{

_id: 1,

host: 'test2.mongo.db:27017',

arbiterOnly: false,

buildIndexes: true,

hidden: false,

priority: 1,

tags: {},

secondaryDelaySecs: Long("0"),

votes: 1

{

_id: 2,

host: 'test3.mongo.db:27017',

arbiterOnly: false,

buildIndexes: true,

hidden: false,

priority: 1,

tags: {},

secondaryDelaySecs: Long("0"),

votes: 1

}

protocolVersion: Long("1"),

writeConcernMajorityJournalDefault: true,

settings: {

chainingAllowed: true,

heartbeatIntervalMillis: 2000,

heartbeatTimeoutSecs: 10,

electionTimeoutMillis: 10000,

catchUpTimeoutMillis: -1,

catchUpTakeoverDelayMillis: 30000,

getLastErrorModes: {},

getLastErrorDefaults: { w: 1, wtimeout: 0 },

replicaSetId: ObjectId("639db0b39d9a0c1c17cfaf5a")

}

rs0 [direct: primary] test>

#查看主节点

rs.status()

rs0 [direct: primary] test> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:06:57.005Z"),

myState: 1,

term: Long("1"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

lastCommittedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

appliedOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

durableOpTime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671278771, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'electionTimeout',

lastElectionDate: ISODate("2022-12-17T12:06:22.806Z"),

electionTerm: Long("1"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:06:22.864Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:06:23.645Z")

members: [

{

_id: 0,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 142,

optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDate: ISODate("2022-12-17T12:06:52.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671278782, i: 1 }),

electionDate: ISODate("2022-12-17T12:06:22.000Z"),

configVersion: 1,

configTerm: 1,

self: true,

lastHeartbeatMessage: ''

{

_id: 1,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 45,

optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDate: ISODate("2022-12-17T12:06:52.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.864Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test1.mongo.db:27017',

syncSourceId: 0,

infoMessage: '',

configVersion: 1,

configTerm: 1

{

_id: 2,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 45,

optime: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDurable: { ts: Timestamp({ t: 1671278812, i: 1 }), t: Long("1") },

optimeDate: ISODate("2022-12-17T12:06:52.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:06:52.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastDurableWallTime: ISODate("2022-12-17T12:06:52.885Z"),

lastHeartbeat: ISODate("2022-12-17T12:06:56.857Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:06:55.860Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test1.mongo.db:27017',

syncSourceId: 0,

infoMessage: '',

configVersion: 1,

configTerm: 1

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671278812, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671278812, i: 1 })

}

3.灾备测试

挂掉1之后到2 上查看集群状态如下，可以发现2自动变成了主


rs0 [direct: primary] test> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:18:35.005Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671279498, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'stepUpRequestSkipDryRun',
    lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 0,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")
  },
  electionParticipantMetrics: {
    votedForCandidate: true,
    electionTerm: Long("1"),
    lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),
    electionCandidateMemberId: 0,
    voteReason: '',
    lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    priorityAtElection: 1
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 0,
      state: 8,
      stateStr: '(not reachable/healthy)',
      uptime: 0,
      optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDate: ISODate("1970-01-01T00:00:00.000Z"),
      optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:17:38.860Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:17:38.860Z"),
      lastHeartbeat: ISODate("2022-12-17T12:18:33.050Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:17:41.884Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: 'Error connecting to test1.mongo.db:27017 (192.168.2.224:27017) :: caused by :: Connection refused',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 839,
      optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:18:28.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671279448, i: 1 }),
      electionDate: ISODate("2022-12-17T12:17:28.000Z"),
      configVersion: 1,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 742,
      optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:18:28.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:18:28.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),
      lastHeartbeat: ISODate("2022-12-17T12:18:34.909Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:18:33.989Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test2.mongo.db:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671279508, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671279508, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

rs0 [direct: primary] test> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:18:35.005Z"),

myState: 1,

term: Long("2"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

lastCommittedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

appliedOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

durableOpTime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671279498, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'stepUpRequestSkipDryRun',

lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),

electionTerm: Long("2"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

priorPrimaryMemberId: 0,

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")

electionParticipantMetrics: {

votedForCandidate: true,

electionTerm: Long("1"),

lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),

electionCandidateMemberId: 0,

voteReason: '',

lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

priorityAtElection: 1

members: [

{

_id: 0,

health: 0,

state: 8,

stateStr: '(not reachable/healthy)',

uptime: 0,

optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDate: ISODate("1970-01-01T00:00:00.000Z"),

optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:17:38.860Z"),

lastDurableWallTime: ISODate("2022-12-17T12:17:38.860Z"),

lastHeartbeat: ISODate("2022-12-17T12:18:33.050Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:17:41.884Z"),

pingMs: Long("0"),

lastHeartbeatMessage: 'Error connecting to test1.mongo.db:27017 (192.168.2.224:27017) :: caused by :: Connection refused',

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

configVersion: 1,

configTerm: 2

{

_id: 1,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 839,

optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:18:28.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671279448, i: 1 }),

electionDate: ISODate("2022-12-17T12:17:28.000Z"),

configVersion: 1,

configTerm: 2,

self: true,

lastHeartbeatMessage: ''

{

_id: 2,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 742,

optime: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279508, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:18:28.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:18:28.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastDurableWallTime: ISODate("2022-12-17T12:18:28.862Z"),

lastHeartbeat: ISODate("2022-12-17T12:18:34.909Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:18:33.989Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test2.mongo.db:27017',

syncSourceId: 1,

infoMessage: '',

configVersion: 1,

configTerm: 2

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671279508, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671279508, i: 1 })

}

重启1之后，再次查看，可以发现1重新加入了集群，但身份是从节点


rs0 [direct: primary] test> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:19:42.878Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671279558, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'stepUpRequestSkipDryRun',
    lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 0,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")
  },
  electionParticipantMetrics: {
    votedForCandidate: true,
    electionTerm: Long("1"),
    lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),
    electionCandidateMemberId: 0,
    voteReason: '',
    lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    priorityAtElection: 1
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 5,
      optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:19:39.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastHeartbeat: ISODate("2022-12-17T12:19:41.318Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:19:41.768Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test3.mongo.db:27017',
      syncSourceId: 2,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 907,
      optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:19:39.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671279448, i: 1 }),
      electionDate: ISODate("2022-12-17T12:17:28.000Z"),
      configVersion: 1,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 810,
      optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:19:39.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),
      lastHeartbeat: ISODate("2022-12-17T12:19:40.962Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:19:42.044Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'test2.mongo.db:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671279579, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671279579, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

rs0 [direct: primary] test> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:19:42.878Z"),

myState: 1,

term: Long("2"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

lastCommittedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

appliedOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

durableOpTime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671279558, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'stepUpRequestSkipDryRun',

lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),

electionTerm: Long("2"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

priorPrimaryMemberId: 0,

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")

electionParticipantMetrics: {

votedForCandidate: true,

electionTerm: Long("1"),

lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),

electionCandidateMemberId: 0,

voteReason: '',

lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

priorityAtElection: 1

members: [

{

_id: 0,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 5,

optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:19:39.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastHeartbeat: ISODate("2022-12-17T12:19:41.318Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:19:41.768Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test3.mongo.db:27017',

syncSourceId: 2,

infoMessage: '',

configVersion: 1,

configTerm: 2

{

_id: 1,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 907,

optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:19:39.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671279448, i: 1 }),

electionDate: ISODate("2022-12-17T12:17:28.000Z"),

configVersion: 1,

configTerm: 2,

self: true,

lastHeartbeatMessage: ''

{

_id: 2,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 810,

optime: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279579, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:19:39.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:19:39.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastDurableWallTime: ISODate("2022-12-17T12:19:39.170Z"),

lastHeartbeat: ISODate("2022-12-17T12:19:40.962Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:19:42.044Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: 'test2.mongo.db:27017',

syncSourceId: 1,

infoMessage: '',

configVersion: 1,

configTerm: 2

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671279579, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671279579, i: 1 })

}

从2写入点新数据，到1查询测试，可以发现是正常的


rs0 [direct: secondary] test> use sample_mflix
switched to db sample_mflix
rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite1" } )
[
  {
    _id: ObjectId("639db429d29c478bd6ec0507"),
    title: 'The Favourite1',
    genres: [ 'Drama', 'History' ],
    runtime: 121,
    rated: 'R',
    year: 2018,
    directors: [ 'Yorgos Lanthimos' ],
    cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],
    type: 'movie'
  }
]
rs0 [direct: secondary] sample_mflix>

rs0 [direct: secondary] test> use sample_mflix

switched to db sample_mflix

rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite1" } )

[

{

_id: ObjectId("639db429d29c478bd6ec0507"),

title: 'The Favourite1',

genres: [ 'Drama', 'History' ],

runtime: 121,

rated: 'R',

year: 2018,

directors: [ 'Yorgos Lanthimos' ],

cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],

type: 'movie'

}

]

rs0 [direct: secondary] sample_mflix>

挂掉现在的从节点3，写入新数据后再重启3


rs0 [direct: primary] sample_mflix> rs.status()
{
  set: 'rs0',
  date: ISODate("2022-12-17T12:22:41.079Z"),
  myState: 1,
  term: Long("2"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    lastCommittedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    appliedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    durableOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
    lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
    lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1671279738, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'stepUpRequestSkipDryRun',
    lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),
    electionTerm: Long("2"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 0,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")
  },
  electionParticipantMetrics: {
    votedForCandidate: true,
    electionTerm: Long("1"),
    lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),
    electionCandidateMemberId: 0,
    voteReason: '',
    lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },
    priorityAtElection: 1
  },
  members: [
    {
      _id: 0,
      name: 'test1.mongo.db:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 183,
      optime: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },
      optimeDurable: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:22:28.000Z"),
      optimeDurableDate: ISODate("2022-12-17T12:22:28.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      lastHeartbeat: ISODate("2022-12-17T12:22:39.460Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:22:39.967Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    },
    {
      _id: 1,
      name: 'test2.mongo.db:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 1086,
      optime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2022-12-17T12:22:38.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1671279448, i: 1 }),
      electionDate: ISODate("2022-12-17T12:17:28.000Z"),
      configVersion: 1,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 2,
      name: 'test3.mongo.db:27017',
      health: 0,
      state: 8,
      stateStr: '(not reachable/healthy)',
      uptime: 0,
      optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDate: ISODate("1970-01-01T00:00:00.000Z"),
      optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),
      lastAppliedWallTime: ISODate("2022-12-17T12:22:28.878Z"),
      lastDurableWallTime: ISODate("2022-12-17T12:22:28.878Z"),
      lastHeartbeat: ISODate("2022-12-17T12:22:39.106Z"),
      lastHeartbeatRecv: ISODate("2022-12-17T12:22:34.175Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: 'Error connecting to test3.mongo.db:27017 (192.168.2.178:27017) :: caused by :: Connection refused',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 2
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1671279758, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1671279758, i: 1 })
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

rs0 [direct: primary] sample_mflix> rs.status()

{

set: 'rs0',

date: ISODate("2022-12-17T12:22:41.079Z"),

myState: 1,

term: Long("2"),

syncSourceHost: '',

syncSourceId: -1,

heartbeatIntervalMillis: Long("2000"),

majorityVoteCount: 2,

writeMajorityCount: 2,

votingMembersCount: 3,

writableVotingMembersCount: 3,

optimes: {

lastCommittedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

lastCommittedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

readConcernMajorityOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

appliedOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

durableOpTime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z")

lastStableRecoveryTimestamp: Timestamp({ t: 1671279738, i: 1 }),

electionCandidateMetrics: {

lastElectionReason: 'stepUpRequestSkipDryRun',

lastElectionDate: ISODate("2022-12-17T12:17:28.838Z"),

electionTerm: Long("2"),

lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1671279442, i: 1 }), t: Long("1") },

numVotesNeeded: 2,

priorityAtElection: 1,

electionTimeoutMillis: Long("10000"),

priorPrimaryMemberId: 0,

numCatchUpOps: Long("0"),

newTermStartDate: ISODate("2022-12-17T12:17:28.858Z"),

wMajorityWriteAvailabilityDate: ISODate("2022-12-17T12:17:29.859Z")

electionParticipantMetrics: {

votedForCandidate: true,

electionTerm: Long("1"),

lastVoteDate: ISODate("2022-12-17T12:06:22.621Z"),

electionCandidateMemberId: 0,

voteReason: '',

lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1671278771, i: 1 }), t: Long("-1") },

priorityAtElection: 1

members: [

{

_id: 0,

health: 1,

state: 2,

stateStr: 'SECONDARY',

uptime: 183,

optime: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },

optimeDurable: { ts: Timestamp({ t: 1671279748, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:22:28.000Z"),

optimeDurableDate: ISODate("2022-12-17T12:22:28.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastHeartbeat: ISODate("2022-12-17T12:22:39.460Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:22:39.967Z"),

pingMs: Long("0"),

lastHeartbeatMessage: '',

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

configVersion: 1,

configTerm: 2

{

_id: 1,

health: 1,

state: 1,

stateStr: 'PRIMARY',

uptime: 1086,

optime: { ts: Timestamp({ t: 1671279758, i: 1 }), t: Long("2") },

optimeDate: ISODate("2022-12-17T12:22:38.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:22:38.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:38.878Z"),

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

electionTime: Timestamp({ t: 1671279448, i: 1 }),

electionDate: ISODate("2022-12-17T12:17:28.000Z"),

configVersion: 1,

configTerm: 2,

self: true,

lastHeartbeatMessage: ''

{

_id: 2,

health: 0,

state: 8,

stateStr: '(not reachable/healthy)',

uptime: 0,

optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },

optimeDate: ISODate("1970-01-01T00:00:00.000Z"),

optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),

lastAppliedWallTime: ISODate("2022-12-17T12:22:28.878Z"),

lastDurableWallTime: ISODate("2022-12-17T12:22:28.878Z"),

lastHeartbeat: ISODate("2022-12-17T12:22:39.106Z"),

lastHeartbeatRecv: ISODate("2022-12-17T12:22:34.175Z"),

pingMs: Long("0"),

lastHeartbeatMessage: 'Error connecting to test3.mongo.db:27017 (192.168.2.178:27017) :: caused by :: Connection refused',

syncSourceHost: '',

syncSourceId: -1,

infoMessage: '',

configVersion: 1,

configTerm: 2

}

ok: 1,

'$clusterTime': {

clusterTime: Timestamp({ t: 1671279758, i: 1 }),

signature: {

hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),

keyId: Long("0")

}

operationTime: Timestamp({ t: 1671279758, i: 1 })

}

可以发现新写入的数据依然可以正常查询


rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite2" } )
[
  {
    _id: ObjectId("639db4b7d29c478bd6ec0508"),
    title: 'The Favourite2',
    genres: [ 'Drama', 'History' ],
    runtime: 121,
    rated: 'R',
    year: 2018,
    directors: [ 'Yorgos Lanthimos' ],
    cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],
    type: 'movie'
  }
]

rs0 [direct: secondary] sample_mflix> db.getMongo().setReadPref('secondary')

rs0 [direct: secondary] sample_mflix> db.movies.find( { title: "The Favourite2" } )

[

{

_id: ObjectId("639db4b7d29c478bd6ec0508"),

title: 'The Favourite2',

genres: [ 'Drama', 'History' ],

runtime: 121,

rated: 'R',

year: 2018,

directors: [ 'Yorgos Lanthimos' ],

cast: [ 'Olivia Colman', 'Emma Stone', 'Rachel Weisz' ],

type: 'movie'

}

]

附录


mongo

https://mirrors.tuna.tsinghua.edu.cn

#分配DNS
192.168.2.224 test1.mongo.db
192.168.2.147 test2.mongo.db
192.168.2.178 test3.mongo.db

#远程ssh+免密+禁用密码登录
ssh mongo1@test1.mongo.db
ssh mongo2@test2.mongo.db
ssh mongo3@test3.mongo.db

#安装mongo
wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -
sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

#单机测试
mongosh
端口 27017
db
use test

#集群配置
修改每个机器的bindIp
sudo vim /etc/mongod.conf
sudo service mongod restart

#尝试远程连接
mongosh test1.mongo.db
mongosh test2.mongo.db
mongosh test3.mongo.db

#配置集群名
  replSetName: "rs0"

sudo vim /etc/mongod.conf
sudo service mongod restart

#初始化集群
rs.initiate( {
   _id : "rs0",
   members: [
      { _id: 0, host: "test1.mongo.db:27017" },
      { _id: 1, host: "test2.mongo.db:27017" },
      { _id: 2, host: "test3.mongo.db:27017" }
   ]
})

rs.conf()

rs.status()

#集群测试
use sample_mflix

db.movies.insertOne(
  {
    title: "The Favourite",
    genres: [ "Drama", "History" ],
    runtime: 121,
    rated: "R",
    year: 2018,
    directors: [ "Yorgos Lanthimos" ],
    cast: [ "Olivia Colman", "Emma Stone", "Rachel Weisz" ],
    type: "movie"
  }
)

db.getMongo().setReadPref('secondary')
db.movies.find( { title: "The Favourite" } )

mongo

https://mirrors.tuna.tsinghua.edu.cn

#分配DNS

192.168.2.224 test1.mongo.db

192.168.2.147 test2.mongo.db

192.168.2.178 test3.mongo.db

#远程ssh+免密+禁用密码登录

ssh mongo1@test1.mongo.db

ssh mongo2@test2.mongo.db

ssh mongo3@test3.mongo.db

#安装mongo

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -

sudo touch /etc/apt/sources.list.d/mongodb-org-6.0.list

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

sudo apt-get update

sudo apt-get install -y mongodb-org

#单机测试

mongosh

端口 27017

use test

#集群配置

修改每个机器的bindIp

sudo vim /etc/mongod.conf

sudo service mongod restart

#尝试远程连接

mongosh test1.mongo.db

mongosh test2.mongo.db

mongosh test3.mongo.db

#配置集群名

replSetName: "rs0"

sudo vim /etc/mongod.conf

sudo service mongod restart

#初始化集群

rs.initiate( {

_id : "rs0",

members: [

{ _id: 0, host: "test1.mongo.db:27017" },

{ _id: 1, host: "test2.mongo.db:27017" },

{ _id: 2, host: "test3.mongo.db:27017" }

]

})

rs.conf()

rs.status()

#集群测试

use sample_mflix

db.movies.insertOne(

{

title: "The Favourite",

genres: [ "Drama", "History" ],

runtime: 121,

rated: "R",

year: 2018,

directors: [ "Yorgos Lanthimos" ],

cast: [ "Olivia Colman", "Emma Stone", "Rachel Weisz" ],

type: "movie"

}

)

db.getMongo().setReadPref('secondary')

db.movies.find( { title: "The Favourite" } )

参考

https://www.mongodb.com/docs/manual/

https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition

https://www.mongodb.com/docs/manual/replication/

https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set/

https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set-with-keyfile-access-control/#std-label-deploy-repl-set-with-auth