956237586 – 第 3 页 – HYL Studio

背景

20241003kasm idea web workspace

20220615自行编译projector-client

拖了好久都没修这两个问题

打包内置静态资源

分离projector中使用的静态资源地址，改成本地打包解决网络加载js、css的速度问题

支持输入框修改token

edge的PWA正常不会记录query string，让projector支持从url鉴权需要动的东西有点多，所以加个输入框支持手动输入重连即可

坑的是kotlin输出$需要用${‘$’}

client版本v1.8.1.12

server版本v1.8.1.12

TODO 动态生成PWA的manifest

org.jetbrains.projector.server.core.websocket.HttpWsServer.Companion#getResourceName

当前实现的是纯静态的服务器，加载一个kotlin编译成的js文件

入口是projector-client-web/src/main/kotlin/org/jetbrains/projector/client/web/Main.kt

org.jetbrains.projector.client.web.Application#start

入口是启动一个状态机做死循环，初始状态是

private var currentState: ClientState = ClientState.UninitializedPage

第一个队列事件是

projector-client-web/src/main/kotlin/org/jetbrains/projector/client/web/state/ClientAction.kt

org.jetbrains.projector.client.web.state.ClientAction.Start

因此处理过程是

org.jetbrains.projector.client.web.state.ClientState.UninitializedPage#consume

下一个状态是WaitingOpening

想让index.html的内容变成ssr，需要从HttpWsServer开始修改成ssr。或者从更前面做拦截把GET请求内置资源的部分转到独立服务器，

官方之前给过electron的客户端，说明分离客户端的网页是可行的，只需要把客户端尝试独立部署即可完成各种自定义开发。且无需修改服务器代码，服务器的内置代码仅作为兜底使用即可

TODO 让projector支持从urlpath鉴权规避PWA的限制

TODO

TODO 待继续研究考虑实现通用的jcef webview显示转发成canvas方案

TODO 再写一个server，接受所有projector client的上报，提供个网页支持动态从服务器获取完成的url并完成本地的端口转发

20241003kasm idea web workspace

背景

在解决continue加载的时候，重新构建的idea web workspace

20241002continue在projector环境下的加载和相关问题

其实是个副产物，之前做飞连镜像的时候也用过

这次是升级了操作系统版本，构建成功了，在22.04的ubuntu上重新打包了workspcae，当开启jcef后可在vnc环境下正常运行浏览器、JCEF App

代码和说明参考 https://github.com/dev-assistant/skykoma-workspace/tree/main/workspace-ubuntu

注意默认带了个idea干净纯净的ubuntu是base的那个镜像

实现细节

TODO 后面有时间再详细补充，难点是跳过idea默认的各种弹框

后续

TODO 找出https://github.com/kasmtech/KasmVNC/issues/207 的原因：https://issues.chromium.org/issues/40418163 修掉这个快十年的老bug感觉没戏

TODO 支持人工指定证书 or 强制使用http 避开浏览器的bug

TODO 基于22.04重做一个飞连的镜像，同时升级到最新的飞连

TODO 是否可能用 https://github.com/gojue/ecapture抓到wg的配置后弃用飞连客户端？

https://github.com/PinkD/corplink-rs/blob/master/src/client.rs#L652

https://github.com/PinkD/corplink-rs/blob/master/src/api.rs#L21

20241002continue在projector环境下的加载和相关问题

懒得排版了，直接去飞书文档看吧

https://paraparty.feishu.cn/docx/JQtkdMWu7oSsjwxwcpYcww7Inyf

代码 https://github.com/continuedev/continue/pull/2439

0.背景

博客地址 https://blog.hylstudio.cn/archives/1350

飞书文档 https://paraparty.feishu.cn/docx/GEwVdulCgoLX7dxSHMecy4mBnch

argocd试玩，对于argocd定位是cd，能否胜任k8s面板存疑

结论：不能当作k8s面板使用，但对小白开发比较友好

这里作为备份，为了最好的阅读体验可以看飞书

1.安装


mkdir argocd cd argocd wget https://raw.githubusercontent.com/argoproj/argo-cd/v2.9.3/manifests/ha/install.yaml

0 1	mkdir argocd cd argocd wget https://raw.githubusercontent.com/argoproj/argo-cd/v2.9.3/manifests/ha/install.yaml

提取镜像路径，做本地化


sudo docker pull ghcr.io/dexidp/dex:v2.37.0
sudo docker tag ghcr.io/dexidp/dex:v2.37.0 harbor.hylstudio.local/dexidp/dex:v2.37.0
sudo docker push harbor.hylstudio.local/dexidp/dex:v2.37.0

sudo docker pull redis:7.0.11-alpine
sudo docker tag redis:7.0.11-alpine harbor.hylstudio.local/library/redis:7.0.11-alpine
sudo docker push harbor.hylstudio.local/library/redis:7.0.11-alpine

sudo docker pull quay.io/argoproj/argocd:v2.9.3
sudo docker tag quay.io/argoproj/argocd:v2.9.3 harbor.hylstudio.local/argoproj/argocd:v2.9.3
sudo docker push harbor.hylstudio.local/argoproj/argocd:v2.9.3

sudo docker pull ghcr.io/dexidp/dex:v2.37.0

sudo docker tag ghcr.io/dexidp/dex:v2.37.0 harbor.hylstudio.local/dexidp/dex:v2.37.0

sudo docker push harbor.hylstudio.local/dexidp/dex:v2.37.0

sudo docker pull redis:7.0.11-alpine

sudo docker tag redis:7.0.11-alpine harbor.hylstudio.local/library/redis:7.0.11-alpine

sudo docker push harbor.hylstudio.local/library/redis:7.0.11-alpine

sudo docker pull quay.io/argoproj/argocd:v2.9.3

sudo docker tag quay.io/argoproj/argocd:v2.9.3 harbor.hylstudio.local/argoproj/argocd:v2.9.3

sudo docker push harbor.hylstudio.local/argoproj/argocd:v2.9.3

替换install.yaml中的镜像路径


kubectl create namespace argocd kubectl apply -n argocd -f install.yaml

0 1	kubectl create namespace argocd kubectl apply -n argocd -f install.yaml

安装后容器列表如图

资源占用如图

2.kubectl配置转发


kubectl -n argocd port-forward service/argocd-server :80

0 1	kubectl -n argocd port-forward service/argocd-server :80

到argocd-initial-admin-secret这里找默认密码，用户名是admin

3.本地命令行配置

mkdir -p /usr/local/argocd/bin cd /usr/local/argocd/bin wget https://localhost:43819/download/argocd-linux-amd64 mv argocd-linux-amd64 argocd cd /usr/local/bin ln -s /usr/local/argocd/bin/argocd

4.first demo

kubectl config set-context –current –namespace=argocd argocd app create guestbook –repo https://github.com/argoproj/argocd-example-apps.git –path guestbook –dest-server https://kubernetes.default.svc –dest-namespace default argocd app get guestbook argocd app sync guestbook

参考

https://argo-cd.readthedocs.io/en/stable/getting_started/

20231125ks-installer追踪

0.背景

博客地址 https://blog.hylstudio.cn/archives/1343

飞书文档 https://paraparty.feishu.cn/docx/FTM7d1TIcoxL83xfRWzc34omnFe

安装k8s的时候最后一步会特别慢，之前追了一半，今天继续追完搞清楚发生了啥

懒得排版了，为了最佳阅读体验可以看飞书，这里做备份

1.追踪过程

前置知识：kubectl、shell-operator、python、ansible、helm

从https://github.com/kubesphere/ks-installer/blob/20a6daa18adf10410a385b48ab2769e55d8bdee2/Dockerfile.complete#L8 我们可以看到主入口的程序来源是https://github.com/flant/shell-operator 这是一个运行在k8s集群中事件驱动脚本的工具。

根据https://github.com/flant/shell-operator/blob/55ca7a92c873cccfdad0c7591048eaeb8cf0dd4b/docs/src/HOOKS.md?plain=1#L11 描述shell-operator默认会递归扫描hooks目录下的文件把按文件的标准输出当作声明监听event

根据https://github.com/kubesphere/ks-installer/blob/20a6daa18adf10410a385b48ab2769e55d8bdee2/Dockerfile#L5 可知controller下的文件会被放到/hooks下作为监听，其中包括installerRunner.py

从 https://github.com/kubesphere/ks-installer/blob/20a6daa18adf10410a385b48ab2769e55d8bdee2/controller/installRunner.py#L30C12-L30C32 可知installRunner.py会在ClusterConfiguration创建或更新的时候被触发。

从 https://github.com/kubesphere/kubekey/blob/ffaa19f430a6b13aa219c7ec699fc2ea705c3a93/cmd/kk/pkg/kubesphere/modules.go#L101 可找到这个触发动作

TODO 待详细查看触发动作中yaml是如何传递kubesphere版本信息的

从 https://github.com/kubesphere/ks-installer/blob/20a6daa18adf10410a385b48ab2769e55d8bdee2/controller/installRunner.py#L338C41-L338C53 可以看到实际执行动作的是ansible，用的是https://ansible.readthedocs.io/projects/runner/en/stable/index.html

run方法文档https://ansible.readthedocs.io/projects/runner/en/stable/ansible_runner/#ansible_runner.interface.run


verbosity (int) – Control how verbose the output of ansible-playbook is
_input (io.FileIO) – An optional file or file-like object for use as input in a streaming pipeline
_output (io.FileIO) – An optional file or file-like object for use as output in a streaming pipeline

verbosity (int) – Control how verbose the output of ansible-playbook is

_input (io.FileIO) – An optional file or file-like object for use as input in a streaming pipeline

_output (io.FileIO) – An optional file or file-like object for use as output in a streaming pipeline

TODO 从ansible playbook中找到kubesphere版本信息是哪个变量
TODO 确认从kubekey到ansible playbook的版本信息传递数据流

参考https://www.cnblogs.com/sylvia-liu/p/14933776.html强制覆盖entrypoint进去修改python


docker pull harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker run -it --entrypoint /bin/bash harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
cp /hooks/kubesphere/installRunner.py /hooks/kubesphere/installRunner.py.bak
vi /hooks/kubesphere/installRunner.py

docker pull harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker run -it --entrypoint /bin/bash harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

cp /hooks/kubesphere/installRunner.py /hooks/kubesphere/installRunner.py.bak

vi /hooks/kubesphere/installRunner.py

进入shell修改python脚本


import io
ansible_log=io.FileIO('/home/kubesphere/ansible.log', 'w')

            _output=ansible_log,
            verbosity=5

import io

ansible_log=io.FileIO('/home/kubesphere/ansible.log', 'w')

_output=ansible_log,

verbosity=5

重新打一个镜像v3.3.3


docker commit 5274b25c35d5 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.3
docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.3

docker commit 5274b25c35d5 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.3

docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.3

安装虚机


./vm.sh -c 4 -m 8 -d 80 -p on k8s1
./vm.sh -c 2 -m 4 -d 40 -p on k8s2
./vm.sh -c 2 -m 4 -d 40 -p on k8s3

./vm.sh -c 4 -m 8 -d 80 -p on k8s1

./vm.sh -c 2 -m 4 -d 40 -p on k8s2

./vm.sh -c 2 -m 4 -d 40 -p on k8s3

分配域名


#dns config
192.168.2.206 k8s-control.hylstudio.local
192.168.2.206 k8s1.k8s.local
192.168.2.177 k8s2.k8s.local
192.168.2.203 k8s3.k8s.local

#dns config

192.168.2.206 k8s-control.hylstudio.local

192.168.2.206 k8s1.k8s.local

192.168.2.177 k8s2.k8s.local

192.168.2.203 k8s3.k8s.local

强行修改ClusterConfiguration发现kk会报错，改回3.3.2先安装


apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.3.2

apiVersion: installer.kubesphere.io/v1alpha1

kind: ClusterConfiguration

metadata:

name: ks-installer

namespace: kubesphere-system

labels:

version: v3.3.2

尝试给v3.3.2替换成v3.3.3的内容，替换之前对v3.3.2备份

#做个假的3.3.2骗掉版本检测


#做个假的3.3.2骗掉版本检测
docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2-bak
docker image rm harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.3 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

#做个假的3.3.2骗掉版本检测

docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2-bak

docker image rm harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.3 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

安装到最后一步的时候进入ks-installer


namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
13:55:05 UTC success: [k8s1]
Please wait for the installation to complete:    >>--->

namespace/kubesphere-system unchanged

serviceaccount/ks-installer unchanged

customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged

clusterrole.rbac.authorization.k8s.io/ks-installer unchanged

clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged

deployment.apps/ks-installer unchanged

clusterconfiguration.installer.kubesphere.io/ks-installer created

13:55:05 UTC success: [k8s1]

Please wait for the installation to complete: >>--->

查看python文件发现不是从本地pull的镜像，但describe显示地址和tag都是对的。查看harbor上是sha也是ok的，怀疑启动时候还有别的逻辑会修改python文件

相关进程如下


ks-installer-566ffb8f44-ml9gm:/kubesphere$ ps aux
PID   USER     TIME  COMMAND
    1 kubesphe  0:00 /shell-operator start
   56 kubesphe  0:06 python3 /hooks/kubesphere/installRunner.py
 2501 kubesphe  1:21 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubespher
 4348 kubesphe  0:01 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubespher
 5261 kubesphe  0:00 /bin/sh -c /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700921243.
 5262 kubesphe  0:00 /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700921243.7008557-434
 5263 kubesphe  0:00 /usr/local/bin/kubectl apply -f /kubesphere/kubesphere/ks-core/crds/iam.kubesphere.io_
 5287 kubesphe  0:00 bash
 5299 kubesphe  0:00 ps aux

ks-installer-566ffb8f44-ml9gm:/kubesphere$ ps aux

PID USER TIME COMMAND

1 kubesphe 0:00 /shell-operator start

56 kubesphe 0:06 python3 /hooks/kubesphere/installRunner.py

2501 kubesphe 1:21 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubespher

4348 kubesphe 0:01 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubespher

5261 kubesphe 0:00 /bin/sh -c /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700921243.

5262 kubesphe 0:00 /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700921243.7008557-434

5263 kubesphe 0:00 /usr/local/bin/kubectl apply -f /kubesphere/kubesphere/ks-core/crds/iam.kubesphere.io_

5287 kubesphe 0:00 bash

5299 kubesphe 0:00 ps aux

日志显示如下


Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
**************************************************
Waiting for all tasks to be completed ...
task network status is successful  (1/4)
task openpitrix status is successful  (2/4)
task multicluster status is successful  (3/4)
只能怀疑是monitoring了

Start installing monitoring

Start installing multicluster

Start installing openpitrix

Start installing network

**************************************************

Waiting for all tasks to be completed ...

task network status is successful (1/4)

task openpitrix status is successful (2/4)

task multicluster status is successful (3/4)

只能怀疑是monitoring了

但从外面执行的时候发现entrypoint已经不对了，用dockerfile重新打一个测试，不要用docker commit

手动cp出来修改过后的python文件


docker cp xxx:/hooks/kubesphere/installRunner.py .

0 1	docker cp xxx:/hooks/kubesphere/installRunner.py .

编写DockerFile


FROM harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2-bak
RUN rm -rf /hooks/kubesphere/installRunner.py
COPY installRunner.py /hooks/kubesphere/

FROM harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2-bak

RUN rm -rf /hooks/kubesphere/installRunner.py

COPY installRunner.py /hooks/kubesphere/

打个新的镜像，注意新建个文件夹减小docker build context体积


mkdir imgbuild
mv installRunner.py DockerFile imgbuild
cd imagebuild
docker build -f DockerFile -t harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.4 .
docker image rm harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.4 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.4
docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

mkdir imgbuild

mv installRunner.py DockerFile imgbuild

cd imagebuild

docker build -f DockerFile -t harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.4 .

docker image rm harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.4 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.4

docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

再试一次，发现kubekey会动harbor上的tag导致最后使用的不是替换的镜像

不加-a参数尝试apply installer，注意观察harbor上的tag是否正确

手动删除节点上的镜像，重试。ansible.log没起作用，后面再看

stdout中输出的确实是-vvvvv的内容了等着看最后结果

还有一些方法更完美：

~~在kuberkey修改harbor之前，apply~~ ~~yaml之前就手动去机器上crictl pull正确的镜像，这样检测到本地有的镜像就不会被错误的远程覆盖掉~~
~~分两次安装，第一次安装k8s，手动crictl pull，第二次安装kubesphere~~
~~追下kubekey代码，绕过校验直接用正确的版本号安装~~
最终根据源码https://github.com/kubesphere/kubekey/blob/e755baf67198d565689d7207378174f429b508ba/cmd/kk/cmd/create/cluster.go#L141C43-L141C59得知可通过--skip-push-images来避免harbor的镜像被覆盖

通过观察前面的详细输出，可以确认不是卡在开头修改的run方法中调用的ansible任务


Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
**************************************************
Waiting for all tasks to be completed ...

Start installing monitoring

Start installing multicluster

Start installing openpitrix

Start installing network

**************************************************

Waiting for all tasks to be completed ...

根据日志输出可找到实际的任务是https://github.com/kubesphere/ks-installer/blob/ef79beead3285698cdce559dd5505c79fe11dbff/controller/installRunner.py#L277C10-L277C20

component List


    readyToEnabledList = [
        'monitoring',
        'multicluster',
        'openpitrix',
        'network']

readyToEnabledList = [

'monitoring',

'multicluster',

'openpitrix',

'network']

根据代码可知执行的是https://github.com/kubesphere/ks-installer/blob/ef79beead3285698cdce559dd5505c79fe11dbff/playbooks/monitoring.yaml

实例化调用的是https://github.com/kubesphere/ks-installer/blob/ef79beead3285698cdce559dd5505c79fe11dbff/controller/installRunner.py#L57

参数来自https://github.com/kubesphere/ks-installer/blob/ef79beead3285698cdce559dd5505c79fe11dbff/controller/installRunner.py#L261

可以看到写死的quiet=True, https://github.com/kubesphere/ks-installer/blob/ef79beead3285698cdce559dd5505c79fe11dbff/controller/installRunner.py#L266C3-L266C24

接下来看看监控安装啥这么慢：

~~静态分析monitoring安装流程~~
~~继续修改这地方增加相同的verbose参数查看输出？异步调用ansible日志会输出到哪？~~
_~~output参数看起来无效，怎么调试。这个参数是给streaming pipeline用的~~
~~status_handler看起来能用~~

TODO kubekey为啥这4个使用异步调用ansible且不输出中间过程？

因quiet=True所以stdout看不到过程，通过查看当前ks-installer中的进程确定执行到哪了


ks-installer-566ffb8f44-zft9h:/hooks/kubesphere$ ps aux|more
PID   USER     TIME  COMMAND
    1 kubesphe  0:00 /shell-operator start
   18 kubesphe  5:49 python3 /hooks/kubesphere/installRunner.py
 2171 kubesphe  0:00 bash
 4053 kubesphe  1:54 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubesphere/config/ks-config.json -e @/kubesphere/config/ks-status.json -e @/kubesphere/results/env/extravars /kubesphere/playbooks/monitoring.yaml
 8876 kubesphe  0:00 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubesphere/config/ks-config.json -e @/kubesphere/config/ks-status.json -e @/kubesphere/results/env/extravars /kubesphere/playbooks/monitoring.yaml
 8899 kubesphe  0:00 /bin/sh -c /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700926548.0542264-8876-155728454178142/AnsiballZ_command.py && sleep 0
 8900 kubesphe  0:01 /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700926548.0542264-8876-155728454178142/AnsiballZ_command.py
 8926 kubesphe  0:00 /usr/local/bin/kubectl apply -f /kubesphere/kubesphere/prometheus/alertmanager
 8957 kubesphe  0:00 ps aux
 8958 kubesphe  0:00 more

ks-installer-566ffb8f44-zft9h:/hooks/kubesphere$ ps aux|more

PID USER TIME COMMAND

1 kubesphe 0:00 /shell-operator start

18 kubesphe 5:49 python3 /hooks/kubesphere/installRunner.py

2171 kubesphe 0:00 bash

4053 kubesphe 1:54 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubesphere/config/ks-config.json -e @/kubesphere/config/ks-status.json -e @/kubesphere/results/env/extravars /kubesphere/playbooks/monitoring.yaml

8876 kubesphe 0:00 {ansible-playboo} /usr/local/bin/python /usr/local/bin/ansible-playbook -e @/kubesphere/config/ks-config.json -e @/kubesphere/config/ks-status.json -e @/kubesphere/results/env/extravars /kubesphere/playbooks/monitoring.yaml

8899 kubesphe 0:00 /bin/sh -c /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700926548.0542264-8876-155728454178142/AnsiballZ_command.py && sleep 0

8900 kubesphe 0:01 /usr/local/bin/python /home/kubesphere/.ansible/tmp/ansible-tmp-1700926548.0542264-8876-155728454178142/AnsiballZ_command.py

8926 kubesphe 0:00 /usr/local/bin/kubectl apply -f /kubesphere/kubesphere/prometheus/alertmanager

8957 kubesphe 0:00 ps aux

8958 kubesphe 0:00 more

monitor的任务定义在https://github.com/kubesphere/ks-installer/blob/ef79beead3285698cdce559dd5505c79fe11dbff/roles/ks-monitor/tasks/main.yaml

通过watch -n 1 'ps aux|more'可以看到执行过程，其实没卡住就是安装的慢

根据https://github.com/ansible/ansible-runner/blob/e81b02cae85f7c3e402fcb1cc1512da5ee3bcf35/src/ansible_runner/interface.py#L228C28-L228C28可知返回值第一个是异步执行的线程，修改代码尝试重定向输出到日志文件

同时修改quiet = False


def generateTaskLists():
    readyToEnabledList, readyToDisableList = getComponentLists()
    tasksDict = {}
    for taskName in readyToEnabledList:
        playbookPath = os.path.join(playbookBasePath, str(taskName) + '.yaml')
        artifactDir = os.path.join(privateDataDir, str(taskName))
        if os.path.exists(artifactDir):
            shutil.rmtree(artifactDir)

        tasksDict[str(taskName)] = component(
            playbook=playbookPath,
            private_data_dir=privateDataDir,
            artifact_dir=artifactDir,
            ident=str(taskName),
            quiet=False,
            rotate_artifacts=1
        )

    return tasksDict
    

    def installRunner(self):
        installer = ansible_runner.run_async(
            playbook=self.playbook,
            private_data_dir=self.private_data_dir,
            artifact_dir=self.artifact_dir,
            ident=self.ident,
            quiet=self.quiet,
            rotate_artifacts=self.rotate_artifacts,
            verbosity=5
        )
        task_name = self.ident
        thread = installer[0]
        log_file = open('/tmp/'+task_name+'.debug.log', 'w')
        thread.stdout = log_file
        return installer[1]

def generateTaskLists():

readyToEnabledList, readyToDisableList = getComponentLists()

tasksDict = {}

for taskName in readyToEnabledList:

playbookPath = os.path.join(playbookBasePath, str(taskName) + '.yaml')

artifactDir = os.path.join(privateDataDir, str(taskName))

if os.path.exists(artifactDir):

shutil.rmtree(artifactDir)

tasksDict[str(taskName)] = component(

playbook=playbookPath,

private_data_dir=privateDataDir,

artifact_dir=artifactDir,

ident=str(taskName),

quiet=False,

rotate_artifacts=1

)

return tasksDict

def installRunner(self):

installer = ansible_runner.run_async(

playbook=self.playbook,

private_data_dir=self.private_data_dir,

artifact_dir=self.artifact_dir,

ident=self.ident,

quiet=self.quiet,

rotate_artifacts=self.rotate_artifacts,

verbosity=5

)

task_name = self.ident

thread = installer[0]

log_file = open('/tmp/'+task_name+'.debug.log', 'w')

thread.stdout = log_file

return installer[1]

最终修改对比


--- a/installRunner.py.bak
+++ b/installRunner.py
@@ -90,8 +90,13 @@ class component():
             artifact_dir=self.artifact_dir,
             ident=self.ident,
             quiet=self.quiet,
-            rotate_artifacts=self.rotate_artifacts
+            rotate_artifacts=self.rotate_artifacts,
+            verbosity=5
         )
+        task_name = self.ident
+        thread = installer[0]
+        log_file = open('/tmp/'+task_name+'.debug.log', 'w')
+        thread.stdout = log_file
         return installer[1]


@@ -263,7 +268,7 @@ def generateTaskLists():
             private_data_dir=privateDataDir,
             artifact_dir=artifactDir,
             ident=str(taskName),
-            quiet=True,
+            quiet=False,
             rotate_artifacts=1
         )

--- a/installRunner.py.bak

+++ b/installRunner.py

@@ -90,8 +90,13 @@ class component():

artifact_dir=self.artifact_dir,

ident=self.ident,

quiet=self.quiet,

- rotate_artifacts=self.rotate_artifacts

+ rotate_artifacts=self.rotate_artifacts,

+ verbosity=5

)

+ task_name = self.ident

+ thread = installer[0]

+ log_file = open('/tmp/'+task_name+'.debug.log', 'w')

+ thread.stdout = log_file

return installer[1]

@@ -263,7 +268,7 @@ def generateTaskLists():

private_data_dir=privateDataDir,

artifact_dir=artifactDir,

ident=str(taskName),

- quiet=True,

+ quiet=False,

rotate_artifacts=1

)

按最小修改打镜像


mkdir imgbuild
mv installRunner.py DockerFile imgbuild
cd imagebuild
docker build -f DockerFile -t harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.7 .
docker image rm harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.7 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.7
docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2
docker image ls --digests|grep installer

mkdir imgbuild

mv installRunner.py DockerFile imgbuild

cd imagebuild

docker build -f DockerFile -t harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.7 .

docker image rm harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker tag harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.7 harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.7

docker push harbor.hylstudio.local/kubesphereio/ks-installer:v3.3.2

docker image ls --digests|grep installer

强制删除kubesphere monitor的namespace，强制删除ks-installer这个pod和宿主机的installer镜像，迫使k8s重建pod，这时候会按harbor的镜像tag重新拉取

harbor看不到imageId，Image ID是docker客户端侧的概念，所以仓库不管= =

https://github.com/goharbor/harbor/issues/10293

https://github.com/goharbor/harbor/issues/2469

~~下次安装在kubekey apply~~ ~~installer之后，手动去edit镜像为指定的tag，其他的不动了~~

修改thread.stdout = log_file并不会输出，根据https://github.com/ansible/ansible-runner/blob/e0371d634426dfbdb9d3bfacb20e2dd4b039b499/src/ansible_runner/runner.py#L155C28-L155C48

可知self.config.suppress_output_file为假才会输出到文件self.config.artifact_dir的/stdout和stderr中，artifact_dir来自/kubesphere/results/{task_name}，suppress_output_file和suppress_ansible_output默认值也都是False

增加--skip-push-images，修改quite=False后去/kubesphere/results下查看是否有stdout和stderr


%s/quiet=True/quiet=False/g

0 1	%s/quiet=True/quiet=False/g


--- a/installRunner.py.bak
+++ b/installRunner.py
@@ -85,6 +85,7 @@ class component():

     def installRunner(self):
         installer = ansible_runner.run_async(
+            verbosity=5,
             playbook=self.playbook,
             private_data_dir=self.private_data_dir,
             artifact_dir=self.artifact_dir,
@@ -263,7 +264,7 @@ def generateTaskLists():
             private_data_dir=privateDataDir,
             artifact_dir=artifactDir,
             ident=str(taskName),
-            quiet=True,
+            quiet=False,
             rotate_artifacts=1
         )

@@ -341,6 +342,7 @@ def preInstallTasks():

     for task, paths in preInstallTasks.items():
         pretask = ansible_runner.run(
+            verbosity=5,
             playbook=paths[0],
             private_data_dir=privateDataDir,
             artifact_dir=paths[1],
@@ -353,11 +355,12 @@ def preInstallTasks():

 def resultInfo(resultState=False, api=None):
     ks_config = ansible_runner.run(
+        verbosity=5,
         playbook=os.path.join(playbookBasePath, 'ks-config.yaml'),
         private_data_dir=privateDataDir,
         artifact_dir=os.path.join(privateDataDir, 'ks-config'),
         ident='ks-config',
-        quiet=True
+        quiet=False
     )

     if ks_config.rc != 0:
@@ -365,11 +368,12 @@ def resultInfo(resultState=False, api=None):
         exit()
         
     result = ansible_runner.run(
+        verbosity=5,
         playbook=os.path.join(playbookBasePath, 'result-info.yaml'),
         private_data_dir=privateDataDir,
         artifact_dir=os.path.join(privateDataDir, 'result-info'),
         ident='result',
-        quiet=True
+        quiet=False
     )

     if result.rc != 0:
@@ -380,6 +384,7 @@ def resultInfo(resultState=False, api=None):

     if "migration" in resource['status']['core'] and resource['status']['core']['migration'] and resultState == False:
         migration = ansible_runner.run(
+            verbosity=5,
             playbook=os.path.join(playbookBasePath, 'ks-migration.yaml'),
             private_data_dir=privateDataDir,
             artifact_dir=os.path.join(privateDataDir, 'ks-migration'),
@@ -395,11 +400,12 @@ def resultInfo(resultState=False, api=None):
             logging.info(info)

     telemeter = ansible_runner.run(
+        verbosity=5,
         playbook=os.path.join(playbookBasePath, 'telemetry.yaml'),
         private_data_dir=privateDataDir,
         artifact_dir=os.path.join(privateDataDir, 'telemetry'),
         ident='telemetry',
-        quiet=True
+        quiet=False
     )

     if telemeter.rc != 0:

--- a/installRunner.py.bak

+++ b/installRunner.py

@@ -85,6 +85,7 @@ class component():

def installRunner(self):

installer = ansible_runner.run_async(

+ verbosity=5,

playbook=self.playbook,

private_data_dir=self.private_data_dir,

artifact_dir=self.artifact_dir,

@@ -263,7 +264,7 @@ def generateTaskLists():

private_data_dir=privateDataDir,

artifact_dir=artifactDir,

ident=str(taskName),

- quiet=True,

+ quiet=False,

rotate_artifacts=1

)

@@ -341,6 +342,7 @@ def preInstallTasks():

for task, paths in preInstallTasks.items():

pretask = ansible_runner.run(

+ verbosity=5,

playbook=paths[0],

private_data_dir=privateDataDir,

artifact_dir=paths[1],

@@ -353,11 +355,12 @@ def preInstallTasks():

def resultInfo(resultState=False, api=None):

ks_config = ansible_runner.run(

+ verbosity=5,

playbook=os.path.join(playbookBasePath, 'ks-config.yaml'),

private_data_dir=privateDataDir,

artifact_dir=os.path.join(privateDataDir, 'ks-config'),

ident='ks-config',

- quiet=True

+ quiet=False

)

if ks_config.rc != 0:

@@ -365,11 +368,12 @@ def resultInfo(resultState=False, api=None):

exit()

result = ansible_runner.run(

+ verbosity=5,

playbook=os.path.join(playbookBasePath, 'result-info.yaml'),

private_data_dir=privateDataDir,

artifact_dir=os.path.join(privateDataDir, 'result-info'),

ident='result',

- quiet=True

+ quiet=False

)

if result.rc != 0:

@@ -380,6 +384,7 @@ def resultInfo(resultState=False, api=None):

if "migration" in resource['status']['core'] and resource['status']['core']['migration'] and resultState == False:

migration = ansible_runner.run(

+ verbosity=5,

playbook=os.path.join(playbookBasePath, 'ks-migration.yaml'),

private_data_dir=privateDataDir,

artifact_dir=os.path.join(privateDataDir, 'ks-migration'),

@@ -395,11 +400,12 @@ def resultInfo(resultState=False, api=None):

logging.info(info)

telemeter = ansible_runner.run(

+ verbosity=5,

playbook=os.path.join(playbookBasePath, 'telemetry.yaml'),

private_data_dir=privateDataDir,

artifact_dir=os.path.join(privateDataDir, 'telemetry'),

ident='telemetry',

- quiet=True

+ quiet=False

)

if telemeter.rc != 0:

可到/kubesphere/results看到当前执行任务的实时日志


drwxr-xr-x    3 kubesphe kubesphe    4.0K Dec  2 13:37 common
drwxr-xr-x    1 kubesphe kubesphe    4.0K Feb  3  2023 env
drwxr-xr-x    3 kubesphe kubesphe    4.0K Dec  2 13:41 ks-core
drwxr-xr-x    3 kubesphe kubesphe    4.0K Dec  2 13:37 metrics_server
drwxr-xr-x    3 kubesphe kubesphe    4.0K Dec  2 13:37 preinstall

drwxr-xr-x 3 kubesphe kubesphe 4.0K Dec 2 13:37 common

drwxr-xr-x 1 kubesphe kubesphe 4.0K Feb 3 2023 env

drwxr-xr-x 3 kubesphe kubesphe 4.0K Dec 2 13:41 ks-core

drwxr-xr-x 3 kubesphe kubesphe 4.0K Dec 2 13:37 metrics_server

drwxr-xr-x 3 kubesphe kubesphe 4.0K Dec 2 13:37 preinstall

综上所述，当kuberkey出现Please wait for the installation to complete时，控制权是交给了ks-installer


namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
13:35:26 UTC success: [k8s1]
Please wait for the installation to complete:    >>--->

namespace/kubesphere-system unchanged

serviceaccount/ks-installer unchanged

customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged

clusterrole.rbac.authorization.k8s.io/ks-installer unchanged

clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged

deployment.apps/ks-installer unchanged

clusterconfiguration.installer.kubesphere.io/ks-installer created

13:35:26 UTC success: [k8s1]

Please wait for the installation to complete: >>--->

当ks-installer出现这样的字样时，可到/kubesphere/results继续查看剩余4个并行执行的任务状态


Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
**************************************************
Waiting for all tasks to be completed ...

Start installing monitoring

Start installing multicluster

Start installing openpitrix

Start installing network

**************************************************

Waiting for all tasks to be completed ...

按之前的速度，/kubesphere/results/monitoring/monitoring是最慢的，着重观察这个。手动观察剩下三个都已经成功了，只剩下monitoring还在安装

突然发现安装过程居然也有helm，简直是究极套娃，技术栈极其复杂


PLAY RECAP *********************************************************************
localhost                  : ok=24   changed=22   unreachable=0    failed=0    skipped=24   rescued=0    ignored=0

task monitoring status is successful  (4/4)

PLAY RECAP *********************************************************************

localhost : ok=24 changed=22 unreachable=0 failed=0 skipped=24 rescued=0 ignored=0

task monitoring status is successful (4/4)

monitoring这么复杂的任务，不知道官方出于什么理由彻底隐藏了安装过程，不给任何进度提示。在monitoring安装完成后，才出现了下面kubesphere-config的apply操作，所以在这之前kubesphere的webui是无法访问的，因为这里包含了重要的jwttoken来访问kubesphere api，容器会一直报错等待这个configmap的初始化，至此曾经的疑惑均完成了解答


changed: [localhost] => (item=kubesphere-config.yaml) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/kubesphere-config.yaml",
    "delta": "0:00:07.874273",
    "end": "2023-12-02 14:09:16.591793",
    "failed_when_result": false,
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/kubesphere-config.yaml",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "item": "kubesphere-config.yaml",
    "rc": 0,
    "start": "2023-12-02 14:09:08.717520",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "configmap/kubesphere-config created",
    "stdout_lines": [
        "configmap/kubesphere-config created"
    ]
}

changed: [localhost] => (item=kubesphere-config.yaml) => {

"ansible_loop_var": "item",

"changed": true,

"cmd": "/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/kubesphere-config.yaml",

"delta": "0:00:07.874273",

"end": "2023-12-02 14:09:16.591793",

"failed_when_result": false,

"invocation": {

"module_args": {

"_raw_params": "/usr/local/bin/kubectl apply -f /kubesphere/kubesphere/kubesphere-config.yaml",

"_uses_shell": true,

"argv": null,

"chdir": null,

"creates": null,

"executable": null,

"removes": null,

"stdin": null,

"stdin_add_newline": true,

"strip_empty_ends": true,

"warn": true

}

"item": "kubesphere-config.yaml",

"rc": 0,

"start": "2023-12-02 14:09:08.717520",

"stderr": "",

"stderr_lines": [],

"stdout": "configmap/kubesphere-config created",

"stdout_lines": [

"configmap/kubesphere-config created"

]

}

TODO 收集results文件夹下所有event的json画散点图观察

TODO 给kubekey增加参数，打印详细的输出。读取参数后参考上面的代码修改即可

TODO 监控对kubesphere ui的启动应该不是强依赖，梳理依赖关系增加参数来支持异步安装监控，提前让kubesphere可用。需要读ansible脚本确认。根据日志显示telemetry的安装就是完全独立于kubesphere的安装，可以参考

20231003DevOps相关特性总结

0.背景

总结一些无需保密的经验作为总结

20230610企业级web开发特性总结

1.DevOps相关特性总结

监听代码变动
打包环境容器化
平滑关机
健康检查
流量路由
多环境打包

配置中心

一般每个独立动作称作action，一次独立完整的流程称作pipeline，一条pipeline由多个action编排组成

pipeline可人为指定触发条件，可以选择手动触发、定时触发、git事件触发等

action之间可串行并行随意组合，action节点可选择自动执行、定时执行、人工执行、人工二次确认等选项。action节点的类型不同平台略有差异。

2.代码变动监听

代码变动监听一般使用git hooks，配置在git server上，可以监听包括但不限于push、merge requests、tag等动作事件并回调给指定的api

一般配合分支和tag规范来共同管理开发到上线的全流程，例如feature分支->dev分支->master分支三层结构或更细化的多层分支结构。在不同类型的分支上触发的pipeline有所区别，例如feature分支可能额外触发代码检查pipeline流程，dev分支额外触发单元测试pipeline流程，master分支或tag触发上线pipeline流程。

3.打包环境容器化

打包的环境通常不在开发机而是专用的打包机，打包机的环境除非特殊需求一般使用docker容器通过镜像统一管理打包环境，一般会选择包含docker环境的docker作为打包基础环境，在此基础上定制各种打包的环境依赖、调整打包依赖的缓存文件夹共享策略，以此来保证打包环境的稳定和可控。通常还会配套使用私有镜像中心（如docker registry）和私有包管理中心（如jfrog）来加速打包过程排除网络的影响

4.平滑关机

对于重要的服务不仅需要滚动重启，还必须实现平滑关机来保证服务滚动重启的过程中的稳定性，一般思路是程序自行实现平滑关机流程并允许通过监听SIGTERM、SIGKILL等信号或单独实现http或rpc接口触发。在上线的pipeline或控制器或上级调度程序提前调整网络流量后，再通知对应的节点执行平滑关机流程并自行退出，在一定时间超时后强行杀死，因此通常配合健康检查和流量路由使用。

5.健康检查

健康检查一般也由应用程序自行实现，可分为可达性检查和可用性检查，其中可达性检查主要用于探测网络链路是否通畅，不关心程序本身是否可用，可用性检查在可达性检查的基础还必须满足可用的定义。

可用的定义在不同的场景下定义不同，对web服务来说比较简单的定义就是可接收http请求并成功回应，再严格一点的是要同时满足依赖的数据库服务、缓存服务、队列服务同时可用，但严格的定义会导致底层服务故障时无法让带有降级逻辑的程序执行降级导致服务彻底不可用，因此一般不轻易采用，而是让程序自行选择是执行降级逻辑还是直接认为服务不可用。

6.流量路由

20230725接入层网络总结

流量路由主要负责处理流量的流向，上一篇总结的路径上的流量通常被称作是南北向流量或者纵向流量，应用程序之间的通信通常被称作东西向流量或横向流量

对某个服务集群来说一般通过网络层路由、客户端或服务器服务发现、负载均衡来完成精确的控制流量流向。特别的对于云原生的服务网格来说主要处理的是东西向流量。

7.多环境打包

针对同一份代码需要运行在多个数据中心、多个公司、多个环境，因此需要通过某种方式来区分运行环境。可选策略一般有几个思路，一种是用同一份产物通过配置中心、配置文件、环境变量等做出区分来改变程序能感知的环境，另一种是打包时根据需要直接按不同环境组合生成多种环境运行的产物。

对于使用配置中心、配置文件、环境变量来处理的方案，需要额外注意启动时候的传入参数和配置中心的设置，方便集中管理但很容易出意外

对于使用直接打包多种产物来处理的方案，虽然不用担心环境参数错误，但需要额外考虑环境参数的保密问题也就是产物的保密问题

20230725接入层网络总结

0.背景

总结一些无需保密的经验作为总结

20230610企业级web开发特性总结

1.接入层网络

一般web系统都是由接入层网络+业务层网络组成，从用户客户端到业务曾服务的入口这段流量路径通常是人们所说的接入层网络

网络层这中间通常会包括：DNS、X层负载均衡

物理链路上通常会包括三大运营商接入网络、小运营商接入网络、机房间专线、核心交换机互联

相关技术包括虚拟IP、4层/7层网络报文解析等

2.相关总结

一个典型的wen服务接入层网络通常是由 DNS系统、4层负载均衡、7层负载均衡组成

无论是哪层，核心思想都是把多个一样的东西对外表现的”像是”一个，理解了这个就能理解接入层网络的大部分逻辑

在DNS层，DNS可以做到分地区、分运营商进行多IP解析、主备策略、BGP机房策略、健康检查、灾备切换

在网络协议栈第4层，可以将多个IP+端口对外当成一个IP+端口使用，多个IP一般叫做L4RS，一个IP一般叫做L4VIP

当然这一层也会包含负载均衡的策略、健康检查、灾备自动切换等功能

在网络协议栈第7曾，可以将多个host+端口对外当成一个host+端口，host的所在一般叫虚拟主机，多个host+端口一般称作L7RS或serviceIP或RealIP

当然这一层也会包含负载均衡策略、健康检查、灾备自动切换

还有一种特殊流程是L4VIP直通serviceIPs，对于IM类特殊应用就不会都使用7层负载均衡

典型拓扑图如下，敏感信息已虚化处理

作者： 956237586

20241005projector新feature

背景

打包内置静态资源

支持输入框修改token

TODO 动态生成PWA的manifest

TODO

20241003kasm idea web workspace

背景

实现细节

后续

20241002continue在projector环境下的加载和相关问题

20231202argocd搭建试玩

0.背景

1.安装

2.kubectl配置转发

3.本地命令行配置

4.first demo

参考

20231125ks-installer追踪

0.背景

1.追踪过程

20231003DevOps相关特性总结

0.背景

1.DevOps相关特性总结

2.代码变动监听

3.打包环境容器化

4.平滑关机

5.健康检查

6.流量路由

7.多环境打包

20230725接入层网络总结

0.背景

1.接入层网络

2.相关总结