k8s集群快速安装及证书更新

1.环境配置

四台服务器 安装克隆后修改ip uuid,mac 等信息

192.168.83.129  master  centos 7.9
192.168.83.130  node1
192.168.83.131  node2
192.168.83.132  client             nfs和kuboard-spray

2.安装docker和kuboard-spray

在client上安装

1.切换阿里源

cp /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
sudo yum clean all
sudo yum makecache
sudo yum repolist

2.安装docker

## 安装Docker所需要的一些工具包
sudo yum install -y yum-utils
 
## 建立Docker仓库 (映射仓库地址)
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install docker-ce docker-ce-cli containerd.io

sudo systemctl start docker

sudo systemctl enable docker

3.安装kuboard-spray

基于 kubespray (opens new window)提供图形化的 K8S 集群离线安装、维护工具。

KuboardSpray | Kuboard Spray

docker run -d \
  --restart=unless-stopped \
  --name=kuboard-spray \
  -p 9988:80/tcp \
  -e TZ=Asia/Shanghai \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v ~/kuboard-spray-data:/data \
  swr.cn-east-2.myhuaweicloud.com/kuboard/kuboard-spray:v1.2.0
  
在浏览器打开地址 http://这台机器的IP，输入用户名 admin，默认密码 Kuboard123，即可登录 Kuboard-Spray 界面

安装出现的问题

FAILED - RETRYING: [node1]: download_file | Copy file from cache to nodes, if it is available (4 retries left).
FAILED - RETRYING: [node2]: download_file | Copy file from cache to nodes, if it is available (4 retries left).
FAILED - RETRYING: [master]: download_file | Copy file from cache to nodes, if it is available (4 retries left).
FAILED - RETRYING: [node1]: download_file | Copy file from cache to nodes, if it is available (3 retries left).
FAILED - RETRYING: [node2]: download_file | Copy file from cache to nodes, if it is available (3 retries left).
FAILED - RETRYING: [master]: download_file | Copy file from cache to nodes, if it is available (3 retries left).
FAILED - RETRYING: [node2]: download_file | Copy file from cache to nodes, if it is available (2 retries left).
FAILED - RETRYING: [node1]: download_file | Copy file from cache to nodes, if it is available (2 retries left).
FAILED - RETRYING: [master]: download_file | Copy file from cache to nodes, if it is available (2 retries left).
FAILED - RETRYING: [node2]: download_file | Copy file from cache to nodes, if it is available (1 retries left).
FAILED - RETRYING: [node1]: download_file | Copy file from cache to nodes, if it is available (1 retries left).
FAILED - RETRYING: [master]: download_file | Copy file from cache to nodes, if it is available (1 retries left).

TASK [container-engine/runc : download_file | Copy file from cache to nodes, if it is available] ***
fatal: [node2]: FAILED! => {
    "attempts": 4,
    "changed": false,
    "cmd": "sshpass -d3 /usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=3 -o UserKnownHostsFile=/dev/null -F /dev/null -o ControlMaster=auto -o ControlPersist=6********m -o ControlPath=/data/cluster/kubility/root@*****-22 ' --out-format='<<CHANGED>>%i %n%L' /data/resource/spray-v2.18.********a-8_k8s-v1.23.5_v1.11-amd64/content/kubespray_cache/runc root@192.168.83.131:/tmp/releases/runc",
    "msg": "",
    "rc": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
    "stdout_lines": []
}
fatal: [node1]: FAILED! => {
    "attempts": 4,
    "changed": false,
    "cmd": "sshpass -d3 /usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=3 -o UserKnownHostsFile=/dev/null -F /dev/null -o ControlMaster=auto -o ControlPersist=6********m -o ControlPath=/data/cluster/kubility/root@********-22 ' --out-format='<<CHANGED>>%i %n%L' /data/resource/spray-v2.18.********a-8_k8s-v1.23.5_v1.11-amd64/content/kubespray_cache/runc root@********:/tmp/releases/runc",
    "msg": "",
    "rc": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
    "stdout_lines": []
}
fatal: [master]: FAILED! => {
    "attempts": 4,
    "changed": false,
    "cmd": "sshpass -d3 /usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=3 -o UserKnownHostsFile=/dev/null -F /dev/null -o ControlMaster=auto -o ControlPersist=6********m -o ControlPath=/data/cluster/kubility/root@****-22 ' --out-format='<<CHANGED>>%i %n%L' /data/resource/spray-v2.18.********a-8_k8s-v1.23.5_v1.11-amd64/content/kubespray_cache/runc root@***:/tmp/releases/runc",
    "msg": "",
    "rc": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
    "stdout_lines": []
}

对比了下同版本下的

/root/kuboard-spray-data/resource/spray-v2.18.0a-8_k8s-v1.23.5_v1.11-amd64/content/3rd/kubespray/roles/download/tasks/download_file.yml

在大概位置:download_file | Copy file from cache to nodes, if it is available
这里添加了个 ignore_errors: true.

3.安装k8s相关的工具

1.安装kuboard

安装 Kubernetes 多集群管理工具 - Kuboard v3 | Kuboard

docker run -d \
  --restart=unless-stopped \
  --name=kuboard \
  -p 9999:80/tcp \
  -p 10081:10081/tcp \
  -e KUBOARD_ENDPOINT="http://192.168.83.132:9999" \
  -e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
  -v /root/kuboard-data:/data \
  swr.cn-east-2.myhuaweicloud.com/kuboard/kuboard:v3

2.安装helm3

Helm | Installing Helm

2.1.版本对应

Helm 版本	支持的 Kubernetes 版本
3.12.x	1.27.x - 1.24.x
3.11.x	1.26.x - 1.23.x
3.10.x	1.25.x - 1.22.x
3.9.x	1.24.x - 1.21.x
3.8.x	1.23.x - 1.20.x
3.7.x	1.22.x - 1.19.x
3.6.x	1.21.x - 1.18.x

2.2.下载配置

wget -O https://get.helm.sh/helm-v3.12.3-linux-amd64.tar.gz

tar -zxvf helm-v3.12.3-linux-amd64.tar.gz

cp -a linux-amd64/helm /usr/bin/helm

2.3.添加国内源

添加国内源
helm repo add stable http://mirror.azure.cn/kubernetes/charts
helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
helm repo update
helm repo list
helm search repo stable

--安装ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm pull ingress-nginx/ingress-nginx
tar xf ingress-nginx-4.12.0.tgz
kubectl taint node k8s-master node-role.kubernetes.io/master-

4.kubectl

1.强制删除pod ,pv ,pvc

kubectl delete pods <pod> --grace-period=0 --force

kubectl patch pv pvc-3c7317ee-aa88-45a4-ba22-631c8c98fb2a  -p '{"metadata":{"finalizers":null}}'

kubectl patch pvc redis-data-test-nn-1 -p '{"metadata":{"finalizers":null}}'

5.Contained

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
            endpoint = ["https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
            endpoint = ["https://gcr.mirrors.ustc.edu.cn"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
            endpoint = ["https://gcr.mirrors.ustc.edu.cn/google-containers/"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
            endpoint = ["https://quay.mirrors.ustc.edu.cn"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.83.129:30012"]
            endpoint = ["http://192.168.83.129:30012/"]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.83.129:30012"]  
          [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.83.129:30012".tls]
            insecure_skip_verify = true  
          [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.83.129:30012".auth]
            username = "admin"  
            password = "Harbor12345"


$ systemctl daemon-reload && systemctl restart containerd

6.更新证书99年

root@k8s-master:~# kubectl get nodes
The connection to the server <master>:6443 was refused - did you specify the right host or port?

1.查看当前证书时间

kubeadm certs check-expiration

2.下载源码

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
git checkout v1.25.9 # 根据自己的版本切换
在使用git切换分支的时候报错：Your local changes to the following files would be overwritten by checkout：解决方法
切换到报错目录下 删除报错的文件

3.修改证书时间

vi cmd/kubeadm/app/constants/constants.go，找到CertificateValidity，修改如下

CertificateValidity = time.Hour * 24 * 365 * 100

 // KubernetesDir is the directory Kubernetes owns for storing various configuration files
        KubernetesDir = "/etc/kubernetes"
        // ManifestsSubDirName defines directory name to store manifests
        ManifestsSubDirName = "manifests"
        // TempDirForKubeadm defines temporary directory for kubeadm
        // should be joined with KubernetesDir.
        TempDirForKubeadm = "tmp"

        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        CertificateValidity = time.Hour * 24 * 365 * 100

        // DefaultCertificateDir defines default certificate directory
        DefaultCertificateDir = "pki"

        // CACertAndKeyBaseName defines certificate authority base name
        CACertAndKeyBaseName = "ca"
        // CACertName defines certificate name
        CACertName = "ca.crt"
        // CAKeyName defines certificate name
        CAKeyName = "ca.key"

vi staging/src/k8s.io/client-go/util/cert/cert.go 修改了,但是好像没管用

NotAfter:              now.Add(duration365d * 100).UTC(),

func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
        now := time.Now()
        tmpl := x509.Certificate{
                SerialNumber: new(big.Int).SetInt64(0),
                Subject: pkix.Name{
                        CommonName:   cfg.CommonName,
                        Organization: cfg.Organization,
                },
                DNSNames:              []string{cfg.CommonName},
                NotBefore:             now.UTC(),
                NotAfter:              now.Add(duration365d * 100).UTC(),
                KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
                BasicConstraintsValid: true,
                IsCA:                  true,
        }

        certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
        if err != nil {
                return nil, err
        }
        return x509.ParseCertificate(certDERBytes)
}

4.安装编译软件

1.安装编译工具

yum -y install gcc automake autoconf libtool make  rsync jq

2.下载并配置 Golang 环境

--查看一下go版本
[root@client ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:16:51Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:08:36Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
## 下载  go1.19.8
$ wget https://dl.google.com/go/go1.19.8.linux-amd64.tar.gz
## 解压并放置在一个目录中
$ tar zxvf go1.19.8.linux-amd64.tar.gz  -C /usr/local
## 编辑 /etc/profile 文件,添加 Go 环境配置内容
$ vi /etc/profile

export GOROOT=/usr/local/go
export GOPATH=/usr/local/gopath
export PATH=$PATH:$GOROOT/bin
## 使配置生效
$ source /etc/profile

## 测试 Go 命令是否配置成功
$ go version

3.编译kubeadm

make all WHAT=cmd/kubeadm GOFLAGS=-v

编译成功后位置

_output/local/bin/linux/amd64

查看自己的kubeadm在哪里 备份然后覆盖生成的kubeadm 在所有节点上也替换原有版本

$ which kubeadm
/usr/local/bin/kubeadm

$ mv /usr/local/bin/kubeadm /usr/local/bin/kubeadm_backup
$ cp _output/local/bin/linux/amd64/kubeadm /usr/local/bin/kubeadm
$ chmod +x /usr/local/bin/kubeadm

备份原来的证书

cp -r /etc/kubernetes/pki{,.bak2025}

5.重新生成证书

kubeadm certs renew all

重启kube-apiserver, kube-controller-manager, kube-scheduler and etcd

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

$ kubectl delete pod  kube-apiserver-k8s-master -n kube-system

$ kubectl delete pod  kube-controller-manager-k8s-master -n kube-system

$ kubectl delete pod  kube-scheduler-k8s-master -n kube-system

$ systemctl stop etcd  
$ systemctl start etcd  
$ systemctl status etcd