华北Kubernetes集群部署记录(多master)

master节点信息： d5-dui-master-001 10.41.225.129 [运维] d5-dui-master-002 10.40.150.141 [运维] d5-dui-master-003 10.41.225.130 [运维]

创建kube-apiserver的负载均衡器SLB

只能选择4层负载均衡器 d5-k8s-apiserver/10.40.150.145 添加TCP监听 TCP/8443 添加后端服务器，后端服务器端口6443。

部署ansible

[root@d5-dui-master-001 ~]# yum install ansible -y
[root@d5-dui-master-001 ~]# cat > /etc/ansible/hosts <<EOF
[k8s-all:children]
k8s-masters
k8s-workers

[k8s-masters]
d5-dui-master-001 ansible_host=10.41.225.129 ansible_port=5837 ansible_user=root
d5-dui-master-002 ansible_host=10.40.150.141 ansible_port=5837 ansible_user=root
d5-dui-master-003 ansible_host=10.41.225.130 ansible_port=5837 ansible_user=root

[k8s-workers]

EOF
[root@d5-dui-master-001 ~]# ansible k8s-all --list
  hosts (3):
    d5-dui-master-001
    d5-dui-master-002
    d5-dui-master-003
[root@d5-dui-master-001 ~]# ansible k8s-all -m yum -a "name=* state=latest"

修改hosts文件：

[root@d5-dui-master-001 ~]# cat > /etc/hosts <<EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.40.150.140  docker.v2.aispeech.com
10.41.225.129  d5-dui-master-001
10.40.150.141  d5-dui-master-002
10.41.225.130  d5-dui-master-003
EOF

配置SSH无密码连接：

[root@d5-dui-master-001 ~]# ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/id_rsa
[root@d5-dui-master-001 ~]# cat .ssh/id_rsa.pub >> .ssh/authorized_keys 
# 其他节点同理
[root@d5-dui-master-001 ~]# ansible k8s-masters -m ping
/usr/lib/python2.7/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)
d5-dui-master-003 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
d5-dui-master-002 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
d5-dui-master-001 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
[root@d5-dui-master-001 ~]# pip install requests

清理老版本的docker、Kubernetes软件包，yum update更新系统版本：

[root@d5-dui-master-001 ~]# ansible k8s-masters -m copy -a "src=/etc/hosts dest=/etc/hosts"
[root@d5-dui-master-001 ~]# rpm -qa|grep -e docker -e kube
docker-engine-17.05.0.ce-1.el7.centos.x86_64
kubectl-1.6.1-0.x86_64
kubeadm-1.6.1-0.x86_64
docker-engine-selinux-17.05.0.ce-1.el7.centos.noarch
kubelet-1.6.1-0.x86_64
nvidia-docker-1.0.1-1.x86_64
kubernetes-cni-0.5.1-0.x86_64
[root@d5-dui-master-001 ~]# ansible k8s-all -m yum -a "name=docker-engine-17.05.0.ce-1.el7.centos.x86_64,kubectl-1.6.1-0.x86_64,kubeadm-1.6.1-0.x86_64,docker-engine-selinux-17.05.0.ce-1.el7.centos.noarch,kubelet-1.6.1-0.x86_64,nvidia-docker-1.0.1-1.x86_64,kubernetes-cni-0.5.1-0.x86_64 state=absent"
[root@d5-dui-master-001 ~]# ansible k8s-masters -m yum -a "name=* state=latest"

创建CA证书和密钥

Kubernetes共有三个CA证书中心，etcd-ca、

安装cfssl工具集

[root@d5-dui-master-001 kubernetes]# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o cfssl
[root@d5-dui-master-001 kubernetes]# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o cfssljson
[root@d5-dui-master-001 kubernetes]# curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o cfssl-certinfo
[root@d5-dui-master-001 kubernetes]# chmod +x cfssl* && mv cfssl* /usr/local/bin/

生成CA配置文件

参考cfssl print-defaults config的格式，生成CA配置文件：

[root@d5-dui-master-001 kubernetes]# mkdir cert && cat > cert/ca-config.json <<EOF
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "kubernetes": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF

signing：表示该证书可用于签名其它证书，生成的 ca.pem 证书中 CA=TRUE；
server auth：表示 client 可以用该该证书对 server 提供的证书进行验证；
client auth：表示 server 可以用该该证书对 client 提供的证书进行验证。

创建etcd-ca证书

证书签名请求文件：

[root@d5-dui-master-001 kubernetes]# cat > cert/etcd-ca-csr.json <<EOF
{
    "CN": "etcd-ca",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "k8s",
            "OU": "aispeech"
        }
    ]
}
EOF

生成etcd-ca证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert -initca cert/etcd-ca-csr.json |cfssljson -bare cert/etcd-ca
2019/05/15 23:45:49 [INFO] generating a new CA key and certificate from CSR
2019/05/15 23:45:49 [INFO] generate received request
2019/05/15 23:45:49 [INFO] received CSR
2019/05/15 23:45:49 [INFO] generating key: rsa-2048
2019/05/15 23:45:50 [INFO] encoded CSR
2019/05/15 23:45:50 [INFO] signed certificate with serial number 708202704218904529399473200337703060460851404501

检查证书信息：

[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/etcd-ca.pem |head -n 11
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7c:0c:e8:57:1d:b0:7b:f6:d7:44:fc:39:e8:5f:64:d3:39:4f:9e:d5
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=etcd-ca
        Validity
            Not Before: May 15 15:41:00 2019 GMT
            Not After : May 13 15:41:00 2024 GMT
        Subject: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=etcd-ca

创建kubernetes-ca证书

证书签名请求文件：

[root@d5-dui-master-001 kubernetes]# cat > cert/ca-csr.json <<EOF
{
    "CN": "kubernetes-ca",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "k8s",
            "OU": "aispeech"
        }
    ]
}
EOF

生成etcd-ca证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert -initca cert/ca-csr.json |cfssljson -bare cert/ca
2019/05/16 00:50:53 [INFO] generating a new CA key and certificate from CSR
2019/05/16 00:50:53 [INFO] generate received request
2019/05/16 00:50:53 [INFO] received CSR
2019/05/16 00:50:53 [INFO] generating key: rsa-2048
2019/05/16 00:50:54 [INFO] encoded CSR
2019/05/16 00:50:54 [INFO] signed certificate with serial number 51833711093873951873764863377897167651873412942

检查证书信息：

[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/ca.pem |head -n 11
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            09:14:4d:d2:6e:b8:de:fb:37:c0:d1:f0:87:02:33:52:3d:86:d7:4e
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kubernetes-ca
        Validity
            Not Before: May 15 16:46:00 2019 GMT
            Not After : May 13 16:46:00 2024 GMT
        Subject: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kubernetes-ca

创建kubernetes-front-proxy-ca证书

证书签名请求文件：

[root@d5-dui-master-001 kubernetes]# cat > cert/front-proxy-ca-csr.json <<EOF
{
    "CN": "kubernetes-front-proxy-ca",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "k8s",
            "OU": "aispeech"
        }
    ]
}
EOF

生成etcd-ca证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert -initca cert/front-proxy-ca-csr.json |cfssljson -bare cert/front-proxy-ca
2019/05/16 01:31:13 [INFO] generating a new CA key and certificate from CSR
2019/05/16 01:31:13 [INFO] generate received request
2019/05/16 01:31:13 [INFO] received CSR
2019/05/16 01:31:13 [INFO] generating key: rsa-2048
2019/05/16 01:31:13 [INFO] encoded CSR
2019/05/16 01:31:13 [INFO] signed certificate with serial number 582881028555629213930575098762998236798490763443

检查证书信息：

[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/front-proxy-ca.pem |head -n 11
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            66:19:4a:0e:ab:eb:c7:d0:1c:d2:d5:8f:64:d6:47:bc:77:ab:e0:b3
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kubernetes-front-proxy-ca
        Validity
            Not Before: May 15 17:26:00 2019 GMT
            Not After : May 13 17:26:00 2024 GMT
        Subject: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kubernetes-front-proxy-ca

创建etcd相关的证书和私钥

创建etcd server证书和私钥

创建证书签名请求：

[root@d5-dui-master-001 kubernetes]# cat > cert/kube-etcd-csr.json <<EOF
{
    "CN": "kube-etcd",
    "hosts": [
        "localhost",
        "127.0.0.1",
        "10.41.225.129",
        "10.40.150.141",
        "10.41.225.130"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "k8s",
            "OU": "aispeech"
        }
    ]
}
EOF

hosts 字段指定授权使用该证书的etcd节点IP或域名列表，这里将etcd集群的三个节点IP都列在其中。

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
    -ca=cert/etcd-ca.pem \
    -ca-key=cert/etcd-ca-key.pem \
    -config=cert/ca-config.json \
    -profile=kubernetes \
    cert/kube-etcd-csr.json | cfssljson -bare cert/kube-etcd

检查证书信息：

[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/kube-etcd.pem |head -n 11
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            48:a6:4f:12:b8:b7:6e:ed:a0:fa:f6:1d:d7:65:47:44:2a:1d:67:1e
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=etcd-ca
        Validity
            Not Before: May 15 15:42:00 2019 GMT
            Not After : May 12 15:42:00 2029 GMT
        Subject: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kube-etcd

创建kube-apiserver-etcd-client证书

创建证书签名请求：

[root@d5-dui-master-001 kubernetes]# cat > cert/apiserver-etcd-client-csr.json <<"EOF"
{
    "CN": "kube-apiserver-etcd-client",
    "hosts": [],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "system:masters",
            "OU": "aispeech"
        }
    ]
}
EOF

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
    -ca=cert/etcd-ca.pem \
    -ca-key=cert/etcd-ca-key.pem \
    -config=cert/ca-config.json \
    -profile=kubernetes \
    cert/apiserver-etcd-client-csr.json| cfssljson -bare cert/apiserver-etcd-client

分发etcd相关证书和私钥到各master节点

[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m file -a "path=/etc/kubernetes/pki/etcd state=directory"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=cert/etcd-ca.pem dest=/etc/kubernetes/pki/etcd/ca.crt"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=cert/etcd-ca-key.pem dest=/etc/kubernetes/pki/etcd/ca.key"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=cert/kube-etcd.pem dest=/etc/kubernetes/pki/etcd/server.crt"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=cert/kube-etcd-key.pem dest=/etc/kubernetes/pki/etcd/server.key"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=cert/apiserver-etcd-client.pem dest=/etc/kubernetes/pki/apiserver-etcd-client.crt"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=cert/apiserver-etcd-client-key.pem dest=/etc/kubernetes/pki/apiserver-etcd-client.key"

部署Kubernetes的etcd集群

下载和分发etcd二进制文件

[root@d5-dui-master-001 kubernetes]# wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz
[root@d5-dui-master-001 kubernetes]# tar -zxf etcd-v3.3.13-linux-amd64.tar.gz
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=etcd-v3.3.13-linux-amd64/etcd dest=/usr/local/bin/kube-etcd mode=a+x"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=etcd-v3.3.13-linux-amd64/etcdctl dest=/usr/local/bin/ mode=a+x"

创建etcd的systemd unit模板文件

etcd.service：

[root@d5-dui-master-001 kubernetes]# cat > kube-etcd.service.j2 <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/kube-etcd/
ExecStart=/usr/local/bin/kube-etcd \
  --data-dir=/var/lib/kube-etcd/ \
  --name={{ ansible_hostname }} \
  --client-cert-auth \
  --cert-file=/etc/kubernetes/pki/etcd/server.crt \
  --key-file=/etc/kubernetes/pki/etcd/server.key \
  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --peer-client-cert-auth \
  --peer-cert-file=/etc/kubernetes/pki/etcd/server.crt \
  --peer-key-file=/etc/kubernetes/pki/etcd/server.key \
  --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --listen-peer-urls=https://{{ ansible_eth0.ipv4.address }}:2380 \
  --initial-advertise-peer-urls=https://{{ ansible_eth0.ipv4.address }}:2380 \
  --listen-client-urls=https://{{ ansible_eth0.ipv4.address }}:2379,https://127.0.0.1:2379 \
  --advertise-client-urls=https://{{ ansible_eth0.ipv4.address }}:2379 \
  --initial-cluster-token=d5-dui-k8s-cluster \
  --initial-cluster=d5-dui-master-001=https://10.41.225.129:2380,d5-dui-master-002=https://10.40.150.141:2380,d5-dui-master-003=https://10.41.225.130:2380 \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=10485760 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

分发etcd.service文件

[root@d5-dui-master-001 kubernetes]# cat > kube-etcd.yaml <<"EOF"
- hosts: k8s-masters
  remote_user: root
  tasks:
  - name: copy kube-etcd.service
    template:
      src: kube-etcd.service.j2
      dest: /usr/lib/systemd/system/kube-etcd.service  
  - name: create the data directory for etcd
    file:
      path: /var/lib/kube-etcd/
      state: directory
  - name: enable and start etcd.service
    systemd:
      name: kube-etcd
      state: restarted
      enabled: yes
      daemon_reload: yes
EOF
[root@d5-dui-master-001 kubernetes]# ansible-playbook kube-etcd.yaml
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m shell -a "systemctl status kube-etcd.service|grep -e Loaded -e Active"
d5-dui-master-003 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-15 23:51:25 CST; 4s ago

d5-dui-master-001 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-15 23:51:25 CST; 4s ago

d5-dui-master-002 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-15 23:51:25 CST; 4s ago

验证etcd集群状态

[root@d5-dui-master-001 kubernetes]# netstat -lnptu|grep etcd
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      23169/etcd          
tcp        0      0 10.41.225.129:2379      0.0.0.0:*               LISTEN      23169/etcd          
tcp        0      0 10.41.225.129:2380      0.0.0.0:*               LISTEN      23169/etcd
[root@d5-dui-master-001 kubernetes]# ETCDCTL_API=3 etcdctl \
  --endpoints=https://10.41.225.129:2379,https://10.40.150.141:2379,https://10.41.225.130:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health
https://10.41.225.130:2379 is healthy: successfully committed proposal: took = 2.611159ms
https://10.41.225.129:2379 is healthy: successfully committed proposal: took = 2.413527ms
https://10.40.150.141:2379 is healthy: successfully committed proposal: took = 1.931428ms

查看当前的leader节点

[root@d5-dui-master-001 kubernetes]# ETCDCTL_API=3 etcdctl \
  --endpoints=https://10.41.225.129:2379,https://10.40.150.141:2379,https://10.41.225.130:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status -w table
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://10.41.225.129:2379 |  c7ab68eac8156e5 |  3.3.13 |   20 kB |     false |         2 |          8 |
| https://10.40.150.141:2379 | 6874af47788ddde9 |  3.3.13 |   20 kB |      true |         2 |          8 |
| https://10.41.225.130:2379 | 85499b97bb010b82 |  3.3.13 |   20 kB |     false |         2 |          8 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+

安装docker

下载和安装docker-engine

[root@d5-dui-master-001 ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0  200G  0 disk 
└─vda1 253:1    0  200G  0 part /
vdb    253:16   0  200G  0 disk 
└─vdb1 253:17   0  200G  0 part /data/resources

使用根目录下的/data目录存放docker数据。

使用阿里云的docker yum源：

[root@d5-dui-master-001 ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker-main]
name=Docker Repository
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/
enabled=1
gpgcheck=0
EOF
[root@d5-dui-master-001 ~]# ansible k8s-masters -m file -a "path=/etc/yum.repos.d/docker.repo state=absent"
[root@d5-dui-master-001 ~]# ansible k8s-masters -m copy -a "src=/etc/yum.repos.d/docker-ce.repo dest=/etc/yum.repos.d/docker-ce.repo"

安装kubernetes v1.14推荐的docker 18.06：

[root@d5-dui-master-001 ~]# ansible k8s-masters -m yum -a "name=docker-ce-18.06.3.ce-3.el7.x86_64 state=present"

配置docker.service

[root@d5-dui-master-001 ~]# mkdir kubernetes && cd kubernetes
[root@d5-dui-master-001 kubernetes]# cat > docker.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H=unix:///var/run/docker.sock --data-root=/data/docker --insecure-registry=docker.v2.aispeech.com  --registry-mirror=https://woom60mt.mirror.aliyuncs.com --group=docker --max-concurrent-downloads=10 --max-concurrent-uploads=10 --live-restore
EOF
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m file -a "path=/etc/systemd/system/docker.service.d/ state=directory"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=docker.conf dest=/etc/systemd/system/docker.service.d/docker.conf"

这里没有修改cgroup驱动为systemd，kubelet使用systemd时有问题，仍旧使用cgroupfs。

启动docker.service

[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m file -a "path=/data/docker/ state=absent"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m systemd -a "name=docker state=restarted enabled=yes daemon_reload=yes"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m shell -a "systemctl status docker.service|grep -e Loaded -e Active"
d5-dui-master-002 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-15 16:09:46 CST; 14s ago

d5-dui-master-003 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-15 16:09:46 CST; 14s ago

d5-dui-master-001 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-15 16:09:47 CST; 13s ago

docker info

[root@d5-dui-master-001 kubernetes]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.06.3-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.639GiB
Name: d5-dui-master-001
ID: RADW:7XCP:QGDU:DGQO:KNWF:ESEU:PYJX:C3WR:UQD7:P75S:MOPR:T7EK
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 docker.v2.aispeech.com
 127.0.0.0/8
Registry Mirrors:
 https://woom60mt.mirror.aliyuncs.com/
Live Restore Enabled: true

部署Kubernetes

安装Kubernetes的yum源

[root@d5-dui-master-001 kubernetes]# ansible k8s-all -m copy -a "src=/etc/yum.repos.d/kubernetes.repo dest=/etc/yum.repos.d/kubernetes.repo"
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装v1.14.1版本kubelet、kubectl和kubeadm

[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m yum -a "name=kubeadm,kubelet,kubectl state=present"

修改系统设置

1）关闭swap

[root@bdp-k8s-master-001 kubernetes]# [root@d5-dui-master-001 ~]# ansible k8s-masters -m shell -a "swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab"

2）关闭firewalld

[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m systemd -a "name=firewalld state=stopped enabled=no"

3）关闭SELinux

[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m shell -a "if [ "$(getenforce)" == "Enforcing" ]; then setenforce 0;fi"
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m lineinfile -a "path=/etc/selinux/config regexp='^SELINUX=' line='SELINUX=disabled'"

4）修改内核参数

[root@bdp-k8s-master-001 kubernetes]# cat >> /etc/sysctl.d/k8s.conf <<EOF
vm.swappiness = 0
EOF
[root@bdp-k8s-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=/etc/sysctl.d/k8s.conf dest=/etc/sysctl.d/k8s.conf"
[root@bdp-k8s-master-001 kubernetes]# ansible k8s-masters -m shell -a "sysctl --system"

5）最大打开文件数量

[root@d5-dui-master-001 kubernetes]# ulimit -n
65535

不用修改。

6）安装常用的软件包

[root@d5-dui-master-001 ~]# ansible k8s-masters -m yum -a "name=net-tools,bash-completion,tree,telnet,nmap-ncat state=present"
[root@d5-dui-master-001 ~]# ansible k8s-masters -m shell -a "echo 'source <(kubectl completion bash)' >> /root/.bashrc"

kubectl completion bash配置kubectl命令的补全。

创建所需的证书和私钥

创建front-proxy-client证书和私钥

[root@d5-dui-master-001 kubernetes]# cat > cert/front-proxy-client-csr.json <<"EOF"
{
    "CN": "front-proxy-client",
    "hosts": [],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "k8s",
            "OU": "aispeech"
        }
    ]
}
EOF

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
    -ca=cert/front-proxy-ca.pem \
    -ca-key=cert/front-proxy-ca-key.pem \
    -config=cert/ca-config.json \
    -profile=kubernetes \
    cert/front-proxy-client-csr.json| cfssljson -bare cert/front-proxy-client

[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/front-proxy-client.pem |head -n11
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2c:a5:7e:7f:b0:82:6a:08:61:ba:77:c9:98:17:e1:85:f9:7b:d3:55
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kubernetes-front-proxy-ca
        Validity
            Not Before: May 15 17:32:00 2019 GMT
            Not After : May 12 17:32:00 2029 GMT
        Subject: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=front-proxy-client

创建apiserver使用的server证书和私钥

创建证书签名请求：

[root@d5-dui-master-001 kubernetes]# cat > cert/apiserver-csr.json <<EOF
{
  "CN": "kube-apiserver",
  "hosts": [
    "127.0.0.1",
    "localhost",
    "10.41.225.129",
    "10.40.150.141",
    "10.41.225.130",
    "10.40.150.145",
    "d5-dui-master-001",
    "d5-dui-master-002",
    "d5-dui-master-003",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Jiangsu",
      "L": "Suzhou",
      "O": "k8s",
      "OU": "aispeech"
    }
  ]
}
EOF

其中CN必须为“kube-apiserver”；
指定授权使用该证书的IP或域名列表，这里列出了apiserver节点IP、kubernetes服务的Service IP和域名；
如果使用了Keepalived等高可用，hosts字段需要加入VIP。

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
    -ca=cert/ca.pem \
    -ca-key=cert/ca-key.pem \
    -config=cert/ca-config.json \
    -profile=kubernetes \
    cert/apiserver-csr.json| cfssljson -bare cert/apiserver
[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/apiserver.pem|head -n11
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0e:b3:ff:b3:1b:71:91:3c:02:e7:e8:c3:53:76:c6:ed:8e:0b:3b:f9
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kubernetes-ca
        Validity
            Not Before: May 15 16:55:00 2019 GMT
            Not After : May 12 16:55:00 2029 GMT
        Subject: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kube-apiserver
[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/apiserver.pem|grep -A1 "X509v3 Subject Alternative Name"
            X509v3 Subject Alternative Name: 
                DNS:localhost, DNS:d5-dui-master-001, DNS:d5-dui-master-002, DNS:d5-dui-master-003, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:10.41.225.129, IP Address:10.40.150.141, IP Address:10.41.225.130, IP Address:10.40.150.145, IP Address:10.96.0.1

证书10年有效期。

创建apiserver访问kubelet的client证书

创建证书签名请求：

[root@d5-dui-master-001 kubernetes]# cat > cert/apiserver-kubelet-client-csr.json <<"EOF"
{
    "CN": "apiserver-kubelet-client",
    "hosts": [],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "system:masters",
            "OU": "aispeech"
        }
    ]
}
EOF

其中"O": "system:masters"对应ClusterRoleBinding:cluster-admin的Group。

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
    -ca=cert/ca.pem \
    -ca-key=cert/ca-key.pem \
    -config=cert/ca-config.json \
    -profile=kubernetes \
    cert/apiserver-kubelet-client-csr.json| cfssljson -bare cert/apiserver-kubelet-client

创建kubectl使用的admin证书和私钥

创建证书签名请求：

[root@d5-dui-master-001 kubernetes]# cat >cert/admin-csr.json <<"EOF"
{
    "CN": "kubernetes-admin",
    "hosts": [],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Jiangsu",
            "L": "Suzhou",
            "O": "system:masters",
            "OU": "aispeech"
        }
    ]
}
EOF

其中"O": "system:masters"对应ClusterRoleBinding:cluster-admin的Group，即该证书拥有Group:system:masters的权限——集群的最高权限。

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
    -ca=cert/ca.pem \
    -ca-key=cert/ca-key.pem \
    -config=cert/ca-config.json \
    -profile=kubernetes \
    cert/admin-csr.json| cfssljson -bare cert/admin

查看证书信息：

[root@d5-dui-master-001 kubernetes]# openssl x509 -noout -text -in cert/admin.pem|head -n11
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            54:bb:c7:03:b3:e2:06:e5:d0:d6:ec:4c:d4:d7:0c:b7:2a:79:b4:76
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=Jiangsu, L=Suzhou, O=k8s, OU=aispeech, CN=kubernetes-ca
        Validity
            Not Before: May 15 16:47:00 2019 GMT
            Not After : May 12 16:47:00 2029 GMT
        Subject: C=CN, ST=Jiangsu, L=Suzhou, O=system:masters, OU=aispeech, CN=kubernetes-admin

创建kubectl使用的kubeconfig文件：

[root@d5-dui-master-001 kubernetes]# kubectl config set-cluster kubernetes \
  --certificate-authority=cert/ca.pem \
  --server=https://10.41.225.129:6443 \
  --embed-certs=true \
  --kubeconfig=admin.kubeconfig
Cluster "kubernetes" set.
[root@d5-dui-master-001 kubernetes]# kubectl config set-credentials kubernetes-admin \
  --client-certificate=cert/admin.pem \
  --client-key=cert/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=admin.kubeconfig
User "kubernetes-admin" set.
[root@d5-dui-master-001 kubernetes]# kubectl config set-context kubernetes-admin@kubernetes \
  --cluster=kubernetes \
  --namespace=default \
  --user=kubernetes-admin \
  --kubeconfig=admin.kubeconfig
Context "kubernetes-admin@kubernetes" set.
[root@d5-dui-master-001 kubernetes]# kubectl config use-context kubernetes-admin@kubernetes \
  --kubeconfig=admin.kubeconfig
Switched to context "kubernetes-admin@kubernetes".

“--embed-certs=true”将kubectl所需的admin证书和私钥嵌入在kubeconfig文件中。

查看kubeconfig文件的配置信息：

[root@d5-dui-master-001 kubernetes]# kubectl config view --minify --kubeconfig=admin.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://10.41.225.129:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: default
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

创建controller-manager使用的证书和私钥

创建证书签名请求：

[root@d5-dui-master-001 kubernetes]# cat > cert/controller-manager-csr.json <<EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "10.41.225.129",
      "10.40.150.141",
      "10.41.225.130"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Jiangsu",
        "L": "Suzhou",
        "O": "k8s",
        "OU": "aispeech"
      }
    ]
}
EOF

hosts 列表包含所有 kube-controller-manager 节点 IP；
CN 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
    -ca=cert/ca.pem \
    -ca-key=cert/ca-key.pem \
    -config=cert/ca-config.json \
    -profile=kubernetes \
    cert/controller-manager-csr.json | cfssljson -bare cert/controller-manager

创建kubeconfig文件：

[root@d5-dui-master-001 kubernetes]# kubectl config set-cluster kubernetes \
  --certificate-authority=cert/ca.pem \
  --server=https://10.41.225.129:6443 \
  --embed-certs=true \
  --kubeconfig=controller-manager.kubeconfig
Cluster "kubernetes" set.

[root@d5-dui-master-001 kubernetes]# kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=cert/controller-manager.pem \
  --client-key=cert/controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=controller-manager.kubeconfig
User "system:kube-controller-manager" set.

[root@d5-dui-master-001 kubernetes]# kubectl config set-context system:kube-controller-manager@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=controller-manager.kubeconfig
Context "system:kube-controller-manager@kubernetes" set.

[root@d5-dui-master-001 kubernetes]# kubectl config use-context system:kube-controller-manager@kubernetes \
  --kubeconfig=controller-manager.kubeconfig
Switched to context "system:kube-controller-manager@kubernetes".

查看kubeconfig文件：

[root@d5-dui-master-001 kubernetes]# kubectl config view --kubeconfig=controller-manager.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://10.41.225.129:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:kube-controller-manager
  name: system:kube-controller-manager@kubernetes
current-context: system:kube-controller-manager@kubernetes
kind: Config
preferences: {}
users:
- name: system:kube-controller-manager
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

创建scheduler使用的证书和私钥

创建证书签名请求：

[root@d5-dui-master-001 kubernetes]# cat >cert/scheduler-csr.json <<EOF
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "10.41.225.129",
      "10.40.150.141",
      "10.41.225.130"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
          "C": "CN",
          "ST": "Jiangsu",
          "L": "Suzhou",
          "O": "k8s",
          "OU": "aispeech"
      }
    ]
}
EOF

hosts 列表包含所有 kube-scheduler 节点 IP；
CN 为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限。

生成证书和私钥：

[root@d5-dui-master-001 kubernetes]# cfssl gencert \
  -ca=cert/ca.pem \
  -ca-key=cert/ca-key.pem \
  -config=cert/ca-config.json \
  -profile=kubernetes \
  cert/scheduler-csr.json | cfssljson -bare cert/scheduler

创建kubeconfig文件：

[root@d5-dui-master-001 kubernetes]# kubectl config set-cluster kubernetes \
  --certificate-authority=cert/ca.pem \
  --server=https://10.41.225.129:6443 \
  --embed-certs=true \
  --kubeconfig=scheduler.kubeconfig
Cluster "kubernetes" set.

[admin@k8s-admin ~]$ kubectl config set-credentials system:kube-scheduler \
  --client-certificate=cert/scheduler.pem \
  --client-key=cert/scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=scheduler.kubeconfig
User "system:kube-scheduler" set.

[admin@k8s-admin ~]$ kubectl config set-context system:kube-scheduler@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=scheduler.kubeconfig
Context "system:kube-scheduler@kubernetes" set.

[admin@k8s-admin ~]$ kubectl config use-context system:kube-scheduler@kubernetes \
  --kubeconfig=scheduler.kubeconfig
Switched to context "system:kube-scheduler@kubernetes".

查看kubeconfig文件：

[root@d5-dui-master-001 kubernetes]# kubectl config view --kubeconfig scheduler.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://10.41.225.129:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:kube-scheduler
  name: system:kube-scheduler@kubernetes
current-context: system:kube-scheduler@kubernetes
kind: Config
preferences: {}
users:
- name: system:kube-scheduler
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

修改controller-manager、scheduler和admin使用的kubeconfig文件

修改controller-manager、scheduler和admin使用的kubeconfig文件，server使用当前节点的apiserver：

for node in d5-dui-master-001:10.41.225.129 d5-dui-master-002:10.40.150.141 d5-dui-master-003:10.41.225.130
do
  node_name=$(echo $node|awk -F':' '$0=$1')
  node_ip=$(echo $node|awk -F':' '$0=$2')
  cp controller-manager.kubeconfig controller-manager-$node_name.kubeconfig
  sed -i "s/10.41.225.129/$node_ip/g" controller-manager-$node_name.kubeconfig
done

for node in d5-dui-master-001:10.41.225.129 d5-dui-master-002:10.40.150.141 d5-dui-master-003:10.41.225.130
do
  node_name=$(echo $node|awk -F':' '$0=$1')
  node_ip=$(echo $node|awk -F':' '$0=$2')
  cp scheduler.kubeconfig scheduler-$node_name.kubeconfig
  sed -i "s/10.41.225.129/$node_ip/g" scheduler-$node_name.kubeconfig
done

for node in d5-dui-master-001:10.41.225.129 d5-dui-master-002:10.40.150.141 d5-dui-master-003:10.41.225.130
do
  node_name=$(echo $node|awk -F':' '$0=$1')
  node_ip=$(echo $node|awk -F':' '$0=$2')
  cp admin.kubeconfig admin-$node_name.kubeconfig
  sed -i "s/10.41.225.129/$node_ip/g" admin-$node_name.kubeconfig
done

创建master节点kubelet使用的kubeconfig文件

[root@d5-dui-master-001 kubernetes]# for node in d5-dui-master-001:10.41.225.129 d5-dui-master-002:10.40.150.141 d5-dui-master-003:10.41.225.130;do
node_name=$(echo $node|awk -F':' '$0=$1')
node_ip=$(echo $node|awk -F':' '$0=$2')
echo ">>> ${node_name}"

# 创建证书签名请求：
cat > kubelet/kubelet-${node_name}-csr.json <<EOF
{
    "CN": "system:node:${node_name}",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
    "127.0.0.1",
    "${node_name}"
    ],
    "names": [
      {
          "C": "CN",
          "ST": "Jiangsu",
          "L": "Suzhou",
          "O": "system:nodes",
          "OU": "aispeech"
      }
    ]
}
EOF

# 生成证书和私钥：
cfssl gencert \
  -ca=cert/ca.pem \
  -ca-key=cert/ca-key.pem \
  -config=cert/ca-config.json \
  -profile=kubernetes \
  kubelet/kubelet-${node_name}-csr.json | cfssljson -bare kubelet/kubelet-${node_name}

# 创建kubeconfig文件：
kubectl config set-cluster kubernetes \
  --certificate-authority=cert/ca.pem \
  --embed-certs=true \
  --server=https://${node_ip}:6443 \
  --kubeconfig=kubelet/kubelet-${node_name}.kubeconfig

kubectl config set-credentials system:node:${node_name} \
  --client-certificate=kubelet/kubelet-${node_name}.pem \
  --client-key=kubelet/kubelet-${node_name}-key.pem \
  --embed-certs=true \
  --kubeconfig=kubelet/kubelet-${node_name}.kubeconfig

kubectl config set-context system:node:${node_name}@kubernetes \
  --cluster=kubernetes \
  --user=system:node:${node_name} \
  --kubeconfig=kubelet/kubelet-${node_name}.kubeconfig

kubectl config use-context system:node:${node_name}@kubernetes \
  --kubeconfig=kubelet/kubelet-${node_name}.kubeconfig

done

分发master节点使用的证书和密钥

[root@d5-dui-master-001 kubernetes]# cat > pb-tls-master.yaml <<EOF
- hosts: k8s-masters
  remote_user: root
  tasks:
  - name: mkdir /etc/kubernetes/pki/
    file:
      path: /etc/kubernetes/pki/
      state: directory
  - name: copy ca.crt
    copy:
      src: cert/ca.pem
      dest: /etc/kubernetes/pki/ca.crt
  - name: copy ca.key
    copy:
      src: cert/ca-key.pem
      dest: /etc/kubernetes/pki/ca.key
  - name: copy apiserver.crt
    copy:
      src: cert/apiserver.pem
      dest: /etc/kubernetes/pki/apiserver.crt
  - name: copy apiserver.key
    copy:
      src: cert/apiserver-key.pem
      dest: /etc/kubernetes/pki/apiserver.key
  - name: copy apiserver-kubelet-client.crt
    copy:
      src: cert/apiserver-kubelet-client.pem
      dest: /etc/kubernetes/pki/apiserver-kubelet-client.crt
  - name: copy apiserver-kubelet-client.key
    copy:
      src: cert/apiserver-kubelet-client-key.pem
      dest: /etc/kubernetes/pki/apiserver-kubelet-client.key
  - name: copy front-proxy-ca.crt
    copy:
      src: cert/front-proxy-ca.pem
      dest: /etc/kubernetes/pki/front-proxy-ca.crt
  - name: copy front-proxy-ca.key
    copy:
      src: cert/front-proxy-ca-key.pem
      dest: /etc/kubernetes/pki/front-proxy-ca.key
  - name: copy front-proxy-client.crt
    copy:
      src: cert/front-proxy-client.pem
      dest: /etc/kubernetes/pki/front-proxy-client.crt
  - name: copy front-proxy-client.key
    copy:
      src: cert/front-proxy-client-key.pem
      dest: /etc/kubernetes/pki/front-proxy-client.key
  - name: copy sa.pub
    copy:
      src: /etc/kubernetes/pki/sa.pub
      dest: /etc/kubernetes/pki/sa.pub
  - name: copy sa.key
    copy:
      src: /etc/kubernetes/pki/sa.key
      dest: /etc/kubernetes/pki/sa.key
  - name: copy admin.conf
    copy:
      src: admin-{{ ansible_hostname }}.kubeconfig
      dest: /etc/kubernetes/admin.conf
  - name: copy controller-manager.conf
    copy:
      src: controller-manager-{{ ansible_hostname }}.kubeconfig
      dest: /etc/kubernetes/controller-manager.conf
  - name: copy scheduler.conf
    copy:
      src: scheduler-{{ ansible_hostname }}.kubeconfig
      dest: /etc/kubernetes/scheduler.conf
  - name: copy kubelet.conf
    copy:
      src: kubelet/kubelet-{{ ansible_hostname }}.kubeconfig
      dest: /etc/kubernetes/kubelet.conf
EOF
[root@d5-dui-master-001 kubernetes]# ansible-playbook pb-tls-master.yaml

初始化第一个master节点

准备kubeadm-config.yaml文件

默认的kubeadm init配置：

[root@d5-dui-master-001 kubernetes]# kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: d5-dui-master-001
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.14.0
networking:
  dnsDomain: cluster.local
  podSubnet: ""
  serviceSubnet: 10.96.0.0/12
scheduler: {}

自定义kubeadm init配置：

[root@d5-dui-master-001 kubernetes]# cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.1
controlPlaneEndpoint: 10.41.225.129:6443
imageRepository: docker.v2.aispeech.com/gcr.io
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
etcd:
    external:
        endpoints:
        - https://10.41.225.129:2379
        - https://10.40.150.141:2379
        - https://10.41.225.130:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
EOF

kind：老版本是MasterConfiguration，新版本是ClusterConfiguration；
kubernetesVersion：部署的kubernetes版本，kubeadm vX.Y 支持 vX.Y 和 vX.(Y-1) 两个版本kubernetes，推荐两者版本一致，这里设置为stable，即 vX.Y 的最新稳定版；
controlPlaneEndpoint：开始设置为阿里云SLB的内网地址和端口，但是阿里云L4 SLB后端服务器无法访问自己，最后修改为d5-dui-master-001的内网地址；
networking.podSubnet：flannel要求kubeadm init设置--pod-network-cidr=10.244.0.0/16。

查看需要下载的镜像：

[root@d5-dui-master-001 kubernetes]# kubeadm config images list --config kubeadm-config.yaml 
I0515 17:55:18.268528   28508 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0515 17:55:18.268626   28508 version.go:97] falling back to the local client version: v1.14.1
docker.v2.aispeech.com/gcr.io/kube-apiserver:v1.14.1
docker.v2.aispeech.com/gcr.io/kube-controller-manager:v1.14.1
docker.v2.aispeech.com/gcr.io/kube-scheduler:v1.14.1
docker.v2.aispeech.com/gcr.io/kube-proxy:v1.14.1
docker.v2.aispeech.com/gcr.io/pause:3.1
docker.v2.aispeech.com/gcr.io/etcd:3.3.10
docker.v2.aispeech.com/gcr.io/coredns:1.3.1

初始化第一个master节点

[root@d5-dui-master-001 kubernetes]# kubeadm init --config=kubeadm-config.yaml
[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing ca certificate authority
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using existing etcd/server certificate and key on disk
[certs] External etcd mode: Skipping etcd/peer certificate authority generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate authority generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.503616 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --experimental-upload-certs
[mark-control-plane] Marking the node d5-dui-master-001 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node d5-dui-master-001 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 5neu4m.85hod58emyyzeua3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities 
and service account keys on each node and then running the following as root:

  kubeadm join 10.41.225.129:6443 --token 5neu4m.85hod58emyyzeua3 \
    --discovery-token-ca-cert-hash sha256:c5faa35ce9fc9b68b053ec7fcbf312500a454ff50b193f331c6c70f238dc46ed \
    --experimental-control-plane    

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.41.225.129:6443 --token 5neu4m.85hod58emyyzeua3 \
    --discovery-token-ca-cert-hash sha256:c5faa35ce9fc9b68b053ec7fcbf312500a454ff50b193f331c6c70f238dc46ed

--experimental-upload-certs：kubeadm自动复制证书和密钥，可以去掉该参数手动复制；
certificate-key：两小时后过期，建议一次性加入所有 master 节点。

检查集群信息

[root@d5-dui-master-001 kubernetes]# kubectl get nodes
NAME                STATUS     ROLES    AGE    VERSION
d5-dui-master-001   NotReady   master   101s   v1.14.1
[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
coredns-7d5c5df7bf-hlm27                    0/1     Pending   0          83s
coredns-7d5c5df7bf-vxqcn                    0/1     Pending   0          82s
kube-apiserver-d5-dui-master-001            1/1     Running   0          24s
kube-controller-manager-d5-dui-master-001   1/1     Running   0          20s
kube-proxy-cfdph                            1/1     Running   0          83s
kube-scheduler-d5-dui-master-001            1/1     Running   0          16s
[root@d5-dui-master-001 kubernetes]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   106s

尚未安装网络插件，节点为“NotReady”状态，CoreDNS为“Pending”状态。

部署CNI网络插件flannel

[root@d5-dui-master-001 kubernetes]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@d5-dui-master-001 kubernetes]# kubectl apply -f kube-flannel.yml 
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

确保 podSubnet 和 kubeadm-config.yaml 文件一致。

等待所有Pods就绪

[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
coredns-7d5c5df7bf-hlm27                    1/1     Running   0          8m8s
coredns-7d5c5df7bf-vxqcn                    1/1     Running   0          8m7s
kube-apiserver-d5-dui-master-001            1/1     Running   0          7m9s
kube-controller-manager-d5-dui-master-001   1/1     Running   0          7m5s
kube-flannel-ds-amd64-rtch2                 1/1     Running   0          73s
kube-proxy-cfdph                            1/1     Running   0          8m8s
kube-scheduler-d5-dui-master-001            1/1     Running   0          7m1s
[root@d5-dui-master-001 kubernetes]# kubectl get nodes
NAME                STATUS   ROLES    AGE     VERSION
d5-dui-master-001   Ready    master   8m30s   v1.14.1

加入其它master节点

由于没有使用vip，每个master节点上的组件配置文件不一样，无法使用kubeadm join加master节点。这里需要修改kube-apiserver.yaml的“--advertise-address”和“livenessProbe”的ip为当前master节点的ip。

准备kube-apiserver.yaml

[root@d5-dui-master-001 kubernetes]# cp /etc/kubernetes/manifests/* .
[root@d5-dui-master-001 kubernetes]# sed -i 's/- kube-apiserver/- kube-apiserver\n    - --apiserver-count=3/g' kube-apiserver.yaml
[root@d5-dui-master-001 kubernetes]# for node in d5-dui-master-001:10.41.225.129 d5-dui-master-002:10.40.150.141 d5-dui-master-003:10.41.225.130
do
  node_name=$(echo $node|awk -F':' '$0=$1')
  node_ip=$(echo $node|awk -F':' '$0=$2')
  cp kube-apiserver.yaml kube-apiserver-$node_name.yaml
  sed -i "s/10.41.225.129/$node_ip/g" kube-apiserver-$node_name.yaml
  ansible $node_name -m copy -a "src=kube-apiserver-$node_name.yaml dest=/etc/kubernetes/manifests/kube-apiserver.yaml"
done

分发manifests yaml文件并重启kubelet

ansible k8s-masters -m copy -a "src=kube-scheduler.yaml dest=/etc/kubernetes/manifests/kube-scheduler.yaml"
ansible k8s-masters -m copy -a "src=kube-controller-manager.yaml dest=/etc/kubernetes/manifests/kube-controller-manager.yaml"
ansible k8s-masters -m copy -a "src=/var/lib/kubelet/config.yaml dest=/var/lib/kubelet/config.yaml"
ansible k8s-masters -m copy -a "src=/var/lib/kubelet/kubeadm-flags.env dest=/var/lib/kubelet/kubeadm-flags.env"
ansible k8s-masters -m copy -a "src=/etc/sysconfig/kubelet dest=/etc/sysconfig/kubelet"
ansible k8s-masters -m systemd -a "name=kubelet state=restarted daemon_reload=yes enabled=yes"

检查master节点和组件状态

节点状态：

[root@d5-dui-master-001 kubernetes]# kubectl get nodes
NAME                STATUS   ROLES    AGE     VERSION
d5-dui-master-001   Ready    master   13h     v1.14.1
d5-dui-master-002   Ready    <none>   13h     v1.14.1
d5-dui-master-003   Ready    <none>   13h     v1.14.1

组件状态：

[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -a "kubectl get componentstatuses"
d5-dui-master-001 | CHANGED | rc=0 >>
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   

d5-dui-master-002 | CHANGED | rc=0 >>
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   

d5-dui-master-003 | CHANGED | rc=0 >>
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}

系统Pods状态：

[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
coredns-7d5c5df7bf-29kw5                    1/1     Running   7          6h50m
coredns-7d5c5df7bf-dc75z                    1/1     Running   4          6h50m
kube-apiserver-d5-dui-master-001            1/1     Running   3          13h
kube-apiserver-d5-dui-master-002            1/1     Running   10         13h
kube-apiserver-d5-dui-master-003            1/1     Running   9          13h
kube-controller-manager-d5-dui-master-001   1/1     Running   3          13h
kube-controller-manager-d5-dui-master-002   1/1     Running   3          13h
kube-controller-manager-d5-dui-master-003   1/1     Running   2          13h
kube-flannel-ds-amd64-2xv2n                 1/1     Running   3          13h
kube-flannel-ds-amd64-cv7sw                 1/1     Running   2          13h
kube-flannel-ds-amd64-rtch2                 1/1     Running   3          13h
kube-proxy-cfdph                            1/1     Running   3          13h
kube-proxy-fcmkv                            1/1     Running   1          6h54m
kube-proxy-ft668                            1/1     Running   3          13h
kube-scheduler-d5-dui-master-001            1/1     Running   2          13h
kube-scheduler-d5-dui-master-002            1/1     Running   3          13h
kube-scheduler-d5-dui-master-003            1/1     Running   2          13h

设置所有master节点NoSchedule

给非kubeadm初始化的master节点添加污点：

[root@d5-dui-master-001 kubernetes]# kubectl taint node d5-dui-master-002 node-role.kubernetes.io/master=:NoSchedule
error: Node d5-dui-master-002 already has node-role.kubernetes.io/master taint(s) with same effect(s) and --overwrite is false
[root@d5-dui-master-001 kubernetes]# kubectl taint node d5-dui-master-003 node-role.kubernetes.io/master=:NoSchedule
error: Node d5-dui-master-002 already has node-role.kubernetes.io/master taint(s) with same effect(s) and --overwrite is false

给非kubeadm初始化的master节点打master标签：

[root@d5-dui-master-001 kubernetes]# kubectl label nodes d5-dui-master-002 node-role.kubernetes.io/master=
node/d5-dui-master-002 labeled
[root@d5-dui-master-001 kubernetes]# kubectl label nodes d5-dui-master-003 node-role.kubernetes.io/master=
node/d5-dui-master-003 labeled
[root@d5-dui-master-001 kubernetes]# kubectl get nodes --show-labels
NAME                STATUS   ROLES    AGE     VERSION   LABELS
d5-dui-master-001   Ready    master   13h     v1.14.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=d5-dui-master-001,kubernetes.io/os=linux,node-role.kubernetes.io/master=
d5-dui-master-002   Ready    master   13h     v1.14.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=d5-dui-master-002,kubernetes.io/os=linux,node-role.kubernetes.io/master=
d5-dui-master-003   Ready    master   13h     v1.14.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=d5-dui-master-003,kubernetes.io/os=linux,node-role.kubernetes.io/master=
d5-nginx-001        Ready    <none>   4h16m   v1.14.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=d5-nginx-001,kubernetes.io/os=linux,nginx-ingress-controller-dds=1
d5-nginx-002        Ready    <none>   4h16m   v1.14.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=d5-nginx-002,kubernetes.io/os=linux,nginx-ingress-controller-dds=1

daoker中添加新集群

创建serviceaccount:daoker-admin：

[root@d5-dui-master-001 kubernetes]# kubectl get clusterrolebindings|grep -e cluster-admin -e NAME
NAME                                                   AGE
cluster-admin                                          8h
[root@d5-dui-master-001 kubernetes]# kubectl create serviceaccount daoker-admin -n kube-system
serviceaccount/daoker-admin created
[root@d5-dui-master-001 kubernetes]# kubectl create clusterrolebinding daoker-admin --clusterrole=cluster-admin --serviceaccount kube-system:daoker-admin
clusterrolebinding.rbac.authorization.k8s.io/daoker-admin created
[root@d5-dui-master-001 kubernetes]# kubectl get clusterrolebindings -o wide|grep -e cluster-admin -e NAME
NAME                AGE   ROLE                            USERS   GROUPS           SERVICEACCOUNTS
cluster-admin       8h    ClusterRole/cluster-admin               system:masters   
daoker-admin        20s   ClusterRole/cluster-admin                                kube-system/daoker-admin

查看token和API Token：

[root@d5-dui-master-001 kubernetes]# kubectl get sa -n kube-system daoker-admin -o jsonpath='{.secrets[0].name}'
daoker-admin-token-c7rkl
[root@d5-dui-master-001 kubernetes]# kubectl get secret -n kube-system $(kubectl get sa -n kube-system daoker-admin -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}'
ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9pSnJkV0psTFhONWMzUmxiU0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVmpjbVYwTG01aGJXVWlPaUprWVc5clpYSXRZV1J0YVc0dGRHOXJaVzR0WXpkeWEyd2lMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObGNuWnBZMlV0WVdOamIzVnVkQzV1WVcxbElqb2laR0Z2YTJWeUxXRmtiV2x1SWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXpaWEoyYVdObExXRmpZMjkxYm5RdWRXbGtJam9pTW1VMVpqVTRaV0V0TnpjNE9DMHhNV1U1TFRrMU1XTXRNREF4TmpObE1EUmlZamt4SWl3aWMzVmlJam9pYzNsemRHVnRPbk5sY25acFkyVmhZMk52ZFc1ME9tdDFZbVV0YzNsemRHVnRPbVJoYjJ0bGNpMWhaRzFwYmlKOS5OR0lrTDRsdVdLZ19KZjd1dVV2djZNR19CV3pKaGQzODZ6bG9SblZEdnJ1NlFuZzFVNUF0cm03UDVoYWhoeFJ1bnNaZzFZckM3NDM4WnRUa2ZCdXh5VTJXcGpLMkRKcTVVazJzeC1GWUJTYjBJQlB6Y1FhVTNtV0FBcVpwX2NxUllfZ3V2VjJjc3Blc000T3kzSHBEeEJLVVJMVTA5Q0ZybG5CZS1BaE54UkF1dzEtSUg5RnZQYkdiaUVZX1hkRTR2VU1KcTZGNTdFSEsyWDN6U0ZnSFpaWUpkc0pGR0dTQ2JUNVVuQXRfdDdoVDNCN3BndTRhMnR1elI5MXJTUzR3X0hPcHBjamtPd3o1eWsycjlIZmNqVXZraU9vOGJST05TbWxBMGJLVTFJcmNySTJweFVkWTk5VzQxV3ZBVk5uTXhka3QtODR1UlZFdXJlcUNBQXFFaGc=

添加worker节点

worker节点使用阿里云SLB的内网IP地址，端口8443：

[root@d5-nginx-001 data]# kubeadm join 10.40.150.145:8443 --token 5neu4m.85hod58emyyzeua3 \
> --discovery-token-ca-cert-hash sha256:c5faa35ce9fc9b68b053ec7fcbf312500a454ff50b193f331c6c70f238dc46ed 
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
▽       [WARNING Service-Kubelet]: kubelet service does not exist
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] couldn't detect a kubelet service, can't make sure the kubelet not running for a short period of time while setting up configuration for it.
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] couldn't detect a kubelet service, can't make sure the kubelet is running properly.
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

部署业务etcd集群

版本保持和d1集群的etcd一样，且不适用https：

[root@d1-k8s-master-001 ~]# etcdctl --version
etcdctl version: 3.2.22
API version: 2

下载和分发etcd二进制文件

[root@d5-dui-master-001 kubernetes]# wget https://github.com/etcd-io/etcd/releases/download/v3.2.22/etcd-v3.2.22-linux-amd64.tar.gz
[root@d5-dui-master-001 kubernetes]# tar -zxf etcd-v3.2.22-linux-amd64.tar.gz
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m copy -a "src=etcd-v3.2.22-linux-amd64/etcd dest=/usr/local/bin/daoker-etcd mode=a+x"

创建etcd的systemd unit模板文件

etcd.service：

[root@d5-dui-master-001 kubernetes]# cat > daoker-etcd.service.j2 <<"EOF"
[Unit]
Description=Etcd Server For Daoker
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/daoker-etcd/
ExecStart=/usr/local/bin/daoker-etcd \
  --data-dir=/var/lib/daoker-etcd/ \
  --name={{ ansible_hostname }} \
  --advertise-client-urls=http://{{ ansible_eth0.ipv4.address }}:2381 \
  --listen-client-urls=http://{{ ansible_eth0.ipv4.address }}:2381,http://127.0.0.1:2381 \
  --initial-advertise-peer-urls=http://{{ ansible_eth0.ipv4.address }}:2382 \
  --listen-peer-urls=http://{{ ansible_eth0.ipv4.address }}:2382 \
  --initial-cluster=d5-dui-master-001=http://10.41.225.129:2382,d5-dui-master-002=http://10.40.150.141:2382,d5-dui-master-003=http://10.41.225.130:2382 \
  --initial-cluster-state=new \
  --initial-cluster-token=d5-dui-daoker-cluster
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

分发daoker-etcd.service文件

[root@d5-dui-master-001 kubernetes]# cat > daoker-etcd.yaml <<"EOF"
- hosts: k8s-masters
  remote_user: root
  tasks:
  - name: copy daoker-etcd.service
    template:
      src: daoker-etcd.service.j2
      dest: /usr/lib/systemd/system/daoker-etcd.service
  - name: create the data directory for daoker-etcd
    file:
      path: /var/lib/daoker-etcd/
      state: directory
  - name: enable and start daoker-etcd.service
    systemd:
      name: daoker-etcd
      state: restarted
      enabled: yes
      daemon_reload: yes
EOF
[root@d5-dui-master-001 kubernetes]# ansible-playbook daoker-etcd.yaml
[root@d5-dui-master-001 kubernetes]# ansible k8s-masters -m shell -a "systemctl status daoker-etcd.service|grep -e Loaded -e Active"
d5-dui-master-001 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/daoker-etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-05-16 14:40:28 CST; 54s ago

d5-dui-master-002 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/daoker-etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-05-16 14:40:28 CST; 54s ago

d5-dui-master-003 | CHANGED | rc=0 >>
   Loaded: loaded (/usr/lib/systemd/system/daoker-etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-05-16 14:40:28 CST; 54s ago

验证业务etcd集群状态

[root@d5-dui-master-001 kubernetes]# netstat -lnptu|grep daoker-etcd
tcp        0      0 127.0.0.1:2381          0.0.0.0:*               LISTEN      21136/daoker-etcd   
tcp        0      0 10.41.225.129:2381      0.0.0.0:*               LISTEN      21136/daoker-etcd   
tcp        0      0 10.41.225.129:2382      0.0.0.0:*               LISTEN      21136/daoker-etcd
[root@d5-dui-master-001 kubernetes]# ETCDCTL_API=3 etcdctl \
  --endpoints=http://10.41.225.129:2381,http://10.40.150.141:2381,http://10.41.225.130:2381 \
  endpoint health
http://10.41.225.130:2381 is healthy: successfully committed proposal: took = 1.850931ms
http://10.41.225.129:2381 is healthy: successfully committed proposal: took = 2.726152ms
http://10.40.150.141:2381 is healthy: successfully committed proposal: took = 2.704009ms

查看当前的leader节点

[root@d5-dui-master-001 kubernetes]# ETCDCTL_API=3 etcdctl \
  --endpoints=http://10.41.225.129:2381,http://10.40.150.141:2381,http://10.41.225.130:2381 \
  endpoint status -w table
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
| http://10.41.225.129:2381 | 3de7f5ce308a5933 |  3.2.22 |   25 kB |     false |         2 |          8 |
| http://10.40.150.141:2381 |  eb5f1aa72077efc |  3.2.22 |   25 kB |     false |         2 |          8 |
| http://10.41.225.130:2381 | 40660789f82c2e0b |  3.2.22 |   25 kB |      true |         2 |          8 |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+

配置d5-dui-master-001的业务etcd外网访问

daoker-etcd.service：

[root@d5-dui-master-001 kubernetes]# cat > /usr/lib/systemd/system/daoker-etcd.service <<EOF
[Unit]
Description=Etcd Server For Daoker
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/daoker-etcd/
ExecStart=/usr/local/bin/daoker-etcd \
  --data-dir=/var/lib/daoker-etcd/ \
  --name=d5-dui-master-001 \
  --advertise-client-urls=http://10.41.225.129:2381 \
  --listen-client-urls=http://0.0.0.0:2381 \
  --initial-advertise-peer-urls=http://10.41.225.129:2382 \
  --listen-peer-urls=http://10.41.225.129:2382 \
  --initial-cluster=d5-dui-master-001=http://10.41.225.129:2382,d5-dui-master-002=http://10.40.150.141:2382,d5-dui-master-003=http://10.41.225.130:2382 \
  --initial-cluster-state=new \
  --initial-cluster-token=d5-dui-daoker-cluster
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

重启daoker-etcd.service：

[root@d5-dui-master-001 kubernetes]# systemctl daemon-reload
[root@d5-dui-master-001 kubernetes]# systemctl restart daoker-etcd

测试外网连接业务etcd：

[root@daoker etcd-v3.2.22-linux-amd64]# ETCDCTL_API=3 ./etcdctl --endpoints=http://39.98.194.75:2381 endpoint status -w table
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| http://39.98.194.75:2381 | 3de7f5ce308a5933 |  3.2.22 |   25 kB |     false |         2 |          9 |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
[root@daoker etcd-v3.2.22-linux-amd64]# ETCDCTL_API=3 ./etcdctl --endpoints=http://39.98.194.75:2381 put test 20190516
OK
[root@daoker etcd-v3.2.22-linux-amd64]# ETCDCTL_API=3 ./etcdctl --endpoints=http://39.98.194.75:2381 get test
test
20190516
[root@daoker etcd-v3.2.22-linux-amd64]# ETCDCTL_API=3 ./etcdctl --endpoints=http://39.98.194.75:2381 del test
1
[root@daoker etcd-v3.2.22-linux-amd64]# ETCDCTL_API=3 ./etcdctl --endpoints=http://39.98.194.75:2381 get / --prefix

集群健壮性测试

部署应用

复制华东1集群etcd、Secret、ConfigMap和Ingress的配置，部署应用。

集群组件修改记录

20190520.修改服务的NodePort可用范围

服务默认的NodePort可用范围为30000-32767，部署服务的时候发现cnluserver使用了28033端口导致业务无法部署。查看华东正式环境，设置的NodePort可用范围是80-32767，于是将华北修改为和华东一样。 NodePort可用范围通过apiserver的“--service-node-port-range”参数设置，修改三个master节点的kube-apiserver的Pod添加“--service-node-port-range=80-32767”参数。

20190521.修改flannel优先级

因资源不足引起flannel的Pod被驱逐，导致节点NotReady，解决办法修改flannel的Pod优先级。

集群中默认的优先级PriorityClass：system-cluster-critical：

[root@d5-dui-master-001 kubernetes]# kubectl get priorityclasses
NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            5d14h
system-node-critical      2000001000   false            5d14h
[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system kube-apiserver-d5-dui-master-001 -o yaml|grep priorityClassName
  priorityClassName: system-cluster-critical
[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system kube-controller-manager-d5-dui-master-001 -o yaml|grep priorityClassName
  priorityClassName: system-cluster-critical
[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system kube-scheduler-d5-dui-master-001 -o yaml|grep priorityClassName
  priorityClassName: system-cluster-critical
[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system kube-proxy-6qr7q -o yaml|grep priorityClassName
  priorityClassName: system-node-critical
[root@d5-dui-master-001 kubernetes]# kubectl get deployment -n kube-system coredns -o yaml|grep priorityClassName
      priorityClassName: system-cluster-critical
[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system kube-flannel-ds-amd64-2wzh2 -o yaml|grep priorityClassName

增加新的优先级PriorityClass：important-kubernetes：

[root@d5-dui-master-001 kubernetes]# cat > kube-flannel-pc.yaml <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: important-kubernetes
value: 1000000000
EOF
[root@d5-dui-master-001 kubernetes]# kubectl apply -f kube-flannel-pc.yaml 
priorityclass.scheduling.k8s.io/important-kubernetes created

修改flannel的定义文件kube-flannel.yaml，在daemonset：kube-flannel-ds-amd64的“spec.template.spec”下添加“priorityClassName: important-kubernetes”，然后应用Yaml文件，并重启flannel：

[root@d5-dui-master-001 kubernetes]# kubectl apply -f kube-flannel.yml 
podsecuritypolicy.extensions/psp.flannel.unprivileged configured
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.extensions/kube-flannel-ds-amd64 configured
daemonset.extensions/kube-flannel-ds-arm64 unchanged
daemonset.extensions/kube-flannel-ds-arm unchanged
daemonset.extensions/kube-flannel-ds-ppc64le unchanged
daemonset.extensions/kube-flannel-ds-s390x unchanged

重启flannel之后可以看到其优先级已经改变，节点资源不足时不会影响到flannel的稳定性：

[root@d5-dui-master-001 kubernetes]# kubectl get ds -n kube-system kube-flannel-ds-amd64 -o yaml|grep priorityClassName
      priorityClassName: important-kubernetes
[root@d5-dui-master-001 kubernetes]# kubectl delete pods -n kube-system --force --grace-period=0 $(kubectl get pods -n kube-system |grep flannel|awk '{print $1}')
[root@d5-dui-master-001 kubernetes]# kubectl get pods -n kube-system kube-flannel-ds-amd64-4hlq2 -o yaml|grep -B1 priorityClassName
  priority: 1000000000
  priorityClassName: important-kubernetes

Space shortcuts

Page tree

华北Kubernetes集群部署记录(多master)

创建kube-apiserver的负载均衡器SLB

部署ansible

创建CA证书和密钥

安装cfssl工具集

生成CA配置文件

创建etcd-ca证书

创建kubernetes-ca证书

创建kubernetes-front-proxy-ca证书

创建etcd相关的证书和私钥

创建etcd server证书和私钥

创建kube-apiserver-etcd-client证书

分发etcd相关证书和私钥到各master节点

部署Kubernetes的etcd集群

下载和分发etcd二进制文件

创建etcd的systemd unit模板文件

分发etcd.service文件

验证etcd集群状态

查看当前的leader节点

安装docker

下载和安装docker-engine

配置docker.service

启动docker.service

docker info

部署Kubernetes

安装Kubernetes的yum源

安装v1.14.1版本kubelet、kubectl和kubeadm

修改系统设置

创建所需的证书和私钥

创建front-proxy-client证书和私钥

创建apiserver使用的server证书和私钥

创建apiserver访问kubelet的client证书

创建kubectl使用的admin证书和私钥

创建controller-manager使用的证书和私钥

创建scheduler使用的证书和私钥

修改controller-manager、scheduler和admin使用的kubeconfig文件

创建master节点kubelet使用的kubeconfig文件

分发master节点使用的证书和密钥

初始化第一个master节点

准备kubeadm-config.yaml文件

初始化第一个master节点

检查集群信息

部署CNI网络插件flannel

等待所有Pods就绪

加入其它master节点

准备kube-apiserver.yaml

分发manifests yaml文件并重启kubelet

检查master节点和组件状态

设置所有master节点NoSchedule

daoker中添加新集群

添加worker节点

部署业务etcd集群

下载和分发etcd二进制文件

创建etcd的systemd unit模板文件

分发daoker-etcd.service文件

验证业务etcd集群状态

查看当前的leader节点

配置d5-dui-master-001的业务etcd外网访问

集群健壮性测试

部署应用

集群组件修改记录

20190520.修改服务的NodePort可用范围

20190521.修改flannel优先级