1、部署业务etcd集群daoker-etcd

版本和Kubernetes集群使用的etcd一样，但不使用https，监听2081和2082端口。

1）准备etcd的systemd unit模板文件

[root@d3-master-001 kubernetes]# cat > daoker-etcd.service.j2 <<EOF
[Unit]
Description=Etcd Server For Daoker
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/daoker-etcd/
ExecStart=/usr/local/bin/daoker-etcd \\
  --data-dir=/var/lib/daoker-etcd/ \\
  --name={{ ansible_hostname }} \\
  --advertise-client-urls=http://{{ ansible_$IFACE.ipv4.address }}:2381 \\
  --listen-client-urls=http://{{ ansible_$IFACE.ipv4.address }}:2381,http://127.0.0.1:2381 \\
  --initial-advertise-peer-urls=http://{{ ansible_$IFACE.ipv4.address }}:2382 \\
  --listen-peer-urls=http://{{ ansible_$IFACE.ipv4.address }}:2382 \\
  --initial-cluster=d3-master-001=http://$MASTER01_IP:2382,d3-master-002=http://$MASTER02_IP:2382,d3-master-003=http://$MASTER03_IP:2382 \\
  --initial-cluster-state=new \\
  --initial-cluster-token=daoker-etcd-cluster
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

2）分发daoker-etcd.service文件并启动etcd

[root@d3-master-001 kubernetes]# cat > pb-daoker-etcd.yaml <<"EOF"
- hosts: k8s-masters
  remote_user: root
  tasks:
  - name: copy etcd binary file
    copy:
      src: etcd-v3.3.13-linux-amd64/etcd
      dest: /usr/local/bin/daoker-etcd
      mode: u+x
  - name: copy daoker-etcd.service
    template:
      src: daoker-etcd.service.j2
      dest: /usr/lib/systemd/system/daoker-etcd.service
  - name: create the data directory for daoker-etcd
    file:
      path: /var/lib/daoker-etcd/
      state: directory
  - name: enable and start daoker-etcd.service
    systemd:
      name: daoker-etcd
      state: restarted
      enabled: yes
      daemon_reload: yes
EOF
[root@d3-master-001 kubernetes]# ansible-playbook pb-daoker-etcd.yaml
[root@d3-master-001 kubernetes]# ansible k8s-masters -m shell -a "systemctl status daoker-etcd.service|grep -e Loaded -e Active"

3）验证etcd集群状态

[root@d3-master-001 kubernetes]# netstat -lnptu|grep etcd
[root@d3-master-001 kubernetes]# netstat -lnptu|grep etcd
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      155563/kube-etcd
tcp        0      0 10.24.10.74:2379        0.0.0.0:*               LISTEN      155563/kube-etcd
tcp        0      0 10.24.10.74:2380        0.0.0.0:*               LISTEN      155563/kube-etcd
tcp        0      0 127.0.0.1:2381          0.0.0.0:*               LISTEN      184534/daoker-etcd
tcp        0      0 10.24.10.74:2381        0.0.0.0:*               LISTEN      184534/daoker-etcd
tcp        0      0 10.24.10.74:2382        0.0.0.0:*               LISTEN      184534/daoker-etcd
[root@d3-master-001 kubernetes]# ETCDCTL_API=3 etcdctl \
  --endpoints=http://$MASTER01_IP:2381,http://$MASTER02_IP:2381,http://$MASTER03_IP:2381 \
  endpoint health
http://10.24.10.76:2381 is healthy: successfully committed proposal: took = 1.469283ms
http://10.24.10.74:2381 is healthy: successfully committed proposal: took = 1.871965ms
http://10.24.10.75:2381 is healthy: successfully committed proposal: took = 1.937508ms

查看集群成员：

[root@d3-master-001 kubernetes]# ETCDCTL_API=3 etcdctl \
  --endpoints=http://$MASTER01_IP:2381,http://$MASTER02_IP:2381,http://$MASTER03_IP:2381 \
  member list -w table
+------------------+---------+---------------+-------------------------+-------------------------+
|        ID        | STATUS  |     NAME      |       PEER ADDRS        |      CLIENT ADDRS       |
+------------------+---------+---------------+-------------------------+-------------------------+
| 12cd34edd4d7594f | started | d3-master-001 | http://10.24.10.74:2382 | http://10.24.10.74:2381 |
| 2a9da1140162f35c | started | d3-master-002 | http://10.24.10.75:2382 | http://10.24.10.75:2381 |
| efd312ff39f47e32 | started | d3-master-003 | http://10.24.10.76:2382 | http://10.24.10.76:2381 |
+------------------+---------+---------------+-------------------------+-------------------------+

查看当前的leader节点：

[root@d3-master-001 kubernetes]# ETCDCTL_API=3 etcdctl \
  --endpoints=http://$MASTER01_IP:2381,http://$MASTER02_IP:2381,http://$MASTER03_IP:2381 \
  endpoint status -w table
+-------------------------+------------------+---------+---------+-----------+-----------+------------+
|        ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-------------------------+------------------+---------+---------+-----------+-----------+------------+
| http://10.24.10.74:2381 | 12cd34edd4d7594f |  3.3.13 |   16 kB |     false |         2 |          8 |
| http://10.24.10.75:2381 | 2a9da1140162f35c |  3.3.13 |   16 kB |     false |         2 |          8 |
| http://10.24.10.76:2381 | efd312ff39f47e32 |  3.3.13 |   16 kB |      true |         2 |          8 |
+-------------------------+------------------+---------+---------+-----------+-----------+------------+

2、daoker中添加新集群

1）daoker中添加新集群

集群名称：D3-DUI-正式环境(国科) 集群简称：d3-gk 集群地址：10.24.10.114:8443 etcd地址：10.24.10.114:8442 token和API token：

// 查询拥有“ClusterRole/cluster-admin”权限的Serviceaccount
[root@d3-master-001 ~]# kubectl get clusterrolebindings -o wide|grep -e cluster-admin -e NAME
NAME           AGE     ROLE                       USERS  GROUPS          SERVICEACCOUNTS
cluster-admin  7h38m   ClusterRole/cluster-admin         system:masters

// 如果没有拥有“ClusterRole/cluster-admin”权限的Serviceaccount，可以手动创建一个。Serviceaccount名字随意，这里设置为admin-daoker
[root@d3-master-001 ~]# kubectl create serviceaccount admin-daoker -n kube-system
serviceaccount/admin-daoker created
[root@d3-master-001 ~]# kubectl create clusterrolebinding admin-daoker --clusterrole=cluster-admin --serviceaccount kube-system:admin-daoker
clusterrolebinding.rbac.authorization.k8s.io/admin-daoker created
[root@d3-master-001 ~]# kubectl get clusterrolebindings -o wide|grep -e cluster-admin -e NAME
NAME           AGE     ROLE                       USERS  GROUPS          SERVICEACCOUNTS
admin-daoker   10s     ClusterRole/cluster-admin                         kube-system/admin-daoker
cluster-admin  7h38m   ClusterRole/cluster-admin         system:masters

// 查询Secret的name，填写在“token”一栏
[root@d3-master-001 ~]# kubectl get sa -n kube-system admin-daoker -o jsonpath='{.secrets[0].name}'
admin-daoker-token-xg8c7

// 查询Secret的token，填写在“API token”一栏
[root@d3-master-001 ~]# kubectl get secret -n kube-system $(kubectl get sa -n kube-system admin-daoker -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}'
ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9pSnJkV0psTFhONWMzUmxiU0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVmpjbVYwTG01aGJXVWlPaUpoWkcxcGJpMWtZVzlyWlhJdGRHOXJaVzR0ZUdjNFl6Y2lMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObGNuWnBZMlV0WVdOamIzVnVkQzV1WVcxbElqb2lZV1J0YVc0dFpHRnZhMlZ5SWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXpaWEoyYVdObExXRmpZMjkxYm5RdWRXbGtJam9pTUROa09UTTRPVFV0WldNeE1DMHhNV1U1TFdFell6TXRNVFF4T0RjM05XTm1PVEpoSWl3aWMzVmlJam9pYzNsemRHVnRPbk5sY25acFkyVmhZMk52ZFc1ME9tdDFZbVV0YzNsemRHVnRPbUZrYldsdUxXUmhiMnRsY2lKOS5MSndmY04waGVHeDRHWnN3RU5tLU94dGNudFlJX2xpaVBPa1BEbEZDOTFEd3FpbVJhUVNscnpjbHB4X2pGM0E1alkwNC1DeGZ2R1YweWJxUEFHU3ZpZnJnNk8tOG1IeFNPQWRmVGNOMnRpeWhZeU1pWTBsVnNQSGw2cWNYdHEyYnZBR2cwMHB2Y2h4c2xsM1ZwVU9Od09DbUJhYzBSVUZIQUduelJsZHpwZnBmbGxMVGtrT2Y1ZmVFcGhGeS15d3lfZWw1a29xYnFfZjBqbE9oWUdVeTU5RUdRb1kwT1gzMjlSY0VxOXF0QTJTMFFyZUtSU2U5YkVERWZLMkxzVWJDTVNLV01FeXZwZmxYTHBoX1FtZXYtdWN3akoxTVBiaC11eWQ2eEg0RE1OSHFIbko5dTVTWlNfNVYwS0Z5R0kwQUJnUmNLeHpHRlcxSWVTWUItcVJyMUE=
  • token：填写Secret名字；
  • API Token：Kubernetes集群中拥有“ClusterRole/cluster-admin”权限的Serviceaccount使用的Secret的token。

2）同步Namespace、ConfigMap、Ingress配置

  • 「集群管理」-「ENV管理」-「NAMESPACE」下的「odcp」、「cloud」名字空间添加新集群「D3-DUI-正式环境(国科)」；
  • 「集群管理」-「ENV管理」-「CONFIGMAP」，「odcp-config」、「cloud-config」添加新集群「D3-DUI-正式环境(国科)」；
  • 「集群管理」-「集群列表」-「D1-DUI-正式环境(华东)」-「配置更新」，复制「cloud-config」的内容，「D3-DUI-正式环境(国科)」-「配置更新」，编辑「cloud-config」粘贴复制的内容；
  • 「集群管理」-「标签管理」-「INGRESS」，添加新集群「D3-DUI-正式环境(国科)」；
  • 「集群管理」-「Ingress管理」-「Ingress」，将「D1-DUI-正式环境(华东)」中的asr-internal-ingress、asr-lite-ingress-normal、asr-rewrite-ingress复制到「D3-DUI-正式环境(国科)」

3）同步etcd配置

同步「D1-DUI-正式环境(华东)」的「/olive/config」到「D3-DUI-正式环境(国科)」

  • No labels