Warning: Cannot modify header information - headers already sent by (output started at /data/web/virtuals/85063/virtual/www/domains/waldauf.org/lib/plugins/color/syntax.php:1) in /data/web/virtuals/85063/virtual/www/domains/waldauf.org/inc/actions.php on line 180
technology:k8s:etcd - WiKi

ETCD

Respect version which is in documentation.

Install etcd from repository:

  • apt install etcd
  • OR download etcd binary from github, unpack and copy it to /usr/local/bin and create systemd service.

Certificates

Cookbooks how to genereate certificates via openssl or cfssl:

  1. https://coreos.com/kubernetes/docs/latest/openssl.html … official manual page by coreos which I used
  • Create a Cluster Root CA
    openssl genrsa -out ca-key.pem 2048
    openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"

It is really important to define all IPs and FQDNs in certificate.

  • Kubernetes API server keypair
    • OpenSSL Config - openssl.cnf:
      [req]
      req_extensions = v3_req
      distinguished_name = req_distinguished_name
      [req_distinguished_name]
      [ v3_req ]
      basicConstraints = CA:FALSE
      keyUsage = nonRepudiation, digitalSignature, keyEncipherment
      subjectAltName = @alt_names
      [alt_names]
      DNS.1 = master01
      DNS.2 = localhost
      DNS.3 = your.alias
      IP.1 = 192.168.56.101
      IP.2 = 10.3.0.1
      IP.3 = 127.0.0.1
      IP.4 = 0.0.0.0
    • Generate the API server keypair
      openssl genrsa -out apiserver-key.pem 2048
      openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=$(hostname)" -config openssl.cnf
      openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 3650 -extensions v3_req -extfile openssl.cnf


  • Kubernetes Worker (Minion/Node) keypairs
    • worker-openssl.cnf:
      [req]
      req_extensions = v3_req
      distinguished_name = req_distinguished_name
      [req_distinguished_name]
      [ v3_req ]
      basicConstraints = CA:FALSE
      keyUsage = nonRepudiation, digitalSignature, keyEncipherment
      subjectAltName = @alt_names
      [alt_names]
      IP.1 = $ENV::WORKER_IP
    • Generate the Kubernetes Worker Keypairs - make for every worker/minion/node.
      Set WORKER_IP=no default and WORKER_FQDN=no default. The IP addresses and fully qualifed hostnames of all worker nodes will be needed. The certificates generated for the worker nodes will need to reflect how requests will be routed to those nodes. In most cases this will be a routable IP and/or a routable hostname. These will be unique per worker; when you see them used below, consider it a loop and do that step for each worker.
      openssl genrsa -out ${WORKER_FQDN}-worker-key.pem 2048
      WORKER_IP=${WORKER_IP} openssl req -new -key $(hostname)-worker-key.pem -out ${WORKER_FQDN}-worker.csr -subj "/CN=$(hostname)" -config worker-openssl.cnf
      WORKER_IP=${WORKER_IP} openssl x509 -req -in $(hostname)-worker.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out $(hostname)-worker.pem -days 3650 -extensions v3_req -extfile worker-openssl.cnf


  • Generate the Cluster Administrator Keypair
    openssl genrsa -out admin-key.pem 2048
    openssl req -new -key admin-key.pem -out admin.csr -subj "/CN=$(hostname)"
    openssl x509 -req -in admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin.pem -days 3650

Configuration

  • Create kubeadm user:
    useradd -u 10000  -m -s /bin/bash kubeadm
  • Create working dir:
    mkdir -p /appl/kubeadm/etcd
    chown -R kubeadm:kubeadm /appl/kubeadm
  • Systemd: /lib/systemd/system/etcd.service:
    [Unit]
    Description=Etcd Server
    After=network.target
    
    [Service]
    Type=simple
    WorkingDirectory=/appl/kubeadm/etcd
    EnvironmentFile=-/etc/etcd/etcd.conf
    User=kubeadm
    ExecStart=/usr/bin/etcd
    Restart=on-failure
    LimitNOFILE=65536
    
    [Install]
    WantedBy=multi-user.target

    … template is from GIT repository platform-infra.

  • Configuration file /etc/etcd/etcd.conf:
    ETCD_NAME=master01
    ETCD_DATA_DIR=/appl/kubeadm/etcd
    #ETCD_SNAPSHOT_COUNTER="10000"
    #ETCD_HEARTBEAT_INTERVAL="100"
    #ETCD_ELECTION_TIMEOUT="1000"
    #ETCD_MAX_SNAPSHOTS="5"
    #ETCD_MAX_WALS="5"
    #ETCD_CORS=""
    
    #[cluster]
    ETCD_INITIAL_ADVERTISE_PEER_URLS=https://master01:2380
    ETCD_INITIAL_CLUSTER=master01=https://master01:2380
    ETCD_INITIAL_CLUSTER_STATE=new
    ETCD_INITIAL_CLUSTER_TOKEN=etcd-k8-cluster
    #ETCD_DISCOVERY=""
    #ETCD_DISCOVERY_SRV=""
    #ETCD_DISCOVERY_FALLBACK="proxy"
    #ETCD_DISCOVERY_PROXY=""
    
    ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
    
    ETCD_ADVERTISE_CLIENT_URLS=https://master01:2379
    ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
    
    #[proxy]
    ETCD_PROXY="off"
    
    #[security]
    ETCD_CA_FILE=/etc/etcd/certs/ca-master01.pem
    ETCD_TRUSTED_CA_FILE=/etc/etcd/certs/ca-master01.pem
    ETCD_CERT_FILE=/etc/etcd/certs/etcd-server.pem
    ETCD_KEY_FILE=/etc/etcd/certs/etcd-server-key.pem
    ETCD_CLIENT_CERT_AUTH=True
    ## Peer CERTS are used if ETCD is running in cluster
    ETCD_PEER_CA_FILE=/etc/etcd/certs/ca.crt
    ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/certs/ca.crt
    ETCD_PEER_CERT_FILE=/etc/etcd/certs/peer.crt
    ETCD_PEER_KEY_FILE=/etc/etcd/certs/peer.key
    ETCD_PEER_CLIENT_CERT_AUTH=True

Run Etcd

Run over systemd:

systemctl start etcd

Enable start etcd during server boot:

systemctl enable etcd

Problems

Cannot start ETCD service

  • In journalctl -f you can see error:
    Jun 14 09:54:02 master01 systemd[1]: Failed to start Etcd Server.
    Jun 14 09:54:02 master01 systemd[1]: etcd.service: Unit entered failed state.
    Jun 14 09:54:02 master01 systemd[1]: etcd.service: Failed with result 'exit-code'.
    Jun 14 09:54:02 master01 systemd[1]: etcd.service: Service hold-off time over, scheduling restart.
    Jun 14 09:54:02 master01 systemd[1]: Stopped Etcd Server.

    … problem could be caused by missing dir /appl/kubeadm/etcd/ or bad permission on it.

Navigation
Print/export
Toolbox