Warning: Cannot modify header information - headers already sent by (output started at /data/web/virtuals/85063/virtual/www/domains/waldauf.org/lib/plugins/color/syntax.php:1) in /data/web/virtuals/85063/virtual/www/domains/waldauf.org/inc/actions.php on line 180
technology:k8s:flanneld - WiKi

Flannel

Source:

Flannel must run on the master(s) and every minion

Installation

Download Flannel binary from repository: https://github.com/coreos/flannel/releases

Unpack and copy to /usr/local/bin.


Certificates

Flanneld connects to etcd so you have to generate client certs:

  • Define used your CA cert and CA key:
    CAcert="/appl/K8s_configuration/certs/ca/ca-master01.pem"
    CAkey="/appl/K8s_configuration/certs/ca/ca-master01-key.pem"
  • Generate certs - MUST BE DONE FOR EVERY MASTER/MINION::
    for HOSTNAME in master01 node01; do
      openssl genrsa -out etcd-client-${HOSTNAME}-key.pem 2048
      openssl req -new -key etcd-client-${HOSTNAME}-key.pem -subj "/CN=${HOSTNAME}" -out etcd-client-${HOSTNAME}.csr
      openssl x509 -req -in etcd-client-${HOSTNAME}.csr -CA ${CAcert} -CAkey ${CAkey} -CAcreateserial -out etcd-client-${HOSTNAME}.pem -days 3650
      openssl x509 -noout -text -in etcd-client-${HOSTNAME}.pem
    done

Copy certificates on every master/minion to /etc/flanneld/certs.

Configuration

If your primary network interface (NI) is another than you use for communication between servers - YOU HAVE TO SPECIFY WHICH NI MUST FLANNELD USE. In my case my primary interface is NAT (for communication with inetrnet) and second VboxHostOnly Adapter (for communication between servers).

Create Flannel network configuration file for K8s: flannel-config.json

cat > /tmp/flannel-config.json << EOF
{
  "Network": "172.17.0.0/16",
  "SubnetLen": 24,
  "Backend": {
    "Type": "vxlan"
  }
}
EOF


Add the network configuration to etcd.

# etcdctl set <CLUSTER_NAME>/network/config < flannel-config.json
  • If etcd is running on TLS you must use etcd certificate:
    etcdctl --ca-file /etc/flanneld/certs/ca-master01.pem --cert-file /etc/flanneld/certs/etcd-client-master01.pem --key-file /etc/flanneld/certs/etcd-client-master01-key.pem --endpoint <ETCD_ADVERTISE_CLIENT_URLS> set <CLUSTER_NAME>/network/config < /tmp/flannel-config.json


You can check validate the etcd entry by running the following command:

  • List your registry:
    etcdctl ls 
  • Display your network configuration:
    etcdctl get <CLUSTER_NAME>/network/config
    {
      "Network": "172.17.0.0/16",
      "SubnetLen": 24,
      "Backend": {
        "Type": "vxlan"
      }
    }

Configuration of Flannel - The /etc/flannel/flanneld configuration file looks like the following:

# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="https://master01:2379"
FLANNEL_ETCD_ENDPOINTS="https://master01:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/<CLUSTER_NAME>/network"
FLANNEL_ETCD_PREFIX="/<CLUSTER_NAME>/network"

FLANNELD_ETCD_CAFILE="/etc/flanneld/certs/ca-master01.pem"
FLANNELD_ETCD_CERTFILE="/etc/flanneld/certs/etcd-client-master01.pem"
FLANNELD_ETCD_KEYFILE="/etc/flanneld/certs/etcd-client-master01-key.pem"

# Any additional options that you want to pass
# By default, we just add a good guess for the network interface on Vbox.  Otherwise, Flannel will probably make the right guess.
# If your NETWORK INTERFACE is not primary - YOU HAVE TO USE option ''--iface=NI|IP''. It says to Flanneld which NI use for communication between servers.
FLANNEL_OPTIONS="--ip-masq -v=10"


Systemd

Configure systemd for Flanneld: /lib/systemd/system/flanneld.service:

[Unit]
After=network-online.target
Wants=network-online.target
Description=flannel is an etcd backed overlay network for containers

[Service]
Type=notify
EnvironmentFile=/etc/flanneld/flanneld
ExecStart=/usr/bin/flanneld --etcd-endpoints=${FLANNEL_ETCD} --etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS
ExecStartPost=/usr/bin/mk-docker-opts.sh -d /run/flannel_docker_opts.env -i

[Install]
WantedBy=multi-user.target

Script for generating Docker daemon options - /usr/local/bin/mk-docker-opts.sh:

#!/bin/sh
 
usage() {
	echo "$0 [-f FLANNEL-ENV-FILE] [-d DOCKER-ENV-FILE] [-i] [-c] [-m] [-k COMBINED-KEY]
 
Generate Docker daemon options based on flannel env file
OPTIONS:
	-f	Path to flannel env file. Defaults to /run/flannel/subnet.env
	-d	Path to Docker env file to write to. Defaults to /run/docker_opts.env
	-i	Output each Docker option as individual var. e.g. DOCKER_OPT_MTU=1500
	-c	Output combined Docker options into DOCKER_OPTS var
	-k	Set the combined options key to this value (default DOCKER_OPTS=)
	-m	Do not output --ip-masq (useful for older Docker version)
" >&2
 
	exit 1
}
 
flannel_env="/run/flannel/subnet.env"
docker_env="/run/docker_opts.env"
combined_opts_key="DOCKER_OPTS"
indiv_opts=false
combined_opts=false
ipmasq=true
 
while getopts "f:d:icmk:?h" opt; do
	case $opt in
		f)
			flannel_env=$OPTARG
			;;
		d)
			docker_env=$OPTARG
			;;
		i)
			indiv_opts=true
			;;
		c)
			combined_opts=true
			;;
		m)
			ipmasq=false
			;;
		k)
			combined_opts_key=$OPTARG
			;;
		[\?h])
			usage
			;;
	esac
done
 
if [ $indiv_opts = false ] && [ $combined_opts = false ]; then
	indiv_opts=true
	combined_opts=true
fi
 
if [ -f "$flannel_env" ]; then
	. $flannel_env
fi
 
if [ -n "$FLANNEL_SUBNET" ]; then
	DOCKER_OPT_BIP="--bip=$FLANNEL_SUBNET"
fi
 
if [ -n "$FLANNEL_MTU" ]; then
	DOCKER_OPT_MTU="--mtu=$FLANNEL_MTU"
fi
 
if [ -n "$FLANNEL_IPMASQ" ] && [ $ipmasq = true ] ; then
	if [ "$FLANNEL_IPMASQ" = true ] ; then
		DOCKER_OPT_IPMASQ="--ip-masq=false"
	elif [ "$FLANNEL_IPMASQ" = false ] ; then
		DOCKER_OPT_IPMASQ="--ip-masq=true"
	else
		echo "Invalid value of FLANNEL_IPMASQ: $FLANNEL_IPMASQ" >&2
		exit 1
	fi
fi
 
eval docker_opts="\$${combined_opts_key}"
 
if [ "$docker_opts" ]; then
	docker_opts="$docker_opts ";
fi
 
echo -n "" >$docker_env
 
for opt in $(set | grep "DOCKER_OPT_"); do
 
	OPT_NAME=$(echo $opt | awk -F "=" '{print $1;}');
	OPT_VALUE=$(eval echo "\$$OPT_NAME");
 
	if [ "$indiv_opts" = true ]; then
		echo "$OPT_NAME=\"$OPT_VALUE\"" >>$docker_env;
	fi
 
	docker_opts="$docker_opts $OPT_VALUE";
 
done
 
if [ "$combined_opts" = true ]; then
	echo "${combined_opts_key}=\"${docker_opts}\"" >>$docker_env
fi

Run Flannel

Run over systemd:

systemctl start flanneld

Enable start flanneld during server boot:

systemctl enable flanneld


If everything is alright then you should see new NI:

X: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 66:d6:29:f9:26:62 brd ff:ff:ff:ff:ff:ff
    inet 10.244.48.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
Navigation
Print/export
Toolbox