CIDR Calculation for AWS VPC Subnet

 Following post provide the formulas for calculating CIDR for VPC Subnet.

Considering a VPC with CIDR 172.31.0.0/16. 

1) What will be the CIDR notation for subnets inside 172.31.0.0/16 VPC?

The VPC can be divided into subnets with CIDR notation starting from /17 to /28.

2) What will be the CIDR notation for subnets if we have to divide 172.31.0.0/16 VPC into 'X' number of subnets?

If we have to divide 172.31.0.0/16 into 4 subnets, then following if the formula to calculate network bits (CIDR notation) for subnets. 

2 ^ (32 minus VPC CIDR Notation) / 2 ^ ( 32 minus SUBNET CIDR Notation)  =  NUMBER OF SUBNETS

considering 'SUBNET CIDR Notation' which is to be derived as a variable x, the above formula will be:

2 ^ ( 32 - 16) / 2 ^ (32 - x) = 4

Now consider 32 - x as variable y, above equation will be:

2 ^ 16 / 2 ^ y = 4

2 ^ (16 - y) = 4

which derives, y as 14. So 'x' will be 32 - 14 = 18.

That means, the CIDR notation of 4 subnets inside 172.31.0.0/16 will be /18.

3) How many subnets can be created with specific CIDR notation?

To find out how many subnets can be created for a given CIDR notation, same formula can be used:

2 ^ (32 minus VPC CIDR Notation) / 2 ^ ( 32 minus SUBNET CIDR Notation)  =  NUMBER OF SUBNETS

Considering 'SUBNET CIDR Notation' to be 20, above formula will be :

 2 ^ ( 32 - 16) / 2 ^ (32 - 20) = NUMBER OF SUBNETS

2 ^ 16 / 2 ^ 12 = NUMBER OF SUBNETS

2 ^ (16-12) = NUMBER OF SUBNETS

2 ^ 4 = NUMBER OF SUBNETS

that means, 16 subnets can be created with CIDR notation of /20.

4) How many IP Addressed will be available in subnet with CIDR notation /20?

Formula: 2 ^ ( 32 minus SUBNET CIDR Notation)

with /20, above will be 2 ^ ( 32 - 20 ) i.e. 2 ^ 12 i.e. 4096 IP Addresses

5) How to calculate Subnet addresses for each subnet?

Considering that we decided to divide 172.31.0.0/16 into subnets with /20 CIDR notation, following can be used to determine subnet address for each subnet. 

First subnet address will be 172.31.0.0/20. Since we know there will be total of 4096 IP addresses in this subnet and the range of IP address is 0 to 255 i.e. 256 values on fourth octet, so 4096 / 256 i.e.16 which mean 0 to 15 third octet value will cover total IP address range. 

So next subnet will start from 172.31.(0 + 16).0/20 and so on.

172.31.0.0/20    (IP Range: 172.31.0.0 to 172.31.15.255) 

172.31.16.0/20  (IP Range: 172.31.16.0 to 172.31.31.255) 

172.31.32.0/20

172.31.48.0/20

........

172.31.240.0/20

Hope this helps ✌

Useful Link: https://www.davidc.net/sites/default/subnets/subnets.html

Kubernetes (k8s) Disaster Recovery with Heptio Velero on AWS

Following post list down the steps to perform manual disaster recovery of an kubernetes cluster on AWS using Velero. 

Pre-requisites: Kubernetes cluster is up and running with one master node and one node acting as worker. Please refer following post  to setup kubernetes cluster with kubeadm on AWS free tier. 

AWS CLI is installed 

Kubernetes Cluster with kubeadm

1) Create s3 bucket to be used for backup.

export BUCKET=velero-backup-bkt

export REGION=ap-southeast-2

aws s3api create-bucket --bucket $BUCKET --region $REGION --create-bucket-configuration LocationConstraint=$REGION 

2) Create IAM role to access bucket to be used by Velero

cat > assume-role-policy-document.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/InstanceRole"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role --role-name velero \
  --assume-role-policy-document \
  file://assume-role-policy-document.json

Where, InstanceRole,  is the role assigned to worker node EC2 instance. If no role is assigned, then create a new role.

 

3) Create and assign policy to above role to access S3

cat > velero-trust-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}
EOF

aws iam put-role-policy \
  --role-name velero \
  --policy-name s3 \
  --policy-document file://velero-trust-policy.json

4) Download velero client

wget https://github.com/vmware-tanzu/velero/releases/download/v1.3.2/velero-v1.3.2-linux-amd64.tar.gz

tar -xvf velero-v1.3.2-linux-amd64.tar.gz -C /tmp

sudo mv /tmp/velero-v1.3.2-linux-amd64/velero /usr/local/bin

5) Install velero

velero install \
	--provider aws \
	--plugins velero/velero-plugin-for-aws:v1.0.1 \
	--bucket ${BUCKET} \
	--backup-location-config region=${REGION} \
	--snapshot-location-config region=${REGION} \
	--pod-annotations iam.amazonaws.com/role=arn:aws:iam::<ACCOUNT_ID>:role/velero \
	--no-secret

kubectl get all -n velero

6) Install sample NGINX application on worker node that will serve as our test application to perform disaster recovery.

cat > nginx-deployment.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-example
  labels:
    app: nginx

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx-example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.17.6
        name: nginx
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: my-nginx
  namespace: nginx-example
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
EOF

kubectl apply -f nginx-deployment.yaml

Verify all the resources are up and running

kubectl get all -n nginx-example

7) Create backup either based on app selector or namespace

velero backup create nginx-backup --selector app=nginx
OR
velero backup create nginx-backup --include-namespaces nginx-example

Check the status. Should show completion time once completed successfully.

velero backup describe nginx-backup

To create scheduled backups

velero create schedule daily-backup-at-7am --schedule="0 7 * * *" --include-namespaces nginx-example

8) Check S3 bucket to confirm backup is created:

9) Now to simulate disaster, lets delete nginx-example namespace. This will delete deployment and all running NGINX pods

kubectl delete namespace nginx-example

10) Restore all resources from backup

velero restore create --from-backup nginx-backup

Check restore status:

velero restore describe nginx-backup-20200930060821

Once restore completes, exactly same nginx resources should be up and running.

To restore scheduled backup. We can either do it using specific backup or with latest backup created by schedule:

#Specific backup
velero restore create --from-backup <BACKUP_NAME>
# Latest backup from schedule
velero restore create --from-schedule <SCHEDULE-NAME>

✌

Kubernetes (k8s) Cluster with Personal Docker Registry on AWS

This post describes the steps to deploy an application on kubernetes cluster after reading from personal docker registry server. 

Pre-requisites: Kubernetes cluster is up and running with one node acting as worker. Please refer following post  to setup kubernetes cluster with kubeadm.

Kubernetes Cluster with kubeadm

1) Install personal docker registry server on master node

Following setup are for insure docker registry for testing purpose only:

a) Setup basic authentication for docker registry login with root as username and welcome as password.

mkdir auth

docker run --entrypoint htpasswd registry:2.7.0 -Bbn root welcome > auth\htpasswd

Following error is encountered with registry:2. So use registry:2.7.0 version

"docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"htpasswd\": executable file not found in $PATH": unknown."

b) Allow insecure access to repository. Edit the daemon.json file at /etc/docker/daemon.json and add following. If the daemon.json file does not exist, we can create it.

{
  "insecure-registries" : ["registrydomain:5000"]
}

where, registrydomain in case of AWS EC2 instance can be instance private IP.

c) Reload docker service.

service docker reload

d) Start docker registry container on master node:

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v "$(pwd)"/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  registry:2.7.0

On AWS EC2 security group, allow inbound access on port 5000. 

e) To run docker registry on one of the worker node as pod, create following deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - name: registry
        image: registry:2.7.0
        volumeMounts:
        - name: auth
          mountPath: /auth
        env: 
        - name: REGISTRY_AUTH
          value: "htpasswd"
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: "Registry Realm"
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: "/auth/htpasswd"
        ports:
        - containerPort: 5000
      volumes:
      - name: auth
        hostPath:
          # directory location on host
          path: /home/ubuntu/auth
          # this field is optional
          type: Directory

htpasswd file must be available on worker node. 

Now personal docker registry is up and running on master node. 

2) Push image into personal docker registry

a) Perform same steps as above to setup personal registry as insure on worker node by editing daemon.json file and adding insecure-registries entry. Reload docker service.

b) Pull the image from docker hub and push it to personal registry:

#Pull image from docker hub
docker pull nginx
# Tag the image
docker tag nginx:latest registrydomain:5000/my-nginx
# Login to personal registry
docker login registrydomain:5000
# Push re-tagged image
docker push registrydomain:5000/my-nginx

3) Verify image in docker registry on master node

# Login to personal registry
docker login registrydomain:5000
# Pull image
docker pull registrydomain:5000/my-nginx

4) Run application inside kubernetes cluster

a) Create deployment to run pod with container image pulled from personal registry:  

cat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: registrydomain:5000/my-nginx
        ports:
        - containerPort: 80
EOF

b) Expose the deployment as NodePort type service

cat <<EOF >service.yaml
apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - nodePort: 31000
    port: 80
    targetPort: 80
EOF

c) Go to AWS console, and enable inbound for port 31000 on worker node EC2 security group

d) Access nginx from browser with worker node public IP.

We now have application pull from personal registry running in kubernetes cluster. ✌

Kubernetes (k8s) Cluster with Kubeadm on AWS

This post list down the steps for creating Kubernetes cluster on AWS EC2 instance using Kubeadm. The setup will include one master node, one worker node and one sample application running on worker node accessible from browser. The setup is eligible for free tier AWS account.

1) Launch Ubuntu EC2 instance for master node

2) Install kubeadm and docker on master node

swapoff -a

sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg2

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF

apt-get update && apt-get install -y kubelet kubeadm kubectl docker.io

service docker start

3) Setup control-plane with kubeadm init

Cluster setup with kubeadm requires a minimum of 2 CPUs on host machine. So running "kubeadm init" on free tier eligible EC2 instance will result in following error:

[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

To setup cluster on machine with 1 CPU, run kubeadm init with --ignore-preflight-errors=all. This will ignore the error and continue cluster setup. 

kubeadm init --ignore-preflight-errors all

Once done, control-plane should get initialized successfully with following message

Now execute the commands as mentioned in above initialization message.

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Next deploy a pod network to cluster as mentioned in cluster initialization message:

Following is the setup for Weave Net. Weave Net provides a network to connect all pods together, implementing the Kubernetes model. Kubernetes uses the Container Network Interface (CNI) to join pods onto Weave Net. Weave Net can be installed onto your CNI-enabled Kubernetes cluster with command:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Check status of master node and all pods. (It took some time for coredns to get started. EC2 instance was also restarted in between)

Now we are ready to join worker nodes in the cluster. 

4) Setup worker node

1) Launch another Ubuntu EC2 instance

 

2) Install kubeadm and docker

swapoff -a

sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg2

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF

apt-get update && apt-get install -y kubelet kubeadm kubectl docker.io

service docker start

For EC2 instance based on Amazon Linux AMI , the run following commands to install kubeadm

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

yum install kubeadm

If above command on Amazon Linux instance fails with following error:

Error: Package: kubelet-1.19.2-0.x86_64 (kubernetes)

           Requires: iptables >= 1.4.21

           Installed: iptables-1.4.18-1.22.amzn1.x86_64 (installed)

               iptables = 1.4.18-1.22.amzn1

then update iptables package with:

yum install 'ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.3/x86_64/os/Packages/iptables-1.4.21-17.el7.x86_64.rpm'

3) Go to AWS console, and enable inbound access for port 6443 on Master EC2 instance security group.

4) Create kube config file. We can either use "kubectl config" command, or copy $HOME/.kube/config file from master node to worker node. So copy file from master node.

5) Now join the worker node to cluster with command displayed in kubeadm init output. If token is not know, then "kubeadm token list" command can be used on master node to list token.

systemctl enable docker.service

kubeadm join 172.31.0.8:6443 --token hkjm3g.kj7wwxj4lcc7x0y2 --discovery-token-ca-cert-hash sha256:88706a6fcd1a5a8c0c35f3311647f25a70359194e40a88ff3a4ace28f238abd9

 

Run "kubectl get nodes" on control plane or worker node to confirm worker node has joined successfully

4) Deploy sample application on worker node

a) Create deployment with nginx image

cat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
EOF

2) Expose the deployment as NodePort type service

cat <<EOF >service.yaml
apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - nodePort: 31000
    port: 80
    targetPort: 80
EOF

3) Go to AWS console, and enable inbound for port 31000 on worker node EC2 security group

4) Access nginx from browser with worker node public IP.

 

We have now a Kubernetes cluster configured with one master node, one worker node and sample application running on worker node. ✌