Kubernetes (k8s) Cluster with Personal Docker Registry on AWS

This post describes the steps to deploy an application on kubernetes cluster after reading from personal docker registry server. 

Pre-requisites: Kubernetes cluster is up and running with one node acting as worker. Please refer following post  to setup kubernetes cluster with kubeadm.

Kubernetes Cluster with kubeadm

1) Install personal docker registry server on master node

Following setup are for insure docker registry for testing purpose only:

a) Setup basic authentication for docker registry login with root as username and welcome as password.

mkdir auth

docker run --entrypoint htpasswd registry:2.7.0 -Bbn root welcome > auth\htpasswd

Following error is encountered with registry:2. So use registry:2.7.0 version

"docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"htpasswd\": executable file not found in $PATH": unknown."

b) Allow insecure access to repository. Edit the daemon.json file at /etc/docker/daemon.json and add following. If the daemon.json file does not exist, we can create it.

{
  "insecure-registries" : ["registrydomain:5000"]
}

where, registrydomain in case of AWS EC2 instance can be instance private IP.

c) Reload docker service.

service docker reload

d) Start docker registry container on master node:

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v "$(pwd)"/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  registry:2.7.0

On AWS EC2 security group, allow inbound access on port 5000. 

e) To run docker registry on one of the worker node as pod, create following deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - name: registry
        image: registry:2.7.0
        volumeMounts:
        - name: auth
          mountPath: /auth
        env: 
        - name: REGISTRY_AUTH
          value: "htpasswd"
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: "Registry Realm"
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: "/auth/htpasswd"
        ports:
        - containerPort: 5000
      volumes:
      - name: auth
        hostPath:
          # directory location on host
          path: /home/ubuntu/auth
          # this field is optional
          type: Directory

htpasswd file must be available on worker node. 

Now personal docker registry is up and running on master node. 

2) Push image into personal docker registry

a) Perform same steps as above to setup personal registry as insure on worker node by editing daemon.json file and adding insecure-registries entry. Reload docker service.

b) Pull the image from docker hub and push it to personal registry:

#Pull image from docker hub
docker pull nginx
# Tag the image
docker tag nginx:latest registrydomain:5000/my-nginx
# Login to personal registry
docker login registrydomain:5000
# Push re-tagged image
docker push registrydomain:5000/my-nginx

3) Verify image in docker registry on master node

# Login to personal registry
docker login registrydomain:5000
# Pull image
docker pull registrydomain:5000/my-nginx

4) Run application inside kubernetes cluster

a) Create deployment to run pod with container image pulled from personal registry:  

cat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: registrydomain:5000/my-nginx
        ports:
        - containerPort: 80
EOF

b) Expose the deployment as NodePort type service

cat <<EOF >service.yaml
apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - nodePort: 31000
    port: 80
    targetPort: 80
EOF

c) Go to AWS console, and enable inbound for port 31000 on worker node EC2 security group

d) Access nginx from browser with worker node public IP.

We now have application pull from personal registry running in kubernetes cluster. ✌