Reducing project infrastructure costs by time based scaling for Kubernetes

January 1, 2021

Kubernetes

Reducing infrastructure costs is quite simple: just make sure your application environments are not running, when you don't need them. Shutting down a single VM has an immediate effect on the operation costs. Undeploying a Kubernetes environment allows the IT team to implement and benefit from a cluster auto-downscaling policy. Even shutting down an environment at night and weekends can save the organization a lot of money.

The Problem

Scaling resources on Kubernetes can be quite tricky. The "Horizontal Pod Autoscaler" is a good solution to scale the number of pods based on observed CPU utilization in Production. Normally ony 1 instance of each pod is required in a development environment. The HPA can down scale your deployment, but you can't set the minimum pods to 0.

So, how can Pods be scaled based on time? What if you want to scale down the pods at night or to scale up a deployment for the same peak hours every day?

How to solve it?

We can use Kubernetes CronJobs to scale deployments up or down. The bitnami/kubectl:latest provides us a pod with the kubectl tool, which allows us to run commands from a cron job directly on the Kubernetes platform. To scale the deployment we use the following command kubectl scale deployment <deployment-name> --replicas=<number-of-pods>. If you need to scale a group of deployment, you can use the selector argument. For example the -l environment=dev will select all deployments with the dev environment label.

The Cron expression is a bit tricky again. The times are based on the timezone of the kube-controller-manager. If not customized the default UTC timezone is used.

Of course the job requires a service account with the permission to manipulate the deployment specification.

Stop all pods at night

Scale down all deployments in the dev environment at 7pm UTC

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: scale-down-job
spec:
  schedule: "0 19 * * 1-5"  # <1>
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure # <2>
          containers:
          - name: kubectl
            image: bitnami/kubectl:latest # <3>
            imagePullPolicy: IfNotPresent
            args: 
              - scale
              - deployment
              - -l
              - environment=dev # <4>
              - --replicas=0 # <5>
          serviceAccountName: cicd # <6>

<1> Cron expression to run every working day at 7pm UTC
<2> The default restartPolicy is not allowed and must be set to OnFailure or Never
<3> Docker image with the kubectl tool
<4> Selector to match all deployments with label environment=dev
<5> Set the number of pods. Use 0 to remove all pods.
<6> Service account to run the job. Requires permission to manipulate the deployment specs.

Scale up all deployments in the dev environment at 7am UTC

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: scale-up-job
spec:
  schedule: "0 7 * * 1-5" # <7>
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: kubectl
            image: bitnami/kubectl:latest
            imagePullPolicy: IfNotPresent
            args: 
              - scale
              - deployment
              - -l
              - environment=dev
              - --replicas=1
          serviceAccountName: cicd

<7> Cron expression to run every working day at 7am UTC