You Select Me - Deploy Applications on desired OpenShift Nodes
This article describes how to deploy pods (applications) on the desired node or nodes. OpenShift uses K8S (Kubernetes) to do that, so we will also cover the K8S basics about that.
- We have one OpenShift node with a persistent storage problem.
- The storage provisioning works, but K8S could not mount the storage from that particular node.
- Our cluster consists of three master nodes and six compute nodes.
- One of the six compute nodes is faulty.
- We have a probability of 1/6 (16.667%) of faulty deployments.
We narrow down the problem to one specific node. Now we are going to troubleshoot the node but must ensure that the deployment gets only on that node.
Table of Contents
Node Management
To list the nodes in the OpenShift cluster, use this command:
oc get nodes
Our third node in data centre 2 is the problem.
NAME STATUS ROLES AGE VERSION
dc-1-node-1 Ready compute 114d v1.11.0+d4cacc0
dc-1-node-2 Ready compute 114d v1.11.0+d4cacc0
dc-1-node-3 Ready compute 114d v1.11.0+d4cacc0
dc-2-node-1 Ready compute 114d v1.11.0+d4cacc0
dc-2-node-2 Ready compute 114d v1.11.0+d4cacc0
dc-2-node-3 Ready,SchedulingDisabled compute 114d v1.11.0+d4cacc0
master-1 Ready infra,master 114d v1.11.0+d4cacc0
master-2 Ready infra,master 114d v1.11.0+d4cacc0
master-3 Ready infra,master 114d v1.11.0+d4cacc0
After we have identified the culprit, we can prevent the deployment of new pods to that node with:
oc adm manage-node dc-2-node-3 --schedulable=false
If we want to troubleshoot, we make it schedulable again with this command:
oc adm manage-node dc-2-node-3 --schedulable
Label Node for troubleshooting
We label the node for the troubleshooting deployment, with the label type and the value problem. You can use any other suitable label and value.
oc label nodes dc-2-node-3 type=problem
node/dc-2-node-3 labeled
Labels are a K8S concept. Labelling an object in OpenShift or Kubernetes is an excellent method to organise, group, or select API objects.
Verify label is applied
After the labelling, we double check by looking into the node description.
oc describe node dc-2-node-3 | head
Name: dc-2-node-3
Roles: compute
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=dc-2-node-3
logging-infra-fluentd=true
node-role.kubernetes.io/compute=true
type=problem
Troubleshooting
We create new test project.
oc new-project friendly-broccoli
We use a PostgreSQL deployment with persistent storage to troubleshoot and test the storage capability of each node. Any other deployment of an application with persistent storage would also do.
- We create a new template for PostgreSQL.
- We use as a basis the template from Origin (the OpenShift OpenSource community).
- Download the template and extend the template with the
nodeSelectorkey to our previously labelled node. - The
nodeSelectorbelongs to thespecroot in theservicedefinition.
{
"spec": {
"nodeSelector": {
"type": "problem"
}
}
}
Create the new template:
oc create -f postgresql-persistent-template.json
We can check the deployment parameters of the template with their default values:
oc process --parameters -n friendly-broccoli postgresql-persistent
NAME DESCRIPTION VALUE
MEMORY_LIMIT Maximum amount of memory the container can use. 512Mi
NAMESPACE The OpenShift Namespace where the ImageStream resides. openshift
DATABASE_SERVICE_NAME The name of the OpenShift Service exposed for the database. postgresql
POSTGRESQL_USER Username for PostgreSQL user that will be used for accessing the database. user[A-Z0-9]{3}
POSTGRESQL_PASSWORD Password for the PostgreSQL connection user. [a-zA-Z0-9]{16}
POSTGRESQL_DATABASE Name of the PostgreSQL database accessed. sampledb
VOLUME_CAPACITY Volume space available for data, e.g. 512Mi, 2Gi. 1Gi
POSTGRESQL_VERSION Version of PostgreSQL image to be used (10 or latest). 10
Test Deployment
We create a new deployment for testing:
oc new-app postgresql-persistent -p POSTGRESQL_USER=friend -p POSTGRESQL_PASSWORD=sweet
The OpenShift cluster logs:
--> Deploying template "friendly-broccoli/postgresql-persistent" to project friendly-broccoli
PostgreSQL
---------
PostgreSQL database service, with persistent storage. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/.
NOTE: Scaling to more than one replica is not supported. You must have persistent volumes available in your cluster to use this template.
The following service(s) have been created in your project: postgresql.
Username: friend
Password: sweet
Database Name: sampledb
Connection URL: postgresql://postgresql:5432/
For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/postgresql-container/.
* With parameters:
* Memory Limit=512Mi
* Namespace=openshift
* Database Service Name=postgresql
* PostgreSQL Connection Username=friend
* PostgreSQL Connection Password=sweet
* PostgreSQL Database Name=sampledb
* Volume Capacity=1Gi
* Version of PostgreSQL Image=10
--> Creating resources ...
secret "postgresql" created
service "postgresql" created
persistentvolumeclaim "postgresql" created
deploymentconfig.apps.openshift.io "postgresql" created
--> Success
Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
'oc expose svc/postgresql'
Run 'oc status' to view your app.
We now can check on which node the pod runs:
oc get pods -o wide
The pod is on the labelled node. OpenShift tries to deploy PostgreSQL on the dedicated node without success.
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
postgresql-1-4kt8t 0/1 ContainerCreating 0 10s <none> dc-2-node-3 <none>
postgresql-1-deploy 1/1 Running 0 12s 10.132.2.83 dc-2-node-3 <none>
We expected the above outcome, and now we can troubleshoot the events:
oc get events
The events indicate our problem.
2m 2m 1 postgresql-1-4kt8t.15b9495f8da0acd4 Pod
Normal SuccessfulAttachVolume attachdetach-controller
AttachVolume.Attach succeeded for volume "pvc-dcf1bb72-bab6-11e9-ac47-005056ab11cb"
45s 45s 1 postgresql-1-4kt8t.15b9497c12ede9b2 Pod
Warning FailedMount kubelet, dc-2-node-3
Unable to mount volumes for pod "postgresql-1-friendly-broccoli(ded32df0-bab6-11e9-a7eb-005056abfe40)":
timeout expired waiting for volumes to attach or mount for pod "friendly-broccoli"/"postgresql-1-4kt8t".
list of unmounted volumes=[postgresql-data]. list of unattached volumes=[postgresql-data default-token-c26w7]
Summary
In OpenShift, it is useful to have pods land on specific sets of nodes for troubleshooting and isolation purposes.
- We use during runtime, labels and selectors to deploy the PostgreSQL example on a specific node.
- We had no downtime! There was no need to restart a node. One major strength of K8s.
- This post demonstrates how to use labels for troubleshooting.
- Another scenario is to label infrastructure or database nodes in production for the distinctive deployment of PostgreSQL.
If you are wondering if we could solve the persistent storage problem, look into this previous blog post.