My experience implementing health check for gRPC microservices using readinessProbe
In this blog, I will share my experience with the readinessProbe for my gRPC-based microservice, running as a Pod in a K98s cluster; wherein my microservice is dependent on multiple external services -such as IBM Cloudant. As we all know production services might take some time to establish the connection to external services and sometimes it’s possible that backend services are down. To provide a better user experience and service availability it’s always a good practice to ensure all the underlying services are up and running before we allow any traffic to the pod. To implement health-checks for external services I am using readinessProbe feature provided by Kubernetes.
I just want to give an overview to all of you about the probe and type of probes provided by Kubernetes.
ProbeProbe describes a health check to be performed against a container to determine whether it is alive or ready to receive traffic. There are three kinds of probes namely, startupProbe, livenessProbe, and readinessProbe described below.
startupProbe: Thekubeletuses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.
readinessProbe: The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.
For example, an application might need to load large data or configuration files during startup or depend on external services like IBM Cloudant in our case after startup. In such cases, you don’t want to kill the application and at the same time don’t want to send requests to the pod. Kubernetes readinessProbe plays a vital role in this case.
Note: Readiness probes run on the container during its whole lifecycle.
livenessProbe : Thekubeletuses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.
Let’s talk about readinessProbenow as we have little understanding of the probes and how I have implemented it for checking the health of my DBaaS dependent services.
How does it work in gRPC servers?
Implementing a probe is a little complex in the case of gRPC servers as compared to application serving HTTP/1.2 traffic or by doing a command execution. To resolve this complexity, We can create a proto file for a health check as mentioned below. I will be implementing only one RPC function “Check” as per my requirement. If you want to use for any streaming services then you can use “Watch” RPC along with “Check”.
Create a proto file:
healthcheck.proto under <git url of root dir>/proto/grpc/health directory
To generate pb file execute the below command:
protoc — go_out=plugins=grpc:. proto/grpc/health/healthcheck.proto
Now it’s time to provide the implementation of the “Check” health service as mentioned below:
To make code more readable we can create a different package “health-service”
You can use the below code format to call the Cloudant database and check the connection with your Cloudant.
Snippets are mentioned below:
Now we have implemented the “NewHealthServiceChecker” function which will return HealthServiceChecker. We can call it from the server module and register as mentioned below:
We are almost ready to test our gRPC health check service. We will be using grpc_health_probeprovided by gRPC Ecosystemin GitHub here. To add binary to your microservice, we need to add the below snippet in the Dockerfile.
The grpc_health_probe utility allows you to query the health of gRPC services that expose service their status through the gRPC Health Checking Protocol.
once the image is built with the grpc_health_probeclient we need to define readinessProbe in the deployment.yaml.
Defining the probes:
To use the binary, we just need to define the exec action for the probes with the grpc_health_probe binary as follows.
Probes have a number of fields that you can use to more precisely control the behavior of liveness and readiness checks:
initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. The minimum value is 0.
periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. The minimum value is 1.timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. The minimum value is 1.
successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness. The minimum value is 1.
failureThreshold: When a probe fails, Kubernetes will tryfailureThresholdtimes before giving up. Giving up in case of a liveness probe means restarting the container. In the case of readiness probe, the Pod will be marked Unready. Defaults to 3. The minimum value is 1.
Based on the result of the probes like SERVING or NOT_SERVING, the kubelet will perform its action further. Thus our services that are built with gRPC are better served with the probes provided by Kubernetes.
I hope the above-mentioned details are sufficient to head-on for health check implementation using readiness probe for gRPC microservices.
References:
https://github.com/grpc-ecosystem/grpc-health-probe
https://github.com/IBM-Cloud/go-cloudant/blob/master/cloudant.go
