In the previous post, we laid out the foundations to create our own custom Kubernetes controller. We detailed what a controller was, and that its only requirement is to be able to communicate with HTTP/JSON. In this post, we are going to finally start developing it.
The technology stack can be Python, NodeJS or Ruby. Because this blog is named "A Java Geek", it’s normal to choose Java.
As a use-case, we will implement the sidecar pattern: every time a pod gets scheduled, a sidecar pod will be scheduled along it as well. If the former is removed, the latter needs to be as well.
Choosing the right tool
In order to execute REST calls in Java, one needs to generate the bindings first. There are several ways to get those: . The most tedious one is to do that manually: one needs to carefully get hold of all possible JSON request and response combinations, develop the corresponding Java objects, choose the JSON serialization framework, as well as the HTTP client. . The next best option is to use proprietary code generators such as Swagger or Apiary. This requires the API provider to provide the model in one of the possible formats. On the downside, one needs to use the relevant tool. Sometimes, the format is more or less open, such as the OpenAPI specification. In that case, the tool can be chosen among those that implement the format. . In the best of cases, the bindings are already provided.
This is the case with Kubernetes: the project provides their own bindings, for a variety of languages. The issue is that the language wrapper is very close to the REST API, too close for my taste. For example, this is how one lists all pods across all namespaces:
ApiClient client = Config.defaultClient();
CoreV1Api core = new CoreV1Api(client);
V1PodList pods =
core.listPodForAllNamespaces(null, null, null, null, null, null, null, null); (1)| 1 | Notice all the null parameters that need to be passed |
This is what was meant by "the wrapper code being very close to the REST API"". Fortunately, there’s another option available. The Fabric8 organization offers a fluent Java API on Github. The code equivalent to the above one is:
KubernetesClient client = new DefaultKubernetesClient();
PodList pods = client.pods().inAnyNamespace().list(); (1)| 1 | Fluent, no need to pass useless null parameters |
A quick overview of Fabric8
In a few words, with Fabric8’s API, all Kubernetes resources are available on the KubernetesClient instance e.g.:
client.namespaces()client.services()client.nodes()- etc.
Depending on the nature of the resource, it can be scoped by a namespace - or not:
client.pods().inAnyNamespace()client.pods().inNamespace("ns")
At that point, the verb can be invoked:
- List all pods in all namespaces
-
client.pods().inAnyNamespace().list(); - Delete all pods in the namespace
ns -
client.pods().delete(client.pods().inNamespace("ns").list().getItems()); - Create a new namespace with the name
ns -
client.namespaces() .createNew() .withApiVersion("v1") .withNewMetadata() .withName("ns") .endMetadata() .done();
Implementing the control loop
Remember that a Kubernetes controller is just a control loop that watches the state of the cluster, and reconciles it with the desired state.
In order to be aware of scheduling/deleting events, one needs the Observer pattern.
The application will subscribe to such events, and the relevant callbacks will be triggered when they happen.
This class diagram is a very simplified diagram of the API:
To actually implement a watcher is just a matter of the following lines:
public class DummyWatcher implements Watcher<Pod> {
@Override
public void eventReceived(Action action, Pod pod) {
switch (action) {
case ADDED: (1)
break;
case MODIFIED: (2)
break;
case DELETED: (3)
break;
case ERROR: (4)
break;
}
}
@Override
public void onClose(KubernetesClientException cause) {
(5)
}
}
client.pods()
.inAnyNamespace()
.watch(DummyWatcher());| 1 | Act when a new pod is added |
| 2 | Act when an existing pod is modified |
| 3 | Act when a pod is deleted |
| 4 | Act in case of error |
| 5 | Clean up any resource.
If the client closes correctly, cause will be null |
The nitty-gritty details
At this point, we have everything that is required to implement the sidecar pattern. I won’t show the whole code - it’s available on GitHub, but a few key things need to be highlighted.
Tagging the sidecar
At its core, the watcher needs to add a sidecar pod when a new pod is added, and remove the sidecare when it’s removed. This basic approach doesn’t work: if a sidecar pod is scheduled, then the watcher will be triggered, and it will add a new sidecar pod to the sidecar. And this will go on and on… Thus, it’s of utmost importance to "tag" sidecars pods. When such a pod is detected, the creation logic shouldn’t be triggered.
There are several ways to tag a sidecar pod:
- Suffixing sidecar pods' name with a specific string e.g.
sidecar - Adding specific labels:
client.pods() .inNamespace("ns") .createNew() .withNewMetadata() .addToLabels("sidecar", "true") .endMetadata() .done();
Removing the sidecar along with the pod
A pod should have one and only one sidecar. It should be created when the pod is added, and should be deleted when the later is deleted as described above.
Hence, a reference to the main pod should be added to the sidecar. This way, when a pod is deleted - and when it’s not a sidecar, we should find the assigned sidecar and delete it as well.
The first naive approach is to explicitly delete the sidecar when the main pod is deleted.
However, this is a lot of work, for not much.
Kubernetes allows to bind the lifecycle a pod to the lifecycle of another.
The deletion logic is then handled by Kubernetes itself.
This is backed by the concept of ownerReference.
The API makes it straightforward to implement:
client.pods()
.inNamespace("ns")
.createNew()
.withNewMetadata()
.addNewOwnerReference()
.withApiVersion("v1")
.withKind("Pod")
.withName(podName)
.withUid(pod.getMetadata().getUid())
.endOwnerReference()
.endMetadata()
.done();Always keep a sidecar
Adding a sidecar doesn’t mean it will stay forever this way. For example, a pod belonging to a deployment can be deleted. It’s the goal of the deployment to re-create a pod to reach the desired number of replicas.
Likewise, if a sidecar is deleted while the main pod is kept, a new sidecar should be spawned with the correct own reference.
Conclusion
In this post, we described how one could implement a Kubernetes controller with the Java language on the JVM. With the help of Fabric8’s API, it has been quite straightforward. The main issues come from the edge cases in the scheduling/deleting logic. In the next and final post of this series, we will finally see how to deploy and run the code.