Werner

Challenge: The client created separate 'customer' environments, but due to the process, it took more than a week to create one. There was a documentation page that listed which steps (ansible playbooks) all had to be executed.

Solution: Processes automated with Terraform for IaC and Ansible for configuration, integrated into a Jenkins pipeline. Via a Jenkins job, the client could generate a basic configuration and create an environment based on this just generated data. Via Terraform, the infrastructure was created on AWS (or Azure), and with Ansible, all applications were implemented with a basic setup. After deployment, this environment was ready to be further fine-tuned for the client's wishes.

Results: Deployment time reduced from more than a week to hours, improved reliability and costs significantly lowered. The engineer could run the job in the background and focus on other tasks.

Lessons Learned: Integrating IaC tools such as Terraform requires initial investment in training, but pays off in reproducible environments. In addition, this also helped in the business case to quickly set up an environment from a disaster recovery and to be able to restore backups.

Technologies: Terraform, Ansible, AWS, Azure, GIT, Jenkins.

Challenge: The client had multiple customer environments and wanted to make "things simpler".

Solution: Created a design and tested with a proof of concept. The goal was to provision an EKS service in 3 different AWS regions. In each EKS, Argo CD would be implemented. Argo CD would use an ApplicationSet that reads information from a GIT repository. This GIT repository had a directory structure with region/customer, and each directory contained a JSON file with the applications and their versions. When a new customer is added to this GIT repository, Argo CD would directly notice this and create a new vCluster. Argo CD would then implement the correct applications and their versions in this new vCluster, making the environment directly available for use.

Results: This would enable a customer to be onboarded and running within 10 minutes after the change was merged in GIT.

Technologies: Kubernetes, AWS, Argo CD, GIT.

Challenge: The client had a Kubernetes running in AWS, but it was not up-to-date. Kubernetes was several versions old, just like the components such as the Ingress controller.

Solution: Version by version, I updated the EKS service by first upgrading the development environment, followed by the production environment. In the meantime, I created an overview of all components, such as the Ingress Controller, Argo CD and various other operators, and made a plan to update each of the components. First to the latest version in the same major version that was already running, then apply the necessary changes as part of the upgrade, and then proceed to either the next major update or the latest available version. In addition, because Kubernetes was running on EKS, it was still using the Amazon Linux 2 nodes. I made a plan to switch to Amazon Linux 2023 and therefore also ensured that it worked successfully in the development environment before it went to production.

Results: Kubernetes and the associated tools were up-to-date. A very critical Nginx Ingress controller CVE issue was found and a patch was made available, fortunately a few weeks after I updated the Nginx Ingress Controller. Because I had ensured that Kubernetes was up-to-date again, we could apply the change within minutes.

Lessons Learned: Keeping environments up-to-date is crucial, especially when security incidents/patches are needed.

Technologies: Kubernetes, AWS (CloudFormation).

Challenge: There was a logging setup, where log events generated from the application sometimes didn't even arrive in the Elasticsearch cloud solution.

Solution: Filebeat was running as a DaemonSet in Kubernetes and sent everything to a single Logstash pod. From there, it went to Elasticsearch. Following a short analysis, I proposed using Fluent-bit and placing these events in Kafka. Subsequently, I deployed multiple Logstash instances that retrieved the data from Kafka and forwarded it to Elasticsearch. In addition, I applied various fine-tuning, so that – if Kafka or Elasticsearch itself was temporarily unavailable – no data was lost and all logging was quickly available in Elasticsearch again.

Results: Since the change has been activated, there has been no single observation that even one event had disappeared. However, it became clear that in the previous setup more logs did not arrive in Elasticsearch than was initially thought.

Lessons Learned: The integration of Kafka added buffering, which proved crucial for scalability, but required careful configuration to minimize both network latency and having sufficient storage.

Technologies: Kubernetes, Filebeat, Logstash, Fluent-bit, Kafka, Elasticsearch, AWS (CloudFormation).