Agoda recently detailed how they integrated their macOS infrastructure with Kubernetes. This was accomplished by developing the macOS Virtualization Kubelet (macOS-vz-Kubelet), a tool designed to manage macOS workloads within a Kubernetes environment.
Vitalii Horbachov, Staff Software Engineer (Cloud Platform) at Agoda elaborated the journey through a blog post. Agoda was heavily reliant on its Apple infrastructure for development, previously managing 200 Mac Minis for continuous integration (CI) processes. However, the traditional methods faced scalability and performance issues, especially after the arrival of Apple Silicon. Existing virtualization frameworks like QEMU/KVM also posed certain limitations. The previous Kubernetes solutions were inadequate as they only supported a portion of the infrastructure, primarily focusing on virtual device farms without addressing the build machines used for CI/CD.
To overcome these challenges, Agoda developed macOS-vz-Kubelet, which allows Kubernetes to manage native macOS workloads directly. This project is built using Go and leverages Apple’s Virtualization Framework to run macOS virtual machines (VMs) that inherit the full performance of Apple Silicon hardware. This solution not only replaces older methods but also enables seamless deployment of Pods with macOS workloads.
Source: How We Integrated Native macOS Workloads with Kubernetes
One of the features is the ability to create hybrid runtime Pods where the first container is a macOS VM, and additional containers can be managed by Docker. This setup allows for efficient CI workflows where a macOS VM serves as the build environment while Docker containers handle auxiliary tasks like logging and artifact uploads.
The project introduces a custom Open Container Initiative (OCI)-compliant format specifically designed for macOS VM images. This format allows for efficient image creation, packaging, and management using tools compliant with Apple’s Virtualization framework – e.g. macosvm.
Agoda was recently in the news as they adopted an unconventional approach to transition from a monolithic GraphQL API to a microservices architecture. Numan Hanif, Associate Development Manager at Agoda, told InfoQ, that this approach has minimized disruptions, empowered their teams with greater control over the full stack, and better aligned the architecture with agile and modern development principles.
Horbachov further elaborated networking challenges inherent in Apple’s Virtualization framework. By default, VMs use NAT networking; however, Agoda implemented bridged networking with DHCP to allow direct access to VMs via remote IPs. This was achieved by assigning custom MAC addresses to VM interfaces and monitoring network traffic to retrieve IP addresses.
When it comes to managing the underlying macOS hosts, Agoda employed Mobile Device Management (MDM) solutions like JAMF for initial configuration and setup of Mac Minis in their data center. This included user account setup, OS management and updates.
Horbachov concluded by mentioning that the introduction of macOS-vz-Kubelet represents an important step in managing macOS workloads at scale within Agoda’s infrastructure.
Interested readers can head over to the project repository, which includes comprehensive documentation on features, usage guides, and examples demonstrating real-world applications of macOS-vz-Kubelet. This journey is also discussed in Agoda’s Plugged In Tech Podcast.
