Migrating Ruby on Rails to Kubernetes: Our Experience and Lessons Learned

In this blog post, we want to share our journey of migrating our Ruby on Rails application to run in containers and on top of Kubernetes. As a company heavily relying on Rails and serving numerous customers, we saw this migration as an opportunity to improve scalability, flexibility, and efficiency within our infrastructure. We'll discuss the challenges we faced, the decisions we made, and the benefits we gained from this move.

Our Original Infrastructure:

Before migrating to Kubernetes, our application stack consisted of various components such as Rails, MySQL, Redis, Memcached, Faye (real-time push server), and multiple Go services. We used AWS EC2, ELB load balancers, and AWS MySQL RDS to run our infrastructure.

Motivation for the Move:

While our previous infrastructure worked well for several years, adding new services became a bottleneck. For example, setting up Sphinx for search required additional servers, load balancing, backups, and firewall configurations. We wanted to streamline the process of adding new components to our infrastructure and reduce the complexity of managing different server types.

Preparing for the Migration:

As a team experienced with containers and Kubernetes, we decided to keep our monolithic Rails application intact, considering its strong conventions and efficiency. We began by creating a base Kubernetes configuration file, defining all services as Kubernetes services to ensure compatibility.

Building the Configuration:

We developed 11 configuration files for our project, addressing different aspects such as setup, security, configuration values, secrets, certificates, and various components' deployment. These files were stored in our git repository, with encrypted secrets and security files for enhanced protection.

Addressing Rails-specific Issues:

Throughout the migration, we encountered specific challenges related to Rails:

1. Database Migration: We leveraged Kubernetes jobs to run the Rails database migrations before each deployment, as we didn't have any backward-incompatible migrations.

2. Asset Pipeline Compilation: To handle asset compilation, we moved the process to our image building flow, ensuring that assets were compiled into the container images.

3. Sidekiq Restarts: Kubernetes posed challenges in managing Sidekiq deployments due to its kill signals. We had to configure Kubernetes to delay killing Sidekiq processes until they finished their current jobs.

4. Unicorn Memory and Thread Model: We transitioned from Unicorn to Puma to better utilize Kubernetes' scalability capabilities. Puma's thread-based model allowed us to run multiple instances efficiently.

Addressing Non-Rails Issues:

One of the significant challenges was managing dynamic shared storage. We decided to refactor our infrastructure to eliminate the need for shared storage folders, as running NFS NAS reliably within containers proved difficult.

Deployment:

Moving to Kubernetes brought new opportunities for rapid deployments. We introduced the concept of Formations, which represented a combination of environment and deployment destination. This allowed for more fine-grained deployment configurations, feature-specific deployments, and separate stacks for each branch and PR.

Build Chain:

We developed our own BuildGrid to build container images triggered by each git commit. These images were then pulled, configurations were generated, secrets were injected using Gifnoc, and finally, the configurations were applied to our Kubernetes cluster.

Migration and Benefits:

Migrating to Kubernetes required careful planning and reduced downtime. Despite some challenges, such as intermittent issues with Redis, we successfully made the transition. In the end, we achieved significant benefits, including reduced infrastructure costs, a more flexible and uniform infrastructure setup, and improved scalability and efficiency.

Migrating our Ruby on Rails application to Kubernetes brought numerous advantages, allowing us to streamline our infrastructure, reduce costs, and improve scalability. While maintaining our monolithic Rails application, we successfully leveraged Kubernetes' capabilities and optimized our deployment processes. With a more flexible and efficient infrastructure in place, we can focus on