Developing In Real Time

One Developer’s Journey Into the World of Continuous Delivery

How I became an evangelist of continuous deployment: a method that speeds up development and makes feature releases less susceptible to bugs.

Throughout my career as a software developer, when I’ve begun a new job, I’ve found that each team has a unique “way of doing things.” I may know the languages in which the organization’s engineering team codes, but I may be less familiar with the frameworks the team uses, or the way the team sets up their various services, or the hierarchical structure of the files in the code base. 

One thing, however, that has remained relatively consistent from place to place has been the general process of releasing code. Yes, one company might use Jenkins for deployment while another uses CircleCI, or one might follow a sprint schedule while another uses more of a Kanban approach, or one might have a stricter code reviewing process than another. But — in my personal experience — the overarching process has been the same:

  1. Code for a feature is written in a feature branch
  2. The resulting behavior is QAed in a staging environment
  3. When (and only when) the feature has been completed and battle-tested, the feature is merged and released into the wilds of production for real, live users to interact with.

Imagine my surprise when I started onboarding at Abacus and quickly discovered that its developers were committing unfinished, unreviewed code to the master branch and deploying it to production… and that they were each doing this several times a day.

CD and Feature Flagging at Abacus

Abacus uses a technique called feature flagging (or feature toggling). The basic idea is that unfinished or untested features can be safely released into production if no one sees them. To achieve this, one uses flags, digital switches which can be flipped on or off depending on the environment in which the code is running. Part of the greater process of continuous delivery, the goal is to get work into production safely, quickly, and in a sustainable way.

Rather than working on a complex feature in the bubble of a feature branch, and merging it in all at once when it’s complete, a developer merges chunks of the work piecemeal or (gasp) commits directly to master. As long as the code is wrapped in conditionals to check that the relevant feature flag is off, the code will not run, and the application will function as if that code isn’t there.

Feature flagging is by no means a new concept and certainly not unique to Abacus’s “way of doing things.” It’s been used by large development teams like those at Flickr, Etsy, Gmail, and Netflix, and has been written about before. But the process was new to me, and frankly seemed quite contrary to the philosophies I had built around safety in the product development lifecycle. 

Is This Safe?

My education about releasing code orthodoxically preached not putting code into production that hadn’t first gone through “the process”: merging, testing, and, more pointedly, being finished. As such, I had one overarching question when exposed to the process of committing unfinished code behind feature flags:

Why on earth would anyone do this?!

Such a technique seemed inside-out to me. Having unfinished code in a feature branch until complete seemed as fundamental to me as coding in a local environment rather than on the production server. In my mind, you were supposed to finish your work in a safe space devoid of real users. You paint your masterpiece in the studio; not inside the gallery at 2 ‘o clock on a busy Saturday afternoon. 

Even behind a flag, it still seemed to me that the code would grow messier, or be more susceptible to issues that would have been caught in the long, smooth, bureaucratic process of branching, code reviewing, and QAing prior to release. It just seemed more “correct” to develop a feature in a safe, encapsulated environment, and to subsequently have the code looked at via a pull request. Circumventing that process by introducing unfinished or non-battle-tested code into an application that is live and being used by customers seemed risky at best and insane at worst.

I quickly learned that feature flagging is not only a faster way to develop, it is also far safer than developing full features in branches. The goals of continuous delivery are to keep the code in a deployable state at all times, and to keep all developers working on the latest possible version of that code. As opposed to a system of periodic releases (where all the contributions in a given span of time are packaged and deployed to production in one fell swoop) continuous delivery is accomplished through all contributors making frequent, small commits to production throughout each day. This enables teams to de-couple deployment from a release. It keeps developers more in sync, and allows for easier rollback of buggy code.

Speed and Safety

Speed is the most obvious benefit to a continuous delivery system. If you are able to frequently deploy to production through continuous delivery, you are not only able to skip the bottleneck of preemptive, routine review steps, but you are able to get immediate feedback on the code you wrote. I think that many folks — myself included, prior to joining Abacus — have an assumption that by following these routine steps and working in a vacuum, they can potentially eliminate any possibility of bugs. 

But bugs are not eradicated through any process, whether feature flagging or feature branching. Bugs are a results of building features. While they should be avoided and promptly fixed, one should not expect to find a process that eliminates the possibility of them. They should instead hope to fix them as quickly as possible.

CI/CD’s philosophy on this issue is to optimize for “time to fix” a bug. The goal is to catch bugs and to roll back the changes that caused them with ease. Since code is deployed regularly and tested intrinsically through use, bugs can be fixed much faster. The overall impact of having a bug is reduced when you don’t need to wait two weeks to release the fix. 

CI/CD doesn’t rule out code reviews. We do still employ them in cases where a developer wants another set of eyes or has a specific question regarding the code they are releasing. We use pull requests to ensure code is clean as well. However, we are by no means bound by them. It’s also much more effective to test for functionality by actually using the feature, than just looking at the code behind it out of context.

Abacus’s feature flagging system is set up in such a way that we can more effectively test out a feature before exposing it to all of our users. This ultimately makes the feature release extremely safe. We are able to toggle a feature in a given environment, but we are also able to whitelist it for a subset of clients or customers, or our internal test accounts in production. When a feature is finally finished, we can turn it on for a subset/percentage of our users, or for a specific client or clients. Once the feature is live, we can use flags to create feature tiers depending on the particular plan a client is on. For example, if we want a feature to only be available for clients using the “premium” product, we can use a flag to expose it only to those particular clients.

Since Abacus uses our own product, we can even just turn the feature on for ourselves. Our co-workers can then report any issues or bugs they find by using it in real life, over a predefined period of time. By bringing a feature for a test-run internally, we are more likely to identify edge cases and pitfalls that we might not have thought to test through a traditional manual QA system.

Avoiding Merge Hell

Atomic commits make rollback of problematic code easier, and they also make the initial merge of code smoother. If you build a big feature in a branch and wait until it’s (hypothetically) complete and bug-free, you are inevitably at the mercy of a monster merge. If your feature ends up breaking something, you will have little choice but to either revert the whole merge, or to look through a massive amount of code to find the source of the issue before more users complain. We do use a staging environment for checking our work, and it is expected that developers run tests and check their work locally before pushing. However, by making smaller commits, any problematic issues are easier to identify and revert.

Furthermore, by committing frequently and atomically behind a flag, we are each able to work on the most recent version of master. Working in a longstanding feature branch may feel safe, but by doing so you are also exposing yourself to problems when it comes time to merge. The longer your code sits safely in a feature branch, the more stale/out of sync your version of the code base will get. As other developers contribute to master without you, your changes will become more susceptible to conflicts when you attempt to merge your branch in. Merge conflicts contribute to the complexity of feature rollouts, and such complexity increases the likelihood that new bugs will appear when the conflicts are resolved incorrectly.

They also contribute to added time of a feature rollout, The world doesn’t stop turning even while you’re resolving the conflicts, so it’s entirely possible that the code base will change further while you’re resolving the conflicts… which could result in more conflicts. In the words of the Swedish software developer and Youtuber Mattias Petter Johansson, “Every hour that we spend in our code cave, the probability of code changing under our feet goes up. And we spent a lot of hours in that cave. So we have to go back into our code cave and restructure our monster [merge] and we pray that the code doesn’t change under our feet when we’re away this time.”

So, What’s the Bad News?

There are certainly downsides to our approach. The biggest one for me has been a lack of transparency to developers not involved in a project. When I complete a feature, it may function per the spec, but the underlying code won’t be clearly distinguishable to my coworkers. Without a branch, the set of changes I introduce for a new feature is hidden in the flurry of unrelated commits to the master branch. The longer it takes to build a feature, the greater the flurry. There are multiple ways to build a feature, and coworker feedback is important to the health of the code base, especially as the team strives to follow styles and conventions. We have some ideas for improving this process by allowing for code reviews after deployment, but that is a topic for a future post.

No workflow is without flaws, and every process, like the code it produces, should be open to fine-tuning. Continuous delivery, and more specifically feature flagging, has allowed the Abacus team to avoid some of the pitfalls and bottlenecks caused by a release-based, branch-only flow. It has also allowed us to fine-tune the code we write with greater speed and ease, resulted in safer deploys, and has enabled us to test out features by putting them where they will ultimately reside: in front of users. After living with this system, I’m a convert — and you should be too.

Related Posts