We often spend a lot of time planning for the perfect deployment process and rolling back strategies, but sometimes we forget that despite the best plans that we have, we still need to address urgent post deployment fixes when rolling back is not an option. Hotfixes may not always be due to critical bug fixes, they can also arise from last minute client commitments or regulatory requirements. I have observed several instances where we had to push back on these changes to the next release cycle resulting in unhappy clients simply because we do not have a process in place to handle hotfixes. You may ask then, why don’t we keep a Release Branch or Hotfix Branch in our development practice? The answer is not always that simple and obvious depending on your Agile practices and build process.
In my environment, we maintain a single code base that is deployed across multiple regions with engineers spread across different countries committing code and deploying to their market as their product owner sees fit. Our build pipeline maintains these different markets and builds our master branch to be deployed to a staging environment before being deployed to production. Keeping these branches synchronised for each region proved to be a challenge and more likely to result in additional overhead for the engineering team.
In the end we simply decided that we just need to make a simple change in our deployment pipeline to allow a specific branch to be built and deployed. To deploy a hotfix solution now simply meant branching off the last release version for our region and applying the fix for our build pipeline to pick it up for deployment. Aside from being able to deploy hotfixes now, this small change in our process actually resulted in another unintentional advantage for us. Having the ability to deploy any branch to our staging environment allowed us to build and preview certain features for the product owners to review and test before they are released. These can be proof of concept designs and may even be full blown MVPs (Minimum Viable Product).
So back to our question, are hotfixes really the way to go ? It depends. The ability to deploy hotfixes is certainly great to have as I have explained earlier, but it should really be used sparingly. It is ultimately up to the team to decide if these deployments have minimal risk but have great impact to the business and should not be a used as a shortcut to speed up your feature releases. Perhaps with our hotfix practice in place, it will allow us to slowly transition into more frequent releases and achieve a state of Continuous Deployment and eliminate hotfixes altogether.