DevOps Modernization Project Wrap Up

by Shayne Babczak on June 23, 2020

Polaris Solutions recently finished a project to modernize an enterprise build and deployment strategy. The project landscape consisted of the following:

  • 30 web applications
  • 12 windows services
  • 12 CMS sites (Umbraco)

For years, a basic build and deployment process was used for the test environment followed by a manual deployment to the production environment. This strategy worked when the number of projects was small but became burdensome when the number of projects, and project dependencies, increased. This build and deployment approach became THE major pain point in their development process.

Their original setup consisted of the following:

  • 2 deployment environments (Test and Production)
    • The test environment consisted of a single server
    • The production environment consisted of 2 load-balanced servers
  • One main branch of source control for each project in TFVC
  • Team City was used for builds and to deploy to the test environment
  • Manual deployments went to the production environment

The above setup created several challenges that the client was unable to solve without a complete DevOps overhaul. The biggest challenges included:

  • The manual deployment process to production was very error prone. The code that was being manually deployed to production was not guaranteed to be the same as the code that was deployed to the test environment. 
  • If a new code check-in occurred between deploying to the test environment and deploying to the production environment, the new code had the potential to get deployed to production completely untested
  • Production hotfix support was non-existent. If a production hotfix was needed while a new feature was being developed, there was no easy way to manage the changes. The new feature code would be rolled back, the hotfix would be developed, tested, and deployed, and then the rolled back new feature code would be reapplied. 

The manual production deployments were both labor-intensive and time-consuming.

  • Following audited best practices, only certain individuals had access to deploy to production. This made it a team effort to get code to Prod. Developers would watch over the shoulder of the deployer to give step-by-step instructions on what and how to deploy.   

As often happens, the existing development teams did not have the capacity or skill set to solve these issues with a modernized DevOps process. Polaris provided the expertise and ability to design and roll out a new build and release pipeline to solve these major pain points:

  • Migrated from TFVC to Git for source control. Since branches are first-class citizens in Git, it is easier to develop features and hotfixes simultaneously/independently and to then merge the changes seamlessly
  • Transitioned project builds from Team City to Azure DevOps. Azure DevOps builds work much better with the various branching provided by Git, which provided a more robust continuous integration solution
  • Moved release logic to Octopus Deploy (their selected vendor). This allowed the client to have confidence the code they were deploying was consistent across all environments. Deployment durations were decreased from an hour to minutes.
  • Doubled the number of environments, adding development and stage to their existing test and production environments. These additional environments provide more This again provided more chances to build confidence that the deployment would be successful

These changes greatly increased the client’s development throughput by letting their developers focus more on crafting code than worrying about deployment considerations. Other benefits included:

  • Significantly decreased release times, which reduced application down time
  • Eliminated user prone errors, which improved the quality of releases
  • Increased user satisfaction and confidence in the IT department
  • Increased developer productivity. Time that was being spent preparing applications for release could now be spent on improving code quality/developing new features


image