There should be no mystery around DevOps (don’t ignore this link, just because it is from Wikipedia. It is actually a very good overview of the practice and culture of DevOps). Formal implementations of DevOps have been around for at least five years – and before – when the practices around using agile methodology to integrate development and IT operations silos into a collaborative organization with the process, tools and coordination began to be recognized. So, before we begin looking at best practices for implementing DevOps – let’s dispel a few common myths:
- The best information about DevOps was written yesterday… Nope. DevOps was first described in 2009 as a formal subject at a conference in Belgium. Over the past six years, there has been a lot of good information from practitioners in the field. There is good information being produced now too – but this is not (primarily) a technical subject. Freshly-minted articles on DevOps, information from the latest conference, and new tools may be great, but if they don’t benefit from a wide range of implementations and field experience, you might want to wait until some of the fog is cleared by early adopters…
- DevOps is a skill. Not really. Implementing DevOps is not your neighbor’s, but both can be very successful in their own environment. You can’t hire your way into a great DevOps team simply by searching resumes. DevOps teams are developed over time, culturally, in much the same way agile practices are integrated into development teams. Great DevOps teams are built out of a common desire to improve and integrate development and operations practices, processes and systems. It doesn’t happen overnight. One person can help lead an effort from past experience, but without commitment and active participation of all team members – the results will be eyewash at best.
- DevOps is about automation. Every tool vendor has to say this, but… There are many good tools for automating development and operational tasks as a part of DevOps, but no branded tool is essential to every implementation. In fact, many of the best-known and widely-used tools are open source and have been around for many years. Automation is part of DevOps – but not the point of the practice.
- If you are going to implement DevOps, you need to throw out everything you know and do – and start over. Actually, just the opposite. The most successful implementations start from practices you already know and use, scaled to the organizational level and consistently improved with best practices. Big bangs – where everything is restarted from scratch – are bound to cause disruption and distraction. Start small, build on what works and continue to examine and improve. Don’t kill the messenger or refuse new ideas outright. Welcome new ideas, but don’t let anyone tell you everything you are doing is wrong and can’t be built upon.
- If you are going to implement DevOps, you need to be ready to virtualize everything. A virtual non-starter for planning. DevOps implementations often start from a need to better manage or take advantage of the tools that are available for virtualized infrastructure but that shouldn’t stop you from implementing DevOps if you are not using virtualization and have found no need to head in that direction. Virtualization, for the sake of using the technology and nothing else, creates confusion and another layer of practices to work through.
- We don’t need DevOps – we’re not in the technology business. Maybe you don’t know what drives your business… Almost all businesses today use software to run their key business processes. In today’s business, nothing stands still, change is the only constant. Organizations that cannot evolve, that are based on systems that are stuck on “version one,” put their future at risk. DevOps, like agile, is a methodology based on helping teams work together towards common goals through iterative operations. Collaboration, communication, and repetitive operational improvement are requirements that are necessary to all organizations leveraging lean, agile and DevOps methodologies.
There are more, but let’s leave the myths to move on to some best practices that are field proven:
- All hands on deck! Just as agile emphasizes direct communication and collaboration between developers and product owners, DevOps emphasizes the same between all stakeholders involved in software development, deployment and operations. Implementations are not built among three key engineers sitting in a room lighted by the glow of monitors with energy drinks in hand. They are team efforts that succeed or fail based on the participation of everyone and the articulation of their own, initially self-centered, needs.
- Leave ad-hoc, one-off processes and systems for repeatable, manageable practices. You’re not going to improve a process or system you’re not going to reuse or repeat. There are good reasons that many common practices have become industry standards. Integrating them into a process flow that crosses silos from development, through deployment, to operations is just the next step. Formalizing them provides an understandable, repeatable process that can be managed across many levels of operations and provides a basis for an examination that leads to incremental improvement. What are the practices to work on you ask?
- Test Automation – To be fair, test-driven development (TDD) at the developer-level, with automated tests run by the developer multiple times a day, is a common practice in agile teams. But from a DevOps point of view, if it promotes early testing and gives developers an opportunity to correct problems during development (rather than later in the process), it lowers the risks of the release process and the issues caused when an app going through release hits a show-stopping fault.
- Continuous Integration – Another result of the constant improvement of agile practices, continuous integration (CI) builds on test automation by leveraging a version control system and automated regression testing to assure as new code is added to an application, tests on existing functionality continue to respond as expected and a method of “roll-back” can be used to revert to an earlier version of working, tested code when necessary. Where developer-level automated testing focuses on newly developed code, CI focuses on quality at the application-level as the new or modified code is integrated. At the DevOps level, this practice again assures the quality of the working code and sets the stage for continuous deployment (which we will discuss shortly).
- Integrated Configuration & Change Management – Some lists of DevOps practices separate these two issues – but in truth, they are joined at the hip in operational practice. Integrated configuration management brings the development team into a wider sphere of operations than the current application they are working on. Instead of building variations of services that already exist (as an example) – integrated configuration management makes them ask – what do we need to do to use an existing service instead of re-inventing wheels. And if a change is needed, change management brings operations into the picture so they can provide input on what other systems could be impacted and what consequences or opportunities a change might expose at a broader level. This level of thinking is often avoided in the enterprise because it can open a “can of worms” as it exposes dependencies and brings stakeholders out of the shadows – but with DevOps implementations – it is exactly what you want to happen.
- Integrated Deployment Planning – In an agile world, incremental deployment of applications becomes a natural outgrowth of automated testing and continuous integration – but operational constraints often limit or stop the capability in its tracks. Leveraging the inclusion of all stakeholders in application development, operations, and support in the deployment process (mentioned in our first point) opens up the opportunity for planning to start at the beginning of development rather than the end of the process and to leverage incremental development as a way to get end-user involvement in application development much earlier in the process. This greatly lowers the risk that a bug-free application will be developed that fails to meet user needs and expectations and assures that organization level integration problems will be detected before the application is rolled into daily operations.
- Continuous Deployment – With all of the above points in place, DevOps sets the stage for one of the most important aspects of implementations, the capability to release new features to existing applications as they are developed, instead of a predetermined “release dates” as is common in enterprise operations. This opens the organization to a virtuous cycle of user-driven improvement and a true “agile enterprise” approach. There are a lot of buzz words in that thought, but in practice, it is the expression of everything the agile and lean movements have come together to provide. But, as you can see – continuous deployment cannot happen without many other steps in place.
- Application Monitoring and Automated Dashboards – Strictly speaking, you might look at these two areas as the icing on the cake. Application monitoring requires that developers build in or provide tools to manage application health throughout the lifecycle – assuring that it scales properly, utilizes resources as expected, and doesn’t become “jammed” when associated systems don’t respond as expected. Automated dashboards are an enterprise level tool to assure that systems like version control and automated builds run properly across many iterations on many projects and provide feedback on how DevOps system operates at the macro-level providing opportunities for general, rather than spots, improvement. Not all DevOps implementations will include these aspects, especially in smaller or single-product focused situations, but they are seen as critical in an increasing number of implementations.
- Automate building environments – When internally proven and successful practices are in place, leaving the building of environments for new projects to manual operations run by a person in the IT shop or a spare developer raises a risk that necessary elements or requirements will be left out or gradually reversed to legacy implementations. In situations where virtualized infrastructure is used, with the many options that come with virtualization, this becomes a critical step in DevOps implementations. Halting development to take care of what should have been standard configurations as a project is ramping up can slow down the process of reaching full production to a large degree.
- Approach DevOps as a cultural change – Successful DevOps implementations requires teams to (want to) leave their silos and work as one collaborative unit to bring capabilities to the point where a virtuous, iterative cycle of improvement can become routine rather than a long, difficult path to one big all-encompassing change. The transition can be threatening to some individuals and groups. The temptation to succumb to a “let’s just adopt these tools and be done with it” is great during the process. But like any cultural change, if DevOps is gradually implemented with input from all stakeholders, the chances for success are greatly improved. It isn’t just the tools. It isn’t just the people and their roles. It is a real change in the way we think about how things are done and the outcomes we need to do things better.
- Don’t succumb to the opportunity to set up new DevOps titles and roles that simply build new silos and fiefdoms – Ok, so you have a working DevOps implementation. Everyone is playing a part in your new way of operating and success is beginning to show. With that success comes a natural tendency to try to protect your advances with titles like, “Automation Architect” or “DevOps Security Engineer.” People who have spent the time to think deeply about an area like automation become natural centers for practices that come out of their work. Of course, why not? The simple reason is that if automation becomes the domain of one engineer or group – it is no longer everyone’s concern. Instead of everyone being involved, everyone looks to the Automation Architect for answers and if they don’t have one – it is assumed there isn’t any. Or – if they have an opinion, they are put in the position of making that opinion reality regardless of other ideas that might come from other members of the team. Developing new “DevOps” positions can easily reverse the cultural change that is needed for success with DevOps. It is a conundrum, but it the same as agile at this point, more positions and titles just assure that individuals don’t have to take as much responsibility and participate in solutions fully. It is a slippery slope.
Are there more best practices in DevOps? Of course. But the chances are that you will find them as they apply to your organization and begin the process of incremental implementation. That is a real secret. Don’t go for everything from the beginning. Work on each area, gain success and acceptance and build on your success as you move forward. Throw the grand DevOps implementation project plan in the trash, or if you are cornered into a plan, keep it high level, not time-dependent and let your team find and fill in the blanks.
While we’re at it – let’s say again you can bring outsourced teams into the mix. We’ve said it before, but if you are considering a DevOps implementation, you can find outsourcing vendors that are experienced and can play an active, partner-level role. If you’re one of the organizations that saw some of yourself in the “we’re not a technology company” statement – an outsourced partner can be a real asset because they can bring in experience and possibilities that can expose options and speed up adoption.
Scio is a provider of nearshore, outsourced software development and DevOps solutions for our clients in North America. We have experience with dedicated teams in DevOps situations and developing DevOps-based solutions for our customers that need to approach continuous release deployment with agile teams. We can work at a partner-level for a long term, sustained improvement – the point where DevOps makes real sense. If you would like to know more about how we could help your project – contact us.