So after my initial post, I decided to dig a bit deeper into the processes of DevOps, rather than go directly into the tools used (Docker, Kubernetes, Jira, etc.). I’m doing this mainly because the general view of many people in IT that DevOps is more about the tools used, rather than the actual processes that are needed to achieve success with DevOps.
So, per Gene Kim’s “The Phoenix Project”, and “The DevOps Handbook”, defines the following four types of work.
|1. Business Projects||Business Initiatives, most of development work|
|2. Internal IT Projects||Infrastructure and IT Operations|
|3. Updates and Changes||Generated by the first two types of work|
|4. Unplanned Work/Recovery Work||Incidents and problems generated by other work|
So, how does this concept assist in attaining DevOps objectives? Well, it’s important to know how much time you or your team is dedicating to work that advances the company business objectives. I personally work in the IT Security group at my employer, and DevSecOps can be considered even more obscure as to what it is.
Out of the four types of work, the one type that’s least wanted is the Fourth. Unplanned Work stops you from proceeding with your company’s own projects or your team’s projects. That’s because you have to stop everything to put out fires. While for Security Teams, working on incidents can be the core work you do, but ask your self this: Are Incident Tickets taking time away from improving the Security Stance of your employer?
Look at how many hours per day are you using on projects that you’re assigned at work, and how much time you’re doing incidents? If you’re using only 20% of your time on project work, and 80% of your time on incidents, then it’s unsustainable. You’re putting out fires all the time, without the opportunity to do much or any Root Cause Analysis in identifying what’s causing these incidents and remediation. In Security, if you’re in this situation, it might be beneficial to look at an MSSP to take over handling of some incidents in order to free up time to implement improvements. If you already have an MSSP handling basic SOC work, and you’re still flooded with tickets, then it might be worth a review of the contract with said MSSP, and either choosing a new one to handle SOC operations, or expand their handling of incidents. An additional benefit of using an MSSP, is that they have more experience in dealing with various architectures and platforms, and can help in implementing better alerts and automation.
The question you have to ask, is this. What is your team’s job description and objectives? Are you building a SOC, or are you running a Security Team that’s supposed to be improving the security stance of a company? While they’re not mutually exclusive, they have two different budgets. As with a SOC, you’re going to need more people to handle the unplanned work, while another team or group would have to handle improvements. Or if you’re working on improving a company’s security stance, you might not need to hire a large team, but you should consider the costs of contracting with an MSSP. In the end its a cost/benefit analysis between the two (having an internal SOC, or using an MSSP).
Security stance improvements are extremely important. If your company is working on a transition to DevOps, then it will lead to many more projects and hours that will be thrown at each team. Some of the changes will be difficult to implement due to company culture, others will just need a lot of time. But this leads to the issue of how much time you and your team dedicate to fire fighting incidents. 80% of your time on incidents, 20% on projects is unsustainable. And ideally, it should be quite the opposite. 20% of your time working on incidents, and 80% working on projects. This aspect shows more the maturity of a security program, rather than concentrating on making more and more tickets, to improving the MTTR (Mean Time to Resolution) metric. But not by closing tickets faster without investigation, but by having the team primarily investigate more difficult tickets that can lead to improvements. Easier and lower priority tickets can be used to get metrics based on quantities and impacted users/devices.
So, after this discussion of the four types of work in DevSecOps, or general Security Operations at a company, this kinda leads us to the actual implementation of DevOps principles in the organization, and the “Three Ways”, which will be covered in future posts.