The fundamental principle of DevOps within an organization

Very often, organizations experience local conflicts between developers and infrastructure teams that need to be resolved afterwards. Since I am often brought in to resolve such conflicts, I would like to share my view on why these conflicts arise and what needs to be done, in my opinion, to significantly reduce their number.

How did DevOps come about?

We all know why such a marketing name as DevOps was invented. In case someone doesn't know...

(I can guess how annoying introductory words about the same thing in texts like this are. For example, let's draw a ship with containers or a set of micro-poops in an article about microservices for the hundredth time, but here the story is needed for further narration, not to introduce something everyone already knows) So, in case someone doesn't know, in distant 2009, the DevOps movement was aimed at stopping the throwing of feces between admins and development teams in the process of delivering the product to the user. The problem arose from the application deployment process at the time. Often, the development team provided an artifact as a result of their activities (for example, a .jar file, .dll, .exe, in some cases an archive with source code). Then this artifact went to the admins, and they had to run it on some environment, be it production for users or a test environment. And here the most interesting began, admins can't run it - they return the artifact to the developers, developers don't know what the admins did - they return it to the admins with the words "you do it yourselves, have you tried this?". And so on in a circle, until the artifact starts or someone runs out of patience. Of course, situations that turned into throwing did not end with just deployment, then came load, errors, logs and other joys of operation, in which some teams threw "information" to other teams, and they in turn responded with other "information" in response.

And then someone said "enough!" and came up with a marketing name for the process that would put an end to the championship for the most accurate throwing of responsibility to another team. They called it DevOps, which briefly means the following: “development teams, let's delve into how your application runs in real environments and monitor its stability and needs, and admin teams, on your side, let's learn to understand the needs of development teams and learn to automate with regard to demand within the context of the current organization".

How should it work in practice?

Through our beloved conversations with each other at the right time. To simplify, instead of the team coming and saying:

"I have made it, take my creation and deliver it to the users and remember that my creation requires 17 cores, a proprietary Taiwanese operating system, 400 environment variables, 2 XML configs, one Prolog config, 2 TB of local disk space, and 4 databases. And yes, the product must be in production in 2 days, and the documentation is available to me through a neural interface."

We just need to change a little:

"I am planning to create it. My creation will require approximately 17 cores, a proprietary Taiwanese operating system, several hundred environment variables, configs in XML and Prolog, a couple of TBs of local disk space, and 4 different databases. And yes, the product must be in production by the dd.mm.yyyy date. We have not planned to write documentation yet."

The situation described above is hyperbolized, but it is an attempt to show the difference visually - instead of putting something in operation and passing it on as a fact, we start the conversation before doing anything.

What is the purpose of this conversation?

The main goal is for the admins to understand the needs of the development team and for the development team to understand the infrastructure capabilities. To simplify, put yourself in the other person's shoes.

That's it. This goal is the basis of the DevOps process itself. In fact, DevOps is just a term that was invented to easily fit into a conference description group further actions, in which processes and tools will be born to solve existing problems at that time.

But if you remove the term from here, the above-described goal is the beginning of healthy and productive team development. And also the reason why increasing the number of people does not lead to proportional growth in speed, but that's another story.

Both admins and developers have those sacred knowledge that is not available to the other side by default. Understanding each other's goals and capabilities, teams can come to a compromise when one doesn't have to sacrifice business requirements, and the other doesn't bleed from their eyes trying to understand what's going on and meet deadlines. The main thing is to take a deep breath before starting...

Where are the tools?

The tools we use today, and which are already included in the term DevOps, were born precisely during those same conversations. That is, deploying Kubernetes and stuffing everything into containers is not the ultimate goal of the process. The goal is to understand that teams create a system in this way because they have reasons for it, and jointly agree on how the demand for infrastructure for this very system can be covered in the development process. As a result of these conversations, both sides should understand what to do next and the reasons behind these actions. Kubernetes themselves were born this way. New developers arrived, the business wanted more, and increasing the number of admins and approaching the configuration of each server with a touch of romance when servers for the same thing were configured slightly differently was not a radical response to such growth. And then they immediately learned how to use Puppet and Chef to distribute identical infrastructure, then added more dynamics and stuffed all applications under orchestrators, in which a whole world for applications is built-in out of the box. But again, all this was a response to specific problems, often in the context of specific organizations. Of course, some problems were more common, so some tools are now included by default in many projects.

Why are Kubernetes and Terraform still not a solution to the original problem?

Despite the fact that from a bird's eye view, the problems of different projects are similar, every day we work at a lower level. We are not talking about continuous delivery as something abstract. We are talking about the fact that we need to deliver applications that we make in the context of the current project, organization, product, etc. In one case, a product may have hundreds of requests per second, and every movement in production needs to be done carefully and distributed step by step, spreading the load increase for the new version over a day or two. In another case, we need the traffic to go to the new version as quickly as possible. And in such cases, we return to the same interaction that was present in 2009, only the subject of the conversation has slightly changed. Communication is not in the direction of "give us different servers, and as many as possible" or "give us the ability to solve task X because it's necessary for our context". Despite the fact that the widespread use of public clouds greatly softens the situation, such conversations still do not disappear anywhere. The reasons may be different: legacy in systems, falling under government regulations, support for various standards, etc.

What does it all mean?

The point is, until we learn to have those very conversations at the right time and come to a decision that satisfies all parties involved, there is no DevOps in the processes. There is just a set of tools that out of the box solve some industry-wide problems.

But if everyone learns to communicate with each other, then the number of effective managers, architects, DevOps teams will drastically decrease, and real productivity and results will begin.

If it's too confusing, here's a simple rule to start with:

As a result of all communication between multiple interested parties, all parties should understand the next steps.

Directions like "please do not touch me..." should not be considered as the next steps.

What does this mean in real-life examples?

Example. Allow this server to access these subnets

Negative scenario:

Development team: We need to call API X from server Y, but they don't have network connectivity. Please add network connectivity between them.

Infrastructure team: No, these networks should not be connected.

Curtain... The development team can't achieve the business goal, everyone is unhappy. The infrastructure team thinks they saved the company from heresy again, and they treat developers like little kids. And the developers think that the admins are completely disconnected from what's going on and it's not clear what they are doing except for prohibiting things. As a result, escalation occurs, and someone starts delving into the problems of both sides and finding a solution that will satisfy both.

Positive scenario:

Development team: We have a task that requires us to call API X from server Y. But we found that there is no network connectivity between them. Can we organize this connectivity?

Infrastructure team: No, we already have segmented trusted networks. There should be no trust between these components, according to the plan. Do you have the ability to complete the task without communicating between these 2 servers?

Then there is a brainstorming session on alternatives, and they realize that the API doesn't necessarily have to be called on the server, and it can be called, for example, from a browser, and for this, the API needs to be exposed through a gateway, and they part ways, continuing to work on their tasks.

In conclusion

The words and examples described here may seem naive and obvious. But I have repeatedly observed and been in similar situations where 2-3 parties get stuck on the first branch of communication, and there needs to be that intermediate link that tries to find a compromise between everyone. If we change slightly how we communicate between infrastructure and development, then this can lead to excellent solutions and, sometimes, even world-class tools.