Hot answers tagged

38

TL;DR: Chaos Monkey was developed in 2010 at Netflix and released into wild in 2012 is part of the Simian Army, wildly popular among devoted followers. Built on principles of chaos engineering, the army increases resiliency to failure by injecting constant failure to the system. Concept Chaos Monkey was developed specifically for AWS where it will randomly ...


18

What is the MMM First I want to explain the context for Brook's Law. What was the assumption that made him create it back in 1975? A Man-month is a hypothetical unit of work representing the work done by one person in one month; Brooks' law says that it is impossible to measure useful work in man-months. source: https://en.wikipedia.org/wiki/...


10

It's not a good idea. In my experience you'll gain the disadvantages of both while any projected advantages will somehow fail to materialize. Itemized: You will lose speed. IT will comply with their own standard. The new task (for them) will follow the same 'sluggish' template all their work now has. Be prepared they will find it challenging - so even ...


9

Some additions to your own answer to this question ... Additional monkeys The article about "How chaos boosts performance" describes a few more of these monkeys, i.e.: 10-18 Monkey: finds configuration and run time problems in instances that serve customers in multiple regions. Chaos Kong: simulates an outage of an Amazon region. Remark: The same ...


9

If the project is for a startup's minimum viable product or some other sort of sanity-check/proof of concept effort - hackathon before DevOps is often recommended - basically in the "fail-fast/fail cheap" lean startup strategy. Some might recommend hackathon before DevOps in other situations as well, for time-to-market reasons. But if DevOps doesn't ...


9

A value stream is a process that adds value to a collection of things that have less inherent value. The canonical example there is an assembly line. I have little interest in a disconnected collection of parts (steering wheel, shifter, gas pedal.) On the other hand, I'll pay good money for a car. Car manufacturers add value to the parts in the form of ...


8

The key to presenting solutions is always to identify problems first; if the team doesn't feel like you're solving any problems, then they're going to dismiss what you're suggesting. The specific problems your Ops team is facing we don't know, but there are a number of common ones: Getting non-actionable alerts Spending lots of time in toil Version ...


8

Establish a wiki (such as Confluence, DokuWiki, etc). Establish a chat/communications channel (such as through Slack or Mattermost). Share some code repositories (such as through GitLab, Bitbucket, GitHub). Share some binary repositories (such as through Artifactory, Nexus, Pulp). Occasionally send out a "newsletter" email that showcases what your ...


7

What is Value Stream Mapping A Value Stream Map is a representation of the flow of work, inventory and information from supplier to the customer through your organization. The VSM enables you to see at a glance where the delays are in your process, any constraints and excessive work or inventory. For IT organizations, one common notation is that inventory ...


6

It's a bit hacky, but you can always do something like: check_process: cmd.run: name: ps aux | grep '[f]oobar' The exit code will be non-0 and the state will fail if foobar doesn't exist.


6

There is no "Deploy-once-a-day" rule in devops philosophy. It's more of: Deploy as soon as possible and as often as possible. Also it calls for decoupling architecture so different parts of it may be released separately and also for decoupling deploys from release. The Deveops Handbook by Gene Kim et al.calls for modifying Definition of Done to include ...


6

We're using jobs in jenkins to trigger deployments. It ensures that no matter who does the deployment, the ansible command that is run will be the same. A nice bonus is the build logs record when deployments were triggered, who triggered them and what exactly happened during the deployment. It's certainly not foolproof, but it's been a nice improvement over ...


6

You can find many of the answers in the result of DevOps survey which you should ask the product owner to read. This is a document written specifically for business people with little technical knowledge talking in the terms he should understand. At average you will need 1 extra developer for every 4 people to keep same level of feature development (38% vs ...


6

We don't use Agile for the DevOps group ourselves, but we do integrate with the normal Scrum Teams. When something is needed by the team from DevOps, such as optimizing the build server, the related team puts a PBI in their backlog with a 'DevOps' label. Our lead has a custom dashboard in Jira with all issues labeled 'DevOps'. They work with the Scrum Master ...


5

DevOps is about the merging of information and processes between those who develop the software and those who deploy the software. If the goal of your hackathon is to produce code which can not, or should not be deployed anywhere, then sure it makes sense to get your proof of concept (PoC) coded before developing the deployment process. However, if you ...


5

What I have done (and this is only subjective) is as follows: When a manager thinking about a due date wishes to add people into my team to cut the time needed and seems under MMM, I first discuss with him or her on why this could be bad. My favorite analogy for this is to remind them that if one woman can have a baby in nine months, nine women won't have ...


4

What you will lose by fitting into the IT organization is the "Dev" part of your little DevOps team. When teams become segmented into artificial roles of NetOps, SysOps and Dev, you introduce the following problems: Un-needed red tape and isolation - To do anything the developers will have to submit a ticket to IT and wait for them to implement it. They ...


4

The guidance that we've always followed in my various scrum teams is developers should get what they've got into testing as soon as possible. It may mean a little rework later on down the line if there are issues found in code reviews or developers haven't completely finished coding, but the earlier developers get feedback on what's been done, the earlier ...


4

Categories of errors There are two ways of looking at human factors that lead to problems and accidents: You can see the human error as the cause of a mishap. In this case "human error," under whatever label—loss of situation awareness, procedural violation, regulatory shortfalls, managerial deficiencies is the conclusion of your investigation. You can see ...


4

Speaking exclusively from a CI perspective, increasing the number of developers working on a project typically translates into more people working in the same branch. Traditional CI systems have a scalability issue in this respect: the probability of breakages/regressions/blockages increases slowing down the integration speed and inviting smaller teams to ...


4

Hackathons are a great way to promote people to share knowledge, collaborate, and share risk and responsibility. The traditional (non-devops) organization is often power-centric and promotes people to avoid talking directly to each other. Allowing people to only communicate to each other via their managers. This type of culture in organizations is sub-...


3

I would suggest that you adopt some of the IT team, and give them thorough training in the new system. Once they understand the system fully then it makes sense to offload it to them. Otherwise, you will become a Support Center for IT - and spend a lot of time firefighting as they learn the intricacies of the new system.


3

One Server to rule them all, One Server to find them, One Server to bring them all and in the outage bind them You, Sauron, forged this One Server, in the Darkness of Mount Doom your Datacenter in a desire to rule All applications. Hopefully the Fellowship of Devops did unite to tell you: After a long fight, Frodo the Chaos Monkey has been able to melt ...


3

As Michael mentioned, offer a standard solution based on release versions/numbers, with a reasonably long lifespan for your industry (maybe interleaved with one or more shorter lifespan intermediate versions, if it makes sense for your typical customers). Give your customers the option to embark on this standard release track, maybe with a decent migration ...


3

Open the PR, even if it's not ready. Basically, allow the QA testing & code reviews to operate in parallel. Pros: code review and testing can operate in parallel. Cons: if issues are found in testing, the code needs to change, even though it has already been reviewed. When code review prompts changes, testing has to be redone. This encourages the ...


2

Some great answers here, but I thought I'd chime in with an example of how we deliver features. "shipping" to production is not the goal, the goal is to deliver value. Shipping code to production is a means to an ends and we usually deploy many times to enable a feature. Here's a contrived example using a 1 week sprint for succinctness. During planning on ...


2

The Agile software development method describes a set of values and principles for software development under which requirements and solutions evolve through the collaborative effort of self-organizing cross-functional teams. It advocates adaptive planning, evolutionary development, early delivery, and continuous improvement, and it encourages rapid and ...


2

In a sense your question exactly underlines the problem faced by teams trying to be agile but without having the benefits of a good DevOps culture in place: there is practically no guarantee that at the end of each sprint they'll be able to deliver to their customers. In some cases teams may end up using more relaxed definitions of done (like passed QA in ...


2

I agree that relying on "developer education" is quite futile, at least when it's the sole course of action. Combine this, however, with automation and you can achieve far greater impact. Both sides, DAST and SAST can be covered with: Unit test authentication: Make sure that you have unit tests that check registration, login, password reset and that it is ...


2

You're basically describing GitOps. Here's how I've followed that for a containerized microservices build and deploy pipeline: Each service repo includes it's own Dockerfile, Kubernetes manifests, and anything else needed to locally build and test the project. The base image for the Dockerfile will likely be managed from another repo but built and hosted by ...


Only top voted, non community-wiki answers of a minimum length are eligible