What Is Chaos Engineering? A Practical Guide to Starting Small

Your system is designed to survive a server failure. The load balancer should route around it, autoscaling should replace it and nobody should notice. But have you ever actually watched that happen? Or are you trusting a diagram?

Chaos engineering is how teams find out. Instead of waiting for a real outage to test their assumptions, they cause small, controlled failures on purpose, watch what happens and fix what breaks. It sounds reckless. Done properly, it is one of the most careful things an engineering team can do.

This guide explains what chaos engineering is, how an experiment works, and how to start without putting customers at risk.

Chaos engineering in plain English

The Principles of Chaos Engineering, a community definition that grew out of Netflix’s work, describes it as the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production.

The key word is experimenting. Chaos engineering is not randomly breaking things. It follows the scientific method:

1. Define what “normal” looks like
2. Predict that normal behavior will continue during a failure
3. Introduce a realistic failure
4. Look for differences that prove the prediction wrong

If the system stays healthy, you have evidence your resilience design works. If it does not, you found a weakness on your schedule, with engineers watching, instead of at 3 a.m. during a real incident.

Key takeaway: Chaos engineering is controlled experimentation, not random destruction. Every experiment starts with a hypothesis, a small blast radius and a way to stop immediately.

Where it came from

Netflix popularized the idea while moving its streaming service to Amazon Web Services. Its engineers built a tool called Chaos Monkey, which randomly terminates virtual machine instances and containers, to force teams to design services that tolerate losing servers at any time. In a July 2011 blog post, Netflix introduced the “Simian Army,” a family of tools that included Chaos Monkey and others simulating bigger failures. Chaos Monkey is now open source.

Since then, the practice has spread well beyond streaming. Cloud providers now offer managed services for it, and AWS’s Well-Architected Framework includes testing resilience with chaos engineering as a reliability best practice.

How a chaos experiment works

1. Define steady state

Steady state is a measurable output that shows the system is behaving normally. Pick business-level or user-facing metrics, not just server stats:

• Successful checkouts per minute
• Request error rate and latency
• Video streams started, messages delivered or orders processed

If you already track service level indicators and objectives, they make excellent steady-state measures.

2. Form a hypothesis

Write down what you expect: “If one of the 3 web servers fails, error rate will stay below 0.5% and the load balancer will stop sending it traffic within 30 seconds.”

A good hypothesis is specific and measurable. “The system will be fine” is not.

3. Choose a realistic failure

Base it on things that really happen or have happened to you. Past incidents and your postmortems are the best source of ideas.

4. Limit the blast radius

The Principles of Chaos Engineering say the experimenter is responsible for keeping fallout from experiments as small and contained as possible. Start with one instance, one service, a small share of traffic or a non-production environment. Set automatic stop conditions, such as “abort if error rate exceeds 1%.”

5. Run, observe and stop if needed

Run the experiment while watching your dashboards. You need good observability to tell whether the system is coping. If a stop condition triggers, halt and restore immediately.

6. Learn and fix

Compare results with your hypothesis. If something broke, treat it like an incident: document it, fix the weakness and rerun the experiment to confirm the fix.

Common failures to inject

Zone and failover tests connect directly to your recovery time and recovery point objectives. They show whether your documented recovery plan actually works.

Tools for chaos engineering

You do not need a dedicated product to start. Stopping a test server by hand during a planned session is a valid first experiment. As you mature, tools add safety controls, scheduling and reporting.

AWS Fault Injection Service, for example, uses experiment templates made of actions, targets and stop conditions tied to CloudWatch alarms, and is billed per minute that actions run. AWS’s documentation warns that it performs real actions on real resources and strongly recommends planning and running experiments in pre-production first. Azure Chaos Studio’s newer Workspaces and Scenarios model, which offers ready-made scenarios such as a compute zone outage, was in public preview as of its latest documentation, so check the current status before relying on it.

Is chaos engineering safe?

It is safe when it is disciplined. The risk is not the practice itself but skipping the controls. Follow these rules:

• Never surprise people. Announce experiments to on-call staff and stakeholders.
• Start outside production. Use staging or a test environment until your process is mature.
• Have a stop button. Automated stop conditions plus a manual abort.
• Keep the blast radius small. One host, one service, a small percentage of traffic.
• Avoid peak times for early production experiments.
• Do not run experiments during active incidents or major launches.
• Get approval from service owners, and document what you will do and how you will undo it.

Running in production is the end goal in the Principles of Chaos Engineering, because staging rarely matches real traffic. But it is a destination, not a starting point.

Game days: the easiest way to start

A game day is a scheduled session where a team runs a chaos experiment together, often in a test environment, and practices its incident response at the same time. AWS’s reliability guidance recommends running game days regularly.

A simple first game day:

1. Pick one service with a clear resilience claim, such as “survives losing one instance.”
2. Gather the team for 2 hours, including whoever is on call.
3. Agree on steady state, hypothesis and stop conditions, and write them down.
4. Inject the failure in staging, for example by stopping one instance.
5. Observe dashboards, alerts and logs. Did alerts fire? Did the runbook help?
6. Restore the system and confirm it returned to steady state.
7. Write up findings and create owned, dated action items.

Game days test people and processes as much as technology. You often learn that an alert went to the wrong channel or a runbook was out of date, which is just as valuable as finding a code bug.

Where to start

1. Get the basics right first. Chaos engineering assumes you have monitoring, alerts and some redundancy. If a single server failure would take you down, fix that before experimenting. Our guide to high availability covers the design side.
2. List your resilience assumptions. “We survive losing a zone.” “The app works if the cache is down.” Each one is a future experiment.
3. Run your first game day in staging using the steps above.
4. Turn successful experiments into automated tests that run on a schedule or in your deployment pipeline, as AWS’s guidance recommends.
5. Move carefully toward production with tiny blast radius and strong stop conditions once staging experiments are routine.

Chaos engineering sits naturally alongside site reliability engineering practices and progressive delivery techniques like canary deployments.

Conclusion

Every system has weaknesses. The only question is whether you find them during a calm, planned experiment or during a real outage. Chaos engineering gives you a structured way to test your resilience assumptions: define normal, predict, inject a small failure, watch and fix. Start with a game day in a test environment, keep the blast radius tiny, and build up from there.

Sources

• Principles of Chaos Engineering — definition, experiment steps and advanced principles including minimizing blast radius
• Netflix Chaos Monkey on GitHub — what Chaos Monkey does and its open source release
• Netflix Tech Blog: The Netflix Simian Army — 2011 introduction of the Simian Army
• AWS Well-Architected Framework: Test resiliency using chaos engineering — chaos engineering as a reliability best practice and tool options
• AWS Fault Injection Service User Guide — experiment templates, stop conditions, pre-production recommendation and pricing model
• Microsoft Learn: What is Azure Chaos Studio? — Workspaces, Scenarios and fault types