1 00:00:00,000 --> 00:00:02,300 Welcome to another episode of Microsoft Knowledge Nuggets. 2 00:00:02,300 --> 00:00:03,540 I'm your host, Mirko Peters. 3 00:00:03,540 --> 00:00:06,000 What if you could break your own cloud apps on purpose 4 00:00:06,000 --> 00:00:08,160 in a controlled way to make them stronger? 5 00:00:08,160 --> 00:00:10,520 Sounds backwards, but it's the same logic firefighters use 6 00:00:10,520 --> 00:00:11,920 when they set a controlled burn. 7 00:00:11,920 --> 00:00:14,720 They start small fires on purpose to clear out dry brush. 8 00:00:14,720 --> 00:00:17,320 So when a real wildfire comes, there's nothing left to burn. 9 00:00:17,320 --> 00:00:19,560 That's what Azure Chaos Studio does for your cloud apps. 10 00:00:19,560 --> 00:00:20,640 By the end of this episode, 11 00:00:20,640 --> 00:00:22,440 you'll understand what the service is, 12 00:00:22,440 --> 00:00:23,720 why you'd want to use it, 13 00:00:23,720 --> 00:00:25,320 and how it helps you build systems 14 00:00:25,320 --> 00:00:27,360 that don't fall over when things go wrong. 15 00:00:27,360 --> 00:00:30,360 So grab your coffee and let's start with the problem. 16 00:00:30,360 --> 00:00:31,520 The problem. 17 00:00:31,520 --> 00:00:33,720 Why traditional testing isn't enough? 18 00:00:33,720 --> 00:00:35,680 Here's the thing about traditional testing. 19 00:00:35,680 --> 00:00:37,240 It checks whether your software works 20 00:00:37,240 --> 00:00:39,280 as designed under normal conditions. 21 00:00:39,280 --> 00:00:40,640 You write a test, run it, 22 00:00:40,640 --> 00:00:42,600 and if the output matches, you pass. 23 00:00:42,600 --> 00:00:43,640 Great for catching bugs, 24 00:00:43,640 --> 00:00:45,040 but it misses how your system behaves 25 00:00:45,040 --> 00:00:46,680 when something unexpected fails, 26 00:00:46,680 --> 00:00:49,440 like a network timeout, a virtual machine that dies, 27 00:00:49,440 --> 00:00:51,440 or an entire availability zone going dark. 28 00:00:51,440 --> 00:00:53,960 Modern cloud apps aren't simple anymore. 29 00:00:53,960 --> 00:00:56,040 They're distributed systems that talk to databases, 30 00:00:56,040 --> 00:00:58,360 caches, identity services, third party APIs, 31 00:00:58,360 --> 00:00:59,360 and other microservices, 32 00:00:59,360 --> 00:01:01,720 and all those pieces have hidden dependencies. 33 00:01:01,720 --> 00:01:04,280 One small failure can cascade through the whole system. 34 00:01:04,280 --> 00:01:05,960 A database gets slow, the app times out, 35 00:01:05,960 --> 00:01:07,440 the load balancer marks it unhealthy, 36 00:01:07,440 --> 00:01:09,560 traffic reroutes, other instances get overwhelmed, 37 00:01:09,560 --> 00:01:11,680 and suddenly your entire application is down. 38 00:01:11,680 --> 00:01:13,160 Traditional testing doesn't catch that 39 00:01:13,160 --> 00:01:14,760 because it tests in perfect conditions 40 00:01:14,760 --> 00:01:17,240 with everything available, zero network latency, 41 00:01:17,240 --> 00:01:19,880 and no one shutting down VMs during your test run. 42 00:01:19,880 --> 00:01:22,080 So you deploy to production feeling confident, 43 00:01:22,080 --> 00:01:23,360 then reality hits, 44 00:01:23,360 --> 00:01:24,920 and that false confidence is dangerous 45 00:01:24,920 --> 00:01:27,680 because it's the difference between knowing your system works 46 00:01:27,680 --> 00:01:29,120 and knowing it survives. 47 00:01:29,120 --> 00:01:31,120 And that's where chaos engineering comes in. 48 00:01:31,120 --> 00:01:32,880 What chaos engineering actually is? 49 00:01:32,880 --> 00:01:34,880 Chaos engineering is a practical method. 50 00:01:34,880 --> 00:01:37,520 You inject real controlled failures into your system 51 00:01:37,520 --> 00:01:38,800 and observe how it reacts. 52 00:01:38,800 --> 00:01:40,800 This doesn't happen in a lab or a simulation. 53 00:01:40,800 --> 00:01:42,600 It happens on your actual resources. 54 00:01:42,600 --> 00:01:44,920 The idea is simple, you start with a hypothesis. 55 00:01:44,920 --> 00:01:47,080 Something like, if we lose one virtual machine, 56 00:01:47,080 --> 00:01:48,880 the load balancer should keep routing traffic 57 00:01:48,880 --> 00:01:50,400 to the remaining instances. 58 00:01:50,400 --> 00:01:51,560 Then you run an experiment, 59 00:01:51,560 --> 00:01:53,480 you shut down that VM and watch what happens. 60 00:01:53,480 --> 00:01:54,960 Does the load balancer do its job? 61 00:01:54,960 --> 00:01:55,840 Does the app stay up? 62 00:01:55,840 --> 00:01:57,320 Do your alerts fire correctly? 63 00:01:57,320 --> 00:01:59,960 You measure everything and then you fix what breaks. 64 00:01:59,960 --> 00:02:03,800 Here's the thing, chaos engineering is not random destruction. 65 00:02:03,800 --> 00:02:05,840 It's not a developer walking into the server room 66 00:02:05,840 --> 00:02:07,360 and pulling cables for fun. 67 00:02:07,360 --> 00:02:09,960 It's scientific, repeatable, and safety constrained. 68 00:02:09,960 --> 00:02:12,160 You define the blast radius and control exactly 69 00:02:12,160 --> 00:02:14,240 what fails when and for how long. 70 00:02:14,240 --> 00:02:15,880 And most importantly, you learn from it. 71 00:02:15,880 --> 00:02:17,480 Every experiment gives you evidence. 72 00:02:17,480 --> 00:02:19,040 Either your hypothesis was right 73 00:02:19,040 --> 00:02:20,640 and your system is resilient, 74 00:02:20,640 --> 00:02:23,000 or it was wrong and you found a weakness to fix. 75 00:02:23,000 --> 00:02:24,160 Either way, you win. 76 00:02:24,160 --> 00:02:26,880 The challenge is that building your own chaos engineering tooling 77 00:02:26,880 --> 00:02:27,400 is hard. 78 00:02:27,400 --> 00:02:30,600 You'd need to write scripts to shut down VMs, inject CPU pressure, 79 00:02:30,600 --> 00:02:33,400 block network traffic, and orchestrate all that safely 80 00:02:33,400 --> 00:02:34,240 and repeatedly. 81 00:02:34,240 --> 00:02:35,280 That's a lot of work. 82 00:02:35,280 --> 00:02:38,080 And that's exactly why Azure built a managed service to handle it. 83 00:02:38,080 --> 00:02:39,800 Introducing Azure Chaos Studio. 84 00:02:39,800 --> 00:02:42,040 Azure Chaos Studio is a fully managed service 85 00:02:42,040 --> 00:02:45,080 for running chaos experiments directly on your Azure resources. 86 00:02:45,080 --> 00:02:47,360 No custom scripts, no third party tools, 87 00:02:47,360 --> 00:02:49,840 no building your own failure injection platform. 88 00:02:49,840 --> 00:02:52,040 It's all in the Azure portal ready to go. 89 00:02:52,040 --> 00:02:53,080 Here's how it works. 90 00:02:53,080 --> 00:02:55,520 Chaos Studio gives you two ways to inject faults. 91 00:02:55,520 --> 00:02:58,560 Service direct faults run through the Azure control plane itself. 92 00:02:58,560 --> 00:03:00,560 You tell Chaos Studio to shut down a VM 93 00:03:00,560 --> 00:03:02,480 and it does it through Azure's own APIs. 94 00:03:02,480 --> 00:03:03,920 No agent needed. 95 00:03:03,920 --> 00:03:05,920 Agent-based faults install a small agent 96 00:03:05,920 --> 00:03:07,360 inside your virtual machine. 97 00:03:07,360 --> 00:03:09,720 Then it can crank up the CPU to 100%, 98 00:03:09,720 --> 00:03:12,360 fill up memory or kill a specific process. 99 00:03:12,360 --> 00:03:14,120 These let you simulate problems that happen 100 00:03:14,120 --> 00:03:17,000 inside the operating system, not just at the resource level. 101 00:03:17,000 --> 00:03:18,000 Here's an analogy. 102 00:03:18,000 --> 00:03:19,960 Imagine you own a large office building. 103 00:03:19,960 --> 00:03:22,320 You want to know if your fire escapes actually work. 104 00:03:22,320 --> 00:03:24,240 Waiting for a real fire is a terrible idea. 105 00:03:24,240 --> 00:03:25,960 So you hire a dedicated disaster crew. 106 00:03:25,960 --> 00:03:27,520 They come in, block off a stairwell, 107 00:03:27,520 --> 00:03:29,240 simulate smoke on the third floor 108 00:03:29,240 --> 00:03:30,440 and watch how people react. 109 00:03:30,440 --> 00:03:31,480 That's Chaos Studio. 110 00:03:31,480 --> 00:03:32,840 It's your disaster crew. 111 00:03:32,840 --> 00:03:34,760 You call them in, they run the drill, 112 00:03:34,760 --> 00:03:36,600 and you learn whether your building is safe 113 00:03:36,600 --> 00:03:38,440 before the real emergency hits. 114 00:03:38,440 --> 00:03:41,840 Chaos Studio supports a wide range of Azure resources. 115 00:03:41,840 --> 00:03:44,440 Virtual machines, VM scale sets, 116 00:03:44,440 --> 00:03:46,440 Azure Kubernetes service clusters, 117 00:03:46,440 --> 00:03:49,240 Azure Cosmos DB, Azure cache for readers, 118 00:03:49,240 --> 00:03:51,400 network security groups, and more. 119 00:03:51,400 --> 00:03:52,560 The fault library keeps growing. 120 00:03:52,560 --> 00:03:54,680 So whether you're running a simple two-tier app 121 00:03:54,680 --> 00:03:57,360 or a complex microservices architecture on AKS, 122 00:03:57,360 --> 00:03:58,280 you can test it. 123 00:03:58,280 --> 00:03:59,120 Here's the good news. 124 00:03:59,120 --> 00:04:01,080 You don't need to build any custom tooling. 125 00:04:01,080 --> 00:04:03,120 Everything is available through the Azure portal, 126 00:04:03,120 --> 00:04:05,920 the CLI, REST APIs, ARM templates, or BICEP. 127 00:04:05,920 --> 00:04:07,560 You can define your experiments as code 128 00:04:07,560 --> 00:04:09,760 and integrate them into your deployment pipelines. 129 00:04:09,760 --> 00:04:11,120 But let's not get ahead of ourselves. 130 00:04:11,120 --> 00:04:12,680 First, let's break down how these experiments 131 00:04:12,680 --> 00:04:14,240 are actually structured. 132 00:04:14,240 --> 00:04:17,160 The building blocks, targets, steps, branches, actions. 133 00:04:17,160 --> 00:04:19,240 Every chaos experiment starts with targets. 134 00:04:19,240 --> 00:04:20,200 What is a target? 135 00:04:20,200 --> 00:04:23,040 It's simply an Azure resource you want to run faults against. 136 00:04:23,040 --> 00:04:25,400 You open Chaos Studio, find your resource, 137 00:04:25,400 --> 00:04:28,720 a VM, a scale set, an AKS cluster, and you onboard it. 138 00:04:28,720 --> 00:04:30,600 That means you tell Chaos Studio, 139 00:04:30,600 --> 00:04:32,840 this resource is available for experiments. 140 00:04:32,840 --> 00:04:34,040 Here's the key. 141 00:04:34,040 --> 00:04:35,320 When you onboard a target, 142 00:04:35,320 --> 00:04:37,560 you also pick which faults you're willing to allow 143 00:04:37,560 --> 00:04:38,320 on that resource. 144 00:04:38,320 --> 00:04:39,360 You don't enable everything. 145 00:04:39,360 --> 00:04:41,320 You choose CPU pressure, yes. 146 00:04:41,320 --> 00:04:42,400 Full VM shutdown? 147 00:04:42,400 --> 00:04:43,480 Maybe not for production? 148 00:04:43,480 --> 00:04:44,320 You decide. 149 00:04:44,320 --> 00:04:46,960 Once your targets are onboarded, you create an experiment. 150 00:04:46,960 --> 00:04:48,480 Think of an experiment as a blueprint. 151 00:04:48,480 --> 00:04:50,600 It describes exactly what failures to inject 152 00:04:50,600 --> 00:04:53,640 in what order, against which targets, and for how long. 153 00:04:53,640 --> 00:04:55,280 You don't just say break something. 154 00:04:55,280 --> 00:04:57,440 You define the entire scenario upfront. 155 00:04:57,440 --> 00:04:58,960 Experiments have a simple structure. 156 00:04:58,960 --> 00:05:00,640 Steps, branches, and actions. 157 00:05:00,640 --> 00:05:02,040 Steps run one after another. 158 00:05:02,040 --> 00:05:04,720 Step one finishes, then step two starts, then step three. 159 00:05:04,720 --> 00:05:06,360 This is how you model a timeline. 160 00:05:06,360 --> 00:05:08,560 First, degrade the network, then stress the CPU, 161 00:05:08,560 --> 00:05:09,800 then shut down a VM. 162 00:05:09,800 --> 00:05:11,760 Each step is a phase in your scenario. 163 00:05:11,760 --> 00:05:14,480 Inside each step, you can have one or more branches. 164 00:05:14,480 --> 00:05:15,760 Here's where things get interesting. 165 00:05:15,760 --> 00:05:17,240 Branches run in parallel. 166 00:05:17,240 --> 00:05:19,120 So inside step one, you could have branch A 167 00:05:19,120 --> 00:05:21,120 adding latency to your database and branch B 168 00:05:21,120 --> 00:05:24,440 cranking up CPU on your web servers, both at the same time. 169 00:05:24,440 --> 00:05:27,000 This lets you simulate complex, multi-fault scenarios 170 00:05:27,000 --> 00:05:28,680 that mirror real-world incidents. 171 00:05:28,680 --> 00:05:29,760 Then you have actions. 172 00:05:29,760 --> 00:05:31,320 Actions are the actual faults. 173 00:05:31,320 --> 00:05:34,920 CPU pressure, memory exhaustion, VM shutdown, network delay, 174 00:05:34,920 --> 00:05:36,800 DNS block, process kill. 175 00:05:36,800 --> 00:05:39,920 Each action targets a specific resource or group of resources. 176 00:05:39,920 --> 00:05:41,520 And again, you can only use actions 177 00:05:41,520 --> 00:05:43,320 that you've enabled on that target. 178 00:05:43,320 --> 00:05:44,240 That's your safety net. 179 00:05:44,240 --> 00:05:46,280 If you didn't enable VM shutdown on a target, 180 00:05:46,280 --> 00:05:48,320 the experiment simply won't run that action. 181 00:05:48,320 --> 00:05:51,080 So the workflow looks like this, onboard your targets. 182 00:05:51,080 --> 00:05:52,920 Enable the faults you're comfortable with. 183 00:05:52,920 --> 00:05:55,560 Create an experiment with steps, branches, and actions. 184 00:05:55,560 --> 00:05:56,240 Then run it. 185 00:05:56,240 --> 00:05:57,600 But before we get into running one, 186 00:05:57,600 --> 00:05:59,240 there's something important to understand. 187 00:05:59,240 --> 00:06:00,840 When chaos studio runs an experiment, 188 00:06:00,840 --> 00:06:02,120 it's not simulating anything. 189 00:06:02,120 --> 00:06:04,120 It's actually doing these things to your resources. 190 00:06:04,120 --> 00:06:06,200 CPU pressure is real CPU pressure. 191 00:06:06,200 --> 00:06:07,720 A VM shutdown is a real shutdown. 192 00:06:07,720 --> 00:06:08,600 That's the whole point. 193 00:06:08,600 --> 00:06:10,160 You're testing how your system behaves 194 00:06:10,160 --> 00:06:11,720 under actual failure conditions. 195 00:06:11,720 --> 00:06:13,080 And that means you need to be careful. 196 00:06:13,080 --> 00:06:14,560 So let's walk through a real experiment 197 00:06:14,560 --> 00:06:16,240 and see how it all comes together. 198 00:06:16,240 --> 00:06:18,240 Running your first chaos experiment. 199 00:06:18,240 --> 00:06:19,240 Let's make this real. 200 00:06:19,240 --> 00:06:21,040 Imagine you have a web application running 201 00:06:21,040 --> 00:06:23,440 on three virtual machines inside a virtual machine 202 00:06:23,440 --> 00:06:24,320 scale set. 203 00:06:24,320 --> 00:06:26,760 Each VM leaves in a different availability zone. 204 00:06:26,760 --> 00:06:28,920 You have a load balancer in front distributing traffic 205 00:06:28,920 --> 00:06:30,040 across all three. 206 00:06:30,040 --> 00:06:31,760 The hypothesis is simple. 207 00:06:31,760 --> 00:06:34,040 If we lose an entire availability zone, 208 00:06:34,040 --> 00:06:35,880 the load balancer should reroute traffic 209 00:06:35,880 --> 00:06:39,120 to the remaining two VMs and the application stays available. 210 00:06:39,120 --> 00:06:41,440 So you design an experiment with two steps. 211 00:06:41,440 --> 00:06:46,120 Step one, stress two of the three VMs with 95% CPU for five minutes. 212 00:06:46,120 --> 00:06:49,760 That simulates a noisy neighbor scenario or a runaway process. 213 00:06:49,760 --> 00:06:53,040 Step two, abruptly shut down all the VMs in zone two. 214 00:06:53,040 --> 00:06:55,120 That simulates a real zone outage. 215 00:06:55,120 --> 00:06:57,520 You run the experiment and then you watch what happens. 216 00:06:57,520 --> 00:06:58,560 Here's what you'd see. 217 00:06:58,560 --> 00:07:02,400 In the first five minutes, two VMs, spike to 100% CPU, 218 00:07:02,400 --> 00:07:04,040 the third one stays normal because you didn't 219 00:07:04,040 --> 00:07:04,640 target it. 220 00:07:04,640 --> 00:07:06,160 You're monitoring dashboards light up. 221 00:07:06,160 --> 00:07:07,040 Alerts fire. 222 00:07:07,040 --> 00:07:09,840 The load balancer notices that those two VMs are struggling, 223 00:07:09,840 --> 00:07:11,440 but because they're still technically running, 224 00:07:11,440 --> 00:07:12,800 it keeps sending them traffic. 225 00:07:12,800 --> 00:07:14,760 Your app slows down but doesn't go offline. 226 00:07:14,760 --> 00:07:15,520 That's interesting. 227 00:07:15,520 --> 00:07:16,920 You learn something already. 228 00:07:16,920 --> 00:07:18,120 Then step two kicks in. 229 00:07:18,120 --> 00:07:20,920 The VMs in zone two are abruptly shut down, not gracefully, 230 00:07:20,920 --> 00:07:21,680 not with a warning. 231 00:07:21,680 --> 00:07:22,560 They just disappear. 232 00:07:22,560 --> 00:07:25,720 One of those two stressed VMs was in that zone, so it's gone. 233 00:07:25,720 --> 00:07:27,720 The load balancer sees the health probe fail, 234 00:07:27,720 --> 00:07:29,320 removes it from the back end pool, 235 00:07:29,320 --> 00:07:31,640 and routes all traffic to the remaining healthy VMs. 236 00:07:31,640 --> 00:07:32,480 Your app stays up. 237 00:07:32,480 --> 00:07:34,040 The hypothesis holds. 238 00:07:34,040 --> 00:07:35,920 But here's what you might not expect. 239 00:07:35,920 --> 00:07:38,680 The remaining VMs now have to handle 100% of the traffic. 240 00:07:38,680 --> 00:07:41,040 If you didn't have enough capacity to absorb that load, 241 00:07:41,040 --> 00:07:42,360 you'd see latency spike. 242 00:07:42,360 --> 00:07:44,280 Maybe the app stays technically online, 243 00:07:44,280 --> 00:07:45,600 but users feel the pain. 244 00:07:45,600 --> 00:07:46,480 That's a finding. 245 00:07:46,480 --> 00:07:49,000 You now know you need either more headroom or faster 246 00:07:49,000 --> 00:07:49,880 auto scaling. 247 00:07:49,880 --> 00:07:51,640 And remember, this isn't a simulation. 248 00:07:51,640 --> 00:07:53,560 When chaos studio runs this experiment, 249 00:07:53,560 --> 00:07:56,400 those VMs actually hit 95% CPU. 250 00:07:56,400 --> 00:07:58,160 That zone two shut down is a real shut down. 251 00:07:58,160 --> 00:08:00,360 The VMs stop the load balancer rebalances, 252 00:08:00,360 --> 00:08:02,480 and your application either survives or it doesn't. 253 00:08:02,480 --> 00:08:03,320 That's the whole point. 254 00:08:03,320 --> 00:08:04,200 You're not guessing. 255 00:08:04,200 --> 00:08:07,600 You're watching real behavior under real failure conditions. 256 00:08:07,600 --> 00:08:10,080 You learn whether your app stays up or falls over. 257 00:08:10,080 --> 00:08:12,240 And if it falls over, you fix it before your customers ever 258 00:08:12,240 --> 00:08:13,040 see it happen. 259 00:08:13,040 --> 00:08:15,320 But this power comes with responsibility. 260 00:08:15,320 --> 00:08:16,880 You need to be careful. 261 00:08:16,880 --> 00:08:19,600 Safety first, avoiding unplanned outages. 262 00:08:19,600 --> 00:08:22,520 Chaos studio has multiple safety mechanisms built in. 263 00:08:22,520 --> 00:08:24,240 And before you run your first experiment, 264 00:08:24,240 --> 00:08:26,200 you need to understand every single one. 265 00:08:26,200 --> 00:08:28,720 The first safety layer is identity and permissions. 266 00:08:28,720 --> 00:08:30,760 Every experiment you create gets its own system 267 00:08:30,760 --> 00:08:32,280 assigned managed identity. 268 00:08:32,280 --> 00:08:35,040 But that identity starts with zero permissions by default. 269 00:08:35,040 --> 00:08:38,160 You have to explicitly granted access to the resources it needs. 270 00:08:38,160 --> 00:08:40,400 And you grant the minimum permissions required. 271 00:08:40,400 --> 00:08:43,680 If an experiment only needs to stress CPU on two VMs, 272 00:08:43,680 --> 00:08:46,080 you scope the permissions down tight instead of allowing 273 00:08:46,080 --> 00:08:48,200 it to shut down everything in the resource group. 274 00:08:48,200 --> 00:08:50,560 Then there's the fault enablement I mentioned earlier. 275 00:08:50,560 --> 00:08:53,160 When you onboard a target, you choose exactly which faults 276 00:08:53,160 --> 00:08:54,240 you're willing to allow. 277 00:08:54,240 --> 00:08:56,880 If you never enable VM shut down on a production scale set, 278 00:08:56,880 --> 00:08:59,520 no experiment can shut down those VMs, 279 00:08:59,520 --> 00:09:01,720 even if someone targets them by accident. 280 00:09:01,720 --> 00:09:04,160 It's a hard block that requires you to deliberately opt 281 00:09:04,160 --> 00:09:05,440 into each fault type. 282 00:09:05,440 --> 00:09:07,600 Now about where to start and the answer is simple. 283 00:09:07,600 --> 00:09:10,160 Start small and start in a non-production environment. 284 00:09:10,160 --> 00:09:12,600 Run your first experiments in dev or staging, 285 00:09:12,600 --> 00:09:14,880 test with a single VM and a single fault, 286 00:09:14,880 --> 00:09:17,720 and validate that your monitoring works, your alerts fire, 287 00:09:17,720 --> 00:09:20,240 and you can cancel the experiment if something goes wrong. 288 00:09:20,240 --> 00:09:23,360 Only after you've built that confidence in a safe environment, 289 00:09:23,360 --> 00:09:25,480 should you even think about touching production. 290 00:09:25,480 --> 00:09:27,000 Microsoft has a warning about this 291 00:09:27,000 --> 00:09:28,320 that I want to share directly. 292 00:09:28,320 --> 00:09:30,840 They call it a resume in generating event. 293 00:09:30,840 --> 00:09:32,600 If you run a chaos experiment in production 294 00:09:32,600 --> 00:09:35,480 without proper safeguards and it takes down your entire application, 295 00:09:35,480 --> 00:09:37,440 that's the kind of mistake that could get you fired. 296 00:09:37,440 --> 00:09:38,600 That's not hyperbole. 297 00:09:38,600 --> 00:09:40,680 Chaos Studio is powerful because it actually 298 00:09:40,680 --> 00:09:43,960 does things to your resources, so use that power wisely. 299 00:09:43,960 --> 00:09:46,760 The last safety net is monitoring and cancellation. 300 00:09:46,760 --> 00:09:50,120 While an experiment runs, you can watch the progress in real time. 301 00:09:50,120 --> 00:09:53,000 You see which steps completed, which branches are active, 302 00:09:53,000 --> 00:09:54,840 and which actions are executing. 303 00:09:54,840 --> 00:09:57,640 If something looks wrong, you hit cancel and the experiment 304 00:09:57,640 --> 00:09:58,760 stops immediately. 305 00:09:58,760 --> 00:10:01,600 The resources that were affected stay in whatever state they're in, 306 00:10:01,600 --> 00:10:03,040 but no new faults get injected. 307 00:10:03,040 --> 00:10:05,760 So keep your dashboards open and your finger on the cancel button. 308 00:10:05,760 --> 00:10:08,520 Start small, be deliberate, and respect the power of the tool. 309 00:10:08,520 --> 00:10:11,080 So now that you know how to stay safe, the question is, 310 00:10:11,080 --> 00:10:13,680 where should you actually run these experiments? 311 00:10:13,680 --> 00:10:15,760 Shifting left versus shifting right. 312 00:10:15,760 --> 00:10:18,120 So where do you actually run these experiments? 313 00:10:18,120 --> 00:10:21,120 There are two main approaches and most teams use both. 314 00:10:21,120 --> 00:10:23,320 The first is shifting left. 315 00:10:23,320 --> 00:10:26,720 You run chaos experiments early in your development life cycle. 316 00:10:26,720 --> 00:10:29,240 As soon as code gets deployed to a test environment, 317 00:10:29,240 --> 00:10:32,640 you hit it with faults and make it part of your CICD pipeline. 318 00:10:32,640 --> 00:10:35,800 The pipeline deploys the app runs a chaos experiment to stress it, 319 00:10:35,800 --> 00:10:39,560 checks that it survives, and only then promotes the build to the next stage. 320 00:10:39,560 --> 00:10:43,800 Think of it as a quality gate, just like unit tests or security scans. 321 00:10:43,800 --> 00:10:47,800 If the app can't handle a VM shutdown in staging, it doesn't go to production. 322 00:10:47,800 --> 00:10:51,880 The second approach is shifting right, which means running experiments in production. 323 00:10:51,880 --> 00:10:54,240 And yes, that sounds terrifying, but hear me out. 324 00:10:54,240 --> 00:10:55,200 You don't start there. 325 00:10:55,200 --> 00:10:58,960 You only shift right after you've built confidence in non-production environments. 326 00:10:58,960 --> 00:11:01,360 You begin with small, limited experiments. 327 00:11:01,360 --> 00:11:04,960 Maybe you stress CPU on one instance during a low traffic period. 328 00:11:04,960 --> 00:11:07,000 You watch closely, you learn, and then you expand. 329 00:11:07,000 --> 00:11:10,320 Maybe you shut down a single VM in production during business hours 330 00:11:10,320 --> 00:11:13,320 and validate that your load balancer actually re-route traffic, 331 00:11:13,320 --> 00:11:15,080 that your users don't notice a thing, 332 00:11:15,080 --> 00:11:16,960 and that your monitoring catches it all. 333 00:11:16,960 --> 00:11:19,960 Over time, you build up to more aggressive experiments. 334 00:11:19,960 --> 00:11:22,600 There's a third approach that sits in the middle, game days. 335 00:11:22,600 --> 00:11:24,720 A game day is a scheduled chaos drill, 336 00:11:24,720 --> 00:11:28,040 where you bring together your development team, operations team, security team, 337 00:11:28,040 --> 00:11:30,280 everyone who would be involved in a real incident. 338 00:11:30,280 --> 00:11:33,080 You run a chaos experiment and practice your incident response. 339 00:11:33,080 --> 00:11:35,160 Who gets the alert, who makes the call to escalate? 340 00:11:35,160 --> 00:11:36,280 What's the communication chain? 341 00:11:36,280 --> 00:11:38,280 How fast can you diagnose the issue? 342 00:11:38,280 --> 00:11:40,440 Game days test the technology, but more importantly, 343 00:11:40,440 --> 00:11:42,720 they test the people and processes around it. 344 00:11:42,720 --> 00:11:46,920 The key idea across all three approaches is that chaos engineering is not a one-time test. 345 00:11:46,920 --> 00:11:50,360 You don't run one experiment, declare your system resilient, and move on. 346 00:11:50,360 --> 00:11:54,160 Systems change, code changes, configurations change, dependencies change. 347 00:11:54,160 --> 00:11:56,600 What was resilient last month might be fragile today. 348 00:11:56,600 --> 00:11:59,600 The goal is continuous resilience validation. 349 00:11:59,600 --> 00:12:03,680 You run experiments regularly, integrate them into your release process, 350 00:12:03,680 --> 00:12:05,680 and make them part of your operational rhythm. 351 00:12:05,680 --> 00:12:08,040 So what's the real value of all this effort? 352 00:12:08,040 --> 00:12:10,160 The real value, building confidence. 353 00:12:10,160 --> 00:12:12,080 Chaos engineering does more than find bugs. 354 00:12:12,080 --> 00:12:14,640 It builds confidence, and that's a completely different thing. 355 00:12:14,640 --> 00:12:16,160 A bug is a specific defect. 356 00:12:16,160 --> 00:12:18,480 You fix it and move on, but confidence is a feeling 357 00:12:18,480 --> 00:12:21,440 it's knowing your system can survive the unexpected, 358 00:12:21,440 --> 00:12:24,800 and you can't get that from unit tests or integration tests alone. 359 00:12:24,800 --> 00:12:28,600 You get it from watching your system handle a real failure and come out the other side. 360 00:12:28,600 --> 00:12:31,880 This process forces you to improve things you might otherwise ignore. 361 00:12:31,880 --> 00:12:33,120 You're monitoring, for example. 362 00:12:33,120 --> 00:12:37,720 When you run a chaos experiment, you quickly discover whether your dashboards show the right metrics, 363 00:12:37,720 --> 00:12:40,040 whether your alerts fire at the right thresholds, 364 00:12:40,040 --> 00:12:43,120 and whether your on-call team gets notified in time. 365 00:12:43,120 --> 00:12:45,800 If you can't see the failure happening, you can't respond to it. 366 00:12:45,800 --> 00:12:47,640 Chaos experiments expose those gaps. 367 00:12:47,640 --> 00:12:49,480 They also uncover hidden assumptions. 368 00:12:49,480 --> 00:12:51,120 Every architect has said something like, 369 00:12:51,120 --> 00:12:55,640 "If the database goes down, the app should just cache the last known data and keep running." 370 00:12:55,640 --> 00:12:58,000 That sounds good on a whiteboard, but does it actually work? 371 00:12:58,000 --> 00:12:59,080 Have you ever tested it? 372 00:12:59,080 --> 00:13:01,920 Chaos experiments answer that question with real evidence. 373 00:13:01,920 --> 00:13:03,800 You shut down the database and watch what happens. 374 00:13:03,800 --> 00:13:05,480 Maybe the app handles it gracefully. 375 00:13:05,480 --> 00:13:08,600 Maybe it crashes, either way you learn something you didn't know before. 376 00:13:08,600 --> 00:13:12,480 Chaos engineering is most powerful when you combine it with other resilience practices. 377 00:13:12,480 --> 00:13:14,000 Load testing, for example. 378 00:13:14,000 --> 00:13:17,760 You can stress your system with simulated traffic and inject faults at the same time, 379 00:13:17,760 --> 00:13:21,040 and that's a much more realistic test than either one alone. 380 00:13:21,040 --> 00:13:24,680 Tools like Azure Monitor and Application Insights give you the data to understand 381 00:13:24,680 --> 00:13:26,160 what happened during the experiment. 382 00:13:26,160 --> 00:13:28,520 Together they form a full resilience toolkit. 383 00:13:28,520 --> 00:13:31,400 Load testing tells you if your system can handle the traffic. 384 00:13:31,400 --> 00:13:33,960 Chaos engineering tells you if it can survive the failures. 385 00:13:33,960 --> 00:13:35,800 The result is fewer surprises. 386 00:13:35,800 --> 00:13:39,320 When a real outage happens, you've already seen something like it before. 387 00:13:39,320 --> 00:13:42,720 Your runbooks are tested, your team knows what to do, and your monitoring is tuned. 388 00:13:42,720 --> 00:13:43,920 You recover faster. 389 00:13:43,920 --> 00:13:45,200 Your user's barely noticed. 390 00:13:45,200 --> 00:13:46,400 That's the whole point. 391 00:13:46,400 --> 00:13:49,480 Not to break things for fun, but to build systems that don't break easily. 392 00:13:49,480 --> 00:13:53,040 And when they do break, to recover so fast nobody remembers it happened. 393 00:13:53,040 --> 00:13:54,680 So that's Azure Chaos Studio. 394 00:13:54,680 --> 00:13:58,000 It lets you test your systems resilience by injecting controlled failures 395 00:13:58,000 --> 00:14:00,400 before your customers ever experience them. 396 00:14:00,400 --> 00:14:03,840 If this episode made sense of another piece of the Microsoft Cloud puzzle, 397 00:14:03,840 --> 00:14:07,200 hit subscribe and share it with someone starting their cloud journey. 398 00:14:07,200 --> 00:14:08,040 They'll thank you later.