1 00:00:00,000 --> 00:00:03,000 So what exactly changes when a company moves from one AI chatbot 2 00:00:03,000 --> 00:00:04,480 to dozens of AI tools? 3 00:00:04,480 --> 00:00:06,240 At first, it feels simple. 4 00:00:06,240 --> 00:00:09,120 One team adds a chatbot for customer support, 5 00:00:09,120 --> 00:00:11,800 another uses AI to search company documents, 6 00:00:11,800 --> 00:00:14,440 and meeting notes get written automatically. 7 00:00:14,440 --> 00:00:16,840 A workflow can read an email, pull out a request, 8 00:00:16,840 --> 00:00:18,440 and send it to the right system. 9 00:00:18,440 --> 00:00:21,000 Before long, the company has several models, agents 10 00:00:21,000 --> 00:00:24,320 that can take actions, and tools that reach into business data. 11 00:00:24,320 --> 00:00:27,320 Each team often connects straight to a different AI provider, 12 00:00:27,320 --> 00:00:29,640 using its own key, its own limits, 13 00:00:29,640 --> 00:00:30,880 and its own safety rules. 14 00:00:30,880 --> 00:00:32,400 Costs become hard to trace. 15 00:00:32,400 --> 00:00:34,080 A key can end up in the wrong place. 16 00:00:34,080 --> 00:00:35,840 And when an agent calls a tool, nobody 17 00:00:35,840 --> 00:00:38,040 may have one clear record of what it tried to do. 18 00:00:38,040 --> 00:00:40,240 This is where Microsoft AI Gateway comes in. 19 00:00:40,240 --> 00:00:42,000 By the end of this episode, you'll understand 20 00:00:42,000 --> 00:00:44,200 what it is, how its building blocks fit together, 21 00:00:44,200 --> 00:00:46,360 and when it starts making sense for your business. 22 00:00:46,360 --> 00:00:49,400 Think of it like an office building with lots of side doors. 23 00:00:49,400 --> 00:00:50,600 You could leave every door open 24 00:00:50,600 --> 00:00:52,240 and hope people use them properly, 25 00:00:52,240 --> 00:00:55,520 or you can bring everyone through one managed entrance. 26 00:00:55,520 --> 00:00:57,000 One front door for AI. 27 00:00:57,000 --> 00:01:00,720 Microsoft AI Gateway is a set of Azure API management features 28 00:01:00,720 --> 00:01:03,280 that gives your AI traffic one control point. 29 00:01:03,280 --> 00:01:04,960 Let's break it down in plain English. 30 00:01:04,960 --> 00:01:07,840 Imagine your company's AI setup as a modern office building. 31 00:01:07,840 --> 00:01:10,600 The AI models are rooms where the thinking happens. 32 00:01:10,600 --> 00:01:12,720 Your agents are workers moving through the building, 33 00:01:12,720 --> 00:01:14,760 deciding which approved task to do next. 34 00:01:14,760 --> 00:01:17,880 Tools are service doors that lead to places like HR, 35 00:01:17,880 --> 00:01:20,840 finance, customer records, and internal files. 36 00:01:20,840 --> 00:01:22,720 The AI Gateway is the reception desk 37 00:01:22,720 --> 00:01:24,760 and the security desk at the front. 38 00:01:24,760 --> 00:01:26,240 It checks who is coming in, 39 00:01:26,240 --> 00:01:28,720 applies the rules and records where requests go 40 00:01:28,720 --> 00:01:31,080 before they reach the rest of the building. 41 00:01:31,080 --> 00:01:33,000 A request follows a simple path. 42 00:01:33,000 --> 00:01:36,720 An app, a co-pilot, or an agent sends a request to the gateway. 43 00:01:36,720 --> 00:01:39,360 The gateway checks the rules that apply to that caller, 44 00:01:39,360 --> 00:01:43,800 then sends it to the approved model, tool, agent, or business API. 45 00:01:43,800 --> 00:01:47,120 And when a response comes back, it passes through the gateway again 46 00:01:47,120 --> 00:01:48,800 before it reaches the user. 47 00:01:48,800 --> 00:01:51,760 That one path gives the business a place to manage access, 48 00:01:51,760 --> 00:01:53,680 limits, safety checks, and logging 49 00:01:53,680 --> 00:01:56,560 without building those controls into every single app. 50 00:01:56,560 --> 00:01:59,560 This also separates two jobs that often get mixed together. 51 00:01:59,560 --> 00:02:02,800 Application teams can focus on the experience people actually use. 52 00:02:02,800 --> 00:02:05,360 They build the support assistant, the document search screen, 53 00:02:05,360 --> 00:02:07,240 or the workflow that saves time. 54 00:02:07,240 --> 00:02:09,240 Meanwhile, the platform team manages 55 00:02:09,240 --> 00:02:13,000 the shared rules behind the scenes, deciding which services are approved, 56 00:02:13,000 --> 00:02:14,800 which identities can use them, 57 00:02:14,800 --> 00:02:17,600 how much traffic a project can send, and what gets logged. 58 00:02:17,600 --> 00:02:20,880 You don't want every team inventing a new security plan for every chatbot. 59 00:02:20,880 --> 00:02:23,360 Microsoft AI Gateway can sit in front of several kinds 60 00:02:23,360 --> 00:02:24,280 of AI traffic. 61 00:02:24,280 --> 00:02:27,480 It can manage calls to AI models and model context protocol, 62 00:02:27,480 --> 00:02:29,080 MCP tool calls. 63 00:02:29,080 --> 00:02:32,160 It can also manage agent to agent, A2A traffic, 64 00:02:32,160 --> 00:02:34,480 and the business APIs your company already uses, 65 00:02:34,480 --> 00:02:37,920 such as an API for checking an order, reading an HR policy, 66 00:02:37,920 --> 00:02:39,320 or opening a support ticket. 67 00:02:39,320 --> 00:02:41,360 That last part matters because most companies 68 00:02:41,360 --> 00:02:44,360 don't need to throw away their existing systems to use AI. 69 00:02:44,360 --> 00:02:47,200 They just need a safe way for AI to work with those systems, 70 00:02:47,200 --> 00:02:50,200 and a gateway helps put controls around that connection. 71 00:02:50,200 --> 00:02:52,480 Now Microsoft AI gateway isn't a model. 72 00:02:52,480 --> 00:02:53,800 It doesn't write answers itself. 73 00:02:53,800 --> 00:02:55,040 It isn't an agent builder either, 74 00:02:55,040 --> 00:02:57,080 so you don't use it to design an HR assistant 75 00:02:57,080 --> 00:02:58,720 or teach a finance agent what to do. 76 00:02:58,720 --> 00:03:00,760 And it doesn't replace Microsoft Foundry, 77 00:03:00,760 --> 00:03:03,440 where teams can build, test, and manage AI projects. 78 00:03:03,440 --> 00:03:05,280 Instead, it sits in the middle. 79 00:03:05,280 --> 00:03:07,480 Foundry can help you work with AI models and agents 80 00:03:07,480 --> 00:03:11,160 while Azure API Management AI Gateway helps control the traffic 81 00:03:11,160 --> 00:03:13,560 those models, agents, and tools create. 82 00:03:13,560 --> 00:03:15,480 You may already know Azure API management 83 00:03:15,480 --> 00:03:17,600 as a way to manage normal APIs. 84 00:03:17,600 --> 00:03:19,880 An API is simply a way for one piece of software 85 00:03:19,880 --> 00:03:22,440 to ask another piece of software for something. 86 00:03:22,440 --> 00:03:25,040 AI Gateway extends that same idea for AI work 87 00:03:25,040 --> 00:03:27,080 because AI requests bring new questions. 88 00:03:27,080 --> 00:03:28,120 Who used which model? 89 00:03:28,120 --> 00:03:29,760 How many tokens did it consume? 90 00:03:29,760 --> 00:03:31,440 Can this agent call that tool? 91 00:03:31,440 --> 00:03:33,320 Should this prompt reach the model at all? 92 00:03:33,320 --> 00:03:35,560 Putting the gateway in the middle gives you one place 93 00:03:35,560 --> 00:03:37,960 to answer those questions, and the first pressure point 94 00:03:37,960 --> 00:03:39,680 appears when every application starts 95 00:03:39,680 --> 00:03:42,040 choosing its own model endpoint. 96 00:03:42,040 --> 00:03:44,280 One endpoint, many models. 97 00:03:44,280 --> 00:03:45,960 When your application connects directly 98 00:03:45,960 --> 00:03:48,120 to a single model endpoint, that choice 99 00:03:48,120 --> 00:03:50,280 gets baked into the code itself. 100 00:03:50,280 --> 00:03:52,360 Often the code is tied to one provider 101 00:03:52,360 --> 00:03:54,920 one specific deployment, one Azure region, 102 00:03:54,920 --> 00:03:56,360 and a fixed set of limits. 103 00:03:56,360 --> 00:03:57,960 That works fine while the project is small, 104 00:03:57,960 --> 00:04:00,360 but imagine your support assistant runs on one model 105 00:04:00,360 --> 00:04:01,280 in one region. 106 00:04:01,280 --> 00:04:03,480 Months later, that model hits its capacity. 107 00:04:03,480 --> 00:04:05,480 A better model comes along for less money, 108 00:04:05,480 --> 00:04:07,680 or your business needs to move to another region. 109 00:04:07,680 --> 00:04:11,200 Without a shared layer, every app team has to change their code, 110 00:04:11,200 --> 00:04:13,320 test it again, and roll out an update. 111 00:04:13,320 --> 00:04:15,240 That makes for slow, messy change. 112 00:04:15,240 --> 00:04:18,640 Microsoft AI Gateway gives your applications a stable front door. 113 00:04:18,640 --> 00:04:19,960 They send requests there, 114 00:04:19,960 --> 00:04:21,800 and the model choice can change behind the scenes 115 00:04:21,800 --> 00:04:23,400 without you having to touch the app. 116 00:04:23,400 --> 00:04:25,120 Think of it like a company phone number. 117 00:04:25,120 --> 00:04:26,760 Customers dial one number, 118 00:04:26,760 --> 00:04:29,240 even if the call gets passed to a different team or office. 119 00:04:29,240 --> 00:04:30,920 They never need to know how it's handled. 120 00:04:30,920 --> 00:04:33,720 With AI Gateway, your app calls one approved endpoint, 121 00:04:33,720 --> 00:04:35,040 and behind that endpoint, 122 00:04:35,040 --> 00:04:37,920 the gateway sends the request to the model that fits the job. 123 00:04:37,920 --> 00:04:39,520 Microsoft also has a new approach 124 00:04:39,520 --> 00:04:41,840 called the unified model API. 125 00:04:41,840 --> 00:04:44,200 In plain English, it lets an application 126 00:04:44,200 --> 00:04:47,040 use an open AI style way of making a request, 127 00:04:47,040 --> 00:04:48,920 while the gateway can connect that request 128 00:04:48,920 --> 00:04:51,280 to different model providers behind the scenes. 129 00:04:51,280 --> 00:04:54,320 So your app sends one familiar kind of request, 130 00:04:54,320 --> 00:04:57,000 and the gateway handles all the provider details. 131 00:04:57,000 --> 00:04:57,960 Depending on your setup, 132 00:04:57,960 --> 00:05:00,440 that request could go to an Azure OpenAI deployment, 133 00:05:00,440 --> 00:05:02,160 a model in Microsoft Foundry, 134 00:05:02,160 --> 00:05:05,040 and Thropic, Google Gemini, or Amazon Bedrock. 135 00:05:05,040 --> 00:05:06,360 The root follows rules you set, 136 00:05:06,360 --> 00:05:09,080 not a hard coded choice buried inside every application. 137 00:05:09,080 --> 00:05:11,480 That gives the platform team room to make changes 138 00:05:11,480 --> 00:05:13,720 without disturbing the people who use the app. 139 00:05:13,720 --> 00:05:16,760 Say your company runs an AI assistant for customer support. 140 00:05:16,760 --> 00:05:18,280 Most questions are simple. 141 00:05:18,280 --> 00:05:19,560 Where is my order? 142 00:05:19,560 --> 00:05:21,240 How do I reset my password? 143 00:05:21,240 --> 00:05:23,920 Those can be answered by a smaller, cheaper model quickly, 144 00:05:23,920 --> 00:05:25,520 without spending more than needed. 145 00:05:25,520 --> 00:05:27,600 Then a customer sends a long, unusual question 146 00:05:27,600 --> 00:05:29,840 that needs careful reasoning or product knowledge. 147 00:05:29,840 --> 00:05:32,720 The gateway routes that type of request to a stronger model. 148 00:05:32,720 --> 00:05:34,800 The support app still sends requests the same way, 149 00:05:34,800 --> 00:05:37,640 so your team never needs to know which model handled the question, 150 00:05:37,640 --> 00:05:39,200 and your developers don't have to wire 151 00:05:39,200 --> 00:05:41,160 up separate connections for every option. 152 00:05:41,160 --> 00:05:43,760 What about capacity? 153 00:05:43,760 --> 00:05:45,480 AI model endpoints have limits. 154 00:05:45,480 --> 00:05:47,320 They can only handle a certain amount of traffic 155 00:05:47,320 --> 00:05:48,560 during a period of time. 156 00:05:48,560 --> 00:05:50,440 If every request hits one deployment, 157 00:05:50,440 --> 00:05:53,640 a single busy hour can slow down or block the whole experience. 158 00:05:53,640 --> 00:05:56,760 A gateway can spread traffic across multiple approved deployments, 159 00:05:56,760 --> 00:05:58,920 rooting to another region when one fills up 160 00:05:58,920 --> 00:06:01,440 or using a backup route if an endpoint goes down. 161 00:06:01,440 --> 00:06:03,720 That doesn't mean requests should bounce around randomly. 162 00:06:03,720 --> 00:06:04,680 You still set the rules. 163 00:06:04,680 --> 00:06:07,720 For example, customer facing traffic gets priority 164 00:06:07,720 --> 00:06:09,960 on your main model while internal testing 165 00:06:09,960 --> 00:06:13,720 uses a different path, or one region handles local users 166 00:06:13,720 --> 00:06:16,160 while another acts as a backup. 167 00:06:16,160 --> 00:06:17,320 The point is simple. 168 00:06:17,320 --> 00:06:20,160 Your applications connect to one controlled front door, 169 00:06:20,160 --> 00:06:22,760 and behind it the platform can adjust models, regions, 170 00:06:22,760 --> 00:06:24,720 or providers with minimal disruption. 171 00:06:24,720 --> 00:06:26,520 But that matters because AI moves fast. 172 00:06:26,520 --> 00:06:29,320 A model that works today might not be the best choice next month. 173 00:06:29,320 --> 00:06:31,440 If every app talks directly to its provider, 174 00:06:31,440 --> 00:06:33,720 changing models becomes a project for every team. 175 00:06:33,720 --> 00:06:37,200 With the gateway, the model decision lives in one shared place. 176 00:06:37,200 --> 00:06:39,880 A quick note on the unified model API. 177 00:06:39,880 --> 00:06:42,240 Microsoft lists it as public preview. 178 00:06:42,240 --> 00:06:43,480 Preview features can change, 179 00:06:43,480 --> 00:06:45,960 and they may not be ready for every production use case. 180 00:06:45,960 --> 00:06:48,680 Still, the larger pattern doesn't depend on that one feature. 181 00:06:48,680 --> 00:06:51,840 As your API management can already sit in front of model endpoints, 182 00:06:51,840 --> 00:06:55,200 apply routing rules, and help you manage more than one back end. 183 00:06:55,200 --> 00:06:59,960 The unified model API aims to make that cross-provider experience easier for applications. 184 00:06:59,960 --> 00:07:01,800 Choosing the right model is only half the job 185 00:07:01,800 --> 00:07:03,680 because a model can answer a question, 186 00:07:03,680 --> 00:07:05,920 but an agent can decide to take an action 187 00:07:05,920 --> 00:07:08,360 that reaches into real company systems. 188 00:07:08,360 --> 00:07:09,920 Governing agents and tools. 189 00:07:09,920 --> 00:07:13,120 An agent takes AI beyond just answering questions. 190 00:07:13,120 --> 00:07:16,240 It's software that can look at a task, choose from approved options, 191 00:07:16,240 --> 00:07:18,480 and call a tool or service to get work done. 192 00:07:18,480 --> 00:07:20,760 An HR agent might look up a leave policy, 193 00:07:20,760 --> 00:07:22,400 a support agent might check an order, 194 00:07:22,400 --> 00:07:24,960 and a finance agent might prepare payment request. 195 00:07:24,960 --> 00:07:26,200 That ability is useful, 196 00:07:26,200 --> 00:07:28,040 but it also changes the risk profile. 197 00:07:28,040 --> 00:07:30,160 A wrong answer from a model confuses someone, 198 00:07:30,160 --> 00:07:32,800 but an agent using the wrong tool can expose data, 199 00:07:32,800 --> 00:07:36,080 change a record, or trigger a process it should never touch. 200 00:07:36,080 --> 00:07:38,160 This is why tools need rules of their own. 201 00:07:38,160 --> 00:07:41,200 You'll hear the term MCP or model context protocol. 202 00:07:41,200 --> 00:07:42,800 Think of it as a standard connector shape 203 00:07:42,800 --> 00:07:44,640 that gives an agent a known way to find 204 00:07:44,640 --> 00:07:46,880 a user company tool like a document search service 205 00:07:46,880 --> 00:07:48,440 or order lookup API. 206 00:07:48,440 --> 00:07:49,920 Rather than building a different connection 207 00:07:49,920 --> 00:07:51,600 for every agent in every system, 208 00:07:51,600 --> 00:07:53,560 MCP gives teams a common pattern. 209 00:07:53,560 --> 00:07:55,800 Then there's A to A, agent to agent, 210 00:07:55,800 --> 00:07:58,120 a standard way for one agent to contact another. 211 00:07:58,120 --> 00:08:00,400 For example, a support agent asks a shipping agent 212 00:08:00,400 --> 00:08:01,840 for delivery details instead of trying 213 00:08:01,840 --> 00:08:03,640 to understand the shipping system itself. 214 00:08:03,640 --> 00:08:05,120 That sounds tidy on paper, 215 00:08:05,120 --> 00:08:06,760 but an agent chain gets complicated 216 00:08:06,760 --> 00:08:08,880 when every connection has different access rules, 217 00:08:08,880 --> 00:08:10,920 hidden credentials, and unknown tools. 218 00:08:10,920 --> 00:08:13,560 Imagine a finance agent that answers invoice questions. 219 00:08:13,560 --> 00:08:16,040 It should only use an approved invoice lookup tool, 220 00:08:16,040 --> 00:08:18,480 but someone gives it broad access to the finance system 221 00:08:18,480 --> 00:08:21,040 because it's faster than defining a narrow tool. 222 00:08:21,040 --> 00:08:22,800 A user sends a harmless looking prompt, 223 00:08:22,800 --> 00:08:24,840 but hidden instructions in an attached document 224 00:08:24,840 --> 00:08:26,920 push the agent toward an unapproved action. 225 00:08:26,920 --> 00:08:28,720 If the agent has too much access, 226 00:08:28,720 --> 00:08:30,600 it might request unnecessary data 227 00:08:30,600 --> 00:08:33,040 or start a workflow that should need human approval. 228 00:08:33,040 --> 00:08:34,440 The agent isn't malicious. 229 00:08:34,440 --> 00:08:36,960 It just had too much freedom when handling untrusted input. 230 00:08:36,960 --> 00:08:39,400 Microsoft AI Gateway lets you put controls 231 00:08:39,400 --> 00:08:42,360 in front of MCP servers and business APIs. 232 00:08:42,360 --> 00:08:45,040 Instead of a direct path from an AI app to a database, 233 00:08:45,040 --> 00:08:48,120 you expose a small control tool for one specific job. 234 00:08:48,120 --> 00:08:49,560 For example, an agent asks, 235 00:08:49,560 --> 00:08:51,320 check the status of this order. 236 00:08:51,320 --> 00:08:53,000 The tool returns just the status, 237 00:08:53,000 --> 00:08:55,360 not every order or the full customer record. 238 00:08:55,360 --> 00:08:56,400 It's a narrow job, 239 00:08:56,400 --> 00:08:59,280 and the gateway checks who's asking before allowing the call. 240 00:08:59,280 --> 00:09:01,480 Identity sits at the center of the setup. 241 00:09:01,480 --> 00:09:03,640 With Microsoft Enter ID, apps sign in 242 00:09:03,640 --> 00:09:06,680 through managed identities instead of sharing provider keys. 243 00:09:06,680 --> 00:09:08,680 A managed identity proves who it is 244 00:09:08,680 --> 00:09:11,400 without a developer copying a secret into code. 245 00:09:11,400 --> 00:09:13,360 Each agent should have its own identity. 246 00:09:13,360 --> 00:09:15,360 That way you can see which agent called a tool, 247 00:09:15,360 --> 00:09:17,640 revoke access for one agent without breaking others 248 00:09:17,640 --> 00:09:18,680 and apply different rules. 249 00:09:18,680 --> 00:09:21,480 An HR agent shouldn't inherit a finance agent's permissions 250 00:09:21,480 --> 00:09:23,200 just because both use AI. 251 00:09:23,200 --> 00:09:24,520 This follows zero trust. 252 00:09:24,520 --> 00:09:25,800 Check every request, 253 00:09:25,800 --> 00:09:27,840 give only the access needed for the current job 254 00:09:27,840 --> 00:09:30,400 and assume any prompt or tool response could be unsafe. 255 00:09:30,400 --> 00:09:31,920 In practice, the agent gets permission 256 00:09:31,920 --> 00:09:33,920 for one specific tool in action. 257 00:09:33,920 --> 00:09:35,160 It can read a leave balance, 258 00:09:35,160 --> 00:09:36,840 but not download the payroll file. 259 00:09:36,840 --> 00:09:39,760 It can create a draft support ticket, but not closer case. 260 00:09:39,760 --> 00:09:41,600 The gateway also applies safety checks 261 00:09:41,600 --> 00:09:43,280 to prompts and tool traffic. 262 00:09:43,280 --> 00:09:45,360 If a request contains harmful content or data 263 00:09:45,360 --> 00:09:46,760 that shouldn't leave the system, 264 00:09:46,760 --> 00:09:48,840 a policy stops it before reaching the backend. 265 00:09:48,840 --> 00:09:50,280 No filter catches everything, 266 00:09:50,280 --> 00:09:52,560 but checking traffic at a shared point is much safer 267 00:09:52,560 --> 00:09:55,320 than hoping each agent builder adds the same protection. 268 00:09:55,320 --> 00:09:57,640 You also need a catalog of approved tools. 269 00:09:57,640 --> 00:10:01,120 API Center acts as a private catalog for APIs and MCP servers, 270 00:10:01,120 --> 00:10:02,360 so teams find the right tool 271 00:10:02,360 --> 00:10:05,400 without searching all docs or copying unverified web addresses 272 00:10:05,400 --> 00:10:08,080 that helps turn which tool can my agent use 273 00:10:08,080 --> 00:10:09,880 into a controlled business decision. 274 00:10:09,880 --> 00:10:11,440 Security controls protect the path, 275 00:10:11,440 --> 00:10:13,160 but they also create a new question, 276 00:10:13,160 --> 00:10:14,760 which teams are using that path 277 00:10:14,760 --> 00:10:17,680 and how much AI usage are they creating. 278 00:10:17,680 --> 00:10:19,800 Token costs, limits and visibility. 279 00:10:19,800 --> 00:10:22,680 AI costs work differently than a normal software license. 280 00:10:22,680 --> 00:10:24,040 The meter runs on tokens. 281 00:10:24,040 --> 00:10:24,880 What's a token? 282 00:10:24,880 --> 00:10:26,040 It's a small piece of text. 283 00:10:26,040 --> 00:10:27,600 A short word might use one token. 284 00:10:27,600 --> 00:10:29,000 A longer word uses more. 285 00:10:29,000 --> 00:10:30,840 The model counts everything you send in. 286 00:10:30,840 --> 00:10:33,000 Your question, the instructions you add, 287 00:10:33,000 --> 00:10:35,320 any documents you attach and the answer it gives back 288 00:10:35,320 --> 00:10:37,200 so a simple question uses very little. 289 00:10:37,200 --> 00:10:39,280 But an agent that reads a long policy document 290 00:10:39,280 --> 00:10:41,120 asks follow-up questions, calls a tool 291 00:10:41,120 --> 00:10:42,920 and then creates a detailed answer 292 00:10:42,920 --> 00:10:44,320 can use far more than you expect. 293 00:10:44,320 --> 00:10:46,280 You've probably seen this with a meeting transcript. 294 00:10:46,280 --> 00:10:47,960 You see a few pages of notes. 295 00:10:47,960 --> 00:10:50,400 The AI system also processes its system instructions, 296 00:10:50,400 --> 00:10:52,440 the transcript, retrieved company data, 297 00:10:52,440 --> 00:10:54,240 tool results and the final response. 298 00:10:54,240 --> 00:10:55,960 Every part adds to the usage. 299 00:10:55,960 --> 00:10:58,040 That's why one busy agent can eat up a shared model 300 00:10:58,040 --> 00:10:59,640 limits surprisingly fast. 301 00:10:59,640 --> 00:11:01,920 Imagine a team testing a new marketing assistant 302 00:11:01,920 --> 00:11:03,080 on a Friday afternoon. 303 00:11:03,080 --> 00:11:05,360 Someone gives it a large set of campaign files 304 00:11:05,360 --> 00:11:08,520 and asks it to compare, rewrite and summarize them in a loop. 305 00:11:08,520 --> 00:11:10,400 Meanwhile, customer support depends 306 00:11:10,400 --> 00:11:12,880 on the same model capacity for people trying to get help. 307 00:11:12,880 --> 00:11:15,120 If the test workload uses the shared quota, 308 00:11:15,120 --> 00:11:17,240 the support assistant may slow down or fail 309 00:11:17,240 --> 00:11:18,240 when customers need it. 310 00:11:18,240 --> 00:11:20,640 The problem isn't just that the company spent money. 311 00:11:20,640 --> 00:11:22,720 The wrong workload took capacity from the workload 312 00:11:22,720 --> 00:11:23,760 that mattered most. 313 00:11:23,760 --> 00:11:26,760 Azure API management AI gateway can apply limits 314 00:11:26,760 --> 00:11:28,080 before that happens. 315 00:11:28,080 --> 00:11:30,480 You can set request limits, token limits 316 00:11:30,480 --> 00:11:33,960 and quotas for an app, team, project or consumer. 317 00:11:33,960 --> 00:11:37,080 A quota can cover a short window, tokens per minute 318 00:11:37,080 --> 00:11:39,960 or a longer period like a daily or monthly allowance. 319 00:11:39,960 --> 00:11:41,720 That gives each workload a boundary. 320 00:11:41,720 --> 00:11:44,760 The marketing assistant can have a budget that fits its purpose. 321 00:11:44,760 --> 00:11:46,520 Customer support can have its own allowance 322 00:11:46,520 --> 00:11:48,080 and a higher priority. 323 00:11:48,080 --> 00:11:49,880 When the marketing assistant reaches its limit, 324 00:11:49,880 --> 00:11:52,560 the gateway can reject or slow its next requests. 325 00:11:52,560 --> 00:11:53,840 Customer support keeps running 326 00:11:53,840 --> 00:11:56,360 because one test project can't use everything available. 327 00:11:56,360 --> 00:11:58,840 This also changes the monthly cost conversation. 328 00:11:58,840 --> 00:12:00,560 Without a shared view, a finance team 329 00:12:00,560 --> 00:12:02,720 may receive one large AI bill and ask, 330 00:12:02,720 --> 00:12:04,240 who used all of this? 331 00:12:04,240 --> 00:12:05,600 The answer can be hard to find 332 00:12:05,600 --> 00:12:07,600 when teams connect directly to different services 333 00:12:07,600 --> 00:12:08,720 with shared keys. 334 00:12:08,720 --> 00:12:11,520 A gateway can record the caller, the model, token use, 335 00:12:11,520 --> 00:12:14,360 response status and the tool calls linked to a request. 336 00:12:14,360 --> 00:12:15,800 That gives you a clearer trail. 337 00:12:15,800 --> 00:12:19,000 You can see whether the spending came from an HR assistant, 338 00:12:19,000 --> 00:12:20,680 a developer test environment, 339 00:12:20,680 --> 00:12:22,880 a customer facing chatbot or an agent 340 00:12:22,880 --> 00:12:25,120 that kept retrying after an error. 341 00:12:25,120 --> 00:12:26,800 You can also spot unusual patterns 342 00:12:26,800 --> 00:12:28,640 like one application sending much larger prompts 343 00:12:28,640 --> 00:12:29,480 than normal. 344 00:12:29,480 --> 00:12:31,160 If your teams start in Microsoft Foundry, 345 00:12:31,160 --> 00:12:33,360 the gateway can connect to that work too. 346 00:12:33,360 --> 00:12:35,640 Foundry projects can use project-level token limits 347 00:12:35,640 --> 00:12:38,000 while the gateway applies the traffic rules around them. 348 00:12:38,000 --> 00:12:40,400 That gives AI builders a simpler starting point 349 00:12:40,400 --> 00:12:42,280 while the platform team still has a shared place 350 00:12:42,280 --> 00:12:43,960 to manage limits and monitoring. 351 00:12:43,960 --> 00:12:46,560 There's also a useful feature called semantic caching. 352 00:12:46,560 --> 00:12:49,960 Say one employee asks, where can I find the travel policy? 353 00:12:49,960 --> 00:12:51,840 A few seconds later, another asks, 354 00:12:51,840 --> 00:12:54,120 where is our policy for business trips? 355 00:12:54,120 --> 00:12:56,440 The words differ, but the meaning may be close enough 356 00:12:56,440 --> 00:12:58,880 that the gateway can reuse a suitable earlier answer 357 00:12:58,880 --> 00:13:00,720 rather than calling the model again. 358 00:13:00,720 --> 00:13:02,800 That can reduce cost and speed up the response 359 00:13:02,800 --> 00:13:04,920 so it only fits questions where reuse makes sense. 360 00:13:04,920 --> 00:13:07,560 You wouldn't cash a request for someone's current leave balance 361 00:13:07,560 --> 00:13:10,320 or an order status because those answers can change. 362 00:13:10,320 --> 00:13:12,120 Routing can help with cost as well. 363 00:13:12,120 --> 00:13:14,760 A company may send suitable work to a lower cost model, 364 00:13:14,760 --> 00:13:17,040 use capacity it already reserved before paying 365 00:13:17,040 --> 00:13:19,520 for extra usage and switch to a backup path 366 00:13:19,520 --> 00:13:20,920 when a limit appears. 367 00:13:20,920 --> 00:13:23,720 Still, a gateway doesn't remove the need for ownership. 368 00:13:23,720 --> 00:13:26,640 Teams need budgets, someone needs to review model answers, 369 00:13:26,640 --> 00:13:29,480 someone needs to decide whether an agent should exist at all. 370 00:13:29,480 --> 00:13:31,520 The gateway gives you the meter, the guardrails, 371 00:13:31,520 --> 00:13:32,880 and the record of traffic. 372 00:13:32,880 --> 00:13:34,760 It doesn't make business decisions for you. 373 00:13:34,760 --> 00:13:36,680 Once you can see and control the traffic, 374 00:13:36,680 --> 00:13:40,000 the whole setup becomes easier to picture in one everyday example. 375 00:13:40,000 --> 00:13:41,360 A simple request journey. 376 00:13:41,360 --> 00:13:43,520 Picture an HR assistant inside a company. 377 00:13:43,520 --> 00:13:45,880 An employee asks, how many days of parental leave 378 00:13:45,880 --> 00:13:48,160 can I take and can you start the request for me? 379 00:13:48,160 --> 00:13:50,880 The aim is simple, give the employee a fast useful answer 380 00:13:50,880 --> 00:13:52,760 then help with the next approved step, 381 00:13:52,760 --> 00:13:55,080 but the HR assistant must not hand out payroll data, 382 00:13:55,080 --> 00:13:58,760 open every HR file or carry a model provider key inside its code. 383 00:13:58,760 --> 00:14:00,840 The request starts when the employee signs in. 384 00:14:00,840 --> 00:14:03,400 Before the assistant sends anything to a model or a tool, 385 00:14:03,400 --> 00:14:05,600 the gateway checks who the employee is, 386 00:14:05,600 --> 00:14:07,280 whether they can use the HR assistant 387 00:14:07,280 --> 00:14:08,920 and which limits apply to that app. 388 00:14:08,920 --> 00:14:10,960 The employee sees one simple chat window 389 00:14:10,960 --> 00:14:12,920 while these checks happen behind the scenes. 390 00:14:12,920 --> 00:14:14,600 Next, the assistant needs an answer. 391 00:14:14,600 --> 00:14:18,400 A basic policy question can go to the company's approved model endpoint. 392 00:14:18,400 --> 00:14:21,480 If that endpoint is busy or reaches a set limit, 393 00:14:21,480 --> 00:14:23,240 the gateway can follow a backup route 394 00:14:23,240 --> 00:14:25,280 that the company already approved. 395 00:14:25,280 --> 00:14:28,120 The employee still gets an answer through the same HR assistant, 396 00:14:28,120 --> 00:14:31,720 rather than seeing an error because one model deployment became too busy. 397 00:14:31,720 --> 00:14:33,960 Then the employee asks to start the leave request. 398 00:14:33,960 --> 00:14:36,280 At this point, the agent needs more than a model. 399 00:14:36,280 --> 00:14:38,480 It needs a tool that can check the employee's details 400 00:14:38,480 --> 00:14:40,000 and create the right request. 401 00:14:40,000 --> 00:14:43,960 The agent asks to use an approved MCP tool for leave management. 402 00:14:43,960 --> 00:14:45,440 The gateway checks that request too. 403 00:14:45,440 --> 00:14:47,560 Can this HR agent call this tool? 404 00:14:47,560 --> 00:14:50,280 Is this employee allowed to start this type of request? 405 00:14:50,280 --> 00:14:53,000 Does the action fit the permissions assigned to the agent? 406 00:14:53,000 --> 00:14:54,920 Only then does the tool receive the call. 407 00:14:54,920 --> 00:14:58,000 The tool does not return a giant list of employee records. 408 00:14:58,000 --> 00:14:59,960 It returns only what the job needs. 409 00:14:59,960 --> 00:15:01,840 The employee's available leave balance, 410 00:15:01,840 --> 00:15:03,280 the policy that applies to them, 411 00:15:03,280 --> 00:15:06,000 and a confirmation that the request was created as a draft. 412 00:15:06,000 --> 00:15:07,400 That narrow response matters. 413 00:15:07,400 --> 00:15:09,040 The agent has enough information to help, 414 00:15:09,040 --> 00:15:12,720 but it does not get broad access just because it asked a question in plain English. 415 00:15:12,720 --> 00:15:16,360 Now, imagine the employee paced text from an outside document into the chat. 416 00:15:16,360 --> 00:15:19,120 The text contains instructions that try to trick the assistant 417 00:15:19,120 --> 00:15:22,160 into ignoring its rules or requesting confidential information. 418 00:15:22,160 --> 00:15:25,080 The gateway can apply safety checks before the prompt reaches the model 419 00:15:25,080 --> 00:15:27,000 or before a tool receives a request. 420 00:15:27,000 --> 00:15:29,240 If the request breaks a rule, it can stop there. 421 00:15:29,240 --> 00:15:33,080 The same kind of check can review a response before it reaches the employee. 422 00:15:33,080 --> 00:15:35,880 Not every problem comes from a bad prompt either. 423 00:15:35,880 --> 00:15:40,240 A model might produce a response that includes information it should not return. 424 00:15:40,240 --> 00:15:44,040 A shared control point gives the company a place to inspect that path and act on it. 425 00:15:44,040 --> 00:15:45,680 Meanwhile, the request creates a record. 426 00:15:45,680 --> 00:15:48,920 The company can link the token used to the HR assistant project. 427 00:15:48,920 --> 00:15:50,920 It can see which model handled the question, 428 00:15:50,920 --> 00:15:53,720 whether the tool called succeeded, whether a request was blocked, 429 00:15:53,720 --> 00:15:57,080 and how long the process took, that makes support in review much easier. 430 00:15:57,080 --> 00:16:00,720 If an HR team reports that employees cannot create leave requests, 431 00:16:00,720 --> 00:16:03,400 the operations team can look for failed tool calls. 432 00:16:03,400 --> 00:16:05,400 If a safety rule blocks valid questions, 433 00:16:05,400 --> 00:16:07,680 they can see the pattern and improve the policy. 434 00:16:07,680 --> 00:16:10,760 If one agent suddenly uses much more AI capacity than normal, 435 00:16:10,760 --> 00:16:12,240 that change becomes visible. 436 00:16:12,240 --> 00:16:16,840 And if the company updates a rule like requiring an extra check before certain leave requests, 437 00:16:16,840 --> 00:16:18,920 it can update the gateway policy once. 438 00:16:18,920 --> 00:16:22,160 It does not need to wait for separate code updates across every HR app, 439 00:16:22,160 --> 00:16:25,080 chatbot, and workflow that uses the same controlled path. 440 00:16:25,080 --> 00:16:28,840 The same request pattern still works when the company adds another model, 441 00:16:28,840 --> 00:16:31,160 a new region, more internal tools, 442 00:16:31,160 --> 00:16:34,400 or another agent that handles a related HR task. 443 00:16:34,400 --> 00:16:39,160 Starts more though, a gateway becomes useful when AI traffic starts to become shared traffic. 444 00:16:39,160 --> 00:16:40,200 Where to start? 445 00:16:40,200 --> 00:16:42,280 So, where do you actually begin? 446 00:16:42,280 --> 00:16:44,520 Start by picking a single real workload, 447 00:16:44,520 --> 00:16:47,080 maybe a support assistant that already calls a model, 448 00:16:47,080 --> 00:16:50,680 or an agent your team plans for HR, finance, or internal documents search. 449 00:16:50,680 --> 00:16:53,080 Don't try to tackle every model and every tool at once, 450 00:16:53,080 --> 00:16:54,440 and that's a fast track to chaos. 451 00:16:54,440 --> 00:16:57,400 Instead of giving provider keys directly to every application, 452 00:16:57,400 --> 00:17:01,560 put that workloads model endpoint behind Azure API management AI gateway, 453 00:17:01,560 --> 00:17:04,640 think of it like a single security checkpoint for your AI traffic. 454 00:17:04,640 --> 00:17:07,400 Then set the basics, name a clear owner for the workload, 455 00:17:07,400 --> 00:17:11,320 use enter-based access so only the right people or services can reach it. 456 00:17:11,320 --> 00:17:13,240 Set a token limit that fits the job. 457 00:17:13,240 --> 00:17:14,240 No more, no less. 458 00:17:14,240 --> 00:17:16,160 Turn on logging so you can see what's happening. 459 00:17:16,160 --> 00:17:18,720 At one content safety rule, your team can test and review. 460 00:17:18,720 --> 00:17:19,760 Start simple. 461 00:17:19,760 --> 00:17:22,160 Once the model path and access boundary are working, 462 00:17:22,160 --> 00:17:24,560 add approved MCP tools one at a time. 463 00:17:24,560 --> 00:17:28,160 Each tool should have a narrow job and a clear reason for the agent to use it. 464 00:17:28,160 --> 00:17:30,000 No unnecessary extras. 465 00:17:30,000 --> 00:17:34,480 From there, you grow from one controlled workload to shared models, tools, and agents. 466 00:17:34,480 --> 00:17:36,160 Without that initial control point, 467 00:17:36,160 --> 00:17:39,600 every new AI app creates another unmanaged path into company data 468 00:17:39,600 --> 00:17:41,360 and another line on the AI bill. 469 00:17:41,360 --> 00:17:45,360 For the next knowledge nugget, look at Azure API management simply explained. 470 00:17:45,360 --> 00:17:48,080 That's the foundation Microsoft AI gateway bill zone.