Azure AI Gateway Tier: Preview Setup, Policies, MCP, and Token Monitoring

Evaluation scope and product identity

This article translates and consolidates an August 20, 2026 technical validation report. The source describes an actual East US 2 deployment and runtime checks using curl, the OpenAI SDK, and MCP JSON-RPC. East US 2 and Sweden Central were the documented preview regions. All timings, counts, status codes, pricing availability, and feature limitations below describe that evaluation, not new tests or promises about the current service. No cloud resources or inference requests were executed while preparing this public edition.

The dedicated AI Gateway tier is an ARM resource of type Microsoft.ApiManagement/service with SKU AIGateway, capacity 1. It is not Microsoft.ApiManagement/gateways with SKU WorkspaceGatewayPremium, which identifies an APIM workspace gateway. Creation and configuration use the standalone portal https://ai.gateway.azure.com/ai-gateways. The created resource can be inspected through ARM:

az resource show \
  --ids "/subscriptions/YOUR_SUBSCRIPTION_ID/resourceGroups/YOUR_RESOURCE_GROUP/providers/Microsoft.ApiManagement/service/YOUR_GATEWAY_NAME" \
  --query "{sku:sku.name, capacity:sku.capacity, url:properties.gatewayUrl, identity:identity.principalId}" -o json
# Expected resource shape: SKU AIGateway, capacity 1, gateway URL, principal ID.

Original portal screenshots are deliberately omitted because they may contain nonpublic resource names and identifiers. The procedures and observed results are reproduced as text. All YOUR_* values identify required replacements, not existing resources.

Dedicated preview tier versus conventional APIM

Conventional APIM adds AI gateway capabilities to a general-purpose API management service: register backends, import or define APIs, and author XML policies such as llm-token-limit and llm-semantic-cache. The dedicated preview tier treats models, MCP servers, token controls, and a self-service catalog as native resources and uses policy cards instead of XML.

The source found no documented requirement to run az feature register. It recommended rechecking IaC support, quota limits, pricing, regions, and feature availability at GA rather than treating portal success as a deployment contract.

Architecture and access boundaries

1. The gateway authenticates the runtime access key from the api-key header.
2. It evaluates policies attached to the requested model or tool asset.
3. It selects the backend using the exact model registration name or tool routing and authenticates upstream using configured credentials.
4. It returns the response and emits supported telemetry.

OpenAI-compatible models share https://YOUR_GATEWAY_HOST/default/models/openai/v1. Anthropic uses the protocol-specific /default/models/anthropic/v1/messages route. MCP servers use /default/toolservers/YOUR_MCP_SERVER_NAME/mcp. The default segment is the workspace discriminator. Assign unique model registration names and send that exact name in the request, not an assumed underlying model ID. Provider-specific features and parameter compatibility still need testing.

Administrators authenticate to the portal with Entra ID. Applications use long-lived runtime access keys created in Keys or Get access keys. These keys have a name and owner and remain valid until explicitly rotated or revoked. A key covers every model and tool in its gateway; separate keys provide attribution, not per-model authorization isolation. Use one key per application and environment, store it in a secret manager or inject it through the environment, and separate gateways when a stronger asset boundary is required. Backend provider keys remain at the gateway rather than in every application.

Prepare Foundry, create the gateway, and import a model

The source used gpt-5-mini, version 2025-08-07, with GlobalStandard capacity 1 to limit evaluation cost. A Foundry account must already contain a deployment before import: selecting an empty account produces no importable models. The account command uses --kind AIServices, SKU S0, and an explicit custom domain.

az cognitiveservices account create \
  --subscription YOUR_SUBSCRIPTION_ID \
  -n YOUR_FOUNDRY_ACCOUNT -g YOUR_RESOURCE_GROUP -l eastus2 \
  --kind AIServices --sku S0 --custom-domain YOUR_FOUNDRY_ACCOUNT --yes
az cognitiveservices account deployment create \
  --subscription YOUR_SUBSCRIPTION_ID \
  -g YOUR_RESOURCE_GROUP -n YOUR_FOUNDRY_ACCOUNT \
  --deployment-name YOUR_DEPLOYMENT_NAME \
  --model-name gpt-5-mini --model-version 2025-08-07 \
  --model-format OpenAI --sku-name GlobalStandard --sku-capacity 1

These are provisioning examples, not commands that were executed for publication. Confirm quota, model availability, permissions, cost, and the intended resource group before use. The source encountered ServiceModelDeprecating when attempting a new gpt-4o-mini deployment; that is an environment/date-specific observation, not a blanket statement about every region or existing deployment.

1. Sign in to the dedicated portal and open Select a gateway. Filter by subscription and resource group, search by name, and use Refresh when necessary.
2. Select Create AI Gateway. Enter YOUR_GATEWAY_NAME; the UI checks availability and derives its runtime URL. Choose an available preview region. Expand Advanced to select or create a resource group.
3. Confirm the system-assigned managed identity. Creation did not ask for capacity or scale units and completed within one minute in the source test.
4. Select the gateway to establish the Models, MCP servers, Policies, Monitoring, and Networking context. Return to the selector when switching gateways.
5. Open Models → Import from Foundry. In Select Resource, choose the subscription and Foundry account, then the discovered deployments.
6. In Provider Details, enter YOUR_PROVIDER_NAME and a display name, then select managed identity or provider-key authentication.
7. Check the role-assignment outcome. The portal attempted to grant Foundry User on the selected Foundry account to the gateway's managed identity. The administrator needs permission to create that role assignment; a provider entry can exist even when inadequate RBAC causes subsequent inference failures.
8. Return to Home or Models and record YOUR_GATEWAY_MODEL_NAME and the endpoint for application configuration.

Import from Foundry means importing the account into the dedicated gateway portal. It does not mean using the similarly named gateway-connection action in the Foundry blade of Azure Portal; the latter connects conventional APIM and is a different workflow.

Register and federate MCP tools

1. Open MCP servers → Add MCP server. The wizard has Source and Confirm stages.
2. Choose a source: remote MCP URL, OpenAPI specification file/URL, or built-in connector. Multiple source types can coexist in one logical server.
3. Give each backend a unique YOUR_MCP_BACKEND_NAME. Its name prefixes generated tool names, for example YOUR_MCP_BACKEND_NAME_addANewPetToTheStore. Renaming it later changes the client-visible names.
4. Configure upstream authentication independently of client runtime-key authentication: None, API Key, OAuth 2.0, or Managed identity. None was appropriate only for the source's unauthenticated demonstration API.
5. Use Add another backend to federate more sources, then confirm the logical server.
6. Review Endpoint URL, Backends, upstream authentication, Streamable HTTP protocol, and Failure mode. A setting such as Fail the request matters when one unavailable backend can fail discovery for the combined server.
7. Review Policies and Monitoring tabs and use Use to obtain connection snippets for application teams.

An upstream MCP endpoint can itself be exposed by an existing APIM service. The source used this arrangement to reuse existing API assets instead of rebuilding them. OpenAPI summaries/descriptions became tool descriptions, and request-body schemas became JSON Schema inputSchema definitions. Specification quality therefore directly affects tool discovery and invocation accuracy.

POST https://YOUR_GATEWAY_HOST/default/toolservers/YOUR_MCP_SERVER_NAME/mcp
api-key: YOUR_RUNTIME_ACCESS_KEY
Content-Type: application/json
Accept: application/json, text/event-stream

{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}

The source received a normal tools/list result. Streamable HTTP required the Accept header to include text/event-stream. Models and tools shared the same gateway host and runtime key. The catalog also offered asset connection examples, favorites, and a Ctrl+K command palette (Command+K on macOS).

Model calls and Responses-based agents

Use the standard OpenAI-compatible chat-completion path rather than a deployment path or a date-based api-version query. The source's successful response returned HTTP 200, model gpt-5-mini-2025-08-07, and content OK. Its usage was 8 prompt tokens, 75 completion tokens, 83 total tokens, with 64 reasoning tokens included within completion. A short visible answer can therefore consume substantial reasoning tokens.

POST https://YOUR_GATEWAY_HOST/default/models/openai/v1/chat/completions
api-key: YOUR_RUNTIME_ACCESS_KEY
Content-Type: application/json

{"model":"YOUR_GATEWAY_MODEL_NAME","messages":[{"role":"user","content":"Say OK"}]}

The request's model registration name and the response's actual provider model ID can differ. Log these as separate concepts. OpenAI SDK code must explicitly set default_headers={"api-key": API_KEY}; passing a key only through the SDK's api_key argument normally produces Authorization: Bearer and does not implement the documented gateway header. The source's later working Responses example used api_key="unused" as an SDK-required dummy and supplied the real key through default_headers; the downloadable script follows that pattern.

The Responses API supported conversation continuation with previous_response_id and a type=mcp tool whose server_url pointed back to the gateway's own MCP endpoint. The model service handled tool execution rather than requiring an application-side tool execution loop. Responses output contained mcp_list_tools and mcp_call entries, exposing the requested tool and its arguments. The original demonstration used require_approval=never; this automatically permits tool actions and is unsuitable as an unnoticed default for write-capable tools. The companion script requires an explicit always/never choice and handles mcp_approval_request responses interactively when always is selected. Do not place personal data or secrets in prompts, tool arguments, or diagnostic logs.

Policy configuration and counter dimensions

Add a policy through Type → Assets → Configure. Policies attach to selected assets, not automatically to the entire gateway; multiple assets can be selected and different models can receive different settings.

Token policies offered minute/hour/day periods in the reviewed documentation. The test used Per minute and Caller identity. Separate application keys align that dimension with application budgets. Caller IP instead groups clients behind the same NAT address, so it is not a reliable per-application boundary in many networks. Apply request controls to MCP servers too: a model-only token policy cannot limit repeated tool calls.

Measured token limiting: post-response accounting and debt

The test attached a 100-token/minute Caller identity limit and sent a request allowing 10,000 completion tokens. The first request succeeded with 12 prompt tokens plus 1,345 completion tokens, including 576 reasoning tokens: 1,357 total. The next request returned HTTP 429 with Retry-After: 747 and {"statusCode":429,"message":"Token limit is exceeded. Try again in 747 seconds."}.

This demonstrates post-response accounting in that evaluation: the gateway let the expensive request finish, updated the counter from its usage, and blocked later requests. It did not predict and reject the oversized first request. The excess behaved like carried debt rather than a clean reset after 60 seconds:

Excess = 1,357 - 100 = 1,257 tokens
Recovery estimate = 1,257 / 100 TPM = 12.57 minutes = 754 seconds
Observed Retry-After = 747 seconds, approximately consistent with time between calls.

Subsequent Retry-After values fell from 708 to 687 over roughly 20 seconds. Even a five-token request was rejected while debt remained, and blocking lasted more than 12 minutes. A very low limit can therefore create a long service interruption after one overrun. Size budgets using average real request tokens × target RPM plus a buffer, include reasoning consumption, constrain client output with max_completion_tokens, and measure recovery under controlled load. This policy is not a strict per-request spending cap.

Use body and header evidence together when classifying errors; the observed header distinction is a diagnostic heuristic, not a universal contract for every provider or future gateway version. Backend throttling and gateway debt can require materially different backoff behavior.

Measured request limiting: admission control and quick recovery

The comparison policy permitted five calls per 60 seconds and targeted both a model and an MCP server. Its UI offered 30 seconds, 60 seconds, two minutes, and five minutes. Seven sequential small model requests produced five HTTP 200 responses, a sixth response of HTTP 429 with Retry-After: 1, and then HTTP 200. The rejected response said Rate limit is exceeded. Try again in 1 seconds. and did not include the backend's served-model header.

Unlike token accounting, request count can be checked before forwarding. The sixth call was denied without creating excess backend consumption, and the source interpreted the quick seventh-call recovery as rolling capacity renewal. The one-second observation is not a guaranteed recovery time for every request. Use request limiting for abuse and agent loops, and token limiting for consumption control; neither alone prevents both high-frequency traffic and a single very expensive completion.

Content safety: gateway rejection is not a model refusal

The observed portal offered only the four category presets and did not ask for a Content Safety resource. Role assignments before and after policy creation still showed only the existing Foundry role, with no added Cognitive Services User role; resource inventory showed no newly created Content Safety account. The source interpreted this as a fully managed gateway capability. This differed from reviewed documentation saying a customer-selected Content Safety backend was required. Prompt Shields, blocklists, log-only mode, and four/eight-level severity controls were not visible in that UI. Recheck current behavior if any is a requirement.

A harmful-content-generation request reached the model, which refused in an HTTP 200 answer. The source recorded 1,165 total tokens, including 832 reasoning tokens, while its prompt-filter violence classification was safe. A model refusing a request does not establish that the gateway policy rejected it; classification of text and inference about intent are different concerns. Model refusal occurs after backend processing and may consume tokens.

A separate test containing content classified as harmful was rejected before the backend with HTTP 403 and {"statusCode":403,"message":"Request failed content safety check."}. The served-model, responsible-AI, rate-limit, and APIM-request headers observed on backend responses were absent. The source recorded no model-token consumption for this gateway-blocked request; that does not establish that every policy feature is free of service charges. Reviewed documentation listed 400, while the measured gateway response was 403.

IP filtering and private networking

The IP filter UI accepts allow/deny actions and a comma-separated list of CIDRs. A single IPv4 host should use YOUR_CLIENT_IPV4/32; do not substitute a broad prefix accidentally. A /10 spans more than four million addresses. Denying a range containing the test caller caused HTTP 403 with {"statusCode":403,"message":"Forbidden"} even though the runtime key was valid. This proves an additional network-based authorization boundary in that test; it does not prove the internal ordering of IP filtering relative to key authentication.

The same 403 status is used by Content safety, while both limit policies use 429. Record the safe error message, not status alone. Include monitoring/administration source ranges in allow-list planning, understand NAT aggregation, and prepare rollback. Blocking your runtime IP does not itself prevent policy repair through the Entra-authenticated management portal.

Networking controls are separate from policy cards. Inbound defaults were Public network access: Enable with no private endpoints; Disable permits access only through private endpoints. Outbound defaults were Public routing, with Private routing available for private backends. Locking inbound access and enabling private backend access are independent changes. It is possible to allow public clients while reaching a private Foundry backend. This is not the same architecture as placing a conventional injected gateway inside a subnet. Review subnet delegation/size, DNS, routing, and the current private-networking documentation before deployment. The source recommended baseline Content safety/IP controls on all relevant assets and rate-limit overrides on expensive models or constrained tools.

Configure telemetry and understand where metrics live

1. Open Monitoring → Configure telemetry.
2. Select New Application Insights, Existing Application Insights, or a custom OTLP-compatible destination.
3. New automatically creates an Insights resource in the gateway's subscription, resource group, and region. These fields were read-only. Create the desired Insights resource separately and choose Existing when different placement is required.
4. Select a system-assigned or user-assigned identity. Verify that Monitoring Metrics Publisher was granted.
5. Inspect the actual scope: in this dedicated tier's test it was a managed Data Collection Rule, not the Insights resource. Check DCR RBAC when troubleshooting ingestion.
6. Inspect the managed Azure Monitor Workspace automatically created in a managed resource group as well as the visible Insights resource.

Although the product architecture uses OpenTelemetry terminology, the source observed only token usage metrics, not request logs, traces, or MCP call metrics. The portal queried a managed Prometheus endpoint with /api/v1/query_range; the metric name was azure.ai_gateway.client.token.usage, an Azure-specific namespace. Query it with PromQL, not KQL:

increase({__name__="azure.ai_gateway.client.token.usage",
          microsoft.appresourceid="/subscriptions/YOUR_SUBSCRIPTION_ID/resourceGroups/YOUR_RESOURCE_GROUP/providers/microsoft.insights/components/YOUR_APPLICATION_INSIGHTS_NAME",
          azure.ai_gateway.workspace="default"}[7200s])

Immediately after configuration, the portal reported HTTP 400 with Monitoring account=YOUR_MONITORING_ACCOUNT_ID not found. In the source this was provisioning propagation, and waiting a few minutes then selecting Retry resolved it; avoid automatically rebuilding the configuration on that symptom. Persistent errors still need RBAC, workspace, and endpoint investigation.

The dashboard offered Normalized tokens and Provider-native tokens, model filters, 1H/24H/7D/30D periods, and Open in Azure Portal. An empty dashboard instructed the user to invoke a model and refresh after ingestion. New model usage appeared within seconds during the later measurement; this is not a guaranteed ingestion latency. Existing Insights keeps data in the selected Azure subscription, while a custom OTLP destination follows its own configuration and provider terms.

Measured token reporting and accounting caveats

After calling two models through two separate runtime keys, the 24-hour Provider-native view showed the following source values. Resource/model-registration/key names have been replaced with placeholders; public underlying model types are retained.

These are displayed source observations, including rounding and fractional query results, not integer per-request ledger entries. Do not force rounded dashboard categories to reconcile or double-add detail series. Reasoning is a subset of output; cached input is a subset of input. Provider-native shows raw gen_ai.token.type fields including total_tokens, whereas Normalized aligns provider input/output field conventions. A normalized display assists cross-provider comparison but does not establish identical tokenization or prices.

Both backends were Foundry, yet the provider heuristic classified one alias as Other and the other as OpenAI. The source attributed this to registration-name and usage-field inference. Use model-ID-like registration names when practical and validate provider grouping rather than trusting inference unconditionally.

The Token usage by API key panel showed YOUR_APP_1_KEY_NAME at 13.6K and YOUR_APP_2_KEY_NAME at 5.6K. Each key called a different model, so key totals matched model totals in that experiment. This corrected an earlier source assumption that application-level reporting was unavailable. The dimension is the key's name, not its secret value. A model-valued x-ratelimit-key response header does not negate key-based dashboard attribution.

Application budgets, limitations, and rollout decisions

1. Issue separate runtime keys per application/environment with a deliberate naming convention such as YOUR_APPLICATION-YOUR_ENVIRONMENT-key.
2. Choose Caller identity for token-limit counters so independent callers do not share a NAT-based budget.
3. Use supported response headers to pace requests; do not treat the model-valued header as the application identity.
4. Correlate requests using apim-request-id or an application correlation value without embedding prompts, personal data, or credentials.
5. Use provider invoices or Azure Cost Management for financial reconciliation. The source documentation called token metrics consumption estimates, not financial-reporting data.

Key/model dimensions are convenient for standard application showback, but team/department/feature dimensions require other instrumentation or conventional APIM policies such as an explicit counter-key expression. Missing usage on streaming or passthrough responses means unknown, not zero. Key-rotation continuity was not verified. Per-model key scoping was unavailable in the report, and client-side tool auditing remained necessary because MCP calls were not represented by token telemetry.

The source recommended conventional APIM for production SLA requirements, Korean/Asian deployment requirements, reproducible IaC pipelines, integration with existing API estates, semantic caching, load balancing/circuit breakers, custom dimensions, and predictable published pricing. It favored the dedicated tier for noncritical multi-provider experiments, quick MCP federation, low-configuration proofs of concept, catalog self-service, and built-in per-key token reporting. Neither round-trip latency nor data-residency requirements can be assumed satisfied by a preview region outside the required geography.

Before advancing beyond a controlled pilot, recheck: SLA and regional availability; pricing and Cost Management attribution; limits on models, tools, keys, and throughput; template-based creation; semantic cache/pool/circuit-breaker support; key rotation continuity; key dimensions in custom OTLP exports; request logs and trace export; MCP traffic reporting; and finer-grained key scopes. Start with noncritical workloads, retain a rollback route, and rerun measured policy tests against the actual target version.

Companion templates and reference documentation

The downloads are sanitized, source-derived templates that require your own values. gateway-requests.http contains chat, MCP discovery, and explicitly labeled policy-test requests. responses-agent.py requires Python and the openai package plus AI_GATEWAY_HOST, AI_GATEWAY_API_KEY, AI_GATEWAY_MODEL_NAME, AI_GATEWAY_MCP_SERVER_NAME, AI_GATEWAY_MCP_SERVER_LABEL, and an explicit AI_GATEWAY_MCP_APPROVAL value. Use always unless automatic tool approval is deliberately justified. resource-commands.sh is a billable provisioning reference and requires all named environment variables; it is not an automatic end-to-end gateway deployment. token-usage.promql contains the source query with parameterized resource identifiers. None has been cloud-tested for this public edition.

Official references: https://learn.microsoft.com/en-us/azure/api-management/ai-gateway-overview; https://learn.microsoft.com/en-us/azure/api-management/quickstart-ai-gateway-create; https://learn.microsoft.com/en-us/azure/api-management/ai-gateway-manage-models-tools; https://learn.microsoft.com/en-us/azure/api-management/ai-gateway-govern-secure-assets; https://learn.microsoft.com/en-us/azure/api-management/ai-gateway-configure-private-networking; https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities; https://learn.microsoft.com/en-us/azure/api-management/llm-token-limit-policy; https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching; https://learn.microsoft.com/en-us/azure/api-management/backends; https://learn.microsoft.com/en-us/cli/azure/apim?view=azure-cli-latest; https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/.

Resources

Source-derived, parameterized resources. Review every placeholder, permission, dependency, and deployment effect before use. These files are not a one-click deployment.

• gateway-requests.http

  Sanitized source-derived chat, MCP discovery, and manually controlled policy-test requests.

• responses-agent.py

  Environment-configured Responses/MCP agent template with explicit tool-approval selection.

• resource-commands.sh

  Source-derived Foundry provisioning and gateway inspection commands; requires values and separate execution approval.

• token-usage.promql

  Parameterized source-derived PromQL token-usage query.

Editorial notes

This page consolidates the following source documents into an English technical guide:

• AI Gateway Tier Technical Validation Report — August 20, 2026

• All 22 original portal screenshots were omitted: creation/selection, Foundry import, Home, MCP sources/overview/list, networking, four-policy setup, telemetry destination/authentication, and monitoring states/charts. Text preserves their procedures and measurements without image identifiers or metadata.
• Repeated summary tables, duplicated call examples, and graphical architecture diagrams were consolidated into text while retaining substantive behavior, evidence, limitations, and decision criteria.
• Original resource, account, provider, gateway-model registration, key-name, MCP-server, principal, monitoring-account, host, and path identifiers were replaced by explicit parameters. Raw source links and author metadata were omitted.
• The source's seven-request shell loop wrote response/header files to machine-local paths; the companion HTTP template instead supports deliberate manual repetition without filesystem output.
• The source's unsafe automatic MCP approval setting is not a default: the companion script requires an explicit choice and adds interactive approval handling. Its incomplete initial SDK authentication snippet is superseded by the source's later working api-key-header pattern.
• The verbatim harmful-content test prompt was omitted; the request class, refusal behavior, measured token count, filtering distinction, and rejection result are retained.

No original credentials or private repository links are included. Do not put populated configuration files or copied production outputs back into this public site.