Enterprise-ready solution accelerator for implementing a centralized AI API gateway that empowers organizations to securely leverage multiple Azure AI services with unified governance, monitoring, and cost management.
🔒 Enterprise Security & Compliance
- PII Detection & Masking - Automatic detection and redaction of sensitive data
- Entra ID Integration - JWT token validation with Zero Trust principles
- Bring Your Own Network - Deploy into existing VNets with private connectivity
🧠 Expanded AI Service Portfolio
- Azure OpenAI Realtime API - WebSocket-based real-time voice & text conversations
- Azure Document Intelligence - Advanced document processing and data extraction
- AI Model Inference - Custom models from Azure AI Foundry integration
- Azure AI Search - Vector, hybrid, and semantic search capabilities
📊 Advanced Monitoring & Operations
- Throttling Events Monitoring - Real-time 429 error tracking with alerts
- Dynamic Throttling Assignment - Intelligent load balancing for PTU models
- Enhanced Power BI Dashboards - Advanced usage analytics with cost allocation
🧩 Use Case Onboarding Automation
- APIM Product + Subscription + KV Secrets (Bicep) - Automate per-use-case onboarding to the AI Gateway; creates per-service products, subscriptions, and writes endpoint + key secrets to Key Vault. Includes a ready-to-use Financial Assistant example.
🏢 Enterprise Governance
- Centralized access control and API key management
- Managed identity integration (no master keys required)
- Multi-tenant isolation with product-based access control
- Per-use-case onboarding automation for APIM Products and Subscriptions
⚡ Intelligent Routing
- Priority-based backend selection with automatic failover
- Regional load balancing across multiple AI backend instances
- Capacity-aware routing with dynamic throttling for PTU models
💰 Cost Management
- Real-time usage tracking and charge-back allocation
- Token/Requests-level monitoring across all AI services
- Flexible json based usage data model that supports extension
- Power BI integration for self-service advanced analytics and reporting
🔐 Security & Compliance
- Private endpoint connectivity for all managed services services
- Network isolation with VNet integration
- Enterprise authentication with Entra ID
- PII detection and processing
- LLM content safety for prompt and content filtering
Deploy enterprise-ready AI governance in minutes with Azure Developer CLI (azd) or Bicep templates.
| Component | Purpose | Enterprise Features |
|---|---|---|
| 🚪 API Management | Central AI gateway with intelligent routing | Load balancing, throttling, JWT validation |
| 📊 Application Insights | Real-time monitoring & analytics | Custom dashboards, throttling alerts |
| 📨 Event Hub | Usage data streaming & processing | Real-time cost tracking, compliance logging |
| 🤖 Azure OpenAI | Multi-region AI deployments (3 regions) | GPT-models, Realtime API, fully private |
| 🛡️ Azure Content Safety | Centralized LLM protection | Prompt Shield and Content Safety protections |
| 💳 Azure Language Service | PII entity detection | Natural language based PII entity detection, anonymization |
| 🗄️ Cosmos DB | Usage analytics & cost allocation | Global distribution, automatic scaling |
| ⚡ Logic App | Event processing & data transformation | Workflow-based processing |
| 🔐 Managed Identity | Zero-credential authentication | Secure service-to-service communication |
| 🔗 Virtual Network | Private connectivity & isolation | BYOVNET support, private endpoints |
Azure Requirements:
- Azure Account with OpenAI access approved
- Subscription with
Microsoft.Authorization/roleAssignments/writepermissions - Sufficient OpenAI capacity in target regions (East US, North Central US, East US 2)
Development Tools:
- Azure Developer CLI (azd)
- Azure CLI
- VS Code (optional)
Review the main.bicep configuration, then deploy:
# Authenticate and setup environment
azd auth login
azd env new ai-hub-gateway-dev
# Deploy everything
azd up💡 Tip: Use Azure Cloud Shell to avoid local setup. If deployment fails, retry
azd up- it may be a transient error.
Once deployed, access your AI Gateway through the Azure API Management portal:
Comprehensive guides to master AI Hub Gateway implementation and operations.
| Guide | Description |
|---|---|
| Architecture Overview | Complete system design and component relationships |
| Deployment Guide | Step-by-step deployment instructions |
| Enterprise Provisioning | NEW: Branch-based deployment strategy, parameter management, and CI/CD automation |
| APIM Configuration | Advanced API Management policies and routing |
| Bring Your Own Network | Deploy into existing VNets |
| Deployment Troubleshooting | Common issues and solutions |
| Guide | Description |
|---|---|
| OpenAI Onboarding | Add new OpenAI instances and models |
| AI Search Integration | Vector search and RAG capabilities |
| AI Foundry Integration | Custom model deployment |
| End-to-end Scenario | Complete chat-with-data implementation |
| Guide | Description |
|---|---|
| PII Detection & Masking | Automated data protection |
| Entra ID Authentication | JWT validation and Zero Trust |
| Use Case Onboarding | Multi-service AI solution patterns |
| Guide | Description |
|---|---|
| Power BI Dashboard | Usage analytics and cost allocation |
| Throttling Events | Real-time 429 error monitoring |
| Dynamic Throttling | Intelligent load balancing |
| Usage Ingestion | Token tracking and billing |
| Guide | Description |
|---|---|
| Hybrid Deployment | Multi-cloud and edge scenarios |
| Use Case Onboarding (APIM Product Automation) | Automate per-use-case APIM Products, Subscriptions, and Key Vault secrets; includes “Financial Assistant” example |