AI Gateways are a new component of the IT estate that enable enterprises to control their use of AI by applying good practices in governance, security, and operationalization. By governing and enhancing the use of Large Language Models (LLMs) and other forms of AI, AI Gateways can accelerate AI maturity and generate benefits in four categories:
1. Centralize AI
Many organizations find that the enthusiasm of early adopters of generative AI within their development groups exceeds their ability to properly manage and govern access to AI services. AI providers often offer free tiers to encourage experimentation. But this results in a plethora of separate subscriptions across an organization's developers, with no visibility into who is using AI services, what they're using them for, or how they're using them.
Introducing an AI Gateway is the first step in curbing this unbridled use of AI. The process varies by provider, but here we will use the AI Gateway available in IBM API Connect as an example.
AI Gateways appear alongside other types of API providers in the tool's API management dashboard.
Typically, you need to apply configuration such as providing the API key (subscription key) for a subscription that is centrally owned by the AI provider, information about where the AI provider endpoint is hosted (e.g. region), and other required technical information.
Creating a centralized AI Gateway in this way not only improves visibility, but also consolidates the purchasing power of the enterprise into a single subscription managed by the AI Gateway, replacing dozens or even hundreds of separate subscriptions created directly by application developers, allowing them to leverage economies of scale and enter into more favorable commercial agreements based on their total expected consumption.
They can also begin to build a catalog of permitted services that have been approved by the enterprise for use by application developers. This form of “reverse API management” provides an inward-facing catalog of approved services that developers can use (as opposed to the typical API management pattern of exposing services to external consumers).
2. Increase developer efficiency
The AI Gateway also makes developers more efficient: an internal catalog saves developers time and effort by granting access to approved services, and it shows developers that the research and sign-up process required for externally hosted services is completed in a central location.
Once developers discover AI services in the developer portal, they can use a self-service onboarding process to leverage those APIs and immediately continue building their applications.
For example, a user on the logistics team can create an application called “Delivery Scheduler”, subscribe it to the watsonx.ai product in the AI Gateway, generate application credentials (client ID) and invoke that service at runtime.
Once the AI Gateway is up and running, developers' existing AI provider accounts need to be migrated and managed through the AI Gateway. Developers need to be able to switch their applications to use the AI Gateway with minimal changes. In this case, it is useful for the AI Gateway to present the same API contract that the AI providers use.
In the example above, the application's request body adheres to the same schema structure as the watsonx.ai API specification. Existing fields such as model_id, input, and other parameters are provided, but the API endpoint is now presented through an AI Gateway (API Connect endpoint) instead of calling the AI provider's endpoint directly.
The only changes required to your application code are to replace the target endpoint and specify the security credentials issued by the AI Gateway. This simplicity makes it easy for application developers to adopt a centrally managed service.
3. Understand AI usage
Deploying an AI Gateway provides visibility and knowledge into how your enterprise is using AI models. By routing all requests through an AI Gateway, it can provide insights into the interactions between your applications and AI providers.
In addition to traditional API management style reporting like number of requests, technical response codes, etc., AI Gateways can provide analytics on AI-specific characteristics like the number of AI tokens sent or received by application requests, which is typically the primary metric providers use to charge for usage.
For example, you can graph the number of request and response tokens generated through your AI Gateway over a period of time to track the overall adoption trends of your AI Gateway service over time.
In addition to providing an aggregate view across the enterprise, the AI Gateway can report analytics at a more granular level per consumer organization (e.g., department within a company) and per individual application registered to use the AI Gateway service based on the amount of AI Tokens they power.
This allows you to identify which users or applications within your enterprise are using AI models most frequently and, if necessary, manage departmental chargebacks to those consumers.
You can also take a deeper look into the usage patterns of your applications, including visualizing which AI models are being used, which can help you understand which models are most important to your business scenario, or if some models might be more expensive than others.
4. Optimize for cost and performance
The AI Gateway also helps optimize the cost and performance of AI-powered applications.
Because many AI providers charge based on the number of tokens sent and received, an AI gateway must be able to set rate limits based on the number of tokens per time interval, going beyond the traditional REST pattern of rate limiting based on the number of requests.
This allows you to prevent cost explosion by imposing limits on AI usage for each application, for example setting smaller limits both during the development and testing phases of a project (such as providing limited access to a large number of application developers) and for production use cases (to protect against sudden spikes in traffic or denial of service attacks).
The AI Gateway also benefits your application performance by providing built-in caching of responses to AI requests.
Depending on the complexity of the request, it may take 5, 10, 15 or more seconds for the AI model to generate an answer. In scenarios where the same request is issued multiple times in quick succession (such as a series of multi-choice prompts with multiple end-users to select), the AI Gateway may be able to cache the response for a period of time. This allows for instant responses to your application and saves costs by preventing requests from being sent multiple times to the AI provider.
summary
The AI Gateway enables you to enhance and manage your use of AI services in four categories:
- Centralize AI
- Increase developer efficiency
- Understanding AI usage
- Cost and performance optimization
Want to learn more?
YOUTUBE.COM/THENEWSTACK
Technology is evolving quickly, so don't miss an episode. Subscribe to our YouTube channel to stream all our podcasts, interviews, demos, and more.
subscribe
