Conversational AI application patterns represent a major evolution in how applications are experienced and built and deployed.
Humans interact with agents all the time in the everyday activities we do as humans. Most of us use a real estate agent when buying or selling a home, and rely on an insurance agent to help us navigate the world of home and car liability. Today, the agents we deal with are almost always human, but as is widely predicted, the advent of generative AI will see AI-powered agents (which are not human) “do the talking.”Leading Intelligence [that] Achieving ResultsCurrently, in many of our everyday interactions involving humans,
Admittedly, this isn't all that surprising. Automation is already used for many simple workflows today. Auction assistants that make intelligent bids on our behalf are one example of an existing automated agent. What I want to focus on in this conversation is not just how generative AI will dramatically increase the prevalence of automated agents, but how those agents will drive new application architecture design patterns and how that will impact application delivery.
Two key developments enabled by generative AI are (a) the ability to use conversational interfaces to interact with digital systems and (b) the ability to autonomously decompose high-level tasks into subtasks. This includes the ability to perform subtasks using APIs for digital workflows. The first point, conversational interface capabilities, is now well understood and accepted. More than 67% of Americans use OpenAI's ChatGPT, and ChatGPT, the most widely adopted GenAI chatbot, has approximately 35 million regular users in North America. However, the second development is less well known but is an emerging trend. Specifically, GenAI systems have given rise to frameworks such as AutoGPT and BabyAGI, which have demonstrated the ability to do high-level task planning and decomposition. We believe that the integration of these two notable developments, intelligent task planners and conversational interfaces, will give rise to a new class of applications.Conversational AI app-using AI Orchestrator Coordinates the internal activity of the application.
Learn more about conversational AI applications
The core value proposition of conversational AI applications will be to dramatically improve the user experience of the application. Today's user experience patterns, where humans identify and execute each step of a digital workflow and then take responsibility for directing and orchestrating the interactions at each step of the workflow, will be disrupted. Instead, we will have a user experience where humans specify the end state or goal: result— the goals and constraints to be met, all using a suitable conversational interface. Breaking down high-level workflows into subtasks, executing those subtasks, and directing the interactions of the detailed subtasks is what we call ” AI orchestrator.
example: Conversational AI Applications The architectural pattern used is depicted in the diagram below. We use trip planning as an example scenario. In this not-so-futuristic story, the user simply provides a statement of intent or outcome.I'm hoping to travel somewhere warm next weekend for under $1000 while eating delicious Chinese food.” of AI Orchestrator It plays a role similar to a human travel agent: (a) identify the necessary subtasks, (b) gather information from various knowledge sources, (c) collate the results, and (d) filter out options that do not satisfy the high-level constraints. If the AI orchestrator is further empowered to perform actions on behalf of its human clients, it can also perform transactions following a similar process, such as obtaining payment and travel loyalty information, making payments, making restaurant reservations, etc. This workflow is visually depicted below.
While the details of the workflow may change, the key message is that identifying subtasks and orchestrating the workflow is the responsibility of the AI orchestrator, not humans. The interface for human clients is intent-driven and conversational.want to go on a trip…Not only are humans not forced into non-intuitive, externally designed workflows, but they also don't have to specify all the “how to” level subtasks, like checking the weather, finding restaurants, and figuring out which destinations' airfare and accommodation costs meet budget constraints.
From the application consumer's perspective, this is a transformation of the user experience: complexity measured in time and human effort is significantly reduced while at the same time improving the quality of results compared to what humans typically achieve. It's important to note that this is not just a theoretical possibility: in conversations with CTOs and CIOs around the world, enterprises are already planning to deploy applications following this pattern within the next 12 months. In fact, Microsoft recently announced a conversational AI app focused on travel use cases.
When we think about how this automated workflow is executed architecturally, we can see that the conversational AI application pattern represents a major evolution not only in how we experience applications, but also in how we build and deploy them. The “core” of the architecture is AI Orchestrator The orchestrator plays the roles of planner, choreographer, organizer and judge mentioned above, but to collect the necessary data and take the required actions, the orchestrator relies on a range of support services.
These supporting services do not need to reside in Orchestrator's local environment (e.g. in the same Kubernetes cluster) – in fact, these services are often located outside of Orchestrator due to concerns about data confidentiality, regulatory compliance, partner business constraints, etc. If the supporting services do not fall into the core competency or value proposition of the company that owns the AI app, they may simply be a black box and abstracted behind a SaaS interface.
While the fundamental premise of “services are everywhere” isn't unique to the design of conversational AI apps, these types of apps accelerate existing evolutionary trends, resulting in changes that are very different from the past. Collective Portfolio If your enterprise applications span multiple public clouds and on-premise environments, Currently, each application itself It is itself a hybrid, multi-cloud deployment.
To contextualize this idea with the travel example, an application owner might keep the core and “most important” intellectual property/functionality of their application “on-premise” (e.g. LLM fine-tuned for travel subtask decomposition) and allow other application functionality related to the underlying services to be deployed in public cloud instances (e.g. databases containing customer account and personalization information). Also, in many cases other functionality that is necessary for the application but is not a core competency of the application owner (e.g. weather forecasts or payment services) may be consumed as SaaS services, but architecturally accessed as if they were “inside” the application, even if API calls are made to external APIs.
This means that what a user perceives as “one” application likely includes software components running in several different environments: a company's on-premise infrastructure, company-owned infrastructure deployed in the public cloud, and external SaaS services deployed in unknown or abstracted locations. Therefore, compared to traditional apps, you will see a more distributed architecture that spans on-premise and multiple public clouds, leverages both AI and traditional services, and uses consumption form factors such as containers, virtual machines and SaaS.
reference: Conversational AI: Improved service at lower cost
The Role of APIs in Conversational AI Apps
The second key theme for application developers is the growing importance of APIs in conversational AI patterns, which arise as a result of the up-leveling of how humans interact with applications. As interaction models move from the granularity of app-per-subtask to a higher level of intent or desired outcome granularity of abstraction, subtask-level workflows that were typically controlled and mediated by humans via a GUI are instead managed by automated AI agents that call APIs directly.
The impact of this massively distributed agent orchestration application model is also far-reaching for operations teams, both in the areas of deployment and security. First, the distributed nature of applications means that no single infrastructure provider can provide holistic observability across applications. This creates even greater complexity challenges in the areas of debuggability, performance management, and operational cost management. Operations teams will need solutions that work consistently and seamlessly across on-premise, public cloud, and SaaS environments. Another key impact resulting from applications needing to securely transport data and make API calls across these different network environments is the increased importance of Multi-Cloud Networking (MCN) solutions.
Security teams must also address multiple challenges. One overarching challenge is similar to the operational concerns of deployment teams: how to apply consistent security controls and governance policies across the multiple diverse environments used to compose a single logical application. A second emerging concern for application security practitioners is the need for greater focus on the threat surface that exists “inside” an application. Historically, an application's core logic has lived within a single environment, such as a single Kubernetes namespace or a single VPC, so compromising the “inside” of an app required sophisticated attack vectors, either attacking the supply chain or compromising internal microservices.
Going forward, the “internal” components of applications will themselves be distributed and likely use different infrastructures, and as a result, the internal attack surface will not only be more directly accessible to attackers, but will also be built using a more diverse technology infrastructure. This combination provides a larger and more accessible threat surface for attackers to exploit. The third challenge is addressing the evolution of bot protection in a future world where AI-powered agents that directly consume APIs are prevalent and, in fact, the most common legitimate clients of APIs. In that environment, the bot challenge will evolve from leveraging human-facing browsers to distinguish between “humans” and “bots,” to technologies that can distinguish between “good” and “bad” automated agents based on observed AI behavior patterns.
A final word on the impact of AI
The advent of generative AI has revolutionized many aspects of the technology sector and will continue to impact it in the future. While it is not possible to predict all these impacts at this time, one transformation that seems imminent is in the area of application experience. GenAI frees humans from the traditional interaction pattern of detailing each step of a complex workflow within the constraints of a highly structured and opinionated GUI. Instead, applications can take a more human-centric approach where outcomes and intent are specified in natural language along with constraints. Recent advances in AI are poised to provide the foundational technology we need today, and compelling improvements in user experience will create strong demand. Therefore, technologists of all kinds, including application developers, operations teams, and security teams, need to prepare for the new challenges that this new architectural pattern brings.
