Operational sustainability is as important as innovation. The choice of AI model determines the explainability and robustness of the output, operational complexity, risk, and monitoring needs. An authoritative response from the manager will clearly explain the model architecture or techniques used, what the key features/inputs were, and why these models were chosen over alternative models. Managers also need to be able to explain the practical trade-offs between performance, interpretability, and operational risk. A weak answer depends on buzzwords.
The key differences between prototype and production-grade systems are governance, measurement discipline, and continuous monitoring. Without these, backtesting can be misleading and performance can degrade without you noticing. Investors should assess whether managers use rigorous methodologies, control common pitfalls (overfitting, look-ahead bias), and can demonstrate ongoing reliability. They need to determine whether their operating model is durable with clear accountability, appropriate staffing, third-party risk management, and appropriate resiliency controls.
Vendor dependence can create concentration risks, and weak governance can lead to model, cyber, and continuity vulnerabilities. It is important to understand the components (data, models, infrastructure) provided by the vendor and what due diligence is performed on the vendor. Consider data sources and licenses. What are usage rights and auditability? Weak data governance creates operational, reputational, and regulatory risks and can lead to weak models and hidden exposures (licenses, privacy, bias).
Our research shows that 63% of administrators use off-the-shelf AI tools from vendors, and 51% use vendor tools with their own customizations. When it comes to data, 58% use vendor-provided data. More than two-thirds (69%) of respondents cited data quality and access as the main barrier to further adoption of AI.
Our research also shows that many programs are operating on lean internal resources, with most companies reporting only a small number of dedicated AI specialists within their investment teams (57% say they have 1-5 full-time employees dedicated to developing, implementing, or monitoring AI).
Assessing AI is not about how advanced the manager’s technology is. It is important that AI addresses a defined investment problem, is supported by robust governance, is validated by empirical discipline, is monitored in an operational environment, and operates under clear governance with resilient controls.
When evaluating asset managers, it is important to distinguish durable and repeatable AI capabilities from marketing narratives and unmanaged risk.
