In this Q&A session, Kumar University Dean Jim Starling chats with Professor Amaury Rendasse, Chair of Engineering Management and Systems Engineering and an expert in machine learning and AI.
sterling: Why should AI research begin by defining the AI persona that will create the prompt?
Rendasse: Defining a persona is a fundamental step because it constrains the response space that the model draws. When you specify a role such as “Senior Quantum Physicist” or “Eminent Mathematician,” you are implicitly setting expectations about the tone, depth, vocabulary, and assumptions that your model allows.
From a benchmarking perspective, persona definitions help ensure that the model is operating within a cognitive frame that is appropriate for the domain required for the task. This reduces drift toward a general least common denominator answer and reduces the risk of producing confident but uncalibrated outputs, which can occur when a model lacks a clear identity and purpose.
Importantly, your persona needs to be more than just a job title. Relevant disciplines, typical reasoning styles, and boundaries of expertise must be encoded so that the model can interpret the prompt in the correct context.
sterling: What are the most popular AI benchmarks?
Rendasse: The most reputable benchmarks tend to share three characteristics: That is, it is widely adopted in the research community, published with a clear scoring protocol, and designed to be difficult to manipulate. In reality, no single test captures true ability, so a full-scale assessment is triangulated across multiple benchmark families.
- General Knowledge and Reasoning: Massive Multitask Language Understanding (MMLU) is still a well-known historical baseline, but many groups now prefer more difficult successor languages such as MMLU-Pro. MMLU-Pro is explicitly designed to be more discriminative and inferential.
- Scientific Reasoning by Experts: Generalized Physics Question Answering (GPQA) is widely used to investigate high-level scientific question-and-answer tasks that aim to prevent shallow pattern matching.
- Coding and Software Engineering: While HumanEval is a classic code generation benchmark, the Software Engineering Benchmark (SWE Bench) is considered a more “realistic” test because it evaluates whether a system can solve real GitHub problems through test-based validation. More and more teams are also using LiveCodeBench to better control test set contamination and track coding performance over time.
- Multimodal Understanding: Multimodal Massive Understanding (MMMU) is a flagship multimodal benchmark focused on university-level multidisciplinary reasoning based on both text and images. It is frequently cited as a reference point for compound progress.
- Interactions and Preferences: Chatbot Arena (LMSYS) is impactful because it measures human preferences at scale through direct model comparisons and Elo-style evaluations.
- Agent and tool usage: General AI Assistant (GAIA) covers common AI assistant functionality, including tool usage and web browsing style behaviors. Especially for web navigation, WebArena and Mind2Web are widely referred to as environments and datasets for evaluating agents working on simulated or real websites. Extensive agent evaluation suites like AgentBench aim to test agent behavior across multiple interactive environments.
sterling: Can you explain why claims are being made that “math” is being solved by AI? What are the benchmarks that demonstrate this?
Rendasse: These claims are usually shorthand for very specific observations. AI has moved from “calculator-style” word problems to near-ceiling performance on standardized competitive math tasks, especially when models can reason or use tools step-by-step. Historically, GSM8K has been noted to measure multi-step elementary school math word problems. Although valuable, it works well enough on modern systems that it is no longer a strong delimiter.
New evidence behind the “math solved” narrative comes from competitive benchmarks like the American Invitational Mathematics Exam (AIME), where recent models focused on inference in OpenAI are reporting very high scores, especially when allowed to use the Python interpreter for validation and calculations. But if you define “solving math” as research-level mathematical reasoning, the primary benchmark is FrontierMath, developed by Epoch AI with expert mathematicians. It uses new and unpublished questions across contemporary mathematics disciplines and is designed to reduce contamination. In FrontierMath’s original report, state-of-the-art systems solved about less than 2% of problems, a clear indication that “math” is not solved in the research sense.
sterling: How can AI best predict investment markets and optimize investment returns?
Rendasse: AI is impacting investing in two main ways: signal discovery and decision optimization. On the signals side, modern models ingest vast streams of structured and unstructured data such as macroeconomic indicators, earnings and guidance, interest rates and inflation expectations, newsflow and alternative data (shipping activity, web traffic, satellite-derived proxies), along with real-time natural language processing sentiment from filings and media. The advantage is that there are no “magic predictions”, but rather the ability to detect weak nonlinear relationships and time-varying situations that are difficult to model with traditional linear factors.
On the optimization side, we use AI to translate these signals into actions. These are portfolio construction (balancing expected returns against risk, drawdowns and constraints), dynamic hedging and execution (minimizing slippage and market impact). In a more agent-like setting, the system performs scenario simulations. “If X happens, what is the distribution of outcomes?” — and continually update the exposure as new information arrives. This is especially true in systematic trading and algorithmic execution, where decisions are made frequently and disciplined risk management is essential.
And it’s kind of interesting to answer this because financial time series forecasting was a central topic for me and my PhD. I’ve seen both sides of this story firsthand. The true power of data-driven models and the harsh reality that markets are non-stationary, adaptive, and extremely tolerant of overfitting.
sterling: Besides benchmarks, what other measures show progress in training AI?
Rendasse: Benchmarking is useful, but it’s not the only way to track progress in the field. More broadly, people focus on predictable scaling behavior, efficiency, and generalization. I would like to add a small caveat in advance. This is a bit outside of my core research area, so rather than claiming deep expertise in each standard, I’m describing the mainstream standards that I see used across the community.
- First, scaling laws capture the empirical regularity that increasing compute, data, and model capacity improves performance in a fairly predictable manner. Even when individual benchmarks become saturated, scaling curves can show whether the improvement is due to a better training recipe, better data, or simply increased computation.
- Second, computing efficiency is just as important as raw power. That is, how much performance can you get per unit of training computing, and how much can you do per unit of inference computing. This includes latency, throughput, memory usage, and energy. Because a model that is “slightly better” but 10 times more expensive may not represent a practical advance.
- Third, the community is increasingly tracking computations during inference, giving models more time or steps to infer before committing to an answer. This is part of a move towards reasoning-oriented systems, where “thinking longer” can significantly improve accuracy on difficult tasks.
- Finally, an important signal is generalization, especially zero-shot (or minimal-shot) behavior. Can a model solve new tasks for which it has not been explicitly trained using only natural language descriptions? Strong zero-shot generalization suggests that the model is learning reusable abstractions rather than simply memorizing patterns associated with narrow datasets.
These criteria are therefore intended to measure not just “how high the score is,” but also how predictably the functionality is extended, how efficiently it is delivered, and how reliably it can be transferred to new tasks.
sterling: What is explainable AI? What recent advances have shown that progress is rapid and significant?
Rendasse: Explainable AI is a discipline and increasingly an engineering practice for making AI systems auditable. You can track what your AI system did, why it did it, what signals drove its decision, and whether it fell within policy. In other words, explainability is moving from “nice-to-have explanations” to operational accountability, especially when introducing agent systems that not only produce text but also perform actions.
This shows why progress in explainable artificial intelligence is currently progressing rapidly. We’re seeing a shift from reactive explanations to continuous monitoring, logging, and policy enforcement, the kind of infrastructure that can actually be deployed in enterprises and regulated workflows. This acceleration is also being driven by governance expectations. The National Institute of Standards and Technology (NIST) explicitly identifies transparency, explainability, and interpretability as core characteristics of trustworthiness, and the European Union’s AI law emphasizes transparency obligations for high-risk systems.
So the real progress is not just “better explanations on a slide,” but the emergence of an end-to-end traceability stack for real-world AI deployments, where decisions can be defended, fraud detected, and compliance demonstrated.
