Before we dive into Investigation 2: Performance Dimensions, let us briefly recall the context established in Issue 9.
The fundamental challenge: Models that perform admirably on traditional training and validation sets can exhibit critical shortcomings during deployment. Real-world conditions rarely mirror a model’s dataset. Statistical rigor alone may not be sufficient for robust performance.
The success factor: The success stories are the ones that build systems designed to fit neatly into real workflows. “The failures are the ones that try to make generic AI into clunky processes and expect transformation to follow.” MIT Report Flags 95% GenAI Failure Rate, but critics say It oversimplifies.
The Discovery Vector Perspective: Technical Discovery operates during the Model Design & Validation phase of Managing Innovative AI Projects, asking one fundamental question: Can we build a model that meets accuracy, fairness, robustness, and scalability goals? It is recursive, iterative, and never truly ends because models drift, new architectures emerge, and validation requirements evolve.
The Six Investigations: Technical Discovery is organized into six interconnected investigations that cover the full spectrum of model design and validation. They are Exploring Architecture Choices, Defining Performance Dimensions, Applying Validation Techniques, Tracking Experiments Systematically, Optimizing Hyperparameters, and Ensuring Reproducibility & Scalability.
We now turn to Investigation 2: Key Performance Dimensions, moving beyond traditional accuracy metrics to define what truly matters for AI system’s success.
Investigation 2: Key Performance Dimensions
Traditional accuracy metrics are insufficient. Modern AI validation must span multiple, often competing, dimensions. The evaluation landscape has matured in 2025–2026, with new frameworks and metrics addressing gaps in earlier approaches.
What to investigate:
- Does the model’s predictive performance meet expectations?
Predictive Performance evaluates the core accuracy and quality of the model’s outputs. This includes traditional metrics such as accuracy, precision, recall, F1-score, and AUC-ROC. These metrics characterize how well the model fits the patterns in its data and how effectively it distinguishes between different classes or outcomes.
- Does the model treat all groups equitably?
Fairness assesses whether the model’s predictions are unbiased across different demographic groups. This includes measuring demographic parity, equalized odds, and group-wise ROC curves. Fairness evaluation ensures that the model does not systematically disadvantage any group, which is critical for compliance and ethical AI deployment.
- Does the model withstand real-world stressors?
Robustness measures the model’s performance under challenging conditions. This includes testing performance under input perturbations, adversarial prompts, and environment shifts. Robustness evaluation ensures that the model can handle the unpredictable nature of real-world deployment, where inputs may differ from training data or be deliberately manipulated.
- Can stakeholders understand why decisions were made?
Explainability evaluates the transparency of the model’s decision-making process. This includes SHAP/LIME reports, attention visualization, and contrastive explanations. Explainability is essential for building trust, debugging failures, and meeting regulatory requirements in sectors like finance, healthcare, and insurance.
- Is generated content accurate, coherent, and contextually appropriate?
GenAI Quality assesses the quality of outputs from generative models. This includes metrics such as perplexity, hallucination rates, coherence scores, output diversity, and semantic accuracy. Unlike traditional systems, generative models rarely produce a single “correct” output, requiring a combination of quantitative indicators and human evaluation.
- Do agents achieve their objectives effectively and efficiently?
Agentic Efficiency and Effectiveness evaluate the performance of autonomous agentic systems. This includes measuring task completion rate, step efficiency, error recovery, goal alignment, and collaboration metrics. Agentic evaluation operates across output evaluation (last answer quality), trace-level evaluation (reasoning and planning quality), and behavioural evaluation (tool selection and error recovery).
Frameworks for measurements
Multiple frameworks have emerged to provide a systematic approach to performance measurement across these dimensions. These frameworks operationalize responsible AI principles into measurable criteria, and they provide quantitative scores or assessments that support decision-making, compliance, and stakeholder communication. The following are the recent key frameworks:
Responsible AI Scoring and Evaluation (RAISE) proposed byNguyen & Do (2025) is a unified framework that systematically quantifies model performance across four foundational dimensions of Responsible AI namely, Explainability, Fairness, Robustness and Sustainability. It aggregates metrics into a single, interpretable Responsibility Score while reporting predictive performance separately to enable analysis of trade-offs between accuracy and responsibility.
RAISE framework for deployment evaluation is proposed by Kajala, (2026). Itshares the same acronym as above but is designed to help organizations systematically evaluate whether an AI system is ready for real-world deployment from a responsible AI perspective. It translates high-level Responsible AII principles into an actionable assessment methodology.
Responsible AI Labs (RAIL) framework emphasizes eight measurable dimensions for assessing the normative behaviour of LLMs namely, Fairness, Safety, Reliability, Transparency, Privacy, Accountability, Inclusivity, and User Impact. The framework is demonstrated on Anthropic’s “Values in the Wild” dataset.
The ARIA (AI Responsibility and Impact Assessment) framework dynamically weighs five responsibility dimensions of performance, fairness, robustness, privacy, and sustainability based on application context and stakeholder profiles. Research shows low correlations between ARIA and traditional rankings, indicating that uniform approaches misrepresent model suitability in up to 60% of scenarios.
NIST’s ARIA (Assessing Risks and Impacts of AI) takes a different approach that of moving beyond accuracy scores to capture risks revealed through model testing, red-teaming, and field trials. This three-tier structure generates a multidimensional view of system behaviour in real-world contexts. NIST’s ARIA performs the measure function described in the NIST AI Risk Management Framework.
A Word of Caution on Interpreting Frameworks for measurements
The nature of these frameworks varies as some aggregate multiple dimensions into a single holistic score (RAISE), others dynamically weigh dimensions based on application context (ARIA), and still others focus on capturing risks through testing and evaluation (NIST’s ARIA). The landscape of frameworks is evolving rapidly leading not only to new frameworks showing up but same acronym getting defined differently. They represent thoughtful and promising efforts to bring structure and measurability to responsible AI evaluation. They offer practical ways for teams to begin assessing ethical dimensions and can serve as useful starting points for internal governance, benchmarking, and model comparison.
At the same time, it is important to acknowledge that these frameworks are still evolving. While they provide helpful model‑based indicators, they do not yet fully align with the formal requirements of software metrology. As such, their outputs should be interpreted as informative signals, not definitive measurements.
Organizations may find value in adopting these frameworks as part of a broader responsible AI strategy, with the understanding that they represent early steps toward more rigorous, metrology‑compliant evaluation methods. Their continued refinement and the industry’s collective learning will help move the field closer to reliable, standardized measurement practices.
Measurement Nuances for GenAI and Agentic AI
The performance dimensions and frameworks discussed above provide a solid foundation for evaluating AI systems. However, Generative AI and Agentic AI introduce fundamentally different evaluation challenges that traditional approaches were not designed to address. Generative models produce open-ended outputs where there is rarely a single “correct” answer, making reference-based metrics inadequate. Agentic systems, by contrast, fail across entire execution paths, not just in final answers. They require evaluation of reasoning, tool selection, and error recovery. This section explores the specialized challenges and evolving approaches that teams must consider when validating GenAI and Agentic AI systems, complementing the frameworks already discussed.
For Generative AI, standard reference‑based metrics like BLEU and ROUGE have low correlation with human judgment. As a result, LLM‑as‑a‑Judge, where a powerful LLM evaluates the output of another model has emerged as a scalable measurement approach. The core measurements obtained from this technique are agreement statistics between the LLM judge and human annotators, such as:
- Cohen’s Kappa (κ) measuring agreement beyond chance. Strong models like GPT‑4 achieve κ ≈ 0.79, while weaker can fall to 0.31, a wide range that underscores the need for human calibration.
- Precision, Recall, and F1 can reach F1 = 81%, precision = 78%, and recall = 85%., in groundedness evaluation, an LLM judge
- Agreement Rate, the raw percentage of direct matches, typically ranging between 66% and 96%.
G‑Eval uses chain‑of‑thought reasoning for structured evaluation and has become the de facto standard for reference‑free quality scoring. Its primary output is the Spearman correlation (ρ) between G‑Eval’s judgments and human evaluations. For example, on summarisation tasks, G‑Eval using GPT‑4 achieves ρ = 0.514, which outperforms earlier automatic scoring methods. Additional correlation measures such as Kendall’s τ and Pearson’s r are also commonly reported to provide a more complete picture of agreement with human judgment.
Regardless of which approach is used, teams must maintain a human‑labelled calibration subset to validate the judge’s performance over time. Reference‑free metrics like RQUGE evaluate question generation quality by testing the answerability of candidate questions. They offer alternatives that correlate better with human judgment without relying on reference texts.
For Agentic AI key indicators include Task Completion Rate (percentage of tasks completed successfully), Action Success Rate (percentage of tool calls that return valid data), Autonomy Score (rate of human intervention required), and Cost per Successful Task. Research shows that most agents complete only about 30% of complex tasks autonomously, underscoring the importance of rigorous agentic evaluation.
Adopt multi-pronged approach for measurement!
No single metric or framework captures the full picture. Technical Discovery must adopt a multi-framework approach, combining traditional metrics with responsible AI frameworks, LLM-as-a-Judge techniques for generative outputs, and layered evaluation for agentic systems. The choice of dimensions and frameworks should be guided by:
- Application risk level (high‑stakes vs. low‑stakes)
- Regulatory requirements (e.g., EU AI Act, NIST, upcoming sector‑specific rules)
- Stakeholder priorities (e.g., some users care more about explainability, others about privacy)
In order to address the two persistent challenges, Saturation, where models score so high that the benchmark can no longer distinguish between good and excellent and Construct Validity whether the benchmark truly measures what it claims to measure, teams should keep the measurement toolkit current and follow evolving benchmarks.
References
- Amironesei, R., Godil, A., Greenberg, C., Greene, K., Hall, P., Jensen, T., Fiscus, J., & Schulman, N. (2025). Assessing Risks and Impacts of AI (ARIA) 0.1: Pilot Evaluation Report. (NIST AI 700-2). National Institute of Standards and Technology, Gaithersburg, MD. November 2025.
- ARIA Framework Authors. (2025). Bridging Context, Statistics, and Practice: A Multi-Dimensional Framework for Responsible LLM Evaluation and Selection. Presented at the 2025 IEEE International Conference on Omni-layer Intelligent Systems (COINS), Valencia, Spain, August 18–21, 2025. Published in IEEE Xplore, November 11, 2025.
- Bean, A. M., Kearns, R. O., Romanou, A., et al. (2025). Measuring What Matters: Construct Validity in Large Language Model Benchmarks. Presented at the 39th Conference on Neural Information Processing Systems (NeurIPS 2025), San Diego, CA.
- BigDATAwire. (2025, September 4). MIT Report Flags 95% GenAI Failure Rate, But Critics Say It Oversimplifies.
- European Commission. (2025). Guidelines on obligations for General-Purpose AI providers. (Under the EU AI Act, came into force on August 2, 2025).
- Forrester Research. (2025). Gen AI is a chaos agent: models are wrong 60% of the time. Presented at the Forrester Security and Risk Summit 2025.
- Kajala, A. (2026). Assessment of Artificial Intelligence Systems Using the Responsible Artificial Intelligence Score Evaluation (RAISE) Framework. In 2026 13th International Conference on Computing for Sustainable Global Development (INDIACom), pp. 1–6. Published April 8, 2026.
- Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv preprint arXiv:2303.16634. Presented at EMNLP 2023.
- MIT NANDA Initiative. (2025). The GenAI Divide: State of AI in Business 2025.
- Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2022). RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question. arXiv preprint arXiv:2211.01482.
- Nguyen, L. P. T., & Do, H. T. (2025). RAISE: A Unified Framework for Responsible AI Scoring and Evaluation. In C. Dima, A. Ferrando, & V. Malvone (Eds.), Lecture Notes in Computer Science (Vol. 16366 LNAI, pp. 453–460). Springer Science and Business Media Deutschland GmbH. Presented at the 26th International Conference on Principles and Practice of Multi-Agent Systems (PRIMA 2025), Modena, Italy.
- NIST. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). Released January 26, 2023.
- Verma, S., Prasun, P., Jaiswal, A., & Kumar, P. (2025). RAIL in the Wild: Operationalizing Responsible AI Evaluation Using Anthropic’s Value Dataset. arXiv preprint arXiv:2505.00204. Submitted April 30, 2025; published May 2, 2025.
© 2026, Jayakumar K R. All rights reserved.
This article and all its contents in this form or any other form, format, or media shall not be copied, reproduced, distributed, adapted, translated, or quoted in whole or in part without the prior explicit written permission of the author.
Unauthorized use, sharing, or reproduction of this material is prohibited.