Investigation 6: Reproducibility & Scalability – Designing for Repeatability and Growth

ByK R Jayakumar

This is the final investigation of Technical Discovery that operates in Phase 3 (Model Design & Validation) of Managing Innovative AI Project (MIAP) framework. It is not about performing reproducibility or operating at scale; those are the concerns of Phase 4 (Deployment & Monitoring) and Phase 5 (Operationalization & Governance). This investigation is about the design decisions, architectural patterns, and governance choices made during model design and validation that determine whether reproducibility and scalability are even possible later. It is the bridge between “we have a working model” and “we have a model that can be trusted to work again and can grow.”

Why This Investigation Matters for Project Leaders

A model that cannot be reproduced cannot be trusted. A model that cannot scale cannot deliver value beyond the lab. These are not technical niceties; they are governance preconditions. If your team cannot reconstruct exactly how a model was built, you cannot audit it. If your architecture cannot scale from one model to a hundred, you cannot justify the investment.

A reproducibility failure discovered during an audit is a compliance incident. A scalability failure discovered during a product launch is a business crisis. Both are preventable through design choices made during Technical Discovery.

This investigation addresses a fundamental management question: Have we made the design decisions that will allow this system to be repeated and to grow?

What to Investigate

1. Reproducibility by Design

The reproducibility of LLM performance is fragile. Research by Yuan et al. (2025) [1] demonstrates that changing system configuration such as evaluation batch size, GPU count, and GPU version can introduce significant differences in generated responses, with up to 9% variation in accuracy under bfloat16 precision. The design question is: What must be true of our architecture for a result to be reproducible six months from now, on different hardware, by a different engineer?

The table below maps the core design decisions to the three system types. This ensures that the investigation provides actionable guidance regardless of the system being built.

These are not decisions to delegate. They determine whether you can answer, “Why did this version perform better, and can we prove it?”, the fundamental governance question for AI systems.

2. Scalability by Design

The fundamental scalability challenge is not handling more load; it is maintaining control, visibility, and governance as the number of models, users, and use cases grow. The design question is ‘What architectural patterns will allow this system to grow without chaos?’

The table below maps scalability design decisions to the three system types.

The patterns above are management controls expressed as technical architecture.

The Design Artifacts: What This Investigation Produces

At the end of this investigation, the team should have:

  1. A reproducibility specification: A document that defines what will be versioned, how lineage will be tracked, what constitutes a configuration, and how a result will be reconstructed. This specification is tailored to the system type (traditional ML, GenAI, or agentic).
  2. A scalability architecture: A design that specifies orchestration patterns, infrastructure abstraction, scaling triggers, and cost optimization strategies. This architecture is tailored to the system type.
  3. A governance boundary map: A clear definition of who can tune what, under what conditions, with what tracking. This map is tailored to the system type.
  4. A set of design decisions that constrain later phases: These are the inputs to Phase 4 (Deployment & Monitoring) and Phase 5 (Operationalization & Governance). They are not implementation plans; they are the architectural commitments that implementation must honor.

Common Pitfalls: A Manager’s Design Red Flag Brief

References

  1. Yuan, J., Li, H., Ding, X., et al. (2025). Understanding and mitigating numerical sources of nondeterminism in LLM inference. NeurIPS 2025. https://neurips.cc/virtual/2025/loc/san-diego/poster/118169
  2. Red Hat Developer. (2025, October 8). One model is not enough, too many models is hard: Technical deep dive. https://developers.redhat.com/articles/2025/10/08/one-model-not-enough-too-many-models-hard-technical-deep-dive
  3. ZenML Blog. (2025, December 20). Airflow vs Kubeflow vs ZenML: Feature-by-feature comparison for modern ML teams. https://www.zenml.io/blog/airflow-vs-kubeflow
  4. Cerebras AI. (2025, September 19). Cerebras and Docker: Speed meets safety. https://www.cerebras.ai/blog/DockerCompose
  5. Cloud Native Computing Foundation. (2025, August 27). How OCI artifacts will drive future AI use cases. https://www.cncf.io/blog/2025/08/27/how-oci-artifacts-will-drive-future-ai-use-cases/
  6. Kubernetes. (2025, May 1). Kubernetes v1.33: New features in DRA. https://kubernetes.io/blog/2025/05/01/kubernetes-v1-33-dra-updates/
  7. KitOps. (2026). KitOps: Standards-based packaging & versioning for AI/ML projects. https://pkg.go.dev/github.com/kitops-ml/kitops
  8. Zhang, Y., Li, X., Chen, J., & Wang, S. (2024). Slice-level scheduling for high-throughput and load-balanced LLM serving. arXiv:2406.13511. https://arxiv.org/abs/2406.13511
  9. Jiang, H., Zhang, G., Guo, W., Tan, Z., Vippagunta, S., Raman, S., Chatterjee, S., Lin, J., Liu, S., Schladenhauffen, M., Luo, J., & Jiang, H. (2025). Serverless GPU architecture for enterprise HR analytics: A production-scale BDaaS implementation. arXiv:2510.19689. https://arxiv.org/abs/2510.19689
  10. Hellert, T., Montenegro, J., & Sulc, A. (2025). Osprey: A scalable framework for the orchestration of agentic systems. arXiv:2508.15066. https://arxiv.org/abs/2508.15066
  11. Krishnan, A., Gupta, R., Sharma, P., & Lee, J. (2025). MAFA: A multi-agent framework for enterprise-scale annotation. arXiv:2510.14184. https://arxiv.org/abs/2510.14184
  12. 12. Forbes. (2025, December 5). AWS re:Invent 2025: How AI agents make enterprise automation scalable.https://www.forbes.com/sites/forbestechcouncil/2025/12/05/aws-reinvent-2025-how-ai-agents-make-enterprise-automation-scalable/

All trademarks, logos, and brand names mentioned are property of their respective owners. References are for identification, commentary, or education only. No endorsement or affiliation is implied.

© 2026, Jayakumar K R. All rights reserved.

This article and all its contents, whether in this form or any other form, format, or media shall not be copied, reproduced, distributed, adapted, translated, or quoted in whole or in part without the prior explicit written permission of the author. Unauthorized use, sharing, or reproduction of this material is strictly prohibited.

Similar Posts