Now-a-days, in the world of enterprise, machine learning workloads have become mainstream. However, there is an abundance of choices that can be made around multi-cloud infrastructure and machine learning toolkits, making it complex to balance their costs and performance. Microservices architecture has been the preferred architecture style for a few years now and there’s been rapid growth in its adoption, never failing to provide exceptionally testable & maintainable services. To have a lot more simplified services management, deployment and to orchestrate tools, Kubernetes is recommended. Kubeflow, a known and widely adopted open source container management platform that manages machine learning stack on Kubernetes. This paper discusses the development and validation of Kubeflow components such as PyTorch, TensorFlow, & Notebook Servers. It includes PodDefault functionalities for notebooks and container builder API to build docker images using Kaniko. Using Helm, Kubeflow upgrade operation is performed to enhance the configured resources whenever required for the distributed training jobs & workloads. Hence, providing data scientists a scalable platform to run machine learning workloads without having to worry about resources, costs, time, and portability.
Development of Kubeflow Components in DevOps Framework for Scalable Machine Learning Systems
