Show HN: Garnet – Distributed Python on Kubernetes https://ift.tt/3a0ibzw
Show HN: Garnet – Distributed Python on Kubernetes Garnet (https://garnet.ai) is a framework to run distributed and parallel Python on Kubernetes. We make it easy to scale from laptop to cluster, without requiring any devops work. Garnet plugs into any existing Python environment (notebook, IDE, CI/CD pipelines, custom apps) using a lightweight client library, and enables users to scale popular data/ml libraries (including pandas, NumPy, scikit-learn, XGBoost, etc.) and custom code to a remote cluster for execution. Under the hood, we natively wire up Dask with Kubernetes for scheduling, which brings advantages such as dynamic resource allocation, ephemeral clusters and autoscaling, while abstracting the devops complexity for developers and data scientists. Our vision is to build the orchestration layer for distributed computing in Python. We’re starting off with Dask, but have goals of supporting other frameworks such as Ray and Modin in the future. Some of the cool use cases we’ve se