Vertex
Deploying the Takeoff server on Vertex AI
Vertex AI is Google Cloud's unified AI platform that provides purpose-built MLOps tools for data scientists and ML engineers to automate, standardize, and manage ML projects. Vertex AI can be useful for anyone looking to build and deploy machine learning models quickly and efficiently, without having to worry about the underlying infrastructure and management of the machine learning pipeline.
Takeoff makes it easy to deploy large language models. Vertex and Takeoff are perfect partners - with Takeoff it is trivial to launch a language model on a single device, and with Vertex it is easy to scale that deployment to many instances dynamically.
When running on Vertex Takeoff doesn't support streaming. Only the /generate
endpoint will work. The Vertex AI instance also cannot communicate with the Titan frontends.
Prerequisites​
To follow along with this guide, you will need these few things set up:
-
Artifact Registry should be set up so you can push the Takeoff Image to this location. Check out this guide for help setting this up.
-
Vertex AI should be set up so you can access the Model Registry, as well as create new endpoints.