Kubernetes

See our full guide on deploying to Kubernetes here.

K8s onboarding

To get set up with Kubernetes, create the following manifest, and save it as a file called manifest.yaml.

In order to download the takeoff server, you’ll need to first create the contents of the secret. To do so, run the following from any machine with the docker daemon running.

If you don’t have a machine with docker available (it doesn’t have to be a production machine) - please reach out and we can create the kubernetes secret for you.

docker login -u takeoffusers

And at the prompt, enter the API token provided by your account manager.

Then, run

cat ~/.docker/config.json | base64

from your command line, to base64 encode your docker token. The resulting string of characters should be entered into the <to-create> field in the following manifest.

apiVersion: v1
kind: Secret
metadata:
  name: regcred
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <to-create>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: takeoff-pro-deployment
  labels:
    app: takeoff-pro
spec:
  replicas: 1
  selector:
    matchLabels:
      app: takeoff-pro
  template:
    metadata:
      labels:
        app: takeoff-pro
    spec:
      containers:
        - name: takeoff-pro-container
          image: tytn/takeoff-pro:0.12.0-gpu
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: TAKEOFF_MODEL_NAME
              value: TitanML/llama2-7b-chat-4bit-AWQ
            - name: TAKEOFF_DEVICE
              value: cuda
      imagePullSecrets:
        - name: regcred

Once you’ve done so, run

kubectl apply -f manifest.yaml