Skip to main content
Version: 0.21.x

Inference Engine Documentation

Overview​

The heart of the Takeoff Stack is a state-of-the-art inference server designed for self-hosting and deploying Large Language Models (LLMs). It combines ease of use with efficient performance, enabling quick deployment using a simple Docker command. This approach saves time and optimizes server performance through techniques like quantization and batching, leading to lower latency and higher throughput.

The server is ideal for users who need on-premise deployment for data privacy reasons or prefer control over their models. With Titan Takeoff, you can use fine-tuned and custom models in-house without relying on external APIs. This is crucial for industries where data sensitivity is a concern or where specialized model tuning is required.

Key Features​

High-Performance Inference​

  • 🚀 Proprietary inference engine backend for best-in-class speed and throughput
  • 🎚️ Seamless multi-GPU and quantization support
  • 🦀 Inference orchestrated by Rust for minimal overhead
  • 📡 Support for streamed responses for interactive applications

Flexible Deployment and Management​

  • 📦 Packaged in a single, easily deployed container for self-hosted and offline machines
  • 🖥️ Handy GUI for testing and managing models
  • 🚀 Simple deployment with a single command
  • 📊 Metrics dashboard for monitoring usage
  • 📚 Deployment guides for AWS, GCP, Kubernetes, Vertex, and Sagemaker

Advanced Model Control and Support​

  • 🎛️ Structured generation controls
  • 🧩 Sophisticated batching behavior adapted to tasks
  • 🏨 Support for hosting multiple copies of a model or multiple models from one instance
  • 🤗 Widespread support for Hugging Face models and custom models

Benefits​

Titan Takeoff reduces the time and effort needed to build and maintain model serving infrastructure, allowing developers to focus on critical aspects of their projects. It provides a straightforward, efficient way to deploy and manage Large Language Models while offering the flexibility and security of on-premise hosting.

Getting Started​

  1. Deploy the Container: Use a simple Docker command to deploy Titan Takeoff.
  2. Configure Models: Utilize the Model Management API to manage models dynamically.
  3. Monitor Performance: Use the built-in metrics dashboard to monitor usage and performance.
  4. Integrate: Leverage integrations with Langchain and Weaviate for enhanced functionality.

Support​

For support from the Titan Takeoff team and the community, contact hello@titanml.co.

For detailed deployment guides, refer to our documentation for AWS, GCP, Kubernetes, Vertex, and Sagemaker.

You're all set! Enjoy using Titan Takeoff for your model deployment needs.