Supported Models & Hardware
Supported Models
Takeoff supports most generation & embedding models natively supported by HuggingFace Transformers, which includes most models available on the HuggingFace Hub.
Models from the Llama-2
, Mistral
or Mixtral
families benefit from further optimisations to the base Takeoff optimisations.
Multi-GPU support is also available for models from these families, enabled by specifying the devices to use with the TAKEOFF_CUDA_VISIBLE_DEVICES
variable.
Models which are quantized using AWQ are supported, with AWQ being the recommended method with which to run large models on smaller hardware. Read more about AWQ and Quantization here.
Selecting the right model requires optimising performance under your hardware constraints. Models are often issued in different sizes, and can be quantized to different levels, each affecting the performance and memory usage. We discuss balancing these factors in more details here.
To help you avoid Out of Memory
errors, we have also created a memory calculator that will estimate the amount of memory a model will use. This can be accessed from the Takeoff inference GUI. You can also specify your hardware's specifications to determine if a specific model can be run on your configuration. See more about using the calculator here.
Supported Models Table
Here's a list of all the supported models categorized by size that are readily available on our Hugging Face Hub. These models are divided into three categories based on their parameter size: Small Models (less than 5 billion parameters), Medium Models (between 5 billion and 13 billion parameters), and Large Models (more than 13 billion parameters).
Our Hugging Face Hub hosts various pre-trained models optimized for different use cases, from natural language processing tasks like text generation, question answering, and embeddings, to advanced multi-modal models for specific domains. This allows developers, researchers, and machine learning engineers to deploy these models directly or fine-tune them according to their needs, saving time and computational resources.
We strive to keep these models updated and continuously add new versions with optimized performance and smaller memory footprints (like quantized models). If you're looking for a model that fits your project’s specific requirements, explore the comprehensive list below:
Supported Hardware
Takeoff is designed to work across as wide a range of hardware as possible, to lower the barrier to start working with LLMs. However, to maximize performance on commonly used hardware, hardware-specific optimizations are sometimes used, which means not all hardware types can support all model optimizations. The biggest difference is between Ampere (and later) generation GPUs vs pre-Ampere generation GPUs and CPUs.
Post-Ampere specific optimizations are used for most commonly used model types and for all AWQ models. This means that on CPUs and pre-Ampere GPUs, you cannot use models like Llama, Mistral, or Mixtral out of the box, or any AWQ model.
Hardware Type | Can Use Base Model? | Can Use AWQ Model? |
---|---|---|
Post-Ampere GPUs | ✔️ | ✔️ |
Pre-Ampere GPUs | ✔️ | ❌ |
CPUs | ✔️ | ❌ |
Pre-Ampere GPUs are those from the Turing and Volta generations or earlier. This includes the T4, V100, and Quadro RTX 8000 GPUs. It also includes any GPU from the 10xx and 20xx series.
Post-Ampere GPUs are those from the Ampere or Hopper generation of GPUs or later. This includes the A10, A6000, A100, H100, L4, and L40S GPUs. It also includes any GPU from the 30xx and 40xx series.