Version: 0.18.x

Rust Server

Titan Takeoff uses an optimised server built with Rust, rather than the FastAPI server used in the open source version of Titan Takeoff. In practice we saw a 10x throughput increase with the rust server.

Why Rust?

The Rust programming language is designed for speed and performance, with a focus on safety and reliability, as well as efficient use of memory. This ensures that your deployments will be able to handle a higher volume of simultaneous requests.

Load Testing Takeoff Rust Server vs FastAPI

Third Party Testing

Using the independent Techempower Framework Testing, we can see that Rust frameworks are consistently scored highest (color coded purple). Specifically, the Rust frameworks excelled at the multiple queries which is the most relevant to Takeoff throughput.

third party testing

Load Testing Results

Configuration

Model Name: opt_125m
Device: cuda (Nvidia GA106 [GeForce RTX 3060 Lite Hash Rate])
Max Batch Size: 50
Batch Timeout: 100ms
Request body:

{
  "text": "Hello World, what is your name dog?",
  "max_new_tokens": 10
}

Both load tests were run on the same machine, with the same configuration. We used Locust to run the load tests. We slowly ramped up to 40 concurrent users in intervals of 5.