Skip to main content

One post tagged with "control-layer"

View All Tags

Benchmarking the Doubleword Control Layer

· 14 min read
Fergus Finn
Fergus Finn
Founder & Member of Technical Staff, Doubleword

Control Layer Benchmarking​

Benchmarking is hard.

We think our Control Layer (dwctl) is the fastest AI gateway around. We believe this because it's written in Rust1, and because we thought about performance a lot while we were building it. We put it in production in our self-hosted inference stack, and we knew that it was fast because we didn't notice it.

It's so good that we are open sourcing it. And once it's out there, it can be used in lots of different places, in lots of different ways. And so, to prove that it will be fast everywhere, we have to do benchmarking2.

Footnotes​

  1. And therefore blazing fast. ↩

  2. The usual caveats about general case benchmarks apply: the only realistic benchmarks are built by you, the user, since only you know what your application looks like. Every highly technical business for whom performance is a proof point eventually releases a weary blog post talking about how performance is multifaceted and can't be captured by simple benchmarks. See here, here, here, here for interesting content. ↩