0.4.0

0.4.0

Features

• Bits and bytes HF 4 bit backend
• Takeoff PRO added to Iris
• Multi GPU support
• Mistral support
• API docs for takeoff
• Redis and Python reader are spun up from rust gateway
• Rust server
• Rust server serves static files
• AWQ Backend
• Batched streaming for AWQ, python reader integrates with Rust gateway
• Integration and benchmark tests for takeoff
• Regex guided generation
• Unify logging formats between rust & python, rationalise log levels
• Change batching behaviour to fix throughput issues
• Manager for redis connections in the rust server
• Conversion entrypoint for AWQ, CT2.
• Model management API PUT /models to spawn new reader with new config
• Added bitsandbytes 4bit backend
• React + Typescript Frontend