Create reader

POST 
/reader

Create reader

Takes in a ReaderConfig payload and loads up new python reader

Request

application/json

Body

required

access_token stringnullable
cold_cache_cpu_size stringnullable
cold_cache_cuda_size stringnullable
constrained_decoding_backend stringnullable
consumer_group stringrequired
cuda_graph_cache_capacity int32nullable
cuda_visible_devices stringnullable
device stringrequired
disable_cuda_graph int32nullable
internal_gateway_ip stringnullable
lmfe_max_consecutive_whitespaces int32nullable
log_level stringnullable
max_batch_size int64nullable
max_sequence_length int32nullable
model_name stringrequired
nvlink_unavailable int32nullable
page_cache_size stringnullable
prefill_chunk_size int64nullable
quant_type stringnullable
quantize_cache_bits int32nullable
reader_log_as_json stringnullable
ssd_cache_size integernullable
tensor_parallel int32nullable

Responses

201

Takes in a JSON payload and loads up new model and/or backend

application/json

Schema

Schema

_reader_config

object

required

access_token stringnullable

cold_cache_cpu_size stringnullable

cold_cache_cuda_size stringnullable

constrained_decoding_backend stringnullable

consumer_group stringrequired

cuda_graph_cache_capacity int32nullable

cuda_visible_devices stringnullable

device stringrequired

disable_cuda_graph int32nullable

internal_gateway_ip stringnullable

lmfe_max_consecutive_whitespaces int32nullable

log_level stringnullable

max_batch_size int64nullable

max_sequence_length int32nullable

model_name stringrequired

nvlink_unavailable int32nullable

page_cache_size stringnullable

prefill_chunk_size int64nullable

quant_type stringnullable

quantize_cache_bits int32nullable

reader_log_as_json stringnullable

ssd_cache_size integernullable

tensor_parallel int32nullable
_reader_id stringrequired

Example (from schema)

{
  "_reader_config": {
    "access_token": "string",
    "cold_cache_cpu_size": "string",
    "cold_cache_cuda_size": "string",
    "constrained_decoding_backend": "string",
    "consumer_group": "string",
    "cuda_graph_cache_capacity": 0,
    "cuda_visible_devices": "string",
    "device": "string",
    "disable_cuda_graph": 0,
    "internal_gateway_ip": "string",
    "lmfe_max_consecutive_whitespaces": 0,
    "log_level": "string",
    "max_batch_size": 0,
    "max_sequence_length": 0,
    "model_name": "string",
    "nvlink_unavailable": 0,
    "page_cache_size": "string",
    "prefill_chunk_size": 0,
    "quant_type": "string",
    "quantize_cache_bits": 0,
    "reader_log_as_json": "string",
    "ssd_cache_size": 0,
    "tensor_parallel": 0
  },
  "_reader_id": "string"
}

422

Malformed request body

Loading...