Skip to main content
Version: 0.21.x

Tokenize

POST 
/tokenize/:reader_id

Tokenize

The tokenization endpoint takes in a text, and returns the tokenized version of the text: the way the text is seen by the model with the path-supplied reader_id.

The reader_id can be found as the model's key in the live_readers object, returned by the status endpoint. It can also be supplied at boot time for each model in the config: see here for more information.

Request​

Path Parameters

    reader_id stringrequired

    The ID of the reader to use for tokenization.

Body

required

    skip_special_tokens booleannullable

    text

    object

    required

    Input Text used for ease of users not to have to use the clunky PayloadText. Mapping provided below to convert InputText to PayloadText.

    oneOf

    string

Responses​

Takes in a JSON payload and returns the response all at once.

Schema

    skip_special_tokens booleannullable

    text

    object

    required

    Input Text used for ease of users not to have to use the clunky PayloadText. Mapping provided below to convert InputText to PayloadText.

    oneOf

    string

Loading...