Picking Generation Parameters
How to Select Generation Params
Choosing the right generation parameters is more of an art than a science. We have some experience with this, and so provide here two examples of sets of parameters that have proven useful to us when using generative models.
The sampling_topk, sampling_topp and sampling_temperature parameters are explained in detail here.
We broadly divide tasks into one of two categories:
- Creative tasks: things like chatbots, story writing, and general assistants. These need to sound plausibly human, and benefit from generating novel text.
- Factual tasks: things like classification, question answering, summarization, extraction, etc. Tasks where repeating from the source material is not as serious. These do not need to generating long form coherent text, benefiting more from textual understanding.
Creative Gen Params:​
- Sampling TopK: 50
- Sampling Temperature: 1.2
Parameters are chosen to give the model a higher chance of generating novel text. Low temperatures and small TopK are associated with repeating phrases and words from the prompt, so are less suited to novel generation.
Factual Gen Params​
- Sampling TopK: 5
- Sampling Temperature: 0.3
Parameters are chosen to strongly bias the most favourable answer.
Generation can be made deterministic (within the bounds of GPU non-determinism) by setting sampling_topk=1.