Picking Generation Parameters
How to Select Generation Params
Choosing the right generation parameters is more of an art than a science. We have some experience with this, and so provide here two examples of sets of parameters that have proven useful to us when using generative models.
The sampling_topk, sampling_topp and sampling_temperature parameters are explained in detail here.
We broadly divide tasks into one of two categories:
- Creative tasks: things like chatbots, story writing, and general assistants. These need to sound plausibly human, and benefit from generating novel text.
- Factual tasks: things like classification, question answering, summarization, extraction, etc. Tasks where repeating from the source material is not as serious. These do not need to generating long form coherent text, benefiting more from textual understanding.
Creative Gen Params:​
- Sampling TopK: 50
- Sampling Temperature: 1.2
- Repetition Penalty: 1.2
- No Repeat N Gram Size: 3
Parameters are chosen to give the model a higher chance of generating novel text. Low temperatures and small TopK are associated with repeating phrases and words from the prompt, so are less suited to novel generation. The small repetition penalty and No-Repeat-N-Gram-Size again aim to penalize repeating words from the prompt.
Factual Gen Params​
- Sampling TopK: 10
- Sampling Temperature: 0.3
- Repetition Penalty: 1.0
- No Repeat N Gram Size: 0
Parameters are chosen as to not penalise repetitive generation. This is important, as often the answers to questions or summaries contain exact phrases and words from the source text. Setting generation parameters that are 'too creative' for factual tasks can lead to unexpected results like mispellings of answers, etc. as the model attempts to avoid repeating ngrams (including those required to spell an entity's name properly).
Generation can be made deterministic (within the bounds of GPU non-determinism) by setting sampling_topk=1.