GenAI Basics · Question 15 · Hard
What is the main pre-training objective of decoder-only (GPT-style) large language models?
- Predicting the next token from the preceding tokens, using large amounts of unlabelled text (self-supervised learning)
- Supervised classification of labelled images
- Reinforcement learning from scratch, without any text data
- Hand-written grammar rules for parsing sentences
Answer and explanation
Answer: A. Predicting the next token from the preceding tokens, using large amounts of unlabelled text (self-supervised learning)
Explanation: GPT-style models are pre-trained with self-supervised next-token prediction on large text corpora. Instruction tuning and preference training such as RLHF come afterwards.
Previous: A model has a context window of 8,000 tokens, shared by the input and the… · Next: Asking an LLM to perform a task with only an instruction and no examples in…
Practise genai basics
Try a fresh set of genai basics questions with an explanation after every answer, free and without signing up: GenAI Basics practice. Or sit a full placement mock test.
More genai basics questions
- In a transformer model, what does the self-attention mechanism do?
- Which order of steps is correct for answering a user question with a typical…
- A model has a context window of 8,000 tokens, shared by the input and the…
- Asking an LLM to perform a task with only an instruction and no examples in…
- What mainly distinguishes generative AI from traditional predictive…
- In the model family name GPT, what does GPT stand for?
- Further training an already pre-trained model on a smaller, task-specific…
- A model that can accept both an image and a text question about that image…
- How do diffusion models, widely used for text-to-image generation, generate an…
- What does top-p (nucleus) sampling do when choosing the next token?
More technical & programming practice
Pseudo-code · C Programming · Java · Python · Data Structures · Algorithms · Object-Oriented Programming · DBMS
LogicWiz Exams home · All practice topics · Placement mock tests · Daily 10 challenge