Introduction to large language models is a short course video from Google Cloud Tech that explains what large language models (LLMs) are, what they can be used for, and how prompting and tuning shape their answers. It is a good first watch for engineers who want the vocabulary before going deeper. This post gives you the video plus a written companion: the core ideas, a few worked numbers and why the topic matters to people who build chips.
Key takeaways
- An LLM is a large neural network, usually a transformer, trained to predict the next token of text.
- Models are first pre-trained on very large general text collections, then adapted for specific tasks with fine-tuning or prompt design.
- Prompt design (what you ask and how) often changes results more than people expect.
- Running and training LLMs is dominated by memory capacity and bandwidth, which is why they matter for semiconductor design.
What the video covers
The video comes from Google Cloud Tech and is part of Google's introductory generative AI learning material. As we recall it, the talk covers:
- A definition of large language models as a subset of deep learning.
- Typical use cases, such as text classification, question answering, summarisation and document generation.
- The idea of pre-training a general model and then tuning it for a domain.
- Prompt design and prompt engineering, and the difference between them.
- An overview of the Google Cloud tools available at the time for building on LLMs.
The product names in the video reflect Google's line-up in 2023, when it was recorded; several of those tools have since been renamed or replaced, but the concepts are unchanged.
How a large language model works
Tokens
Text is split into tokens, which are whole words or pieces of words. A common rule of thumb for English is that one token is about three quarters of a word, so 1,000 words is roughly 1,300 tokens.
The transformer
Most current LLMs use the transformer architecture introduced by Vaswani and colleagues in the 2017 paper Attention Is All You Need. Its key part, self-attention, lets each token weigh every other token in the context when building its representation. That is what lets the model track meaning across long passages.
Pre-training, fine-tuning and prompting
| Stage | What happens | Data and cost |
|---|---|---|
| Pre-training | The model learns to predict the next token over a very large text collection | Trillions of tokens, very large compute budget |
| Fine-tuning | Weights are adjusted on a smaller, task-specific or instruction data set | Thousands to millions of examples, much cheaper |
| Parameter-efficient tuning | Only a small set of extra parameters is trained, the base model stays fixed | Cheap, quick to switch between tasks |
| Prompt design | No training; the input is written to guide the model, often with examples | No training cost |
Worked numbers: why hardware matters
Illustrative numbers, for a model with 7 billion parameters.
- Memory for weights: at 16 bits (2 bytes) per parameter, 7 billion parameters need about 14 GB. Quantised to 4 bits they need about 3.5 GB, which is why quantisation lets such models run on a laptop or phone.
- Training compute: a widely used estimate is about 6 × parameters × tokens floating-point operations. Training on 2 trillion tokens gives 6 × 7×109 × 2×1012, about 8.4×1022 operations.
- Bandwidth for generation: generating each token reads all the weights once. At 14 GB per token and 1 TB/s of memory bandwidth, the upper limit is about 70 tokens per second for a single user.
The last point is the reason AI accelerators use HBM and advanced packaging: for text generation, memory bandwidth, not arithmetic, is usually the limit.
Common mistakes when starting out
- Treating output as fact. LLMs can produce fluent but wrong answers, often called hallucinations. Check anything that matters.
- Fine-tuning too early. Better prompts or retrieval of relevant documents often solve the problem more cheaply.
- Ignoring context length. A model can only see a fixed number of tokens at once; long documents may need to be split.
FAQ
Is an LLM the same as generative AI?
No. LLMs are one kind of generative AI, focused on text. Generative AI also includes image, audio and video models.
Do I need to know machine learning maths to follow the video?
No. It is an introduction aimed at a general technical audience.
How long is the video?
About 16 minutes. Google Cloud Tech published it in May 2023.
Your comments will be moderated before it appears here.