Tokens and Parameters in LLM

- Overview

Tokens are the basic units of text or data processed by an LLM (roughly equal to 3 ÷ 4 characters or 0.75 words). Parameters are the internal variables that the model learns during training, representing its overall size and "memory" capacity. Tokens are the input/output experience; parameters dictate the model's intelligence.

1. Tokens: The Model's "Words": 

• Definition: Tokens are chunks of characters or words that the model reads and generates. For instance, the word "chatbot" might be split into two tokens: "chat" and "bot".
• Tokenization: Before an LLM can read human text, it converts the text into numbers (tokens) . Every model has a fixed "vocabulary" of these tokens.
• Cost and Limits: API usage is billed by the number of tokens processed (input + output). Furthermore, models have a "context window" limit (e.g., 128,000 tokens), which restricts how much text you can send in a single prompt.

2. Parameters: The Model's "Memory": 

• Definition: Parameters are the numerical values (weights and biases) adjusted during the training process. They act as memory slots that capture relationships between data.
• Model Size: When you hear about a "70B" or "8B" model, it means the LLM contains 70 billion or 8 billion parameters. Generally, a higher parameter count equates to a smarter model capable of handling more nuanced tasks, though it requires more computing power.
• Run-time Parameters: During generation, users can also adjust settings (sometimes referred to as parameters) like Temperature to control how creative and predictable the AI's responses are.

- How Parameters and Tokens Interact during Processing

In AI, tokens are the fundamental units of data (words or word parts) the model reads and generates, while parameters are the learned internal settings used to process them. During processing, input tokens are translated into mathematical vectors, passed through the model's parameters, and used to sequentially predict the next token. 

1. To understand how these concepts interact: 

• Tokenization: Your input text is broken down into numerical fragments called tokens (where roughly 100 tokens equal 75 English words).
• Embedding & Contextual Processing: These tokens are converted into dense vectors that represent their meaning . As these vectors pass through the model's layers, they are transformed and evaluated by the AI's parameters (also known as weights), which adjust the influence of one token on another. 
• Generation: Using these adjusted parameters, the model calculates probabilities to generate the most likely next token, repeating the cycle sequentially to build words and sentences.

[More to come ...]