DATA REPRESENTATION
From text to training inputs.
A compact reference to the representations between raw records and model computation. These are general concepts; they do not specify the preprocessing recipe of any model in the observatory.
Tokenization
A tokenizer turns text into vocabulary IDs. Its pipeline can normalize text, split it before tokenization, apply a learned tokenization model, and add special tokens. Keep the tokenizer configuration alongside the dataset so those IDs retain their meaning.
Packing
Packing places shorter examples into a shared sequence to reduce padding. Record where each example begins and ends. When examples must remain independent, the attention implementation must respect those boundaries; concatenating token IDs alone does not express that requirement.
Masking
Attention masks control which positions can attend to which other positions. Loss masks select the targets that contribute to the objective. Ignoring a target in the loss does not hide its input from attention. Keep causal visibility, padding, example boundaries, and target selection explicit.
Storage formats
Arrow defines columnar data in memory and its serialization. Parquet is a columnar file format for stored data. Safetensors stores tensors, such as checkpoint weights. Choose the representation for the object being stored, and keep the schema and preprocessing configuration with it. A file extension alone does not specify tokenization, packing, or masking.
Apache Arrow columnar format · Apache Parquet overview · Hugging Face Safetensors
← Home · Explore Modeling Foundations →
Source links require connectivity. This reference works offline and does not certify a model’s training pipeline.