BERT

Short for Bidirectional Encoder Representations from Transformers, BERT was introduced in 2018 to pretrain language representations with an encoder-only Transformer. Its masked language modeling objective hides some tokens and uses context from both directions to reconstruct them; the original work also used a sentence-pair relationship objective. The pretrained model can then be adapted to tasks such as classification, named-entity extraction, and extractive question answering through fine-tuning.

BERT produces contextual representations for each token, so the same word may receive different vectors depending on its sentence. The original checkpoint was not designed for autoregressive generation or efficient vector retrieval. Retrieval systems commonly adapt BERT-family encoders as bi-encoders, or process query and document jointly as cross-encoders. BERT identifies an architectural family and pretraining method; embeddings, retrieval, and reranking behavior depend on the chosen variant and training procedure.