Introduction


Embeddings are the foundation of modern AI applications — powering semantic search, recommendation systems, clustering, and classification. At their core, embeddings convert text, images, or other data into dense vector representations that capture semantic meaning. This guide explains what embeddings are, how they work, and how to use them effectively.


What Are Embeddings?


An embedding is a numerical representation of data — typically a list of floating-point numbers — where the position in vector space encodes semantic meaning. Similar concepts cluster together, while dissimilar concepts are far apart.


For example, the embeddings for "king" and "queen" would be closer to each other than to "bicycle." More importantly, embeddings capture analogical relationships: the vector operation `king - man + woman` produces a vector close to `queen`.


Modern embedding models produce vectors with 384 to 3072 dimensions. The trade-off is speed versus information density: smaller vectors are faster to compare but capture less nuance.


How Embeddings Are Generated


Embedding models are trained using contrastive learning. The model learns to pull semantically similar texts together in vector space while pushing dissimilar texts apart.


**The training signal is typically:**

  • **If two texts are similar** (paraphrases, translations, query-document pairs): minimize the distance between their embeddings
  • **If two texts are dissimilar**: maximize the distance

    • The most common loss function is **contrastive loss** or **InfoNCE loss**, which uses in-batch negative sampling to provide training signal at scale.


    Popular Embedding Models


    | Model | Dimensions | Best For | Size |

    |-------|-----------|----------|------|

    | text-embedding-3-small | 512-1536 | General purpose | API |

    | text-embedding-3-large | 256-3072 | High accuracy | API |

    | BAAI/bge-large-en-v1.5 | 1024 | RAG and search | 1.3 GB |

    | sentence-transformers/all-MiniLM-L6-v2 | 384 | Speed-critical apps | 80 MB |

    | intfloat/e5-mistral-7b-instruct | 4096 | Highest quality | 14 GB |


    For production RAG systems, `bge-large-en-v1.5` offers an excellent balance of quality and speed. For mobile or latency-sensitive applications, `all-MiniLM-L6-v2` is the standard choice.


    Using Embeddings in Practice


    Generating Embeddings


    
from sentence_transformers import SentenceTransformer



model = SentenceTransformer("BAAI/bge-large-en-v1.5")

documents = [

    "Embeddings represent text as dense vectors.",

    "Semantic search uses vector similarity to find relevant results.",

    "Vector databases store and index embeddings for fast retrieval."

]



embeddings = model.encode(documents, normalize_embeddings=True)

print(f"Shape: {embeddings.shape}")  # (3, 1024)

    Measuring Similarity


    Cosine similarity is the standard metric for comparing embeddings:


    
import numpy as np



def cosine_similarity(a, b):

    return np.dot(a, b)  # For normalized vectors, dot = cosine



query_embedding = model.encode("How do vectors represent meaning?", normalize_embeddings=True)

scores = [cosine_similarity(query_embedding, doc_emb) for doc_emb in embeddings]

    When using normalized embeddings, dot product and cosine similarity are identical, simplifying computation.


    Dimensionality Reduction


    For visualization, reduce embeddings to 2D or 3D using UMAP or t-SNE:


    
import umap



reducer = umap.UMAP(n_components=2, random_state=42)

embeddings_2d = reducer.fit_transform(embeddings)

    UMAP preserves more global structure than t-SNE and is significantly faster for large datasets.


    Advanced Techniques


    Multi-Task Embeddings


    Some models can produce different embeddings for different tasks by prepending task-specific prefixes. For example, `bge` models use:

  • `"Represent this sentence for searching relevant passages: "` for query encoding
  • No prefix for document encoding

    • This simple technique improves retrieval accuracy by 2-5% in production systems.


    Matryoshka Embeddings


    Matryoshka embedding models (like OpenAI's text-embedding-3 series) produce vectors where the first N dimensions form a valid, lower-quality embedding. You can truncate the vector at inference time to trade accuracy for speed and storage — using 256 dimensions for initial retrieval and 1536 for re-ranking.


    Common Pitfalls


  • **Not normalizing embeddings**: Always normalize unless you have a specific reason not to

    • 2. **Using raw cosine similarity with unnormalized vectors**: Results are dominated by vector magnitude

    3. **Ignoring domain mismatch**: Embedding models trained on general web text may perform poorly on medical or legal text

    4. **Fixed chunk sizes in RAG**: Large chunks dilute meaning; small chunks lose context


    Conclusion


    Embeddings are a versatile and powerful tool in the AI practitioner's toolkit. They enable semantic search, document clustering, recommendation systems, and are the backbone of RAG architectures. Choosing the right model, normalizing properly, and understanding the trade-offs between dimension count and performance will determine the success of your embedding-based application.