ChromaDB Guide: Running Modes, Embedding Functions, Filtering, and Taking Chroma to Production

ChromaDB (Chroma) is an open-source vector database designed to make it easy to store documents with embeddings and query them by similarity. Developers can start with a few lines of Python or JavaScript, which is why it is widely used in tutorials, prototypes, and retrieval-augmented generation (RAG) experiments.

Running modes

Embedding functions

If you add documents without vectors, Chroma uses an embedding function to create them. The default is a small local model, which is convenient for experiments. In production, choose the model deliberately (see choosing an embedding model), configure it explicitly on the collection, and use the same model for documents and queries. You can also compute embeddings yourself and pass them in.

Filtering

• Metadata filters with where, for example {"team": "support"}, or operators such as $eq, $in, $gte, $and, and $or.
• Document content filters with where_document, for example {"$contains": "refund"}.

Our RAG implementation guide includes a full Chroma example with metadata filtering.

Distance metrics

Chroma supports squared Euclidean (L2), cosine, and inner product distance, configured when the collection is created. Match the metric to what your embedding model was trained for. Many text embedding models are designed for cosine similarity.

Taking Chroma to production: a checklist

• Run in client-server mode or Chroma Cloud rather than embedding the database in several application processes.
• Secure the server with authentication, TLS, and network restrictions. Never expose it publicly without protection.
• Back up the persistence directory, and test restores.
• Record the embedding model and version with each collection.
• Load-test with realistic data volume, filters, and concurrency.
• Plan for high availability. If you need replicated, distributed clusters, compare with Qdrant, Weaviate, or a managed option such as Pinecone.

Working with collections

A Chroma collection groups documents with their embeddings, metadata, and IDs. Create one collection per logical dataset, such as a product's documentation or a department's policies, and keep its embedding function and distance metric consistent. Each record needs a unique ID; use deterministic IDs built from your document ID and chunk number, so you can update or delete chunks when source documents change. Chroma supports adding records, upserting them to insert or replace by ID, updating documents or metadata, and deleting by ID or by metadata filter. Use upsert in ingestion pipelines, so re-running a job does not create duplicates.

Chunking and metadata for good results

Chroma stores and searches whatever you give it, so retrieval quality depends on how you prepare content. Split documents into chunks of a few hundred tokens, ideally along headings or paragraphs rather than arbitrary character counts, and include the document title and section heading in each chunk's text. Store useful metadata with every chunk: source URL or path, title, section, document type, language, last updated date, and any access group. This metadata lets you filter results, show citations, and remove outdated content. Our production RAG guide covers chunking strategies in more depth.

Querying and interpreting results

A query returns the nearest records along with their distances, documents, and metadata. Lower distance means more similar for distance metrics such as L2 and cosine distance. Request more results than you need, for example ten, and rerank or filter them before passing the best few to a language model. Combine where metadata filters with where_document content filters to narrow results, for example to a product and a keyword. Inspect distances on a sample of real queries to understand what a "good" match looks like for your data, and consider a threshold below which results are treated as not relevant, so your application can say it found nothing rather than returning weak matches.

Using Chroma with LangChain and LlamaIndex

Chroma integrates with popular frameworks such as LangChain and LlamaIndex, which provide document loaders, text splitters, embedding wrappers, and retrievers that use Chroma as the vector store. These integrations let you build a working RAG prototype quickly. As the project matures, make sure you understand and control the settings the framework applies by default, such as chunk sizes, the embedding model, the number of results, and how metadata is stored, because defaults that work for a demo may not suit production data. Pin framework and Chroma versions, since APIs evolve across releases.

Running Chroma as a server

For anything beyond a single local process, run Chroma in client-server mode, typically as a Docker container, and connect from your applications with the HTTP client. This allows several services to share one database and separates the database lifecycle from your application deployments. Mount persistent storage for the data directory, configure resource limits, enable authentication, put the server on a private network or behind a reverse proxy with TLS, and set up health checks and monitoring. Back up the data directory or use Chroma's recommended backup approach, and rehearse restoring it. If you prefer not to operate the server, Chroma Cloud offers a managed option.

Performance and scaling considerations

Chroma performs well for collections of modest size and typical query volumes, which covers many internal tools, prototypes, and departmental assistants. As collections grow into many millions of vectors, or query volume rises sharply, measure latency, memory usage, and ingestion speed carefully on your own hardware. Plan capacity for the HNSW index in memory, and avoid running multiple application processes that open the same persistent database directly, which client-server mode prevents. If your requirements include high availability with automatic failover, horizontal scaling across many nodes, or advanced multi-tenancy, compare Chroma with distributed databases before committing.

Evaluating retrieval quality

Before relying on Chroma in an application, build a small evaluation set: 50 to 200 real questions, each with the chunks or documents that should be returned. Measure how often the right result appears in the top three, five, and ten results, and how high it ranks. Use the evaluation set to compare embedding models, chunk sizes, metadata filters, and distance metrics. Rerun it whenever you change any of these settings or re-index content. This small investment turns tuning from guesswork into measurement, and it remains valuable if you later move to a different vector database, because the same questions can be used to compare systems.

Example: an internal IT help assistant

An IT team wants staff to get instant answers about VPN setup, device policies, and software requests. A developer loads around 600 help articles into a Chroma collection using a persistent client, splitting each article by heading and storing the title, URL, and category as metadata. A simple web interface sends questions to a small API that queries Chroma for the top eight chunks, filters by category when the user selects one, and asks a language model to answer with links to the source articles. After a successful pilot, the team sets the embedding model explicitly, moves Chroma to a server container on the internal network with authentication and nightly backups, adds an evaluation set built from real help desk tickets, and schedules a nightly job that re-indexes changed articles and deletes removed ones. The whole system runs comfortably on modest infrastructure.

Migrating from Chroma when you outgrow it

If your needs grow beyond what Chroma comfortably supports, migration is straightforward when you have kept a clean source of truth. Store original documents, chunk text, metadata, and the embedding model version outside Chroma. Export or regenerate embeddings, load them into the new database, rerun your evaluation set to compare results, and switch your retrieval code behind a single interface. Teams that build this separation from the start can change databases in days rather than weeks.

Common Chroma mistakes

• Relying on the default embedding function in production without choosing and recording a model deliberately.
• Using the in-memory client and losing data when the process restarts.
• Several processes writing to the same persistent directory instead of using client-server mode.
• Random IDs that create duplicates when ingestion jobs are rerun.
• Exposing the server without authentication on a shared or public network.
• No backups of the data directory.
• Never removing deleted documents, so outdated answers keep appearing.

Chroma compared with other options

Chroma's main strength is simplicity: it is one of the fastest ways to get semantic search working in Python or JavaScript, and it scales comfortably for many internal and departmental applications. Qdrant and Weaviate offer more advanced filtering, distributed clustering, quantization, and multi-tenancy for larger production systems. Pinecone offers a fully managed serverless service. PostgreSQL with pgvector keeps vectors alongside relational data in a database many teams already run. Many projects start with Chroma for prototyping, then either keep it for modest production workloads or migrate to one of these options as requirements grow.

Summary

ChromaDB makes it easy to build semantic search and RAG prototypes, and with the right setup it can serve modest production workloads well. Choose the embedding model deliberately, design chunking and metadata carefully, use deterministic IDs and upserts, run in client-server mode with authentication and backups, and measure retrieval quality with a real evaluation set. For a broader view, read our vector database comparison and our guide to choosing an embedding model, and see what a vector database is for the fundamentals.

A practical path from prototype to production

Stage 1, prototype: use the persistent client locally, load a representative sample of documents, and confirm that semantic search returns useful results for real questions. Stage 2, pilot: choose and record the embedding model explicitly, build an evaluation set, run Chroma as a server container on a private network with authentication, and give a small group of users access. Stage 3, production: automate ingestion with upserts and deletions, add backups and restore tests, monitor latency and errors, and review evaluation results before every change. Stage 4, review: after a few months of real usage, check whether data volume, query load, availability needs, or multi-tenancy requirements have grown beyond what your Chroma deployment handles comfortably. If they have, use your evaluation set and clean source of truth to compare and migrate to a distributed option. If not, keep Chroma: simplicity is a real advantage when it meets your needs.

Key points to remember

Chroma rewards a simple, disciplined setup. Pick and record the embedding model, chunk content along its natural structure, keep rich metadata for filtering and citations, use deterministic IDs with upserts, run the server with authentication and backups, and measure retrieval with real questions. Those habits matter far more to answer quality than any single configuration setting, and they make it easy to grow or switch databases later without losing work, because your content, metadata, and evaluation set remain the real foundation of the system.

Start simple, measure honestly, and let real usage tell you when it is time to move to something larger and more complex.

For many teams, that moment never comes, and Chroma remains the right tool.

Build with Vertex Cyber Tech

We help teams move from Chroma prototypes to production RAG systems, choosing the right vector database for their scale. Explore our AI/ML services or book a consultation.

Sources and further reading

• Chroma documentation
• Chroma on GitHub