Larger Embedding Models Aren't Always Better: Our Experience with Corporate Documents

What are embeddings and why their dimension impacts your business

When an AI system searches for answers within your documents, it doesn't read every file every time. Instead, it pre-processes each text chunk into a numerical "fingerprint" – a vector. This is an embedding.

Imagine each paragraph of your contract or instruction manual is transformed into a coordinate in a multi-dimensional space. When an employee asks a question, the system searches based on the similarity of these coordinates, not just keywords. This is why AI search can find the right contract clause even if the query is phrased differently than the text itself.

Dimension refers to the number of values in such a vector. OpenAI's text-embedding-3-small generates vectors with 1536 values, while text-embedding-3-large uses 3072. It seems intuitive: more numbers mean a more precise search. However, in practice, it's more complex.

Why is this important for businesses? Because dimension directly affects three things: search speed, server memory usage, and document processing costs. If you choose a dimension "just in case," you pay for it every day.

Why companies overpay for dimension they don't need

When we start working with a new client, one of the first questions we discuss is which vector dimension will be suitable for their archive. Almost every time, we hear the same sentiment: "Let's go with 3072 – bigger is better, right?"

This logic is perfectly understandable. More dimensions seem to imply more details and a more accurate search. But in practice, this isn't always the case. We explain why – with numbers.

According to a study on optimizing vector storage in RAG systems (arXiv, 2025), one million documents with 1536-dimensional vectors occupy about 6.1 GB of RAM. With 3072-dimensional vectors, it's twice as much, over 12 GB. In cloud infrastructure, RAM is a direct driver of monthly costs.

Meanwhile, the increase in search quality is minimal. A comparative analysis of embedding models in 2026 shows that moving from 1536 to 3072 dimensions yields only a 2-4 point increase in nDCG on retrieval benchmarks. "The quality curve flattens very quickly after 768 dimensions for most tasks," yet storage costs are six times higher.

Here's how the comparison looks in numbers:

For an SMB company with an archive of 50,000–200,000 documents – a typical scale for a law firm, medical center, or distributor – the difference in annual costs can amount to several thousand dollars. Without any noticeable improvement in response quality.

The conclusion is simple: "more" in the case of embeddings is not always "better." It's "more expensive" and "slower." And for most corporate documents, the search quality with 1536 dimensions is entirely sufficient.

What we achieved with 1536 in a real product

At AskYourDocs, we went through this decision not in theory, but with a real product and real clients. We'll be honest: how we arrived at 1536, what it gave us, and where the challenges were.

Our stack: Spring Boot, Java 21, PostgreSQL with the pgvector extension, OpenRouter as a gateway to LLMs. Infrastructure – Railway EU West (Amsterdam), file storage – Cloudflare R2 in the EU jurisdiction. For embeddings, we chose text-embedding-3-small with a dimension of 1536.

Why 1536, not 3072

Initially, we also considered: perhaps we should use a larger model – just in case? But when we calculated the actual costs at scale and looked at benchmarks for our type of documents (legal contracts, internal regulations, HR documents, instructions), it became clear: the difference in search quality between 1536 and 3072 for homogeneous corporate texts in a single language is minimal. But the difference in costs is significant.

What changed after choosing 1536

An Important Lesson: Dimension is Not the Main Variable

When search accuracy was low (17%), the problem wasn't that we chose 1536 instead of 3072. The problem was the quality of OCR recognition – scanned PDFs provided "garbage" as input, and no embedding dimension could save that.

After implementing Vision OCR (GPT-4o-mini for recognition with automatic retry for 90°/180°/270° for inverted scans), search accuracy increased to an acceptable level – with the same 1536 dimensions.

This confirms a simple idea: the quality of input data is more important than the vector dimension. Garbage in, garbage out, regardless of whether it's 1536 or 3072.

Note: The figures provided are the result of our specific deployment and the types of documents our clients have. Your results will depend on archive volume, document language, and input file quality.

When 3072 is truly justified – and when it's an unnecessary expense

We're deliberately not saying that 3072 is always bad. That would be dishonest. There are scenarios where a larger dimension is indeed justified – and we ourselves recommend it to clients when we see the right conditions. However, such cases are significantly less common among SMB companies than one might think.

When 3072 Makes Sense

Multilingual documents within the same chunk. If your archive contains documents where multiple languages are mixed in a single paragraph – for example, a technical specification with English terms and a Ukrainian description, or a contract with quotes from foreign legislation – a larger dimension better captures semantic links between languages. 1536 can handle it, but with noticeable losses in cross-lingual search.

Complex domain terminology. Medical protocols, scientific articles, patent documentation – texts where phrasing is critically important and terms that sound similar have different meanings. Here, a higher dimension provides better "resolution" between closely related concepts.

Archives of millions of documents with business-critical search. If you have 2-5 million documents, and even one missed relevant result costs money or reputation – a 2-4 point nDCG increase on benchmarks becomes tangible in real scenarios. At this scale, it's worth testing both options.

There is an infrastructure budget. If RAM and storage are not a limitation, and the team is prepared for higher operational costs – 3072 provides a certain margin of accuracy "just in case.".

When 1536 is Entirely Sufficient

Homogeneous documents in a single language. Internal regulations, employment contracts, HR documents, employee instructions, price lists – this is the most common type of archive among our clients. Here, 1536 provides search quality indistinguishable from 3072 in practice.

Archives up to 500,000 documents. The typical scale for a law firm, medical center, or distributor. At this volume, the difference between models in real queries is statistical noise, not a business effect.

Deployments on limited resources. If you are deploying the system on your own servers or in an isolated environment (Hetzner, on-premise) – the twice-smaller RAM footprint from 1536 could mean the difference between one and two servers in a rack.

Response speed is important. Smaller dimension = faster similarity search with the same hardware. For products where users expect real-time responses, this is noticeable.

An Interesting Fact That Changes Intuition

As noted in the official OpenAI documentation, text-embedding-3-large, when reduced to 256 dimensions, still outperforms the full-size text-embedding-ada-002 (1536) on the MTEB benchmark. This means modern models have learned to pack information more efficiently – and the maximum dimension is no longer synonymous with maximum quality.

It's important to understand this: we are no longer in an era where "more dimensions = better." We are in an era of efficient encoding, where a well-trained model with 1536 dimensions wins against an outdated model with 3072.

In Short: The Choice Matrix

How to choose the dimension for your document archive – a checklist

This is the most practical section of the article. If you are currently choosing an embedding model for your AI system, answer the six questions below. After each question, there are specific examples from our experience so you can identify with your scenario.

Question 1: What is the language of your documents?

This is the first and most important question. The embedding dimension significantly impacts search quality, especially in a linguistic context.

• Single archive – single language (all contracts in Ukrainian, all instructions in German) → 1536 is sufficient. The model handles homogeneous language content well.
• Documents in different languages, but in separate files (part of the archive in Ukrainian, part in English) → 1536 will cope if the search is conducted in the document's language.
• Mix of languages within a single chunk (technical documentation where one paragraph contains English terms and Ukrainian explanations) → it's worth testing 3072.

Example from practice: a client – a Ukrainian law firm, with an archive of 40,000 contracts in Ukrainian. They chose 1536. Searching by the content of contract clauses works correctly, the client is satisfied with the result.

Another example: a distributor with documentation from foreign suppliers – some files in English, some in Ukrainian, some with mixed languages in specification tables. Here, we recommended testing both options before making the final choice.

Question 2: What is the volume of your archive?

The archive volume affects two factors simultaneously: indexing cost (API calls) and the cost of storing vectors in the database.

Important: indexing cost is a one-time expense. However, the cost of storing vectors in RAM is a monthly expense. With 3072 dimensions, it's twice as high. Also read: how to correctly prepare documents for AI search – file structure affects chunk size and, consequently, the number of vectors in the database.

Question 3: What document formats are in your archive?

This is a question often overlooked when choosing dimensions – and it shouldn't be. The quality of the input text is more important than the vector size.

• Digital PDFs and Word files → clean text, direct indexing. 1536 will provide full quality.
• Scanned PDFs (scanned paper documents) → the main issue here is not the dimension, but the OCR quality. Poor OCR provides "garbage" as input – and no dimension can save that. First, address the recognition issue, and only then evaluate the vectors. More details in our case study: how Vision OCR improves RAG accuracy on scanned documents.
• Mixed archive (part digital, part scanned) → a separate pipeline for scanned documents is needed before indexing.

Example from practice: we had a case where search accuracy on a scanned archive was 17%. We tried changing parameters – it didn't help. The problem was that the OCR provided garbled text with substitute characters. After implementing Vision OCR, accuracy increased to ~50% – with the same 1536 dimensions. The dimension was not a factor at all here.

Question 4: What is your infrastructure budget?

This question is not just about API costs, but about the total cost of ownership (TCO): server RAM, storage in the vector database, hosting costs.

Question 5: How critical is search accuracy for your business?

Let's be honest: for most corporate tasks, the difference between 1536 and 3072 is practically imperceptible. But there are exceptions.

• Search within internal regulations, HR documents, instructions → 1536 accuracy is more than sufficient. Employees will find the necessary clause.
• Legal search in contracts → 1536 handles it. More critical are correct chunking and OCR quality.
• Medical protocols with narrow terminology → it's worth testing 3072, especially if similar terms refer to different procedures.
• Business-critical search where a missed result means financial or legal risk → test both options on your real archive before launching.

Question 6: Do you have resources for testing?

If you have doubts – the most honest answer is: conduct an A/B test on your actual archive. Take 50-100 typical queries from future users, index a sample of documents with both models, and compare the results. This will take a few hours and provide a more accurate outcome than any benchmark.

If you don't have resources for testing – focus on the table below: