LLMs and Vector Databases: How Does Semantic Search Work?

An AI application's ability to produce accurate answers depends on more than the large language model it uses. How the model accesses the information it needs is also crucial. Users' questions may not match the wording in stored records exactly. This is where the distinction between keyword-based search and search that assesses semantic similarity becomes important. When evaluating database options for LLM applications, the starting question is not which product is more powerful, but what information needs to be accessed and how.

It would be an inaccurate generalization to dismiss SQL and NoSQL databases as inadequate for AI. The limitation often lies in the search method rather than the database itself. For example, a record described as a red sports car will not be found by a SQL LIKE query looking for the exact phrase fast car. Full-text search can improve word-based retrieval; however, word matching alone is not always enough to identify similar concepts expressed in different ways.

Semantic search is built on embedding models that convert content into numerical vectors. Using an appropriate model, texts are transformed into multidimensional coordinates that represent semantic relationships. The user's question is also vectorized using the same or a compatible model. The query vector is then compared with the vectors of the stored records. This allows a search for fast car to include a record for a red sports car among the relevant results, even though it does not contain the same words. Success depends on the quality of the model and the data.

As the number of vectors grows, comparing each query against every record individually can become costly. ANN, or approximate nearest neighbor search, aims to find similar vectors more efficiently. HNSW is one of the indexing approaches used for this purpose. The word approximate matters here: a balance is struck between search speed, resource usage, and the ability to retrieve relevant results. An index should therefore be assessed not just on speed, but also on the result quality the application requires.

This retrieval approach plays an important role in RAG architectures. RAG retrieves content relevant to the user's question from an information source and provides it to the language model as context. Vector search is one method that can be used to select this content; it is not the same as training the model. If the retrieved documents are incorrect, incomplete, or irrelevant, answer quality may also be affected. RAG design therefore requires attention not only to model selection, but also to the preparation of source content and the relevance of search results.

In practice, replacing the existing database is not necessarily required. Extensions that provide vector support, such as pgvector, and vector-focused solutions such as Pinecone, Weaviate, or Milvus can be evaluated for different needs. The decision should consider integration with existing systems, data volume, filtering requirements, and operating conditions together. At X Mind Solutions, we approach this as a matter of building an architecture suited to AI's information retrieval needs, rather than excluding traditional databases.