🚀 LiteDB v6.0 Prerelease#52: Introducing Vector Search!

This release marks a significant milestone for LiteDB with the introduction of Vector Search! This powerful new feature enables you to build modern AI-powered applications, such as semantic search, recommendation engines, and Retrieval-Augmented Generation (RAG) systems, directly within LiteDB.

✨ Major New Feature: Vector Search

• Native Vector Storage & Indexing: We've added a native BsonVector type (for float[]) and a high-performance vector index based on the HNSW algorithm. You can now perform fast Approximate Nearest Neighbor (ANN) searches using standard distance metrics: Cosine (default), Euclidean, and DotProduct.

• New Fluent Query API: The query API has been extended with intuitive methods for vector operations.

  using LiteDB.Vector; // New namespace for vector extensions!
  
  // Create a vector index on your embedding property
  var docs = db.GetCollection<Document>("docs");
  docs.EnsureIndex(x => x.Embedding, new VectorIndexOptions(256));
  
  // Find the top 5 most similar documents to your query vector
  var results = docs.Query()
      .TopKNear(x => x.Embedding, queryVector, k: 5)
      .ToList();
  

• SQL Support & Demo: Vector searches are also supported in SQL via the VECTOR_SIM() function. To see a complete, end-to-end example, check out the new LiteDB.Demo.Tools.VectorSearch project in the repository, which builds a semantic search engine using Google Gemini for embeddings.

🛠️ Other Improvements & Fixes

• Reliability: Improved data integrity for external file storage (LiteFileStorage) by ensuring streams are correctly checkpointed on transaction commit.
• Build: The build process has been optimized to better support shallow clones in CI/CD environments.

What's Changed

• feat: Vector Search and Similarity Indexing by @JKamsker in https://github.com/litedb-org/LiteDB/pull/2678
• Ensure external streams checkpoint on commit by @JKamsker in https://github.com/litedb-org/LiteDB/pull/2652
• Disable GitVersion in shallow clones by @JKamsker in https://github.com/litedb-org/LiteDB/pull/2656

Full Changelog: https://github.com/litedb-org/LiteDB/compare/v6.0.0-prerelease.0015...v6.0.0-prerelease.0052