Add support for full text queries and hybrid search queries #303
+1,331
−329
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rbs333
reviewed
Mar 26, 2025
rbs333
reviewed
Mar 26, 2025
|
@justin-cechmanek you should rebase your branch and change this pr to be against |
tylerhutcherson
force-pushed
the
feat/RAAE-227/hybrid-query
branch
from
7f959f1 to
7e0f24d
Compare
…ept aggregate queries
Prior to RedisVL 0.4.0, we validated passed-in Redis clients when the user called `set_client()`. This PR reintroduces similar behavior by validating all clients, whether we created them or not, on first access through the lazy-client mechanism. Closes RAAE-694.
Add `batch_search` and `batch_query` methods to `SearchIndex` and `AsyncSearchIndex`. These methods run search commands in a pipeline and return correctly-parsed and post-processed results.
This PR implements a layered architecture for managing and validating
searchable data in Redis, with clear separation of concerns between
schema definition, data validation, and storage operations.
- `IndexSchema` provides the blueprint for data structure and
constraints
- Defines fields with specific types (TEXT, TAG, NUMERIC, GEO, VECTOR)
- Supports different storage types (HASH, JSON) with appropriate
configuration
- `SchemaModelGenerator` dynamically creates Pydantic models from schema
definitions
- Implements a caching mechanism to avoid redundant model generation
- Maps Redis field types to appropriate Python/Pydantic types
- Provides type-specific validators:
- VECTOR: validates dimensions and value ranges (e.g., INT8 range
checks)
- GEO: validates geographic coordinate format
- NUMERIC: prevents boolean values
- `BaseStorage` is the abstract class provides the foundation for Redis
operations
- Specialized implementations (HashStorage, JsonStorage) for different
Redis data types
- Enforces schema validation during write operations when set to True
- Implements optimized batch operations using Redis pipelines
- Supports both synchronous and asynchronous interfaces
- Handles key generation, preprocessing, and error handling
The `SearchIndex` contains the setting `validate_on_load`, which
defaults on `False`.
Objects are preprocessed and validated against the schema
Objects are prepared with appropriate keys
Batch writing occurs using Redis pipelines for efficiency
TTL (expiration) can be applied if specified
Keys are fetched in batches using pipelines
Data is converted from Redis format to Python objects
Bytes are automatically converted to appropriate types
Run API-dependent tests once per matrix run.
This pr accomplishes 2 goals: 1. Add an option for users to easily get back a similarity value between 0 and 1 that they might expect to compare against other vector dbs. 2. Fix the current bug that `distance_threshold` is validated to be between 0 and 1 when in reality it can take values between 0 and 2. > Note: after much careful thought I believe it is best that for `0.5.0` we do **not** start enforcing all distance_thresholds between 0 and 1 and move to this option as default behavior. Ideally this metric would be consistent throughout our code and I don't love supporting this flag but I think it provides the value that is scoped for this ticket while inflicting the least amount of pain and confusion. Changes: 1. Adds the `normalize_vector_distance` flag to VectorQuery and VectorRangeQuery. Behavior: - If set to `True` it normalizes values returned from redis to a value between 0 and 1. - For cosine similarity, it applies `(2 - value)/2`. - For L2 distance, it applies normalization `(1/(1+value))`. - For IP, it does nothing and throws a warning since normalized IP is cosine by definition. - For VectorRangeQuery, if `normalize_vector_distance=True` the distance threshold is now validated to be between 0 and 1 and denormalized for execution against the database to make consistent. 2. Relaxes validation for semantic caching and routing to be between 0 and 2 fixing the current bug and aligning with how the database actually functions.
…ept aggregate queries
justin-cechmanek
changed the title
rbs333
reviewed
Apr 2, 2025
rbs333
reviewed
Apr 2, 2025
tylerhutcherson
approved these changes
Apr 3, 2025
tylerhutcherson
changed the title
justin-cechmanek
changed the title
abrookins
approved these changes
Apr 4, 2025
abrookins
added a commit
that referenced
this pull request
Apr 4, 2025
Co-authored-by: Robert Shelton <[email protected]> Co-authored-by: Andrew Brookins <[email protected]> Co-authored-by: Tyler Hutcherson <[email protected]> Co-authored-by: Robert Shelton <[email protected]>
abrookins
added a commit
that referenced
this pull request
Apr 4, 2025
Co-authored-by: Robert Shelton <[email protected]> Co-authored-by: Andrew Brookins <[email protected]> Co-authored-by: Tyler Hutcherson <[email protected]> Co-authored-by: Robert Shelton <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.