Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix resource issues with embeddings indexing components backed by databases #831

Closed
davidmezzetti opened this issue Dec 6, 2024 · 0 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@davidmezzetti
Copy link
Member

Currently, there are scenarios where embeddings index components backed by a database (i.e. pgvector) have issues with upserts that delete all existing data.

The following issues have been identified.

  • Passing the SQLAlchemy engine to table DDL statements. This wraps the operation with another layered transaction.
  • Passing the SQLAlchemy engine to the database session. This is causing locking behavior within the same database component.
  • For ANNs backed by databases, the close method must be run before recreating a new ANN. Logic should be added to ensure this.

This work will address these issues and ensure that database-connected indexing components have all their actions run through a single transaction until a save is called. This ensures consistency with file-based components.

@davidmezzetti davidmezzetti added the bug Something isn't working label Dec 6, 2024
@davidmezzetti davidmezzetti added this to the v8.1.0 milestone Dec 6, 2024
@davidmezzetti davidmezzetti self-assigned this Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant