Is it possible to run a custom get_queryset method with django-auto-prefetch?

[nickmccullum]

Python Version

3.10

Django Version

4.2.6

Package Version

1.7.0

Description

I am trying to integrate django-auto-prefetch with a Django project that has a soft deletion framework.

This doesn't work (i.e., it uses the regular Django query functionality.

import auto_prefetch


class NotDeletableManager(auto_prefetch.Manager):
    def get_queryset(self) -> models.QuerySet:
        return super().get_queryset().filter(deleted_at__isnull=True)

any tips?

[tolomea]

I'm going to need more detail about your current setup. Is the soft deletion thing a library or just in the project? How is it enabled / used at present? In what contexts are you getting the full results instead of the soft deletion filtered ones?

…

On Mon, Oct 9, 2023, 13:49 Nick McCullum ***@***.***> wrote: Python Version 3.10 Django Version 4.2.6 Package Version 1.7.0 Description I am trying to integrate django-auto-prefetch with a Django project that has a soft deletion framework. This doesn't work (i.e., it uses the regular Django query functionality. import auto_prefetch class NotDeletableManager(auto_prefetch.Manager): def get_queryset(self) -> models.QuerySet: return super().get_queryset().filter(deleted_at__isnull=True) any tips? — Reply to this email directly, view it on GitHub <#275>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGERUCEM2GRJYU463LGDQ3X6PXG7AVCNFSM6AAAAAA5YYB6CWVHI2DSMVQWIX3LMV43ASLTON2WKOZRHEZTEOJZGY4DQMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[nickmccullum]

Here are answers to your questions:

• This is custom code in our project, used in an abstract model that is inherited by many other models. It looks like this (before the django-auto-prefetch integration):

class DeletedObjectsManager(models.Manager):
    def get_queryset(self) -> models.QuerySet:
        return super().get_queryset().filter(deleted_at__isnull=False)


class NotDeletableManager(models.Manager):
    def get_queryset(self) -> models.QuerySet:
        return super().get_queryset().filter(deleted_at__isnull=True)


class NotDeletableMixin(models.Model):
    """
    A framework for soft deletion in Django.
    """

    objects = NotDeletableManager()
    deleted_objects = DeletedObjectsManager()
    all_objects = models.Manager()

• That NotDeletableMixin is inherited by other models
• We use django-perf-rec to monitor the SQL queries performed across our entire test suite. This setup did not create any changes to the SQL used in our app:

import auto_prefetch


class DeletedObjectsManager(auto_prefetch.Manager):
    def get_queryset(self) -> models.QuerySet:
        return super().get_queryset().filter(deleted_at__isnull=False)


class NotDeletableManager(auto_prefetch.Manager):
    def get_queryset(self) -> models.QuerySet:
        return super().get_queryset().filter(deleted_at__isnull=True)


class NotDeletableMixin(auto_prefetch.Model):
    """
    A framework for soft deletion in Django.
    """

    objects = NotDeletableManager()
    deleted_objects = DeletedObjectsManager()
    all_objects = models.Manager()

However, this setup did result in new SQL being generated:

import auto_prefetch


class NotDeletableMixin(models.Model):
    """
    A framework for soft deletion in Django.
    """

    objects = auto_prefetch.Manager()
    deleted_objects = auto_prefetch.Manager()
    all_objects = auto_prefetch.Manager()

which leads me to believe there might be a problem with the get_queryset method.

Hopefully this help!

[tolomea]

That all sounds like it could be expected. It's common that adding auto prefetch doesn't change perf rec results. Anywhere you are doing perf rec I expect you have already also added appropriate select/prefetch_related calls. With those in place auto prefetch had nothing to do in those cases. In the second case you are loosing the soft delete filtering which will change your pref rec output, although probably not the number of queries, just the details of the queries.

…

On Mon, Oct 9, 2023, 14:11 Nick McCullum ***@***.***> wrote: Here are answers to your questions: - This is custom code in our project, used in an abstract model that is inherited by many other models. It looks like this (before the django-auto-prefetch integration): class DeletedObjectsManager(models.Manager): def get_queryset(self) -> models.QuerySet: return super().get_queryset().filter(deleted_at__isnull=False) class NotDeletableManager(models.Manager): def get_queryset(self) -> models.QuerySet: return super().get_queryset().filter(deleted_at__isnull=True) class NotDeletableMixin(models.Model): """ A framework for soft deletion in Django. """ objects = NotDeletableManager() deleted_objects = DeletedObjectsManager() all_objects = models.Manager() - That NotDeletableMixin is inherited by other models - We use django-perf-rec to monitor the SQL queries performed across our entire test suite. This setup did not create any changes to the SQL used in our app: import auto_prefetch class DeletedObjectsManager(auto_prefetch.Manager): def get_queryset(self) -> models.QuerySet: return super().get_queryset().filter(deleted_at__isnull=False) class NotDeletableManager(auto_prefetch.Manager): def get_queryset(self) -> models.QuerySet: return super().get_queryset().filter(deleted_at__isnull=True) class NotDeletableMixin(auto_prefetch.Model): """ A framework for soft deletion in Django. """ objects = NotDeletableManager() deleted_objects = DeletedObjectsManager() all_objects = models.Manager() However, this setup *did* result in new SQL being generated: import auto_prefetch class NotDeletableMixin(models.Model): """ A framework for soft deletion in Django. """ objects = auto_prefetch.Manager() deleted_objects = auto_prefetch.Manager() all_objects = auto_prefetch.Manager() which leads me to believe there might be a problem with the get_queryset method. Hopefully this help! — Reply to this email directly, view it on GitHub <#275 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGERUCAOKMWZT7C2ITR3ATX6PZXPAVCNFSM6AAAAAA5YYB6CWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJSHE4DMNBSGY> . You are receiving this because you commented.Message ID: ***@***.***>

[nickmccullum]

that's a good point.

I am going to specifically set up a test case that should trigger an N+1 query, and see what the behavior is. I'll update you eventually.

[nickmccullum]

Bah, you're right. this is working as intended.

I used django-perf-rec to profile this code:

with django_perf_rec.record():
    rental_units = RentalUnit.objects.all()
    [rental_unit.building.property_management_company for rental_unit in rental_units]

which should absolutely create an N+1 query under Django's standard configuration. However, only 3 queries were performed. Here's the .yml output:

test_auto_prefetch:
- db: 'SELECT ... FROM "rental_unit" WHERE "rental_unit"."deleted_at" IS #'
- db: SELECT ... FROM "building" WHERE "building"."id" IN (...)
- db: SELECT ... FROM "property_management_company" WHERE "property_management_company"."id" IN (...)

Closing this issue now. Thanks for your time.

[tolomea]

Glad I could help

…

On Mon, Oct 9, 2023 at 5:07 PM Nick McCullum ***@***.***> wrote: Closed #275 <#275> as completed. — Reply to this email directly, view it on GitHub <#275 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGERUD6HLSXRYVMOHQNTTDX6QOMFAVCNFSM6AAAAAA5YYB6CWVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJQGU4TGMJZGQZTANI> . You are receiving this because you commented.Message ID: ***@***.*** com>