Running test with mysql database throws too many connections or memory issue

[walterdis]

Pulse Version

latest

Laravel Version

10.32

PHP Version

8.1.9

Livewire Version

dont know

Database Driver & Version

MySQL 8.0.31 for macOS 14.0 on arm64 (Docker)

Description

Hello,

When I run tests using the MySQL database with --parallel, even with Redis, the tests stop in the middle, giving me the error message: "Too many connections."

[1040] Too many connections (Connection: mysql, SQL: drop database if exists `project_testing_test_2`))

If i run without --parallel, it's giving me the memory limit exception

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/vendor/laravel/framework/src/Illuminate/Collections/Arr.php on line 565

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/vendor/laravel/framework/src/Illuminate/Collections/Arr.php on line 565
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/vendor/pestphp/pest/src/Plugins/Actions/CallsShutdown.php on line 1

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/vendor/pestphp/pest/src/Plugins/Actions/CallsShutdown.php on line 1

Am I doing something wrong?

Steps To Reproduce

Configure tests to use mysql

• php artisan test --parallel (too many connections)
• php artisan test (throws memory exception)

[jbrooksuk]

Disable Pulse in your tests. You can add PULSE_ENABLED=false in your phpunit.xml file.

[walterdis]

Hello.

I've already done that and the problem still persists

[timacdonald]

@walterdis can you run the following terminal command in your application and let me know the output.

composer outdated -- laravel/pulse | grep beta

[jessarcher]

Have you configured a different database connection for Pulse but maybe pointing to the same MySQL server? It's the only reason I can think why Pulse's existence would increase the number of database connections.

[timacdonald]

Although I'm not entirely sure the root cause of this issue, I've created a PR that might resolve it by not hitting the ingest unless there are entries to store.

#212

[walterdis]

@walterdis can you run the following terminal command in your application and let me know the output.

composer outdated -- laravel/pulse | grep beta

@timacdonald Hello ;)

versions : * v1.0.0-beta6
latest   : v1.0.0-beta6

@jessarcher Hi, i have configured it to use Redis. I was hoping that when I did this, Laravel Pulse would start getting the data through the worker and not open connections to the database all the time

[timacdonald]

@walterdis can you please upgrade to v1.0.0-beta7 and see if that fixes your issue.

Please ensure that PULSE_ENABLED is set to false in your phpunit.xml.

[walterdis]

@walterdis can you please upgrade to v1.0.0-beta7 and see if that fixes your issue.

Please ensure that PULSE_ENABLED is set to false in your phpunit.xml.

@timacdonald Hi, thx for the answer ;)

I've tested 2 scenarios:

php artisan test --parallel
The problem with too many connections disappeared even with pulse_enabled = true ;)

php artisan test
Keeps throwing memory exception ;(

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/vendor/pestphp/pest/src/Plugins/Actions/CallsShutdown.php on line 1

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/vendor/pestphp/pest/src/Plugins/Actions/CallsShutdown.php on line 1```

[Andilutten]

This just happened to us in production...
Laravel pulse was using 150 / 150 DB connections while updating something in the pulse_aggregates table. 1.5 hours of outage..

[taylorotwell]

@Andilutten were you using Octane?

[Andilutten]

@taylorotwell Nope

[timacdonald]

@Andilutten could you provide information here on what your setup looks like?

1. What Ingest driver are you using?
2. Are you running the pulse:work command? How many instances of pulse:work are you running across how many servers?
3. Are you running the pulse:check command? How many instances are you running across how many servers?
4. Do you know if the issue originated in the pulse:work or pulse:check command, during a request, or on the queue?
5. Which version of the beta you had installed at the time?

Some other information that would help:
6. What queue driver are you using?
7. How many queue workers do you have active and how many processes does each have?

[timacdonald]

@walterdis if you disable pulse AND telescope in your test suite by putting the following in your phpunit.xml...

<?xml version="1.0" encoding="UTF-8"?>
<phpunit>
    <!-- ... -->
    <php>
        <!-- ... -->
        <env name="TELESCOPE_ENABLED" value="false"/>
        <env name="PULSE_ENABLED" value="false"/>
    </php>
</phpunit>

do you still get the memory / db issues in your tests? Just trying to nail down where things could be falling apart in the tests.

[DaanTisg]

We have the same issue. Both Telescope and Pulse are disabled in the CI environment:

PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 565248 bytes) in /var/www/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php on line 56

[walterdis]

@walterdis if you disable pulse AND telescope in your test suite by putting the following in your phpunit.xml...

<?xml version="1.0" encoding="UTF-8"?>
<phpunit>
    <!-- ... -->
    <php>
        <!-- ... -->
        <env name="TELESCOPE_ENABLED" value="false"/>
        <env name="PULSE_ENABLED" value="false"/>
    </php>
</phpunit>

do you still get the memory / db issues in your tests? Just trying to nail down where things could be falling apart in the tests.

@timacdonald Yes, i've tested all scenarios i could think of.

My current phpunit.xml (yes, i've tried without the clockwork extension)

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.3/phpunit.xsd" backupGlobals="false"
         bootstrap="vendor/autoload.php" colors="true" stopOnFailure="true" cacheDirectory=".phpunit.cache" processIsolation="false">
    <testsuites>
        <testsuite name="Modules">
            <directory suffix="Test.php">./app-modules/*/tests/Feature</directory>
            <directory suffix="Test.php">./app-modules/*/tests/Unit</directory>
        </testsuite>
    </testsuites>
    <php>
        <server name="APP_ENV" value="testing"/>
        <server name="BCRYPT_ROUNDS" value="4"/>
        <server name="CACHE_DRIVER" value="array"/>
        <server name="MAIL_MAILER" value="array"/>
        <server name="QUEUE_CONNECTION" value="sync"/>
        <server name="SESSION_DRIVER" value="array"/>
        <server name="TELESCOPE_ENABLED" value="false"/>
        <server name="DB_FOREIGN_KEYS" value="true"/>
        <server name="PULSE_ENABLED" value="false"/>
        <ini name="error_reporting" value="E_ALL"/>
        <ini name="display_errors" value="1"/>
    </php>

    <extensions>
        <bootstrap class="Clockwork\Support\Laravel\Tests\ClockworkExtension"/>
    </extensions>

</phpunit>

.env.testing

CLOCKWORK_ENABLE=false
CLOCKWORK_TESTS_COLLECT=false

PULSE_INGEST_DRIVER=redis
PULSE_REDIS_CONNECTION=pulse
REDIS_PULSE_DB=1
PULSE_ENABLED=false

Still getting the memory exception running php artisan test but, when i run php artisan test --parallel the memory issue don't happen

[Andilutten]

@Andilutten could you provide information here on what your setup looks like?

1. What Ingest driver are you using?

2. Are you running the pulse:work command? How many instances of pulse:work are you running across how many servers?

3. Are you running the pulse:check command? How many instances are you running across how many servers?

4. Do you know if the issue originated in the pulse:work or pulse:check command, during a request, or on the queue?

5. Which version of the beta you had installed at the time?

Some other information that would help:

6. What queue driver are you using?

7. How many queue workers do you have active and how many processes does each have?

I wasnt using redis ingest. The pulse setup was pretty much straight out of the box. I was using the applications MySQL database, (hence the mentioned outage). I was not running either the check or work command, and the application is only using nightly sync jobs. So no queue workers required. (Atleast not right now)

I don't have access to the code right now since I'm out and about , but I will post the version info when I get back home.

[DaanTisg]

For me the issue has been resolved by updating all composer packages. Here are the changed package versions. It may be of use to someone:

[timacdonald]

Got some more PRs coming in that should help address the memory issues. Will likely tag them early next week. Still working on understanding the DB connection issues. If there is any additional information you think could be useful to help us replicate the issue, that would be amazing 🤘

[Andilutten]

@timacdonald
I was using version v1.0.0-beta2

[tasarsu]

Hi, I have same issue here, tested with v1.0.0-beta5 and v1.0.0-beta7

There are 800 tests in my project, test are running on mysql, phpunit.xml file with PULSE_ENABLED as false. No redis. After 400 tests I get this error:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 266240 bytes) in .... each time in different file

[ngl5000]

I'm experiencing memory issues as well. my db is postgres. I have a command which loops over a dbf file ~70k rows and saves the data in two tables using updateorcreate. when i have pulse enabled the mem required goes from a starting point of ~40mb to 250mb. with pulse diabled mem stays ~ 40mbs through the entire process

[walterdis]

Got some more PRs coming in that should help address the memory issues. Will likely tag them early next week. Still working on understanding the DB connection issues. If there is any additional information you think could be useful to help us replicate the issue, that would be amazing 🤘

@timacdonald Hi.
I've created a test project for the memory issue.

https://github.com/walterdis/pulse-test

[timacdonald]

Hey folks, I've had a potential breakthrough on the memory issue. Will have something new tomorrow.

In the meantime, @ngl5000 and @tasarsu, can you confirm if the issue exists on the latest beta when you have PULSE_ENABLED set to false in your PHPUnit.xml?

[timacdonald]

Hey folks,

The memory issue should be resolved with a buffer limit introduced in #250

The DB connection issue in testsuites will be resolved using the 'null' Ingest driver introduced in #245

This means that you can put a cap on the amount of memory Pulse will capture in it's buffer and also keep your tests fast by not requiring Pulse to actually write to the DB during tests - but keep it enabled to reflect production.

All ingesting is wrapped in rescue calls, so having it write to the database doesn't really serve much purpose in a testsuite IMO.

[timacdonald]

@Andilutten do you know what kind of load you application was under around that time?

Was it a substantial amount of requests per minute?

I'm thinking perhaps you had a lot of traffic and the upsert queries were locking each other out and you had enough traffic for it to be problematic.

For high-traffic applications we recommend the Redis driver for this reason.

I'm sorry again that this impacted your production app.

[walterdis]

Hey folks,

The memory issue should be resolved with a buffer limit introduced in #250

The DB connection issue in testsuites will be resolved using the 'null' Ingest driver introduced in #245

This means that you can put a cap on the amount of memory Pulse will capture in it's buffer and also keep your tests fast by not requiring Pulse to actually write to the DB during tests - but keep it enabled to reflect production.

All ingesting is wrapped in rescue calls, so having it write to the database doesn't really serve much purpose in a testsuite IMO.

Hello @timacdonald

I've done a composer update but the package was not updated.
Tried composer remove laravel/pulse and composer require laravel/pulse too

What i`m doing wrong?

Thanks

[timacdonald]

@Andilutten we will tag a release next week that should allow you to update. Will keep ya posted.

[timacdonald]

v1.0.0-beta8 has been tagged which should introduce these fixes.

[ngl5000]

@timacdonald i ran the monthly upload with the new beta release and it worked great this time. Thank you!!

[walterdis]

It's working here as well.

Thanks! ;)

[timacdonald]

Great to hear!

[taghwo]

@tasarsu did the beta8 push fix your issue? Having same memory error as you commented. I have more than 800 test cases, it timesout with error:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 65536 bytes) in /var/www/vendor/laravel/framework/src/Illuminate/Support/ViewErrorBag.php on line 11

Laravel pulse version:

versions : * v1.0.0-beta8
latest   : v1.0.0-beta8

phpunit.xml

    <php>
        <env name="APP_ENV" value="testing"/>
        <env name="BCRYPT_ROUNDS" value="4"/>
        <env name="CACHE_DRIVER" value="array"/>
        <env name="MAIL_MAILER" value="array"/>
        <env name="QUEUE_CONNECTION" value="sync"/>
        <env name="SESSION_DRIVER" value="array"/>
        <env name="TELESCOPE_ENABLED" value="false"/>
        <env name="PULSE_ENABLED" value="false"/>
    </php>

@timacdonald I don't know if I need to open another issue.

[timacdonald]

@taghwo I think I know what could be impacting your testsuite.

Could you try something for me to help confirm: would you search of all usages of the AboutCommand::add function and comment them out?

You should at least find them in the following files:

• vendor/laravel/pulse/src/PulseServiceProvider.php
• vendor/livewire/livewire/src/LivewireServiceProvider.php

Then could you try running the testsuite again and let me know if there is an improvement?

That function has a memory leak that is potentially impacting your testsuite. I've submitted fix / work around to the framework (laravel/framework#49557).

[tasarsu]

Hi @taghwo and @timacdonald
I have just tested beta 8, and all the tests were completed without any memory errors. Thanks for the fix.

[timacdonald]

Thank you so much for trying that out.

We'll merge the framework fix and encourage everyone to update to get the fix across the ecosystem.

I'll close this and we can track the framework PR.

[taghwo]

@taghwo I think I know what could be impacting your testsuite.

Could you try something for me to help confirm: would you search of all usages of the AboutCommand::add function and comment them out?

You should at least find them in the following files:

• vendor/laravel/pulse/src/PulseServiceProvider.php
• vendor/livewire/livewire/src/LivewireServiceProvider.php

Then could you try running the testsuite again and let me know if there is an improvement?

That function has a memory leak that is potentially impacting your testsuite. I've submitted fix / work around to the framework (laravel/framework#49557).

@timacdonald Unfortunately after commenting those lines out I still get the memory issue. Uninstalling Laravel Pulse the test suite runs successfully.

[timacdonald]

@taghwo I've found another memory leak that could be casing the issue; this time in Livewire.

I've submitted a fix (livewire/livewire#7677).

Could you please try commenting out the AboutCommand::add functions again in:

• vendor/laravel/pulse/src/PulseServiceProvider.php
• vendor/livewire/livewire/src/LivewireServiceProvider.php

and then opening the vendor/laravel/pulse/src/PulseServiceProvider.php and commenting out the line $livewire->addPersistentMiddleware($app->make('config')->get('pulse.middleware', [])); as seen here:

pulse/src/PulseServiceProvider.php

Line 170 in 32fb030

$livewire->addPersistentMiddleware($app->make('config')->get('pulse.middleware', []));

Would love to know if that helps the situation at all. Thank you again for helping out with this one.

[timacdonald]

I've also found another memory leak in the framework that could have been impacting you due to the introduction of Livewire.

Could you make sure to update to at least laravel/framework v10.37.3 and try again (also with the above ideas).

[taghwo]

Will try these out over the weekend and get back on Monday.

[taghwo]

Hello @timacdonald, I have some update.

Current package versions:



Laravel Pulse


versions : * v1.0.0-beta8
latest   : v1.0.0-beta8



Laravel framework

versions : * v10.39.0



Test Suite Result:

Ran successful without memory issues after commenting out the lines of code specified.



Current Issue:

I cannot seem to update laravel/pulse to v1.0.0-beta8 without adding —with-dependencies. It throws the error
:

    laravel/pulse v1.0.0-beta8 requires illuminate/auth ^10.34 -> 
    found illuminate/auth[v10.34.0, ..., v10.39.0] but these were not loaded, 
    likely because it conflicts with another require.


If I add —with-dependencies after "update laravel/pulse:1.0.0-beta8" it updates the Laravel/framework to v10.39 plus other dependencies which I am not in control of.
Is this the expected behaviour because I am not sure I want to update to the latest version of the framework yet. Would like you to guide me on this part.
Thanks.

[timacdonald]

@taghwo, Pulse now requires a minimum framework version of 10.34. If your application has not yet updated to 10.34 or greater you will not be able to install the latest beta.

I recommend updating at least to 10.37.3 to address the memory leak issue reported.

I'm glad to hear the memory issue was resolved.

[taghwo]

Worked fine now. Thanks @timacdonald.

[RomkaLTU]

I have same issue, it's not resolved or re-apeared again. Memory leaks when running test everything is disabled in phpunit.xml.

Pulse version: beta15
Framework version: 10.48.2

[timacdonald]

@RomkaLTU, are you saying you are experiencing memory leaks in your testsuite on those versions?