Lots of proc files left open

[ratetel]

Hello... we are running the latest rtpengine with recording in debian 11 lxcs. Under high loads, we are seeing several million open files left behind with:

~# lsof |grep rtpengine | wc -l
15452848

As a result we are getting tons of "Too many open files" errors and rtpengine fails to engage.

We've set respectable ulimits for the LXC and host OS.

Anyone have an ideas?

[rfuchs]

Which files are those?

Are you closing your calls correctly?

[ratetel]

Hi Richard... These are all produced by rtpengine-recording. When we restart rtpengine and rtpengine-recording, the count drops and stays to around 16k until we put calls on it. It seems to be a leak in the recording module. To be clear... We are using proc, merging both legs, and outputting to mp3. Alex

…

On Fri, Jan 20, 2023, 8:09 AM Richard Fuchs ***@***.***> wrote: Which files are those? Are you closing your calls correctly? — Reply to this email directly, view it on GitHub <#1597 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNDBBDN3W4QOCHOTRMJXY3WTKMBPANCNFSM6AAAAAAT7XACUU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[rfuchs]

Can you be more specific? A concrete example maybe?

[vladpaiu]

Hello,

On a server not running traffic any more, we're running

lsof | grep rtpengine > /tmp/lsof_out

Then for a callid that's ended, we do

grep -nrH "74eb216b2a77a9ba5f43b2604ca41937" /tmp/lsof_out | wc -l
1828

so we get a huge number of open /proc/rtpengine/3/calls/74eb216b2a77a9ba5f43b2604ca41937 type of files..

The strange thing is that lsof also outputs the thread ID that is using the open files, and in the /tmp/lsof_out we see loads of threads that are not currently running ( from the output of ps -T -p [rtpengine-recording-pid-here] ).

[rfuchs]

And these open files are ... ??