Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Seeing Segmentation fault usign pyroscope-io python package for profiling a python app #164

Open
venu-shapley opened this issue Apr 25, 2024 · 7 comments
Labels
question Further information is requested

Comments

@venu-shapley
Copy link

We are using pyroscope-io package to profile a python application which is a fastAPI server.

During the initialization, I configure the pyroscope but I am seeing Segmentation faults and pointed to the following difference when the service starts up vs running into a segmentation fault.

image

Can you please help me understand what is going on?

@venu-shapley venu-shapley added the question Further information is requested label Apr 25, 2024
@korniltsev
Copy link
Collaborator

What is the pip package version? Does it crash without pyroscope?

Just to clarify, does it crash every time at the startup? or some times it does not crash?

Can you run the python app with env RUST_BACKTRACE=full ?

Another thing that could help in theory is full debug which you can enable like this

l = logging.getLogger()
l.setLevel(logging.DEBUG)

pyroscope.configure(
# ...
	enable_logging = True,
)

Can you attach lldb to a running python process like this lldb -p $(echo $YOUR_PYTHON_PID) --one-line continue and then when you get the Segmentation fault in the debugger you could run two commands bt and reg read - that could give some starting point for investigation. And then share the backtrace and regs dump.
Or if the crash happens too quickly before you can attach with the lldb debugger, you can start it from the debugger like this lldb /Users/korniltsev/.pyenv/versions/3.11.9/bin/python3 --one-line 'run your_script.py' and again once you get the sgementation fault do bt and reg read and share here.
You may need adjust the commands little for your app.

@venu-shapley
Copy link
Author

The package version we are using is pyroscope-io = "0.8.7".
It doesn't crash when I don't initialize the pyroscope profiler.
It doesn't crash everytime, it does it sometimes, but it is not rare to crash.
The crash is happening very early, even before the fast api server is ready to serve requests. I'll try to get the lldb debug info shortly.

@ThisIsANiceName
Copy link

I got the same issue with the 0.8.7 version.

As mentioned before it doesn't happen every time.

Here is the backtrace from gdb in case this helps. (I did not do more investigration into the issue, and downgraded to 0.8.5)
pyroscope_segfault

@vishal-wadhwa
Copy link

Facing similar issue
RUST backtrace
MacOS 14.5 (Sonoma) M1

 2024-06-03T08:09:47.973Z WARN  py_spy::sampler > Failed to open process 66517: Undefined error: 0 (os error 0)
 2024-06-03T08:09:48.012Z WARN  py_spy::sampler > Failed to create spy for 66517: Undefined error: 0 (os error 0)
 2024-06-03T08:09:48.117Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.223Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.324Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.425Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.526Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.628Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.733Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.838Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
 2024-06-03T08:09:48.942Z WARN  py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pretty_env_logger-0.4.0/src/lib.rs:74:22
stack backtrace:
   0:        0x17c8bb580 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h819e9cbdf1a9e730
   1:        0x17c78b548 - core::fmt::write::ha5e9bf3131ecb7c0
   2:        0x17c8a10ac - std::io::Write::write_fmt::h414ce9994bf17404
   3:        0x17c8be880 - std::sys_common::backtrace::print::h8072db0bbd5bcc3d
   4:        0x17c8be4e0 - std::panicking::default_hook::{{closure}}::h2c85c5b0c2ede151
   5:        0x17c8bf3cc - std::panicking::rust_panic_with_hook::h023af7f90b47eb8b
   6:        0x17c8bef7c - std::panicking::begin_panic_handler::{{closure}}::h14283519edc1d634
   7:        0x17c8beeec - std::sys_common::backtrace::__rust_end_short_backtrace::hc366c0b0cef5b747
   8:        0x17c8beee0 - _rust_begin_unwind
   9:        0x17c8f9128 - core::panicking::panic_fmt::h324f50b29db90195
  10:        0x17c8f9374 - core::result::unwrap_failed::hf783e6a14bbaf60b
  11:        0x17c77e380 - _initialize_logging
  12:        0x1a4981050 - <unknown>
  13:        0x1a4989ae0 - <unknown>
  14:        0x17c189478 - _cdata_call
  15:        0x102289020 - __PyObject_MakeTpCall
  16:        0x10235ebdc - _call_function
  17:        0x10235bed4 - __PyEval_EvalFrameDefault
  18:        0x10235f898 - __PyEval_EvalCodeWithName
  19:        0x102289ba4 - __PyFunction_Vectorcall
  20:        0x10235eb3c - _call_function
  21:        0x10235bfbc - __PyEval_EvalFrameDefault
  22:        0x102289a04 - _function_code_fastcall
  23:        0x10235eb3c - _call_function
  24:        0x10235bed4 - __PyEval_EvalFrameDefault
  25:        0x10235f898 - __PyEval_EvalCodeWithName
  26:        0x102356150 - _PyEval_EvalCode
  27:        0x102353420 - _builtin_exec
  28:        0x1022cbab4 - _cfunction_vectorcall_FASTCALL
  29:        0x102289384 - _PyVectorcall_Call
  30:        0x10235c26c - __PyEval_EvalFrameDefault
  31:        0x10235f898 - __PyEval_EvalCodeWithName
  32:        0x102289ba4 - __PyFunction_Vectorcall
  33:        0x10235eb3c - _call_function
  34:        0x10235bed4 - __PyEval_EvalFrameDefault
  35:        0x102289a04 - _function_code_fastcall
  36:        0x10235eb3c - _call_function
  37:        0x10235beb0 - __PyEval_EvalFrameDefault
  38:        0x102289a04 - _function_code_fastcall
  39:        0x10235eb3c - _call_function
  40:        0x10235bf48 - __PyEval_EvalFrameDefault
  41:        0x102289a04 - _function_code_fastcall
  42:        0x10235eb3c - _call_function
  43:        0x10235bf48 - __PyEval_EvalFrameDefault
  44:        0x102289a04 - _function_code_fastcall
  45:        0x10235eb3c - _call_function
  46:        0x10235bf48 - __PyEval_EvalFrameDefault
  47:        0x10235f898 - __PyEval_EvalCodeWithName
  48:        0x102289ba4 - __PyFunction_Vectorcall
  49:        0x10235eb3c - _call_function
  50:        0x10235bed4 - __PyEval_EvalFrameDefault
  51:        0x10235f898 - __PyEval_EvalCodeWithName
  52:        0x102289ba4 - __PyFunction_Vectorcall
  53:        0x10235eb3c - _call_function
  54:        0x10235bed4 - __PyEval_EvalFrameDefault
  55:        0x102289a04 - _function_code_fastcall
  56:        0x10235eb3c - _call_function
  57:        0x10235bf48 - __PyEval_EvalFrameDefault
  58:        0x102289a04 - _function_code_fastcall
  59:        0x10235eb3c - _call_function
  60:        0x10235beb0 - __PyEval_EvalFrameDefault
  61:        0x10229b8f4 - _gen_send_ex
  62:        0x102daf01c - _task_step
  63:        0x102289020 - __PyObject_MakeTpCall
  64:        0x102378394 - _context_run
  65:        0x1022cbbdc - _cfunction_vectorcall_FASTCALL_KEYWORDS
  66:        0x102289384 - _PyVectorcall_Call
  67:        0x10235c26c - __PyEval_EvalFrameDefault
  68:        0x102289a04 - _function_code_fastcall
  69:        0x10235eb3c - _call_function
  70:        0x10235beb0 - __PyEval_EvalFrameDefault
  71:        0x102289a04 - _function_code_fastcall
  72:        0x10235eb3c - _call_function
  73:        0x10235beb0 - __PyEval_EvalFrameDefault
  74:        0x102289a04 - _function_code_fastcall
  75:        0x10235eb3c - _call_function
  76:        0x10235beb0 - __PyEval_EvalFrameDefault
  77:        0x102289a04 - _function_code_fastcall
  78:        0x10235eb3c - _call_function
  79:        0x10235beb0 - __PyEval_EvalFrameDefault
  80:        0x10235f898 - __PyEval_EvalCodeWithName
  81:        0x102289ba4 - __PyFunction_Vectorcall
  82:        0x10235eb3c - _call_function
  83:        0x10235beb0 - __PyEval_EvalFrameDefault
  84:        0x10235f898 - __PyEval_EvalCodeWithName
  85:        0x102289ba4 - __PyFunction_Vectorcall
  86:        0x10235eb3c - _call_function
  87:        0x10235bfbc - __PyEval_EvalFrameDefault
  88:        0x10235f898 - __PyEval_EvalCodeWithName
  89:        0x102356150 - _PyEval_EvalCode
  90:        0x10239ca78 - _PyRun_SimpleFileExFlags
  91:        0x1023bb078 - _Py_RunMain
  92:        0x1023bb508 - _pymain_main
  93:        0x1023bb5a8 - _Py_BytesMain
fatal runtime error: failed to initiate panic, error 5
[1]    66516 abort      RUST_BACKTRACE=full python app_fastapi.py

@korniltsev
Copy link
Collaborator

korniltsev commented Jul 8, 2024

thread '' panicked at 'called Result::unwrap() on an Err value: SetLoggerError(())', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pretty_env_logger-0.4.0/src/lib.rs:74:22

This error is happening if pyroscope is attempted to initialize second time in the same process.

The pyroscope should definetely not unwrap and / or panic in this case, i will improve it.

In the mean time, users can just make sure to not call pyroscope.configure second time to avoid this issue

Also this is not segmentation fault

@korniltsev
Copy link
Collaborator

Here is the backtrace from gdb in case this helps. (I did not do more investigration into the issue, and downgraded to 0.8.5)

I don't see the segmentation fault. Please include register values, at least RIP / PC next time

@jacky-annalise
Copy link

jacky-annalise commented Jul 16, 2024

Overview

We're currently using wolfi-base, as lldb isn't available in Wolfi's APK, we had to replace it with gdb, I hope there is enough info to further debug the issue.

We also notice that it will sometimes cause the main process to get stuck on Failed to get interpreter from binary BSS, scanning libpython BSS when it doesn't segmentation fault.

I've attached details for 5x segmentation faults, 1x Pyroscope hanging, and 1x Successful run, let me know if I can add further information.

Questions

Version of SDK you are importing:

  • pyroscope-io = "0.8.7"
  • pyroscope-otel = "0.1.0"

Does the segmentation fault happen on startup or in the middle of app run randomly or happen at build time?

  • Happens during startup of the application.

what python version are you using?

  • 3.11

Segmentation Fault 1

Log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: File "/usr/lib/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /usr/lib/libstdc++.so.6.0.32-gdb.py
line to your configuration file "/home/nonroot/.config/gdb/gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/nonroot/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff4a3ff6c0 (LWP 205)]
[New Thread 0x7fff47bfe6c0 (LWP 206)]
[New Thread 0x7fff473fd6c0 (LWP 207)]
[New Thread 0x7fff42bfc6c0 (LWP 208)]
[New Thread 0x7fff403fb6c0 (LWP 209)]
[New Thread 0x7fff3dbfa6c0 (LWP 210)]
[New Thread 0x7fff3d3f96c0 (LWP 211)]
[New Thread 0x7fff38bf86c0 (LWP 212)]
[New Thread 0x7fff363f76c0 (LWP 213)]
[New Thread 0x7fff33bf66c0 (LWP 214)]
[New Thread 0x7fff313f56c0 (LWP 215)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff21bff6c0 (LWP 216)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff005ff6c0 (LWP 217)]
[New Thread 0x7ffeffdfe6c0 (LWP 218)]
[New Thread 0x7ffeff5fd6c0 (LWP 219)]
[New Thread 0x7ffefedfc6c0 (LWP 220)]
[New Thread 0x7ffefe5fb6c0 (LWP 221)]
[New Thread 0x7ffefddfa6c0 (LWP 222)]
[New Thread 0x7ffefd5f96c0 (LWP 223)]
[New Thread 0x7ffefcdf86c0 (LWP 224)]
[New Thread 0x7ffefc5f76c0 (LWP 225)]
[New Thread 0x7ffefbdf66c0 (LWP 226)]
[New Thread 0x7ffefb5f56c0 (LWP 227)]
[Detaching after vfork from child process 228]
[Detaching after vfork from child process 229]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[New Thread 0x7ffee800d6c0 (LWP 230)]
[Thread 0x7ffee800d6c0 (LWP 230) exited]
[New Thread 0x7ffee800d6c0 (LWP 231)]
[New Thread 0x7ffee77fc6c0 (LWP 232)]
[New Thread 0x7ffee6ffb6c0 (LWP 233)]
[New Thread 0x7ffee67fa6c0 (LWP 234)]
[New Thread 0x7ffee5ff96c0 (LWP 235)]
[New Thread 0x7ffee57f86c0 (LWP 236)]
[New Thread 0x7ffee4ff76c0 (LWP 237)]
[New Thread 0x7ffec7fff6c0 (LWP 238)]
[New Thread 0x7ffec77fe6c0 (LWP 239)]
[New Thread 0x7ffec6ffd6c0 (LWP 240)]
[New Thread 0x7ffec67fc6c0 (LWP 241)]
[New Thread 0x7ffec5ffb6c0 (LWP 242)]
[New Thread 0x7ffec57fa6c0 (LWP 243)]
[New Thread 0x7ffec4ff96c0 (LWP 244)]
[New Thread 0x7ffea7fff6c0 (LWP 245)]
[New Thread 0x7ffea77fe6c0 (LWP 246)]
[New Thread 0x7ffea6ffd6c0 (LWP 247)]
[New Thread 0x7ffee47f66c0 (LWP 248)]
[New Thread 0x7ffee45f56c0 (LWP 249)]
[New Thread 0x7ffee43f46c0 (LWP 250)]
 2024-07-16T02:19:45.102Z INFO  Pyroscope::Timer > Initializing Timer
[New Thread 0x7ffec47f86c0 (LWP 251)]
 2024-07-16T02:19:45.102Z INFO  Pyroscope::Session > Creating SessionManager
[New Thread 0x7ffec45f76c0 (LWP 252)]
[New Thread 0x7ffec43f66c0 (LWP 253)]
[New Thread 0x7ffea67fc6c0 (LWP 254)]
[New Thread 0x7ffea65fb6c0 (LWP 255)]
 2024-07-16T02:19:45.107Z INFO  py_spy::python_spy > Got virtual memory maps from pid 202:
 2024-07-16T02:19:45.108Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T02:19:45.114Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T02:19:45.116Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T02:19:45.116Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T02:19:45.116Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T02:19:45.116Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T02:19:45.116Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T02:19:45.116Z WARN  py_spy::python_spy > Interpreter address from _PyRuntime symbol is invalid 00007ffff7f62798
 2024-07-16T02:19:45.116Z INFO  py_spy::python_spy > Failed to get interp_head from symbols, scanning BSS section from main binary
 2024-07-16T02:19:45.116Z INFO  py_spy::python_spy > Failed to get interpreter from binary BSS, scanning libpython BSS

Thread 47 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec45f76c0 (LWP 252)]
0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so

Backtrace

#0  0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#1  0x00007ffee9160c4d in py_spy::python_spy::check_interpreter_addresses () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#2  0x00007ffee9162b38 in py_spy::python_spy::get_interpreter_address_from_binary () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#3  0x00007ffee915b49a in py_spy::python_spy::PythonSpy::new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#4  0x00007ffee9170d11 in py_spy::python_spy::PythonSpy::retry_new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#5  0x00007ffee91a5504 in std::sys_common::backtrace::__rust_begin_short_backtrace () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#6  0x00007ffee91a5317 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#7  0x00007ffee93b2c56 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#8  alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#9  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#10 0x00007ffff7915d68 in start_thread () from /lib/libc.so.6
#11 0x00007ffff7994ccc in clone3 () from /lib/libc.so.6

Registers

rax            0x200000001         8589934593
rbx            0x200000000         8589934592
rcx            0x1                 1
rdx            0xffffffffffffffff  -1
rsi            0x7ffe8017aaa8      140731047455400
rdi            0x7ffec45d96f0      140732192888560
rbp            0x2                 0x2
rsp            0x7ffec45d83f0      0x7ffec45d83f0
r8             0x2f990             194960
r9             0x0                 0
r10            0x7ffec45d83d8      140732192883672
r11            0x246               582
r12            0x7ffff79265b0      140737346954672
r13            0x7ffe801bca08      140731047725576
r14            0x7ffe801dca30      140731047856688
r15            0x7ffec45f5440      140732193002560
rip            0x7ffee913260c      0x7ffee913260c <core::ptr::drop_in_place<goblin::error::Error>+188>
eflags         0x10202             [ IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
fs_base        0x7ffec45f76c0      140732193011392
gs_base        0x0                 0

Segmentation Fault 2

Log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: File "/usr/lib/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff4a3ff6c0 (LWP 839)]
[New Thread 0x7fff47bfe6c0 (LWP 840)]
[New Thread 0x7fff453fd6c0 (LWP 841)]
[New Thread 0x7fff42bfc6c0 (LWP 842)]
[New Thread 0x7fff403fb6c0 (LWP 843)]
[New Thread 0x7fff3dbfa6c0 (LWP 844)]
[New Thread 0x7fff3b3f96c0 (LWP 845)]
[New Thread 0x7fff38bf86c0 (LWP 846)]
[New Thread 0x7fff363f76c0 (LWP 847)]
[New Thread 0x7fff33bf66c0 (LWP 848)]
[New Thread 0x7fff333f56c0 (LWP 849)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff21bff6c0 (LWP 850)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff005ff6c0 (LWP 851)]
[New Thread 0x7ffeffdfe6c0 (LWP 852)]
[New Thread 0x7ffeff5fd6c0 (LWP 853)]
[New Thread 0x7ffefedfc6c0 (LWP 854)]
[New Thread 0x7ffefe5fb6c0 (LWP 855)]
[New Thread 0x7ffefddfa6c0 (LWP 856)]
[New Thread 0x7ffefd5f96c0 (LWP 857)]
[New Thread 0x7ffefcdf86c0 (LWP 858)]
[New Thread 0x7ffefc5f76c0 (LWP 859)]
[New Thread 0x7ffefbdf66c0 (LWP 860)]
[New Thread 0x7ffefb5f56c0 (LWP 861)]
[Detaching after vfork from child process 862]
[Detaching after vfork from child process 863]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[New Thread 0x7ffee800d6c0 (LWP 864)]
[Thread 0x7ffee800d6c0 (LWP 864) exited]
[New Thread 0x7ffee800d6c0 (LWP 865)]
[New Thread 0x7ffee780c6c0 (LWP 866)]
[New Thread 0x7ffee6ffb6c0 (LWP 867)]
[New Thread 0x7ffee67fa6c0 (LWP 868)]
[New Thread 0x7ffee5ff96c0 (LWP 869)]
[New Thread 0x7ffee57f86c0 (LWP 870)]
[New Thread 0x7ffee4ff76c0 (LWP 871)]
[New Thread 0x7ffec7fff6c0 (LWP 872)]
[New Thread 0x7ffec77fe6c0 (LWP 873)]
[New Thread 0x7ffec6ffd6c0 (LWP 874)]
[New Thread 0x7ffec67fc6c0 (LWP 875)]
[New Thread 0x7ffec5ffb6c0 (LWP 876)]
[New Thread 0x7ffec57fa6c0 (LWP 877)]
[New Thread 0x7ffec4ff96c0 (LWP 878)]
[New Thread 0x7ffeaffff6c0 (LWP 879)]
[New Thread 0x7ffeaf7fe6c0 (LWP 880)]
[New Thread 0x7ffeaeffd6c0 (LWP 881)]
[New Thread 0x7ffee47f66c0 (LWP 882)]
[New Thread 0x7ffee45f56c0 (LWP 883)]
[New Thread 0x7ffee43f46c0 (LWP 884)]
 2024-07-16T02:34:25.527Z INFO  Pyroscope::Timer > Initializing Timer
[New Thread 0x7ffec45f76c0 (LWP 886)]
 2024-07-16T02:34:25.528Z INFO  Pyroscope::Session > Creating SessionManager
[New Thread 0x7ffec47f86c0 (LWP 885)]
[New Thread 0x7ffec43f66c0 (LWP 887)]
[New Thread 0x7ffeae7fc6c0 (LWP 888)]
[New Thread 0x7ffeae5fb6c0 (LWP 889)]
 2024-07-16T02:34:25.532Z INFO  py_spy::python_spy > Got virtual memory maps from pid 838:
 2024-07-16T02:34:25.533Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T02:34:25.539Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T02:34:25.540Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T02:34:25.540Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T02:34:25.540Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T02:34:25.540Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T02:34:25.540Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T02:34:25.540Z WARN  py_spy::python_spy > Interpreter address from _PyRuntime symbol is invalid 00007ffff7f62798
 2024-07-16T02:34:25.540Z INFO  py_spy::python_spy > Failed to get interp_head from symbols, scanning BSS section from main binary
 2024-07-16T02:34:25.540Z INFO  py_spy::python_spy > Failed to get interpreter from binary BSS, scanning libpython BSS

Thread 47 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec47f86c0 (LWP 885)]
0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so

Backtrace

#0  0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> ()
   from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#1  0x00007ffee9160c4d in py_spy::python_spy::check_interpreter_addresses ()
   from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#2  0x00007ffee9162b38 in py_spy::python_spy::get_interpreter_address_from_binary ()
   from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#3  0x00007ffee915b49a in py_spy::python_spy::PythonSpy::new ()
   from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#4  0x00007ffee9170d11 in py_spy::python_spy::PythonSpy::retry_new ()
   from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#5  0x00007ffee91a5504 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
   from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#6  0x00007ffee91a5317 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
   from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#7  0x00007ffee93b2c56 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#8  alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#9  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#10 0x00007ffff7915d68 in start_thread () from /lib/libc.so.6
#11 0x00007ffff7994ccc in clone3 () from /lib/libc.so.6

Registers

(gdb) info registers
rax            0x200000001         8589934593
rbx            0x200000000         8589934592
rcx            0x1                 1
rdx            0xffffffffffffffff  -1
rsi            0x7ffe8817aaa8      140731181673128
rdi            0x7ffec47da6f0      140732194989808
rbp            0x2                 0x2
rsp            0x7ffec47d93f0      0x7ffec47d93f0
r8             0x2f990             194960
r9             0x0                 0
r10            0x7ffec47d93d8      140732194984920
r11            0x246               582
r12            0x7ffff79265b0      140737346954672
r13            0x7ffe881bca08      140731181943304
r14            0x7ffe881dca30      140731182074416
r15            0x7ffec47f6440      140732195103808
rip            0x7ffee913260c      0x7ffee913260c <core::ptr::drop_in_place<goblin::error::Error>+188>
eflags         0x10202             [ IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
fs_base        0x7ffec47f86c0      140732195112640
gs_base        0x0                 0

Segmentation Fault 3

Log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: File "/usr/lib/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff4a3ff6c0 (LWP 891)]
[New Thread 0x7fff47bfe6c0 (LWP 892)]
[New Thread 0x7fff453fd6c0 (LWP 893)]
[New Thread 0x7fff42bfc6c0 (LWP 894)]
[New Thread 0x7fff403fb6c0 (LWP 895)]
[New Thread 0x7fff3dbfa6c0 (LWP 896)]
[New Thread 0x7fff3b3f96c0 (LWP 897)]
[New Thread 0x7fff38bf86c0 (LWP 898)]
[New Thread 0x7fff363f76c0 (LWP 899)]
[New Thread 0x7fff33bf66c0 (LWP 900)]
[New Thread 0x7fff333f56c0 (LWP 901)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff21bff6c0 (LWP 902)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff005ff6c0 (LWP 903)]
[New Thread 0x7ffeffdfe6c0 (LWP 904)]
[New Thread 0x7ffeff5fd6c0 (LWP 905)]
[New Thread 0x7ffefedfc6c0 (LWP 906)]
[New Thread 0x7ffefe5fb6c0 (LWP 907)]
[New Thread 0x7ffefddfa6c0 (LWP 908)]
[New Thread 0x7ffefd5f96c0 (LWP 909)]
[New Thread 0x7ffefcdf86c0 (LWP 910)]
[New Thread 0x7ffefc5f76c0 (LWP 911)]
[New Thread 0x7ffefbdf66c0 (LWP 912)]
[New Thread 0x7ffefb5f56c0 (LWP 913)]
[Detaching after vfork from child process 914]
[Detaching after vfork from child process 915]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[New Thread 0x7ffee800d6c0 (LWP 916)]
[Thread 0x7ffee800d6c0 (LWP 916) exited]
[New Thread 0x7ffee800d6c0 (LWP 917)]
[New Thread 0x7ffee780c6c0 (LWP 918)]
[New Thread 0x7ffee6ffb6c0 (LWP 919)]
[New Thread 0x7ffee67fa6c0 (LWP 920)]
[New Thread 0x7ffee5ff96c0 (LWP 921)]
[New Thread 0x7ffee57f86c0 (LWP 922)]
[New Thread 0x7ffee4ff76c0 (LWP 923)]
[New Thread 0x7ffec7fff6c0 (LWP 924)]
[New Thread 0x7ffec77fe6c0 (LWP 925)]
[New Thread 0x7ffec6ffd6c0 (LWP 926)]
[New Thread 0x7ffec67fc6c0 (LWP 927)]
[New Thread 0x7ffec5ffb6c0 (LWP 928)]
[New Thread 0x7ffec57fa6c0 (LWP 929)]
[New Thread 0x7ffec4ff96c0 (LWP 930)]
[New Thread 0x7ffea7fff6c0 (LWP 931)]
[New Thread 0x7ffea77fe6c0 (LWP 932)]
[New Thread 0x7ffea6ffd6c0 (LWP 933)]
[New Thread 0x7ffee47f66c0 (LWP 934)]
[New Thread 0x7ffee45f56c0 (LWP 935)]
[New Thread 0x7ffee43f46c0 (LWP 936)]
 2024-07-16T03:30:39.877Z INFO  Pyroscope::Timer > Initializing Timer
[New Thread 0x7ffec47f86c0 (LWP 937)]
[New Thread 0x7ffec45f76c0 (LWP 938)]
 2024-07-16T03:30:39.878Z INFO  Pyroscope::Session > Creating SessionManager
[New Thread 0x7ffec43f66c0 (LWP 939)]
[New Thread 0x7ffea67fc6c0 (LWP 940)]
[New Thread 0x7ffea65fb6c0 (LWP 941)]
 2024-07-16T03:30:39.886Z INFO  py_spy::python_spy > Got virtual memory maps from pid 890:
 2024-07-16T03:30:39.887Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T03:30:39.898Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T03:30:39.902Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T03:30:39.902Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T03:30:39.902Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T03:30:39.902Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T03:30:39.902Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:30:39.902Z WARN  py_spy::python_spy > Interpreter address from _PyRuntime symbol is invalid 00007ffff7f62798
 2024-07-16T03:30:39.902Z INFO  py_spy::python_spy > Failed to get interp_head from symbols, scanning BSS section from main binary
 2024-07-16T03:30:39.903Z INFO  py_spy::python_spy > Failed to get interpreter from binary BSS, scanning libpython BSS
INFO:     Will watch for changes in these directories: ['/app']
2024-07-16 03:30:39,891 INFO [uvicorn.error] [config.py:359] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024-07-16 03:30:39,907 INFO [uvicorn.error] [config.py:585] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [890] using StatReload
2024-07-16 03:30:39,907 INFO [uvicorn.error] [basereload.py:72] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Started reloader process [890] using StatReload

Thread 46 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec47f86c0 (LWP 937)]
0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so

Backtrace

#0  0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#1  0x00007ffee9160c4d in py_spy::python_spy::check_interpreter_addresses () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#2  0x00007ffee9162b38 in py_spy::python_spy::get_interpreter_address_from_binary () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#3  0x00007ffee915b49a in py_spy::python_spy::PythonSpy::new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#4  0x00007ffee9170d11 in py_spy::python_spy::PythonSpy::retry_new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#5  0x00007ffee91a5504 in std::sys_common::backtrace::__rust_begin_short_backtrace () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#6  0x00007ffee91a5317 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#7  0x00007ffee93b2c56 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#8  alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#9  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#10 0x00007ffff7915d68 in start_thread () from /lib/libc.so.6
#11 0x00007ffff7994ccc in clone3 () from /lib/libc.so.6

Registers

rax            0x200000001         8589934593
rbx            0x200000000         8589934592
rcx            0x1                 1
rdx            0xffffffffffffffff  -1
rsi            0x7ffe8c17abe8      140731248782312
rdi            0x7ffec47da6f0      140732194989808
rbp            0x2                 0x2
rsp            0x7ffec47d93f0      0x7ffec47d93f0
r8             0x2f990             194960
r9             0x0                 0
r10            0x7ffec47d93d8      140732194984920
r11            0x246               582
r12            0x7ffff79265b0      140737346954672
r13            0x7ffe8c1bcb48      140731249052488
r14            0x7ffe8c1dcb70      140731249183600
r15            0x7ffec47f6440      140732195103808
rip            0x7ffee913260c      0x7ffee913260c <core::ptr::drop_in_place<goblin::error::Error>+188>
eflags         0x10202             [ IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
fs_base        0x7ffec47f86c0      140732195112640
gs_base        0x0                 0

Segmentation Fault 4

Log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: File "/usr/lib/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff4a3ff6c0 (LWP 943)]
[New Thread 0x7fff49bfe6c0 (LWP 944)]
[New Thread 0x7fff453fd6c0 (LWP 945)]
[New Thread 0x7fff42bfc6c0 (LWP 946)]
[New Thread 0x7fff403fb6c0 (LWP 947)]
[New Thread 0x7fff3dbfa6c0 (LWP 948)]
[New Thread 0x7fff3b3f96c0 (LWP 949)]
[New Thread 0x7fff38bf86c0 (LWP 950)]
[New Thread 0x7fff383f76c0 (LWP 951)]
[New Thread 0x7fff33bf66c0 (LWP 952)]
[New Thread 0x7fff313f56c0 (LWP 953)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff21bff6c0 (LWP 954)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff005ff6c0 (LWP 955)]
[New Thread 0x7ffeffdfe6c0 (LWP 956)]
[New Thread 0x7ffeff5fd6c0 (LWP 957)]
[New Thread 0x7ffefedfc6c0 (LWP 958)]
[New Thread 0x7ffefe5fb6c0 (LWP 959)]
[New Thread 0x7ffefddfa6c0 (LWP 960)]
[New Thread 0x7ffefd5f96c0 (LWP 961)]
[New Thread 0x7ffefcdf86c0 (LWP 962)]
[New Thread 0x7ffefc5f76c0 (LWP 963)]
[New Thread 0x7ffefbdf66c0 (LWP 964)]
[New Thread 0x7ffefb5f56c0 (LWP 965)]
[Detaching after vfork from child process 966]
[Detaching after vfork from child process 967]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[New Thread 0x7ffee800d6c0 (LWP 968)]
[Thread 0x7ffee800d6c0 (LWP 968) exited]
[New Thread 0x7ffee800d6c0 (LWP 969)]
[New Thread 0x7ffee77fc6c0 (LWP 970)]
[New Thread 0x7ffee6ffb6c0 (LWP 971)]
[New Thread 0x7ffee67fa6c0 (LWP 972)]
[New Thread 0x7ffee5ff96c0 (LWP 973)]
[New Thread 0x7ffee57f86c0 (LWP 974)]
[New Thread 0x7ffee4ff76c0 (LWP 975)]
[New Thread 0x7ffec7fff6c0 (LWP 976)]
[New Thread 0x7ffec77fe6c0 (LWP 977)]
[New Thread 0x7ffec6ffd6c0 (LWP 978)]
[New Thread 0x7ffec67fc6c0 (LWP 979)]
[New Thread 0x7ffec5ffb6c0 (LWP 980)]
[New Thread 0x7ffec57fa6c0 (LWP 981)]
[New Thread 0x7ffec4ff96c0 (LWP 982)]
[New Thread 0x7ffea7fff6c0 (LWP 983)]
[New Thread 0x7ffea77fe6c0 (LWP 984)]
[New Thread 0x7ffea6ffd6c0 (LWP 985)]
[New Thread 0x7ffee47f66c0 (LWP 986)]
[New Thread 0x7ffee45f56c0 (LWP 987)]
[New Thread 0x7ffee43f46c0 (LWP 988)]
 2024-07-16T03:40:43.152Z INFO  Pyroscope::Timer > Initializing Timer
[New Thread 0x7ffec47f86c0 (LWP 989)]
[New Thread 0x7ffec45f76c0 (LWP 990)]
 2024-07-16T03:40:43.153Z INFO  Pyroscope::Session > Creating SessionManager
[New Thread 0x7ffec43f66c0 (LWP 991)]
[New Thread 0x7ffea67fc6c0 (LWP 992)]
[New Thread 0x7ffea65fb6c0 (LWP 993)]
 2024-07-16T03:40:43.157Z INFO  py_spy::python_spy > Got virtual memory maps from pid 942:
 2024-07-16T03:40:43.158Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T03:40:43.164Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T03:40:43.165Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T03:40:43.165Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T03:40:43.165Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T03:40:43.165Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T03:40:43.165Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:40:43.165Z WARN  py_spy::python_spy > Interpreter address from _PyRuntime symbol is invalid 00007ffff7f62798
 2024-07-16T03:40:43.165Z INFO  py_spy::python_spy > Failed to get interp_head from symbols, scanning BSS section from main binary
 2024-07-16T03:40:43.165Z INFO  py_spy::python_spy > Failed to get interpreter from binary BSS, scanning libpython BSS

Thread 46 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec47f86c0 (LWP 989)]
0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so

Backtrace

#0  0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#1  0x00007ffee9160c4d in py_spy::python_spy::check_interpreter_addresses () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#2  0x00007ffee9162b38 in py_spy::python_spy::get_interpreter_address_from_binary () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#3  0x00007ffee915b49a in py_spy::python_spy::PythonSpy::new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#4  0x00007ffee9170d11 in py_spy::python_spy::PythonSpy::retry_new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#5  0x00007ffee91a5504 in std::sys_common::backtrace::__rust_begin_short_backtrace () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#6  0x00007ffee91a5317 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#7  0x00007ffee93b2c56 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#8  alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#9  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#10 0x00007ffff7915d68 in start_thread () from /lib/libc.so.6
#11 0x00007ffff7994ccc in clone3 () from /lib/libc.so.6

Registers

rax            0x200000001         8589934593
rbx            0x200000000         8589934592
rcx            0x1                 1
rdx            0xffffffffffffffff  -1
rsi            0x7ffe8c17abe8      140731248782312
rdi            0x7ffec47da6f0      140732194989808
rbp            0x2                 0x2
rsp            0x7ffec47d93f0      0x7ffec47d93f0
r8             0x2f990             194960
r9             0x0                 0
r10            0x7ffec47d93d8      140732194984920
r11            0x246               582
r12            0x7ffff79265b0      140737346954672
r13            0x7ffe8c1bcb48      140731249052488
r14            0x7ffe8c1dcb70      140731249183600
r15            0x7ffec47f6440      140732195103808
rip            0x7ffee913260c      0x7ffee913260c <core::ptr::drop_in_place<goblin::error::Error>+188>
eflags         0x10202             [ IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
fs_base        0x7ffec47f86c0      140732195112640
gs_base        0x0                 0

Segmentation Fault 5

Log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: File "/usr/lib/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff4a3ff6c0 (LWP 1263)]
[New Thread 0x7fff47bfe6c0 (LWP 1264)]
[New Thread 0x7fff453fd6c0 (LWP 1265)]
[New Thread 0x7fff44bfc6c0 (LWP 1266)]
[New Thread 0x7fff423fb6c0 (LWP 1267)]
[New Thread 0x7fff3dbfa6c0 (LWP 1268)]
[New Thread 0x7fff3b3f96c0 (LWP 1269)]
[New Thread 0x7fff38bf86c0 (LWP 1270)]
[New Thread 0x7fff383f76c0 (LWP 1271)]
[New Thread 0x7fff35bf66c0 (LWP 1272)]
[New Thread 0x7fff333f56c0 (LWP 1273)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff21bff6c0 (LWP 1274)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff005ff6c0 (LWP 1275)]
[New Thread 0x7ffeffdfe6c0 (LWP 1276)]
[New Thread 0x7ffeff5fd6c0 (LWP 1277)]
[New Thread 0x7ffefedfc6c0 (LWP 1278)]
[New Thread 0x7ffefe5fb6c0 (LWP 1279)]
[New Thread 0x7ffefddfa6c0 (LWP 1280)]
[New Thread 0x7ffefd5f96c0 (LWP 1281)]
[New Thread 0x7ffefcdf86c0 (LWP 1282)]
[New Thread 0x7ffefc5f76c0 (LWP 1283)]
[New Thread 0x7ffefbdf66c0 (LWP 1284)]
[New Thread 0x7ffefb5f56c0 (LWP 1285)]
[Detaching after vfork from child process 1286]
[Detaching after vfork from child process 1287]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[New Thread 0x7ffee800d6c0 (LWP 1288)]
[Thread 0x7ffee800d6c0 (LWP 1288) exited]
[New Thread 0x7ffee800d6c0 (LWP 1289)]
[New Thread 0x7ffee77fc6c0 (LWP 1290)]
[New Thread 0x7ffee6ffb6c0 (LWP 1291)]
[New Thread 0x7ffee67fa6c0 (LWP 1292)]
[New Thread 0x7ffee5ff96c0 (LWP 1293)]
[New Thread 0x7ffee57f86c0 (LWP 1294)]
[New Thread 0x7ffee4ff76c0 (LWP 1295)]
[New Thread 0x7ffec7fff6c0 (LWP 1296)]
[New Thread 0x7ffec77fe6c0 (LWP 1297)]
[New Thread 0x7ffec6ffd6c0 (LWP 1298)]
[New Thread 0x7ffec67fc6c0 (LWP 1299)]
[New Thread 0x7ffec5ffb6c0 (LWP 1300)]
[New Thread 0x7ffec57fa6c0 (LWP 1301)]
[New Thread 0x7ffec4ff96c0 (LWP 1302)]
[New Thread 0x7ffea7fff6c0 (LWP 1303)]
[New Thread 0x7ffea77fe6c0 (LWP 1304)]
[New Thread 0x7ffea6ffd6c0 (LWP 1305)]
[New Thread 0x7ffee47f66c0 (LWP 1306)]
[New Thread 0x7ffee45f56c0 (LWP 1307)]
[New Thread 0x7ffee43f46c0 (LWP 1308)]
 2024-07-16T03:48:28.870Z INFO  Pyroscope::Timer > Initializing Timer
[New Thread 0x7ffec47f86c0 (LWP 1309)]
[New Thread 0x7ffec45f76c0 (LWP 1310)]
 2024-07-16T03:48:28.870Z INFO  Pyroscope::Session > Creating SessionManager
[New Thread 0x7ffec43f66c0 (LWP 1311)]
[New Thread 0x7ffea67fc6c0 (LWP 1312)]
[New Thread 0x7ffea65fb6c0 (LWP 1313)]
 2024-07-16T03:48:28.875Z INFO  py_spy::python_spy > Got virtual memory maps from pid 1262:
 2024-07-16T03:48:28.875Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T03:48:28.881Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T03:48:28.882Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T03:48:28.882Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T03:48:28.882Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T03:48:28.882Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T03:48:28.882Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:48:28.882Z WARN  py_spy::python_spy > Interpreter address from _PyRuntime symbol is invalid 00007ffff7f62798
 2024-07-16T03:48:28.882Z INFO  py_spy::python_spy > Failed to get interp_head from symbols, scanning BSS section from main binary
 2024-07-16T03:48:28.882Z INFO  py_spy::python_spy > Failed to get interpreter from binary BSS, scanning libpython BSS

Thread 46 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec47f86c0 (LWP 1309)]
0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so

Backtrace

#0  0x00007ffee913260c in core::ptr::drop_in_place<goblin::error::Error> () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#1  0x00007ffee9160c4d in py_spy::python_spy::check_interpreter_addresses () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#2  0x00007ffee9162b38 in py_spy::python_spy::get_interpreter_address_from_binary () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#3  0x00007ffee915b49a in py_spy::python_spy::PythonSpy::new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#4  0x00007ffee9170d11 in py_spy::python_spy::PythonSpy::retry_new () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#5  0x00007ffee91a5504 in std::sys_common::backtrace::__rust_begin_short_backtrace () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#6  0x00007ffee91a5317 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-37m-x86_64-linux-gnu.so
#7  0x00007ffee93b2c56 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#8  alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2015
#9  std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#10 0x00007ffff7915d68 in start_thread () from /lib/libc.so.6
#11 0x00007ffff7994ccc in clone3 () from /lib/libc.so.6

Registers

rax            0x200000001         8589934593
rbx            0x200000000         8589934592
rcx            0x1                 1
rdx            0xffffffffffffffff  -1
rsi            0x7ffe8c17abe8      140731248782312
rdi            0x7ffec47da6f0      140732194989808
rbp            0x2                 0x2
rsp            0x7ffec47d93f0      0x7ffec47d93f0
r8             0x2f990             194960
r9             0x0                 0
r10            0x7ffec47d93d8      140732194984920
r11            0x246               582
r12            0x7ffff79265b0      140737346954672
r13            0x7ffe8c1bcb48      140731249052488
r14            0x7ffe8c1dcb70      140731249183600
r15            0x7ffec47f6440      140732195103808
rip            0x7ffee913260c      0x7ffee913260c <core::ptr::drop_in_place<goblin::error::Error>+188>
eflags         0x10202             [ IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
fs_base        0x7ffec47f86c0      140732195112640
gs_base        0x0                 0

Pyroscope Hang

Pyroscope hangs the process, getting stuck at Failed to get interpreter from binary BSS, scanning libpython BSS.

Log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: File "/usr/lib/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff4a3ff6c0 (LWP 995)]
[New Thread 0x7fff47bfe6c0 (LWP 996)]
[New Thread 0x7fff453fd6c0 (LWP 997)]
[New Thread 0x7fff42bfc6c0 (LWP 998)]
[New Thread 0x7fff403fb6c0 (LWP 999)]
[New Thread 0x7fff3dbfa6c0 (LWP 1000)]
[New Thread 0x7fff3b3f96c0 (LWP 1001)]
[New Thread 0x7fff38bf86c0 (LWP 1002)]
[New Thread 0x7fff363f76c0 (LWP 1003)]
[New Thread 0x7fff33bf66c0 (LWP 1004)]
[New Thread 0x7fff333f56c0 (LWP 1005)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff21bff6c0 (LWP 1006)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff005ff6c0 (LWP 1007)]
[New Thread 0x7ffeffdfe6c0 (LWP 1008)]
[New Thread 0x7ffeff5fd6c0 (LWP 1009)]
[New Thread 0x7ffefedfc6c0 (LWP 1010)]
[New Thread 0x7ffefe5fb6c0 (LWP 1011)]
[New Thread 0x7ffefddfa6c0 (LWP 1012)]
[New Thread 0x7ffefd5f96c0 (LWP 1013)]
[New Thread 0x7ffefcdf86c0 (LWP 1014)]
[New Thread 0x7ffefc5f76c0 (LWP 1015)]
[New Thread 0x7ffefbdf66c0 (LWP 1016)]
[New Thread 0x7ffefb5f56c0 (LWP 1017)]
[Detaching after vfork from child process 1018]
[Detaching after vfork from child process 1019]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[New Thread 0x7ffee800d6c0 (LWP 1020)]
[Thread 0x7ffee800d6c0 (LWP 1020) exited]
[New Thread 0x7ffee800d6c0 (LWP 1021)]
[New Thread 0x7ffee77fc6c0 (LWP 1022)]
[New Thread 0x7ffee6ffb6c0 (LWP 1023)]
[New Thread 0x7ffee67fa6c0 (LWP 1024)]
[New Thread 0x7ffee5ff96c0 (LWP 1025)]
[New Thread 0x7ffee57f86c0 (LWP 1026)]
[New Thread 0x7ffee4ff76c0 (LWP 1027)]
[New Thread 0x7ffecffff6c0 (LWP 1028)]
[New Thread 0x7ffecf7fe6c0 (LWP 1029)]
[New Thread 0x7ffeceffd6c0 (LWP 1030)]
[New Thread 0x7ffece7fc6c0 (LWP 1031)]
[New Thread 0x7ffecdffb6c0 (LWP 1032)]
[New Thread 0x7ffecd7fa6c0 (LWP 1033)]
[New Thread 0x7ffeccff96c0 (LWP 1034)]
[New Thread 0x7ffeabfff6c0 (LWP 1035)]
[New Thread 0x7ffeab7fe6c0 (LWP 1036)]
[New Thread 0x7ffeaaffd6c0 (LWP 1037)]
[New Thread 0x7ffee47f66c0 (LWP 1038)]
[New Thread 0x7ffee45f56c0 (LWP 1039)]
[New Thread 0x7ffee43f46c0 (LWP 1040)]
 2024-07-16T03:43:23.816Z INFO  Pyroscope::Timer > Initializing Timer
[New Thread 0x7ffecc7f86c0 (LWP 1041)]
[New Thread 0x7ffecc5f76c0 (LWP 1042)]
 2024-07-16T03:43:23.816Z INFO  Pyroscope::Session > Creating SessionManager
[New Thread 0x7ffecc3f66c0 (LWP 1043)]
[New Thread 0x7ffeaa7fc6c0 (LWP 1044)]
[New Thread 0x7ffeaa5fb6c0 (LWP 1045)]
 2024-07-16T03:43:23.821Z INFO  py_spy::python_spy > Got virtual memory maps from pid 994:
 2024-07-16T03:43:23.822Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T03:43:23.828Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T03:43:23.829Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T03:43:23.829Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T03:43:23.829Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T03:43:23.829Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T03:43:23.829Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:43:23.829Z INFO  py_spy::python_spy > Found interpreter at 0x00007ffff7f62798
 2024-07-16T03:43:23.829Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:43:23.829Z INFO  py_spy::python_spy > Found _PyRuntime @ 0x00007ffff7f54160, getting gilstate.tstate_current from offset 0x240
INFO:     Will watch for changes in these directories: ['/app']
2024-07-16 03:43:23,826 INFO [uvicorn.error] [config.py:359] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024-07-16 03:43:23,841 INFO [uvicorn.error] [config.py:585] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [994] using StatReload
2024-07-16 03:43:23,841 INFO [uvicorn.error] [basereload.py:72] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Started reloader process [994] using StatReload
[Detaching after vfork from child process 1046]
[Detaching after vfork from child process 1047]
[New Thread 0x7ffeaa3fa6c0 (LWP 1073)]
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
 2024-07-16T03:43:29.673Z INFO  Pyroscope::Timer > Initializing Timer
 2024-07-16T03:43:29.673Z INFO  Pyroscope::Session > Creating SessionManager
 2024-07-16T03:43:29.678Z INFO  py_spy::python_spy > Got virtual memory maps from pid 1047:
 2024-07-16T03:43:29.678Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T03:43:29.684Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T03:43:29.685Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T03:43:29.685Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T03:43:29.685Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T03:43:29.685Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T03:43:29.685Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:43:29.685Z WARN  py_spy::python_spy > Interpreter address from _PyRuntime symbol is invalid 00007ffff7f62798
 2024-07-16T03:43:29.685Z INFO  py_spy::python_spy > Failed to get interp_head from symbols, scanning BSS section from main binary
 2024-07-16T03:43:29.685Z INFO  py_spy::python_spy > Failed to get interpreter from binary BSS, scanning libpython BSS
 2024-07-16T03:43:30.000Z INFO  Pyroscope::Session > Creating Session
 2024-07-16T03:43:30.001Z INFO  Pyroscope::Session > Sending Session: 1721101400 - 1721101410
[New Thread 0x7ffea9bf96c0 (LWP 1100)]
[New Thread 0x7ffea99f86c0 (LWP 1101)]
[Thread 0x7ffea99f86c0 (LWP 1101) exited]
 2024-07-16T03:43:33.804Z ERROR pyroscope::session > SessionManager - Failed to send session: error
...

Successful Run

Application successfully running (Please ignore the Error loading ASGI app).

Log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
warning: File "/usr/lib/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff4a3ff6c0 (LWP 1117)]
[New Thread 0x7fff49bfe6c0 (LWP 1118)]
[New Thread 0x7fff453fd6c0 (LWP 1119)]
[New Thread 0x7fff44bfc6c0 (LWP 1120)]
[New Thread 0x7fff403fb6c0 (LWP 1121)]
[New Thread 0x7fff3dbfa6c0 (LWP 1122)]
[New Thread 0x7fff3b3f96c0 (LWP 1123)]
[New Thread 0x7fff3abf86c0 (LWP 1124)]
[New Thread 0x7fff383f76c0 (LWP 1125)]
[New Thread 0x7fff33bf66c0 (LWP 1126)]
[New Thread 0x7fff313f56c0 (LWP 1127)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff21bff6c0 (LWP 1128)]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
[New Thread 0x7fff005ff6c0 (LWP 1129)]
[New Thread 0x7ffeffdfe6c0 (LWP 1130)]
[New Thread 0x7ffeff5fd6c0 (LWP 1131)]
[New Thread 0x7ffefedfc6c0 (LWP 1132)]
[New Thread 0x7ffefe5fb6c0 (LWP 1133)]
[New Thread 0x7ffefddfa6c0 (LWP 1134)]
[New Thread 0x7ffefd5f96c0 (LWP 1135)]
[New Thread 0x7ffefcdf86c0 (LWP 1136)]
[New Thread 0x7ffefc5f76c0 (LWP 1137)]
[New Thread 0x7ffefbdf66c0 (LWP 1138)]
[New Thread 0x7ffefb5f56c0 (LWP 1139)]
[Detaching after vfork from child process 1140]
[Detaching after vfork from child process 1141]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[New Thread 0x7ffee800d6c0 (LWP 1142)]
[Thread 0x7ffee800d6c0 (LWP 1142) exited]
[New Thread 0x7ffee800d6c0 (LWP 1143)]
[New Thread 0x7ffee77fc6c0 (LWP 1144)]
[New Thread 0x7ffee6ffb6c0 (LWP 1145)]
[New Thread 0x7ffee67fa6c0 (LWP 1146)]
[New Thread 0x7ffee5ff96c0 (LWP 1147)]
[New Thread 0x7ffee57f86c0 (LWP 1148)]
[New Thread 0x7ffee4ff76c0 (LWP 1149)]
[New Thread 0x7ffec7fff6c0 (LWP 1150)]
[New Thread 0x7ffec77fe6c0 (LWP 1151)]
[New Thread 0x7ffec6ffd6c0 (LWP 1152)]
[New Thread 0x7ffec67fc6c0 (LWP 1153)]
[New Thread 0x7ffec5ffb6c0 (LWP 1154)]
[New Thread 0x7ffec57fa6c0 (LWP 1155)]
[New Thread 0x7ffec4ff96c0 (LWP 1156)]
[New Thread 0x7ffea7fff6c0 (LWP 1157)]
[New Thread 0x7ffea77fe6c0 (LWP 1158)]
[New Thread 0x7ffea6ffd6c0 (LWP 1159)]
[New Thread 0x7ffee47f66c0 (LWP 1160)]
[New Thread 0x7ffee45f56c0 (LWP 1161)]
[New Thread 0x7ffee43f46c0 (LWP 1162)]
 2024-07-16T03:44:56.800Z INFO  Pyroscope::Timer > Initializing Timer
[New Thread 0x7ffec47f86c0 (LWP 1163)]
 2024-07-16T03:44:56.800Z INFO  Pyroscope::Session > Creating SessionManager
[New Thread 0x7ffec45f76c0 (LWP 1164)]
[New Thread 0x7ffec43f66c0 (LWP 1165)]
[New Thread 0x7ffea67fc6c0 (LWP 1166)]
[New Thread 0x7ffea65fb6c0 (LWP 1167)]
 2024-07-16T03:44:56.806Z INFO  py_spy::python_spy > Got virtual memory maps from pid 1116:
 2024-07-16T03:44:56.806Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T03:44:56.814Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T03:44:56.815Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T03:44:56.815Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T03:44:56.815Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T03:44:56.815Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T03:44:56.815Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:44:56.816Z INFO  py_spy::python_spy > Found interpreter at 0x00007ffff7f62798
 2024-07-16T03:44:56.816Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:44:56.816Z INFO  py_spy::python_spy > Found _PyRuntime @ 0x00007ffff7f54160, getting gilstate.tstate_current from offset 0x240
INFO:     Will watch for changes in these directories: ['/app']
2024-07-16 03:44:56,810 INFO [uvicorn.error] [config.py:359] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024-07-16 03:44:56,827 INFO [uvicorn.error] [config.py:585] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [1116] using StatReload
2024-07-16 03:44:56,827 INFO [uvicorn.error] [basereload.py:72] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Started reloader process [1116] using StatReload
[Detaching after vfork from child process 1168]
[Detaching after vfork from child process 1169]
 2024-07-16T03:45:00.000Z INFO  Pyroscope::Session > Creating Session
 2024-07-16T03:45:00.000Z INFO  Pyroscope::Session > Sending Session: 1721101490 - 1721101500
[New Thread 0x7ffea63fa6c0 (LWP 1182)]
[New Thread 0x7ffea61f96c0 (LWP 1183)]
[New Thread 0x7ffea5ff86c0 (LWP 1197)]
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_versions" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/venv/lib/python3.11/site-packages/pydantic/_internal/_fields.py:161: UserWarning: Field "model_version" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
 2024-07-16T03:45:02.363Z INFO  Pyroscope::Timer > Initializing Timer
 2024-07-16T03:45:02.363Z INFO  Pyroscope::Session > Creating SessionManager
 2024-07-16T03:45:02.367Z INFO  py_spy::python_spy > Got virtual memory maps from pid 1169:
 2024-07-16T03:45:02.368Z INFO  py_spy::python_spy > Found libpython binary @ /usr/lib/libpython3.11.so.1.0
 2024-07-16T03:45:02.374Z INFO  py_spy::python_spy > Getting version from python binary BSS
 2024-07-16T03:45:02.375Z INFO  py_spy::python_spy > Failed to get version from BSS section: failed to find version string
 2024-07-16T03:45:02.375Z INFO  py_spy::python_spy > Getting version from libpython BSS
 2024-07-16T03:45:02.375Z INFO  py_spy::version    > Found matching version string '3.11.9 (tags/v3.11.9-0-gde54cf5-dirty:de54cf5, Jun 26 2024, 12:07:40) ['
 2024-07-16T03:45:02.375Z INFO  py_spy::python_spy > python version 3.11.9 detected
 2024-07-16T03:45:02.375Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:45:02.375Z INFO  py_spy::python_spy > Found interpreter at 0x00007ffff7f62798
 2024-07-16T03:45:02.375Z INFO  py_spy::python_spy > got symbol _PyRuntime (0x00007ffff7f54160) from libpython binary
 2024-07-16T03:45:02.375Z INFO  py_spy::python_spy > Found _PyRuntime @ 0x00007ffff7f54160, getting gilstate.tstate_current from offset 0x240
ERROR:    Error loading ASGI app. Could not import module "main".
2024-07-16 03:45:02,378 ERROR [uvicorn.error] [config.py:481] [trace_id=0 span_id=0 resource.service.name=ctc_api trace_sampled=False] - Error loading ASGI app. Could not import module "main".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

5 participants