Segfaults in createTiledPlans

[cgbriggs99]

I've been getting some weird segfaults when I try to run the example test. Here's a valgrind run that shows it.

==1423119== Memcheck, a memory error detector
==1423119== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==1423119== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==1423119== Command: tests/example
==1423119== 
==1423119== Warning: set address range perms: large range [0x59c92000, 0x159e93000) (noaccess)
==1423119== Warning: set address range perms: large range [0x159e00000, 0x218c01000) (noaccess)
==1423119== Warning: set address range perms: large range [0x15a000000, 0x218c00000) (defined)
==1423119== Warning: set address range perms: large range [0x218c00000, 0x2d7a01000) (noaccess)
==1423119== Warning: set address range perms: large range [0x218e00000, 0x2d7a00000) (defined)

4-byte test
Number of elements 216
Dimensions
3 3 3 2 4 
Permutation
1 3 0 4 2 
==1423119== Invalid read of size 1
==1423119==    at 0x494F4F5: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x494F67C: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x49531F0: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x490741D: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x4ABAAEA: hipOccupancyMaxActiveBlocksPerMultiprocessor (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x303993: getNumActiveBlock(int, int, LaunchConfig const&, int, hipDeviceProp_t const&) (hip_runtime_api.h:0)
==1423119==    by 0x30DCA0: librettPlan_t::createTiledPlans(int, int const*, int const*, unsigned long, int, hipDeviceProp_t const&, std::__cxx11::list<librettPlan_t, std::allocator<librettPlan_t> >&) (src/plan.cpp:520)
==1423119==    by 0x30EC01: librettPlan_t::createPlans(int, int const*, int const*, int, int const*, int const*, unsigned long, int, hipDeviceProp_t const&, std::__cxx11::list<librettPlan_t, std::allocator<librettPlan_t> >&) (src/plan.cpp:712)
==1423119==    by 0x2FBE23: librettPlan(unsigned int*, int, int*, int*, unsigned long, ihipStream_t*&) (src/librett.cpp:168)
==1423119==    by 0x2F8BC5: bool tensor_transpose<int>(std::vector<int, std::allocator<int> >&, std::vector<int, std::allocator<int> >&, ihipStream_t*&) (tests/example.cpp:158)
==1423119==    by 0x2F8657: main (tests/example.cpp:96)
==1423119==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
==1423119== 
==1423119== 
==1423119== Process terminating with default action of signal 11 (SIGSEGV)
==1423119==  Access not within mapped region at address 0x20
==1423119==    at 0x494F4F5: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x494F67C: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x49531F0: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x490741D: ??? (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x4ABAAEA: hipOccupancyMaxActiveBlocksPerMultiprocessor (in /opt/rocm-5.5.3/lib/libamdhip64.so.5.5.50503)
==1423119==    by 0x303993: getNumActiveBlock(int, int, LaunchConfig const&, int, hipDeviceProp_t const&) (hip_runtime_api.h:0)
==1423119==    by 0x30DCA0: librettPlan_t::createTiledPlans(int, int const*, int const*, unsigned long, int, hipDeviceProp_t const&, std::__cxx11::list<librettPlan_t, std::allocator<librettPlan_t> >&) (src/plan.cpp:520)
==1423119==    by 0x30EC01: librettPlan_t::createPlans(int, int const*, int const*, int, int const*, int const*, unsigned long, int, hipDeviceProp_t const&, std::__cxx11::list<librettPlan_t, std::allocator<librettPlan_t> >&) (src/plan.cpp:712)
==1423119==    by 0x2FBE23: librettPlan(unsigned int*, int, int*, int*, unsigned long, ihipStream_t*&) (src/librett.cpp:168)
==1423119==    by 0x2F8BC5: bool tensor_transpose<int>(std::vector<int, std::allocator<int> >&, std::vector<int, std::allocator<int> >&, ihipStream_t*&) (tests/example.cpp:158)
==1423119==    by 0x2F8657: main (tests/example.cpp:96)
==1423119==  If you believe this happened as a result of a stack
==1423119==  overflow in your program's main thread (unlikely but
==1423119==  possible), you can try to increase the size of the
==1423119==  main thread stack using the --main-stacksize= flag.
==1423119==  The main thread stack size used in this run was 100003840.
==1423119== 
==1423119== HEAP SUMMARY:
==1423119==     in use at exit: 2,008,681 bytes in 15,735 blocks
==1423119==   total heap usage: 298,876 allocs, 283,141 frees, 153,337,969 bytes allocated
==1423119== 
==1423119== LEAK SUMMARY:
==1423119==    definitely lost: 584 bytes in 4 blocks
==1423119==    indirectly lost: 2,908 bytes in 54 blocks
==1423119==      possibly lost: 12,348 bytes in 88 blocks
==1423119==    still reachable: 1,992,841 bytes in 15,589 blocks
==1423119==                       of which reachable via heuristic:
==1423119==                         multipleinheritance: 112 bytes in 1 blocks
==1423119==         suppressed: 0 bytes in 0 blocks
==1423119== Rerun with --leak-check=full to see details of leaked memory
==1423119== 
==1423119== For lists of detected and suppressed errors, rerun with: -s
==1423119== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

It should be noted that I am using an older version of HIP, though the functions in question have not changed in use, and have not changed significantly in implementation. I have tracked the issue down to when HIP tries to build the kernel in order to figure out the number of blocks it needs to achieve maximum occupancy. If anyone has any insight, I would be happy to hear it.

[cgbriggs99]

Specifically, here's a sort of stack trace for this issue, figured out through Ghidra.
https://github.com/ROCm/clr/blob/fa63919a6339ea2a61111981ba2362c97fbdf743/hipamd/src/hip_fatbin.cpp#L439
https://github.com/ROCm/clr/blob/fa63919a6339ea2a61111981ba2362c97fbdf743/hipamd/src/hip_fatbin.cpp#L457
https://github.com/ROCm/clr/blob/fa63919a6339ea2a61111981ba2362c97fbdf743/hipamd/src/hip_global.cpp#L165
https://github.com/ROCm/clr/blob/73ed8adfd08c38b6a40638c9d26ace5ec862ba68/hipamd/src/hip_platform.cpp#L513