Missing .so lib for training pybinding

[ZiyueXu77]

🐛 Describe the bug

Installed with the current main following:

git pull
./install_executorch.sh --clean
git submodule sync
git submodule update --init
./install_executorch.sh --pybind training

Then try the test script

python pip-out/lib.linux-x86_64-cpython-310/executorch/extension/training/pybindings/test/test.py 

Error message:

Traceback (most recent call last):
  File "/media/ziyuexu/Research/Experiment/NVFlare/executorch/pip-out/lib.linux-x86_64-cpython-310/executorch/extension/training/pybindings/test/test.py", line 14, in <module>
    from executorch.extension.training import (
  File "/home/ziyuexu/anaconda3/envs/nvflare_edge/lib/python3.10/site-packages/executorch/extension/training/__init__.py", line 9, in <module>
    from executorch.extension.training.pybindings._training_lib import get_sgd_optimizer
ImportError: libextension_module.so: cannot open shared object file: No such file or directory

Versions

Collecting environment information...
PyTorch version: 2.7.0.dev20250311+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.2 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.39

Python version: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-6.8.0-55-generic-x86_64-with-glibc2.39
Is CUDA available: False
CUDA runtime version: 12.6.68
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA RTX 6000 Ada Generation
Nvidia driver version: 560.35.03
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz
CPU family: 6
Model: 79
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
Stepping: 1
CPU(s) scaling MHz: 46%
CPU max MHz: 3800.0000
CPU min MHz: 1200.0000
BogoMIPS: 6796.02
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti intel_ppin ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
L1d cache: 192 KiB (6 instances)
L1i cache: 192 KiB (6 instances)
L2 cache: 1.5 MiB (6 instances)
L3 cache: 15 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-11
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+320d555
[pip3] mypy==1.10.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==2.2.4
[pip3] nvidia-cublas-cu11==11.10.3.66
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu11==11.7.101
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu11==11.7.99
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu11==11.7.99
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu11==8.5.0.96
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu11==10.9.0.58
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu11==10.2.10.91
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu11==11.4.0.1
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu11==11.7.4.91
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu11==2.14.3
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu11==11.7.91
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] onnx==1.14.0
[pip3] torch==2.7.0.dev20250311+cpu
[pip3] torchao==0.10.0+git923242e2
[pip3] torchaudio==2.6.0.dev20250311+cpu
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0.dev20250311+cpu
[pip3] triton==3.2.0
[conda] executorch 0.6.0a0+320d555 pypi_0 pypi
[conda] numpy 2.2.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
[conda] torch 2.7.0.dev20250311+cpu pypi_0 pypi
[conda] torchao 0.10.0+git923242e2 pypi_0 pypi
[conda] torchaudio 2.6.0.dev20250311+cpu pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchvision 0.22.0.dev20250311+cpu pypi_0 pypi
[conda] triton 3.2.0 pypi_0 pypi

cc @JacobSzwejbka

[JacobSzwejbka]

looking

[JacobSzwejbka]

Are you using editable mode of the pip release @ZiyueXu77 ?

I am unable to repro this failure when using the git clone flow. Both the xor example and the pybindings test succeed for me locally.

[ZiyueXu77]

thanks for the prompt response! let me start from scratch and clean, could be due to my previous installations

[ZiyueXu77]

start from a fresh conda env with python 3.11 and a new clone, now the previous error is gone, but new error
ModuleNotFoundError: No module named 'executorch.extension.training.pybindings._training_lib'
the full process:

conda create -n et python=3.11
conda activate et
pip install cmake
git clone --branch main https://github.com/pytorch/executorch.git
cd executorch
git submodule sync
git submodule update --init
./install_requirements.sh
./install_executorch.sh --editable --pybind training
python extension/training/pybindings/test/test.py

[jathu]

@ZiyueXu77 I think when you run pip install -e . you are re-installing ET without the training pybindings. If you want to install ET in editable mode with the training pybindings, I think all you'll have to do is:

$ ./install_executorch.sh --editable --pybind training
$ python extension/training/pybindings/test/test.py

[ZiyueXu77]

Thanks! Somehow this gives me the original error
from executorch.extension.training.pybindings._training_lib import get_sgd_optimizer ImportError: libextension_module.so: cannot open shared object file: No such file or directory

[jathu]

I suspect this is an issue with how this extension is being linked on macOS vs. Linux

executorch/extension/module/CMakeLists.txt

Lines 20 to 29 in fc25829

	if(CMAKE_TOOLCHAIN_IOS
	OR CMAKE_TOOLCHAIN_ANDROID
	OR APPLE
	)
	# Building a share library on iOS requires code signing On Android we see
	# duplicated registration when using shared lib
	add_library(extension_module STATIC ${_extension_module__srcs})
	else()
	add_library(extension_module SHARED ${_extension_module__srcs})
	endif()

cc @JacobSzwejbka @GregoryComer

[JacobSzwejbka]

cc @larryliu0820 for some more pip editable mode issues.

@ZiyueXu77 could you try cloning and building from source instead of using pip? Editable mode is still wip I think.

[ZiyueXu77]

cc @larryliu0820 for some more pip editable mode issues.

@ZiyueXu77 could you try cloning and building from source instead of using pip? Editable mode is still wip I think.

I revised my steps according to @jathu 's comments and removed the pip line (updated above), but it gave me the libextension_module.so not found error

[larryliu0820]

I can take a look at this one. I haven't extensively tested training so yet.