framework espidf not finding python module pip

[db4rne]

Operating system:
Arch Linux

PlatformIO Version (platformio --version):
PlatformIO Core, version 6.1.16

Description of problem

is soon as I try to build anything for esp32 with espidf framework, I get an error of python not findig the pip module. If I start python by hand, it works.

Steps to Reproduce

1. pacman -S platformio-core
2. git clone https://github.com/platformio/platform-espressif32.git
3. cd platform-espressif32/examples/espidf-hello-world
4. pio run

Actual Results

Log

❯ pio run
Processing esp32dev (board: esp32dev; platform: espressif32; framework: espidf)
---------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (6.5.0) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-espidf @ 3.50102.240122 (5.1.2)
 - tool-cmake @ 3.16.4
 - tool-esptoolpy @ 1.40501.0 (4.5.1)
 - tool-ninja @ 1.7.1
 - tool-riscv32-esp-elf-gdb @ 12.1.0+20221002
 - tool-xtensa-esp-elf-gdb @ 12.1.0+20221002
 - toolchain-esp32ulp @ 1.23500.220830 (2.35.0)
 - toolchain-xtensa-esp32 @ 12.2.0+20230208
/home/arne/.platformio/penv/.espidf-5.1.2/bin/python: No module named pip
CalledProcessError: Command '['/home/arne/.platformio/penv/.espidf-5.1.2/bin/python', '-m', 'pip', 'list', '--format=json', '--disable-pip-version-check']' returned non-zero exit status 1.:
  File "/usr/lib/python3.13/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/main.py", line 346:
    target_elf = env.BuildProgram()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 684:
    return method(*args, **kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1252:
    install_python_deps()
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1129:
    installed_packages = _get_installed_pip_packages(python_exe_path)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1093:
    pip_output = subprocess.check_output(
  File "/usr/lib/python3.13/subprocess.py", line 472:
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.13/subprocess.py", line 577:
    raise CalledProcessError(retcode, process.args,
======================================================== [FAILED] Took 0.44 seconds ========================================================

Processing esp32-s2-kaluga-1 (board: esp32-s2-kaluga-1; platform: espressif32; framework: espidf)
---------------------------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing espressif/toolchain-xtensa-esp32s2 @ 12.2.0+20230208
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Tool Manager: toolchain-xtensa-esp32s2@12.2.0+20230208 has been installed!
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32-s2-kaluga-1.html
PLATFORM: Espressif 32 (6.5.0) > Espressif ESP32-S2-Kaluga-1 Kit
HARDWARE: ESP32S2 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (ftdi) On-board (ftdi) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-espidf @ 3.50102.240122 (5.1.2)
 - tool-cmake @ 3.16.4
 - tool-esptoolpy @ 1.40501.0 (4.5.1)
 - tool-ninja @ 1.7.1
 - tool-riscv32-esp-elf-gdb @ 12.1.0+20221002
 - tool-xtensa-esp-elf-gdb @ 12.1.0+20221002
 - toolchain-esp32ulp @ 1.23500.220830 (2.35.0)
 - toolchain-riscv32-esp @ 12.2.0+20230208
 - toolchain-xtensa-esp32s2 @ 12.2.0+20230208
/home/arne/.platformio/penv/.espidf-5.1.2/bin/python: No module named pip
CalledProcessError: Command '['/home/arne/.platformio/penv/.espidf-5.1.2/bin/python', '-m', 'pip', 'list', '--format=json', '--disable-pip-version-check']' returned non-zero exit status 1.:
  File "/usr/lib/python3.13/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/main.py", line 346:
    target_elf = env.BuildProgram()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 684:
    return method(*args, **kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1252:
    install_python_deps()
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1129:
    installed_packages = _get_installed_pip_packages(python_exe_path)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1093:
    pip_output = subprocess.check_output(
  File "/usr/lib/python3.13/subprocess.py", line 472:
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.13/subprocess.py", line 577:
    raise CalledProcessError(retcode, process.args,
======================================================== [FAILED] Took 23.67 seconds ========================================================

Processing esp32-c3-devkitm-1 (board: esp32-c3-devkitm-1; platform: espressif32; framework: espidf)
---------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32-c3-devkitm-1.html
PLATFORM: Espressif 32 (6.5.0) > Espressif ESP32-C3-DevKitM-1
HARDWARE: ESP32C3 160MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-builtin, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-espidf @ 3.50102.240122 (5.1.2)
 - tool-cmake @ 3.16.4
 - tool-esptoolpy @ 1.40501.0 (4.5.1)
 - tool-ninja @ 1.7.1
 - tool-riscv32-esp-elf-gdb @ 12.1.0+20221002
 - tool-xtensa-esp-elf-gdb @ 12.1.0+20221002
 - toolchain-esp32ulp @ 1.23500.220830 (2.35.0)
 - toolchain-riscv32-esp @ 12.2.0+20230208
/home/arne/.platformio/penv/.espidf-5.1.2/bin/python: No module named pip
CalledProcessError: Command '['/home/arne/.platformio/penv/.espidf-5.1.2/bin/python', '-m', 'pip', 'list', '--format=json', '--disable-pip-version-check']' returned non-zero exit status 1.:
  File "/usr/lib/python3.13/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/main.py", line 346:
    target_elf = env.BuildProgram()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 684:
    return method(*args, **kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1252:
    install_python_deps()
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1129:
    installed_packages = _get_installed_pip_packages(python_exe_path)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1093:
    pip_output = subprocess.check_output(
  File "/usr/lib/python3.13/subprocess.py", line 472:
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.13/subprocess.py", line 577:
    raise CalledProcessError(retcode, process.args,
======================================================== [FAILED] Took 0.41 seconds ========================================================

Processing esp32-c6-devkitc-1 (board: esp32-c6-devkitc-1; platform: espressif32; framework: espidf)
---------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32-c6-devkitc-1.html
PLATFORM: Espressif 32 (6.5.0) > Espressif ESP32-C6-DevKitC-1
HARDWARE: ESP32C6 160MHz, 512KB RAM, 8MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-builtin, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-espidf @ 3.50102.240122 (5.1.2)
 - tool-cmake @ 3.16.4
 - tool-esptoolpy @ 1.40501.0 (4.5.1)
 - tool-ninja @ 1.7.1
 - tool-riscv32-esp-elf-gdb @ 12.1.0+20221002
 - tool-xtensa-esp-elf-gdb @ 12.1.0+20221002
 - toolchain-esp32ulp @ 1.23500.220830 (2.35.0)
 - toolchain-riscv32-esp @ 12.2.0+20230208
/home/arne/.platformio/penv/.espidf-5.1.2/bin/python: No module named pip
CalledProcessError: Command '['/home/arne/.platformio/penv/.espidf-5.1.2/bin/python', '-m', 'pip', 'list', '--format=json', '--disable-pip-version-check']' returned non-zero exit status 1.:
  File "/usr/lib/python3.13/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/main.py", line 346:
    target_elf = env.BuildProgram()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Util/envs.py", line 252:
    return self.method(*nargs, **kwargs)
  File "/usr/lib/python3.13/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 684:
    return method(*args, **kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 620:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/arne/.platformio/packages/tool-scons/scons-local-4.8.1/SCons/Script/SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1252:
    install_python_deps()
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1129:
    installed_packages = _get_installed_pip_packages(python_exe_path)
  File "/home/arne/.platformio/platforms/espressif32/builder/frameworks/espidf.py", line 1093:
    pip_output = subprocess.check_output(
  File "/usr/lib/python3.13/subprocess.py", line 472:
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.13/subprocess.py", line 577:
    raise CalledProcessError(retcode, process.args,
======================================================== [FAILED] Took 0.41 seconds ========================================================

Environment         Status    Duration
------------------  --------  ------------
esp32dev            FAILED    00:00:00.443
esp32-s2-kaluga-1   FAILED    00:00:23.673
esp32-c3-devkitm-1  FAILED    00:00:00.408
esp32-c6-devkitc-1  FAILED    00:00:00.414
=================================================== 4 failed, 0 succeeded in 00:00:24.938 ===================================================

Expected Results

should build executables

Additional info

it looks to me like this call is failing: /home/arne/.platformio/penv/.espidf-5.1.2/bin/python -m pip ...
so here is some more info on it:

❯ ls -l /home/arne/.platformio/penv/.espidf-5.1.2/bin/python
lrwxrwxrwx 1 arne arne 15 Nov 29 15:52 /home/arne/.platformio/penv/.espidf-5.1.2/bin/python -> /usr/bin/python

(seems to be a symlink, ok)

now by hand:

❯ /usr/bin/python -m pip --version
pip 24.3.1 from /usr/lib/python3.13/site-packages/pip (python 3.13)

❯ /home/arne/.platformio/penv/.espidf-5.1.2/bin/python -m pip --version
/home/arne/.platformio/penv/.espidf-5.1.2/bin/python: No module named pip

so, for whatever reason, python doesn't like the symlink? I have no idea whats going on.

edit 1:
so I found this: https://unix.stackexchange.com/questions/645554/symlink-to-python-exec-and-the-python-exec-have-different-sys-paths-this-causes
which I verified:

# sys.path variable when calling python normally:
❯ python
Python 3.13.1 (main, Dec  4 2024, 18:05:56) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python313.zip', '/usr/lib/python3.13', '/usr/lib/python3.13/lib-dynload', '/usr/lib/python3.13/site-packages']

# sys.path with the symlink:
❯ /home/arne/.platformio/penv/.espidf-5.1.2/bin/python
Python 3.13.1 (main, Dec  4 2024, 18:05:56) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python313.zip', '/usr/lib/python3.13', '/usr/lib/python3.13/lib-dynload']

so, the path /usr/lib/python3.13/site-packages is missing. still no idea how to fix that

[db4rne]

ok, so I dug a bit deeper and I found the file .platformio/penv/.espidf-5.1.2/pyvenv.cfg, which had an interesting config:

home = /usr/bin
include-system-site-packages = false
version = 3.12.7
executable = /usr/bin/python3.12
command = /usr/bin/python -m venv --clear /home/arne/.platformio/penv/.espidf-5.1.2

I changed it (by hand) to:

home = /usr/bin
include-system-site-packages = true
version = 3.13.1
executable = /usr/bin/python
command = /usr/bin/python -m venv --clear /home/arne/.platformio/penv/.espidf-5.1.2

now it works :D, although I'm not sure if this is still a problem with platformio or my setup, so I will leave the bug open for now, maybe someone more involved knows whats up.

[valeros]

Hi @db4rne, I've just tried to reproduce the issue on a fresh Arch Linux machine and everything worked as expected. What's also strange is that in your case the script complains that the pip is missing, even though the virtual environment for the IDF dependencies is created via the built-in venv module which installs pip by default via ensurepip. How many Python interpreters with different versions do you have in your system?

[db4rne]

Hi, I have two: python 3 (3.13.1) and 2.7. /usr/bin/python points to python3. what I think what happened is that the environment was created before (also, when I made the bug, I didn't realize it was user-wide installed, not per project), and probably with python 3.12.7 (as it said in the file), and then it didn't update this environment after the update? just guessing here, but I think it makes sense?

[valeros]

I see, so just to sum up: some time ago, you installed PlatformIO and the espressif32 dev-platform using Python 3.12.7. Over time, you updated your Python to 3.13.1 and since that moment all IDF projects have been broken. Is that correct?

[db4rne]

yes, that is correct. and as soon as I updated the file .platformio/penv/.espidf-5.1.2/pyvenv.cfg as prevoiusly mentioned it works again.