Custom default sysconfig INSTALL_SCHEME leads to superfulous "Virtual environment creation failed"

[hroncok]

Hello,

With the proposal in https://bugs.python.org/issue43976 we are using an approach in Fedora, that changes the default install scheme to use /usr/local paths. Essentially, we set:

if (not (hasattr(sys, 'real_prefix') or
    sys.prefix != sys.base_prefix) and
    'RPM_BUILD_ROOT' not in os.environ):
        _INSTALL_SCHEMES['posix_prefix'] = {
            'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
            'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
            'purelib': '{base}/local/lib/python{py_version_short}/site-packages',
            'platlib': '{platbase}/local/{platlibdir}/python{py_version_short}/site-packages',
            'include':
                '{installed_base}/include/python{py_version_short}{abiflags}',
            'platinclude':
                '{installed_platbase}/include/python{py_version_short}{abiflags}',
            'scripts': '{base}/local/bin',
            'data': '{base}/local',
        }

However, we have realized this breaks standard assumptions about virtual environments created by virtualenv.

In virtualenv, we have fixed this in pypa/virtualenv#2208 by using a dedicated installation scheme called "venv". The standard library venv module does not read install schemes at all, but we have opened https://bugs.python.org/issue45413 anyway.

In build, this causes problems that @vstinner reported to me on Fedora's Python IRC channel. tl;dr in here:

build/src/build/env.py

Line 306 in 96f9188

paths = sysconfig.get_paths(vars=config_vars)

The default install scheme is used. Hence on Fedora, build errors with:

RuntimeError: Virtual environment creation failed, executable /tmp/build-env-.../local/bin/python missing

That is correct, /tmp/build-env-.../local/bin/python is missing, but there is /tmp/build-env-.../bin/python which is completely OK.

We'd like build to use the "venv" scheme if it exists, as does virtualenv itself:

https://github.com/pypa/virtualenv/blob/bb2eeaca5a5c9818bf3b8b047f0e859c16905d26/src/virtualenv/discovery/py_info.py#L76-L80

(To check this out, use Python 3.10 on Fedora 36 or Python 3.11 on any Fedora version (installed via dnf from the official Fedora repositories. I think that build itself must not be installed in a virtual environment to reproduce this.)

$ python3.11 -m build
* Creating venv isolated environment...

Traceback (most recent call last):
  File "/home/vstinner/.local/lib/python3.11/site-packages/build/__main__.py", line 372, in main
    built = build_call(
            ^^^^^^^^^^^
  File "/home/vstinner/.local/lib/python3.11/site-packages/build/__main__.py", line 229, in build_package_via_sdist
    sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/.local/lib/python3.11/site-packages/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/.local/lib/python3.11/site-packages/build/__main__.py", line 104, in _build_in_isolated_env
    with _IsolatedEnvBuilder() as env:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/.local/lib/python3.11/site-packages/build/env.py", line 104, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/.local/lib/python3.11/site-packages/build/env.py", line 258, in _create_isolated_env_venv
    executable, script_dir, purelib = _find_executable_and_scripts(path)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/.local/lib/python3.11/site-packages/build/env.py", line 303, in _find_executable_and_scripts
    raise RuntimeError(f'Virtual environment creation failed, executable {executable} missing')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Virtual environment creation failed, executable /tmp/build-env-7jymb7oz/local/bin/python missing

ERROR Virtual environment creation failed, executable /tmp/build-env-7jymb7oz/local/bin/python missing

Would you accept a PR that fixes this? This should be fairly easy to fix, but I am unsure if I will be able to provide tests.

[hroncok]

#310 is kinda related

[hroncok]

A draft PR without tests #434

[layday]

Previous discussion: #215.