Skip to content

Error on creation of wheel with a name that already exists, and add Py_LIMITED_API to the docs' FAQ #569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
12 changes: 12 additions & 0 deletions cibuildwheel/linux.py
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,18 @@ def build(options: BuildOptions) -> None:
# clean up test environment
docker.call(['rm', '-rf', venv_dir])

existing_output_files = docker.call(['ls'], capture_output=True, cwd=container_output_dir).split('\n')
for repaired_wheel in repaired_wheels:
if repaired_wheel.name in existing_output_files:
message = f'Created wheel ({repaired_wheel}) already exists in output directory.'
if 'abi3' in repaired_wheel.name:
message += ('\n'
"It looks like you are building wheels against Python's limited API/stable ABI;\n"
'please limit your Python selection to a single version when building limited API\n'
'wheels, with CIBW_BUILD.')
log.error(message)
sys.exit(1)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want sys.exit, or an exception? Or downgrade this to just a warning (though Windows will error, anyway?).

And if sys.exit: just 1, or a new/specific error code?

Copy link
Contributor

@henryiii henryiii Jan 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How hard would it be to just skip the build here and go directly to tests when a abi3 wheel is already built? It should likely also be an error if tests are empty, as you should not "build" with multiple versions but not test? That would provide the "full support" solution.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've already done the build at this point. To do the 'full support' thing, we'd need to predict the produced wheel name without actually doing the build.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first time, you just build like normal. Then you could register that you produced a abi3 wheel, and future builds would skip building. For python-requires, you have to read the config, since you don't know what you are allowed to produce. For ABI3, you could just use the technique above and react to the built, rather than trying to compute what will happen first.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, that should be possible! The main question is how close this is to cibuildwheel's other behavior. Most of the time, you have to explicitly opt-in to things and cibuildwheel won't say "ah, you probably meant ...".

On a more practical note: how would you decide which Python version to build on? I don't fully understand Py_LIMITED_API, but it seems to also include a "minimum version"?

Copy link
Contributor

@henryiii henryiii Jan 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you decide which Python version to build on?

Via Requires-Python and/or build selectors. cibuildwheel builds the oldest Python selected, and works it's way forward. Say it builds 2.7, normal wheel. 3.5, normal wheel. Then 3.6 is a stable ABI build. From then on, it just tests the existing wheel: 3.7, 3.8, and 3.9 are just tests of the ABI wheel.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would even work if someone had special logic in setup.py for the limited_abi only working after some version of minimum required Python, while any other non-reactive method would not. You don't need to worry about the other direction (yet?), since the limited ABI works on all future Pythons (at least until Python 4).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, OK, that could work, indeed :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea, yeah! Avoiding the metadata read would be great.


# move repaired wheels to output
docker.call(['mkdir', '-p', container_output_dir])
docker.call(['mv', *repaired_wheels, container_output_dir])
Expand Down
10 changes: 10 additions & 0 deletions cibuildwheel/macos.py
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,16 @@ def call_with_arch(args: Sequence[PathOrStr], **kwargs: Any) -> int:
# clean up
shutil.rmtree(venv_dir)

if (options.output_dir / repaired_wheel.name).exists():
message = f'Created wheel ({repaired_wheel}) already exists in output directory.'
if 'abi3' in repaired_wheel.name:
message += ('\n'
"It looks like you are building wheels against Python's limited API/stable ABI;\n"
'please limit your Python selection to a single version when building limited API\n'
'wheels, with CIBW_BUILD.')
log.error(message)
sys.exit(1)

# we're all done here; move it to output (overwrite existing)
shutil.move(str(repaired_wheel), options.output_dir)
log.build_end()
Expand Down
10 changes: 10 additions & 0 deletions cibuildwheel/windows.py
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,16 @@ def build(options: BuildOptions) -> None:
# clean up
shutil.rmtree(venv_dir)

if (options.output_dir / repaired_wheel.name).exists():
message = f'Created wheel ({repaired_wheel}) already exists in output directory.'
if 'abi3' in repaired_wheel.name:
message += ('\n'
"It looks like you are building wheels against Python's limited API/stable ABI;\n"
'please limit your Python selection to a single version when building limited API\n'
'wheels, with CIBW_BUILD.')
log.error(message)
sys.exit(1)

# we're all done here; move it to output (remove if already exists)
shutil.move(str(repaired_wheel), options.output_dir)
log.build_end()
Expand Down
60 changes: 60 additions & 0 deletions test/test_limited_api.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
import subprocess
import textwrap

import pytest

from . import test_projects, utils

limited_api_project = test_projects.new_c_project(
setup_cfg_add=textwrap.dedent(r'''
[bdist_wheel]
py_limited_api=cp36
''')
)


def test_setup_cfg(tmp_path):
project_dir = tmp_path / 'project'
limited_api_project.generate(project_dir)

# build the wheels
actual_wheels = utils.cibuildwheel_run(project_dir, add_env={
'CIBW_BUILD': 'cp27-* cp36-*', # PyPy does not have a Py_LIMITED_API equivalent
})

# check that the expected wheels are produced
expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0', limited_api='cp36')
if '-pp' not in w]
assert set(actual_wheels) == set(expected_wheels)


def test_build_option_env(tmp_path, capfd):
project_dir = tmp_path / 'project'
test_projects.new_c_project().generate(project_dir)

# build the wheels
actual_wheels = utils.cibuildwheel_run(project_dir, add_env={
'CIBW_ENVIRONMENT': 'PIP_BUILD_OPTION="--py-limited-api=cp36"',
'CIBW_BUILD': 'cp27-* cp36-*', # PyPy does not have a Py_LIMITED_API equivalent
})

# check that the expected wheels are produced
expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0', limited_api='cp36')
if '-pp' not in w]
assert set(actual_wheels) == set(expected_wheels)


def test_duplicate_wheel_error(tmp_path, capfd):
project_dir = tmp_path / 'project'
limited_api_project.generate(project_dir)

with pytest.raises(subprocess.CalledProcessError):
utils.cibuildwheel_run(project_dir, add_env={
'CIBW_BUILD': 'cp36-* cp37-*',
})

captured = capfd.readouterr()
print('out', captured.out)
print('err', captured.err)
assert "already exists in output directory" in captured.err
assert "It looks like you are building wheels against Python's limited API" in captured.err
8 changes: 6 additions & 2 deletions test/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ def _get_arm64_macosx_deployment_target(macosx_deployment_target: str) -> str:

def expected_wheels(package_name, package_version, manylinux_versions=None,
macosx_deployment_target='10.9', machine_arch=None, *,
exclude_27=IS_WINDOWS_RUNNING_ON_TRAVIS):
exclude_27=IS_WINDOWS_RUNNING_ON_TRAVIS,
limited_api=None):
'''
Returns a list of expected wheels from a run of cibuildwheel.
'''
Expand All @@ -106,7 +107,10 @@ def expected_wheels(package_name, package_version, manylinux_versions=None,
else:
manylinux_versions = ['manylinux2014']

python_abi_tags = ['cp35-cp35m', 'cp36-cp36m', 'cp37-cp37m', 'cp38-cp38', 'cp39-cp39']
if limited_api:
python_abi_tags = [f'{limited_api}-abi3']
else:
python_abi_tags = ['cp35-cp35m', 'cp36-cp36m', 'cp37-cp37m', 'cp38-cp38', 'cp39-cp39']

if machine_arch in ['x86_64', 'AMD64', 'x86']:
python_abi_tags += ['cp27-cp27m', 'pp27-pypy_73', 'pp36-pypy36_pp73', 'pp37-pypy37_pp73']
Expand Down