Skip to content

Ensure make -j uses a reasonable argument #1541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 18, 2025
Merged

Ensure make -j uses a reasonable argument #1541

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4be7737
Ensure ``make -j`` uses a reasonable argument
AA-Turner Apr 18, 2025
c2784fa
Simplify
AA-Turner Apr 18, 2025
1b10013
Expand
AA-Turner Apr 18, 2025
9ce0c3c
Do not suggest `make -j` for building CPython. `make -j` means _no_ l…
Yhg1s Apr 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions developer-workflow/extension-modules.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -547,8 +547,11 @@ Now that the configuration is in place, it remains to compile the project:

.. tip::

Use ``make -j`` to speed-up compilation by utilizing as many CPU cores
as possible or ``make -jN`` to allow at most *N* concurrent jobs.
Use ``make -jN`` to speed-up compilation by utilizing as many CPU cores
as possible, where *N* is as many CPU cores you want to spare (and have
memory for). Be careful using ``make -j`` with no argument, as this puts
no limit on the number of jobs, and compilation can sometimes use up a
lot of memory (like when building with LTO).

* ``make regen-configure`` updates the :cpy-file:`configure` script.

Expand Down
13 changes: 6 additions & 7 deletions getting-started/setup-building.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,14 +203,13 @@ do to get a pydebug build of CPython.

Once ``configure`` is done, you can then compile CPython with::

$ make -s -j2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that nproc is part of GNU coreutils and may not necessarily be available on all Unix-like systems (namely, macOS). So Mac users should first do brew install coreutils or use sysctl instead to get the number of physical or logical cores (I don't remember which one nproc returns)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've merged, as Hugo uses macOS and didn't complain, but we can open a follow-up if need be.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I do have coreutils installed via Homebrew, which is why it worked for me. Let's not require or assume others do.

Shall we use a hardcoded value under the macOS tabs? Something like 8 feels like a good default?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TL;DR - this is good enough.

I feel like people who understand how to run make already understand what $(nproc) can be substituted with manually if they don't have the command. if we wanted to be ironic we'd suggest $(python3 -c 'import os; print(os.cpu_count())') to the person who is building python and may not yet have a built python3 (but realistically do anyways because all Linux distros offer it and macOS xcode comes with one). 😛

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like people who understand how to run make already understand what $(nproc) can be substituted with

That seems like a rash assumption to me; I would expect anyone with that level of comfort with make to not need to be walked through "run ./configure && make". There's also the point that make -j $(nproc) in the absense of a working nproc will leave the user with make -j , which is what we were trying to avoid with this change.

Copy link
Member

@picnixz picnixz Apr 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assumption would be correct if we were in the extension-modules.rst file, but this file is for the "getting started". I don't think we should assume that the reader is extra familiar with make and even knows about nproc and co (this section would be read by newcomers I think).

By the way, nproc being part of coreutils and not necessarily available on macOS caused some issues in other projects: https://www.drupal.org/project/drupal/issues/3407360. So it's not necessarily a theoretical use case.

$ make -s -j $(nproc)

This will build CPython with only warnings and errors being printed to
stderr and utilize up to 2 CPU cores. If you are using a multi-core machine
with more than 2 cores (or a single-core machine), you can adjust the number
passed into the ``-j`` flag to match the number of cores you have (or if your
version of Make supports it, you can use ``-j`` without a number and Make
will not limit the number of steps that can run simultaneously.).
stderr. The ``-j`` argument means that :program:`make` will concurrently run
tasks, limiting the number of parallel jobs to the number of CPU cores in your
computer. You can adjust the number passed to the ``-j`` flag to change
the limit on parallel jobs, which can trade RAM usage versus compilation time.

At the end of the build you should see a success message, followed
by a list of extension modules that haven't been built because their
Expand Down Expand Up @@ -875,7 +874,7 @@ some of CPython's modules (for example, ``zlib``).

And finally, run ``make``::

$ make -s -j2
$ make -s -j $(nproc)

There will sometimes be optional modules added for a new release which
won't yet be identified in the OS-level build dependencies. In those cases,
Expand Down
4 changes: 2 additions & 2 deletions index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@ instructions please see the :ref:`setup guide <setup>`.

.. code-block:: shell

./configure --with-pydebug && make -j
./configure --with-pydebug && make -j $(nproc)

.. tab:: macOS

.. code-block:: shell

./configure --with-pydebug && make -j
./configure --with-pydebug && make -j $(nproc)

.. tab:: Windows

Expand Down
Loading