3

I'm working on a Python project with a setup.py that has something like this1:

setup(
    cmdclass={"build_ext": my_build_ext},
    ext_modules=[
        Extension("A", ["a.c", "common.c"]),
        Extension("B", ["b.c", "common.c"])
    ]
)

I'm running into a problem when building the modules in parallel where it seems like one module tries to read common.o/common.obj while another is compiling it, and it fails. Is there some way to get setuptools to compile the C files for each module into their own build directories so that they aren't overwriting each other?

 

  1. The actual project is more complicated with more modules and source files.
1

1 Answer 1

3

I found a potential solution by overriding build_extension() in a custom build_ext class:

import copy, os
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext
class my_build_ext(build_ext):
    def build_extension(self, ext):
        # Append the extension name to the temp build directory
        # so that each module builds to its own directory.
        # We need to make a (shallow) copy of 'self' here
        # so that we don't overwrite this value when running in parallel.
        self_copy = copy.copy(self)
        self_copy.build_temp = os.path.join(self.build_temp, ext.name)
        build_ext.build_extension(self_copy, ext)
setup(
    cmdclass={"build_ext": my_build_ext},
    ext_modules=[
        Extension("A", ["a.c", "common.c"]),
        Extension("B", ["b.c", "common.c"])
    ]
)

I've also since been told of the (currently undocumented) libraries parameter for setup():

from setuptools import Extension, setup
setup(
    libraries=[
        ("common", {"sources": ["common.c"]}),
    ],
    ext_modules=[
        Extension("A", sources=["a.c"], libraries=["common"]),
        Extension("B", sources=["b.c"], libraries=["common"]),
    ],
)

Both solutions worked for me, but in slightly different ways. The first solution recompiles the code for each module, which allows you to specify different parameters to use for each module (ex. different defs). The second solution only has to compile to code once and it will reuse that for every module.

Sign up to request clarification or add additional context in comments.

1 Comment

I think that is the exactly proper way to handle that since build root is not exposed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.