Using SCons as a build engine for distutils

Question

I have a python package with some C code needed to create an extension (with some non-trivial building needs). I used SCons as my build system because it is really good and flexible.

I am looking for a way to compile my python extensions with SCons, ready for distribution with distutils. I want the user to simply type setup.py install and get the extension compiled using SCons, and not the default for the distutils build engine.

The idea that comes to mind is to override the build_ext command in distutils, but I cannot find extensive documentation for it.

Any suggestion?

Answer 1

The enscons package seems to be designed for tasks. An example of using it to build a package with C extensions is here .

You may have a basic package structure, for example:

pkgroot/ pyproject.toml setup.py SConstruct README.md pkgname/ __init__.py pkgname.py cfile.c 

In this, the pyproject.toml file might look something like this:

 [build-system] requires = ["enscons"] [tool.enscons] name = "pkgname" description = "My nice packahe" version = "0.0.1" author = "Me" author_email = "[e-mail]" keywords = ["spam"] url = "https://github.com/me/pkgname" src_root = "" packages = ["pkgname"] 

where the [tool.enscons] section contains a lot of things familiar with the setuptools / distutils setup functions. Copying from here , the setup.py function may contain something like:

 #!/usr/bin/env python # Call enscons to emulate setup.py, installing if necessary. import sys, subprocess, os.path sys.path[0:0] = ['setup-requires'] try: import enscons.setup except ImportError: requires = ["enscons"] subprocess.check_call([sys.executable, "-m", "pip", "install", "-t", "setup-requires"] + requires) del sys.path_importer_cache['setup-requires'] # needed if setup-requires was absent import enscons.setup enscons.setup.setup() 

Finally, the SConstruct file might look something like this:

 # Build pkgname import sys, os import pytoml as toml import enscons, enscons.cpyext metadata = dict(toml.load(open('pyproject.toml')))['tool']['enscons'] # most specific binary, non-manylinux1 tag should be at the top of this list import wheel.pep425tags full_tag = next(tag for tag in wheel.pep425tags.get_supported() if not 'manylinux' in tag) env = Environment(tools=['default', 'packaging', enscons.generate, enscons.cpyext.generate], PACKAGE_METADATA=metadata, WHEEL_TAG=full_tag) ext_filename = os.path.join('pkgname', 'libcfile') extension = env.SharedLibrary(target=ext_filename, source=['pkgname/cfile.c']) py_source = Glob('pkgname/*.py') platlib = env.Whl('platlib', py_source + extension, root='') whl = env.WhlFile(source=platlib) # Add automatic source files, plus any other needed files. sdist_source=list(set(FindSourceFiles() + ['PKG-INFO', 'setup.py'] + Glob('pkgname/*', exclude=['pkgname/*.os']))) sdist = env.SDist(source=sdist_source) env.Alias('sdist', sdist) install = env.Command("#DUMMY", whl, ' '.join([sys.executable, '-m', 'pip', 'install', '--no-deps', '$SOURCE'])) env.Alias('install', install) env.AlwaysBuild(install) env.Default(whl, sdist) 

After that you can run

 sudo python setup.py install 

to compile the C extension and create the wheel, as well as install the python package or

 python setup.py sdist 

to create the source distribution.

I think you can basically do everything you can with the help of SCons in the SConstruct file.

Answer 2

See page: http://www.scons.org/wiki/PythonExtensions

I am using a slightly modified version to create pyrex-c extensions for python.

Answer 3

I use scons to create the setup.py file. So I created a template called setup.py.in and used scons to extend the template to create setup.py.

Here are some links to my project that does this:

setup.py.in template

The sconstruct

This computes a dictionary of key pairs, values ​​to replace in the setup.py.in template to generate setup.py.

So, the end user does two things:

 scons setup.py python setup.py install 

Warning: my tweak things are a bit messy, as I wrote this a while ago, but it should demonstrate the concept.

If you learn how to write the right scons tools, then this can be converted to a single target, for example:

 scons --pymod 

Here is a scons tool that creates a Python wrapper with SWIG:

 import SCons.Action from SCons.Script import EnsureSConsVersion SCons.Script.EnsureSConsVersion(0,96,92) SwigGenAction = SCons.Action.Action('$SWIGGENCOM', '$SWIGGENCOMSTR') def emitter(target, source, env): """ Add dependency from target to source """ env.Depends(target, source) return target, source def generate(env): """ Add builders and construction variables for the SwigGen builder. """ if 'SWIGCOM' not in env: raise SystemError("SCons build environment could not detect tool: swig") bld = env.Builder( action = SwigGenAction, emitter = emitter, target_factory = env.fs.File) env['BUILDERS']['SwigGen'] = bld env['SWIGGENCOM'] = env['SWIGCOM'] def exists(env): return env.Detect('swig') 

Now you can create shell code from the SConstruct file:

 foobar_cc = env.SwigGen("foobar_wrap.cc", "foobar.i") 

If you change foobar.i, it will regenerate foobar_wrap.cc. After that, you can write other tools to actually complete the python setup.py installation for you, so when -pymod is provided, it will build a python module.

Answer 4

The solution is to provide a custom cmdclass derived from distutils.cmd.Commmand that builds the module the way you want it:

 import distutils.cmd class build_py_cmd(distutils.cmd.Command): def initialize_options(self): pass def finalize_options(self): pass def run(self): print("Calling SCons to build the module") pbs.scons() setup(name = 'riak3k', packages = ['riak3k'], package_dir = {'': 'build/release'}, cmdclass = {'build_py': build_py_cmd},