pkgsrc is a framework for building over 17,000 open source software packages. It is the native package manager on SmartOS, NetBSD, and Minix, and is portable across 23 different operating systems. Use one package manager across all of your systems!
Joyent provide binary packages for SmartOS/illumos, Mac OS X, and Linux.
Once you are up and running with pkgsrc, one of the most common requests is a way to automatically build a number of packages, either a specific list plus their dependencies, or everything currently available.
There are a number of ways to accomplish this, but for this tutorial we will
pbulk, as it is used by a number of pkgsrc developers, and has
support for distributed and chrooted builds.
In this example we will build a set of
pkgsrc-2013Q4 packages, which has been
Please let us know if it doesn't work correctly on your platform.
First, have a think about where you will store pkgsrc, source tarballs,
packages, etc. This guide puts everything under
/content which makes it easy
to then mount just that directory and have everything below it available:
/content/bulklogis where pbulk saves the per-package build logs
/content/distfilesis where source tarballs are kept
makefragment files for configuration
/content/packagesis the top-level directory of binary packages
/content/pkgsrcis where pkgsrc is located
/content/scriptsto hold some miscellaneous scripts
Pre-create some required directories:
Then write a couple of
mk.conf files which will be used by the packaging tools.
Either use git..
At the present time (
2013Q4) there are a couple of patches you need to apply,
one for mksandbox to support some additional features, and one for pbulk to
support chroots and a couple of other bits we've developed at Joyent.
pbulk needs to be installed to its own prefix, from where it will manage the main build.
Then build the necessary pacakges
It is recommended that builds are done as an unprivileged user, which is normally
pbulk, so now would be a good time to create that user, usually with
something like this:
useradd/groupadd syntax for your system. The user can be set to
no-password, it will only be used via
su from the root user.
Next, check that the mksandbox script works on your system. It is designed to
be cross-platform, but on certain systems (e.g. OSX) there is no native support
for loopback mounts, and so you will first need to configure NFS in order to
share system directories to the chroot, usually with
/ -alldirs -maproot=root
Once you are happy the chroot is working as expected, write a couple of wrapper scripts to create and delete them with an optional argument with the name of the chroot, which will be used by pbulk. Below are the scripts I use.
Next step is to build the bootstrap for the target packages, i.e. the main
prefix you will be using. Again we use the bootstrap script, but here you
may want to tweak the settings - check the pkgsrc guide or the
output for more information.
If the prefix you want to build for (i.e.
/usr/pkg) is already in use on the
system, simply do the bootstrap inside a chroot.
I use something like this:
Now we're finally ready to configure pbulk. There is a single configuration file you need to edit, and I will show the changes I have made to it.
This section adds a
ulimit to stop runaway processes from hanging the build.
This section configures the location of the bulk build report. I upload my results to Joyent's Manta object store as it allows arbitrary storage plus distributed Unix queries on the data at a later time.
reuse_scan_results, it makes subsequent runs faster.
In this example I am using a single host which will perform concurrent builds
inside chroots, and so I need to unset
master_ip to localhost.
If you have multiple hosts, simple set
master_ip to a public address, and add
the list of slave IP addresses to
*_clients. They will need to be accessible
via SSH as root from the master, and will need to have their own installs of
/usr/pbulk as well as sharing the same
/content mount as the master, most
likely over NFS.
If you wish to completely disable any concurrency or distributed builds, set
master_mode=no, though note that the build with then run completely
single-threaded and will be much slower.
If you wish to publish to Manta, here are the settings you will need. I have
installed a local copy of the Manta tools to
/content/manta, as the upload
script will need them.
Configure the location where to rsync packages to and where to send the report.
If you are not using Manta, then you will want to set
an appropriate location.
Where to find the
/usr/pkg bootstrap tarball:
Configure build chroots. Here we set the paths to the
rmsandbox scripts we created earlier, and provide a basename of the chroot
directory. By setting
chroot_dir=/chroot/pkgsrc-2013Q4, pbulk will actually
You will want to experiment with the tradoffs between
MAKE_JOBS and the
number of chroots. Generally it will be better to have more chroots compared
to an increase in
MAKE_JOBS, as certain parts of the build will be single
threaded anyway (e.g. large configure scripts). However, you also need to be
aware of the increased disk I/O caused by too many chroots.
As long as you have everything correctly shared, there is nothing stopping you using distributed hosts and chroots, and it is highly recommended if you can as clearly it provides the best performance. With such a setup, at Joyent we are able to do full bulk builds of all 12,000 packages in pkgsrc in under 12 hours.
Finally, configure paths to the ones we have chosen.
One option not mentioned above is
limited_list. If you only want to build a
subset of packages rather than run a full bulk build, simply set
to a file containing paths to packages you want. It is worth doing this
initially anyway, just to check that everything is working fine, e.g.:
And add to pbulk.conf:
Assuming everything was done correctly, it should now just be a matter of
running the bulkbuild. If you have set the
chroot_* variables then this will
run chrooted at the appropriate places, so that your host system's
is not affected.
One of the benefits of the Joyent patch is that it adds support for different
configuration files, so if you really want to you can run concurrent instances
of pbulk. Just write separate
pbulk.conf files and then pass them as
bulkbuild. Again, we use this to run multiple builds across the
same hosts, all thanks to the chroot support.
There are some known issues, I will document them here as they are found.
On OSX, name resolution is broken inside a chroot. This is due to
mDNSResponder being used for DNS lookups, which relies on the
/var/run/mDNSResponder UNIX socket. Unfortunately, making that socket
available in the chroot (either by mounting or creating a proxy with
does not fix the issue, so I would welcome input on this.
For now you need to set
MASTER_SITE_OVERRIDE and then ensure that the IP
address for that mirror is set in
As you can see in my example
mksandbox script, I have to work around a race
condition where a previous scan run may still be cleaning up whilst a new one
is starting. For now I am simply sleeping until the chroot is free, but this
should be fixed properly, probably with process groups and waiting for them to