Portupgrade for binary upgrades

A number of the applications on the FreeBSD mail/web server, crimson, were significantly out of date. Not the applications, such as apache or sendmail that were exposed to the outside world, but the applications used internally. Since crimson has a lowly 120MHz Pentium, performing a ports-based (i.e. source) upgrade via portupgrade would take far too long.

Portupgrade offers a binary-only upgrade option via the -PP command argument, which locates and downloads pre-compiled packages from a FreeBSD FTP site. Portupgrade uses the ports tree to determine the version of the latest packages required. On trying this approach, I found that portupgrade was unable to locate any of the up-to-date packages. This seemed to be because portupgrade was looking for the packages in directories named:

  ftp://ftp.uk.FreeBSD.org/pub/FreeBSD/ports/i386/packages-4.10-release/All
  ftp://ftp.uk.FreeBSD.org/pub/FreeBSD/ports/i386/packages-4.10-release/Latest

As far as I can tell, these are the locations where the binary packages were placed at the time of the 4.10 release. The latest set of packages, related to the current state of the ports tree can be found in:

  ftp://ftp.uk.FreeBSD.org/pub/FreeBSD/ports/i386/packages-4-stable/All
  ftp://ftp.uk.FreeBSD.org/pub/FreeBSD/ports/i386/packages-4-stable/Latest

I needed to tell portupgrade to use the stable FTP directories. First, I tried setting the OS_PKGBRANCH variable in the /usr/local/etc/pkgtools.conf file to "4-stable". This had absolutely no effect. Delving into the portupgrade code, I discovered that portupgrade used uname to identify the running FreeBSD version. The documentation for uname, indicated that it was possible to control its output by setting environment variables. To persuade portupgrade that it was running on a stable branch, I set the environment variable UNAME_r to 4.10-STABLE, prior to running portupgrade.

This had the desired effect, and portupgrade -aPP was able to find most of the packages related to the current ports tree.

In summary, the commands necessary to perform a binary upgrade of the installed ports on a 4.10 FreeBSD system are as follows:

  $ su 
  [root password]
  # setenv UNAME_r "4.10-STABLE"
  # portupgrade -aPP

The -PP argument is likely to leave a number of ports un-upgraded, as the packages are not available. It's probably best to use the -P argument, as this will compile from ports when a package is not available.

If you want to run an upgrade via portupgrade without any user interaction, you need to set the argument BATCH=yes in the /usr/local/etc/pkgtools.conf file. Note that the MAKE_ARGS array also includes an option to compile sendmail without IPV6 support (to workaround the broken DNS in the Alcatel Speedtouch router - see this).

  MAKE_ARGS = {
    '*' => 'BATCH=yes',
    'mail/sendmail => 'SENDMAIL_WITHOUT_IPV6=yes'
  }

While the portupgrade suite is extremely useful, there is something I wanted to add: an ability to find out what packages and/or ports would also need to be installed to support a newly installed package. I wrote portquery to satisfy this need. Here's the source code. It's my first ruby program, so it's pretty clunky. It also needs portupgrade to be installed, as it uses a lot of features offered by the package.

Addendum - 15th February, 2005

A more recent version of portupgrade made the decision that any custom MAKE_ARGS in the /usr/local/etc/pkgtools.conf file meant that ports should be used instead of packages, unless the port name was defined in the USE_PKGS_ONLY array. That is, the -P argument was ignored because I had specified BATCH=yes in MAKE_ARGS.

While there might be good reasons for this decision in general, it doesn't make much sense when the only MAKE_ARGS setting for a port is BATCH=yes. To work around this, I removed the BATCH=yes from /usr/local/etc/pkgtools.conf and added it to /etc/make.conf, which contains all the local make defaults.

Background on ports and packages

The above investigation had the side-effect of clearing up my hazy view of how the various tools related to ports and packages were connected.

The ports tree contains a file called INDEX, which records all the information related to all the ports. This information is sourced from the ports themselves, and hence, as ports are modified, the INDEX file becomes out of date. This is because the INDEX file on cvsup servers is only updated once a week or so (that's what the man page for portsdb says).

It's possible to rebuild the local INDEX file by:

  cd /usr/ports
  make index

However, on my server this can take four or five hours to complete. Using make fetchindex, you can retrieve the latest pre-built INDEX file, but this (possibly) will not reflect the current state of your ports tree. It's worked for me, so far.

Portupgrade (and its friends, portsclean and portversion), use a fast access version of the INDEX file, called INDEX.db. This is created (and updated) by portsdb -u. It is recommended in the portsdb manpage to rebuild the INDEX file after every cvsup, using portsdb -Uu. Remember, the -U option is the same as make index, and can take an inordinately long time.

The other key database related to packages is located in /var/db/pkg/pkgdb.db. This database records information regarding the packages actually installed on the machine. The package database is maintained by pkgdb, usually automatically run by portupgrade as required. It is recommended that pkgdb -F be run after a binary portupgrade to ensure all dependencies are recorded properly.

This diagram might be useful to understand the relationships between the various ports and package utilities and their support files.