Upgrading to FreeBSD 6.3

Recently, FreeBSD 6.3 was released, and I upgraded all my machines to run this new version. Since I now use a fast machine to build the world and kernels for all machines, that is a relatively quick process.

However, when I'd last upgraded (6.1 -> 6.2) I'd not bothered to update any of the ports, since it takes ages on the crimson (120MHz) and chrome (100MHz). I'd also pruned my ports by getting rid of portupgrade (which meant I could get rid of ruby), so I looked for another way of mass updating ports. I tried portmaster, since this had no pre-requisites. I found it somewhat quirky and it lacked the ability of portupgrade to use packages instead of ports.

I figured I'd develop my own simple package upgrade procedure, using the existing base tools (pkg_info and the like). I wanted to use packages as much as possible; much quicker to download than to compile, but there were some ports I had to compile as otherwise they required too many other packages; emacs for example, which I compile WITHOUT_X11.

The result is a shell script pkg_upgrade.sh, which acts in the following way:

It identifies all currently installed packages/ports, and issues pkg_delete -f commands to remove them. Next, it identifies leaf ports (that is, those with no other packages dependent on them), excluding those which have been defined to be built from ports. Once this winnowing is complete, it issues a pkg_add -r command for each package. Finally, those packages that must be build from ports are compiled and installed.

Since the information defined in the package database is used to drive the script, the commands to perform the actions described above are not executed right away, but written to a temporary script file. pkg_upgrade.sh will execute this temporary script, unless told not to, in which case you can examine the generated commands prior to running them.

The script is shown below:

#     pkg_upgrade
#     pkg_upgrade.sh [-n] [-f] [-p "port list"]
#     If the switch -n is specified, the commands necessary to perform the
#     upgrade are not executed.
#     -f will cause pkg_upgrade to unconditionally overwrite the save file
#      containing the current list of packages. 
#     -p "port list" allows the definition of a list of ports that
#     must be compiled, rather than downloaded as packages.  If more
#     than one port is specified, the list must be whitespace separated
#     and enclosed in quotes.  It will override any script set list
#     of ports.
#    This script is intended to be run following a FreeBSD OS
#    upgrade.  It will delete all the installed packages, then
#    re-install all leaf packages (i.e. those with no dependencies)
#    Where packages cannot be used, due to the need to specify compile-
#    time options, a list of ports may be specified.  An attempt is made
#    to identify these and not download the corresponding package.  
#    Such ports are built using the normal make && make install.
#    This script is best run following a single-user reboot into the
#    new version of the OS.  Then, issue the following commands:
#      fsck -p
#      mount -u /
#      mount -t ufs -a
#      swapon -a
#      /etc/rc.d/hostname start
#      /etc/rc.d/netif start
#      /etc/rc.d/routing start
#      csup -g -L 2 /usr/local/etc/cvsup/ports-supfile # update ports tree
#   The pkg_update.sh script can then be run.
#  Mnemonic    Date     Rel   Who
#  pkg_upgrade 20090129 1.0   mpw
#    Created.
#  pkg_upgrade 20100405 1.1   mpw
#    Issue 'make clean' before building ports
#    If port name and package name are different, try to add both

today=`date +'%y%m%d'`
host=`uname -n`

if [ "$[host}" = "" ]; then
    echo "${script}: no host name set; not proceeding."
    exit 1

# define packages that must be built from ports, for each host supported.
# this section needs to be deleted/modifed for different hosts.
if [ "${host}" = "crimson" ]; then
    ports="emacs ispell samba34 cyrus-sasl2-saslauthd py-libxml2"
elif [ "${host}" = "chrome" ]; then
    ports="emacs ispell samba3 cyrus-sasl2-saslauthd py-libxml2"
elif [ "${host}" = "topaz" ]; then
    ports="samba34 ispell"

# get arguments
while [ $# -gt 0 ]; do
    case $1 in
        "-p" ) ports=$2;shift
        "-n" ) execute=false
        "-f" ) safe=false
        * ) echo "$1: unrecognised switch, quitting..."
            exit 1

# get list of all installed packages
pkgs=`pkg_info -E -x '.*'`

# remember them in case of disaster
if [ -r ${backup_file} ] && ${safe}; then
    echo "${script}: package list backup file already exists; use -f to force overwrite"
    exit 1
    echo ${pkgs} >${backup_file}

# ensure exec shell script file is empty
echo "#!/bin/sh" >${exec}

# delete all packages
echo pkg_delete -af >>${exec}

# for each package, identify those that are leaves and pkg_add them
# if a package exists in the list of ports, it will not be pkg_add'ed,
# but built from ports later
for pkg in ${pkgs}
    deps=`pkg_info -q -R ${pkg}`
    if [ "${deps}" = "" ]; then
        origin=`pkg_info -q -o ${pkg}`
        for port in ${ports}
          if [ ${port} = ${basepkg} ]; then
              portlist=${origin}" "${portlist}
        if [ "${build}" = "${basepkg}" ]; then
            if [ "${basepkg}" = "${pkg%-*}" ]; then
                echo "pkg_add -r ${basepkg}" >>${exec} 
                echo "pkg_add -r ${basepkg} || pkg_add -r ${pkg%-*}" >>${exec}

# now build from ports
for port in ${portlist}
  echo cd ${ports_dir}/${port} >>${exec}
  echo "make clean && make && make install" >>${exec}

# the commands necessary for the package upgrade are now in the 
# script ${exec}
if [ -r ${exec} ]; then
    if  ${execute} ; then
        echo "${script}: upgrade running... logfile in ${logfile}"
        sh -x ${exec} | tee ${logfile}
        echo "${script}: pkg_upgrade commands are in ${exec}"
    echo "${script}: package upgrade script (${exec}) is unreadable!"
    exit 1

Addendum - 5th April, 2010

You may be of the opinion that the pkg_upgrade.sh script is a bit hokey. In this you would be right. After this entry was originally written, I'd modifed it a bit, but since I don't fully understand the ports infrastructure, it is still a bit of a bodge.

So, since portmaster had been enhanced to support the installation of ports via packages, I decided to give it another try. The portmaster man page documents the steps to re-install all ports from scratch, but this needs some slight modifications to make use of the new package functionality:

  1. portmaster --list-origins > ~/installed-port-list
  2. Update your ports tree.
  3. portmaster --clean-distfiles-all # remove all distfiles
  4. portmaster --check-port-dbdir # check for stale entries
  5. pkg_delete '*'
  6. rm -rf /usr/local/lib/compat/pkg
  7. Manually check /usr/local and /var/db/pkg to make sure that they are really empty
  8. Re-install portmaster.
  9. Remove any ports that _must_ be compiled from installed-ports-list, say port1 and port2.
  10. portmaster -PD `cat ~/installed-ports-list`
  11. portmaster port1 port2

I tried this on topaz after upgrading to FreeBSD 7.3. It worked very well, except for some reason the mutt port had a corrupt record (pkg_info complained about a pkgdep line without argument). Anyway, I didn't want mutt, so I deleted it and with it the problem.

Addendum 2012-09-12

I've been messing about at this again. Latest attempt is in this entry.