Upgrading to FreeBSD 7.1

I finally had some time to upgrade my FreeBSD servers to 7.1. This followed the upgrade process I originally outlined after my first attempt. However, I've learnt a few things since then...

First, there's no need for the cvsup port, as a similar capability is part of the base system, called csup, which operates identically (for all practical purposes). So the upgrade steps now are:

  1. Get the sources:
          csup -g -L 2 /usr/local/etc/cvsup/src-supfile
        
  2. Then proceed with the following steps:
          cd /usr/src
          ./usr.sbin/mergemaster/mergemaster.sh -p
          make buildworld
          make buildkernel KERNCONF=CRIMSON
          make installkernel KERNCONF=CRIMSON
          [reboot to single-user ]
          fsck -p
          mount -u /
          mount -a -t ufs
          swapon -a
          cd /usr/src
          make installworld
          mergemaster -U # replaces unaltered files automatically
          # get rid of old, unused, files
          make -DBATCH_DELETE_OLD_FILES delete-old
          make -DBATCH_DELETE_OLD_FILES delete-old-libs
          # Enable networking so that package downloads will work
          /etc/rc.d/hostname start
          /etc/rc.d/netif start
          /etc/rc.d/routing start
          # get ports tree up-to-date
          csup -g -L 2 /usr/local/etc/cvsup/ports-supfile
          # replace all packages 
          sh /home/mark/bin/pkg_upgrade.sh
          [Reboot into multi-user (normality has now been resumed...)]
        

Some warnings about the above process:

So far, FreeBSD 7.1 has been working just fine. However, I have noticed that the memory usage, as shown by top, is much more than it used to be on FreeBSD 6.3. This is very noticable on crimson, as it only has 64MB. Here's a typical top display:

  last pid: 71173;  load averages:  0.05,  0.03,  0.01   up 17+19:50:57  17:01:35
  47 processes:  1 running, 46 sleeping
  CPU:  0.8% user,  0.0% nice,  0.8% system,  0.8% interrupt, 97.7% idle
  Mem: 21M Active, 11M Inact, 21M Wired, 2060K Cache, 14M Buf, 3012K Free
  Swap: 128M Total, 19M Used, 109M Free, 14% Inuse

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  71173 mark        1  96    0  3488K  1428K RUN      0:00  1.24% top
  17349 root        1  96    0  4664K  1172K select   5:45  0.00% ntpd
  17418 root        1  96    0  6836K  1780K select   4:19  0.00% sendmail
  17395 root        1  96    0  6852K  2172K select   3:53  0.00% httpd
  17166 root        1  96    0  3176K   740K select   0:43  0.00% syslogd
  17428 root        1   8    0  3204K   740K nanslp   0:37  0.00% cron
  17722 bind        4  96    0 26464K  8056K select   0:29  0.00% named
  17241 root        1  96    0  3204K   832K select   0:08  0.00% rpcbind
  17295 root        1   4    0  3120K   500K -        0:05  0.00% nfsd
  17422 smmsp       1  20    0  5948K  1396K pause    0:05  0.00% sendmail
  17382 dhcpd       1  96    0  3120K  1268K select   0:05  0.00% dhcpd
  71024 mark        1  96    0  8352K  3012K select   0:02  0.00% sshd
  71026 mark        1  20    0  4448K  2052K pause    0:01  0.00% tcsh
  17294 root        1   4    0  3120K   592K accept   0:01  0.00% nfsd
  17413 root        1  96    0  5700K  1268K select   0:01  0.00% sshd
  71021 root        1   4    0  8352K  3028K sbwait   0:01  0.00% sshd
  66257 www         1   4    0  6852K  2488K accept   0:01  0.00% httpd

You can see that 19M of swap is in use. With 6.3, the amount of swap was normally around 2-3M. However, this does not seem to cause any performance problems, and may be just down to how the new malloc works in FreeBSD 7.1, or possibly because the new version of bind needs 26M!

New FreeBSD kernel configuration files

As part of the upgrade to FreeBSD 7.1, I re-worked the kernel configuration files for crimson and chrome to use the config "include file" facility, along with nooptions, nodevice etc. This allows a configuration file to be based on GENERIC, but removes elements which are not required. I was in two minds about doing this, as the level of removal from the GENERIC config file for both crimson and chrome is large, but it's done now.

One tool I would have liked, but could not find, was something that would output the final configuration file, with elements added and removed. I couldn't find anything that did this, so I wrote one in python. Using this tool, the CRIMSON effective configuration file looks like:

  ident CRIMSON
  #
  machine i386
  #
  #
  cpu I486_CPU
  cpu I586_CPU
  #
  option UFS_ACL
  option KBD_INSTALL_CDEV
  option KSE
  option _KPOSIX_PRIORITY_SCHEDULING
  option STOP_NMI
  option PREEMPTION
  option MD_ROOT
  option MSDOSFS
  option NFSLOCKD
  option SOFTUPDATES
  option ATA_STATIC_ID
  option PSEUDOFS
  option SYSVSEM
  option SCSI_DELAY=5000
  option STACK
  option INET
  option SCHED_ULE
  option COMPAT_FREEBSD6
  option NFS_ROOT
  option UFS_GJOURNAL
  option COMPAT_FREEBSD5
  option COMPAT_FREEBSD4
  option ISAPNP
  option GEOM_BSD
  option SYSVMSG
  option SYSVSHM
  option PROCFS
  option CD9660
  option GEOM_PART_GPT
  option COMPAT_43TTY
  option FFS
  option NFSCLIENT
  option UFS_DIRHASH
  option SCTP
  option GEOM_MBR
  option GEOM_LABEL
  option ADAPTIVE_GIANT
  option NFSSERVER
  #
  makeoption DEBUG=-g
  #
  device ppc
  device eisa
  device random
  device pty
  device atkbd
  device pci
  device psm
  device bpf
  device md
  device atadisk
  device uart_ns8250
  device rl
  device mem
  device ata
  device uart
  device ppbus
  device loop
  device isa
  device kbdmux
  device ppi
  device ether
  device atkbdc
  device splash
  device atapifd
  device fdc
  device miibus
  device io
  device atapicd
  device sis
  device plip
  device vga
  device sio
  device npx
  device lpt
  device sc

Here's the python code:

  """
  NAME
    config.py - Displays effective version of FreeBSD kernel config

  SYNOPSIS
    python config.py [-n] config_file

    -n - do not attempt to process the DEFAULT configuration file

  DESCRIPTION
    config.py will process a FreeBSD kernel configuration file and
    output the effective configuration generated, taking account of
    values provided in any include files, and the impact of the
    various 'no..' keywords which are used to remove previously defined
    values.

  NOTES

  MODIFICATION HISTORY
    Mnemonic   Rel   Date     Who
    config     1.0   20090216 mpw
       Created.

  """
  import sys
  import getopt
  import os
  import shlex


  class Lex(shlex.shlex):
      def sourcehook(self,filename):
          f = open(filename)
          return (filename,f)

  class Config:
      def __init__(self):
          self.lex = None
          self.elements = ((("ident",),dict(),1),
                           (("machine",),dict(),1),
                           (("maxusers",),dict(),1),
                           (("cpu","nocpu"),dict(),0),
                           (("option","options","nooption","nooptions"),dict(),0),
                           (("makeoption","makeoptions","nomakeoption",
                             "nomakeoptions"),dict(),0),
                           (("device","nodevice","nodevices"),dict(),0))
          return

      def process_element(self,tok,target):
          for e in self.elements:
              if tok in e[0]:
                  if e[2] == 1: e[1].clear() # only one element value allowed
                  if tok[0:2] == "no":
                      if target in e[1]: del e[1][target]
                  else:
                      e[1][target] = 0
          return

      def build(self,filename,process_default):
          try:
              f = open(filename)
          except IOError:
              return 1

          self.lex = Lex(f)
          # extend word token characters for makeoptions
          self.lex.wordchars = self.lex.wordchars+"-="
          self.lex.source = "include"     # keyword for include file
          self.lex.dir = os.path.dirname(filename)
          self.lex.commenters = "#"       # comment character
          if process_default:
              try:
                  fd = open("DEFAULTS")
              except IOError:
                  return 2
              self.lex.push_source(fd)
          tok = "start"
          while len(tok) != 0:
              tok = self.lex.get_token()
              target = self.lex.get_token()
              self.process_element(tok,target)
          return 0

      def display(self):
          for e in self.elements:
              for k in e[1].keys():
                  print e[0][0],k
              print "#"
          return

  if __name__ == '__main__':
      process_default = True
      try:
          opts,args = getopt.getopt(sys.argv[1:],'-n')
          for o,v in opts:
              if o == '-n': process_default = False
      except getopt.GetoptError,e:
          print "config.py: illegal argument:",e.opt
          sys.exit(1)

      if len(args) == 0:
          print >>sys.stderr,"config.py: usage: python [-n] config.py "\
                "<CONFIG_FILE>"
          sys.exit(1)

      config = Config()
      result = config.build(args[0],process_default)
      if result == 0:
          config.display()
      elif  result == 1:
          print >>sys.stderr,"config.py: Unable to open",args[0]
      elif result == 2:
          print >>sys.stderr,"config.py: Unable to open DEFAULT config file"\
                " (consider -n)"