Practical rc.d scripting in BSD

The historical BSD had a monolithic startup script, /etc/rc. It was invoked by init(8) at system boot time and performed all userland tasks required for multi-user operation: checking and mounting file systems, setting up the network, starting daemons, and so on. The precise list of tasks was not the same in every system; admins needed to customize it. With few exceptions, /etc/rc had to be modified, and true hackers liked it.

The real problem with the monolithic approach was that it provided no control over the individual components started from /etc/rc. For instance, /etc/rc could not restart a single daemon. The system admin had to find the daemon process by hand, kill it, wait until it actually exited, then browse through /etc/rc for the flags, and finally type the full command line to start the daemon again. The task would become even more difficult and prone to errors if the service to restart consisted of more than one daemon or demanded additional actions. In a few words, the single script failed to fulfil what scripts are for: to make the system admin’s life easier.

Later there was an attempt to split out some parts of /etc/rc for the sake of starting the most important subsystems separately. The notorious example was /etc/netstart to bring up networking. It did allow for accessing the network from single-user mode, but it did not integrate well into the automatic startup process because parts of its code needed to interleave with actions essentially unrelated to networking. That was why /etc/netstart mutated into /etc/rc.network. The latter was no longer an ordinary script; it comprised of large, tangled sh(1) functions called from /etc/rc at different stages of system startup. However, as the startup tasks grew diverse and sophisticated, the "quasi-modular" approach became even more of a drag than the monolithic /etc/rc had been.

Without a clean and well-designed framework, the startup scripts had to bend over backwards to satisfy the needs of rapidly developing BSD-based operating systems. It became obvious at last that more steps are necessary on the way to a fine-grained and extensible rc system. Thus BSD rc.d was born. Its acknowledged fathers were Luke Mewburn and the NetBSD community. Later it was imported into FreeBSD. Its name refers to the location of system scripts for individual services, which is in /etc/rc.d. Soon we will learn about more components of the rc.d system and see how the individual scripts are invoked.

The basic ideas behind BSD rc.d are fine modularity and code reuse. Fine modularity means that each basic "service" such as a system daemon or primitive startup task gets its own sh(1) script able to start the service, stop it, reload it, check its status. A particular action is chosen by the command-line argument to the script. The /etc/rc script still drives system startup, but now it merely invokes the smaller scripts one by one with the start argument. It is easy to perform shutdown tasks as well by running the same set of scripts with the stop argument, which is done by /etc/rc.shutdown. Note how closely this follows the Unix way of having a set of small specialized tools, each fulfilling its task as well as possible. Code reuse means that common operations are implemented as sh(1) functions and collected in /etc/rc.subr. Now a typical script can be just a few lines' worth of sh(1) code. Finally, an important part of the rc.d framework is rcorder(8), which helps /etc/rc to run the small scripts orderly with respect to dependencies between them. It can help /etc/rc.shutdown, too, because the proper order for the shutdown sequence is opposite to that of startup.

The BSD rc.d design is described in the original article by Luke Mewburn, and the rc.d components are documented in great detail in the respective manual pages. However, it might not appear obvious to an rc.d newbie how to tie the numerous bits and pieces together to create a well-styled script for a particular task. Therefore this article will try a different approach to describe rc.d. It will show which features should be used in a number of typical cases, and why. Note that this is not a how-to document because our aim is not at giving ready-made recipes, but at showing a few easy entrances into the rc.d realm. Neither is this article a replacement for the relevant manual pages. Do not hesitate to refer to them for more formal and complete documentation while reading this article.

There are prerequisites to understanding this article. First of all, you should be familiar with the sh(1) scripting language to master rc.d. In addition, you should know how the system performs userland startup and shutdown tasks, which is described in rc(8).

This article focuses on the FreeBSD branch of rc.d. Nevertheless, it may be useful to NetBSD developers, too, because the two branches of BSD rc.d not only share the same design but also stay similar in their aspects visible to script authors.

A little consideration before starting $EDITOR will not hurt. To write a well-tempered rc.d script for a system service, we should be able to answer the following questions first:

From the examples that follow we will see why it is important to know the answers to these questions.

The following script just emits a message each time the system boots up:

Things to note are:

➊ An interpreted script should begin with the magic "shebang" line. That line specifies the interpreter program for the script. Due to the shebang line, the script can be invoked exactly like a binary program provided that it has the execute bit set. (See chmod(1).) For example, a system admin can run our script manually, from the command line:

➋ In /etc/rc.subr, a number of sh(1) functions are defined for an rc.d script to use. The functions are documented in rc.subr(8). While it is theoretically possible to write an rc.d script without ever using rc.subr(8), its functions prove extremely handy and make the job an order of magnitude easier. So it is no surprise that everybody resorts to rc.subr(8) in rc.d scripts. We are not going to be an exception.

An rc.d script must "source"/etc/rc.subr (include it using “.”) before it calls rc.subr(8) functions so that sh(1) has an opportunity to learn the functions. The preferred style is to source /etc/rc.subr first of all.

➌ The mandatory variable name specifies the name of our script. It is required by rc.subr(8). That is, each rc.d script must set name before it calls rc.subr(8) functions.

Now it is the right time to choose a unique name for our script once and for all. We will use it in a number of places while developing the script. The content of the name variable needs to match the script name, some parts of FreeBSD (e.g., service jails and the cpuset feature of the rc framework) depend upon this. As such the filename shall also not contain characters which may be troublesome in scripting (e.g., do not use a hyphen "-" and others).

➍ The main idea behind rc.subr(8) is that an rc.d script provides handlers, or methods, for rc.subr(8) to invoke. In particular, start, stop, and other arguments to an rc.d script are handled this way. A method is a sh(1) expression stored in a variable named argument_cmd, where argument corresponds to what can be specified on the script’s command line. We will see later how rc.subr(8) provides default methods for the standard arguments.

➎ We should keep in mind that rc.subr(8) provides default methods for the standard arguments. Consequently, we must override a standard method with a no-op sh(1) expression if we want it to do nothing.

➏ The body of a sophisticated method can be implemented as a function. It is a good idea to make the function name meaningful.

➐ This call to rc.subr(8) loads rc.conf(5) variables. Our script makes no use of them yet, but it still is recommended to load rc.conf(5) because there can be rc.conf(5) variables controlling rc.subr(8) itself.

➑ Usually this is the last command in an rc.d script. It invokes the rc.subr(8) machinery to perform the requested action using the variables and methods our script has provided.

Now let us add some controls to our dummy script. As you may know, rc.d scripts are controlled with rc.conf(5). Fortunately, rc.subr(8) hides all the complications from us. The following script uses rc.conf(5) via rc.subr(8) to see whether it is enabled in the first place, and to fetch a message to show at boot time. These two tasks in fact are independent. On the one hand, an rc.d script can just support enabling and disabling its service. On the other hand, a mandatory rc.d script can have configuration variables. We will do both things in the same script though:

What changed in this example?

➊ The variable rcvar specifies the name of the ON/OFF knob variable.

➋ Now load_rc_config is invoked earlier in the script, before any rc.conf(5) variables are accessed.

➌ A warning will be emitted by run_rc_command if rcvar itself is set, but the indicated knob variable is unset. If your rc.d script is for the base system, you should add a default setting for the knob to /etc/defaults/rc.conf and document it in rc.conf(5). Otherwise it is your script that should provide a default setting for the knob. The canonical approach to the latter case is shown in the example.

➍ Now the message to be shown at boot time is no longer hard-coded in the script. It is specified by an rc.conf(5) variable named dummy_msg. This is a trivial example of how rc.conf(5) variables can control an rc.d script.

➎ Here we use dummy_msg to actually control our script, i.e., to emit a variable message. Use of a shell function is overkill here, since it only runs a single command; an equally valid alternative is:

We said earlier that rc.subr(8) could provide default methods. Obviously, such defaults cannot be too general. They are suited for the common case of starting and shutting down a simple daemon program. Let us assume now that we need to write an rc.d script for such a daemon called mumbled. Here it is:

Pleasingly simple, isn’t it? Let us examine our little script. The only new thing to note is as follows:

➊ The command variable is meaningful to rc.subr(8). If it is set, rc.subr(8) will act according to the scenario of serving a conventional daemon. In particular, the default methods will be provided for such arguments: start, stop, restart, poll, and status.

The daemon will be started by running $command with command-line flags specified by $mumbled_flags. Thus all the input data for the default start method are available in the variables set by our script. Unlike start, other methods may require additional information about the process started. For instance, stop must know the PID of the process to terminate it. In the present case, rc.subr(8) will scan through the list of all processes, looking for a process with its name equal to procname. The latter is another variable of meaning to rc.subr(8), and its value defaults to that of command. In other words, when we set command, procname is effectively set to the same value. This enables our script to kill the daemon and to check if it is running in the first place.

For more detailed information on default methods, refer to rc.subr(8).

Let us add some meat onto the bones of the previous script and make it more complex and featureful. The default methods can do a good job for us, but we may need some of their aspects tweaked. Now we will learn how to tune the default methods to our needs.

➊ Additional arguments to $command can be passed in command_args. They will be added to the command line after $mumbled_flags. Since the final command line is passed to eval for its actual execution, input and output redirections can be specified in command_args.

➋ A good-mannered daemon should create a pidfile so that its process can be found more easily and reliably. The variable pidfile, if set, tells rc.subr(8) where it can find the pidfile for its default methods to use.

➌ If the daemon cannot run unless certain files exist, just list them in required_files, and rc.subr(8) will check that those files do exist before starting the daemon. There also are required_dirs and required_vars for directories and environment variables, respectively. They all are described in detail in rc.subr(8).

➍ We can customize signals to send to the daemon in case they differ from the well-known ones. In particular, sig_reload specifies the signal that makes the daemon reload its configuration; it is SIGHUP by default. Another signal is sent to stop the daemon process; the default is SIGTERM, but this can be changed by setting sig_stop appropriately.

➎➏ Performing additional tasks before or after the default methods is easy. For each command-argument supported by our script, we can define argument_precmd and argument_postcmd. These sh(1) commands are invoked before and after the respective method, as it is evident from their names.

➐ If we would like to implement custom arguments, which can also be thought of as commands to our script, we need to list them in extra_commands and provide methods to handle them.

➑⓮ Our script supports two non-standard commands, plugh and xyzzy. We saw them listed in extra_commands, and now it is time to provide methods for them. The method for xyzzy is just inlined while that for plugh is implemented as the mumbled_plugh function.

Non-standard commands are not invoked during startup or shutdown. Usually they are for the system admin’s convenience. They can also be used from other subsystems, e.g., devd(8) if specified in devd.conf(5).

The full list of available commands can be found in the usage line printed by rc.subr(8) when the script is invoked without arguments. For example, here is the usage line from the script under study:

⓭ A script can invoke its own standard or non-standard commands if needed. This may look similar to calling functions, but we know that commands and shell functions are not always the same thing. For instance, xyzzy is not implemented as a function here. In addition, there can be a pre-command and post-command, which should be invoked orderly. So the proper way for a script to run its own command is by means of rc.subr(8), as shown in the example.

➒ A handy function named checkyesno is provided by rc.subr(8). It takes a variable name as its argument and returns a zero exit code if and only if the variable is set to YES, or TRUE, or ON, or 1, case insensitive; a non-zero exit code is returned otherwise. In the latter case, the function tests the variable for being set to NO, FALSE, OFF, or 0, case insensitive; it prints a warning message if the variable contains anything else, i.e., junk.

Keep in mind that for sh(1) a zero exit code means true and a non-zero exit code means false.

➓ We can affect the flags to be passed to $command by modifying rc_flags in $start_precmd.

⓫ In certain cases we may need to emit an important message that should go to syslog as well. This can be done easily with the following rc.subr(8) functions: debug, info, warn, and err. The latter function then exits the script with the code specified.

⓬ The exit codes from methods and their pre-commands are not just ignored by default. If argument_precmd returns a non-zero exit code, the main method will not be performed. In turn, argument_postcmd will not be invoked unless the main method returns a zero exit code.

After a script has been written, it needs to be integrated into rc.d. The crucial step is to install the script in /etc/rc.d (for the base system) or /usr/local/etc/rc.d (for ports). Both bsd.prog.mk and bsd.port.mk provide convenient hooks for that, and usually you do not have to worry about the proper ownership and mode. System scripts should be installed from src/libexec/rc/rc.d through the Makefile found there. Port scripts can be installed using USE_RC_SUBR as described in the Porter’s Handbook.

However, we should consider beforehand the place of our script in the system startup sequence. The service handled by our script is likely to depend on other services. For instance, a network daemon cannot function without the network interfaces and routing up and running. Even if a service seems to demand nothing, it can hardly start before the basic filesystems have been checked and mounted.

We mentioned rcorder(8) already. Now it is time to have a close look at it. In a nutshell, rcorder(8) takes a set of files, examines their contents, and prints a dependency-ordered list of files from the set to stdout. The point is to keep dependency information inside the files so that each file can speak for itself only. A file can specify the following information:

It is no surprise that rcorder(8) can handle only text files with a syntax close to that of sh(1). That is, special lines understood by rcorder(8) look like sh(1) comments. The syntax of such special lines is rather rigid to simplify their processing. See rcorder(8) for details.

Besides using rcorder(8) special lines, a script can insist on its dependency upon another service by just starting it forcibly. This can be needed when the other service is optional and will not start by itself because the system admin has disabled it mistakenly in rc.conf(5).

With this general knowledge in mind, let us consider the simple daemon script enhanced with dependency stuff:

As before, detailed analysis follows:

➊ That line declares the names of "conditions" our script provides. Now other scripts can record a dependency on our script by those names.

➋➌ So our script indicates which "conditions" provided by other scripts it depends on. According to the lines, our script asks rcorder(8) to put it after the script(s) providing DAEMON and cleanvar, but before that providing LOGIN.

➍ As we remember from the above text, rcorder(8) keywords can be used to select or leave out some scripts. Namely any rcorder(8) consumer can specify through -k and -s options which keywords are on the "keep list" and "skip list", respectively. From all the files to be dependency sorted, rcorder(8) will pick only those having a keyword from the keep list (unless empty) and not having a keyword from the skip list.

In FreeBSD, rcorder(8) is used by /etc/rc and /etc/rc.shutdown. These two scripts define the standard list of FreeBSD rc.d keywords and their meanings as follows:

➎ To begin with, force_depend should be used with much care. It is generally better to revise the hierarchy of configuration variables for your rc.d scripts if they are interdependent.

If you still cannot do without force_depend, the example offers an idiom of how to invoke it conditionally. In the example, our mumbled daemon requires that another one, frotz, be started in advance. However, frotz is optional, too; and rcorder(8) knows nothing about such details. Fortunately, our script has access to all rc.conf(5) variables. If frotz_enable is true, we hope for the best and rely on rc.d to have started frotz. Otherwise we forcibly check the status of frotz. Finally, we enforce our dependency on frotz if it is found to be not running. A warning message will be emitted by force_depend because it should be invoked only if a misconfiguration has been detected.

When invoked during startup or shutdown, an rc.d script is supposed to act on the entire subsystem it is responsible for. E.g., /etc/rc.d/netif should start or stop all network interfaces described by rc.conf(5). Either task can be uniquely indicated by a single command argument such as start or stop. Between startup and shutdown, rc.d scripts help the admin to control the running system, and it is when the need for more flexibility and precision arises. For instance, the admin may want to add the settings of a new network interface to rc.conf(5) and then to start it without interfering with the operation of the existing interfaces. Next time the admin may need to shut down a single network interface. In the spirit of the command line, the respective rc.d script calls for an extra argument, the interface name.

Fortunately, rc.subr(8) allows for passing any number of arguments to script’s methods (within the system limits). Due to that, the changes in the script itself can be minimal.

How can rc.subr(8) gain access to the extra command-line arguments. Should it just grab them directly? Not by any means. Firstly, an sh(1) function has no access to the positional parameters of its caller, but rc.subr(8) is just a sack of such functions. Secondly, the good manner of rc.d dictates that it is for the main script to decide which arguments are to be passed to its methods.

So the approach adopted by rc.subr(8) is as follows: run_rc_command passes on all its arguments but the first one to the respective method verbatim. The first, omitted, argument is the name of the method itself: start, stop, etc. It will be shifted out by run_rc_command, so what is $2 in the original command line will be presented as $1 to the method, and so on.

To illustrate this opportunity, let us modify the primitive dummy script so that its messages depend on the additional arguments supplied. Here we go:

What essential changes can we notice in the script?

➊ All arguments you type after start can end up as positional parameters to the respective method. We can use them in any way according to our task, skills, and fancy. In the current example, we just pass all of them to echo(1) as one string in the next line - note $* within the double quotes. Here is how the script can be invoked now:

➋ The same applies to any method our script provides, not only to a standard one. We have added a custom method named kiss, and it can take advantage of the extra arguments not less than start does. E.g.:

➌ If we want just to pass all extra arguments to any method, we can merely substitute "$@" for "$1" in the last line of our script, where we invoke run_rc_command.

Scripts which start a long running service are suitable for service jails, and should come with a suitable service jail configuration.

Some examples of scripts which are not suitable to run in a service jail:

Scripts not suitable to run in a service jail need to prevent the use within service jails.

A script with a long running service which needs to do something listed above before the start or after the stop, can either be split-up into two scripts with dependencies, or use the precommand and postcommand parts of the script to perform this action.

By default, only the start and stop parts of a script are run within a service jail, the rest is run outside the jail. As such any setting used in the start/stop parts of the script can not be set from e.g. a precommand.

To make a script ready for use with Service Jails, only one more config line needs to be inserted:

➊ If it makes sense that the script runs in a jail, it must have an overridable service jails configuration. If it does not need network access or access to any other resource which is restricted in jails, an empty config like displayed is enough.

Strictly speaking an empty config is not needed, but it explicitly describes that the script is service jails ready, and that it does not need additional jail permissions. As such it is highly recommended to add such an empty config in such a case. The most common option to use is "net_basic", which enables the use of the hosts IPv4 and IPv6 addresses. All possible options are explained in rc.conf(5).

If a setting for the start/stop depends on variables from the rc-framework (e.g., set inside rc.conf(5)), this needs to be handled by load_rc_config and run_rc_command instead of inside a precommand.

If for some reason a script can not be run within a service jail, e.g., because it is not possible to run or it does not make sense to run it in a jail, use the following:

➊ The disabling needs to happen after the load_rc_config call, else a rc.conf(5) setting may override it.

Sometimes it is useful to run several instances of a service. Typically you want to be able to start/stop such instances independently, and you want to have a separate config file for each instance. Each instance should be started at boot, survive updates, and benefit from updates.

Here is an example of a rc script which supports this:

➊ and ➋ make sure to set the name variable to the basename(1) of the script name. If the filename is /usr/local/etc/rc.d/dummy, name is set to dummy. This way changing the filename of the rc script changes automatically the content of the name variable.

➌ specifies the variable name which is used in rc.conf to enable this service based upon the filename of this script. In this example this resolves to dummy_enable.

➍ makes sure the default for the _enable variable is NO.

➎ is an example of having some defaults for service specific framework variables, in this case the service jails options.

➏ and ➐ set variables internal to the script (pay attention to the underscore in front of _dummy_user to make it different from dummy_user which can be set in rc.conf).

The part in ➎ is for variables which are not used inside the script itself but in the rc framework. All the variables which are used as parameters somewhere in the script are assigned to a generic variable like in ➐ to make it more easy to reference them (no need to eval them at each place of use).

This script will now behave differently if the start script has a different name. This allows to create symlinks to it:

The above creates an instance of the dummy service with the name dummy_foo. It does not use the config file /usr/local/etc/dummy.cfg but the config file /usr/local/etc/dummy_foo.cfg (➐), and it uses the PID file /var/run/dummy/dummy_foo.pid instead of /var/run/dummy/dummy.pid.

The services dummy and dummy_foo can be managed independently of each other, while having the start script update itself on package update (due to the symlink). This does not update the REQUIRE line, as such there is no easy way of depending on a specific instance. To depend upon a specific instance in the startup order a copy needs to be made instead of using a symlink. This prevents the automatic pick-up of changes to the start script when an update is installed.

The original article by Luke Mewburn offers a general overview of rc.d and detailed rationale for its design decisions. It provides insight on the whole rc.d framework and its place in a modern BSD operating system.

The manual pages rc(8), rc.subr(8), and rcorder(8) document the rc.d components in great detail. You cannot fully use the rc.d power without studying the manual pages and referring to them while writing your own scripts.

The major source of working, real-life examples is /etc/rc.d in a live system. Its contents are easy and pleasant to read because most rough corners are hidden deep in rc.subr(8). Keep in mind though that the /etc/rc.d scripts were not written by angels, so they might suffer from bugs and suboptimal design decisions. Now you can improve them!