Mirroring FreeBSD

This is one of the most basic services, and it is required for each mirror offering public FTP distributions. FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway). Upload capability is not required (and must never be allowed for the FreeBSD file space). Also the FreeBSD archive should be available under the path /pub/FreeBSD.

There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order).

FreeBSD's ftpd, proftpd, wu-ftpd and maybe ncftpd are among the most commonly used FTPds. The others do not have a large userbase among mirror sites. One thing to consider is that you may need flexibility in limiting how many simultaneous connections are allowed, thus limiting how much network bandwidth and system resources are consumed.

Rsync is often offered for access to the contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source. The protocol is different from FTP in many ways. It is much more bandwidth friendly, as only differences between files are transferred instead of whole files when they change. Rsync does require a significant amount of memory for each instance. The size depends on the size of the synced module in terms of the number of directories and files. Rsync can use rsh and ssh (now default) as a transport, or use its own protocol for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package available:

If you want to offer the FreeBSD web pages, you will need to install a web server. You may optionally offer the FTP fileset via HTTP. The choice of web server software is left up to the mirror administrator. Some of the most popular choices are:

CVSup is a very efficient way of distributing files. It works similar to rsync, but was specially designed for use with CVS repositories. If you want to offer the FreeBSD CVS repository, you really want to consider offering it via CVSup. It is possible to offer the CVS repository via AnonCVS, FTP, rsync or HTTP, but people would benefit much more from CVSup access. CVSup was developed by John Polstra. It is a bit tricky to install on non-FreeBSD platforms, since it is written in Modula-3 and therefore requires a Modula-3 environment. John Polstra has built a stripped down version of M3 that is sufficient to run CVSup, and can be installed much easier. See Ezm3 for details. Related ports are:

There are a few more like net/cvsup-without-gui you might want to have a look at. If you prefer a static binary package, take a look here. This page still refers to the S1G bug that was present in CVSup. Maybe John will set up a generic download-site to get static binaries for various platforms.

It is possible to use CVSup to offer any kind of fileset, not just CVS repositories, but configuration can be complex. CVSup is known to eat some CPU on both the server and the client, since it needs to compare lots of files.

If you have the CVS repository, you may want to offer anonymous CVS access. A short warning first: There is not much demand for it, it requires some experience, and you need to know what you are doing.

Generally there are two ways to access a CVS repository remotely: via pserver or via ssh (we do not consider rsh). For anonymous access, pserver is very well suited, but some still offer ssh access as well. There is a custom crafted wrapper in the CVS repository, to be used as a login-shell for the anonymous ssh account. It does a chroot, and therefore requires the CVS repository to be available under the anonymous user's home-directory. This may not be possible for all sites. If you just offer pserver this restriction does not apply, but you may run with more security risks. You do not need to install any special software, since cvs(1) comes with FreeBSD. You need to enable access via inetd, so add an entry into your /etc/inetd.conf like this:

cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
          

See the manpage for details of the options. Also see the CVS info page about additional ways to make sure access is read-only. It is advised that you create an unprivileged account, preferably called anoncvs. Also you need to create a file passwd in your /home/ncvs/CVSROOT and assign a CVS password (empty or anoncvs) to that user. The directory /anoncvstmp is a special purpose memory based file system. It is not required but advised since cvs(1) creates a shadow directory structure in your /tmp which is not used after the operation but slows things dramatically if real disk operations are required. Here is an excerpt from /etc/fstab, how to set up such a MFS:

/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid,nodev 0 0
          

This is (of course) tuned a lot, and was suggested by John Polstra.

You can use a FTP mirror program to get the files. Some of the most commonly used are:

ftp/mirror was very popular, but seemed to have some drawbacks, as it is written in perl(1), and had real problems with mirroring large directories like a FreeBSD site. There are rumors that the current version has fixed this by allowing a different algorithm for comparing the directory structure to be specified.

In general FTP is not really good for mirroring. It transfers the whole file if it has changed, and does not create a single data stream which would benefit from a large TCP congestion window.

A better way to mirror the FTP area is rsync. You can install the port net/rsync and then use rsync to sync with your upstream host. rsync is already mentioned in 節 2.4.2, “Rsync (optional for FTP fileset)”. Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access.

A command line to mirror FreeBSD might look like:

% rsync -vaz --delete ftp4.de.FreeBSD.org::FreeBSD/ /pub/FreeBSD/
          

Consult the documentation for rsync, which is also available at http://rsync.samba.org/, about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via cron(8).

A few sites, including the one-and-only ftp-master.FreeBSD.org even offer CVSup to mirror the contents of the FTP space. You need to install a CVSup client, preferably from the port net/cvsup. (Also reread 節 2.4.4, “CVSup (desired for CVS repository)”.) A sample supfile suitable for ftp-master.FreeBSD.org looks like this:

          #
          # FreeBSD archive supfile from master server
          #
          *default host=ftp-master.FreeBSD.org
          *default base=/usr
          *default prefix=/pub
          #*default release=all
          *default delete use-rel-suffix
          *default umask=002

          # If your network link is a T1 or faster, comment out the following line.
          #*default compress

          FreeBSD-archive release=all preserve
          

It seems CVSup would be the best way to mirror the archive in terms of efficiency, but it is only available from few sites.

CVSup is described in some detail in 節 2.4.4, “CVSup (desired for CVS repository)” and 節 3.1.3, “With CVSup”.

It is very easy to setup a CVSup mirror. Installing net/cvsup-mirror will make sure all of the needed programs are installed and then gather all the needed information to configure the mirror.

Using other methods than CVSup is generally not recommended. We describe them in short here anyway. Since most sites offer the CVS repository as part of the FTP fileset under the path /pub/FreeBSD/development/FreeBSD-CVS, the following methods could be used.

Since the documentation is referenced a lot from the web pages, it is recommended that you mirror the FreeBSD documentation as well. However, this is not as trivial as the www-pages alone.

First of all, you should get the doc sources, again preferably via CVSup. Here is a corresponding sample supfile:

         #
         # FreeBSD documentation supfile
         #
         *default host=cvsup3.de.FreeBSD.org
         *default base=/usr
         *default prefix=/usr/share
         *default release=cvs tag=.
         *default delete use-rel-suffix

         # If your network link is a T1 or faster, comment out the following line.
         #*default compress

         # This will retrieve the entire doc branch of the FreeBSD repository.
         # This includes the handbook, FAQ, and translations thereof.
         doc-all
         

Then you need to install a couple of ports. You are lucky, there is a meta-port: textproc/docproj to do the work for you. You need to set up some environment variables, like SGML_CATALOG_FILES. Also have a look at your /etc/make.conf (copy /usr/share/examples/etc/make.conf if you do not have one), and look at the DOC_LANG variable. Now you are probably ready to run make in your doc directory (/usr/share/doc by default) and build the documentation. Again you need to make it accessible for your web server and make sure the links point to the right location.

If you have no special intentions or requirements, the statement in 節 4.2, “Ok, but where should I get the stuff now?” applies. This means:

In general the description in 節 4.2.1, “I just want to mirror from somewhere!” still applies. Of course you may want to put some weight on the fact that your upstream should be of a low tier. There are some other considerations about official mirrors that are described in 節 5, “Official Mirrors”.

If you have good reasons and good prerequisites, you may want and get access to one of the master sites. Access to these sites is generally restricted, and there are special policies for access. If you are already an official mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. If it already has three or more, ask the “zone administrator” (<hostmaster@CC.FreeBSD.org>) or FreeBSD mirror sites 郵遞論壇s first.

Whoever helped you become, an official should have helped you gain access to an appropriate upstream host, either one of the master sites or a suitable Tier-1 site. If not, you can send email to <mirror-admin@FreeBSD.org> to request help with that.

There are three master sites for the FTP fileset and one for the CVS repository (the web pages and docs are obtained from CVS, so there is no need for master).