pantz.org banner
Running and tuning OpenBSD network servers in a production environment
Posted on 12-09-2005 02:13:00 UTC | Updated on 12-09-2005 02:13:00 UTC
Section: /software/openbsd/ | Permanent Link

Philipp Buhler
sysfive.com GmbH
[email protected]

Henning Brauer
BS Web Services
[email protected]

October 8, 2002

Converted from PDF to HTML by Pantz.org

Abstract

Heavily loaded network servers can experience resource exhastion. At best, resource exhaustion will slow server response, but left uncorrected, it can result in a crash of the server.

In order to understand and prevent such situations, a knowledge of the internal operation of the operating system is required, especially how memory management works.

This paper will provide an understanding of the memory management of OpenBSD, how to monitor the current status of the system, why crashes occur and how to prevent them.

1. Motivation

Our main motivation for this paper was the lack of comprehensive documentation about tuning network servers running under OpenBSD [Ope02], especially with regard to the memory usage of the networking code in the kernel.

Either one can get general information, or is "left alone" with the source code. This paper outlines how to deal with these issues, without reading the source code. At least one does not need to start in "nowhere-land" and dig through virtually everything.

This paper aims to give a deeper understanding on how the kernel handles connections and interacts with userland applications like the Apache webserver.

2. Resource Exhaustions

Running a publicly accessible server can always lead to unexpected problems. Typically it hap- pens that resources get exhausted. There are numerous reasons for this, including:

Low Budget: There's not enough money to buy "enough" hardware which would run an untuned OS.

Peaks: Overload situations which can be expected (e. g. special use) or not (e. g. get- ting "slashdotted").

DoS: Denial-of-Service by attackers flooding the server.

No matter what reason leads to an exhaustion, there are also different types of resources which can suffer from such a situation. We briefly show common types and countermeasures. Afterwards we go into detail about memory exhaustion.

2.1 I/O Exhaustion

It's very typical for network servers to suffer in this area. Often people just add more CPU to "help" a slowly reacting server, but this wouldn't help in such a case.

Usually one can detect such an exhaustion by using vmstat(8) or systat(8). Detailed usage is shown in Section 5.1 There are also numerous I/O "bottlenecks" possible, but one typical indication is the CPU being mostly idle and blocked processes waiting for resources. Further distinctions can be made:

Disk

The process is waiting for blocks from (or to) the disk and cannot run on the CPU, even if the CPU is idle. This case could be resolved by moving from IDE to SCSI, and/or using RAID technology. If repetitive writes/reads are being done an increase of the filesystem-cache could also help (Though this has implications on the KVM, see the appropriate section). Filesystem-cache can be configured with the kernel option BUFCACHEPERCENT (for most kernel configurations, see options(4) and config(8)).

NIC

Choosing the right network card is important for busy servers. There are lots of low-end models like the whole Realtek range. These cards are relatively dumb themselves. On the other hand, there are chipsets with more intelligence. DEC's 21143, supported by the dc(4) driver, and Intel's newer chipsets, supported by the fxp(4) driver, have been proven to work well in high-load circumstances.

Low-end cards usually generate an interrupt for every packet received, which leads to the problems we describe in the next subsection. By using better cards, like the mentioned DEC and Intel ones, packets are getting combined, thus reducing the amount of interrupts.

Another important point is the physical media interface, e. g. sqphy(4). Noise and distortion is a normal part of network communications, a good PHY will do a better job of extracting the data from the noise on the wire than a poor PHY will, reducing the number of network retransmissions required.

It might be a good idea to use Gigabit cards, even when running 100 MBit/s only. They are obviously built for much higher packet rates (and this is the real problem, not bandwidth) than FastEthernet ones, thus have more own intelligence and deal better with high loads.

IRQ

Every interrupt requires a context switch, from the process running when the IRQ took place, to the interrupt handler. As a number of things must be done upon entering the interrupt handler, a large quantity of interrupts can result in excess time required for context switching. One non-obvious way to reduce this load is to share interrupts between the network adapters, something permitted on the PCI bus. As many people are not even aware of the the possibility of interrupt sharing, and the benefits are not obvious, let's look at this a little closer.

With separate adapters on separate interrupt lines, when the first interrupt comes in, a context switch to the interrupt handler takes place. If another interrupt comes in from the other adapter while the first interrupt is still being handled, it will either interrupt the first handler, or be delayed until the first handler has completed, depending on priority, but regardless, two additional context switches will take place � one into the second handler, one back out.

In the case of the PCI and EISA busses, interrupts are level triggered, not edge triggered, which makes interrupt sharing possible. As long as the interrupt line is held active, a device needs servicing, even if the first device which triggered the interrupt has already been serviced. So, in this case, when the first adapter triggers the interrupt, there will be a context switch to the handler. Before the handler returns, it will see if any other devices need servicing, before doing a context switch back to the previous process.

In a busy environment, when many devices are needing service, saving these context switches can significantly improve performance by permitting the processor to spend more time processing data, rather than switching between tasks. In fact, in a very high load situation, it may be desireable to switch the adapters and drivers from an interrupt driven mode to a polling mode, though this is not supported on OpenBSD at this time.

2.2 CPU Exhaustion

Of course the CPU can be overloaded also while other resources are still fine. Besides buying more CPU power, which is not always possible, there are other ways to resolve this problem. Most common cases for this are:

CGI: Excessive usage of CGI scripts, usually written in interpreter languages like PHP or Perl. Better (resource-wise) coding can help, as well as using modules like mod perl (This can have security implications, but that is another story) to reduce load.

RDBM: Usually those CGI scrips use a database. Optimization of the connections and queries (Indexing, ..) is one way. There is also the complete offloading of the database to a different machine (This could be unfeasible due to an already overloaded network or due to budget constraints).

SSL: Especially e-commerce systems or online banking sites suffer here. OpenBSD supports hardware-accelerators (crypto(4)). Typical cryptographic routines used for SSL/TLS can be offloaded to such cards in a transparent manner, thus freeing CPU time for processing requests.

3. Memory Exhaustion

Another case of overloading can be the exhaustion of memory resources. Also the speed of the allocator for memory areas has significant influence on the overall performance of the system.

3.1 Virtual Memory (VM)

VM is comprised of the physical RAM and possible swap space(s). Processes are loaded into this area and use it for their data structures. While the kernel doesn't really care about the current location of the process' memory space (or address space) it is recommended that especially the most active tasks (like the webserver application) never be swapped out or even subjected to paging.

With regard to reliability it's not critical if the amount of physical RAM is exhausted and heavy paging occurs, but performance-wise this should not happen. The paging could compete for Disk I/O with the server task, thus slowing down the general performance of the server. And, naturally, harddisks are slower than RAM by magnitudes.

It's most likely that countermeasures are taken after the server starts heavy paging, but it could happen that also the swap space, and thus the whole VM, is exhausted. If this occurs, sooner or later the machine will crash.

Even if one doesn't plan for the server starting to page out memory from RAM to swap, there should be some swap space. This prevents a direct crash, if the VM is exhausted. If swap is being used, one has to determine if this was a one-time-only peak, or if there is a general increase of usage on the paging server. In the latter case one should upgrade RAM as soon as possible.

In general it's good practice to monitor the VM usage, especially to track down when the swap space is being touched. See section 5 for details.

3.2 Kernel Virtual Memory (KVM)

Besides VM there is a reserved area solely for kernel tasks. On the common i386 architecture (IA-32) the virtual address space is 4GB. The OpenBSD/i386 kernel reserves 768MB since the 3.2 release (formerly 512MB) of this space for kernel structures, called KVM.

KVM is used for addressing the needs of managing any hardware in the system and small allocations (like pathname translations) being needed by syscalls. The biggest chunks being used are the management of the VM (RAM and swap), filesystem-cache and storage of network buffers (mbuf).

Contrary to userland the kernel allocations can not be paged out ("wired pages"). Actually it's possible to have pageable kernel memory, but this is rarely used (e. g. for pipe buffers) and not a concern in the current context. Thus, if the KVM is exhausted, the server will immediatly crash. Of course 768MB is the limit, but if there is less RAM available, this is the absolute limit for wired pages then. Non-interrupt-safe pages could be paged out, but this is a rare exception.

Since RAM has to be managed by kernel maps also, it's not wise to just upgrade RAM without need. More RAM leaves less space for other maps in KVM. Monitoring the "really" needed amount of RAM is recommended, if KVM exhaustions occur. For example, 128MB for a firewall is usually more than enough. Look at Section 7.2 for a typical hardware setup of a busy firewall.

This complete area is called kernel map in the source and has several "submaps" (see /sys/uvm/uvm km.c). One main reason for this is the locking of the address space. By this mapping other areas of the kernel can stay unlocked while another map is locked.

Main submaps are kmem map, pager map, mb map and exec map. The allocation is done at boot-time and is never freed, the size is either a compile-time or boot-time option to the kernel.

4. Resource Allocation

Since the exhaustion of KVM is the most crit- ical situation one can encounter, we will now concentrate on how those memory areas are allocated.

Userland applications cannot allocate KVM needed for network routines directly. KVM is protected from userland processes completely, thus there have to be routines to pass data over this border. The userland can use a syscall(2) to accomplish that. For the case of networking the process would use socket(2) related calls, like bind(2), recv(2), etc.

Having this layer between userland and kernel, we will concentrate on how the kernel is allocat- ing memory; the userland process has no direct influence on this. The indirect influence is the sending and receiving of data to or from the kernel by the userland process. For example the server handles a lot of incoming network data, which will fill up buffer space (mbufs) within the KVM. If the userland process is not handling this data fast enough, KVM could be exhausted. Of course the same is true if the process is sending data faster than the kernel can release it to the media, thus freeing KVM buffers.

4.1 mbuf

Historically, BSD uses mbuf(9) (memory buffer) routines to handle network related data. An mbuf is a data structure of fixed size of 256 bytes (defined by MSIZE). Since there is overhead for the mbuf header (m hdr{}) itself, the payload is reduced by at least 20 bytes and up to 40 bytes (see /usr/include/sys/mbuf.h for details).

Those additional 20 bytes overhead appear, if the requested data doesn't fit within two mbufs. In such a case an external buffer, called cluster, with a size of 2048 bytes (defined by MCLBYTES), is allocated and referenced by the mbuf (m ext{}).

Mbufs belonging to one payload packet are "chained" together by a pointer mh next. mh nextpkt points to the next chain, forming a queue of network data which can be processed by the kernel. The first member of such a chain has to be a "packet header" (mh_type M_PKTHDR).

Allocation of mbufs and clusters are obtained by macros (MGET, MCLGET, ..). Before the release of OpenBSD 3.0 those macros used malloc(9) to obtain memory resources.

If there were a call to MGET but no more space is left in the corresponding memory map, the kernel would panic ("malloc: out of space in kmem map").

4.2 pool

Nowadays OpenBSD uses pool(9) routines to allocate kernel memory. This system is designed for fast allocation (and freeing) of fixedsize structures, like mbufs.

There are several advantages in using pool(9) routines instead of the ones around malloc(9):

* faster than malloc by caching constructed objects

* cache coloring (using offsets to more efficiently use processor cache with real-world hardware and programming techniques)

* avoids heavy fragmentation of available memory, thus wasting less of it

* provides watermarks and callbacks, giving feedback about pool usage over time

* only needs to be in kmem map if used from interrupts

* can use different backend memory alloca- tors per pool

* VM can reclaim free chunks before paging occurs, not more than to a limit (Maxpg) though

If userland applications are running on OpenBSD (> 3.0), pool(9) routines will be used automatically. But it's interesting for people who plan (or do so right now) to write own kernel routines where using pool(9) could gain significant performance improvements.

Additionally large chunks formerly in the kmem map have been relocated to the ker- nel map by using pools. Allocations for inodes, vnodes, .. have been removed from kmem map, thus there is more space for mbufs, which need protection against interrupt reentrancy, if used for e.g. incoming network data from the NIC (kmem map has to be protected by splvm(), see spl(9)).

5. Memory Measurement

Obviously one wants to know about memory exhaustion before it occurs. Additionally it can be of interest, which process or task is using memory. There are several tools provided in the base OpenBSD system for a rough monitoring of what is going on. For detailed analysis one has to be able to read and interpret the values provided by those tools, but sometimes one needs more details and can rely on 3rd party tools then.

Example outputs of the tools mentioned can be found in the Appendix.

5.1 Common tools

These are tools provided with OpenBSD, where some are rather well-known, but some are not. In any case, we have found that often the tools are used in a wrong fashion or the outputs are misinterpreted. It's quite important to understand what is printed out, even if it's a "known tool".

top

One of the most used tools is top(1). It shows the current memory usage of the system. In detail one could see the following entries:

Real: 68M/117M act/tot, where 68MB are currently used and another 49MB are allocated, but not currently used and may be subject to be freed.

Free: 3724K, shows the amount of free physical RAM

Swap: 24M/256M used/tot, 24MB of 256MB currently available swap space is used.

If one adds 3724kB to 117MB, the machine would have nearly 122MB RAM. This is, of course, not true. It has 128MB of RAM; the "missing" 6MB are used as filesystem-cache (dmesg: using 1658 buffers containing 6791168 bytes (6632K) of memory).

Besides this rough look on the memory usage of the system, there are indicators for other resource exhaustions. In the line CPU states: there is an entry x.y% interrupt. See how to resolve high values, they slow down the perfor- mance.

Blocking disks can be detected in the WAIT col- umn. For example an entry getblk shows that the process is waiting for data from a disk (or any other block device).

ps

Another very common tool is ps(1) and it's related to top(1). Where top(1) is usually used for an overview of the system, one can use ps(1) for detailed picking on the exact state of a process (or process group).

Additionally it can be closer to reality and the output is more flexible, thus one can do better post-processing in scripts or similar.

Probably most interesting are the options showing how much percentage CPU and VM a process is using. One can sort by CPU ('u') or VM usage ('v') to find a hogging process quickly.

vmstat

vmstat(8) is the traditional "swiss army knife" for detailed looks on the systems current usage. It's perfect for a first glance on potential bottlenecks.

A vmstat-newbie will probably be baffled by the output, but with some experience it's rather easy to find out, what's happening and where potential problems are located.

The default output consists of six areas (procs, memory, page, disks, faults, cpu). Each areas has columns for related values:

procs r b w, shows how many processes are (r)unning, are being (b)locked or are (w)aiting. Blocked processes cannot change to running before the block is resolved, e.g. a process "hangs" in a getblk state and waits for disk I/O. Waiting means that the process is ready to run, but has still not been scheduled, most likely because the CPU is overloaded with processes.

memory avm fre, number of pages (1024b) being allocated and on the free list. The avm value gives a better insight on the allocation, than the values from top(1).

page flt re at pi po fr sr, page-in (pi) and page-out (po) are most interesting here. It indicates if, and how much, paging (or even swapping) occurs.

disks sd0 cd0, the columns here depend on the disk setup of course. Values are transfer per seconds on this device. If high values here correspond with blocked processes below procs this is a good indication that the disk subsystem could be too slow.

faults in sys cs, can indicate too many in- terrupts and context switches on the CPU. sys counts syscalls brought to the kernel, a rather hard value to interpret with regard to bottlenecks, but one can get an idea of how much traffic has to pass between userland and kernel for completing the task.

cpu us sy id, looked at separately not too in- formative, but in combination with other values it's one keypoint in figuring out the bottleneck. If processes are in `w' state and `id' is very low, a CPU exhaustion occurs. Processes being (b)locked and having high (id)le values detect I/O exhaustions. Having high (sy)stem values and (w)aiting and/or (b)locked processes indicate that the kernel is busy with itself too much; this is usually because of "bad" drivers. Compare to `faults in' to find out if interrupts are killing the performance. If not it's still possible that the CPU is busy transfering blocks from disk devices, indicated by low disk transfers and blocked processes.

Already impressive diagnostic possibilities, but vmstat(8) can show even more interesting things.

Besides the options -i to show summaries about interrupt behaviour and -s to get information about the swap area, vmstat -m can provide a very detailed look on the current memory usage.

Like we already have shown OpenBSD uses pool(9) for network data, thus we concentrate now on the last chunk vmstat -m is reporting. Most interesting are the lines mbpl and mclpl, which represent the memory usage for mbufs (mbpl) and clusters (mclpl).

Interesting columns are Size, Pgreq, Pgrel, Npage and Maxpg. One can obtain the follow- ing information from that:

Size: the size of a pool item

Pgreq: reports how many pages have ever been allocated by this pool.

Pgrel: the pool freed those pages to the system.

Npage: currently allocated/used pages by the pool.

Maxpg: maximum number of pages the pool can use, even if paging would occur. More precise: the pool can grow over this limit, but the pagedaemon can reclaim free pages being over this limit, if VM is running low.

netstat

Usually netstat(1) is used for gathering network configurations, but it also provides information about different memory usages.

netstat -f inet (or -f inet6) shows information about current network activity. With regard to memory consumption the columns Recv-Q and Send-Q are of major interest.

Typically one will encounter entries in Send-Q for a busy webserver with a good network connection. Clients usually have significant smaller bandwith, thus the provided data of the webserver application cannot "leave" the system. It's getting queued on the network stack, eating up mbuf clusters.

Pending requests will show up in Recv-Q, indicating that the userland cannot process the data as fast as it is coming in over the network.

The latter case should be resolved, even if memory is not running low, since the system would appear sluggish to the client, which is usually not appreciated (by the admin and/or client).

In addition to vmstat -m, netstat -m can report further values about current mbuf and cluster usage. Most notably it reports how much memory is "really" used. vmstat -m shows how many pool items are allocated, but netstat -m then reports how many pool items are actually filled with data to be processed.

In fact one could calculate this in vmstat -m by substracting Releases from Requests, but with numbers like 10599250 and 10599245, this is not really practical. Another pitfall is that vmstat -m reports memory pages, where netstat -m reports pool items (usually a factor of two) used, despite its output of mapped pages in use.

Furthermore it splits up what type of, and how many, mbufs are used (packet headers, sockets, data, ..), and it gives a summary about how much memory is needed by the network stack, which would be rather tedious to calculate from the vmstat -m output.

systat

This tool provides a top(1) like display of information the previous tools would provide. Especially systat vmstat is a perfect overview about load, disk usage, interrupts, CPU and VM usage.

One can monitor the system in intervals, or col- lect the information over time.

5.2 Special tools

Besides the tools we have shown so far, there are additional possibilities to monitor the system. symon and pftop are in the ports collec- tion. KVMspy is not even published for now, but it shows that it's possible to write own tools for specific monitorings without enormous effort (the source code is below 300 lines).

symon

For monitoring overall resource usage over time frames, symon [Dij02] is a perfect tool. It queries the kernel via sysctl about common resources. It uses rrdtool [Oet02] as data storage backend. There is a data collector daemon, called symon, which runs on every monitored machine, sending the collected data to symux, usually running on a central machine, which stores them on disk. Additionally there is a web-interface, symon-web, providing graphical representation of the collected data.

After machines have been set up with detailed analysis, this output is enough to detect highload situations and trigger countermeasures before exhaustion occurs.

If one wants a long-term analysis of detailed data, it's relativly easy to extend this tool. Symon is pretty new and under active development by Willem Dijkstra, but already very useful.

pftop

If one wants to monitor specific areas, like pf(4), pftop [Aca02] is a curses-based, realtime monitoring application providing that.

One can consider it as a netstat-variant, providing similar information, about the paket filter.

KVMspy

For the absolute curious one, there will be KVMspy. Currently it shows a bit more (offsets) information than vmstat -m about pools and a bit less (only current and highwater).

But, for the interested hacker, this is maybe better example code how to poll the kernel states via kvm(3) routines. Queries via sysctl(3) can be found in symon or are added to KVMspy in the future.

6. Countermeasures

And finally we come to the interesting pieces. Several ways to determine where a lack of KVM resources occurs have been shown. So, what to do if it actually happens?

There are three important kernel options defining the KVM layout with regard to networking. NMBCLUSTERS and NKMEMPAGES are compiletime options, but can be set via config(8) as well. MAX KMAPENT can only be set at compile time.

6.1 NMBCLUSTERS

The maximum number of clusters for network data can be defined here. Naturally, it's difficult to calculate this value in advance. Most tuning guides recommend a value of 8192 here. We usually use this value, too.

People tend to raise this value further, not knowing what implications this can have on the system. A value of 8192 potentially uses 16MB for mb map: 8192 2048 = 16777216 (MCLBYTES is usually 2048).

Since this is only a "pre-allocation" and not real usage in the first place, this value can be sane. On the other hand, if there are other problems with KVM, this value may be lowered.

Looking at real-life usage of busy webservers (see 7.1) the high watermark of mclpl is 524 (1048 clusters), thus even the default of 2048 clusters would be sufficient. This high watermark (Hiwat in vmstat -m) is also perfect to determine the mclpl size for load-balanced servers.

Imagine a Hiwat of 1000 on both machines. If one machine has to go out of service, due to a crash or simply hardware maintenance, a pool size of >4000 would ensure that the remaining machine doesn't run out of clusters. Remember that vmstat -m reports pages, not items, thus one has to calculate 100022 for NMBCLUSTERS.

Additionally it's important to track why clus- ters are used in larger numbers. We have shown in 5.1/netstat that it is important to have a quick passing from the Recv-Q to the server application. It's a better idea to improve the application performance in this area, than increasing NMBCLUSTERS and let the data sit in KVM. At least a rather empty Recv-Q leaves more space for the Send-Q, which cannot be influenced directly to free clusters.

After all, it's dangerous to use high-values for this (and the following) options without very detailed knowledge about what is happening in the kernel. A "just to be safe" tuning can easily lead to an unstable machine. We have seen people using a value of 65535 for NMBCLUSTERS, rendering a pre-allocation of 128MB � not a good idea and usually it doesn't gain anything, except problems. Think twice about those values.

6.2 NKMEMPAGES

This option defines the total size of kmem map. Since this is not exclusively used for networking data, it is a bit difficult to calculate the value for this option.

Since kmem map was freed from other usage (4.2) and the introduction of pool(9) ensures that there is more space here for mbufs anyway, so an exhaustion of kmem map is less likely than before.

Tracking of the usage is still possible, though. Looking again at tt vmstat -m, this time at mbpl, one can see a correlation between mbpl and mclpl. It's common that the page value is usually half (or less) the value from mclpl. Yet again, one has to take care of "items vs page-size". Mbufs are way smaller then a cluster, thus 16 mbufs fit in one page of memory.

A network connection using clusters needs at least two mbufs, one for the paket header and one for the reference to the cluster. Since not every connection uses clusters it's sane to assume that a value for NKMEMPAGES being twice the value of NMBCLUSTERS is a good starting point.

Again, one should raise this value very carefully. Blindly changing these values can introduce more problems, than are solved.

Additionally, if the option is not touched, the kernel gets a sane default value for NKMEMPAGES at compile-time, based on RAM available in the system. If the kernel is compiled on a different machine with a different amount of RAM, this option should be used. A typical calculation value is 8162 for a machine with 128MB of RAM; this can be determined by sysctl -n vm.nkmempages.

6.3 MAX KMAPENT

Definition of the number of static entries in kmem map. Like NKMEMPAGES, the value is calculated at compile-time if unset. The default of 1000 (at least, it is based on "maxusers") is usually enough.

Raising this value is discouraged, but could be needed, if panics (uvm mapent alloc: out of static map entries ..) occur. Usually this happens if kmem map is highly fragmented, for example by a lot of small allocations.

7. Real-life Examples

Putting everything together, we provide two examples of machines running OpenBSD under high load. It shows that a careful kernel configuration and hardware selection has great influence on the performance and reliability.

7.1 chat4free.de Webserver

This machine, hosted by BSWS, is running the webserver for one of Germany's biggest chat systems, chat4free.de.

The site consists of static pages and public fo- rums. The unusual problem here is the both the overall load and the enormous peaks which happen when numbers of users are disconnected from the chat server due to external network problems or crashes of the server itself.

Unlike many web applications, this server has a huge volume of small packets, which demonstrates that loading is more an issue of users and packet counts than raw data transfer.

Originally, it was running one Apache instance for the entire application, on an 700MHz Athlon system with 1.5G RAM, running a highly modified OpenBSD 3.0. Unfortunately, this system sometimes crashed due to KVM exhaustion.

To address this problem, the system was switched to a new system, again an 700MHz Athlon with 512M RAM, running two Apache instances in chroot jails, on a fairly stock OpenBSD 3.1 system. The system has a network adapter based on a DEC/Intel 21143, with a Seeq 84220 PHY, and runs "headless" with a serial console.

One of the two Apache instances is stripped down as much as I could make it, and serves the static pages. This httpd binary is only 303k in size, compared to the almost 600k of the stock Apache. The second instance of Apache is much bigger, as it has PHP compiled in. I always use static httpds, rather than Dynamic Shared Objects (DSOs).

The kernel configuration is fairly stock. All unused hardware support and emulations are removed, option DUMMY NOPS is enabled. NMBCLUSTERS is bumped to 8192, NKMEMPAGES to 16384. I considered raising MAX KMAPENT from its default of 1000 to 1500 or so to be able to have even more concurrent Apache processes running, though there has been no real need yet in this application. The machine has an ordinary IDE hard disk for the system, content and logs are on a separate machine's RAID subsystem, mounted via NFS. Most static content ends up being cached, reducing network traffic.

The "lean" httpd instance is configured for up to 1000 concurrent httpd tasks, the "fat" one for up to 600. I've seen both reach their maximum limits at the same time, and the smaller machine handles this load without incident. This is due to the superior memory management in OpenBSD 3.1 and the smaller Apache configurations.

Detailed kernel configuration and dmesg(8) can be found in the Appendix.

7.2 A firewall at BSWS

One important fact about firewalling and filter- ing is that the bandwidth isn't the important issue, the issue is the packet rate (i.e., packets per second). Each packet needs to be handled by the network adapter, the TCP/IP stack and the filter, which each need to do roughly the same amount of work whether the packet is small or large.

The firewall that protects a number of the servers at BSWS is under rather heavy load, not really due to total bandwidth, but the large number of small packets involved. It is running on a 700MHz Duron with 128M RAM and three DEC/Intel 21143-based NICs (one is currently not in use). It boots from a small IDE hard disk, which is quite unimportant to this application.

The machine is running a highly customized version of OpenBSD. The base system is OpenBSD 3.0, but many pieces of what became OpenBSD 3.1 were imported, including 3.1's packet filter pf(4). At the time this was put together, there was no other option for this application. Many of pf's newer features were needed, but it was not possible to wait for 3.1 Release, as the previous OpenBSD 2.9 firewall running IPFilter had saturated the processor at near 100% utilization at peak usage times, and delays were being noticed. The kernel configuration has had all uneeded hardware support and binary emulations removed, and the always famous NKMEMCLUSTERS=16384 and NM- BCLUSTERS=8192 modifications. The number of VLAN interfaces was raised to 20 (from 2 in GENERIC).

As of October 5, the expanded ruleset has 1132 rules. The "quick" keyword is used in most places to reduce the number of rules that must be evaluated for each packet, otherwise the entire ruleset must be evaluated for each packet. The rules are ordered so that the ones I expect the most matches from are towards the top of the file. All pass rules keep state; not only is this good practice for security, but with pf, state table lookups are usually much faster than rule evaluation. No NAT takes place on this machine, only packet filtering.

On the external interface, there is only spoofing protection taking place. Incoming packets with a source IP of the internal networks, outgoing packets with an IP which is not from one of the internal networks, and all 127.0.0.0/8 traffic is dropped. Normally, one would also drop packets with RFC1918 ("private IP space"), however in this case, it is handled externally by the BSWS core routers, because there is valid traffic with RFC1918 IPs from other internal networks crossing this firewall.

The actual filtering policies are enforced on the inside (VLAN) interfaces, which has the benefit that packets attempting to cross between VLANs encounter the same rules as packets from the outside. Every packet passing the firewall is normalized using the scrub directive. OpenBSD 3.2 will support multiple scrub methods besides the classic buffering fragment cache. One of the more interesting is the crop method, which almost completely avoids buffering fragments.

The results have been impressive. In September, 2002, the state table reached a peak size of 29,390, with an average size of 11,000. Up to 15,330 state table lookups per second were performed with average of 5600. State table inserts and removals peaked at slightly over 200 per second each. The CPU load seldom exceeds 10%. Compare this to the old IPFilter solution running on the same hardware doing much the same task, where the CPU was maxed out with only 600 rules and a peak of 15,000 packets per second. pf has permitted considerable growth in the complexity of the rule sets and traffic, and as you can see, still leaves BSWS considerable room to grow. Since this firewall went into operation in March, 2002, there hasn't been a single problem with its hardware or software.

8. Conclusions

Running OpenBSD servers under high load is pretty safe nowadays. We have shown that the introduction of pool(9) made operation way better with regard to memory usage and per- formance.

We have shown how network traffic influences the memory usage of the kernel and how the pieces are related together.

The provided knowledge about monitoring a running system and potential countermeasures against resource exhaustions should help to deal with high-load situations better.

9. Acknowledgements

A big "thank you" goes to Nick Holland, who turned our crappy english into something useful and provided a lot of input on how to explain this difficult area better.

Thanks also to Artur Grabowski for implementing pool(9) in the OpenBSD kernel and for further explanations about KVM.

Several proof-readers helped on finding spelling errors and inconsistencies within the paper, a special thanks here for Daniel Lucq, who also wrote KVMspy.

And, of course, thanks to the OpenBSD developer team for working on a system which provides already sane defaults for operating a high-load server, and, not to forget, a very high level of security.

References

[Aca02] Can E. Acar. Openbsd pf state viewer. http://www.eee.metu.edu.tr/~canacar/pftop/, 2002.

[Dij02] Willem Dijkstra. The small and secure active system monitor. http://www.xs4all.nl/~wpd/symon/ ,2002.

[McK96] Marshall Kirk (et. al.) McKusick. The design and implementation of the 4.4BSD operating system. Addison-Wesley, 1996.

[Oet02] Tobi Oetiker. Round robin database. http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/ ,2002.

[Ope02] OpenBSD. http://www.openbsd.org/ ,2002.

[Ste94] W. Richard Stevens. TCP/IP Illustrated, Vol. 2. Addison-Wesley, 1994.




A. top

This machine is the main server of sysfive.com GmbH, slightly tuned it is really idle.

load averages:    0.19,    0.12,    0.09         14:19:57
68 processes:    1 running, 64 idle, 3 zombie
CPU states:    0.3% user,    0.9% nice,    0.3% system,    0.0% interrupt,98.4% idle
Memory: Real: 49M/80M act/tot    Free: 41M    Swap: 0K/256M used/tot


  PID USERNAME PRI NICE  SIZE  RES STATE  WAIT    TIME    CPU   COMMAND
15902 root     2    0   2308K 1832K idle  select  19:39   0.00% isakmpd
27679 pb       2    0    964K 1468K sleep select  7:00    0.00% screen-3.9.11
19945 gowry    2    0   4644K 5096K idle  select  4:30    0.00% screen-3.9.11
3605  postfix  2    0   304K   736K sleep select  4:29    0.00% qmgr
22360 root    18    0   640K  9944K sleep pause   2:53    0.00% ntpd
11827 pb      2     0   516K  1312K sleep poll    2:18    0.00% stunnel
[..]

B. ps

Same machine, same processes reported by ps -axv

USER      PID   %CPU   %MEM    VSZ     RSS TT    STAT STARTED    TIME     COMMAND
root    22360    0.0    7.6    640    9944 ??    Ss    8Aug02    2:48.24  ntpd -c /etc/ntp.conf
gowry   19945    0.0    3.9    4644   5096 ??    Ss    9Aug02    4:30.56  SCREEN (screen-3.9.11)
root    15902    0.0    1.4    2308   1832 ??    Is   31Jul02    19:39.33 isakmpd
pb      27679    0.0    1.1    964    1468 ??    Ss   13Jul02    6:59.75  SCREEN (screen-3.9.11)
pb      11827    0.0    1.0    516    1312 ??    Ss   13Jul02    2:15.55  stunnel
postfix  3605    0.0    0.6    304     736 ??    S     6Aug02    4:30.29  qmgr -l -t fifo -u

C. vmstat

Current vmstat output of the same machine (vmstat 1 5)

procs     memory           page                         disks        faults        cpu
r b w      avm      fre    flt  re    pi  po   fr   sr  cd0  sd0     in      sy    cs    us   sy  id
1 0 0    50324    41608    14    0    0    0    0    0    0    1    234    7151    160    0    0  99
0 0 0    50324    41608    10    0    0    0    0    0    0    0    233    1602    165    0    0  100
0 0 0    50324    41608    6     0    0    0    0    0    0    0    233    1589    165    0    1  99

If the machine would have disk I/O blocking problems, the output could look like this. Note the idle CPU, but blocked processes are waiting for blocks from the busy drive.

procs      memory         page                          disks      faults         cpu
r b w      avm      fre   flt   re   pi   po   fr   sr  cd0 sd0    in       sy    cs    us   sy id
1 2 0    50324    41608    14    0    0    0    0    0    0 271    234    7151    160    1    3 96
0 1 0    50324    41608    10    0    0    0    0    0    0 312    233    1602    165    0    4 96
0 1 0    50324    41608     6    0    0    0    0    0    0 150    233    1589    165    0    2 98

Worst-case scenario, the machine does heavy paging, thus overloading the disk subsystem. Additionally the CPU is maxed out. Processes are waiting, interrupts cause massive context-switching. The values are arbitrary.

procs   memory         page                            disks       faults       cpu
r b w    avm    fre    flt    re   pi    po   fr   sr  cd0 sd0     in      sy   cs us sy id
1 2 1    324    608    314    0    25    35    0    0    0 271    412    7151 1931 80 19  1
1 3 2    324    608    310    0    28    42    0    0    0 312    501    1602 1876 81 19  0
1 2 1    324    608    306    0    21    38    0    0    0 150    467    1589 1911 85 12  3

Now let's have a look at the pool situation of a firewall. A nice example that the pool can grow over the initial limit (Maxpg 512, Hiwat 516), but somehow KVM is low, since a lot of requests are failing (Fail 14725). The kernel should be reconfigured with NMBCLUSTERS > 1024 (vmstat -m | grep mclpl).

Name     Size   Requests  Fail  Releases Pgreq Pgrel  Npage  Hiwat Minpg  Maxpg Idle
mclpl    2048    1758499 14725   1757480   518     2    516    516     4    512    4

D. netstat

All packet data is getting delivered to/from the sshd fast enough, so no queuing occurs.

Active Internet connections
Proto Recv-Q Send-Q    Local Address    Foreign Address     (state)
tcp        0      0    172.23.1.1.22    10.172.2.32.1156    ESTABLISHED
tcp        0      0    172.23.1.1.22    172.23.1.3.39679    ESTABLISHED
tcp        0      0    172.23.1.1.22    192.168.1.5.42456   ESTABLISHED

Somehow either the uplink is saturated, or the remote clients are not retrieving data fast enough, thus the Send-Q is growing.

Active Internet connections
Proto Recv-Q  Send-Q    Local Address    Foreign Address     (state)
tcp        0    5346    172.23.1.1.22    10.172.2.32.1156    ESTABLISHED
tcp        0       0    172.23.1.1.22    172.23.1.3.39679    ESTABLISHED
tcp        0    7159    172.23.1.1.22    192.168.1.5.42456   ESTABLISHED

For whatever reason, sshd is not processing data fast enough. Maybe the deciphering needs more CPU then available?

Active Internet connections
Proto Recv-Q Send-Q    Local Address    Foreign Address     (state)
tcp     8811      0    172.23.1.1.22    10.172.2.32.1156    ESTABLISHED
tcp     5820      0    172.23.1.1.22    172.23.1.3.39679    ESTABLISHED
tcp    11631      0    172.23.1.1.22    192.168.1.5.42456   ESTABLISHED

Let's have a look at the memory usage with netstat -m. The stack has to keep 85 clusters in KVM, somehow the application is processing data either too fast (Send-Q) or too slow (Recv-Q).

384 mbufs in use:
        100 mbufs allocated to data
        178 mbufs allocated to packet headers
        106 mbufs allocated to socket names and addresses
85/1048 mapped pages in use
3144 Kbytes allocated to network (8% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

E. systat

Looks like the machine is doing nothing? Wrong, look at the interrupt counting for dc0 and dc2. It's the BSWS' firewall described earlier.

       1 users    Load    0.05    0.08    0.08    Sat Oct    5 17:22:05 2002

             memory totals (in KB)            PAGING     SWAPPING        Interrupts
            real   virtual     free           in    out  in    out       7903 total
Active     91472     93712    10848    ops                               100  clock
All       116216    118456   270684    pages                                  pccom0
                                                                         128  rtc

Proc:r    d    s    w    Csw    Trp    Sys    Int    Sof    Flt    forks 3669 dc0
          1    9           6      5     21   7936      4      2    fkppw      dc1
                                                                   fksvm      pciide0
     0.0% Sys    0.0% User    0.0% Nice    90.0% Idle              pwait 4006 dc2
  |    |    |     |    |    |    |      |     |    |    |          relck
                                                                   rlkok
                                                                   noram
  Namei          Sys-cache     Proc-cache      No-cache            ndcpy
      Calls      hits    %     hits     %      miss   %            fltcp
          2         2  100                                         zfod
                                                                   cow
  Discs  wd0                                                   128 fmin
  seeks                                                        170 ftarg
  xfers                                                       8446 itarg
  Kbyte                                                         39 wired
    sec                                                            pdfre
  pdscn

F. iostat

Medium, but constant, traffic on sd0. In fact I was generating traffic with dd(1).

       tty               cd0               sd0               sd1               fd0                     cpu
tin   tout    KB/t  t/s MB/s    KB/t t/s  MB/s    KB/t  t/s MB/s    KB/t  t/s MB/s    us   ni   sy   in id
  0    540    0.00    0 0.00    0.50 2614 1.28    0.00    0 0.00    0.00    0 0.00     1    1    5    3 90
  0    179    0.00    0 0.00    0.50 2560 1.25    0.00    0 0.00    0.00    0 0.00     0    0    2    2 95
  0    344    0.00    0 0.00    0.50 2601 1.27    0.00    0 0.00    0.00    0 0.00     0    0    3    5 92
  0    181    0.00    0 0.00    0.50 2601 1.27    0.00    0 0.00    0.00    0 0.00     0    1    5    3 91

G. pftop

Easy and quick overview about current traffic filtering:

pfTop: Up State 1-3/64, View: default, Order: none

PR    DIR SRC                   DEST                 STATE    AGE    EXP    PKTS BYTES
icmp  Out 192.168.100.32:361    192.168.100.22:361    0:0       9      1       2    96
icmp  Out 192.168.100.32:361    192.168.100.23:361    0:0       9      1       2    96
tcp    In 192.168.100.7:1029    192.168.100.32:443    4:4    4165  86302   25871 9251K

H. KVMspy

The full output would be too long, thus shortened to relevant pools/maps. Somehow this machine is not really exhausted, even with the default settings.

_kmem_map @ 0xd0518cdc: total size = 33431552 bytes, [0xd0890000, 0xd2872000]
_kmem_map @ 0xd0518cdc: 103 entries, actual size = 2453504 bytes (7.34%)
_mb_map @ 0xd0890c00: total size = 4194304 bytes, [0xda63e000, 0xdaa3e000]
_mb_map @ 0xd0890c00: 5 entries, actual size = 118784 bytes (2.83%)
_socket_pool @ 0xd05424c8: currently has 6 pages (24576 bytes)
_socket_pool @ 0xd05424c8: high water mark of 12 pages (49152 bytes)
_nkmempages @ 0xd05029d4: 8162 (_nkmempages * PAGE_SIZE = 33431552 bytes)
_nmbclust @ 0xd04fb278: 2048 (_nmbclust * MCLBYTES = 4194304 bytes)

I. chat4free.de Webserver

I'm using a bit more aggressive timeouts on this machine to lower the number of concurrent connections. This inlcudes a shortened KeepAliveTimeout to 10 seconds in apache's config and the following addition to /etc/sysctl.conf:

net.inet.tcp.keepinittime=10
net.inet.tcp.keepidle=30
net.inet.tcp.keepintvl=30
net.inet.tcp.rstppslimit=400
net.inet.ip.redirect=0
net.inet.ip.maxqueue=1000
kern.somaxconn=256

The timeouts depend heavily on your usage profile and need to be tried. The above ones work fine here, and should fit for most well connected webservers. dmesg:

OpenBSD 3.1 (windu) #0: Wed Apr 17 20:10:40 CEST 2002
    root@ozzel:/usr/src/sys/arch/i386/compile/windu
cpu0: AMD Athlon Model 4 (Thunderbird) ("AuthenticAMD" 686-class) 700 MHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SYS,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR
real mem    = 536457216 (523884K)
avail mem = 494899200 (483300K)
using 5689 buffers containing 26927104 bytes (26296K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(86) BIOS, date 04/02/02, BIOS32 rev. 0 @ 0xfb210
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
pcibios0 at bios0: rev. 2.1 @ 0xf0000/0xb690
pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xfdbd0/176 (9 entries)
pcibios0: PCI Exclusive IRQs: 11
pcibios0: PCI Interrupt Router at 000:07:0 ("VIA VT82C596A PCI-ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "VIA VT8363 Host" rev 0x03
ppb0 at pci0 dev 1 function 0 "VIA VT8363 PCI-AGP" rev 0x00
pci1 at ppb0 bus 1
pcib0 at pci0 dev 7 function 0 "VIA VT82C686 PCI-ISA" rev 0x40
pciide0 at pci0 dev 7 function 1 "VIA VT82C571 IDE" rev 0x06: ATA100, channel 0
\configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0: 
wd0: 16-sector PIO, LBA, 58644MB, 16383 cyl, 16 head, 63 sec, 120103200 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
pchb1 at pci0 dev 7 function 4 "VIA VT82C686 SMBus" rev 0x40
dc0 at pci0 dev 8 function 0 "DEC 21142/3" rev 0x41: irq 11 address 00:00:cb:53:62:c3
sqphy0 at dc0 phy 17: Seeq 84220 10/100 media interface, rev. 0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom0: console
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
biomask 4000 netmask 4800 ttymask 4802
pctr: user-level cycle counter enabled
mtrr: Pentium Pro MTRR support
dkcsum: wd0 matched BIOS disk 80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302



Kernel config:

machine i386 # architecture, used by config; REQUIRED
option DIAGNOSTIC # internal consistency checks
option CRYPTO # Cryptographic framework
option SYSVMSG # System V-like message queues
option SYSVSEM # System V-like semaphores
option SYSVSHM # System V-like memory sharing
option FFS # UFS
option FFS_SOFTUPDATES # Soft updates
option QUOTA # UFS quotas
option MFS # memory file system
option TCP_SACK # Selective Acknowledgements for TCP
option NFSCLIENT # Network File System client
option NFSSERVER # Network File System server
option FIFO # FIFOs; RECOMMENDED
option KERNFS # /kern
option NULLFS # loopback file system
option UMAPFS # NULLFS + uid and gid remapping
option INET # IP + ICMP + TCP + UDP
option INET6 # IPv6 (needs INET)
option PULLDOWN_TEST # use m_pulldown for IPv6 packet parsing
pseudo-device pf 1 # packet filter
pseudo-device pflog 1 # pf log if
pseudo-device loop 2 # network loopback
pseudo-device bpfilter 8 # packet filter
pseudo-device vlan    2 # IEEE 802.1Q VLAN
pseudo-device pty 64 # pseudo-terminals
pseudo-device tb 1 # tablet line discipline
pseudo-device vnd 4 # paging to files
#pseudo-device ccd 4 # concatenated disk devices
pseudo-device ksyms 1 # kernel symbols device
option BOOT_CONFIG # add support for boot -c
option I686_CPU
option USER_PCICONF # user-space PCI configuration
option DUMMY_NOPS # speed hack; recommended
option COMPAT_LINUX # binary compatibility with Linux
option COMPAT_BSDOS # binary compatibility with BSD/OS

option    NMBCLUSTERS=8192
option    NKMEMPAGES=16384

maxusers 64 # estimated number of users
config bsd swap generic

mainbus0 at root
bios0 at mainbus0
apm0 at bios0 flags 0x0000 # flags 0x0101 to force protocol version 1.1
pcibios0 at bios0 flags 0x0000 # use 0x30 for a total verbose
isa0 at mainbus0
isa0 at pcib?
pci* at mainbus0 bus ?
option PCIVERBOSE
pchb* at pci? dev ? function ? # PCI-Host bridges
ppb* at pci? dev ? function ? # PCI-PCI bridges
pci* at ppb? bus ?
pci* at pchb? bus ?
pcib* at pci? dev ? function ? # PCI-ISA bridge
npx0 at isa? port 0xf0 irq 13 # math coprocessor
isadma0 at isa?
isapnp0 at isa?
option WSDISPLAY_COMPAT_USL # VT handling
option WSDISPLAY_COMPAT_RAWKBD # can get raw scancodes
option WSDISPLAY_DEFAULTSCREENS=6
option WSDISPLAY_COMPAT_PCVT # emulate some ioctls
pckbc0 at isa? # PC keyboard controller
pckbd* at pckbc? # PC keyboard
vga* at pci? dev ? function ?
wsdisplay* at vga? console ?
wskbd* at pckbd? console ?
pcppi0 at isa?
sysbeep0 at pcppi?
pccom0 at isa? port 0x3f8 irq 4 # standard PC serial ports
pccom1 at isa? port 0x2f8 irq 3
pciide* at pci ? dev ? function ? flags 0x0000
wd* at pciide? channel ? drive ? flags 0x0000
dc* at pci? dev ? function ? # 21143, "tulip" clone ethernet
sqphy* at mii? phy ? # Seeq 8x220 PHYs
pseudo-device pctr 1
pseudo-device mtrr 1 # Memory range attributes control
pseudo-device sequencer 1
pseudo-device wsmux 2
pseudo-device crypto 1

Reddit!

Related stories


RSS Feed RSS feed logo

About


3com

3ware

alsa

alsactl

alsamixer

amd

android

apache

areca

arm

ati

auditd

awk

badblocks

bash

bind

bios

bonnie

cable

carp

cat5

cdrom

cellphone

centos

chart

chrome

chromebook

cifs

cisco

cloudera

comcast

commands

comodo

compiz-fusion

corsair

cpufreq

cpufrequtils

cpuspeed

cron

crontab

crossover

cu

cups

cvs

database

dbus

dd

dd_rescue

ddclient

debian

decimal

dhclient

dhcp

diagnostic

diskexplorer

disks

dkim

dns

dos

dovecot

drac

dsniff

dvdauthor

e-mail

echo

editor

emerald

encryption

ethernet

expect

ext3

ext4

fat32

fedora

fetchmail

fiber

filesystems

firefox

firewall

flac

flexlm

floppy

flowtools

fonts

format

freebsd

ftp

gdm

gmail

gnome

google

gpg

greasemonkey

greylisting

growisofs

grub

hacking

hadoop

harddrive

hba

hex

hfsc

html

html5

http

https

hulu

idl

ie

ilo

intel

ios

iperf

ipmi

iptables

ipv6

irix

javascript

kde

kernel

kickstart

kmail

kprinter

krecord

kubuntu

kvm

lame

ldap

linux

logfile

lp

lpq

lpr

maradns

matlab

memory

mencoder

mhdd

mkinitrd

mkisofs

moinmoin

motherboard

mouse

movemail

mplayer

multitail

mutt

myodbc

mysql

mythtv

nagios

nameserver

netflix

netflow

nginx

nic

ntfs

ntp

nvidia

odbc

openbsd

openntpd

openoffice

openssh

openssl

openvpn

opteron

parted

partimage

patch

perl

pf

pfflowd

pfsync

photorec

php

pop3

pop3s

ports

postfix

power

procmail

proftpd

proxy

pulseaudio

putty

pxe

python

qemu

r-studio

raid

recovery

redhat

router

rpc

rsync

ruby

saltstack

samba

schedule

screen

scsi

seagate

seatools

sed

sendmail

sgi

shell

siw

smtp

snort

solaris

soundcard

sox

spam

spamd

spf

spotify

sql

sqlite

squid

srs

ssh

ssh.com

ssl

su

subnet

subversion

sudo

sun

supermicro

switches

symbols

syslinux

syslog

systemd

systemrescuecd

t1

tcpip

tcpwrappers

telnet

terminal

testdisk

tftp

thttpd

thunderbird

timezone

ting

tls

tools

tr

trac

tuning

tunnel

ubuntu

unbound

vi

vpn

wget

wiki

windows

windowsxp

wireless

wpa_supplicant

x

xauth

xfree86

xfs

xinearama

xmms

youtube

zdump

zeromq

zic

zlib