File descriptor limits are a common problem for new Squid users. This happens because some operating systems have relatively low per-process and system-wide limits. In some cases, you must take steps to tune your system before compiling Squid.A file descriptor is simply a number that represents an open file or socket. Every time a process opens a new file or socket, it allocates a new file descriptor. These descriptors are reused after the file or socket is closed. Most Unix systems place a limit on the number of simultaneously open file descriptors. There are both per-process and per-system limits. How many file descriptors does Squid need? The answer depends on how many users you have, the size of your cache, and which particular features that you have enabled. Here are some of the things that consume file descriptors in Squid:

  • Client-side TCP connections
  • Server-side TCP connections
  • Writing cachable responses to disk
  • Reading cache hits from disk
  • Log files
  • Communication with external helper processes, such as redirectors and authenticators
  • Idle (persistent) HTTP connections

Even when Squid is not doing anything, it has some number of file descriptors open for log files and helpers. In most cases, this is between 10 and 25, so it’s probably not a big deal. If you have a lot of external helpers, that number goes up. However, the file descriptor count really goes up once Squid starts serving requests. In the worst case, each concurrent request requires three file descriptors: the client-side connection, a server-side connection for cache misses, and a disk file for reading hits or writing misses.A Squid cache with just a few users might be able to get by with a file descriptor limit of 256. For a moderately busy Squid, 1024 is a better limit. Very busy caches should use 4096 or more. One thing to keep in mind is that file descriptor usage often surges above the normal level for brief amounts of time. This can happen during short, temporary network outages or other interruptions in service.There are a number of ways to determine the file descriptor limit on your system. One is to use the built-in shell commands limit or ulimit.For Bourne shell users: 

root# ulimit -n1024

For C shell users: 

root# limit descdescriptors     1024

If you already have Squid compiled and installed, you can just look at the cache.log file for a line like this: 

2003/12/12 11:10:54| With 1024 file descriptors available

If Squid detects a file descriptor shortage while it is running, you’ll see a warning like this in cache.log: 

WARNING! Your cache is running out of file descriptors

If you see the warning, or know in advance that you’ll need more file descriptors, you should increase the limits. The technique for increasing the file descriptor limit varies between operating systems.

For Linux Users

Linux users need to edit one of the system include files and twiddle one of the system parameters via the /proc interface. First, edit /usr/include/bits/types.h and change the value for __FD_SETSIZE. Then, give the kernel a new limit with this command: 

root# echo 1024 > /proc/sys/fs/file-max

Finally, before compiling or running Squid, execute this shell command to set the process limit equal to the kernel limit: 

root# ulimit -Hn 1024

After you have set the limit in this manner, you’ll need to reconfigure, recompile, and reinstall Squid. Also note that these two commands do not permanently set the limit. They must be executed each time your system boots. You’ll want to add them to your system startup scripts.

For NetBSD/OpenBSD/FreeBSD Users

On BSD-based systems, you’ll need to compile a new kernel. The kernel configuration file lives in a directory such as /usr/src/sys/i386/conf or /usr/src/sys/arch/i386/conf. There you’ll find a file, possibly named GENERIC, to which you should add a line like this: 

options       MAXFILES=8192

For OpenBSD, use option instead of options. Reboot your system after you’ve finished configuring, compiling, and installing your new kernel. Then, reconfigure, recompile, and reinstall Squid.

For Solaris Users

Add this line to your /etc/system file: 

set rlim_fd_max = 1024

Then, reboot the system, reconfigure, recompile, and reinstall Squid.For further information on file descriptor limits, see Chapter 3, “Compiling and Installing”, of Squid: The Definitive Guide or section 11.4 of the Squid FAQ.