Performance Tuning Tips for Digital Unix

### Apache HTTP Server Version 1.3

# Performance Tuning Tips for Digital Unix

Below is a set of newsgroup posts made by an engineer from DEC in response to queries about how to modify DEC's Digital Unix OS for more heavily loaded web sites. Copied with permission.

## Update

From: Jeffrey Mogul <mogul@pa.dec.com>
Date: Fri, 28 Jun 96 16:07:56 MDT

1. The advice given in the README file regarding the "tcbhashsize" variable is incorrect. The largest value this should be set to is 1024. Setting it any higher will have the perverse result of disabling the hashing mechanism.
2. Patch ID OSF350-146 has been superseded by

   > Patch ID OSF350-195 for V3.2C
   > Patch ID OSF360-350195 for V3.2D

   Patch IDs for V3.2E and V3.2F should be available soon. There is no known reason why the Patch ID OSF360-350195 won't work on these releases, but such use is not officially supported by Digital. This patch kit will not be needed for V3.2G when it is released.

    From           mogul@pa.dec.com (Jeffrey Mogul)
    Organization   DEC Western Research
    Date           30 May 1996 00:50:25 GMT
    Newsgroups     comp.unix.osf.osf1
    Message-ID     <4oirch$bc8@usenet.pa.dec.com>
    Subject        Re: Web Site Performance
    References     1

    In article <skoogDs54BH.9pF@netcom.com> skoog@netcom.com (Jim Skoog) writes:
    >Where are the performance bottlenecks for Alpha AXP running the
    >Netscape Commerce Server 1.12 with high volume internet traffic?
    >We are evaluating network performance for a variety of Alpha AXP
    >runing DEC UNIX 3.2C, which run DEC's seal firewall and behind
    >that Alpha 1000 and 2100 webservers.

    Our experience (running such Web servers as altavista.digital.com
    and www.digital.com) is that there is one important kernel tuning
    knob to adjust in order to get good performance on V3.2C.  You
    need to patch the kernel global variable "somaxconn" (use dbx -k
    to do this) from its default value of 8 to something much larger.

    How much larger?  Well, no larger than 32767 (decimal).  And
    probably no less than about 2048, if you have a really high volume
    (millions of hits per day), like AltaVista does.

    This change allows the system to maintain more than 8 TCP
    connections in the SYN_RCVD state for the HTTP server.  (You
    can use "netstat -An |grep SYN_RCVD" to see how many such
    connections exist at any given instant).

    If you don't make this change, you might find that as the load gets
    high, some connection attempts take a very long time.  And if a lot
    of your clients disconnect from the Internet during the process of
    TCP connection establishment (this happens a lot with dialup
    users), these "embryonic" connections might tie up your somaxconn
    quota of SYN_RCVD-state connections.  Until the kernel times out
    these embryonic connections, no other connections will be accepted,
    and it will appear as if the server has died.

    The default value for somaxconn in Digital UNIX V4.0 will be quite
    a bit larger than it has been in previous versions (we inherited
    this default from 4.3BSD).

    Digital UNIX V4.0 includes some other performance-related changes
    that significantly improve its maximum HTTP connection rate.  However,
    we've been using V3.2C systems to front-end for altavista.digital.com
    with no obvious performance bottlenecks at the millions-of-hits-per-day
    level.

    We have some Webstone performance results available at
            http://www.digital.com/info/alphaserver/news/webff.html

    [The document referenced above is no longer at that URL -- Ed.]

    I'm not sure if these were done using V4.0 or an earlier version
    of Digital UNIX, although I suspect they were done using a test
    version of V4.0.

    -Jeff

    ----------------------------------------------------------------------------

    From           mogul@pa.dec.com (Jeffrey Mogul)
    Organization   DEC Western Research
    Date           31 May 1996 21:01:01 GMT
    Newsgroups     comp.unix.osf.osf1
    Message-ID     <4onmmd$mmd@usenet.pa.dec.com>
    Subject        Digital UNIX V3.2C Internet tuning patch info

    ----------------------------------------------------------------------------

    Something that probably few people are aware of is that Digital
    has a patch kit available for Digital UNIX V3.2C that may improve
    Internet performance, especially for busy web servers.

    This patch kit is one way to increase the value of somaxconn,
    which I discussed in a message here a day or two ago.

    I've included in this message the revised README file for this
    patch kit below.  Note that the original README file in the patch
    kit itself may be an earlier version; I'm told that the version
    below is the right one.

    Sorry, this patch kit is NOT available for other versions of Digital
    UNIX.  Most (but not quite all) of these changes also made it into V4.0,
    so the description of the various tuning parameters in this README
    file might be useful to people running V4.0 systems.

    This patch kit does not appear to be available (yet?) from
            http://www.service.digital.com/html/patch_service.html
    so I guess you'll have to call Digital's Customer Support to get it.

    -Jeff

    DESCRIPTION: Digital UNIX Network tuning patch

                 Patch ID: OSF350-146

                 SUPERSEDED PATCHES: OSF350-151, OSF350-158

            This set of files improves the performance of the network
            subsystem on a system being used as a web server.  There are
            additional tunable parameters included here, to be used
            cautiously by an informed system administrator.

    TUNING

            To tune the web server, the number of simultaneous socket
            connection requests are limited by:

            somaxconn               Sets the maximum number of pending requests
                                    allowed to wait on a listening socket.  The
                                    default value in Digital UNIX V3.2 is 8.
                                    This patch kit increases the default to 1024,
                                    which matches the value in Digital UNIX V4.0.

            sominconn               Sets the minimum number of pending connections
                                    allowed on a listening socket.  When a user
                                    process calls listen with a backlog less
                                    than sominconn, the backlog will be set to
                                    sominconn.  sominconn overrides somaxconn.
                                    The default value is 1.

            The effectiveness of tuning these parameters can be monitored by
            the sobacklog variables available in the kernel:

            sobacklog_hiwat         Tracks the maximum pending requests to any
                                    socket.  The initial value is 0.

            sobacklog_drops         Tracks the number of drops exceeding the
                                    socket set backlog limit.  The initial
                                    value is 0.

            somaxconn_drops         Tracks the number of drops exceeding the
                                    somaxconn limit.  When sominconn is larger
                                    than somaxconn, tracks the number of drops
                                    exceeding sominconn.  The initial value is 0.

            TCP timer parameters also affect performance. Tuning the following
            require some knowledge of the characteristics of the network.

            tcp_msl                 Sets the tcp maximum segment lifetime.
                                    This is the maximum lifetime in half
                                    seconds that a packet can be in transit
                                    on the network.  This value, when doubled,
                                    is the length of time a connection remains
                                    in the TIME_WAIT state after a incoming
                                    close request is processed.  The unit is
                                    specified in 1/2 seconds, the initial
                                    value is 60.

            tcp_rexmit_interval_min
                                    Sets the minimum TCP retransmit interval.
                                    For some WAN networks the default value may
                                    be too short, causing unnecessary duplicate
                                    packets to be sent.  The unit is specified
                                    in 1/2 seconds, the initial value is 1.

            tcp_keepinit            This is the amount of time a partially
                                    established connection will sit on the listen
                                    queue before timing out (e.g., if a client
                                    sends a SYN but never answers our SYN/ACK).
                                    Partially established connections tie up slots
                                    on the listen queue.  If the queue starts to
                                    fill with connections in SYN_RCVD state,
                                    tcp_keepinit can be decreased to make those
                                    partial connects time out sooner.  This should
                                    be used with caution, since there might be
                                    legitimate clients that are taking a while
                                    to respond to SYN/ACK.  The unit is specified
                                    in 1/2 seconds, the default value is 150
                                    (ie. 75 seconds).

            The hashlist size for the TCP inpcb lookup table is regulated by:

            tcbhashsize             The number of hash buckets used for the
                                    TCP connection table used in the kernel.
                                    The initial value is 32.  For best results,
                                    should be specified as a power of 2.  For
                                    busy Web servers, set this to 2048 or more.

            The hashlist size for the interface alias table is regulated by:

            inifaddr_hsize          The number of hash buckets used for the
                                    interface alias table used in the kernel.
                                    The initial value is 32.  For best results,
                                    should be specified as a power of 2.

            ipport_userreserved     The maximum number of concurrent non-reserved,
                                    dynamically allocated ports.  Default range
                                    is 1025-5000.  The maximum value is 65535.
                                    This limits the numer of times you can
                                    simultaneously telnet or ftp out to connect
                                    to other systems.

            tcpnodelack             Don't delay acknowledging TCP data; this
                                    can sometimes improve performance of locally
                                    run CAD packages.  Default is value is 0,
                                    the enabled value is 1.

                               Digital UNIX version:

                                      V3.2C
    Feature                    V3.2C  patch  V4.0
    =======                    =====  =====  ====
    somaxconn                   X      X      X
    sominconn                   -      X      X
    sobacklog_hiwat             -      X      -
    sobacklog_drops             -      X      -
    somaxconn_drops             -      X      -
    tcpnodelack                 X      X      X
    tcp_keepidle                X      X      X
    tcp_keepintvl               X      X      X
    tcp_keepcnt                 -      X      X
    tcp_keepinit                -      X      X
    TCP keepalive per-socket    -      -      X
    tcp_msl                     -      X      -
    tcp_rexmit_interval_min     -      X      -
    TCP inpcb hashing           -      X      X
    tcbhashsize                 -      X      X
    interface alias hashing     -      X      X
    inifaddr_hsize              -      X      X
    ipport_userreserved         -      X      -
    sysconfig -q inet           -      -      X
    sysconfig -q socket         -      -      X

### Apache HTTP Server Version 1.3