tcp_tunning.txt

Kamailio TCP Tunning/monitoring for lots of open connections
==============================================================

0. Introduction
----------------

This document describes very briefly various settings that should improve
Kamailio TCP performance for sites handling a lot of TCP traffic (> 1000 open
connections or very high connection/disconnection rates).

For now it deals only with Linux specific optimizations.


1. Useful Linux kernel settings
-------------------------------

1.1 Connection rate/pending connections: by default the connection rate is
 too small

net.core.somaxconn             -  limit of the listen() backlog, default 128
net.ipv4.tcp_max_syn_backlog   -  default 1024 or 128
net.ipv4.tcp_timestamps        - default on., should be on (along with
                                 tcp_tw_recycle and timestamp supporting
                                 peers allows for fast connections rates)

1.2 Connection in close_wait

  Connection should stay as little as possible
  in close_wait to quickly free the fd/resources for new connections attempts
  WARNING: this could break normal TCP use, use it only if you know what you are
  doing

net.ipv4.tcp_max_tw_buckets - maximum number of timewait sockets
                              (the default seems to be ok)
net.ipv4.tcp_tw_recycle     - enables fast time wait sockets recycling (default
                              off), should be enabled if you have lots of short
                              lived connections
                              WARNING: see the above warning
net.ipv4.tcp_tw_reuse       - allows reusing of time-wait sockets (default off)
                              WARNING: see above

net.ipv4.tcp_syncookies     - default off, in this case it's probably better to
                              keep it off

1.3 Port range

net.ipv4.ip_local_port_range - should be increased (e.g. 4096-65534)

1.4 Open file descriptors

fs.file-max                 - maximum number of fds that will be allocated
                              (you probably need to increase it, default
                               depends on installed memory)

1.5 Buffers size

  When dealing with large data, check the values for network sockets buffers
  to be sufficient, among them:

net.core.rmem_max
net.core.wmem_max
net.core.rmem_default
net.core.wmem_default
net.ipv4.tcp_mem
net.ipv4.tcp_rmem
net.ipv4.tcp_wmem

  Note: sysctl -w net.ipv4.route.flush=1 - ensure that immediately subsequent
  connections use the values

1.6 Other sysctl that might affect tcp connection rate or the maximum number
    of open connections

fs.epoll.max_user_instances - maximum number of devices - per user (2.6.27.8+)
fs.epoll.max_user_watches   - maximum number of "watched" fds - per user
                              (2.6.27.8+)

net.ipv4.tcp_max_orphans    - might be worth a look if things go wrong
net.core.netdev_max_backlog - maximum device backlog

Related applications
--------------------
ifconfig <dev> txqueuelen <val> - set device transmission queue len

iptables  - remove the ip_conntrack module (it limits the maximum tcp
  connections, adds extra overhead (slow)). It's probably better to remove
  all the iptables modules.


2. Monitoring (values to watch for)
-----------------------------------

2.1 File descriptors

fs.dentry-state  - format: nr. dentries, nr. unused, age_limit, want_pages
fs.file-nr       - format: allocated, unused, max (==fs.file-max)
fs.inode-state  -  format: nr. allocated, nr. free, preshrink

2.2 TCP

/proc/net/netstat - the TW, TWRecycled, TWKilled, PAWPassive, PAWActive,
                    PASWEstab fields
                   ( cat /proc/net/netstat |cut -d" " -f12-17 ;
                     cat /proc/net/sockstat)
/proc/net/sockstat


3. Kamailio settings
----------------------

- Don't forget to increase tcp_max_connections and the amount of shared memory
- You should increase the number of Kamailio "tcp_children" processes (-N no)
  As a rule of thumb, (maximum simultaneous connections)/2000 should be ok
- You might have to decrease TCP_BUF_SIZE to a smaller value (e.g 8K)
- You might want to increase PKG_MEM_POOL_SIZE (for large queues)

- You might need to increase the maximum open fds limit before starting Kamailio
  (e.g. ulimit -n 1000000)