This section contains 'cookbook' entries which may help you solve problems. A cookbook is no replacement for understanding however, so try and comprehend what is going on.
You can do this in several ways. Apache has some support for this with a module, but we'll show how Linux can do this for you, and do so for other services as well. These commands are stolen from a presentation by Jamal Hadi that's referenced below.
Let's say we have two customers, with http, ftp and streaming audio, and we want to sell them a limited amount of bandwidth. We do so on the server itself.
Customer A should have at most 2 megabits, customer B has paid for 5 megabits. We separate our customers by creating virtual IP addresses on our server.
# ip address add 188.177.166.1 dev eth0 # ip address add 188.177.166.2 dev eth0
It is up to you to attach the different servers to the right IP address. All popular daemons have support for this.
We first attach a CBQ qdisc to eth0:
# tc qdisc add dev eth0 root handle 1: cbq bandwidth 10Mbit cell 8 avpkt 1000 \ mpu 64
We then create classes for our customers:
# tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 10Mbit rate \ 2MBit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21 # tc class add dev eth0 parent 1:0 classid 1:2 cbq bandwidth 10Mbit rate \ 5Mbit avpkt 1000 prio 5 bounded isolated allot 1514 weight 1 maxburst 21
Then we add filters for our two classes:
##FIXME: Why this line, what does it do?, what is a divisor?: ##FIXME: A divisor has something to do with a hash table, and the number of ## buckets - ahu # tc filter add dev eth0 parent 1:0 protocol ip prio 5 handle 1: u32 divisor 1 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.1 flowid 1:1 # tc filter add dev eth0 parent 1:0 prio 5 u32 match ip src 188.177.166.2 flowid 1:2
And we're done.
FIXME: why no token bucket filter? is there a default pfifo_fast fallback somewhere?
From Alexey's iproute documentation, adapted to netfilter and with more plausible paths. If you use this, take care to adjust the numbers to reasonable values for your system.
If you want to protect an entire network, skip this script, which is best suited for a single host.
It appears that you need the very latest version of the iproute2 tools to get this to work with 2.4.0.
#! /bin/sh -x # # sample script on using the ingress capabilities # this script shows how one can rate limit incoming SYNs # Useful for TCP-SYN attack protection. You can use # IPchains to have more powerful additions to the SYN (eg # in addition the subnet) # #path to various utilities; #change to reflect yours. # TC=/sbin/tc IP=/sbin/ip IPTABLES=/sbin/iptables INDEV=eth2 # # tag all incoming SYN packets through $INDEV as mark value 1 ############################################################ $iptables -A PREROUTING -i $INDEV -t mangle -p tcp --syn \ -j MARK --set-mark 1 ############################################################ # # install the ingress qdisc on the ingress interface ############################################################ $TC qdisc add dev $INDEV handle ffff: ingress ############################################################ # # # SYN packets are 40 bytes (320 bits) so three SYNs equals # 960 bits (approximately 1kbit); so we rate limit below # the incoming SYNs to 3/sec (not very useful really; but #serves to show the point - JHS ############################################################ $TC filter add dev $INDEV parent ffff: protocol ip prio 50 handle 1 fw \ police rate 1kbit burst 40 mtu 9k drop flowid :1 ############################################################ # echo "---- qdisc parameters Ingress ----------" $TC qdisc ls dev $INDEV echo "---- Class parameters Ingress ----------" $TC class ls dev $INDEV echo "---- filter parameters Ingress ----------" $TC filter ls dev $INDEV parent ffff: #deleting the ingress qdisc #$TC qdisc del $INDEV ingress
Recently, distributed denial of service attacks have become a major nuisance on the internet. By properly filtering and ratelimiting your network, you can both prevent becoming a casualty or the cause of these attacks.
You should filter your networks so that you do not allow non-local IP source addressed packets to leave your network. This stops people from anonymously sending junk to the internet.
Rate limiting goes much as shown earlier. To refresh your memory, our ASCIIgram again:
[The Internet] ---<E3, T3, whatever>--- [Linux router] --- [Office+ISP] eth1 eth0
We first set up the prerequisite parts:
# tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000 # tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10Mbit rate \ 10Mbit allot 1514 prio 5 maxburst 20 avpkt 1000
If you have 100Mbit, or more, interfaces, adjust these numbers. Now you need to determine how much ICMP traffic you want to allow. You can perform measurements with tcpdump, by having it write to a file for a while, and seeing how much ICMP passes your network. Do not forget to raise the snapshot length!
If measurement is impractical, you might want to choose 5% of your available bandwidth. Let's set up our class:
# tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10Mbit rate \ 100Kbit allot 1514 weight 800Kbit prio 5 maxburst 20 avpkt 250 \ bounded
This limits at 100Kbit. Now we need a filter to assign ICMP traffic to this class:
# tc filter add dev eth0 parent 10:0 protocol ip prio 100 u32 match ip protocol 1 0xFF flowid 10:100
If lots of data is coming down your link, or going up for that matter, and you are trying to do some maintenance via telnet or ssh, this may not go too well. Other packets are blocking your keystrokes. Wouldn't it be great if there were a way for your interactive packets to sneak past the bulk traffic? Linux can do this for you!
As before, we need to handle traffic going both ways. Evidently, this works best if there are Linux boxes on both ends of your link, although other UNIX's are able to do this. Consult your local Solaris/BSD guru for this.
The standard pfifo_fast scheduler has 3 different 'bands'. Traffic in band 0 is transmitted first, after which traffic in band 1 and 2 gets considered. It is vital that our interactive traffic be in band 0!
We blatantly adapt from the (soon to be obsolete) ipchains HOWTO:
There are four seldom-used bits in the IP header, called the Type of Service (TOS) bits. They effect the way packets are treated; the four bits are "Minimum Delay", "Maximum Throughput", "Maximum Reliability" and "Minimum Cost". Only one of these bits is allowed to be set. Rob van Nieuwkerk, the author of the ipchains TOS-mangling code, puts it as follows:
Especially the "Minimum Delay" is important for me. I switch it on for "interactive" packets in my upstream (Linux) router. I'm behind a 33k6 modem link. Linux prioritizes packets in 3 queues. This way I get acceptable interactive performance while doing bulk downloads at the same time.
The most common use is to set telnet & ftp control connections to "Minimum Delay" and FTP data to "Maximum Throughput". This would be done as follows, on your upstream router:
# iptables -A PREROUTING -t mangle -p tcp --sport telnet \ -j TOS --set-tos Minimize-Delay # iptables -A PREROUTING -t mangle -p tcp --sport ftp \ -j TOS --set-tos Minimize-Delay # iptables -A PREROUTING -t mangle -p tcp --sport ftp-data \ -j TOS --set-tos Maximize-Throughput
Now, this only works for data going from your telnet foreign host to your local computer. The other way around appears to be done for you, ie, telnet, ssh & friends all set the TOS field on outgoing packets automatically.
Should you have an application that does not do this, you can always do it with netfilter. On your local box:
# iptables -A OUTPUT -t mangle -p tcp --dport telnet \ -j TOS --set-tos Minimize-Delay # iptables -A OUTPUT -t mangle -p tcp --dport ftp \ -j TOS --set-tos Minimize-Delay # iptables -A OUTPUT -t mangle -p tcp --dport ftp-data \ -j TOS --set-tos Maximize-Throughput
This section was sent in by reader Ram Narula from Internet for Education (Thailand).
The regular technique in accomplishing this in Linux is probably with use of ipchains AFTER making sure that the "outgoing" port 80(web) traffic gets routed through the server running squid.
There are 3 common methods to make sure "outgoing" port 80 traffic gets routed to the server running squid and 4th one is being introduced here.
If you can tell your gateway router to match packets that has outgoing destination port of 80 to be sent to the IP address of squid server.
BUT
This would put additional load on the router and some commercial routers might not even support this.
Layer 4 switches can handle this without any problem.
BUT
The cost for this equipment is usually very high. Typical layer 4 switch would normally cost more than a typical router+good linux server.
You can force ALL traffic through cache server.
BUT
This is quite risky because Squid does utilize lots of cpu power which might result in slower over-all network performance or the server itself might crash and no one on the network will be able to access the internet if that occurs.
By using NetFilter another technique can be implemented which is using NetFilter for "mark"ing the packets with destination port 80 and using iproute2 to route the "mark"ed packets to the Squid server.
First, make all traffic pass through naret by making sure it is the default gateway except for silom. Silom's default gateway has to be donmuang (10.0.0.3) or this would create web traffic loop.|----------------| | Implementation | |----------------| Addresses used 10.0.0.1 naret (NetFilter server) 10.0.0.2 silom (Squid server) 10.0.0.3 donmuang (Router connected to the internet) 10.0.0.4 kaosarn (other server on network) 10.0.0.5 RAS 10.0.0.0/24 main network 10.0.0.0/19 total network |---------------| |Network diagram| |---------------| Internet | donmuang | ------------hub/switch---------- | | | | naret silom kaosarn RAS etc.
(all servers on my network had 10.0.0.1 as the default gateway which was the former IP address of donmuang router so what I did was changed the IP address of donmuang to 10.0.0.3 and gave naret ip address of 10.0.0.1)
Silom ----- -setup squid and ipchains
Setup Squid server on silom, make sure it does support transparent caching/proxying, the default port is usually 3128, so all traffic for port 80 has to be redirected to port 3128 locally. This can be done by using ipchains with the following:
silom# ipchains -N allow1 silom# ipchains -A allow1 -p TCP -s 10.0.0.0/19 -d 0/0 80 -j REDIRECT 3128 silom# ipchains -I input -j allow1
Or, in netfilter lingo:
silom# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
(note: you might have other entries as well)
For more information on setting Squid server please refer to Squid faq page on http://squid.nlanr.net).
Make sure ip forwarding is enabled on this server and the default gateway for this server is donmuang router (NOT naret).
Naret ----- -setup iptables and iproute2 -disable icmp REDIRECT messages (if needed)
naret# iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 80 \ -j MARK --set-mark 2
naret# echo 202 www.out >> /etc/iproute2/rt_tables naret# ip rule add fwmark 2 table www.out naret# ip route add default via 10.0.0.2 dev eth0 table www.out naret# ip route flush cache
If donmuang and naret is on the same subnet then naret should not send out icmp REDIRECT messages. In this case it is, so icmp REDIRECTs has to be disabled by:
naret# echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects naret# echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects naret# echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
The setup is complete, check the configuration
On naret: naret# iptables -t mangle -L Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- anywhere anywhere tcp dpt:www MARK set 0x2 Chain OUTPUT (policy ACCEPT) target prot opt source destination naret# ip rule ls 0: from all lookup local 32765: from all fwmark 2 lookup www.out 32766: from all lookup main 32767: from all lookup default naret# ip route list table www.out default via 203.114.224.8 dev eth0 naret# ip route 10.0.0.1 dev eth0 scope link 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.1 127.0.0.0/8 dev lo scope link default via 10.0.0.3 dev eth0 (make sure silom belongs to one of the above lines, in this case it's the line with 10.0.0.0/24) |------| |-DONE-| |------|
|-----------------------------------------| |Traffic flow diagram after implementation| |-----------------------------------------| INTERNET /\ || \/ -----------------donmuang router--------------------- /\ /\ || || || || || \/ || naret silom || *destination port 80 traffic=========>(cache) || /\ || || || \/ \/ \\===================================kaosarn, RAS, etc.
Note that the network is asymmetric as there is one extra hop on general outgoing path.
Here is run down for packet traversing the network from kaosarn to and from the internet. For web/http traffic: kaosarn http request->naret->silom->donmuang->internet http replies from internet->donmuang->silom->kaosarn For non-web/http requests(eg. telnet): kaosarn outgoing data->naret->donmuang->internet incoming data from internet->donmuang->kaosarn
For sending bulk data, the internet generally works better when using larger packets. Each packet implies a routing decision, when sending a 1 megabyte file, this can either mean around 700 packets when using packets that are as large as possible, or 4000 if using the smallest default.
However, not all parts of the internet support full 1460 bytes of payload per packet. It is therefore necessary to try and find the largest packet that will 'fit', in order to optimize a connection.
This process is called 'Path MTU Discovery', where MTU stands for 'Maximum Transfer Unit.'
When a router encounters a packet that's too big too send in one piece, AND it has been flagged with the "Don't Fragment" bit, it returns an ICMP message stating that it was forced to drop a packet because of this. The sending host acts on this hint by sending smaller packets, and by iterating it can find the optimum packet size for a connection over a certain path.
This used to work well until the internet was discovered by hooligans who do their best to disrupt communications. This in turn lead administrators to either block or shape ICMP traffic in a misguided attempt to improve security or robustness of their internet service.
What has happened now is that Path MTU Discovery is working less and less well and fails for certain routes, which leads to strange TCP/IP sessions which die after a while.
Although I have no proof for this, two sites who I used to have this problem with both run Alteon Acedirectors before the affected systems - perhaps somebody more knowledgeable can provide clues as to why this happens.
When you encounter sites that suffer from this problem, you can disable Path MTU discovery by setting it manually. Koos van den Hout, slightly edited, writes:
The following problem: I set the mtu/mru of my leased line running ppp to 296 because it's only 33k6 and I cannot influence the queueing on the other side. At 296, the response to a keypress is within a reasonable timeframe.
And, on my side I have a masqrouter running (of course) Linux.
Recently I split 'server' and 'router' so most applications are run on a different machine than the routing happens on.
I then had trouble logging into irc. Big panic! Some digging did find out that I got connected to irc, even showed up as 'connected' on irc but I did not receive the motd from irc. I checked what could be wrong and noted that I already had some previous trouble reaching certain websites related to the MTU, since I had no trouble reaching them when the MTU was 1500, the problem just showed when the MTU was set to 296. Since irc servers block about every kind of traffic not needed for their immediate operation, they also block icmp.
I managed to convince the operators of a webserver that this was the cause of a problem, but the irc server operators were not going to fix this.
So, I had to make sure outgoing masqueraded traffic started with the lower mtu of the outside link. But I want local ethernet traffic to have the normal mtu (for things like nfs traffic).
Solution:ip route add default via 10.0.0.1 mtu 296(10.0.0.1 being the default gateway, the inside address of the masquerading router)
In general, it is possible to override PMTU Discovery by setting specific routes. For example, if only a certain subnet is giving problems, this should help:
ip route add 195.96.96.0/24 via 10.0.0.1 mtu 1000
As explained above, Path MTU Discovery doesn't work as well as it should anymore. If you know for a fact that a hop somewhere in your network has a limited (<1500) MTU, you cannot rely on PMTU Discovery finding this out.
Besides MTU, there is yet another way to set the maximum packet size, the so called Maximum Segment Size. This is a field in the TCP Options part of a SYN packet.
Recent Linux kernels, and a few pppoe drivers (notably, the excellent Roaring Penguin one), feature the possibility to 'clamp the MSS'.
The good thing about this is that by setting the MSS value, you are telling the remote side unequivocally 'do not ever try to send me packets bigger than this value'. No ICMP traffic is needed to get this to work.
The bad thing is that it's an obvious hack - it breaks 'end to end' by modifying packets. Having said that, we use this trick in many places and it works like a charm.
In order for this to work you need at least iptables-1.2.1a and Linux 2.4.3 or higher. The basic commandline is:
# iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
This calculates the proper MSS for your link. If you are feeling brave, or think that you know best, you can also do something like this:
# iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
This sets the MSS of passing SYN packets to 128. Use this if you have VoIP with tiny packets, and huge http packets which are causing chopping in your voice calls.