Linux Network Queuing
technial linux sysadmin network qos
Published: 2021-03-27

Classify and Prioritize Linux Network Traffic with PRIO qdisc and iptables.

Linux Network Queuing

Summary

This post:

  • setup outbound network traffic control on Linux
  • classify network traffic (“filter”) by iptables
  • prioritize traffic by filter

Motivation

I have a huge HDD Xen VPS (8TB, 4 shared CPU, 8GB RAM) from Servarica (Affiliate Link) for 20 USD/month.

Since the traffic is unlimited for 100mbps, I installed torrent client to [REDACTED].

I want to:

  • Prioritize outbound non torrent traffic,
    • VPN traffic (wireguard, nebula) first, followed by other traffic (e.g. http), and lastly torrent
  • Absolute prioritized, no fair queues
    • i.e. I don’t care if torrent speed drops to 0,
    • if VPN requested all 100mbps, it can have 100mbps, torrent “Windows 10 [REDACTED].iso” can “let them eat cake”

Backgrounds Study

I recommend the following websites.

ArchWiki is usually a good starting point.

[1] ArchWiki: Advanced traffic control

[2] Linux Advanced Routing & Traffic Control HOWTO, Ch 9.5. Classful Queueing Disciplines

[3] Linux Advanced Routing & Traffic Control HOWTO, Ch 9.6. Classifying packets with filters

This is an advanced article; you are expected to have certain knowledge of network devices, iptables, etc.

Bruh, I am an assistant system engineer (SRE), and I have no idea what I am doing. (Benny (author), 2021)

Qdisc

A Network scheduler.

A network scheduler, also called packet scheduler, queueing discipline, qdisc or queueing algorithm, is an arbiter on a node in packet switching communication network. It manages the sequence of network packets in the transmit and receive queues of the network interface controller.

Details are out of the scope of this post.

PRIO qdisc

To “Absolute prioritized, no fair queues”, PRIO qdisc is chosen.

From [3],

In formal words, the PRIO qdisc is a Work-Conserving scheduler.

Nice.

Class, filters

From [3],

When traffic enters a classful qdisc, it needs to be sent to any of the classes within - it needs to be ‘classified’. To determine what to do with a packet, the so called ‘filters’ are consulted. It is important to know that the filters are called from within a qdisc, and not the other way around!

The filters attached to that qdisc then return with a decision, and the qdisc uses this to enqueue the packet into one of the classes. Each subclass may try other filters to see if further instructions apply. If not, the class enqueues the packet to the qdisc it contains.

Since I finds iptables is more easy (palatable), iptables is used to filter.

If you don’t want to understand the full tc filter syntax, just use iptables, and only learn to select on fwmark. You can also have iptables print basic statistics that will help you debug your rules.

Steps

Environment

  • Debain 10 in Xen, other Linux Distro should works.

  • Public network interface: eth0

  • Root

Setup outbound network traffic control (qdisc)

Remove any existing qdisc from the root: tc qdisc del root dev eth0

If none exists, expect returning an error.

To setup a PRIO qdisc for eth0:

tc qdisc add dev eth0 root handle 1: prio
tc qdisc add dev eth0 parent 1:1 handle 10: fq_codel
tc qdisc add dev eth0 parent 1:2 handle 20: fq_codel
tc qdisc add dev eth0 parent 1:3 handle 30: fq_codel
  • The root qdisc for eth0 is now PRIO
  • Classes 1:1, 1:2, 1:3 are created by the first command
  • For 2nd - 3rd command, one child class with fq_codel qdisc is created for each class
    • fq_codel: “Since systemd 217, fq_codel is the default.” See [1].

The tree is as follows [2],

            1:   root qdisc
         /  |  \
        /   |   \
       /    |    \
     1:1   1:2    1:3    classes
      |     |      |
     10:   20:    30:    qdiscs    qdiscs
fq_codel fq_codel fq_codel
band  0    1    2
  • Class 1:1 has the highest priority, 1:3 lowest.
  • Class 1:2 is the default (unsure, maybe depends on the order of tc -s -d qdisc ls dev eth0 output. See below)

Classify network traffic (“filter”) by iptables

To classify which traffic needs prioritized,

  • iptables MARK packet with mark n
  • packet with mark n is then assign to defined classes
iptables -A OUTPUT -p udp --sport 12312 -j MARK --set-mark 6
tc filter add dev eth0 protocol ip parent 1: prio 1 handle 6 fw flowid 1:1

iptables -A DOCKER-USER -s 10.10.10.2 -j MARK --set-mark 7
tc filter add dev eth0 protocol ip parent 1: prio 1 handle 7 fw flowid 1:3

From the commands above,

  • OUTPUT from local port 12312 UDP, is marked with 6, an arbitrary number
    • 12312 is my VPN port, Wiregurad
    • For you, mark any hightest priority traffic with 6
  • packet marked with 6 is assigned to class 1:1, highest priority
  • FORWARD traffic from IP 10.10.10.2 is marked with 7
    • (Updated May 26, 2021) restarting a container will has a new interface. Static IP is used instead
      • IP 10.10.10.2 is the static assigned container IP for torrent client
      • For you, mark any lowest priority traffic with 7
  • packet marked with 7 is assigned to class 1:3, lowest priority

Verify

qdisc

To verify traffic are passed through desired class:

tc -s -d qdisc ls dev eth0

I recommend append watch to the command to view the output in real-time.

  • For root qdisc,
qdisc prio 1: root refcnt 5 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 701913253 bytes 498099 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  • For class 1:3, lowest priority, i.e. torrent traffic:

Notices the Sent 648 bytes 12 pkt

qdisc fq_codel 30: parent 1:3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn                                            
 Sent 648 bytes 12 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 54 drop_overlimit 0 new_flow_count 12 ecn_mark 0
  new_flows_len 0 old_flows_len 0
  • For class 1:1, highest priority, i.e. VPN traffic:
qdisc fq_codel 10: parent 1:1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn                                                                  
 Sent 46 bytes 1 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 46 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 0 old_flows_len 0
  • For class 1:2, unspecified traffic. Seems to be default, as the order of this output is 1:3 1:1 1:2.
    • Setup additional “catch all” filter if needed, e.g. iptables rules to mark all packets
qdisc fq_codel 20: parent 1:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn                                                                  
 Sent 264573127 bytes 186932 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 3028 drop_overlimit 0 new_flow_count 156856 ecn_mark 0
  new_flows_len 0 old_flows_len 0

Filter

To verify iptables is marking packets:

iptables -L OUTPUT -nv

Change OUTPUT to chain you need, e.g. DOCKER-USER

Again, I recommend appending watch to view output in real-time.

Test

watch the both commands above.

iperf3 is used to generated traffic.

  • Run iperf3 -s on the server ip, and iperf3 -c ip on the client.

    • ip is the public ip, packets are not mark
    • For class 1:2, Sent ... bytes ... pkt should increases with the rate of the transfer
    • Before running iperf3 -c, Sent n bytes
    • After running iperf3 -c, Sent n+x+y bytes, where x is the transferred bytes shown on output of iperf3 -c, y is other traffic
    • Also observe iptables pkts and bytes columns

  • Repeat iperf3 -c with VPN (marked with 6)

  • Start downloading/seeding torrent

Useful info

iptables chain graph

Contact

Anything wrong?

Email me: adam \a\ 2isk.in, replace \a\ with @