UA 1278534 - Performance Analysis Document

# Summary

It was brought to my attention that network namespace creation scalability was affected during kernel development.

The following script was used for all the tests and charts generation:

#!/bin/bash

IP=/sbin/ip

function add_fake_router_uuid() {

j=`uuidgen`

$IP netns add bar-${j}

$IP netns exec bar-${j} $IP link set lo up

$IP netns exec bar-${j} sysctl -w net.ipv4.ip_forward=1 > /dev/null

k=`echo $j | cut -b -11`

$IP link add qro-${k} type veth peer name qri-${k} netns bar-${j}

$IP link add qgo-${k} type veth peer name qgi-${k} netns bar-${j}

}

for i in `seq 1 $1`; do

if [ `expr $i % 250` -eq 0 ]; then

echo "$i by `date +%s`"

add_fake_router_uuid

done

I measured how many "fake routers" (above script) could be added per second from 0 to 4000 created routers mark. Using this script and a git bisect on kernel tree I was led to one specific commit causing regression: #911af50 "rcu: Provide compile-time control for no-CBs CPUs". Even Though this change was experimental at that point, it introduced a performance scalability regression (explained below) that still last and seems to be the default option for distributions nowadays.

RCU related code looked like to be responsible for the problem. With that, every commit from tag v3.8..master that changed any of this files: "kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h include/trace/events/rcu.h include/linux/rcupdate.h" was tested. The idea was to check performance regression during rcu development. In the worst case, the regression not being related to rcu, I would still have data to interpret the performance/scalability regression.

All text below this refer to 2 groups of charts, generated during the study:

1) Kernel git tags from 3.8 to 3.14.

2) Kernel git commits for rcu development (111 commits).

Since there was difference in results depending on how many cpus or how the no-cb cpus were configured, 3 kernel config options were used on every measure:

- CONFIG_RCU_NOCB_CPU (disabled): nocbno

- CONFIG_RCU_NOCB_CPU_ALL (enabled): nocball

- CONFIG_RCU_NOCB_CPU_NONE (enabled): nocbnone

Obs: For 1 cpu cases: nocbno, nocbnone, nocball behaves the same since w/ only 1 cpu there is no no-cb cpu

After charts being generated it was clear that NOCB_CPU_ALL (4 cpus) affected the "fake routers" creation process performance and this regression continues up to upstream version. It was also clear that, after commit #911af50, having more than 1 cpu does not improve performance/scalability for netns, makes it worse.

#911af50

...

+#ifdef CONFIG_RCU_NOCB_CPU_ALL

+ pr_info("\tExperimental no-CBs for all CPUs\n");

+ cpumask_setall(rcu_nocb_mask);

+#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */

...

Comparing standing out points (see charts):

#81e5949 - good

#911af50 - bad

I was able to see that, from the script above, the following lines causes major impact on netns scalability/performance:

1) ip netns add -> huge performance regression:

1 cpu: no regression

4 cpu: regression for NOCB_CPU_ALL

obs: regression from 250 netns/sec to 50 netns/sec on 500 netns already created mark

2) ip netns exec -> some performance regression

1 cpu: no regression

4 cpu: regression for NOCB_CPU_ALL

obs: regression from 40 netns (+1 exec per netns creation) to 20 netns/sec on 500 netns created mark

# Assumption (to be confirmed)

rcu callbacks being offloaded to other cpus caused regression in copy_net_ns<-created_new_namespaces or unshare(clone_newnet).

# Kernel .config

# Ubuntu default config file for kernel

config-3.8.0-39-generic does not configure NOCB

# CONFIG_RCU_NOCB_CPU is not set

CONFIG_RCU_FAST_NO_HZ=y

config-3.11.0-20-generic configures NOCB and NOCB_ALL

CONFIG_RCU_NOCB_CPU=y

# CONFIG_RCU_NOCB_CPU_NONE is not set

# CONFIG_RCU_NOCB_CPU_ZERO is not set

CONFIG_RCU_NOCB_CPU_ALL=y

CONFIG_RCU_FAST_NO_HZ=yls

config-3.13.0-24-generic configures NOCB and NOCB_ALL

CONFIG_RCU_NOCB_CPU=y

CONFIG_RCU_NOCB_CPU_ALL=y

CONFIG_RCU_FAST_NO_HZ=y

# Test case config file for kernel

All kernels ending with "-nocb" and tests like "nocbno-xxx"

# CONFIG_RCU_NOCB_CPU is not set

CONFIG_RCU_FAST_NO_HZ=y

All kernels ending with "-nonone" and tests like "nocbnone-xxx"

CONFIG_RCU_NOCB_CPU=y

CONFIG_RCU_NOCB_CPU_NONE=y

# CONFIG_RCU_NOCB_CPU_ZERO is not set

# CONFIG_RCU_NOCB_CPU_ALL is not set

CONFIG_RCU_FAST_NO_HZ=y

All kernels ending "-nocball" and tests like "nocball-xxx"

CONFIG_RCU_NOCB_CPU=y

# CONFIG_RCU_NOCB_CPU_NONE is not set

# CONFIG_RCU_NOCB_CPU_ZERO is not set

CONFIG_RCU_NOCB_CPU_ALL=y

CONFIG_RCU_FAST_NO_HZ=y

# How to see charts

In charts/250.html, if you check only blue, orange, "nocbnone-1cpu-250" and "nocbnone-4cpu-250" it will give you an overview of the performance regression from kernel version 3.9-rc2 to 3.15-rc5. 1 cpu is performing better than 4 cpus. Moving to the next chart (NEXT->), on the 500 fake routers created mark, the chart continues to show this behavior.

This procedure can be repeated comparing "nocbno", "nocbnone" and "nocball" for 1 and 4 cpus.

# notes

- 1 cpu performance is better than 4 cpu performance (does not scale)

# General Observation

1) Before #911af50 (#007) there was no such option as NOCB_CPU_ALL or NOCB_CPU_NONE. You can check that up to that commit the 1 cpu performance was lower than 4 cpu performance and the right throughput would be only those for "nocbno-1cpu-XXX" and "nocbno-4cpu-XXX".

2) The commits showed on the charts are commits that touched following files: kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h include/trace/events/rcu.h include/linux/rcupdate.h. Of course some other change/commit on the source code in between this commits can cause some kind of regression but, since the git bisect bad started with 911af50 (#007) the decision was to test all rcu changes.

3) All the observations were made on the "250 fake routers created" mark. It means that the test had just started. Sometimes it scales more (up to 2500 fake routers) sometimes not. The objective here was to set a tangible parameter for the analysis.

# This hole test started as a kernel bisect, resulting in a bad commit: 911af50 (#007).

# Group of changes that introduced performance regression for this testcase:

All comments (nocbno, nocbnone, nocball) are in the following format:

# 1st: performance regression 4 cpus: #007 (3.9.0-rc2)

---- * 65d798f rcu: Kick adaptive-ticks CPUs that are holding up RCU grace periods

#021 | *-. 6d87669 Merge branches 'doc.2013.03.12a', 'fixes.2013.03.13a' and 'idlenocb.2013.03.26b' into HEAD

---- | |\ \

---- |/ / /

#020 | | * 910ee45 rcu: Make rcu_accelerate_cbs() note need for future grace periods

#019 | | * 0446be4 rcu: Abstract rcu_start_future_gp() from rcu_nocb_wait_gp()

#018 | | * 8b425aa rcu: Rename n_nocb_gp_requests to need_future_gp

#017 | | * b846208 rcu: Push lock release to rcu_start_gp()'s callers

#016 | | * bd9f068 rcu: Repurpose no-CBs event tracing to future-GP events

#015 | | * b92db6c rcu: Rearrange locking in rcu_start_gp()

#014 | | * c0f4dfd rcu: Make RCU_FAST_NO_HZ take advantage of numbered callbacks

#013 | | * b11cc57 rcu: Accelerate RCU callbacks at grace-period end

#012 | | * 5e44ce3 rcu: Export RCU_FAST_NO_HZ parameters to sysfs

#011 | | * a488985 rcu: Distinguish "rcuo" kthreads by RCU flavor

#010 | | * 09c7b89 rcu: Add event tracing for no-CBs CPUs' grace periods

#009 | | * 21e7a60 rcu: Add event tracing for no-CBs CPUs' callback registration

#008 | | * dae6e64 rcu: Introduce proper blocking to no-CBs kthreads GP waits

#007 | | * 911af50 rcu: Provide compile-time control for no-CBs CPU

---- | | * 34ed624 rcu: Remove restrictions on no-CBs CPUs

---- | |/

---- |/|

#006 | * 81e5949 rcu: Tone down debugging during boot-up and shutdown.

#005 | * 6231069 rcu: Add softirq-stall indications to stall-warning messages

#004 | * 6f0a6ad rcu: Delete unused rcu_node "wakemask" field

#002 | * b0f7403 rcu: Avoid invoking RCU core on offline CPUs

---- *---. 40393f5 Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b'

OBS:

!!!! 1 CPU - good performance for nocbno, nocbnone, nocball

!!!! 4 CPU - good performance for nocbno, nocbnone

???? 4 CPU - bad performance for nocball

### 2nd: performance regression 4 cpus: #014 (3.9.0-rc2)

---- * 65d798f rcu: Kick adaptive-ticks CPUs that are holding up RCU grace periods

#021 | *-. 6d87669 Merge branches 'doc.2013.03.12a', 'fixes.2013.03.13a' and 'idlenocb.2013.03.26b' into HEAD

---- | |\ \

---- |/ / /

#020 | | * 910ee45 rcu: Make rcu_accelerate_cbs() note need for future grace periods

#019 | | * 0446be4 rcu: Abstract rcu_start_future_gp() from rcu_nocb_wait_gp()

#018 | | * 8b425aa rcu: Rename n_nocb_gp_requests to need_future_gp

#017 | | * b846208 rcu: Push lock release to rcu_start_gp()'s callers

#016 | | * bd9f068 rcu: Repurpose no-CBs event tracing to future-GP events

#015 | | * b92db6c rcu: Rearrange locking in rcu_start_gp()

#014 | | * c0f4dfd rcu: Make RCU_FAST_NO_HZ take advantage of numbered callbacks

#013 | | * b11cc57 rcu: Accelerate RCU callbacks at grace-period end

#012 | | * 5e44ce3 rcu: Export RCU_FAST_NO_HZ parameters to sysfs

#011 | | * a488985 rcu: Distinguish "rcuo" kthreads by RCU flavor

#010 | | * 09c7b89 rcu: Add event tracing for no-CBs CPUs' grace periods

#009 | | * 21e7a60 rcu: Add event tracing for no-CBs CPUs' callback registration

#008 | | * dae6e64 rcu: Introduce proper blocking to no-CBs kthreads GP waits

#007 | | * 911af50 rcu: Provide compile-time control for no-CBs CPUs

---- | | * 34ed624 rcu: Remove restrictions on no-CBs CPUs

---- | |/

---- |/|

---- | * 81e5949 rcu: Tone down debugging during boot-up and shutdown.

OBS:

!!!! 1 CPU - good performance for nocbno, nocbnone, nocball

???? 4 CPU - bad performance for nocbno, nocbnone (introduced on commit #014)

???? 4 CPU - bad performance for nocball (introduced on commit #007)

### 3rd: bad performance stall 4 cpus: between #014 and #021 (3.9.0-rc2 and 3.9.0)