RPi5: Pipe IPC & GPU Performance Regression – Kernel 6.6 → 6.12

[Kletternaut]

Platform

• Hardware: Raspberry Pi 5 (8 GB RAM), NVMe boot, active chassis cooling
• Kernels compared: 6.6.74+rpt-rpi-2712 vs 6.12.62+rpt-rpi-2712
• Both kernels tested on the same physical system — kernel 6.6.74 was booted via tryboot (sudo reboot "0 tryboot") with matching DTB and overlays extracted from the original kernel package. No reinstall, no hardware change.
• Date: April 2026

---

Background

Over the past 1.5 years I have been developing rpicam-gui, a dual-camera preview and capture application in C for the Raspberry Pi 5, using two IMX477 sensors at 2028×1520 @ 40 fps, alongside an active RDP session (two monitors) and a PiSP-accelerated AI inference pipeline (Hailo AIKit).

When I started this project on kernel 6.6.x the system handled this workload at roughly ~80–83% CPU load — challenging but workable. Over time, as the kernel progressed through the 6.12.x series, the same workload became increasingly difficult to run: frame drops, sluggish UI response over RDP, and eventually the application reaching its practical limit on hardware it ran fine on before.

The benchmarks below document a reproducible regression, most prominently in pipe IPC throughput (−26%), with additional degradation in GPU rendering and scheduler latency. These metrics compound in a multi-threaded camera pipeline and collectively explain the observed real-world performance loss.

---

Test Conditions

• Same Raspberry Pi 5 (8 GB), same NVMe drive, active chassis cooling
• No thermal throttling occurred on either kernel (max 64.2 °C / 63.1 °C)
• Overclocking identical on both: arm_freq=2800, over_voltage=4, gpu_freq=1000
• RDP session active (xrdp 0.9.21.1) — matching real-world workload
• All non-essential services stopped before each run

Note: Tests were performed on kernel 6.12.62, not the latest 6.12.x release.
I rely on a proprietary kernel module driver not yet recompiled for newer kernels,
so 6.12.62 was the highest testable version. Based on the consistent trend across
the 6.12.x series, the regression on the latest kernel is likely equal or worse.

---

Results

Benchmark                          Kernel 6.6.74       Kernel 6.12.62      Change
------------------------------------------------------------------------------------------
CPU single-thread (sysbench)       3191 ev/s           3189 ev/s           ≈ 0%
CPU multi-thread, 4 cores          12426 ev/s          12435 ev/s          ≈ 0%
RAM sequential read                64334 MiB/s         64569 MiB/s         ≈ 0%
RAM sequential write               14586 MiB/s         17102 MiB/s         +17%  (6.12 faster)
RAM random access                  959 MiB/s           874 MiB/s           -9%   (6.12 slower)
GPU glmark2-es2 total score        229                 222                 -3%
GPU bump scene                     318 FPS             299 FPS             -6%
GPU pulsar scene                   214 FPS             203 FPS             -5%
Context switching (stress-ng)      8648 ops/s          8581 ops/s          -0.8%
sched_yield latency                1905 ns/call        2054 ns/call        +7.8% (6.12 slower)
>>> Pipe IPC throughput <<<        2,353,000 ops/s     1,750,000 ops/s     -26%  <<<
>>> Pipe IPC throughput <<<        287 MB/s            214 MB/s            -26%  <<<
NVMe sequential write              355 MB/s            379 MB/s            +7%   (6.12 faster)
Max temperature (active cooling)   64.2 °C             63.1 °C             —
Thermal throttle events            0                   0                   —

---

Key Findings

• CPU raw compute is identical — this is not a clock speed or compiler regression.
• Pipe IPC throughput dropped 26% on 6.12.62. Linux pipes are the primary IPC mechanism between camera capture, ISP, preview and display threads in libcamera / rpicam-apps. A 26% drop here has a direct, measurable impact on any camera-intensive workload.
• sched_yield latency increased 7.8% — relevant for tight producer/consumer loops such as frame delivery between threads.
• GPU performance dropped 3–6% across glmark2-es2 scenes.
• RAM random access dropped 9% — relevant for ISP frame buffer access patterns.

The individual numbers may seem small in isolation, but they compound in a multi-threaded camera pipeline and all point in the same direction: 6.12.x is slower for IPC-heavy, display-intensive workloads on the RPi5.

---

Reproduction

Dependencies:

sudo apt install sysbench stress-ng glmark2-es2

The benchmark script used for both runs is available at:
https://forums.raspberrypi.com/viewtopic.php?t=397426

Key commands for the critical metric:

# Pipe IPC throughput
stress-ng --pipe 4 --timeout 20s --metrics-brief

# Scheduler yield latency
stress-ng --yield 4 --timeout 30s --metrics-brief

---

Request

Could someone with kernel access investigate the pipe IPC throughput regression between 6.6.x and 6.12.x on the RPi5 platform?

A git bisect between the last 6.6.x and first 6.12.x rpi tag on the stress-ng --pipe metric would likely narrow this down quickly.

Full raw benchmark output files are available on request.

Thank you for the continued work on the Raspberry Pi kernel.

Steps to reproduce the behaviour

https://forums.raspberrypi.com/viewtopic.php?t=397426

Device (s)

Raspberry Pi 5

System

$ cat /etc/rpi-issue
Raspberry Pi reference 2024-07-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 48efb5fc5485fafdc9de8ad481eb5c09e1182656, stage4

$ vcgencmd version
2026/02/06 14:31:40
Copyright (c) 2012 Broadcom
version 8124798b (release) (embedded)

$ uname -a
Linux raspi5 6.12.62+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.12.62-1+rpt1~bookworm (2026-01-19) aarch64 GNU/Linux

$ raspinfo | pastebinit
/usr/local/bin/raspinfo: Zeile 103: tvservice: Kommando nicht gefunden.
https://pastebin.com/DP60BvRq

Logs

No response

Additional context

No response

[popcornmix]

I added your benchmark script to my framework and ran it across 4 kernel versions on a Pi500+:

63228.0 / 18843.0 / 848.0 / 3516.0 / 2189.0 : MINOR=6,MAJOR=6
65280.0 / 19422.0 / 848.0 / 3511.0 / 2745.0 : MINOR=12,MAJOR=6
65571.0 / 19318.0 / 845.0 / 3938.0 / 2841.0 : MINOR=0,MAJOR=7
64522.0 / 22373.0 / 847.0 / 3941.0 / 3013.0 : MINOR=18,MAJOR=6

The numbers captured are:
sysbench: Memory read (sequential)/sysbench: Memory write (sequential)/sysbench: Memory random access/yield Kops/s (real time)/IPC Kops/s (real time)

For my tests, 6.12 looks better than 6.6, and the upcoming 6.18 is better still.

[Kletternaut]

Thanks for looking into this, @popcornmix!

However, I'd like to flag a methodological difference that makes your results and mine hard to compare directly — hence the title of this comment: apples to apples.

Differences between your test and mine

	@popcornmix	This report
Hardware	Pi 500	RPi 5 (8 GB), two independent units
Test framework	Your own framework	bench_rpi5.sh — identical script, both boards
stress-ng pipe stressor	Not mentioned	4 workers × 20 s
Kernels compared	6.6 vs 6.12	6.6.74 vs 6.12.62 vs 6.12.75

The Pi 500 uses the RP1 south bridge in a different board layout than the RPi 5. It is not the same platform, and IPC-heavy tests like stress-ng pipe are sensitive to memory subsystem and interconnect latency differences.

---

Independent replication on a second RPi 5 (kernel 6.12.75)

To rule out a flaky unit, I ran the identical script on a second RPi 5 (8 GB, hostname raspi6), configured identically:

Board	arm_freq	gpu_freq	over_voltage	governor
Pi #1 (raspi5)	2800 MHz	1000 MHz	4	ondemand
Pi #2 (raspi6)	2800 MHz	1000 MHz	4	ondemand

Results — three-way comparison (same script, same config)

Metric	Pi #1 · K 6.6.74 (baseline)	Pi #1 · K 6.12.62	Δ vs baseline	Pi #2 · K 6.12.75	Δ vs baseline
sysbench CPU 1T (events/s)	3 191	3 189	≈ 0%	3 188	≈ 0%
sysbench CPU 4T (events/s)	12 426	12 435	≈ 0%	12 733	≈ 0%
stress-ng pipe (ops/s)	2 352 633	1 750 101	−26%	1 891 030	−20%
stress-ng pipe (MB/s)	287	214	−26%	231	−20%
stress-ng swapcontext (calls/s)	2 372 054	2 353 090	≈ 0%	2 662 185	+12%
sched_yield latency (ns/call)	1 905	2 054	+7.8% (slower)	1 780	−6.5% (faster)
RAM random read (MiB/s)	959	874	−8.8%	963	≈ 0%
NVMe seq. write (MB/s) ¹	355	379	—	664 ²	—
No throttle events	✓	✓		✓

¹ NVMe write varies per device; not a kernel regression indicator here.
² Pi #2 uses a PCIe Gen3 dual-M.2 adapter — hardware difference, not comparable.

---

Key observations

1. CPU compute is identical across all three runs (sysbench 1T/4T within <0.1%). The clock and compiler are not responsible for any difference.

2. Pipe IPC regresses by ~26% on 6.12.62 and ~20% on 6.12.75, confirmed on two independent RPi 5 units with identical configuration. This is not a hardware fault.

3. sched_yield latency improved in 6.12.75 vs 6.12.62 (1780 ns vs 2054 ns), which may indicate partial progress. The pipe regression partly correlates with this.

4. RAM random read regressed in 6.12.62 (−8.8%) but recovered in 6.12.75 — suggests some changes between the two 6.12.x releases are relevant.

5. The stress-ng pipe stressor involves kernel pipe buffer management, scheduler wakeups, and inter-process data transfer — exactly the kind of workload affected by changes to the scheduler, pipe buffer sizing, or memory allocation paths between 6.6 and 6.12.

---

Request

Would it be possible to reproduce the stress-ng pipe test specifically on an RPi 5 (not Pi 500) using the same parameters?

stress-ng --pipe 4 --metrics-brief --timeout 20s 2>&1 | grep -E "pipe|metrc"

That would give us a clean apples-to-apples comparison. The difference in your IPC numbers (6.6=2189 Kops, 6.12=2745 Kops) vs mine (6.6=2353 Kops, 6.12=1750 Kops) is striking and the platform difference seems the most likely explanation.

Thanks for the continued attention on this!

[popcornmix]

I am using your bench_pi5.sh script untouched in my framework.
(The framework just allow config to be set, like switching kernel, a reboot and a custom script to be run, with output captured).

My kernels are built from top of tree kernel branches today.

Can you check if your Pi5 is a C1 or D0 revision? You can read that from the 2712 CPU text, but it may be easier to run

 $ sudo vclog -m  | grep bcm2712d0
006148.887: Loaded overlay 'bcm2712d0'

(a C1 won't have that line).

The chip revision and the RAM size are the only things that are likely to affect these results. Pi500+ is 16G.
I'm just running the test a few more times to see if there's much variation in scores.

I can later try to find a Pi5 8GB (with fan).

Can you confirm vcgencmd version, and any custom settings in rpi-eeprom-config, config.txt or cmdline.txt.

[popcornmix]

And can you repeat without overclocking?
I suspect you'll get similar results, but as overclocking is unsupported/ymmv any issue that only affects that will be low priority.

[popcornmix]

I've now done 5 runs. You can see there is a fair amount of noise on the results, so it's best not to try to draw conclusions on a single run.

64129.8 / 19538.4 / 848.8 / 3588.8 / 2626.4 : MINOR=12,MAJOR=6 [65294, 62264, 65230, 65326, 62535] / [18504, 20287, 19417, 18636, 20848] / [849, 849, 849, 848, 849] / [3513, 3566, 3649, 3526, 3690] / [2700, 2809, 2784, 2466, 2373]
65351.2 / 20612.0 / 847.4 / 3537.6 / 2674.6 : MINOR=6,MAJOR=6 [64569, 65095, 66675, 65311, 65106] / [19996, 21087, 22213, 19025, 20739] / [848, 846, 848, 847, 848] / [3519, 3559, 3511, 3514, 3585] / [2885, 2594, 2900, 2860, 2134]
64927.2 / 20873.2 / 846.6 / 4010.0 / 2878.0 : MINOR=0,MAJOR=7 [66344, 64845, 62003, 66464, 64980] / [21397, 19548, 20416, 21131, 21874] / [848, 848, 847, 845, 845] / [3936, 4077, 3977, 4035, 4025] / [2981, 2780, 2985, 2812, 2832]
63898.8 / 21147.0 / 844.0 / 3816.6 / 3056.0 : MINOR=18,MAJOR=6 [65173, 61605, 62953, 65210, 64553] / [24016, 21732, 19036, 19919, 21032] / [838, 839, 848, 848, 847] / [3909, 3880, 3841, 3747, 3706] / [3060, 3096, 3033, 3033, 3058]

[Kletternaut]

Thanks for the follow-up questions. Here are the answers and the stock-clock results.

Chip revision

$ sudo vclog -m | grep bcm2712d0
(no output)

This is a C1 revision (no D0 overlay loaded).

System info as requested

vcgencmd version:

2026/02/06 14:31:40
Copyright (c) 2012 Broadcom
version 8124798b (release) (embedded)

rpi-eeprom-config:

[all]
BOOT_UART=1
BOOT_ORDER=0xf146
NET_INSTALL_AT_POWER_ON=1
POWER_OFF_ON_HALT=1
WAKE_ON_GPIO=0

cmdline.txt:

console=tty1 root=PARTUUID=aef3a64d-02 rootfstype=ext4 fsck.repair=yes rootwait

config.txt (relevant excerpt — OC lines are commented out for the stock runs):

dtparam=pciex1_gen=3
#arm_freq=2800
#gpu_freq=1000
#over_voltage=4

---

Results without overclocking (arm_freq=2400, over_voltage=0)

I repeated the benchmark on both kernels with all OC disabled, VS Code closed, and background services minimised (fan controller and LCD display service stopped). xrdp was active on both runs, matching real-world conditions.

Metric	6.6.74 · stock	6.12.62 · stock	Δ
sysbench CPU 1T (events/s)	2 737	2 735	≈ 0%
sysbench CPU 4T (events/s)	10 709	10 752	≈ 0%
stress-ng pipe (ops/s)	1 996 034	1 580 693	−21%
stress-ng pipe (MB/s)	243	193	−21%
sched_yield latency (ns/call)	2 180	2 365	+8.5% (slower)
RAM random read (MiB/s)	826	768	−7%
glmark2-es2 score	214	205	−4%
Throttle events	0	0

Combined overview — all four runs on the same RPi 5 C1 unit

Metric	6.6.74 OC	6.12.62 OC	Δ (OC)	6.6.74 stock	6.12.62 stock	Δ (stock)
sysbench CPU 1T	3 191	3 189	≈ 0%	2 737	2 735	≈ 0%
sysbench CPU 4T	12 426	12 435	≈ 0%	10 709	10 752	≈ 0%
pipe (ops/s)	2 352 633	1 750 101	−26%	1 996 034	1 580 693	−21%
pipe (MB/s)	287	214	−26%	243	193	−21%
sched_yield (ns)	1 905	2 054	+7.8%	2 180	2 365	+8.5%
RAM random (MiB/s)	959	874	−9%	826	768	−7%

The regression is consistent regardless of overclocking: −21% at stock, −26% with OC. CPU compute stays flat in all four runs, confirming this is not a clock-speed or frequency-scaling issue.

---

Note: The Raspberry Pi forum thread was closed by a moderator with "Let's keep the discussion to the GitHub issue" — so continuing here.

[popcornmix]

Here's scores for multiple runs on an 4GB C1 Pi5.
6.18 kernel looks like the best to me.
Note: this will become the default apt kernel very soon.
Can you test with that?

63721.4 / 18473.5 / 844.2 / 3658.6 / 2495.8 : MAJOR=7 [63393, 67055, 56450, 64531, 66733, 64247, 65314, 64181, 65180, 61974, 64180, 64373, 66403, 62470, 65773, 61704, 62598, 65833, 65581, 56456] / [20323, 19861, 19434, 22175, 15780, 19050, 16091, 16010, 19067, 19558, 17218, 19341, 15640, 21619, 19921, 16713, 18840, 15751, 19987, 17091] / [847, 848, 849, 835, 849, 846, 848, 848, 841, 849, 849, 827, 848, 831, 849, 845, 847, 846, 841, 841] / [3650, 3674, 3600, 3646, 3609, 3698, 3745, 3755, 3633, 3702, 3653, 3701, 3681, 3623, 3592, 3694, 3591, 3680, 3646, 3599] / [2226, 2380, 2755, 2329, 2705, 2588, 2351, 2439, 2771, 2841, 2260, 2413, 2435, 2402, 2307, 2288, 2380, 2713, 2556, 2776] -3.9%
64519.8 / 16809.3 / 842.5 / 3370.4 / 2597.5 : MINOR=6,MAJOR=6 [63264, 64000, 64966, 66348, 63727, 64875, 64253, 64301, 65698, 63946, 64968, 62364, 66173, 63262, 64806, 65918, 64594, 66251, 63347, 63335] / [14012, 15094, 14601, 17599, 15467, 19563, 15954, 17833, 14971, 20949, 16185, 17890, 17204, 14682, 16937, 20005, 17553, 17959, 15217, 16512] / [846, 848, 847, 823, 844, 846, 848, 844, 842, 847, 842, 837, 844, 846, 828, 840, 849, 845, 848, 835] / [3375, 3370, 3337, 3476, 3313, 3364, 3279, 3299, 3341, 3392, 3378, 3293, 3402, 3436, 3427, 3406, 3298, 3398, 3531, 3293] / [2478, 2779, 2791, 2762, 2779, 2731, 2585, 2637, 2612, 2556, 2662, 2175, 2638, 2769, 2565, 2268, 2651, 2775, 2455, 2282]
63399.9 / 18695.7 / 845.0 / 3334.6 / 2682.6 : MINOR=12,MAJOR=6 [57169, 65922, 66913, 59739, 63312, 61758, 64876, 64537, 63670, 64677, 62876, 65520, 65569, 60616, 62444, 62787, 63025, 64161, 65882, 62546] / [13509, 18106, 19838, 18061, 19817, 15008, 17992, 19163, 18598, 18640, 19617, 21306, 15529, 21631, 19575, 17910, 20967, 18571, 19592, 20484] / [845, 844, 849, 849, 834, 850, 844, 846, 848, 838, 850, 848, 849, 850, 848, 833, 844, 848, 840, 844] / [3323, 3408, 3573, 3221, 3281, 3463, 3304, 3368, 3251, 3279, 3275, 3405, 3330, 3288, 3260, 3292, 3319, 3332, 3259, 3461] / [2562, 2859, 2247, 2352, 2806, 2586, 2823, 2387, 2453, 2851, 2843, 2732, 2847, 2830, 2826, 2804, 2543, 2823, 2813, 2664] +3.3%
63454.9 / 19174.1 / 843.0 / 3607.3 / 3134.8 : MINOR=18,MAJOR=6 [63079, 59181, 66965, 66726, 60004, 57889, 66363, 61682, 65214, 61334, 65362, 64227, 64965, 64389, 60846, 60168, 65927, 65919, 62297, 66562] / [12659, 17600, 18508, 15865, 14990, 19269, 20773, 19976, 19116, 19757, 19775, 22227, 19272, 21276, 20920, 21082, 20596, 19868, 21333, 18620] / [844, 846, 841, 847, 832, 848, 847, 848, 847, 834, 837, 844, 843, 846, 846, 847, 841, 835, 840, 846] / [3639, 3629, 3737, 3592, 3626, 3695, 3588, 3538, 3546, 3614, 3636, 3697, 3594, 3674, 3439, 3450, 3771, 3544, 3550, 3588] / [3161, 3137, 3120, 3129, 3120, 3155, 3110, 3112, 3162, 3124, 3138, 3145, 3150, 3143, 3134, 3152, 3090, 3158, 3113, 3143] +20.7%!

[Kletternaut]

Thank you for looking into this and for encouraging the 6.18 test — I went ahead and ran the full benchmark suite on 6.18.21-v8-16k+ as well, both with and without overclocking, on the same RPi 5 C1 unit as before.

I have to say the results are still disappointing. The pipe regression hasn't fully recovered — it's better than 6.12.x, but still clearly below the 6.6.74 baseline. Here's the full picture:

---

Results without overclocking (arm_freq=2400, over_voltage=0)

All runs: OC disabled, VS Code closed, background services minimised (fan controller and LCD display service stopped). xrdp active on all runs to match real-world conditions.

Metric	6.6.74 · stock	6.12.62 · stock	6.12.75 · stock	6.18.21 · stock	Δ (6.6→6.18.21)
sysbench CPU 1T (events/s)	2 737	2 735	2 735	2 738	≈ 0%
sysbench CPU 4T (events/s)	10 709	10 752	10 791	10 841	+1%
stress-ng pipe (ops/s)	1 996 034	1 580 693	1 676 578	1 796 734	−10%
stress-ng pipe (MB/s)	243	193	204	219	−10%
sched_yield latency (ns/call)	2 180	2 365	2 240	2 073	−5% (faster)
RAM random read (MiB/s)	826	768	778	836	+1%
glmark2-es2 score	214	205	205	213	≈ 0%
Throttle events	0	0	0	0

Combined overview — all eight runs on the same RPi 5 C1 unit

Metric	6.6.74 OC	6.12.62 OC	6.12.75 OC	6.18.21 OC	6.6.74 stock	6.12.62 stock	6.12.75 stock	6.18.21 stock
sysbench CPU 1T	3 191	3 189	3 209	3 195	2 737	2 735	2 735	2 738
sysbench CPU 4T	12 426	12 435	12 607	12 562	10 709	10 752	10 791	10 841
pipe (ops/s)	2 352 633	1 750 101	1 651 427	2 049 701	1 996 034	1 580 693	1 676 578	1 796 734
pipe (MB/s)	287	214	202	250	243	193	204	219
sched_yield (ns)	1 905	2 054	1 900	1 800	2 180	2 365	2 240	2 073
RAM random (MiB/s)	959	874	900	985	826	768	778	836

The trend is encouraging — the pipe regression is recovering kernel by kernel (6.12.62: −21%, 6.12.75: −16%, 6.18.21: −10%) — but it is still measurably below 6.6.74 in every single run. CPU compute, memory bandwidth and GPU score are completely flat across all eight runs, which rules out clock speed, DVFS or load-related effects. This really does seem to be a scheduler/pipe throughput regression introduced somewhere after 6.6.

I really appreciate the time you're putting into this!

[popcornmix]

Can you confirm exactly which value you are using for pipe (ops/s) row?

I have (on 6.18 kernel):

--- stress-ng: Pipe IPC throughput (KEY METRIC) ---
stress-ng: info:  [2833] setting to a 20 secs run per stressor
stress-ng: info:  [2833] dispatching hogs: 4 pipe
stress-ng: info:  [2833] note: /proc/sys/kernel/sched_autogroup_enabled is 1 and this can impact scheduling throughput for processes not attached to a tty. Setting this to 0 may improve performance metrics
stress-ng: info:  [2833] note: boost is disabled and this may impact top performance; setting /sys/devices/system/cpu/cpufreq/boost to 1 may improve performance.
stress-ng: metrc: [2833] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2833]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2833] pipe           63226427     20.00      8.18     71.45   3161294.56      794007.77
stress-ng: info:  [2833] skipped: 0
stress-ng: info:  [2833] passed: 4: pipe (4)
stress-ng: info:  [2833] failed: 0
stress-ng: info:  [2833] metrics untrustworthy: 0
stress-ng: info:  [2833] successful run completed in 20.01 secs

none of which seem close to your scores.

[Kletternaut]

Thanks for the follow-up questions. Here are the answers and the stock-clock results.

Note on stress-ng version and which value is reported

The pipe (ops/s) value in all tables is the bogo ops/s (real time) column — the second-to-last field in the metrc line. Example from the 6.6.78 stock run:

stress-ng: metrc: [4948] pipe  42587445  20.00  21.26  57.28  2129330.67  542250.77
                                                                ^^^^^^^^^^
                                                        bogo ops/s (real time) — this is the value used

All runs on this machine use stress-ng 0.15.06 (the version shipped with Raspberry Pi OS Bookworm). The numbers are therefore not directly comparable with results from newer stress-ng versions, which changed how the pipe stressor counts operations (later versions count individual read/write syscalls rather than round-trips, roughly doubling the reported bogo ops/s without any actual throughput change). The meaningful signal here is the relative difference between kernel versions on the same installation — which is consistent and reproducible across all ten runs.

---

Results without overclocking (arm_freq=2400, over_voltage=0)

I repeated the benchmark on all kernels with all OC disabled and background services minimised (fan controller and LCD display service stopped). xrdp was active on all runs, matching real-world conditions.

Metric	6.6.74 · stock	6.6.78 · stock	Δ (6.6.74→6.6.78)	6.12.62 · stock	6.12.75 · stock	6.18.21 · stock
sysbench CPU 1T (events/s)	2 737	2 728	≈ 0%	2 735	2 735	2 738
sysbench CPU 4T (events/s)	10 709	10 769	+1%	10 752	10 791	10 841
stress-ng pipe (ops/s)	1 996 034	2 129 331	+7%	1 580 693	1 676 578	1 796 734
stress-ng pipe (MB/s)	243	260	+7%	193	204	219
sched_yield latency (ns/call)	2 180	2 072	−5% (faster)	2 365	2 240	2 073
RAM random read (MiB/s)	826	823	≈ 0%	768	778	836
glmark2-es2 score	214	217	+1%	205	205	213
Throttle events	0	0		0	0	0

Combined overview — all ten runs on the same RPi 5 C1 unit

Metric	6.6.74 OC	6.6.78 OC	6.12.62 OC	6.12.75 OC	6.18.21 OC	6.6.74 stock	6.6.78 stock	6.12.62 stock	6.12.75 stock	6.18.21 stock
sysbench CPU 1T	3 191	3 194	3 189	3 209	3 195	2 737	2 728	2 735	2 735	2 738
sysbench CPU 4T	12 426	12 562	12 435	12 607	12 562	10 709	10 769	10 752	10 791	10 841
pipe (ops/s)	2 352 633	2 487 746	1 750 101	1 651 427	2 049 701	1 996 034	2 129 331	1 580 693	1 676 578	1 796 734
pipe (MB/s)	287	304	214	202	250	243	260	193	204	219
sched_yield (ns)	1 905	1 831	2 054	1 900	1 800	2 180	2 072	2 365	2 240	2 073
RAM random (MiB/s)	959	985	874	900	985	826	823	768	778	836

Update — 6.6.78-v8-16k+ added (self-compiled from rpi-6.6.y):

The 6.6.78 results are the most interesting data point yet. The pipeline regression is completely absent: stock pipe throughput is 2 129 331 ops/s — +6.7% above the 6.6.74 baseline, and OC reaches 2 487 746 ops/s (+5.7% vs 6.6.74 OC). The 6.6.y stable branch has never shown this regression.

This strongly suggests the regression was introduced in the 6.12 kernel series and is not present in the 6.6.y LTS branch at all. The partial recovery seen across 6.12.62 → 6.12.75 → 6.18.21 (−21% → −16% → −10%) does not yet close the gap. CPU compute, memory bandwidth and GPU score remain flat across all ten runs, ruling out clock-speed or DVFS effects.

[popcornmix]

Some runs on bookworm:
sysbench: Memory read (sequential)/sysbench: Memory write (sequential)/sysbench: Memory random access/yield Kops/s (real time)/IPC Kops/s (real time)

65899.9 / 16804.7 / 851.3 / 3903.7 / 1570.2 : MINOR=0,MAJOR=7 / DISTRO=1 [64868, 66029, 65795, 65854, 67026, 68953, 65517, 66319, 66429, 66748, 67185, 65248, 65614, 63899, 66663, 63914, 65482, 65043, 66172, 65241] / [16397, 15691, 16816, 15533, 16554, 15207, 24288, 16144, 16864, 16488, 15820, 15408, 18464, 17388, 16198, 16147, 16560, 17043, 16124, 16959] / [851, 845, 845, 853, 853, 854, 853, 853, 855, 848, 853, 851, 853, 853, 853, 836, 861, 853, 851, 852] / [3894, 3857, 3850, 3885, 3832, 3915, 3918, 3981, 3914, 3899, 3845, 3863, 3953, 3909, 3871, 3919, 3927, 3935, 3978, 3929] / [1422, 1671, 1712, 1419, 1423, 1678, 1698, 1419, 1698, 1705, 1652, 1674, 1684, 1438, 1445, 1415, 1395, 1716, 1688, 1451] -13.0%!
63097.8 / 19357.3 / 850.5 / 3365.5 / 1662.0 : MINOR=12 / DISTRO=1 [64284, 63200, 65800, 61892, 64536, 61059, 62968, 63112, 57216, 63413, 62819, 64270, 62239, 61742, 63965, 65442, 63984, 65141, 64051, 60823] / [18569, 17042, 19951, 21467, 16379, 21296, 17238, 18896, 22997, 19955, 19108, 17077, 22313, 16475, 17459, 21001, 16235, 20379, 22552, 20758] / [851, 846, 852, 856, 845, 852, 852, 850, 852, 852, 854, 854, 853, 853, 846, 853, 852, 852, 844, 842] / [3369, 3623, 3359, 3419, 3212, 3389, 3420, 3185, 3384, 3381, 3361, 3399, 3428, 3439, 3424, 3363, 3398, 3325, 3193, 3239] / [1656, 1441, 1831, 1390, 1433, 1401, 1941, 1937, 1988, 1960, 1456, 1976, 1439, 1439, 1527, 1539, 1911, 1893, 1650, 1432] -7.9%!
65704.4 / 18530.6 / 853.1 / 3691.6 / 1805.0 : DISTRO=1 [64489, 65711, 66823, 68248, 66925, 66217, 65514, 63794, 66030, 65226, 65596, 66229, 67169, 66794, 65619, 65621, 62954, 63848, 65551, 65730] / [19546, 18667, 23195, 16780, 19468, 18136, 20021, 19626, 16902, 18880, 18046, 17908, 15798, 17599, 20003, 16907, 19039, 18146, 16141, 19804] / [853, 860, 851, 853, 853, 852, 853, 852, 851, 852, 853, 854, 853, 854, 851, 854, 853, 852, 854, 854] / [3699, 3723, 3704, 3645, 3514, 3832, 3744, 3745, 3571, 3555, 3611, 3699, 3807, 3733, 3728, 3721, 3542, 3717, 3721, 3821] / [1807, 1831, 1807, 1827, 1834, 1850, 1805, 1821, 1813, 1830, 1818, 1822, 1803, 1825, 1642, 1830, 1841, 1818, 1652, 1823]
65117.7 / 21699.2 / 849.2 / 3398.1 / 2064.8 : MINOR=6 / DISTRO=1 [64276, 65414, 65262, 65136, 63935, 65082, 65067, 64576, 64621, 65547, 64782, 66270, 63128, 65242, 65702, 66994, 65353, 64367, 65363, 66236] / [20489, 19907, 20750, 23310, 22075, 23395, 19964, 21894, 22808, 22298, 21387, 20796, 20867, 23274, 23331, 19022, 23346, 23396, 20169, 21507] / [853, 851, 852, 850, 851, 855, 851, 849, 850, 851, 852, 852, 852, 851, 851, 831, 830, 850, 851, 851] / [3506, 3415, 3405, 3403, 3493, 3450, 3402, 3286, 3288, 3465, 3356, 3421, 3280, 3375, 3414, 3466, 3350, 3415, 3300, 3472] / [2087, 1957, 2125, 2073, 1968, 2078, 2047, 2124, 2005, 2048, 2051, 2180, 2079, 2088, 2019, 2084, 2033, 2129, 2045, 2077] +14.4%!

The final number is Pipe IPC and the numbers are similar to yours.

[techyminati]

@popcornmix @Kletternaut

Hi,
I wanted to share my personal experience here, although I haven’t done deep benchmarking you all in this thread.

I recently bought a Raspberry Pi 500 and have been using it as a normal desktop machine with Raspberry Pi OS (Bookworm). I’m not running any heavy workloads, I run mostly just a browser, some basic editing (like notepad), and general day-to-day usage.

I don’t have much experience running stress tests like stress-ng, but I did try a few basic runs. On kernel 6.12, the system felt noticeably sluggish to me overall. Even outside benchmarks, the UI responsiveness and general smoothness didn’t feel great.
I also tried Raspberry Pi OS Trixie (with similar newer kernels), and the experience was even worse in terms of perceived smoothness.
However, when I rolled back to an older OS and setup with kernel 6.6, the difference was quite clear that the system felt much smoother and more responsive in normal usage. Same hardware, same usage pattern.
So while I can’t provide detailed metrics or analysis, from a regular desktop-user perspective, I can confirm that something does feel off with newer kernels compared to 6.6.

Just wanted to share my experience (was searching for similar issue and found this).

Thanks for all the work on this!

[Kletternaut]

Bookworm results confirmed — and a note on real-world impact

Thank you for the Bookworm re-runs, @popcornmix. With DISTRO=1 your IPC numbers now align closely with mine (your 6.6: ~2065 Kops vs my 1996 Kops; 6.12: ~1662 vs 1581; 6.18: ~1805 vs 1797 — differences within noise). This effectively confirms the regression is reproducible on the same OS/stress-ng version combination.

One new data point from your run stands out: Linux 7.0 shows 1570 Kops — lower than 6.12, and the lowest of all four versions. The partial recovery in 6.18 does not continue into 7.0. That's worth noting if 6.18 is becoming the default apt kernel soon.

Real-world impact: I2C motor control and camera calibration

Beyond the synthetic benchmarks, I want to describe one concrete real-world manifestation that I think illustrates the regression well.

My setup includes a custom DIY camera system built on top of rpicam-gui, using two IMX477 sensors. Lens focus and zoom are controlled by an ESP32-based motor controller, connected to the Raspberry Pi 5 via I2C. Control commands flow through a custom V4L2 subdevice kernel module (hocusfocus.ko) — an I2C driver that translates v4l2-ctl commands into motor positions over the I2C bus, with libcamera and rpicam-gui providing the live preview feedback loop.

The calibration workflow is a tight, interactive feedback loop:

1. A command is issued via v4l2-ctl → kernel module sends I2C write → motor moves
2. The new lens position is captured by libcamera via the camera pipeline (V4L2 → ISP → pipe IPC → rpicam-gui)
3. The live preview reflects the result and the user issues the next command

Under kernel 6.6, this loop felt tight and predictable: commands translated into motor movements with consistent, smooth response. The VCM actuator sound — audible on each move — had a sharp, well-defined character.

Under 6.12+, the same loop is noticeably degraded: motor responses feel less precise and less repeatable. The feedback between command and visible result in the preview has increased latency and jitter. Calibration results are less reproducible run-to-run, and the process requires more correction iterations to converge. The VCM actuator sound changed character too — less crisp, which I interpret as command timing becoming less deterministic.

This is consistent with what the benchmarks show: pipe IPC regression means slower frame delivery between libcamera threads and the preview; increased sched_yield latency means the producer/consumer loop between capture, ISP and display is less tight. Together they degrade any workflow that relies on low-latency feedback between a hardware actuator and a camera preview.

I suspect this class of workload — tight I2C/V4L2 control loops with real-time visual feedback — is common in industrial and embedded deployments of the RPi 5: machine vision, pick-and-place, optical inspection, laser alignment. For those use cases, a −10% to −21% pipe throughput regression combined with increased scheduler latency is not a benchmark curiosity but a direct reliability and throughput concern. The RPi 5 is increasingly being evaluated for exactly these applications, and a kernel regression in this area could become a significant adoption barrier.

---

One potentially useful data point

The fact that 6.6.78 (self-compiled from rpi-6.6.y) shows no regression — scoring +7% above the 6.6.74 baseline — might give you a useful anchor if you're looking for where this was introduced. The regression pattern 7.0 < 6.12 < 6.18 < 6.6 also suggests it may be an interaction of multiple mainline changes rather than a single commit.

I'm happy to run any specific test or provide raw benchmark files if that would help.

---

@techyminati — your description matches my experience exactly. The subjective feel of the system under 6.12+ is noticeably different from 6.6, even for workloads that don't show up in CPU benchmarks. I'm glad someone else noticed it independently.

[popcornmix]

I should say that we would have changed nothing that is likely to affect these tests.
The pipe test is likely a combination of context switch, task wakeup and memory copy of the piped data.

So any regression will be upstream, and would need to be reported and fixed there.

But I did have an idea and ran a benchmark run with mitigations=off added to command line.
mitigations (i.e. workarounds for vulnerabilities like spectre) have been measured to be low for cpu bound benchmarks, but the workarounds are typically handled at context switch (e.g. flushing caches) so could come into play here.

There was an improvement (2.6% on 7.0 kernel), but it's not the only cause.

65883.2 / 20497.6 / 853.0 / 3380.2 / 1666.8 : DISTRO=1 [64630, 65913, 66253, 66856, 65764] / [23623, 19191, 21937, 15754, 21983] / [852, 852, 857, 853, 851] / [3380, 3486, 3356, 3198, 3481] / [1705, 1693, 1682, 1697, 1557]
65909.2 / 20738.8 / 852.0 / 3397.6 / 1691.2 : MITIGATIONS0=1 / DISTRO=1 [64243, 66252, 65917, 66966, 66168] / [22087, 21269, 21033, 18551, 20754] / [849, 855, 851, 854, 851] / [3319, 3452, 3385, 3383, 3449] / [1741, 1598, 1632, 1746, 1739] +0.5%
64134.6 / 20780.6 / 853.6 / 3920.4 / 1575.6 : MINOR=0,MAJOR=7 / DISTRO=1 [64723, 64682, 65576, 61618, 64074] / [23224, 22518, 19732, 22661, 15768] / [853, 853, 854, 855, 853] / [3861, 3981, 3916, 3880, 3964] / [1620, 1622, 1392, 1615, 1629] +16.0%!
63670.6 / 21447.6 / 851.8 / 4010.0 / 1625.2 : MITIGATIONS0=1 / MINOR=0,MAJOR=7 / DISTRO=1 [66651, 60390, 65457, 64990, 60865] / [21815, 22687, 19211, 20794, 22731] / [850, 852, 852, 852, 853] / [4021, 4063, 4015, 3994, 3957] / [1740, 1449, 1705, 1499, 1733] +18.6%!