CPU clock frequency scales up and down with the CPU usage as a way to reduce power usage. On Linux, this behavior is governed by the CPU governor.
The CPU frequency also scales down as a response to high temperature, or if it's running certain instructions like AVX-512.
Linux's CPU frequency governor[edit | edit source]
You can check your current CPU governor by running:
# cpupower frequency-info --governors analyzing CPU 0: available cpufreq governors: performance powersave
Alternatively, you may check
/sys/devices/system/cpu and see the setting for each CPU core.
# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor performance performance performance performance performance performance
Types of scaling governors[edit | edit source]
Scaling governors are policies that affect the CPU frequency scaling algorithm.
|Governor||A very brief description|
|performance||Run the CPU at the highest frequency, as defined within the scaling_max_freq limit.|
|powersave||Run the CPU at the lowest frequency, as defined within the scaling_min_freq limit.|
|ondemand||Dynamically sets frequency proportional to the load of the CPUs.|
|conservative||Like ondemand, but frequency changes happen in smaller steps.|
|userspace||Run the CPU at user specified frequency, controlled by writing to the |
Intel Turbo Boost frequency[edit | edit source]
Some Intel CPUs support a turbo boost frequency where a CPU can clock faster than its stock speed depending on available power and thermal conditions. Refer to the CPU's specification to determine a particular CPU's turbo boost frequency.
I have observed on some compute servers that the CPU will run somewhere between the base frequency and the turbo boost frequency when the temperature is around the mid 70's Celsius when under full load. I'm not sure why this is the case.
Some helpful commands[edit | edit source]
|Get all CPU core frequencies|
|Get average CPU core frequency|
|Get max CPU core frequency|
|Get min CPU core frequency|
|Get average CPU temperature|
|Get max CPU temperature|
Issues[edit | edit source]
My CPU is stuck at a very low clock speed[edit | edit source]
I've had servers stuck at around 100MHz which is well below the minimum clock speed of the CPU. I wrote about it on Troubleshooting a Slow Linux System as well.
Some tips if this happens to you are:
- Do a BIOS update. A bug with the BIOS could make the CPU think there isn't enough power, for example.
- Check your power supplies. A failing PSU could cause the CPU to throttle.
- Check CPU temperature. A blocked inlet, or not enough airflow could possibly be an issue.
- Try to powercycle the server. If it uses a SMASH-CLP DRAC, try to
/system1/pwrmanagement1target (this seemed to help on a problematic server)
My CPU clocks down under load[edit | edit source]
If your CPU clocks down while under load, check if it is:
- a thermal issue
- a power issue
- due to slow SIMD instructions like AVX2 or AVX-512, which will cause the core to clock down. See: https://stackoverflow.com/questions/56852812/simd-instructions-lowering-cpu-frequency/56861355#56861355
CPU clocking down to the minimum clock speed[edit | edit source]
At RCS, the servers are configured to be using the performance CPU governor in order to eek as much performance as we can from the hardware. Because of the governor setting, we expect the CPU frequency to sit somewhere between the base CPU frequency and the turbo boost frequency. However, I have observed some idle nodes at 1GHz, the minimum limit for these processors.
The minimum clock speed can be found with
cpupower. A properly working node shows the following output, including the governor in use.
# cpupower frequency-info analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: Cannot determine or is not supported. hardware limits: 1000 MHz - 3.70 GHz available cpufreq governors: performance powersave current policy: frequency should be within 1000 MHz and 3.70 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency: Unable to call hardware current CPU frequency: 3.10 GHz (asserted by call to kernel) boost state support: Supported: yes Active: yes
On nodes that clock to the minimum 1GHz,
cpupower does not show the governor:
# cpupower frequency-info analyzing CPU 0: no or unknown cpufreq driver is active on this CPU CPUs which run at the same hardware frequency: Not Available CPUs which need to have their frequency coordinated by software: Not Available maximum transition latency: Cannot determine or is not supported. Not Available available cpufreq governors: Not Available Unable to determine current policy current CPU frequency: Unable to call hardware current CPU frequency: Unable to call to kernel boost state support: Supported: yes Active: yes # grep MHz /proc/cpuinfo cpu MHz : 999.870 cpu MHz : 999.851 cpu MHz : 999.849 cpu MHz : 999.870 cpu MHz : 1000.130 cpu MHz : 999.914 cpu MHz : 999.870 cpu MHz : 999.926 cpu MHz : 1000.030 cpu MHz : 999.845 cpu MHz : 999.967 cpu MHz : 999.921 ...