Kernelstub Usage

What is Kernelstub?

Overview

Kernelstub is a utility to automatically manage your OS’s EFI System Partition (ESP). It makes it simple to copy the current kernel and initramfs image onto the ESP so that they are automatically probeable by most EFI boot loaders as well as the EFI firmware itself. It can also set up the system’s NVRAM to add entries to the firmware boot menu for the kernel (and keep these options up to date when new kernel versions are installed).

From the kernelstub Github page

Developed by System76, kernelstub is included with Pop!_OS to manage kernel settings and boot files with systemd-boot. The .deb file can be downloaded on Github here along with source code. kernelstub is designed to make working with systemd-boot easy.

Common Uses

The benefit of kernelstub is the ease with which kernel options and boot parameters can be managed. You can quickly add (or remove) options that are used by the Linux kernel on boot. kernelstub will make sure to only have a single instance of options added.

For example, the command below adds both “quiet” and “splash” to the kernel options:

sudo kernelstub -a "quiet splash"

A full set of available options are listed on the Github page. They can also be printed out in the Terminal by passing:

sudo kernelstub --help

OR

sudo kernelstub -h

Or by running kernelstub without any other options.

Most of the options available have short and long-form options. Output of --help produced at the time of this writing is included below.

Troubleshooting

kernelstub is also a useful troubleshooting tool, allowing quick changes to kernel options used on boot.

Verbose Output

To see more detailed information about what kernelstub is doing, add the -v or --verbose flags to your command. To save the output to a log pass the -g or --log-file flags followed by the save location:

sudo kernelstub -g ~/kernelstub.log

This command creates a log of kernelstub output and saves it in the Home folder. The name, file extension and save-location can all be chosen by the user.

“Dry Run” and Return Codes

To see if a command will do something intended or something harmful, you can run kernelstub with a “dry run” instruction added.

sudo kernelstub --dry-run

If your dry run yields errors, you can look up the error codes here and adjust accordingly.

Troubleshooting Examples

Drive Latency Changes

If a system exhibits drive, or drive power issues, the drive may not be correctly resuming from suspend. Drive latency can be adjusted using kernelstub.

Finding the latency settings from the drive controller to limit the drive from the deepest drive sleep states can be found from the following commands:

# make sure smartctl is installed
sudo apt install smartmontools
sudo smartctl -a /dev/nvme0n1

Full example output can be found below, but the information we need is shown here:

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     3.50W    2.10W       -    0  0  0  0        0       0
 1 +     2.40W    1.60W       -    0  0  0  0        0       0
 2 +     1.90W    1.50W       -    0  0  0  0        0       0
 3 -   0.0250W       -        -    3  3  3  3     3900   11000
 4 -   0.0050W       -        -    4  4  4  4     5000   39000

Since we don’t want the drive to go into it’s deepest sleep state, we need to choose the second lowest Ex_Lat, or in this case 11000.

The latency can be set with this kernelstub command:

sudo kernelstub -a "nvme_core.default_ps_max_latency_us=12000"

The important information here is the Max Watts. We want to slightly increase the power draw from the drive, without having it draw so much power that it heats up, or doesn’t fully suspend. In this case, that is the second-to-last latency setting with a Max wattage less than 1W.

Cstate Changes

Likewise, CPU performance or power issues can be mitigated by changing C-state settings.

sudo kernelstub -a "intel_idle.max_cstate=4"

The higher the cstate number, the lower the CPU power is set. cstate 0 means the CPU cores are running. cstate 1 is halt. cstate 2 is stop-clock or partial sleep, cstate 3 is sleep and cstate 4 is deep sleep. The maximum cstate number is determined by the processor model.

In our example, setting the maximum cstate to 4 means that the computer can go into deep sleep, but any additinoal power-saving measures will not be used. This can counteract bugs where deeper sleep states cause unexpected behavior, but can increase battery consumption.

Backlight Controls

Kernelstub can be used to change ACPI settings for backlight control in the event of function keys misbehaving.

sudo kernelstub -a "acpi_backlight=vendor”

This command tells the Linux kernel to defer to firmware/hardware controls provided by the vendor of the keyboard for adjusting backlight.

Set Default Boot Kernel

If you need to change the default kernel version used when the system boots, this can be done with this kernelstub command:

sudo kernelstub -v -k /boot/vmlinuz-5.8.18-63.71-care -i /boot/initrd.img-5.8.18-63.71-care

In this example, we have set the kernel to a 5.8 version compiled by the support team. Substitute the desired version. To see which kernel versions are installed and determine the exact version numbers run:

dpkg --list | grep linux-image*

If a specific kernel version is the suspected source of a bug, try a different version. Then a known-good version can be set as the default.

--help Output

user@pop-os:~$ kernelstub --help
usage: kernelstub [-h] [-c] [-p] [-e ESP,] [--esp-path ESP] [-r ROOT]
                  [--root-path ROOT] [-k PATH,] [--kernel-path PATH]
                  [-i PATH,] [--initrd-path PATH] [-o "OPTIONS",]
                  [--options "OPTIONS"] [-a "OPTIONS",]
                  [--add-options "OPTIONS"] [-d "OPTIONS"]
                  [--delete-options "OPTIONS"] [-g LOG] [--log-file LOG]
                  [-l | -n  -s | -m] [-f] [-v]

Automatic Kernel EFIstub manager

optional arguments:
  -h, --help            show this help message and exit
  -c, --dry-run         Don't perform any actions, just simulate them.
  -p, --print-config    Print the current configuration and exit
  -e ESP,
  --esp-path ESP        Manually specify the path to the ESP. Default is
                        /boot/efi
  -r ROOT
  --root-path ROOT      The path where the root filesystem to use is mounted.
  -k PATH,
  --kernel-path PATH    The path to the kernel image.
  -i PATH,
  --initrd-path PATH    The path to the initrd image.
  -o "OPTIONS",
  --options "OPTIONS"   The total boot options to be passed to the kernel
  -a "OPTIONS",
  --add-options "OPTIONS"
                        Boot options to add to the configuration (if they
                        aren't already present)
  -d "OPTIONS"
  --delete-options "OPTIONS"
                        Boot options to remove from the configuration (if
                        they're present already)
  -g LOG
  --log-file LOG        The path to the log file to use. Defaults to
                        /var/log/kernelstub.log
  -l, --loader          Creates a systemd-boot compatible loader configuration
  -n, --no-loader       Turns off creating loader configuration
  -s, --stub            Set up NVRAM entries for the copied kernel
  -m, --manage-only     Only copy entries, don't set up the NVRAM
  -f, --force-update    Forcibly update any loader.conf to set the new entry
                        as the default
  -v, --verbose         Increase program verbosity and display extra output.

smartctl Output

nathaniel@pop-os:~$ sudo smartctl -a /dev/nvme0n1
[sudo] password for nathaniel:
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.25-xanmod1] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       WDC WDS500G2B0C-00PXH0
Serial Number:                      21122J804372
Firmware Version:                   211210WD
PCI Vendor/Subsystem ID:            0x15b7
IEEE OUI Identifier:                0x001b44
Total NVM Capacity:                 500,107,862,016 [500 GB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          500,107,862,016 [500 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            001b44 8b48c4d578
Local Time is:                      Fri Mar  4 13:04:18 2022 MST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x1e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     80 Celsius
Critical Comp. Temp. Threshold:     85 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     3.50W    2.10W       -    0  0  0  0        0       0
 1 +     2.40W    1.60W       -    0  0  0  0        0       0
 2 +     1.90W    1.50W       -    0  0  0  0        0       0
 3 -   0.0250W       -        -    3  3  3  3     3900   11000
 4 -   0.0050W       -        -    4  4  4  4     5000   39000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        41 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    1%
Data Units Read:                    5,169,633 [2.64 TB]
Data Units Written:                 16,273,614 [8.33 TB]
Host Read Commands:                 44,228,094
Host Write Commands:                180,122,354
Controller Busy Time:               325
Power Cycles:                       212
Power On Hours:                     2,059
Unsafe Shutdowns:                   57
Media and Data Integrity Errors:    0
Error Information Log Entries:      1
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged