michaelkuc6/blog_os
1
0
Fork 0
mirror of https://github.com/phil-opp/blog_os.git synced 2025-12-17 06:47:49 +00:00
Code Issues Projects Releases Wiki Activity

Format edition-3 markdown as 'one sentence per line'

Browse Source 
This makes the diffs nicer when we change something in the future.
This commit is contained in:
Philipp Oppermann
2022-12-18 12:09:00 +01:00
parent c40ac1b1d7
commit ce0c6c133b
3 changed files with 782 additions and 265 deletions
Download Patch File Download Diff File Expand all files Collapse all files

382
blog/content/edition-3/posts/02-booting/index.md
View File

@@ -17,14 +17,19 @@ icon = '''
extra_content = ["uefi/index.md"]
+++
In this post, we explore the boot process on both BIOS and UEFI-based systems. We combine the [minimal kernel] created in the previous post with a bootloader to create a bootable disk image. We then show how this image can be started in the [QEMU] emulator and run on real hardware.
In this post, we explore the boot process on both BIOS and UEFI-based systems.
We combine the [minimal kernel] created in the previous post with a bootloader to create a bootable disk image.
We then show how this image can be started in the [QEMU] emulator and run on real hardware.
[minimal kernel]: @/edition-3/posts/01-minimal-kernel/index.md
[QEMU]: https://www.qemu.org/
<!-- more -->
This blog is openly developed on [GitHub]. If you have any problems or questions, please open an issue there. You can also leave comments [at the bottom]. The complete source code for this post can be found in the [`post-02`][post branch] branch.
This blog is openly developed on [GitHub].
If you have any problems or questions, please open an issue there.
You can also leave comments [at the bottom].
The complete source code for this post can be found in the [`post-02`][post branch] branch.
[GitHub]: https://github.com/phil-opp/blog_os
[at the bottom]: #comments
@@ -34,47 +39,66 @@ This blog is openly developed on [GitHub]. If you have any problems or questions
<!-- toc -->
## The Boot Process
When you turn on a computer, it begins executing firmware code that is stored in motherboard [ROM]. This code performs a [power-on self-test], detects available RAM, and pre-initializes the CPU and other hardware. Afterwards it looks for a bootable disk and starts booting the operating system kernel.
When you turn on a computer, it begins executing firmware code that is stored in motherboard [ROM].
This code performs a [power-on self-test], detects available RAM, and pre-initializes the CPU and other hardware.
Afterwards it looks for a bootable disk and starts booting the operating system kernel.
[ROM]: https://en.wikipedia.org/wiki/Read-only_memory
[power-on self-test]: https://en.wikipedia.org/wiki/Power-on_self-test
On x86, there are two firmware standards: the “Basic Input/Output System“ (**[BIOS]**) and the newer “Unified Extensible Firmware Interface” (**[UEFI]**). The BIOS standard is old and outdated, but simple and well-supported on any x86 machine since the 1980s. UEFI, in contrast, is more modern and has much more features, but also more complex.
On x86, there are two firmware standards: the “Basic Input/Output System“ (**[BIOS]**) and the newer “Unified Extensible Firmware Interface” (**[UEFI]**).
The BIOS standard is old and outdated, but simple and well-supported on any x86 machine since the 1980s.
UEFI, in contrast, is more modern and has much more features, but also more complex.
[BIOS]: https://en.wikipedia.org/wiki/BIOS
[UEFI]: https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface
### BIOS
Almost all x86 systems have support for BIOS booting, including most UEFI-based machines that support an emulated BIOS. This is great, because you can use the same boot logic across all machines from the last centuries. The drawback is that the standard is very old, for example the CPU is put into a 16-bit compatibility mode called [real mode] before booting so that archaic bootloaders from the 1980s would still work. Also, BIOS-compatibility will be slowly removed on newer UEFI machines over the next years (see below).
Almost all x86 systems have support for BIOS booting, including most UEFI-based machines that support an emulated BIOS.
This is great, because you can use the same boot logic across all machines from the last centuries.
The drawback is that the standard is very old, for example the CPU is put into a 16-bit compatibility mode called [real mode] before booting so that archaic bootloaders from the 1980s would still work.
Also, BIOS-compatibility will be slowly removed on newer UEFI machines over the next years (see below).
#### Boot Process
When you turn on a BIOS-based computer, it first loads the BIOS firmware from some special flash memory located on the motherboard. The BIOS runs self test and initialization routines of the hardware, then it looks for bootable disks. For that it loads the first disk sector (512 bytes) of each disk into memory, which contains the [_master boot record_] (MBR) structure. This structure has the following general format:
When you turn on a BIOS-based computer, it first loads the BIOS firmware from some special flash memory located on the motherboard.
The BIOS runs self test and initialization routines of the hardware, then it looks for bootable disks.
For that it loads the first disk sector (512 bytes) of each disk into memory, which contains the [_master boot record_] (MBR) structure.
This structure has the following general format:
[_master boot record_]: https://en.wikipedia.org/wiki/Master_boot_record
Offset | Field | Size
-------|-------|-----
0 | bootstrap code | 446
446 | partition entry 1 | 16
462 | partition entry 2 | 16
478 | partition entry 3 | 16
444 | partition entry 4 | 16
510 | boot signature | 2
| Offset | Field | Size |
| ------ | ----------------- | ---- |
| 0 | bootstrap code | 446 |
| 446 | partition entry 1 | 16 |
| 462 | partition entry 2 | 16 |
| 478 | partition entry 3 | 16 |
| 444 | partition entry 4 | 16 |
| 510 | boot signature | 2 |
The bootstrap code is commonly called the _bootloader_ and responsible for loading and starting the operating system kernel. The four partition entries describe the [disk partitions] such as the `C:` partition on Windows. The boot signature field at the end of the structure specifies whether this disk is bootable or not. If it is bootable, the signature field must be set to the [magic bytes] `0xaa55`. It's worth noting that there are [many extensions][mbr-extensions] of the MBR format, which for example include a 5th partition entry or a disk signature.
The bootstrap code is commonly called the _bootloader_ and responsible for loading and starting the operating system kernel.
The four partition entries describe the [disk partitions] such as the `C:` partition on Windows.
The boot signature field at the end of the structure specifies whether this disk is bootable or not.
If it is bootable, the signature field must be set to the [magic bytes] `0xaa55`.
It's worth noting that there are [many extensions][mbr-extensions] of the MBR format, which for example include a 5th partition entry or a disk signature.
[disk partitions]: https://en.wikipedia.org/wiki/Disk_partitioning
[magic bytes]: https://en.wikipedia.org/wiki/Magic_number_(programming)
[mbr-extensions]: https://en.wikipedia.org/wiki/Master_boot_record#Sector_layout
The BIOS itself only cares for the boot signature field. If it finds a disk with a boot signature equal to `0xaa55`, it directly passes control to the bootloader code stored at the beginning of the disk. This bootloader is then responsible for multiple things:
The BIOS itself only cares for the boot signature field.
If it finds a disk with a boot signature equal to `0xaa55`, it directly passes control to the bootloader code stored at the beginning of the disk.
This bootloader is then responsible for multiple things:
- **Loading the kernel from disk:** The bootloader has to determine the location of the kernel image on the disk and load it into memory.
- **Initializing the CPU:** As noted above, all `x86_64` CPUs start up in a 16-bit [real mode] to be compatible with older operating systems. So in order to run current 64-bit operating systems, the bootloader needsn to switch the CPU from the 16-bit [real mode] first to the 32-bit [protected mode], and then to the 64-bit [long mode], where all CPU registers and the complete main memory are available.
- **Querying system information:** The third job of the bootloader is to query certain information from the BIOS and pass it to the OS kernel. This, for example, includes information about the available main memory and graphical output devices.
- **Setting up an execution environment:** Kernels are typically stored as normal executable files (e.g. in the [ELF] or [PE] format), which require some loading procedure. This includes setting up a [call stack] and a [page table].
- **Initializing the CPU:** As noted above, all `x86_64` CPUs start up in a 16-bit [real mode] to be compatible with older operating systems.
So in order to run current 64-bit operating systems, the bootloader needsn to switch the CPU from the 16-bit [real mode] first to the 32-bit [protected mode], and then to the 64-bit [long mode], where all CPU registers and the complete main memory are available.
- **Querying system information:** The third job of the bootloader is to query certain information from the BIOS and pass it to the OS kernel.
This, for example, includes information about the available main memory and graphical output devices.
- **Setting up an execution environment:** Kernels are typically stored as normal executable files (e.g. in the [ELF] or [PE] format), which require some loading procedure.
This includes setting up a [call stack] and a [page table].
[ELF]: https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
[PE]: https://en.wikipedia.org/wiki/Portable_Executable
@@ -85,43 +109,60 @@ The BIOS itself only cares for the boot signature field. If it finds a disk with
[memory segmentation]: https://en.wikipedia.org/wiki/X86_memory_segmentation
[page table]: https://en.wikipedia.org/wiki/Page_table
Some bootloaders also include a basic user interface for [choosing between multiple installed OSs][multi-booting] or entering a recovery mode. Since it is not possible to do all that within the available 446 bytes, most bootloaders are split into a small first stage, which is as small as possible, and a second stage, which is subsequently loaded by the first stage.
Some bootloaders also include a basic user interface for [choosing between multiple installed OSs][multi-booting] or entering a recovery mode.
Since it is not possible to do all that within the available 446 bytes, most bootloaders are split into a small first stage, which is as small as possible, and a second stage, which is subsequently loaded by the first stage.
[multi-booting]: https://en.wikipedia.org/wiki/Multi-booting
Writing a BIOS bootloader is cumbersome as it requires assembly language and a lot of non insightful steps like _“write this magic value to this processor register”_. Therefore we don't cover bootloader creation in this post and instead use the existing [`bootloader`] crate to make our kernel bootable. If you are interested in building your own BIOS bootloader: Stay tuned, a set of posts on this topic is already planned! <!-- , check out our “_[Writing a Bootloader]_” posts, where we explain in detail how a bootloader is built. -->
Writing a BIOS bootloader is cumbersome as it requires assembly language and a lot of non insightful steps like _“write this magic value to this processor register”_.
Therefore we don't cover bootloader creation in this post and instead use the existing [`bootloader`] crate to make our kernel bootable.
If you are interested in building your own BIOS bootloader: Stay tuned, a set of posts on this topic is already planned! <!-- , check out our “_[Writing a Bootloader]_” posts, where we explain in detail how a bootloader is built.
-->
#### The Future of BIOS
As noted above, most modern systems still support booting operating systems written for the legacy BIOS firmware for backwards-compatibility. However, there are [plans to remove this support soon][end-bios-support]. Thus, it is strongly recommended to make operating system kernels compatible with the newer UEFI standard too. Fortunately, it is possible to create a kernel that supports booting on both BIOS (for older systems) and UEFI (for modern systems).
As noted above, most modern systems still support booting operating systems written for the legacy BIOS firmware for backwards-compatibility.
However, there are [plans to remove this support soon][end-bios-support].
Thus, it is strongly recommended to make operating system kernels compatible with the newer UEFI standard too.
Fortunately, it is possible to create a kernel that supports booting on both BIOS (for older systems) and UEFI (for modern systems).
[end-bios-support]: https://arstechnica.com/gadgets/2017/11/intel-to-kill-off-the-last-vestiges-of-the-ancient-pc-bios-by-2020/
### UEFI
The Unified Extensible Firmware Interface (UEFI) replaces the classical BIOS firmware on most modern computers. The specification provides lots of useful features that make bootloader implementations much simpler:
The Unified Extensible Firmware Interface (UEFI) replaces the classical BIOS firmware on most modern computers.
The specification provides lots of useful features that make bootloader implementations much simpler:
- It supports initializing the CPU directly into 64-bit mode, instead of starting in a DOS-compatible 16-bit mode like the BIOS firmware.
- It understands disk partitions and executable files. Thus it is able to fully load the bootloader from disk into memory (no 512-byte "first stage" is required anymore).
- A standardized [specification][uefi-specification] minimizes the differences between systems. This isn't the case for the legacy BIOS firmware, so that bootloaders often have to try different methods because of hardware differences.
- It understands disk partitions and executable files.
Thus it is able to fully load the bootloader from disk into memory (no 512-byte "first stage" is required anymore).
- A standardized [specification][uefi-specification] minimizes the differences between systems.
This isn't the case for the legacy BIOS firmware, so that bootloaders often have to try different methods because of hardware differences.
- The specification is independent of the CPU architecture, so that the same interface can be used to boot on `x86_64` and e.g. `ARM` CPUs.
- It natively supports network booting without requiring additional drivers.
[uefi-specification]: https://uefi.org/specifications
The UEFI standard also tries to make the boot process safer through a so-called _"secure boot"_ mechanism. The idea is that the firmware only allows loading bootloaders that are signed by a trusted [digital signature]. Thus, malware should be prevented from compromising the early boot process.
The UEFI standard also tries to make the boot process safer through a so-called _"secure boot"_ mechanism.
The idea is that the firmware only allows loading bootloaders that are signed by a trusted [digital signature].
Thus, malware should be prevented from compromising the early boot process.
[digital signature]: https://en.wikipedia.org/wiki/Digital_signature
#### Issues & Criticism
While most of the UEFI specification sounds like a good idea, there are also many issues with the standard. The main issue for most people is the fear that the _secure boot_ mechanism can be used to [lock users into the Windows operating system][uefi-secure-boot-lock-in] and thus prevent the installation of alternative operating systems such as Linux.
While most of the UEFI specification sounds like a good idea, there are also many issues with the standard.
The main issue for most people is the fear that the _secure boot_ mechanism can be used to [lock users into the Windows operating system][uefi-secure-boot-lock-in] and thus prevent the installation of alternative operating systems such as Linux.
[uefi-secure-boot-lock-in]: https://arstechnica.com/information-technology/2015/03/windows-10-to-make-the-secure-boot-alt-os-lock-out-a-reality/
Another point of criticism is that the large number of features make the UEFI firmware very complex, which increases the chance that there are some bugs in the firmware implementation itself. This can lead to security problems because the firmware has complete control over the hardware. For example, a vulnerability in the built-in network stack of an UEFI implementation can allow attackers to compromise the system and e.g. silently observe all I/O data. The fact that most UEFI implementations are not open-source makes this issue even more problematic, since there is no way to audit the firmware code for potential bugs.
Another point of criticism is that the large number of features make the UEFI firmware very complex, which increases the chance that there are some bugs in the firmware implementation itself.
This can lead to security problems because the firmware has complete control over the hardware.
For example, a vulnerability in the built-in network stack of an UEFI implementation can allow attackers to compromise the system and e.g. silently observe all I/O data.
The fact that most UEFI implementations are not open-source makes this issue even more problematic, since there is no way to audit the firmware code for potential bugs.
While there are open firmware projects such as [coreboot] that try to solve these problems, there is no way around the UEFI standard on most modern consumer computers. So we have to live with these drawbacks for now if we want to build a widely compatible bootloader and operating system kernel.
While there are open firmware projects such as [coreboot] that try to solve these problems, there is no way around the UEFI standard on most modern consumer computers.
So we have to live with these drawbacks for now if we want to build a widely compatible bootloader and operating system kernel.
[coreboot]: https://www.coreboot.org/
@@ -129,8 +170,12 @@ While there are open firmware projects such as [coreboot] that try to solve thes
The UEFI boot process works in the following way:
- After powering on and self-testing all components, the UEFI firmware starts looking for special bootable disk partitions called [EFI system partitions]. These partitions must be formatted with the [FAT file system] and assigned a special ID that indicates them as EFI system partition. The UEFI standard understands both the [MBR] and [GPT] partition table formats for this, at least theoretically. In practice, some UEFI implementations seem to [directly switch to BIOS-style booting when an MBR partition table is used][mbr-csm], so it is recommended to only use the GPT format with UEFI.
- If the firmware finds a EFI system partition, it looks for an executable file named `efi\boot\bootx64.efi` (on x86_64 systems) in it. This executable must use the [Portable Executable (PE)] format, which is common in the Windows world.
- After powering on and self-testing all components, the UEFI firmware starts looking for special bootable disk partitions called [EFI system partitions].
These partitions must be formatted with the [FAT file system] and assigned a special ID that indicates them as EFI system partition.
The UEFI standard understands both the [MBR] and [GPT] partition table formats for this, at least theoretically.
In practice, some UEFI implementations seem to [directly switch to BIOS-style booting when an MBR partition table is used][mbr-csm], so it is recommended to only use the GPT format with UEFI.
- If the firmware finds a EFI system partition, it looks for an executable file named `efi\boot\bootx64.efi` (on x86_64 systems) in it.
This executable must use the [Portable Executable (PE)] format, which is common in the Windows world.
- It then loads the executable from disk to memory, sets up the execution environment (CPU state, page tables, etc.) in a standardized way, and finally jumps to the entry point of the loaded executable.
[MBR]: https://en.wikipedia.org/wiki/Master_boot_record
@@ -140,50 +185,73 @@ The UEFI boot process works in the following way:
[FAT file system]: https://en.wikipedia.org/wiki/File_Allocation_Table
[Portable Executable (PE)]: https://en.wikipedia.org/wiki/Portable_Executable
From this point on, the loaded executable has control. Typically, this executable is a bootloader that then loads the actual operating system kernel. Theoretically, it would also be possible to let the UEFI firmware load the kernel directly without a bootloader in between, but this would make it more difficult to port the kernel to other architectures.
From this point on, the loaded executable has control.
Typically, this executable is a bootloader that then loads the actual operating system kernel.
Theoretically, it would also be possible to let the UEFI firmware load the kernel directly without a bootloader in between, but this would make it more difficult to port the kernel to other architectures.
Bootloaders and kernels typically need additional information about the system, for example the amount of available memory. For this reason, the UEFI firmware passes a pointer to a special _system table_ as an argument when invoking the bootloader entry point function. Using this table, the bootloader can query various system information and even invoke special functions provided by the UEFI firmware, for example for accessing the hard disk.
Bootloaders and kernels typically need additional information about the system, for example the amount of available memory.
For this reason, the UEFI firmware passes a pointer to a special _system table_ as an argument when invoking the bootloader entry point function.
Using this table, the bootloader can query various system information and even invoke special functions provided by the UEFI firmware, for example for accessing the hard disk.
#### How we will use UEFI
As it is probably clear at this point, the UEFI interface is very powerful and complex. The wide range of functionality makes it even possible to write an operating system directly as an UEFI application, using the UEFI services provided by the system table instead of creating own drivers. In practice, however, most operating systems use UEFI only for the bootloader since own drivers give you better performance and more control over the system. We will also follow this path for our OS implementation.
As it is probably clear at this point, the UEFI interface is very powerful and complex.
The wide range of functionality makes it even possible to write an operating system directly as an UEFI application, using the UEFI services provided by the system table instead of creating own drivers.
In practice, however, most operating systems use UEFI only for the bootloader since own drivers give you better performance and more control over the system.
We will also follow this path for our OS implementation.
To keep this post focused, we won't cover the creation of an UEFI bootloader here. Instead, we will use the already mentioned [`bootloader`] crate, which allows loading our kernel on both UEFI and BIOS systems. If you're interested in how to create an UEFI bootloader yourself, check out our extra post about [**UEFI Booting**].
To keep this post focused, we won't cover the creation of an UEFI bootloader here.
Instead, we will use the already mentioned [`bootloader`] crate, which allows loading our kernel on both UEFI and BIOS systems.
If you're interested in how to create an UEFI bootloader yourself, check out our extra post about [**UEFI Booting**].
[**UEFI Booting**]: @/edition-3/posts/02-booting/uefi/index.md
### The Multiboot Standard
To avoid that every operating system implements its own bootloader that is only compatible with a single OS, the [Free Software Foundation] created an open bootloader standard called [Multiboot] in 1995. The standard defines an interface between the bootloader and operating system, so that any Multiboot compliant bootloader can load any Multiboot compliant operating system on both BIOS and UEFI systems. The reference implementation is [GNU GRUB], which is the most popular bootloader for Linux systems.
To avoid that every operating system implements its own bootloader that is only compatible with a single OS, the [Free Software Foundation] created an open bootloader standard called [Multiboot] in 1995.
The standard defines an interface between the bootloader and operating system, so that any Multiboot compliant bootloader can load any Multiboot compliant operating system on both BIOS and UEFI systems.
The reference implementation is [GNU GRUB], which is the most popular bootloader for Linux systems.
[Free Software Foundation]: https://en.wikipedia.org/wiki/Free_Software_Foundation
[Multiboot]: https://www.gnu.org/software/grub/manual/multiboot2/multiboot.html
[GNU GRUB]: https://en.wikipedia.org/wiki/GNU_GRUB
To make a kernel Multiboot compliant, one just needs to insert a so-called [Multiboot header] at the beginning of the kernel file. This makes it very easy to boot an OS in GRUB. However, GRUB and the Multiboot standard have some problems too:
To make a kernel Multiboot compliant, one just needs to insert a so-called [Multiboot header] at the beginning of the kernel file.
This makes it very easy to boot an OS in GRUB.
However, GRUB and the Multiboot standard have some problems too:
[Multiboot header]: https://www.gnu.org/software/grub/manual/multiboot/multiboot.html#OS-image-format
- The standard is designed to make the bootloader simple instead of the kernel. For example, the kernel needs to be linked with an [adjusted default page size], because GRUB can't find the Multiboot header otherwise. Another example is that the [boot information], which is passed to the kernel, contains lots of architecture dependent structures instead of providing clean abstractions.
- The standard supports only the 32-bit protected mode on BIOS systems. This means that you still have to do the CPU configuration to switch to the 64-bit long mode.
- The standard is designed to make the bootloader simple instead of the kernel.
For example, the kernel needs to be linked with an [adjusted default page size], because GRUB can't find the Multiboot header otherwise.
Another example is that the [boot information], which is passed to the kernel, contains lots of architecture dependent structures instead of providing clean abstractions.
- The standard supports only the 32-bit protected mode on BIOS systems.
This means that you still have to do the CPU configuration to switch to the 64-bit long mode.
- For UEFI systems, the standard provides very little added value as it simply exposes the normal UEFI interface to kernels.
- Both GRUB and the Multiboot standard are only sparsely documented.
- GRUB needs to be installed on the host system to create a bootable disk image from the kernel file. This makes development on Windows or Mac more difficult.
- GRUB needs to be installed on the host system to create a bootable disk image from the kernel file.
This makes development on Windows or Mac more difficult.
[adjusted default page size]: https://wiki.osdev.org/Multiboot#Multiboot_2
[boot information]: https://www.gnu.org/software/grub/manual/multiboot/multiboot.html#Boot-information-format
Because of these drawbacks we decided to not use GRUB or the Multiboot standard for this series. However, we plan to add Multiboot support to our [`bootloader`] crate, so that it becomes possible to load your kernel on a GRUB system too. If you're interested in writing a Multiboot compliant kernel, check out the [first edition] of this blog series.
Because of these drawbacks we decided to not use GRUB or the Multiboot standard for this series.
However, we plan to add Multiboot support to our [`bootloader`] crate, so that it becomes possible to load your kernel on a GRUB system too.
If you're interested in writing a Multiboot compliant kernel, check out the [first edition] of this blog series.
[first edition]: @/edition-1/_index.md
## Bootable Disk Image
We now know that most operating system kernels are loaded by bootloaders, which are small programs that initialize the hardware to reasonable defaults, load the kernel from disk, and provide it with some fundamental information about the underlying system. In this section, we will learn how to combine the [minimal kernel] we created in the previous post with the `bootloader` crate in order to create a bootable disk image.
We now know that most operating system kernels are loaded by bootloaders, which are small programs that initialize the hardware to reasonable defaults, load the kernel from disk, and provide it with some fundamental information about the underlying system.
In this section, we will learn how to combine the [minimal kernel] we created in the previous post with the `bootloader` crate in order to create a bootable disk image.
### The `bootloader` Crate
Since bootloaders quite complex on their own, we won't create our own bootloader here (but we are planning a separate series of posts on this). Instead, we will boot our kernel using the [`bootloader`] crate. This crate supports both BIOS and UEFI booting, provides all the necessary system information we need, and creates a reasonable default execution environment for our kernel. This way, we can focus on the actual kernel design in the following posts instead of spending a lot of time on system initialization.
Since bootloaders quite complex on their own, we won't create our own bootloader here (but we are planning a separate series of posts on this).
Instead, we will boot our kernel using the [`bootloader`] crate.
This crate supports both BIOS and UEFI booting, provides all the necessary system information we need, and creates a reasonable default execution environment for our kernel.
This way, we can focus on the actual kernel design in the following posts instead of spending a lot of time on system initialization.
[`bootloader`]: https://crates.io/crates/bootloader
@@ -196,45 +264,60 @@ To use the `bootloader` crate, we first need to add a dependency on it:
bootloader = "0.10.1"
```
For normal Rust crates, this step would be all that's needed for adding them as a dependency. However, the `bootloader` crate is a bit special. The problem is that it needs access to our kernel _after compilation_ in order to create a bootable disk image. However, cargo has no support for automatically running code after a successful build, so we need some manual build code for this. (There is a proposal for [post-build scripts] that would solve this issue, but it is not clear yet whether the Cargo team wants to add such a feature.)
For normal Rust crates, this step would be all that's needed for adding them as a dependency.
However, the `bootloader` crate is a bit special.
The problem is that it needs access to our kernel _after compilation_ in order to create a bootable disk image.
However, cargo has no support for automatically running code after a successful build, so we need some manual build code for this.
(There is a proposal for [post-build scripts] that would solve this issue, but it is not clear yet whether the Cargo team wants to add such a feature.)
[post-build scripts]: https://github.com/rust-lang/cargo/issues/545
#### Receiving the Boot Information
Before we look into the bootable disk image creation, we update need to update our `_start` entry point to be compatible with the `bootloader` crate. As we already mentioned above, bootloaders commonly pass additional system information when invoking the kernel, such as the amount of available memory. The `bootloader` crate also follows this convention, so we need to update our `_start` entry point to expect an additional argument.
Before we look into the bootable disk image creation, we update need to update our `_start` entry point to be compatible with the `bootloader` crate.
As we already mentioned above, bootloaders commonly pass additional system information when invoking the kernel, such as the amount of available memory.
The `bootloader` crate also follows this convention, so we need to update our `_start` entry point to expect an additional argument.
The [`bootloader` documentation][`BootInfo`] specifies that a kernel entry point should have the following signature:
[`BootInfo`]: https://docs.rs/bootloader/0.10.1/bootloader/boot_info/struct.BootInfo.html
```rust
extern "C" fn(boot_info: &'static mut bootloader::BootInfo) -> ! { ... }
extern "C" fn(boot_info: &'static mut bootloader::BootInfo) -> ! { ...
}
```
The only difference to our `_start` entry point is the additional `boot_info` argument, which is passed by the `bootloader` crate. This argument is a mutable reference to a [`bootloader::BootInfo`] type, which provides various information about the system.
The only difference to our `_start` entry point is the additional `boot_info` argument, which is passed by the `bootloader` crate.
This argument is a mutable reference to a [`bootloader::BootInfo`] type, which provides various information about the system.
[`bootloader::BootInfo`]: https://docs.rs/bootloader/0.10.1/bootloader/boot_info/struct.BootInfo.html
<div class="note"><details>
<summary><h5>About <code>extern "C"</code> and <code>!</code></h5></summary>
The [`extern "C"`] qualifier specifies that the function should use the same [ABI] and [calling convention] as C code. It is common to use this qualifier when communicating across different executables because C has a stable ABI that is guaranteed to never change. Normal Rust functions, on the other hand, don't have a stable ABI, so they might change it the future (e.g. to optimize performance) and thus shouldn't be used across different executables.
The [`extern "C"`] qualifier specifies that the function should use the same [ABI] and [calling convention] as C code.
It is common to use this qualifier when communicating across different executables because C has a stable ABI that is guaranteed to never change.
Normal Rust functions, on the other hand, don't have a stable ABI, so they might change it the future (e.g. to optimize performance) and thus shouldn't be used across different executables.
[`extern "C"`]: https://doc.rust-lang.org/reference/items/functions.html#extern-function-qualifier
[ABI]: https://en.wikipedia.org/wiki/Application_binary_interface
[calling convention]: https://en.wikipedia.org/wiki/Calling_convention
The `!` return type indicates that the function is [diverging], which means that it must never return. The `bootloader` requires this because its code might no longer be valid after the kernel modified the system state such as the [page tables].
The `!` return type indicates that the function is [diverging], which means that it must never return.
The `bootloader` requires this because its code might no longer be valid after the kernel modified the system state such as the [page tables].
[diverging]: https://doc.rust-lang.org/rust-by-example/fn/diverging.html
[page tables]: @/edition-2/posts/08-paging-introduction/index.md
</details></div>
While we could simply add the additional argument to our `_start` function, it would result in very fragile code. The problem is that because the `_start` function is called externally from the bootloader, no checking of the function signature occurs. So no compilation error occurs, even if the function signature completely changed after updating to a newer `bootloader` version. At runtime, however, the code would fail or introduce undefined behavior.
While we could simply add the additional argument to our `_start` function, it would result in very fragile code.
The problem is that because the `_start` function is called externally from the bootloader, no checking of the function signature occurs.
So no compilation error occurs, even if the function signature completely changed after updating to a newer `bootloader` version.
At runtime, however, the code would fail or introduce undefined behavior.
To avoid these issues and make sure that the entry point function has always the correct signature, the `bootloader` crate provides an [`entry_point`] macro that provides a type-checked way to define a Rust function as the entry point. This way, the function signature is checked at compile time so that no runtime error can occur.
To avoid these issues and make sure that the entry point function has always the correct signature, the `bootloader` crate provides an [`entry_point`] macro that provides a type-checked way to define a Rust function as the entry point.
This way, the function signature is checked at compile time so that no runtime error can occur.
[`entry_point`]: https://docs.rs/bootloader/0.6.4/bootloader/macro.entry_point.html
@@ -252,25 +335,34 @@ fn kernel_main(boot_info: &'static mut BootInfo) -> ! {
}
```
We no longer need to use `extern "C"` or `no_mangle` for our entry point, as the macro defines the actual lower-level `_start` entry point for us. The `kernel_main` function is now a completely normal Rust function, so we can choose an arbitrary name for it. Since the signature of the function is enforced by the macro, a compilation error occurs when it e.g. has the wrong argument type.
We no longer need to use `extern "C"` or `no_mangle` for our entry point, as the macro defines the actual lower-level `_start` entry point for us.
The `kernel_main` function is now a completely normal Rust function, so we can choose an arbitrary name for it.
Since the signature of the function is enforced by the macro, a compilation error occurs when it e.g. has the wrong argument type.
After adjusting our entry point for the `bootloader` crate, we can now look into how to create a bootable disk image from our kernel.
### Creating a Disk Image
The [docs of the `bootloader` crate][`bootloader` docs] describes how to create a bootable disk image for a kernel. The first step is to find the directory where cargo placed the source code of the `bootloader` dependency. Then, a special build command needs to be executed in that directory, passing the paths to the kernel binary and its `Cargo.toml` as arguments. This will result in multiple disk image files as output, which can be used to boot the kernel on BIOS and UEFI systems.
The [docs of the `bootloader` crate][`bootloader` docs] describes how to create a bootable disk image for a kernel.
The first step is to find the directory where cargo placed the source code of the `bootloader` dependency.
Then, a special build command needs to be executed in that directory, passing the paths to the kernel binary and its `Cargo.toml` as arguments.
This will result in multiple disk image files as output, which can be used to boot the kernel on BIOS and UEFI systems.
[`bootloader` docs]: https://docs.rs/bootloader/0.10.1/bootloader/
#### A `boot` crate
Since following these steps manually is cumbersome, we create a script to automate it. For that we create a new `boot` crate in a subdirectory, in which we will implement the build steps:
Since following these steps manually is cumbersome, we create a script to automate it.
For that we create a new `boot` crate in a subdirectory, in which we will implement the build steps:
```
cargo new --bin boot
```
This command creates a new `boot` subfolder with a `Cargo.toml` and a `src/main.rs` in it. Since this new cargo project will be tightly coupled with our main project, it makes sense to combine the two crates as a [cargo workspace]. This way, they will share the same `Cargo.lock` for their dependencies and place their compilation artifacts in a common `target` folder. To create such a workspace, we add the following to the `Cargo.toml` of our main project:
This command creates a new `boot` subfolder with a `Cargo.toml` and a `src/main.rs` in it.
Since this new cargo project will be tightly coupled with our main project, it makes sense to combine the two crates as a [cargo workspace].
This way, they will share the same `Cargo.lock` for their dependencies and place their compilation artifacts in a common `target` folder.
To create such a workspace, we add the following to the `Cargo.toml` of our main project:
[cargo workspace]: https://doc.rust-lang.org/cargo/reference/workspaces.html
@@ -281,18 +373,24 @@ This command creates a new `boot` subfolder with a `Cargo.toml` and a `src/main.
members = ["boot"]
```
After creating the workspace, we can begin the implementation of the `boot` crate. Note that the crate will be invoked as part as our build process, so it can be a normal Rust executable that runs on our host system. This means that is has a classical `main` function and can use standard library types such as [`Path`] or [`Command`] without problems.
After creating the workspace, we can begin the implementation of the `boot` crate.
Note that the crate will be invoked as part as our build process, so it can be a normal Rust executable that runs on our host system.
This means that is has a classical `main` function and can use standard library types such as [`Path`] or [`Command`] without problems.
[`Path`]: https://doc.rust-lang.org/std/path/struct.Path.html
[`Command`]: https://doc.rust-lang.org/std/process/struct.Command.html
#### Locating the `bootloader` Source
The first step in creating the bootable disk image is to to locate where cargo put the source code of the `bootloader` dependency. For that we can use cargo's [`cargo metadata`] subcommand, which outputs all kinds of information about a cargo project as a JSON object. Among other things, it contains the manifest path (i.e. the path to the `Cargo.toml`) of all dependencies, including the `bootloader` crate.
The first step in creating the bootable disk image is to to locate where cargo put the source code of the `bootloader` dependency.
For that we can use cargo's [`cargo metadata`] subcommand, which outputs all kinds of information about a cargo project as a JSON object.
Among other things, it contains the manifest path (i.e. the path to the `Cargo.toml`) of all dependencies, including the `bootloader` crate.
[`cargo metadata`]: https://doc.rust-lang.org/cargo/commands/cargo-metadata.html
To keep this post short, we won't include the code to parse the JSON output and to locate the right entry here. Instead, we created a small crate named [`bootloader-locator`] that wraps the needed functionality in a simple [`locate_bootloader`] function. Let's add that crate as a dependency and use it:
To keep this post short, we won't include the code to parse the JSON output and to locate the right entry here.
Instead, we created a small crate named [`bootloader-locator`] that wraps the needed functionality in a simple [`locate_bootloader`] function.
Let's add that crate as a dependency and use it:
[`bootloader-locator`]: https://docs.rs/bootloader-locator/0.0.4/bootloader_locator/index.html
[`locate_bootloader`]: https://docs.rs/bootloader-locator/0.0.4/bootloader_locator/fn.locate_bootloader.html
@@ -315,17 +413,24 @@ pub fn main() {
}
```
The `locate_bootloader` function takes the name of the bootloader dependency as argument to allow alternative bootloader crates that are named differently. Since the function might fail, we use the [`unwrap`] method to [panic] on an error. Panicking is ok here because the `boot` crate is only part of our build process.
The `locate_bootloader` function takes the name of the bootloader dependency as argument to allow alternative bootloader crates that are named differently.
Since the function might fail, we use the [`unwrap`] method to [panic] on an error.
Panicking is ok here because the `boot` crate is only part of our build process.
[`unwrap`]: https://doc.rust-lang.org/std/result/enum.Result.html#method.unwrap
[panic]: https://doc.rust-lang.org/stable/book/ch09-01-unrecoverable-errors-with-panic.html
If you're interested in how the `locate_bootloader` function works, [check out its source code][locate_bootloader source]. It first executes the `cargo metadata` command and parses it's result as JSON using the [`json` crate]. Then it traverses the parsed metadata to find the `bootloader` dependency and return its manifest path.
If you're interested in how the `locate_bootloader` function works, [check out its source code][locate_bootloader source].
It first executes the `cargo metadata` command and parses it's result as JSON using the [`json` crate].
Then it traverses the parsed metadata to find the `bootloader` dependency and return its manifest path.
[locate_bootloader source]: https://docs.rs/crate/bootloader-locator/0.0.4/source/src/lib.rs
[`json` crate]: https://docs.rs/json/0.12.4/json/
Let's try to run it to see whether it works. If everything succeeds, the [`dbg!`] macro should print the path to the `bootloader` source code. Note that we need to run the `boot` binary from the root directory of our workspace, not from within the `boot` directory. Otherwise the `locate_bootloader` function would operate on the `boot/Cargo.toml`, where it won't find a bootloader dependency.
Let's try to run it to see whether it works.
If everything succeeds, the [`dbg!`] macro should print the path to the `bootloader` source code.
Note that we need to run the `boot` binary from the root directory of our workspace, not from within the `boot` directory.
Otherwise the `locate_bootloader` function would operate on the `boot/Cargo.toml`, where it won't find a bootloader dependency.
[`dbg!`]: https://doc.rust-lang.org/std/macro.dbg.html
@@ -338,11 +443,14 @@ To run the `boot` crate from our workspace root (i.e. the kernel directory), we
[boot/src/main.rs:5] bootloader_manifest = "/.../.cargo/.../bootloader-.../Cargo.toml"
```
It worked! We see that the bootloader source code lives somewhere in the `.cargo` directory in our user directory. By querying the source code for the exact bootloader version that our kernel is using, we ensure that the bootloader and the kernel use the exact same version of the `BootInfo` type. This is important because the `BootInfo` type is not stable yet, so undefined behavior can occur when when using different `BootInfo` versions.
It worked! We see that the bootloader source code lives somewhere in the `.cargo` directory in our user directory.
By querying the source code for the exact bootloader version that our kernel is using, we ensure that the bootloader and the kernel use the exact same version of the `BootInfo` type.
This is important because the `BootInfo` type is not stable yet, so undefined behavior can occur when when using different `BootInfo` versions.
#### Running the Build Command
The next step is to run the build command of the bootloader. From the [`bootloader` docs] we learn that the crate requires the following build command:
The next step is to run the build command of the bootloader.
From the [`bootloader` docs] we learn that the crate requires the following build command:
```
cargo builder --kernel-manifest path/to/kernel/Cargo.toml \
@@ -351,7 +459,8 @@ cargo builder --kernel-manifest path/to/kernel/Cargo.toml \
In addition, the docs recommend to use the `--target-dir` and `--out-dir` arguments when building the bootloader as a dependency to override where cargo places the compilation artifacts.
Let's try to invoke that command from our `main` function. For that we use the [`process::Command`] type of the standard library, which allows us to spawn new processes and wait for their results:
Let's try to invoke that command from our `main` function.
For that we use the [`process::Command`] type of the standard library, which allows us to spawn new processes and wait for their results:
[`process::Command`]: https://doc.rust-lang.org/std/process/struct.Command.html
@@ -393,25 +502,34 @@ pub fn main() {
}
```
We use the [`Command::new`] function to create a new [`process::Command`]. Instead of hardcoding the command name "cargo", we use the [`CARGO` environment variable] that cargo sets when compiling the `boot` crate. This way, we ensure that we use the exact same cargo version for compiling the `bootloader` crate, which is important when using non-standard cargo versions, e.g. through rustup's [toolchain override shorthands]. Since the environment variable is set at compile time, we use the compiler-builtin [`env!`] macro to retrieve its value.
We use the [`Command::new`] function to create a new [`process::Command`].
Instead of hardcoding the command name "cargo", we use the [`CARGO` environment variable] that cargo sets when compiling the `boot` crate.
This way, we ensure that we use the exact same cargo version for compiling the `bootloader` crate, which is important when using non-standard cargo versions, e.g. through rustup's [toolchain override shorthands].
Since the environment variable is set at compile time, we use the compiler-builtin [`env!`] macro to retrieve its value.
[`Command::new`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.new
[`CARGO` environment variable]: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-crates
[toolchain override shorthands]: https://rust-lang.github.io/rustup/overrides.html#toolchain-override-shorthand
[`env!`]: https://doc.rust-lang.org/std/macro.env.html
After creating the `Command` type, we pass all the required arguments by calling the [`Command::arg`] method. Most of the paths are still set to [`todo!()`] as a placeholder and will be filled out in a moment.
After creating the `Command` type, we pass all the required arguments by calling the [`Command::arg`] method.
Most of the paths are still set to [`todo!()`] as a placeholder and will be filled out in a moment.
[`Command::arg`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.arg
[`todo!()`]: https://doc.rust-lang.org/std/macro.todo.html
Since the build command needs to be run inside the source directory of the `bootloader` crate, we use the [`Command::current_dir`] method to set the working directory accordingly. We can determine the `bootloader_dir` path from the `bootloader_manifest` path by using the [`Path::parent`] method. Since not all paths have a parent directory (e.g. the path `/` has not), the `parent()` call can fail. However, this should never happen for the `bootloader_manifest` path, so we use the [`Option::unwrap`] method that panics on `None`.
Since the build command needs to be run inside the source directory of the `bootloader` crate, we use the [`Command::current_dir`] method to set the working directory accordingly.
We can determine the `bootloader_dir` path from the `bootloader_manifest` path by using the [`Path::parent`] method.
Since not all paths have a parent directory (e.g. the path `/` has not), the `parent()` call can fail.
However, this should never happen for the `bootloader_manifest` path, so we use the [`Option::unwrap`] method that panics on `None`.
[`Command::current_dir`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.current_dir
[`Path::parent`]: https://doc.rust-lang.org/std/path/struct.Path.html#method.parent
[`Option::unwrap`]: https://doc.rust-lang.org/std/option/enum.Option.html#method.unwrap
After setting the arguments and the working directory, we use the [`Command::status`] method to execute the command and wait for its exit status. Through the [`ExitStatus::success`] method we verify that the command was successful. If not we use the [`panic!`] macro to cause a panic.
After setting the arguments and the working directory, we use the [`Command::status`] method to execute the command and wait for its exit status.
Through the [`ExitStatus::success`] method we verify that the command was successful.
If not we use the [`panic!`] macro to cause a panic.
[`Command::current_dir`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.current_dir
[`Command::status`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.status
@@ -420,7 +538,8 @@ After setting the arguments and the working directory, we use the [`Command::sta
#### Filling in the Paths
We still need to fill in the paths we marked as `todo!` above. We start with the path to the kernel binary:
We still need to fill in the paths we marked as `todo!` above.
We start with the path to the kernel binary:
```rust
// in `main` in boot/src/main.rs
@@ -431,9 +550,12 @@ use std::path::Path;
let kernel_binary = Path::new("target/x86_64-blog_os/debug/blog_os").canonicalize().unwrap();
```
By default, cargo places our compiled kernel executable in a subdirectory of the `target` folder. The `x86_64_blog_os` is the name of our target JSON file and the `debug` indicates that this was a build with debug information and without optimizations. For now we simply hardcode the path to keep things simple, but we will make it more flexible later in this post.
By default, cargo places our compiled kernel executable in a subdirectory of the `target` folder.
The `x86_64_blog_os` is the name of our target JSON file and the `debug` indicates that this was a build with debug information and without optimizations.
For now we simply hardcode the path to keep things simple, but we will make it more flexible later in this post.
Since we're going to need an absolute path, we use the [`Path::canonicalize`] method to get the full path to the file. We use [`unwrap`] to panic if the file doesn't exist.
Since we're going to need an absolute path, we use the [`Path::canonicalize`] method to get the full path to the file.
We use [`unwrap`] to panic if the file doesn't exist.
[`Path::canonicalize`]: https://doc.rust-lang.org/std/path/struct.Path.html#method.canonicalize
[`Result`]: https://doc.rust-lang.org/std/result/enum.Result.html
@@ -455,17 +577,21 @@ let target_dir = kernel_dir.join("target");
let out_dir = kernel_binary.parent().unwrap();
```
The [`CARGO_MANIFEST_DIR`] environment variable always points to the `boot` directory, even if the crate is built from a different directory (e.g. via cargo's `--manifest-path` argument). This gives use a good starting point for creating the paths we care about since we know that our kernel lives in the [parent][`Path::parent`] directory.
The [`CARGO_MANIFEST_DIR`] environment variable always points to the `boot` directory, even if the crate is built from a different directory (e.g. via cargo's `--manifest-path` argument).
This gives use a good starting point for creating the paths we care about since we know that our kernel lives in the [parent][`Path::parent`] directory.
[`CARGO_MANIFEST_DIR`]: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-crates
From the `kernel_dir`, we can then construct the `kernel_manifest` and `target_dir` paths using the [`Path::join`] method. For the `out_dir` binding, we use the parent directory of the `kernel_binary` path. This way, the bootloader will create the disk image files next to our kernel executable.
From the `kernel_dir`, we can then construct the `kernel_manifest` and `target_dir` paths using the [`Path::join`] method.
For the `out_dir` binding, we use the parent directory of the `kernel_binary` path.
This way, the bootloader will create the disk image files next to our kernel executable.
[`Path::join`]: https://doc.rust-lang.org/std/path/struct.Path.html#method.join
#### Creating the Disk Images
There is one last step before we can create the bootable disk images: The `bootloader` build requires the [rustup component] `llvm-tools-preview`. To install it, we can either run `rustup component add llvm-tools-preview` or specify it in our `rust-toolchain` file:
There is one last step before we can create the bootable disk images: The `bootloader` build requires the [rustup component] `llvm-tools-preview`.
To install it, we can either run `rustup component add llvm-tools-preview` or specify it in our `rust-toolchain` file:
[rustup component]: https://rust-lang.github.io/rustup/concepts/components.html
@@ -484,7 +610,12 @@ After that can finally use our `boot` crate to create some bootable disk images
> cargo run --package boot
```
We first compile our kernel through `cargo kbuild` to ensure that the kernel binary is up to date. Then we run our `boot` crate through `cargo run --package boot`, which takes the kernel binary and builds the bootloader around it. The result are some disk image files named `bootimage-*` next to our kernel binary inside `target/x86_64-blog_os/debug`. Note that the command will only work from the root directory of our project. This is because we hardcoded the `kernel_binary` path in our `main` function. We will fix this later in the post, but first it is time to actually run our kernel!
We first compile our kernel through `cargo kbuild` to ensure that the kernel binary is up to date.
Then we run our `boot` crate through `cargo run --package boot`, which takes the kernel binary and builds the bootloader around it.
The result are some disk image files named `bootimage-*` next to our kernel binary inside `target/x86_64-blog_os/debug`.
Note that the command will only work from the root directory of our project.
This is because we hardcoded the `kernel_binary` path in our `main` function.
We will fix this later in the post, but first it is time to actually run our kernel!
From the [`bootloader` docs], we learn that the bootloader the following disk images:
@@ -494,24 +625,30 @@ From the [`bootloader` docs], we learn that the bootloader the following disk im
- A FAT partition image named `bootimage-uefi-<bin_name>.fat`, which contains the EFI executable under `efi\boot\bootx64.efi`.
- A GPT disk image named `bootimage-uefi-<bin_name>.img`, which contains the FAT image as EFI system partition.
In general, the `.img` files are the ones that you want to copy to an USB stick in order to boot from it. The other files are useful for booting the kernel in virtual machines such as [QEMU]. The `<bin_name>` placeholder is the binary name of the kernel, i.e. `blog_os` or the crate name you chose.
In general, the `.img` files are the ones that you want to copy to an USB stick in order to boot from it.
The other files are useful for booting the kernel in virtual machines such as [QEMU].
The `<bin_name>` placeholder is the binary name of the kernel, i.e. `blog_os` or the crate name you chose.
## Running our Kernel
After creating a bootable disk image for our kernel, we are finally able to run it. Before we learn how to run it on real hardware, we start by running it inside the [QEMU] system emulator. This has multiple advantages:
After creating a bootable disk image for our kernel, we are finally able to run it.
Before we learn how to run it on real hardware, we start by running it inside the [QEMU] system emulator.
This has multiple advantages:
- We can't break anything: Our kernel has full hardware access, so that a bug might have serious consequences on read hardware.
- We don't need a separate computer: QEMU runs as a normal program on our development computer.
- The edit-test cycle is much faster: We don't need to copy the disk image to bootable usb stick on every kernel change.
- It's possible to debug our kernel via QEMU's debug tools and GDB.
We will still learn how to boot our kernel on real hardware later in this post, but for now we focus on QEMU. For that you need to install QEMU on your machine as described on the [QEMU download page].
We will still learn how to boot our kernel on real hardware later in this post, but for now we focus on QEMU.
For that you need to install QEMU on your machine as described on the [QEMU download page].
[QEMU download page]: https://www.qemu.org/download/
### Running in QEMU
After installing QEMU, you can run `qemu-system-x86_64 --version` in a terminal to verify that it is installed. Then you can run the BIOS disk image of our kernel through the following command:
After installing QEMU, you can run `qemu-system-x86_64 --version` in a terminal to verify that it is installed.
Then you can run the BIOS disk image of our kernel through the following command:
```
qemu-system-x86_64 -drive \
@@ -522,20 +659,29 @@ As a result, you should see a window open that looks like this:
![QEMU printing several `INFO:` log messages](qemu-bios.png)
This output comes from the bootloader. As we see, the last line is _"Jumping to kernel entry point at […]"_. This is the point where the `_start` function of our kernel is called. Since we currently only `loop {}` in that function nothing else happens, so it is expected that we don't see any additional output.
This output comes from the bootloader.
As we see, the last line is _"Jumping to kernel entry point at […]"_.
This is the point where the `_start` function of our kernel is called.
Since we currently only `loop {}` in that function nothing else happens, so it is expected that we don't see any additional output.
Running the UEFI disk image works in a similar way, but we need to pass some additional files to QEMU to emulate an UEFI firmware. This is necessary because QEMU does not support emulating an UEFI firmware natively. The files that we need are provided by the [Open Virtual Machine Firmware (OVMF)][OVMF] project, which is a sub-project of [TianoCore] and implements UEFI support for virtual machines. Unfortunately, the project is only [sparsely documented][ovmf-whitepaper] and does not even have a clear homepage.
Running the UEFI disk image works in a similar way, but we need to pass some additional files to QEMU to emulate an UEFI firmware.
This is necessary because QEMU does not support emulating an UEFI firmware natively.
The files that we need are provided by the [Open Virtual Machine Firmware (OVMF)][OVMF] project, which is a sub-project of [TianoCore] and implements UEFI support for virtual machines.
Unfortunately, the project is only [sparsely documented][ovmf-whitepaper] and does not even have a clear homepage.
[OVMF]: https://github.com/tianocore/tianocore.github.io/wiki/OVMF
[TianoCore]: https://www.tianocore.org/
[ovmf-whitepaper]: https://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt
The easiest way to work with OVMF is to download pre-built images of the code. We provide such images in the [`rust-osdev/ovmf-prebuilt`] repository, which is updated daily from [Gerd Hoffman's RPM builds](https://www.kraxel.org/repos/). The compiled OVMF are provided as [GitHub releases][ovmf-prebuilt-releases].
The easiest way to work with OVMF is to download pre-built images of the code.
We provide such images in the [`rust-osdev/ovmf-prebuilt`] repository, which is updated daily from [Gerd Hoffman's RPM builds](https://www.kraxel.org/repos/).
The compiled OVMF are provided as [GitHub releases][ovmf-prebuilt-releases].
[`rust-osdev/ovmf-prebuilt`]: https://github.com/rust-osdev/ovmf-prebuilt/
[ovmf-prebuilt-releases]: https://github.com/rust-osdev/ovmf-prebuilt/releases/latest
To run our UEFI disk image in QEMU, we need the `OVMF_pure-efi.fd` file (other files might work as well). After downloading it, we can then run our UEFI disk image using the following command:
To run our UEFI disk image in QEMU, we need the `OVMF_pure-efi.fd` file (other files might work as well).
After downloading it, we can then run our UEFI disk image using the following command:
```
qemu-system-x86_64 -drive \
@@ -548,17 +694,22 @@ If everything works, this command opens a window with the following content:
![QEMU printing several `INFO:` log messages](qemu-uefi.png)
The output is a bit different than with the BIOS disk image. Among other things, it explicitly mentions that this is an UEFI boot right on top.
The output is a bit different than with the BIOS disk image.
Among other things, it explicitly mentions that this is an UEFI boot right on top.
### Screen Output
While we see some screen output from the bootloader, our kernel still does nothing. Let's fix this by trying to output something to the screen from our kernel too.
While we see some screen output from the bootloader, our kernel still does nothing.
Let's fix this by trying to output something to the screen from our kernel too.
Screen output works through a so-called [_framebuffer_]. A framebuffer is a memory region that contains the pixels that should be shown on the screen. The graphics card automatically reads the contents of this region on every screen refresh and updates the shown pixels accordingly.
Screen output works through a so-called [_framebuffer_].
A framebuffer is a memory region that contains the pixels that should be shown on the screen.
The graphics card automatically reads the contents of this region on every screen refresh and updates the shown pixels accordingly.
[_framebuffer_]: https://en.wikipedia.org/wiki/Framebuffer
Since the size, pixel format, and memory location of the framebuffer can vary between different systems, we need to find out these parameters first. The easiest way to do this is to read it from the [boot information structure][`BootInfo`] that the bootloader passes as argument to our kernel entry point:
Since the size, pixel format, and memory location of the framebuffer can vary between different systems, we need to find out these parameters first.
The easiest way to do this is to read it from the [boot information structure][`BootInfo`] that the bootloader passes as argument to our kernel entry point:
```rust
// in src/main.rs
@@ -572,18 +723,23 @@ fn kernel_main(boot_info: &'static mut BootInfo) -> ! {
}
```
Even though most systems support a framebuffer, some might not. The [`BootInfo`] type reflects this by specifying its `framebuffer` field as an [`Option`]. Since screen output won't be essential for our kernel (there are other possible communication channels such as serial ports), we use an [`if let`] statement to run the framebuffer code only if a framebuffer is available.
Even though most systems support a framebuffer, some might not.
The [`BootInfo`] type reflects this by specifying its `framebuffer` field as an [`Option`].
Since screen output won't be essential for our kernel (there are other possible communication channels such as serial ports), we use an [`if let`] statement to run the framebuffer code only if a framebuffer is available.
[`Option`]: https://doc.rust-lang.org/std/option/enum.Option.html
[`if let`]: https://doc.rust-lang.org/reference/expressions/if-expr.html#if-let-expressions
The [`FrameBuffer`] type provides two methods: The `info` method returns a [`FrameBufferInfo`] instance with all kinds of information about the framebuffer format, including the pixel type and the screen resolution. The `buffer` method returns the actual framebuffer content in form of a mutable byte [slice].
The [`FrameBuffer`] type provides two methods: The `info` method returns a [`FrameBufferInfo`] instance with all kinds of information about the framebuffer format, including the pixel type and the screen resolution.
The `buffer` method returns the actual framebuffer content in form of a mutable byte [slice].
[`FrameBuffer`]: https://docs.rs/bootloader/0.10.1/bootloader/boot_info/struct.FrameBuffer.html
[`FrameBufferInfo`]: https://docs.rs/bootloader/0.10.1/bootloader/boot_info/struct.FrameBufferInfo.html
[slice]: https://doc.rust-lang.org/std/primitive.slice.html
We will look into programming the framebuffer in detail in the next post. For now, let's just try setting the whole screen to some color. For this, we just set every pixel in the byte slice to some fixed value:
We will look into programming the framebuffer in detail in the next post.
For now, let's just try setting the whole screen to some color.
For this, we just set every pixel in the byte slice to some fixed value:
```rust
@@ -601,7 +757,8 @@ fn kernel_main(boot_info: &'static mut BootInfo) -> ! {
While it depends on the pixel color format how these values are interpreted, the result will likely be some shade of gray since we set the same value for every color channel (e.g. in the RGB color format).
After running `cargo kbuild` and then our `boot` script again, we can boot the new version in QEMU. We see that our guess that the whole screen would turn gray was right:
After running `cargo kbuild` and then our `boot` script again, we can boot the new version in QEMU.
We see that our guess that the whole screen would turn gray was right:
![QEMU showing a gray screen](qemu-gray.png)
@@ -624,17 +781,24 @@ fn kernel_main(boot_info: &'static mut BootInfo) -> ! {
}
```
We use the [`wrapping_add`] method here because Rust panics on implicit integer overflow (at least in debug mode). By adding a prime number, we try to add some variety. The result looks as follows:
We use the [`wrapping_add`] method here because Rust panics on implicit integer overflow (at least in debug mode).
By adding a prime number, we try to add some variety.
The result looks as follows:
![QEMU showing repeating gradient columns](qemu-wrapping-add.png)
### Booting on Real Hardware
To boot on real hardware, you first need to write either the `bootimage-uefi-blog_os.img` or the `bootimage-bios-blog_os.img` disk image to an USB stick. This deletes everything on the stick, so be careful. The actual steps to do this depend on your operating system.
To boot on real hardware, you first need to write either the `bootimage-uefi-blog_os.img` or the `bootimage-bios-blog_os.img` disk image to an USB stick.
This deletes everything on the stick, so be careful.
The actual steps to do this depend on your operating system.
#### Unix-like
On any Unix-like host OS (including both Linux and macOS), you can use the `dd` command to write the disk image directly to a USB drive. First run either `sudo fdisk -l` (on Linux) or `diskutil list` (on a Mac) to get info about where in `/dev` the file representing your device is located. After that, open a terminal window and run either of the following commands:
On any Unix-like host OS (including both Linux and macOS), you can use the `dd` command to write the disk image directly to a USB drive.
First run either `sudo fdisk -l` (on Linux) or `diskutil list` (on a Mac) to get info about where in `/dev` the file representing your device is located.
After that, open a terminal window and run either of the following commands:
##### Linux
```
@@ -648,11 +812,14 @@ $ sudo dd if=boot-uefi-blog_os.img of=/dev/sdX
$ sudo dd if=boot-uefi-blog_os.img of=/dev/diskX
```
**WARNING**: Be very careful when running this command. If you specify the wrong device as the `of=` parameter, you could end up wiping your system clean, so make sure the device you run it on is a removable one.
**WARNING**: Be very careful when running this command.
If you specify the wrong device as the `of=` parameter, you could end up wiping your system clean, so make sure the device you run it on is a removable one.
#### Windows
On Windows, you can use the [Rufus] tool, which is developed as an open-source project [on GitHub][rufus-github]. After downloading it you can directly run it, there's no installation necessary. In the interface, you select the USB stick you want to write to
On Windows, you can use the [Rufus] tool, which is developed as an open-source project [on GitHub][rufus-github].
After downloading it you can directly run it, there's no installation necessary.
In the interface, you select the USB stick you want to write to
[Rufus]: https://rufus.ie/
[rufus-github]: https://github.com/pbatard/rufus
@@ -693,7 +860,8 @@ On Windows, you can use the [Rufus] tool, which is developed as an open-source p
For running `bootimage` and building the bootloader, you need to have the `llvm-tools-preview` rustup component installed. You can do so by executing `rustup component add llvm-tools-preview`.
For running `bootimage` and building the bootloader, you need to have the `llvm-tools-preview` rustup component installed.
You can do so by executing `rustup component add llvm-tools-preview`.
@@ -720,9 +888,12 @@ It is also possible to write it to an USB stick and boot it on a real machine:
> dd if=target/x86_64-blog_os/debug/bootimage-blog_os.img of=/dev/sdX && sync
```
Where `sdX` is the device name of your USB stick. **Be careful** to choose the correct device name, because everything on that device is overwritten.
Where `sdX` is the device name of your USB stick.
**Be careful** to choose the correct device name, because everything on that device is overwritten.
After writing the image to the USB stick, you can run it on real hardware by booting from it. You probably need to use a special boot menu or change the boot order in your BIOS configuration to boot from the USB stick. Note that it currently doesn't work for UEFI machines, since the `bootloader` crate has no UEFI support yet.
After writing the image to the USB stick, you can run it on real hardware by booting from it.
You probably need to use a special boot menu or change the boot order in your BIOS configuration to boot from the USB stick.
Note that it currently doesn't work for UEFI machines, since the `bootloader` crate has no UEFI support yet.
### Using `cargo run`
@@ -735,9 +906,15 @@ To make it easier to run our kernel in QEMU, we can set the `runner` configurati
runner = "bootimage runner"
```
The `target.'cfg(target_os = "none")'` table applies to all targets that have set the `"os"` field of their target configuration file to `"none"`. This includes our `x86_64-blog_os.json` target. The `runner` key specifies the command that should be invoked for `cargo run`. The command is run after a successful build with the executable path passed as first argument. See the [cargo documentation][cargo configuration] for more details.
The `target.'cfg(target_os = "none")'` table applies to all targets that have set the `"os"` field of their target configuration file to `"none"`.
This includes our `x86_64-blog_os.json` target.
The `runner` key specifies the command that should be invoked for `cargo run`.
The command is run after a successful build with the executable path passed as first argument.
See the [cargo documentation][cargo configuration] for more details.
The `bootimage runner` command is specifically designed to be usable as a `runner` executable. It links the given executable with the project's bootloader dependency and then launches QEMU. See the [Readme of `bootimage`] for more details and possible configuration options.
The `bootimage runner` command is specifically designed to be usable as a `runner` executable.
It links the given executable with the project's bootloader dependency and then launches QEMU.
See the [Readme of `bootimage`] for more details and possible configuration options.
[Readme of `bootimage`]: https://github.com/rust-osdev/bootimage
@@ -745,4 +922,5 @@ Now we can use `cargo run` to compile our kernel and boot it in QEMU.
## What's next?
In the next post, we will explore the VGA text buffer in more detail and write a safe interface for it. We will also add support for the `println` macro.
In the next post, we will explore the VGA text buffer in more detail and write a safe interface for it.
We will also add support for the `println` macro.

342
blog/content/edition-3/posts/02-booting/uefi/index.md
View File

@@ -24,11 +24,14 @@ This post is an addendum to our main [**Booting**] post.
[**Booting**]: @/edition-3/posts/02-booting/index.md
-->
This post explains how to create a basic UEFI application from scratch that can be directly booted on modern x86_64 systems. This includes creating a minimal application suitable for the UEFI environment, turning it into a bootable disk image, and interacting with the hardware through the UEFI system tables and the `uefi` crate.
This post explains how to create a basic UEFI application from scratch that can be directly booted on modern x86_64 systems.
This includes creating a minimal application suitable for the UEFI environment, turning it into a bootable disk image, and interacting with the hardware through the UEFI system tables and the `uefi` crate.
<!-- more -->
This blog is openly developed on [GitHub]. If you have any problems or questions, please open an issue there. You can also leave comments [at the bottom].
This blog is openly developed on [GitHub].
If you have any problems or questions, please open an issue there.
You can also leave comments [at the bottom].
[GitHub]: https://github.com/phil-opp/blog_os
[at the bottom]: #comments
@@ -38,7 +41,8 @@ This blog is openly developed on [GitHub]. If you have any problems or questions
## Minimal UEFI App
We start by creating a new `cargo` project with a `Cargo.toml` and a `src/main.rs`. You can run `cargo new uefi_app` for that or create the files manually:
We start by creating a new `cargo` project with a `Cargo.toml` and a `src/main.rs`.
You can run `cargo new uefi_app` for that or create the files manually:
```toml
# in Cargo.toml
@@ -80,7 +84,9 @@ fn panic(_info: &PanicInfo) -> ! {
}
```
The `#![no_std]` attribute disables the linking of the Rust standard library, which is not available on bare metal. Through the `#![no_main]` attribute, we disable the normal entry point function that based on the C runtime. The `#[panic_handler]` attribute specifies which function should be called when a panic occurs.
The `#![no_std]` attribute disables the linking of the Rust standard library, which is not available on bare metal.
Through the `#![no_main]` attribute, we disable the normal entry point function that based on the C runtime.
The `#[panic_handler]` attribute specifies which function should be called when a panic occurs.
Next, we create an entry point function named `efi_main`:
@@ -100,16 +106,24 @@ pub extern "efiapi" fn efi_main(
}
```
This function signature is standardized by the UEFI specification, which is available [in PDF form][uefi-pdf] on [_uefi.org_]. You can find the signature of the entry point function in section 4.1. The function name `efi_main` is not required by the standard, but it is the common convention for UEFI applications and the Rust compiler will look for a function with that name by default.
This function signature is standardized by the UEFI specification, which is available [in PDF form][uefi-pdf] on [_uefi.org_].
You can find the signature of the entry point function in section 4.1.
The function name `efi_main` is not required by the standard, but it is the common convention for UEFI applications and the Rust compiler will look for a function with that name by default.
Since UEFI also defines a specific [calling convention] (in section 2.3), we set the [`efiapi` calling convention] for our function. Support for this calling function is still unstable in Rust, so we need to add `#![feature(abi_efiapi)]` at the very top of our file.
Since UEFI also defines a specific [calling convention] (in section 2.3), we set the [`efiapi` calling convention] for our function.
Support for this calling function is still unstable in Rust, so we need to add `#![feature(abi_efiapi)]` at the very top of our file.
[uefi-pdf]: https://uefi.org/sites/default/files/resources/UEFI%20Spec%202.8B%20May%202020.pdf
[_uefi.org_]: https://uefi.org/specifications
[calling convention]: https://en.wikipedia.org/wiki/Calling_convention
[`efiapi` calling convention]: https://github.com/rust-lang/rust/issues/65815
The function takes two arguments: an _image handle_ and a _system table_. The image handle is a firmware-allocated handle that identifies the UEFI image. The system table contains some input and output handles and provides access to various functions provided by the UEFI firmware. The function returns an `EFI_STATUS` integer to signal whether the function was successful. It is normally only returned by UEFI apps that are not bootloaders, e.g. UEFI drivers or apps that are launched manually from the UEFI shell. Bootloaders typically pass control to a OS kernel and never return.
The function takes two arguments: an _image handle_ and a _system table_.
The image handle is a firmware-allocated handle that identifies the UEFI image.
The system table contains some input and output handles and provides access to various functions provided by the UEFI firmware.
The function returns an `EFI_STATUS` integer to signal whether the function was successful.
It is normally only returned by UEFI apps that are not bootloaders, e.g. UEFI drivers or apps that are launched manually from the UEFI shell.
Bootloaders typically pass control to a OS kernel and never return.
### UEFI Target
@@ -174,9 +188,12 @@ From the output we can derive multiple properties of the target:
- The `exe-suffix` is `.efi`, which means that all executables compiled for this target have the suffix `.efi`.
- As it's typical for [kernel targets][custom target], both the redzone and SSE are disabled.
- The `is-like-windows` is an indicator that the target uses the conventions of Windows world, e.g. [PE] instead of [ELF] executables.
- The [LLD linker] is used, which ships with Rust. The linker has native support for cross-linking, which means that we can link Windows executables on non-Windows systems without any problems.
- The [LLD linker] is used, which ships with Rust.
The linker has native support for cross-linking, which means that we can link Windows executables on non-Windows systems without any problems.
- Like for most bare-metal targets, the `panic-strategy` is set to `abort` to disable unwinding.
- Various linker arguments are specified. For example, the `/entry` argument sets the name of the entry point function. This is the reason that we named our entry point function `efi_main` and applied the `#[no_mangle]` attribute above.
- Various linker arguments are specified.
For example, the `/entry` argument sets the name of the entry point function.
This is the reason that we named our entry point function `efi_main` and applied the `#[no_mangle]` attribute above.
[PE]: https://en.wikipedia.org/wiki/Portable_Executable
[ELF]: https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
@@ -184,14 +201,16 @@ From the output we can derive multiple properties of the target:
[custom target]: @/edition-2/posts/02-minimal-rust-kernel/index.md#target-specification
If you're interested in understanding all these fields, check out the docs for Rust's internal [`Target`] and [`TargetOptions`] types. These are the types that the above JSON is converted to.
If you're interested in understanding all these fields, check out the docs for Rust's internal [`Target`] and [`TargetOptions`] types.
These are the types that the above JSON is converted to.
[`Target`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_target/spec/struct.Target.html
[`TargetOptions`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_target/spec/struct.TargetOptions.html
### Building
Even though the `x86_64-unknown-uefi` target is a built-in of Rust, there are no precompiled versions of the `core` library available for it. This means that we need to use cargo's [`build-std` feature] as described in the [_Minimal Kernel_][minimal-kernel-build-std] post.
Even though the `x86_64-unknown-uefi` target is a built-in of Rust, there are no precompiled versions of the `core` library available for it.
This means that we need to use cargo's [`build-std` feature] as described in the [_Minimal Kernel_][minimal-kernel-build-std] post.
[`build-std` feature]: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#build-std
@@ -200,24 +219,29 @@ Even though the `x86_64-unknown-uefi` target is a built-in of Rust, there are no
-->
[minimal-kernel-build-std]: @/edition-2/posts/02-minimal-rust-kernel/index.md#the-build-std-option
A nightly Rust compiler is required for building, so we need to set up a [rustup override] for the directory. We can do this either by running a [`rustup override` command] or by adding a [`rust-toolchain.toml` file].
A nightly Rust compiler is required for building, so we need to set up a [rustup override] for the directory.
We can do this either by running a [`rustup override` command] or by adding a [`rust-toolchain.toml` file].
[rustup override]: https://rust-lang.github.io/rustup/overrides.html
[`rustup override` command]: https://rust-lang.github.io/rustup/overrides.html#directory-overrides
[`rust-toolchain.toml` file]: https://rust-lang.github.io/rustup/overrides.html#the-toolchain-file
After doing that, we can finally build our UEFI app. The full build command looks like this:
After doing that, we can finally build our UEFI app.
The full build command looks like this:
```bash
cargo build --target x86_64-unknown-uefi -Z build-std=core \
-Z build-std-features=compiler-builtins-mem
```
This results in a `uefi_app.efi` file in our `x86_64-unknown-uefi/debug` folder. Congratulations! We just created our own minimal UEFI app.
This results in a `uefi_app.efi` file in our `x86_64-unknown-uefi/debug` folder.
Congratulations! We just created our own minimal UEFI app.
## Bootable Disk Image
To make our minimal UEFI app bootable, we need to create a new [GPT] disk image with a [EFI system partition]. On that partition, we need to put our `.efi` file under `efi\boot\bootx64.efi`. Then the UEFI firmware should automatically detect and load it when we boot from the corresponding disk.
To make our minimal UEFI app bootable, we need to create a new [GPT] disk image with a [EFI system partition].
On that partition, we need to put our `.efi` file under `efi\boot\bootx64.efi`.
Then the UEFI firmware should automatically detect and load it when we boot from the corresponding disk.
[GPT]: https://en.wikipedia.org/wiki/GUID_Partition_Table
@@ -234,7 +258,8 @@ To create this disk image, we create a new `disk_image` executable:
> cargo new --bin disk_image
```
This creates a new cargo project in a `disk_image` subdirectory. To share the `target` folder and `Cargo.lock` file with our `uefi_app` project, we set up a cargo workspace:
This creates a new cargo project in a `disk_image` subdirectory.
To share the `target` folder and `Cargo.lock` file with our `uefi_app` project, we set up a cargo workspace:
```toml
# in Cargo.toml
@@ -245,7 +270,9 @@ members = ["disk_image"]
### FAT Filesystem
The first step to create an EFI system partition is to create a new partition image formatted with the [FAT] file system. The reason for using FAT is that this is the only file system that the UEFI standard requires. In practice, most UEFI firmware implementations also support the [NTFS] filesystem, but we can't rely on that since this is not required by the standard.
The first step to create an EFI system partition is to create a new partition image formatted with the [FAT] file system.
The reason for using FAT is that this is the only file system that the UEFI standard requires.
In practice, most UEFI firmware implementations also support the [NTFS] filesystem, but we can't rely on that since this is not required by the standard.
[FAT]: https://en.wikipedia.org/wiki/File_Allocation_Table
[NTFS]: https://en.wikipedia.org/wiki/NTFS
@@ -301,27 +328,33 @@ fn create_fat_filesystem(fat_path: &Path, efi_file: &Path) {
}
```
We first use [`fs::metadata`] to query the size of our `.efi` file and then round it up to the next megabyte. We then use this rounded size to create a new FAT filesystem image file.
We first use [`fs::metadata`] to query the size of our `.efi` file and then round it up to the next megabyte.
We then use this rounded size to create a new FAT filesystem image file.
<span class="gray">
(I'm not sure if the rounding is really necessary, but I had some problems with the `fatfs` crate when trying to use the unaligned size.)
</span>
[`fs::metadata`]: https://doc.rust-lang.org/std/fs/fn.metadata.html
After creating the file that should hold the FAT filesystem image, we use the [`format_volume`] function of `fatfs` to create the new FAT filesystem. After creating it, we use the [`FileSystem::new`] function to open it. The last step is to create the `efi/boot` directory and the `bootx64.efi` file on the filesystem. To write our `.efi` file to the filesystem image, we use the [`io::copy`] function of the Rust standard library.
After creating the file that should hold the FAT filesystem image, we use the [`format_volume`] function of `fatfs` to create the new FAT filesystem.
After creating it, we use the [`FileSystem::new`] function to open it.
The last step is to create the `efi/boot` directory and the `bootx64.efi` file on the filesystem.
To write our `.efi` file to the filesystem image, we use the [`io::copy`] function of the Rust standard library.
[`format_volume`]: https://docs.rs/fatfs/0.3.5/fatfs/fn.format_volume.html
[`FileSystem::new`]: https://docs.rs/fatfs/0.3.5/fatfs/struct.FileSystem.html#method.new
[`io::copy`]: https://doc.rust-lang.org/std/io/fn.copy.html
Note that we're not doing any error handling here to keep the code short. This is not that problematic because the `disk_image` crate is only part of our build process, but you still might want to use at least [`expect`] instead of `unwrap()` or an error handling crate such as [`anyhow`].
Note that we're not doing any error handling here to keep the code short.
This is not that problematic because the `disk_image` crate is only part of our build process, but you still might want to use at least [`expect`] instead of `unwrap()` or an error handling crate such as [`anyhow`].
[`expect`]: https://doc.rust-lang.org/std/result/enum.Result.html#method.expect
[`anyhow`]: https://docs.rs/anyhow/1.0.38/anyhow/
### GPT Disk Image
To make the FAT filesystem that we just created bootable, we need to place it as an [EFI system partition] on a [`GPT`]-formatted disk. To create the GPT disk image, we use the [`gpt`] crate:
To make the FAT filesystem that we just created bootable, we need to place it as an [EFI system partition] on a [`GPT`]-formatted disk.
To create the GPT disk image, we use the [`gpt`] crate:
[`GPT`]: https://en.wikipedia.org/wiki/GUID_Partition_Table
[`gpt`]: https://docs.rs/gpt/2.0.0/gpt/
@@ -388,33 +421,46 @@ fn create_gpt_disk(disk_path: &Path, fat_image: &Path) {
}
```
First, we create a new disk image file at the given `disk_path`. We set its size to the size of the FAT partition plus some extra amount to account for the GPT structure itself.
First, we create a new disk image file at the given `disk_path`.
We set its size to the size of the FAT partition plus some extra amount to account for the GPT structure itself.
To ensure that the disk image is not detected as an unformatted disk on older systems and accidentally overwritten, we create a so-called [_protective MBR_]. The idea is to create a normal [master boot record] structure on the disk that specifies a single partition that spans the whole disk. This way, older systems that don't know the `GPT` format see a disk formatted with an unknown parititon type instead of an unformatted disk.
To ensure that the disk image is not detected as an unformatted disk on older systems and accidentally overwritten, we create a so-called [_protective MBR_].
The idea is to create a normal [master boot record] structure on the disk that specifies a single partition that spans the whole disk.
This way, older systems that don't know the `GPT` format see a disk formatted with an unknown parititon type instead of an unformatted disk.
[_protective MBR_]: https://en.wikipedia.org/wiki/GUID_Partition_Table#Protective_MBR_(LBA_0)
[master boot record]: https://en.wikipedia.org/wiki/Master_boot_record
Next, we create the actual [`GPT`] structure through the [`GptConfig`] type and its [`create_from_device`] method. The result is a [`GptDisk`] type that writes to our `disk` file. Since we want to start with an empty partition table, we use the [`update_partitions`] method to reset the partition table. This isn't strictly necessary since we create a completely new GPT disk, but it's better to be safe.
Next, we create the actual [`GPT`] structure through the [`GptConfig`] type and its [`create_from_device`] method.
The result is a [`GptDisk`] type that writes to our `disk` file.
Since we want to start with an empty partition table, we use the [`update_partitions`] method to reset the partition table.
This isn't strictly necessary since we create a completely new GPT disk, but it's better to be safe.
[`GptConfig`]: https://docs.rs/gpt/2.0.0/gpt/struct.GptConfig.html
[`create_from_device`]: https://docs.rs/gpt/2.0.0/gpt/struct.GptConfig.html#method.create_from_device
[`GptDisk`]: https://docs.rs/gpt/2.0.0/gpt/struct.GptDisk.html
[`update_partitions`]: https://docs.rs/gpt/2.0.0/gpt/struct.GptDisk.html#method.update_partitions
After resetting the new partition table, we create a new partition named `boot` in the partition table. This operation only looks for a free region on the disk and stores the offset and size of that region in the table, together with the partition name and type (an [EFI system partition] in this case). It does not write any bytes to the partition itself. To do that later, we keep track of the `start_offset` of the partition.
After resetting the new partition table, we create a new partition named `boot` in the partition table.
This operation only looks for a free region on the disk and stores the offset and size of that region in the table, together with the partition name and type (an [EFI system partition] in this case).
It does not write any bytes to the partition itself.
To do that later, we keep track of the `start_offset` of the partition.
At this point, we are done with the GPT structure. To write it out to our `disk` file, we use the [`GptDisk::write`] function.
At this point, we are done with the GPT structure.
To write it out to our `disk` file, we use the [`GptDisk::write`] function.
[`GptDisk::write`]: https://docs.rs/gpt/2.0.0/gpt/struct.GptDisk.html#method.write
The final step is to write our `FAT` filesystem image to the newly created partition. For that we use the [`Seek::seek`] function to move the file cursor to the `start_offset` of the parititon. We then use the [`io::copy`] function to copy all the bytes from our `FAT` image file to the disk partition.
The final step is to write our `FAT` filesystem image to the newly created partition.
For that we use the [`Seek::seek`] function to move the file cursor to the `start_offset` of the parititon.
We then use the [`io::copy`] function to copy all the bytes from our `FAT` image file to the disk partition.
[`Seek::seek`]: https://doc.rust-lang.org/std/io/trait.Seek.html#tymethod.seek
### Putting it Together
We now have functions to create the FAT filesystem and GPT disk image. We just need to put them together in our `main` function:
We now have functions to create the FAT filesystem and GPT disk image.
We just need to put them together in our `main` function:
```rust
// in disk_image/src/main.rs
@@ -436,17 +482,24 @@ fn main() {
}
```
To be flexible, we take the path to the `.efi` file as command line argument. For retrieving the arguments we use the [`env::args`] function. The first argument is always set to the path of the `disk_image` executable itself by the operating system, even if the executable is invoked without arguments. We don't need it, so we prefix the variable name with an underscore to silence the "unused variable" warning.
To be flexible, we take the path to the `.efi` file as command line argument.
For retrieving the arguments we use the [`env::args`] function.
The first argument is always set to the path of the `disk_image` executable itself by the operating system, even if the executable is invoked without arguments.
We don't need it, so we prefix the variable name with an underscore to silence the "unused variable" warning.
[`env::args`]: https://doc.rust-lang.org/std/env/fn.args.html
Note that this is a very rudimentary way of doing argument parsing. There are a lot of crates out there that provide nice abstractions for this, for example [`clap`], [`structopt`], or [`argh`]. It is strongly recommend to use such a crate instead of writing your own argument parsing.
Note that this is a very rudimentary way of doing argument parsing.
There are a lot of crates out there that provide nice abstractions for this, for example [`clap`], [`structopt`], or [`argh`].
It is strongly recommend to use such a crate instead of writing your own argument parsing.
[`clap`]: https://docs.rs/clap/2.33.3/clap/index.html
[`structopt`]: https://docs.rs/structopt/0.3.21/structopt/
[`argh`]: https://docs.rs/argh/0.1.4/argh/
From the `efi_path` given as argument, we construct the `fat_path` and `disk_path`. By changing only the file extension using [`Path::with_extension`], we place the FAT and GPT image file next to our `.efi` file. The final step is to invoke our `create_fat_filesystem` and `create_gpt_disk` functions with the corresponding paths as argument.
From the `efi_path` given as argument, we construct the `fat_path` and `disk_path`.
By changing only the file extension using [`Path::with_extension`], we place the FAT and GPT image file next to our `.efi` file.
The final step is to invoke our `create_fat_filesystem` and `create_gpt_disk` functions with the corresponding paths as argument.
[`Path::with_extension`]: https://doc.rust-lang.org/std/path/struct.Path.html#method.with_extension
@@ -456,15 +509,22 @@ Now we can run our `disk_image` executable to create the bootable disk image fro
cargo run --package disk_image -- target/x86_64-unknown-uefi/debug/uefi_app.efi
```
Note the additional `--` argument. The `cargo run` uses this special argument to separate `cargo run` arguments from the arguments that should be passed to the compiled executable. The path of course depends on your working directory, i.e. whether you run it from the project root or from the `disk_image` subdirectory. It also depends on whether you compiled the `uefi_app` in debug or `--release` mode.
Note the additional `--` argument.
The `cargo run` uses this special argument to separate `cargo run` arguments from the arguments that should be passed to the compiled executable.
The path of course depends on your working directory, i.e. whether you run it from the project root or from the `disk_image` subdirectory.
It also depends on whether you compiled the `uefi_app` in debug or `--release` mode.
<!-- TODO uncomment as soon as Booting post is ready
The result of this command is a `.fat` and a `.gdt` file next to the given `.efi` executable. These files can be launched in QEMU and on real hardware [as described][run-instructions] in the main _Booting_ post. The result should look something like this:
The result of this command is a `.fat` and a `.gdt` file next to the given `.efi` executable.
These files can be launched in QEMU and on real hardware [as described][run-instructions] in the main _Booting_ post.
The result should look something like this:
[run-instructions]: @/edition-3/posts/02-booting/index.md#running-our-kernel
-->
The result of this command is a `.fat` and a `.gdt` file next to the given `.efi` executable. These files can be booted on real hardware, but it's easier and safer to start them in a virtual machine first. In this post, we're using the [**QEMU**](https://www.qemu.org/) emulator.
The result of this command is a `.fat` and a `.gdt` file next to the given `.efi` executable.
These files can be booted on real hardware, but it's easier and safer to start them in a virtual machine first.
In this post, we're using the [**QEMU**](https://www.qemu.org/) emulator.
### Running in QEMU
@@ -474,18 +534,23 @@ First, you need to install QEMU on your machine as described on the [QEMU downlo
After installing QEMU, you can run `qemu-system-x86_64 --version` in a terminal to verify that it is installed.
Since QEMU does not support emulating an UEFI firmware natively, we need to download some additional files to emulate an UEFI firmware. The files that we need for that are provided by the [Open Virtual Machine Firmware (OVMF)][OVMF] project, which is a sub-project of [TianoCore] and implements UEFI support for virtual machines. Unfortunately, the project is only [sparsely documented][ovmf-whitepaper] and does not even have a clear homepage.
Since QEMU does not support emulating an UEFI firmware natively, we need to download some additional files to emulate an UEFI firmware.
The files that we need for that are provided by the [Open Virtual Machine Firmware (OVMF)][OVMF] project, which is a sub-project of [TianoCore] and implements UEFI support for virtual machines.
Unfortunately, the project is only [sparsely documented][ovmf-whitepaper] and does not even have a clear homepage.
[OVMF]: https://github.com/tianocore/tianocore.github.io/wiki/OVMF
[TianoCore]: https://www.tianocore.org/
[ovmf-whitepaper]: https://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt
The easiest way to work with OVMF is to download pre-built images of the code. We provide such images in the [`rust-osdev/ovmf-prebuilt`] repository, which is updated daily from [Gerd Hoffman's RPM builds](https://www.kraxel.org/repos/). The compiled OVMF are provided as [GitHub releases][ovmf-prebuilt-releases].
The easiest way to work with OVMF is to download pre-built images of the code.
We provide such images in the [`rust-osdev/ovmf-prebuilt`] repository, which is updated daily from [Gerd Hoffman's RPM builds](https://www.kraxel.org/repos/).
The compiled OVMF are provided as [GitHub releases][ovmf-prebuilt-releases].
[`rust-osdev/ovmf-prebuilt`]: https://github.com/rust-osdev/ovmf-prebuilt/
[ovmf-prebuilt-releases]: https://github.com/rust-osdev/ovmf-prebuilt/releases/latest
To run our UEFI disk image in QEMU, we need the **`OVMF_pure-efi.fd`** file (other files might work as well). After downloading it, we can then run our UEFI disk image using the following command:
To run our UEFI disk image in QEMU, we need the **`OVMF_pure-efi.fd`** file (other files might work as well).
After downloading it, we can then run our UEFI disk image using the following command:
```
qemu-system-x86_64 -drive \
@@ -498,7 +563,8 @@ The result should look something like this:
![QEMU screenshot showing some UEFI firmware output](minimal-uefi-qemu.png)
We don't see any output from our `uefi_app` on the screen yet since we only `loop {}` in our `efi_main`. Instead, we see some output from the UEFI firmware itself that was created before our application was started.
We don't see any output from our `uefi_app` on the screen yet since we only `loop {}` in our `efi_main`.
Instead, we see some output from the UEFI firmware itself that was created before our application was started.
[`uefi`]: https://docs.rs/uefi/0.8.0/uefi/
@@ -506,11 +572,16 @@ Let's try to improve this by printing something to the screen from our `uefi_app
## The `uefi` Crate
In order to print something to the screen, we need to call some functions provided by the UEFI firmware. These functions can be invoked through the `system_table` argument passed to our `efi_main` function. This table provides [function pointers] for all kinds of functionality, including access to the screen, disk, or network.
In order to print something to the screen, we need to call some functions provided by the UEFI firmware.
These functions can be invoked through the `system_table` argument passed to our `efi_main` function.
This table provides [function pointers] for all kinds of functionality, including access to the screen, disk, or network.
[function pointers]: https://en.wikipedia.org/wiki/Function_pointer
Since the system table has a standardized format that is identical on all systems, it makes sense to create an abstraction for it. This is what the `uefi` crate does. It provides a [`SystemTable`] type that abstracts the UEFI system table functions as normal Rust methods. It is not complete, but the most important functions are all available.
Since the system table has a standardized format that is identical on all systems, it makes sense to create an abstraction for it.
This is what the `uefi` crate does.
It provides a [`SystemTable`] type that abstracts the UEFI system table functions as normal Rust methods.
It is not complete, but the most important functions are all available.
[`SystemTable`]: https://docs.rs/uefi/0.8.0/uefi/table/struct.SystemTable.html
@@ -537,12 +608,16 @@ pub extern "efiapi" fn efi_main(
}
```
Instead of using raw pointers and an anonymous `usize` return type, we now use the [`Handle`], [`SystemTable`], and [`Status`] abstraction types provided by the `uefi` crate. This way, we can use the higher-level API provided by the crate instead of carefully calculating pointer offsets to access the system table manually.
Instead of using raw pointers and an anonymous `usize` return type, we now use the [`Handle`], [`SystemTable`], and [`Status`] abstraction types provided by the `uefi` crate.
This way, we can use the higher-level API provided by the crate instead of carefully calculating pointer offsets to access the system table manually.
[`Handle`]: https://docs.rs/uefi/0.8.0/uefi/data_types/struct.Handle.html
[`Status`]: https://docs.rs/uefi/0.8.0/uefi/struct.Status.html
While the above function signature works, it is very fragile because the Rust compiler is not able to typecheck the function signature of entry point functions. Thus, we could accidentally use the wrong signature (e.g. after updating the `uefi` crate), which would cause undefined behavior. To prevent this, the `uefi` crate provides an [`entry` macro] to enforce the correct signature. To use it, we change our entry point function in the following way:
While the above function signature works, it is very fragile because the Rust compiler is not able to typecheck the function signature of entry point functions.
Thus, we could accidentally use the wrong signature (e.g. after updating the `uefi` crate), which would cause undefined behavior.
To prevent this, the `uefi` crate provides an [`entry` macro] to enforce the correct signature.
To use it, we change our entry point function in the following way:
[`entry` macro]: https://docs.rs/uefi/0.8.0/uefi/prelude/attr.entry.html
@@ -560,11 +635,15 @@ fn efi_main(
}
```
The macro already inserts the `#[no_mangle]` attribute and the `pub extern "efiapi"` modifiers for us, so we no longer need them. We will now get a compile error if the function signature is not correct (try it if you like).
The macro already inserts the `#[no_mangle]` attribute and the `pub extern "efiapi"` modifiers for us, so we no longer need them.
We will now get a compile error if the function signature is not correct (try it if you like).
### Printing to Screen
The UEFI standard supports multiple interfaces for printing to the screen. The most simple one is the _Simple Text Output_ protocol, which provides a console-like output interface. It is described in section 11.4 of the UEFI specification ([PDF][uefi-pdf]). We can use it through the [`SystemTable::stdout`] method provided by the `uefi` crate:
The UEFI standard supports multiple interfaces for printing to the screen.
The most simple one is the _Simple Text Output_ protocol, which provides a console-like output interface.
It is described in section 11.4 of the UEFI specification ([PDF][uefi-pdf]).
We can use it through the [`SystemTable::stdout`] method provided by the `uefi` crate:
[`SystemTable::stdout`]: https://docs.rs/uefi/0.8.0/uefi/table/struct.SystemTable.html#method.stdout
@@ -586,7 +665,11 @@ fn efi_main(
}
```
We first use the [`SystemTable::stdout`] method to get an [`Output`] reference. Through this reference, we can then [`clear`] the screen and write a "Hello World!" message through Rust's [`writeln`] macro. In order to be able to use the macro, we need to import the [`fmt::Write`] trait. Since this is only prototype code, we use the [`Result::unwrap`] method to panic on errors. For the `clear` call, we additionally call the [`Completion::unwrap`] method to ensure that the UEFI firmware did not throw any warnings.
We first use the [`SystemTable::stdout`] method to get an [`Output`] reference.
Through this reference, we can then [`clear`] the screen and write a "Hello World!" message through Rust's [`writeln`] macro.
In order to be able to use the macro, we need to import the [`fmt::Write`] trait.
Since this is only prototype code, we use the [`Result::unwrap`] method to panic on errors.
For the `clear` call, we additionally call the [`Completion::unwrap`] method to ensure that the UEFI firmware did not throw any warnings.
[`Output`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/text/struct.Output.html
[`clear`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/text/struct.Output.html#method.clear
@@ -612,7 +695,10 @@ The [`Output`] type also allows to use different colors through its [`set_color`
[`set_color`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/text/struct.Output.html#method.set_color
All of these functions are directly provided by the UEFI firmware, the `uefi` crate just provides some abstractions for this. By looking at the source code of the `uefi` crate, we see that the [`SystemTable`][system-table-src] is just a pointer to a [`SystemTableImpl`] struct, which is created by the UEFI firmware in a standardized format (see section _4.3_ of the UEFI specification ([PDF][uefi-pdf])). It has a `stdout` field, which is a pointer to an [`Output`][output-src] table. The methods of the `Output` type are just [small wrappers] around these function pointers, so all of the functionality is implemented directly in the UEFI firmware.
All of these functions are directly provided by the UEFI firmware, the `uefi` crate just provides some abstractions for this.
By looking at the source code of the `uefi` crate, we see that the [`SystemTable`][system-table-src] is just a pointer to a [`SystemTableImpl`] struct, which is created by the UEFI firmware in a standardized format (see section _4.3_ of the UEFI specification ([PDF][uefi-pdf])).
It has a `stdout` field, which is a pointer to an [`Output`][output-src] table.
The methods of the `Output` type are just [small wrappers] around these function pointers, so all of the functionality is implemented directly in the UEFI firmware.
[system-table-src]: https://docs.rs/uefi/0.8.0/src/uefi/table/system.rs.html#44-47
[`SystemTableImpl`]: https://docs.rs/uefi/0.8.0/src/uefi/table/system.rs.html#209-230
@@ -621,25 +707,42 @@ All of these functions are directly provided by the UEFI firmware, the `uefi` cr
### Boot Services
When we take a closer look at the documentation of the [`SystemTable`] type, we see that it has a generic `View` parameter. The documentation provides a good explanation why this parameter is needed:
When we take a closer look at the documentation of the [`SystemTable`] type, we see that it has a generic `View` parameter.
The documentation provides a good explanation why this parameter is needed:
> [...] Not all UEFI services will remain accessible forever. Some services, called "boot services", may only be called during a bootstrap stage where the UEFI firmware still has control of the hardware, and will become unavailable once the firmware hands over control of the hardware to an operating system loader. Others, called "runtime services", may still be used after that point [...]
> [...] Not all UEFI services will remain accessible forever.
Some services, called "boot services", may only be called during a bootstrap stage where the UEFI firmware still has control of the hardware, and will become unavailable once the firmware hands over control of the hardware to an operating system loader.
Others, called "runtime services", may still be used after that point [...]
>
> We handle this state transition by providing two different views of the UEFI system table, the "Boot" view and the "Runtime" view.
The distinction between "boot" and "runtime" services is defined directly by the UEFI standard ( in section 6), the `uefi` crate just provides an abstraction for this. The distinction is necessary because the UEFI firmware provides such a wide range of functionality, for example a memory allocator or access to network devices. These functions can easily conflict with operating system functionality, so they are only available before an operating system is loaded. To hand over hardware control from the UEFI firmware to an operating system, the UEFI standard provides an `ExitBootServices` function. The `uefi` crate abstracts this function as an [`SystemTable::exit_boot_services`] method.
The distinction between "boot" and "runtime" services is defined directly by the UEFI standard ( in section 6), the `uefi` crate just provides an abstraction for this.
The distinction is necessary because the UEFI firmware provides such a wide range of functionality, for example a memory allocator or access to network devices.
These functions can easily conflict with operating system functionality, so they are only available before an operating system is loaded.
To hand over hardware control from the UEFI firmware to an operating system, the UEFI standard provides an `ExitBootServices` function.
The `uefi` crate abstracts this function as an [`SystemTable::exit_boot_services`] method.
[`SystemTable::exit_boot_services`]: https://docs.rs/uefi/0.8.0/uefi/table/struct.SystemTable.html#method.exit_boot_services
### Interesting UEFI Protocols
The UEFI firmware supports many different hardware functions through so-called protocols. Most of them are not used by traditional operating systems, which instead implement their own drivers and access the different hardware devices directly. There are multiple reasons for this. For one, many protocols are no longer available after exiting boot services, so using the protocols is only possible as long as UEFI stays in control of the hardware (including physical memory allocation). Other reasons are performance (most drivers provided by UEFI are not optimized), control (not all device features are supported in UEFI), and compatibility (most operating systems want to run on non-UEFI systems too).
The UEFI firmware supports many different hardware functions through so-called protocols.
Most of them are not used by traditional operating systems, which instead implement their own drivers and access the different hardware devices directly.
There are multiple reasons for this.
For one, many protocols are no longer available after exiting boot services, so using the protocols is only possible as long as UEFI stays in control of the hardware (including physical memory allocation).
Other reasons are performance (most drivers provided by UEFI are not optimized), control (not all device features are supported in UEFI), and compatibility (most operating systems want to run on non-UEFI systems too).
Even if most operating systems quickly use the `ExitBootServices` function to take over hardware control, there are still a few useful UEFI protocols that are useful when implementing a bootloader. In the following, we present a few useful protocols and show how to use them.
Even if most operating systems quickly use the `ExitBootServices` function to take over hardware control, there are still a few useful UEFI protocols that are useful when implementing a bootloader.
In the following, we present a few useful protocols and show how to use them.
### Memory Allocation
As already mentioned above, the UEFI firmware is in control of memory until we use `ExitBootServices`. To supply additional memory to applications, the UEFI standard defines different memory allocation functions, which are defined in section _6.2_ of the standard ([PDF][uefi-pdf]). The `uefi` crate supports them too: We have to use the [`SystemTable::boot_services`] function to get access to the [`BootServices`] table. Then we can call the [`allocate_pool`] method to allocate a number of bytes from a UEFI-managed memory pool. Alternatively, we can allocate a number of 4KiB pages through [`allocate_pages`]. To free allocated memory again, we can use the [`free_pool`] and [`free_pages`] methods.
As already mentioned above, the UEFI firmware is in control of memory until we use `ExitBootServices`.
To supply additional memory to applications, the UEFI standard defines different memory allocation functions, which are defined in section _6.2_ of the standard ([PDF][uefi-pdf]).
The `uefi` crate supports them too: We have to use the [`SystemTable::boot_services`] function to get access to the [`BootServices`] table.
Then we can call the [`allocate_pool`] method to allocate a number of bytes from a UEFI-managed memory pool.
Alternatively, we can allocate a number of 4KiB pages through [`allocate_pages`].
To free allocated memory again, we can use the [`free_pool`] and [`free_pages`] methods.
[`SystemTable::boot_services`]: https://docs.rs/uefi/0.8.0/uefi/table/struct.SystemTable.html#method.boot_services
[`BootServices`]: https://docs.rs/uefi/0.8.0/uefi/table/boot/struct.BootServices.html
@@ -648,7 +751,8 @@ As already mentioned above, the UEFI firmware is in control of memory until we u
[`free_pool`]: https://docs.rs/uefi/0.8.0/uefi/table/boot/struct.BootServices.html#method.free_pool
[`free_pages`]: https://docs.rs/uefi/0.8.0/uefi/table/boot/struct.BootServices.html#method.free_pages
Using these methods, it is possible to create a Rust-compatible [`GlobalAlloc`], which allows linking the [`alloc`] crate (see the other posts on this blog). The `uefi` crate already provides such an allocator if we enable its `alloc` feature:
Using these methods, it is possible to create a Rust-compatible [`GlobalAlloc`], which allows linking the [`alloc`] crate (see the other posts on this blog).
The `uefi` crate already provides such an allocator if we enable its `alloc` feature:
[`GlobalAlloc`]: https://doc.rust-lang.org/nightly/core/alloc/trait.GlobalAlloc.html
[`alloc`]: https://doc.rust-lang.org/nightly/alloc/index.html
@@ -678,7 +782,8 @@ fn efi_main(
image: uefi::Handle,
system_table: uefi::table::SystemTable<uefi::table::Boot>,
) -> uefi::Status {
// ... (as before)
// ...
(as before)
// initialize the allocator
unsafe {
@@ -711,11 +816,17 @@ cargo build --target x86_64-unknown-uefi -Z build-std=core,alloc \
The only change is that `build-std` is now set to `core,alloc` instead of just `core`.
Note that the UEFI-provided allocation functions are only usable until `ExitBootServices` is called. This is the reason that the `uefi::alloc::init` function requires `unsafe`.
Note that the UEFI-provided allocation functions are only usable until `ExitBootServices` is called.
This is the reason that the `uefi::alloc::init` function requires `unsafe`.
### Locating the ACPI Tables
The [ACPI] standard is used to discover and configure hardware devices. It consists of multiple tables that are placed somewhere in memory by the firmware. To find out where in memory these tables are, we can use the UEFI configuration table, which is defined in section _4.6_ of the standard ([PDF][uefi-pdf]). To access it with the `uefi` crate, we use the [`SystemTable::config_table`] method, which returns a slice of [`ConfigTableEntry`] structs. To find the relevant ACPI [RSDP] table, we look for an entry with a [GUID] that is equal to [`ACPI_GUID`] or [`ACPI2_GUID`]. The `address` field of that entry then tells us the memory address of the RSPD table.
The [ACPI] standard is used to discover and configure hardware devices.
It consists of multiple tables that are placed somewhere in memory by the firmware.
To find out where in memory these tables are, we can use the UEFI configuration table, which is defined in section _4.6_ of the standard ([PDF][uefi-pdf]).
To access it with the `uefi` crate, we use the [`SystemTable::config_table`] method, which returns a slice of [`ConfigTableEntry`] structs.
To find the relevant ACPI [RSDP] table, we look for an entry with a [GUID] that is equal to [`ACPI_GUID`] or [`ACPI2_GUID`].
The `address` field of that entry then tells us the memory address of the RSPD table.
[ACPI]: https://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface
[`SystemTable::config_table`]: https://docs.rs/uefi/0.8.0/uefi/table/struct.SystemTable.html#method.config_table
@@ -741,7 +852,10 @@ We won't do anything with RSDP table here, but bootloaders typically provide it
### Graphics Output
As noted above, the text-based output protocol is only available until exiting UEFI boot services. Another drawback of it is that in only provides a text-based interface instead of allowing to set individual pixels. Fortunately, UEFI also supports a _Graphics Output Protocol_ (GOP) that fixes both of these problems. We can use it in the following way:
As noted above, the text-based output protocol is only available until exiting UEFI boot services.
Another drawback of it is that in only provides a text-based interface instead of allowing to set individual pixels.
Fortunately, UEFI also supports a _Graphics Output Protocol_ (GOP) that fixes both of these problems.
We can use it in the following way:
```rust
use uefi::proto::console::gop::GraphicsOutput;
@@ -753,18 +867,24 @@ writeln!(stdout, "current gop mode: {:?}", gop.current_mode_info()).unwrap();
writeln!(stdout, "framebuffer at: {:#p}", gop.frame_buffer().as_mut_ptr()).unwrap();
```
The [`locate_protocol`] method can be used to locate any protocol that implements the [`Protocol`] trait, including [`GraphicsOutput`]. Not all protocols are available on all systems though. In our case, we use `unwrap` to panic if the GOP protocol is not available.
The [`locate_protocol`] method can be used to locate any protocol that implements the [`Protocol`] trait, including [`GraphicsOutput`].
Not all protocols are available on all systems though.
In our case, we use `unwrap` to panic if the GOP protocol is not available.
[`locate_protocol`]: https://docs.rs/uefi/0.8.0/uefi/table/boot/struct.BootServices.html#method.locate_protocol
[`GraphicsOutput`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/gop/struct.GraphicsOutput.html
[`Protocol`]: https://docs.rs/uefi/0.8.0/uefi/proto/trait.Protocol.html
Since the UEFI-provided functions are neither thread-safe nor reentrant, the `locate_protocol` method returns an [`&UnsafeCell`], which is unsafe to access. We are sure that this is the first and only time that we use the GOP protocol, so we directly convert it to a `&mut` reference by using the [`UnsafeCell::get`] method and then converting the resulting `*mut` pointer via `&mut *`.
Since the UEFI-provided functions are neither thread-safe nor reentrant, the `locate_protocol` method returns an [`&UnsafeCell`], which is unsafe to access.
We are sure that this is the first and only time that we use the GOP protocol, so we directly convert it to a `&mut` reference by using the [`UnsafeCell::get`] method and then converting the resulting `*mut` pointer via `&mut *`.
[`&UnsafeCell`]: https://doc.rust-lang.org/nightly/core/cell/struct.UnsafeCell.html
[`UnsafeCell::get`]: https://doc.rust-lang.org/nightly/core/cell/struct.UnsafeCell.html#method.get
The [`GraphicsOutput`] type provides a wide range of functionality for configuring a pixel-based framebuffer. Through [`current_mode_info`], [`modes`], and [`set_mode`] we can query the currently active graphics mode, get a list of all supported modes, and enable a different mode. The [`frame_buffer`] method gives us direct access to the framebuffer through a [`FrameBuffer`] abstraction type. We can then read the raw pointer and size of the framebuffer via [`FrameBuffer::as_mut_ptr`] and [`FrameBuffer::size`].
The [`GraphicsOutput`] type provides a wide range of functionality for configuring a pixel-based framebuffer.
Through [`current_mode_info`], [`modes`], and [`set_mode`] we can query the currently active graphics mode, get a list of all supported modes, and enable a different mode.
The [`frame_buffer`] method gives us direct access to the framebuffer through a [`FrameBuffer`] abstraction type.
We can then read the raw pointer and size of the framebuffer via [`FrameBuffer::as_mut_ptr`] and [`FrameBuffer::size`].
[`current_mode_info`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/gop/struct.GraphicsOutput.html#method.current_mode_info
[`modes`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/gop/struct.GraphicsOutput.html#method.modes
@@ -774,7 +894,8 @@ The [`GraphicsOutput`] type provides a wide range of functionality for configuri
[`FrameBuffer::as_mut_ptr`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/gop/struct.FrameBuffer.html#method.as_mut_ptr
[`FrameBuffer::size`]: https://docs.rs/uefi/0.8.0/uefi/proto/console/gop/struct.FrameBuffer.html#method.size
As already mentioned, the GOP framebuffer stays available even after exiting boot services. Thus we can simply pass the framebuffer pointer, its mode info, and its size to the kernel, which can then easily write to screen, as we show in our upcoming _Screen Output_ post.
As already mentioned, the GOP framebuffer stays available even after exiting boot services.
Thus we can simply pass the framebuffer pointer, its mode info, and its size to the kernel, which can then easily write to screen, as we show in our upcoming _Screen Output_ post.
<!-- TODO: uncomment when post is ready
[_Screen Output_]: @/edition-3/posts/03-screen-output/index.md
@@ -782,12 +903,20 @@ As already mentioned, the GOP framebuffer stays available even after exiting boo
### Physical Memory Map
When the kernel takes control of memory management, it needs to know which physical memory areas are freely usable, which are still in use, and which are reserved by some hardware devices. To query this _memory map_ from the UEFI firmware, we can use the [`SystemTable::memory_map`] method. However the resulting memory map might still change as long as the UEFI firmware has control over memory and we still call other UEFI functions. For this reason, the UEFI firmware also returns an up-to-date memory map when [exiting boot services], which is the recommended way of retrieving the memory map.
When the kernel takes control of memory management, it needs to know which physical memory areas are freely usable, which are still in use, and which are reserved by some hardware devices.
To query this _memory map_ from the UEFI firmware, we can use the [`SystemTable::memory_map`] method.
However the resulting memory map might still change as long as the UEFI firmware has control over memory and we still call other UEFI functions.
For this reason, the UEFI firmware also returns an up-to-date memory map when [exiting boot services], which is the recommended way of retrieving the memory map.
[`SystemTable::memory_map`]: https://docs.rs/uefi/0.8.0/uefi/table/boot/struct.BootServices.html#method.memory_map
[exiting boot services]: https://docs.rs/uefi/0.8.0/uefi/table/struct.SystemTable.html#method.exit_boot_services
To use the [`exit_boot_services`], we need to provide a buffer that is big enough to hold the memory map. To find out how large the buffer needs to be, we can use the [`BootServices::memory_map_size`] method. Then we can use the [`allocate_pool`] method to allocate a buffer region of that size. However, since the `allocate_pool` call might change the memory map, it might become a bit larger than returned by `memory_map_size`. For this reason, we need to allocate a bit extra space. This can be implemented in the following way:
To use the [`exit_boot_services`], we need to provide a buffer that is big enough to hold the memory map.
To find out how large the buffer needs to be, we can use the [`BootServices::memory_map_size`] method.
Then we can use the [`allocate_pool`] method to allocate a buffer region of that size.
However, since the `allocate_pool` call might change the memory map, it might become a bit larger than returned by `memory_map_size`.
For this reason, we need to allocate a bit extra space.
This can be implemented in the following way:
[`exit_boot_services`]: https://docs.rs/uefi/0.8.0/uefi/table/struct.SystemTable.html#method.exit_boot_services
[`BootServices::memory_map_size`]: https://docs.rs/uefi/0.8.0/uefi/table/boot/struct.BootServices.html#method.memory_map_size
@@ -811,23 +940,34 @@ let (system_table, memory_map) = system_table.exit_boot_services(image, mmap_sto
.unwrap().unwrap()
```
This returns a new [`SystemTable`] instance that no longer provides access to the boot services. The `memory_map` return type is an iterator of [`MemoryDescriptor`] instances, which describe the physical start address, size, and type of each memory region.
This returns a new [`SystemTable`] instance that no longer provides access to the boot services.
The `memory_map` return type is an iterator of [`MemoryDescriptor`] instances, which describe the physical start address, size, and type of each memory region.
[`MemoryDescriptor`]: https://docs.rs/uefi/0.8.0/uefi/table/boot/struct.MemoryDescriptor.html
Note that we also need to call `uefi::alloc::exit_boot_services()` before exiting boot services to uninitialize the heap allocator again. Otherwise undefined behavior might occur if we accidentally use the `alloc` crate again afterwards.
Note that we also need to call `uefi::alloc::exit_boot_services()` before exiting boot services to uninitialize the heap allocator again.
Otherwise undefined behavior might occur if we accidentally use the `alloc` crate again afterwards.
## Creating a Bootloader
Now that we know how to set up a framebuffer and query relevant system information, we're only missing one crucial function to turn our UEFI application into a bootloader: loading a kernel. This includes loading a kernel executable into memory, setting up an execution environment, and passing control to the kernel's entry point function. Unfortunately, this process can be quite complex so that we cannot cover it here. However, we will give some high-level instructions in the following.
Now that we know how to set up a framebuffer and query relevant system information, we're only missing one crucial function to turn our UEFI application into a bootloader: loading a kernel.
This includes loading a kernel executable into memory, setting up an execution environment, and passing control to the kernel's entry point function.
Unfortunately, this process can be quite complex so that we cannot cover it here.
However, we will give some high-level instructions in the following.
### Loading the Kernel from Disk
The first step is to load the kernel executable from disk into main memory. One approach for including our kernel could be to place it in the FAT partition created by our `disk_image` crate. Then we could use the _simple file system_ protocol of UEFI (see section 12.3 of the standard ([PDF][uefi-pdf])) to load it from disk into memory. The `uefi` crate supports this protocol through its [`SimpleFileSystem`] type.
The first step is to load the kernel executable from disk into main memory.
One approach for including our kernel could be to place it in the FAT partition created by our `disk_image` crate.
Then we could use the _simple file system_ protocol of UEFI (see section 12.3 of the standard ([PDF][uefi-pdf])) to load it from disk into memory.
The `uefi` crate supports this protocol through its [`SimpleFileSystem`] type.
[`SimpleFileSystem`]: https://docs.rs/uefi/0.8.0/uefi/proto/media/fs/struct.SimpleFileSystem.html
To keep things simple, we will use a different appoach here. Instead of loading the kernel separately, we place its bytes as a `static` variable inside our bootloader executable. This way, the UEFI firmware directly loads it into memory when launching the bootloader. To implement this, we can use the [`include_bytes`] macro of Rust's `core` library:
To keep things simple, we will use a different appoach here.
Instead of loading the kernel separately, we place its bytes as a `static` variable inside our bootloader executable.
This way, the UEFI firmware directly loads it into memory when launching the bootloader.
To implement this, we can use the [`include_bytes`] macro of Rust's `core` library:
[`include_bytes`]: https://doc.rust-lang.org/nightly/core/macro.include_bytes.html
@@ -837,17 +977,32 @@ static KERNEL: &[u8] = include_bytes!("path/to/the/kernel/executable");
### Parsing the Kernel Executable
After loading the kernel executable into memory (one way or another), we need to parse it. In the following, we assume that the kernel uses the [ELF] executable format, which is popular in the Linux world. This is also the excutable format that the kernel created in this blog series uses.
After loading the kernel executable into memory (one way or another), we need to parse it.
In the following, we assume that the kernel uses the [ELF] executable format, which is popular in the Linux world.
This is also the excutable format that the kernel created in this blog series uses.
The ELF format consists of several headers that describe the executable and define a number of sections. Typically, there is a section called `.text` that contains the actual executable code. Immutable values such as string constants are placed in a section named `.rodata` ("read-only data"). For mutable data (e.g. a `static` containing a `Mutex`), a section named `.data` is used. There is also a section named `.bss` that stores all data that is initialized with zero values (this allows to reduce the size of the binary).
The ELF format consists of several headers that describe the executable and define a number of sections.
Typically, there is a section called `.text` that contains the actual executable code.
Immutable values such as string constants are placed in a section named `.rodata` ("read-only data").
For mutable data (e.g. a `static` containing a `Mutex`), a section named `.data` is used.
There is also a section named `.bss` that stores all data that is initialized with zero values (this allows to reduce the size of the binary).
The various ELF headers are useful in different situations. For loading the executable into memory, the _program header_ is most relevant. It basically groups all the sections of the executable into different groups by their access permissions. There are typically four groups:
The various ELF headers are useful in different situations.
For loading the executable into memory, the _program header_ is most relevant.
It basically groups all the sections of the executable into different groups by their access permissions.
There are typically four groups:
- Read-only and executable: This contains the `.text` section and all other executable code.
- Read-only: This contains the `.rodata` section and all other sections with immutable, non-executable data.
- Read-write: This includes the `.data` section and `.bss` sections. The zeroes of the `.bss` section are not actually stored, only its size is listed. Thus, no memory is wasted for storing zeroes.
- Read-write: This includes the `.data` section and `.bss` sections.
The zeroes of the `.bss` section are not actually stored, only its size is listed.
Thus, no memory is wasted for storing zeroes.
There are various tools to analyze ELF files and read out most headers. The classical tools are `readelf` and `objdump`. There are also several Rust crates for parsing an ELF files, so we don't need to to implement it on our own. Some examples are [`goblin`], [`elf`], and [`xmas-elf`]. The `xmas-elf` crate works quite well in `no_std` environments, so that's the one I would recommend for a bootloader implementation.
There are various tools to analyze ELF files and read out most headers.
The classical tools are `readelf` and `objdump`.
There are also several Rust crates for parsing an ELF files, so we don't need to to implement it on our own.
Some examples are [`goblin`], [`elf`], and [`xmas-elf`].
The `xmas-elf` crate works quite well in `no_std` environments, so that's the one I would recommend for a bootloader implementation.
[`goblin`]: https://docs.rs/goblin/0.3.4/goblin/
[`elf`]: https://docs.rs/elf/0.0.10/elf/
@@ -869,14 +1024,19 @@ for segment in elf_file_program_iter() {
### Virtual Memory Mapping
In order to run multiple programs isolated from each other in parallel, modern computers use a technique called [_virtual memory_]. We will cover virtual memory in detail later in this series, but the basic idea is to provide a _virtual address space_ split in 4KiB large blocks called _pages_. A [_page table_] maps each page to an arbitrary block of physical memory. This way, multiple programs can run at the same virtual address without conflict because they map to different physical memory behind the scenes.
In order to run multiple programs isolated from each other in parallel, modern computers use a technique called [_virtual memory_].
We will cover virtual memory in detail later in this series, but the basic idea is to provide a _virtual address space_ split in 4KiB large blocks called _pages_.
A [_page table_] maps each page to an arbitrary block of physical memory.
This way, multiple programs can run at the same virtual address without conflict because they map to different physical memory behind the scenes.
[_virtual memory_]: https://en.wikipedia.org/wiki/Virtual_memory
[_page table_]: https://en.wikipedia.org/wiki/Page_table
Virtual memory also has lots of other advantages such as fine-grained access control (read/write/execute permissions per page), support for safe shared memory (multiple read-only pages can be mapped to the same frame), and transparent swapping (moving some memory content to disk when main memory becomes too full).
For loading our kernel into virtual memory, we first need to create a new page table. In it, we add mappings for all segments of the kernel executable at their specified virtual addresses. We already loaded the kernel into physical memory, so we can calculate the corresponding frame for each page by adding the segment offset to the physical start address of the `KERNEL` static.
For loading our kernel into virtual memory, we first need to create a new page table.
In it, we add mappings for all segments of the kernel executable at their specified virtual addresses.
We already loaded the kernel into physical memory, so we can calculate the corresponding frame for each page by adding the segment offset to the physical start address of the `KERNEL` static.
Put together, the mapping process could look like this:
@@ -903,27 +1063,45 @@ if let Type::Load = segment.get_type()? {
}
```
As mentioned above, `.bss`-like sections are not stored in the executable since storing their null bytes would only bloat the executable. This results in ELF segments whose `mem_size()` (i.e. size in memory) is larger than their `file_size()` (i.e. segment size in the executable file). These segments require special handling: We need to allocate additional unused physical frames from the memory map we created above and initialize them with zero. Then we can map the additional `mem_size() - file_size()` bytes to these frames.
As mentioned above, `.bss`-like sections are not stored in the executable since storing their null bytes would only bloat the executable.
This results in ELF segments whose `mem_size()` (i.e. size in memory) is larger than their `file_size()` (i.e. segment size in the executable file).
These segments require special handling: We need to allocate additional unused physical frames from the memory map we created above and initialize them with zero.
Then we can map the additional `mem_size() - file_size()` bytes to these frames.
### Creating a Stack
After creating the page table mappings for the kernel, we need to allocate a [execution stack] for it. For that, we choose a region of unused physical memory from the physical memory map and map it to some virtual address. Ideally, we choose the virtual address range in a way that the page immediately before it is not mapped. Thus, we create a so-called _guard page_ that ensures that stack overflows lead to a CPU exception (a page fault) instead of corrupting other data.
After creating the page table mappings for the kernel, we need to allocate a [execution stack] for it.
For that, we choose a region of unused physical memory from the physical memory map and map it to some virtual address.
Ideally, we choose the virtual address range in a way that the page immediately before it is not mapped.
Thus, we create a so-called _guard page_ that ensures that stack overflows lead to a CPU exception (a page fault) instead of corrupting other data.
[execution stack]: https://en.wikipedia.org/wiki/Call_stack
### Switching to Kernel
The final step is to switch to the kernel address space and jump to its entry point function. For this, we need to fill the [`CR3`] register with the address of the created kernel page table and the `rsp` stack pointer register with the end address of the stack (the stack grows downwards on x86_64). Then we can use the `jmp` or `call` instruction to jump to the kernel entry point function. These steps require [inline assembly] and should be done directly after each other (in one `asm` block) because changing the `cr3` and `rsp` registers will break any following Rust code in the bootloader.
The final step is to switch to the kernel address space and jump to its entry point function.
For this, we need to fill the [`CR3`] register with the address of the created kernel page table and the `rsp` stack pointer register with the end address of the stack (the stack grows downwards on x86_64).
Then we can use the `jmp` or `call` instruction to jump to the kernel entry point function.
These steps require [inline assembly] and should be done directly after each other (in one `asm` block) because changing the `cr3` and `rsp` registers will break any following Rust code in the bootloader.
[`CR3`]: https://en.wikipedia.org/wiki/Control_register#CR3
[inline assembly]: https://doc.rust-lang.org/unstable-book/library-features/asm.html
The context switch function itself must be mapped to both the kernel and bootloader address spaces at the exact same address. This is required because the address space switch happens directly when reloading the `CR3` register, so while the code is still executing the code of the context switch function. So the context switch function must be mapped in the new address space too. The kernel can of course remove this mapping later.
The context switch function itself must be mapped to both the kernel and bootloader address spaces at the exact same address.
This is required because the address space switch happens directly when reloading the `CR3` register, so while the code is still executing the code of the context switch function.
So the context switch function must be mapped in the new address space too.
The kernel can of course remove this mapping later.
## Summary
We saw that the UEFI standard already implements lots of functionality. Rust's built-in support for the `x86_64-unknown-uefi` target makes it quite easy to create a minimal UEFI application. To turn the UEFI application into a bootable disk image, we created a `disk_image` builder binary that uses the [`fatfs`] and [`gpt`] crates.
We saw that the UEFI standard already implements lots of functionality.
Rust's built-in support for the `x86_64-unknown-uefi` target makes it quite easy to create a minimal UEFI application.
To turn the UEFI application into a bootable disk image, we created a `disk_image` builder binary that uses the [`fatfs`] and [`gpt`] crates.
The easiest way to access the services of the UEFI system table is the [`uefi`] crate. It provides abstractions for all kinds of UEFI protocols, including graphics output (text and framebuffer-based), memory allocation, and various system information (e.g. memory map and RSDP address).
The easiest way to access the services of the UEFI system table is the [`uefi`] crate.
It provides abstractions for all kinds of UEFI protocols, including graphics output (text and framebuffer-based), memory allocation, and various system information (e.g. memory map and RSDP address).
To turn the UEFI application into a bootloader, we first need to load the kernel executable from disk into memory. We then parse it and create virtual memory mappings for its segments in a new page table. We also need to allocate and map an execution stack for the kernel. The final step is to load the `CR3` and `rsp` registers accordingly and invoke the kernel's entry point function.
To turn the UEFI application into a bootloader, we first need to load the kernel executable from disk into memory.
We then parse it and create virtual memory mappings for its segments in a new page table.
We also need to allocate and map an execution stack for the kernel.
The final step is to load the `CR3` and `rsp` registers accordingly and invoke the kernel's entry point function.