diff --git a/blog/content/edition-2/posts/02-minimal-rust-kernel/index copy.md b/blog/content/edition-2/posts/02-minimal-rust-kernel/index copy.md new file mode 100644 index 00000000..4819ecc8 --- /dev/null +++ b/blog/content/edition-2/posts/02-minimal-rust-kernel/index copy.md @@ -0,0 +1,493 @@ ++++ +title = "A Minimal Rust Kernel" +weight = 2 +path = "minimal-rust-kernel" +date = 2018-02-10 + +[extra] +chapter = "Bare Bones" ++++ + +In this post we create a minimal 64-bit Rust kernel for the x86 architecture. We build upon the [freestanding Rust binary] from the previous post to create a bootable disk image, that prints something to the screen. + +[freestanding Rust binary]: @/edition-2/posts/01-freestanding-rust-binary/index.md + + + +This blog is openly developed on [GitHub]. If you have any problems or questions, please open an issue there. You can also leave comments [at the bottom]. The complete source code for this post can be found in the [`post-02`][post branch] branch. + +[GitHub]: https://github.com/phil-opp/blog_os +[at the bottom]: #comments +[post branch]: https://github.com/phil-opp/blog_os/tree/post-02 + + + +## The Boot Process +When you turn on a computer, it begins executing firmware code that is stored in motherboard [ROM]. This code performs a [power-on self-test], detects available RAM, and pre-initializes the CPU and hardware. Afterwards it looks for a bootable disk and starts booting the operating system kernel. + +[ROM]: https://en.wikipedia.org/wiki/Read-only_memory +[power-on self-test]: https://en.wikipedia.org/wiki/Power-on_self-test + +On x86, there are two firmware standards: the “Basic Input/Output System“ (**[BIOS]**) and the newer “Unified Extensible Firmware Interface” (**[UEFI]**). The BIOS standard is old and outdated, but simple and well-supported on any x86 machine since the 1980s. UEFI, in contrast, is more modern and has much more features, but is more complex to set up (at least in my opinion). + +[BIOS]: https://en.wikipedia.org/wiki/BIOS +[UEFI]: https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface + +Currently, we only provide BIOS support, but support for UEFI is planned, too. If you'd like to help us with this, check out the [Github issue](https://github.com/phil-opp/blog_os/issues/349). + +### BIOS Boot +Almost all x86 systems have support for BIOS booting, including newer UEFI-based machines that use an emulated BIOS. This is great, because you can use the same boot logic across all machines from the last centuries. But this wide compatibility is at the same time the biggest disadvantage of BIOS booting, because it means that the CPU is put into a 16-bit compatibility mode called [real mode] before booting so that archaic bootloaders from the 1980s would still work. + +But let's start from the beginning: + +When you turn on a computer, it loads the BIOS from some special flash memory located on the motherboard. The BIOS runs self test and initialization routines of the hardware, then it looks for bootable disks. If it finds one, the control is transferred to its _bootloader_, which is a 512-byte portion of executable code stored at the disk's beginning. Most bootloaders are larger than 512 bytes, so bootloaders are commonly split into a small first stage, which fits into 512 bytes, and a second stage, which is subsequently loaded by the first stage. + +The bootloader has to determine the location of the kernel image on the disk and load it into memory. It also needs to switch the CPU from the 16-bit [real mode] first to the 32-bit [protected mode], and then to the 64-bit [long mode], where 64-bit registers and the complete main memory are available. Its third job is to query certain information (such as a memory map) from the BIOS and pass it to the OS kernel. + +[real mode]: https://en.wikipedia.org/wiki/Real_mode +[protected mode]: https://en.wikipedia.org/wiki/Protected_mode +[long mode]: https://en.wikipedia.org/wiki/Long_mode +[memory segmentation]: https://en.wikipedia.org/wiki/X86_memory_segmentation + +Writing a bootloader is a bit cumbersome as it requires assembly language and a lot of non insightful steps like “write this magic value to this processor register”. Therefore we don't cover bootloader creation in this post and instead provide a tool named [bootimage] that automatically prepends a bootloader to your kernel. + +[bootimage]: https://github.com/rust-osdev/bootimage + +If you are interested in building your own bootloader: Stay tuned, a set of posts on this topic is already planned! + +#### The Multiboot Standard +To avoid that every operating system implements its own bootloader, which is only compatible with a single OS, the [Free Software Foundation] created an open bootloader standard called [Multiboot] in 1995. The standard defines an interface between the bootloader and operating system, so that any Multiboot compliant bootloader can load any Multiboot compliant operating system. The reference implementation is [GNU GRUB], which is the most popular bootloader for Linux systems. + +[Free Software Foundation]: https://en.wikipedia.org/wiki/Free_Software_Foundation +[Multiboot]: https://wiki.osdev.org/Multiboot +[GNU GRUB]: https://en.wikipedia.org/wiki/GNU_GRUB + +To make a kernel Multiboot compliant, one just needs to insert a so-called [Multiboot header] at the beginning of the kernel file. This makes it very easy to boot an OS in GRUB. However, GRUB and the Multiboot standard have some problems too: + +[Multiboot header]: https://www.gnu.org/software/grub/manual/multiboot/multiboot.html#OS-image-format + +- They support only the 32-bit protected mode. This means that you still have to do the CPU configuration to switch to the 64-bit long mode. +- They are designed to make the bootloader simple instead of the kernel. For example, the kernel needs to be linked with an [adjusted default page size], because GRUB can't find the Multiboot header otherwise. Another example is that the [boot information], which is passed to the kernel, contains lots of architecture dependent structures instead of providing clean abstractions. +- Both GRUB and the Multiboot standard are only sparsely documented. +- GRUB needs to be installed on the host system to create a bootable disk image from the kernel file. This makes development on Windows or Mac more difficult. + +[adjusted default page size]: https://wiki.osdev.org/Multiboot#Multiboot_2 +[boot information]: https://www.gnu.org/software/grub/manual/multiboot/multiboot.html#Boot-information-format + +Because of these drawbacks we decided to not use GRUB or the Multiboot standard. However, we plan to add Multiboot support to our [bootimage] tool, so that it's possible to load your kernel on a GRUB system too. If you're interested in writing a Multiboot compliant kernel, check out the [first edition] of this blog series. + +[first edition]: @/edition-1/_index.md + +### UEFI + +(We don't provide UEFI support at the moment, but we would love to! If you'd like to help, please tell us in the [Github issue](https://github.com/phil-opp/blog_os/issues/349).) + +## A Minimal Kernel +Now that we roughly know how a computer boots, it's time to create our own minimal kernel. Our goal is to create a disk image that prints a “Hello World!” to the screen when booted. For that we build upon the [freestanding Rust binary] from the previous post. + +As you may remember, we built the freestanding binary through `cargo`, but depending on the operating system we needed different entry point names and compile flags. That's because `cargo` builds for the _host system_ by default, i.e. the system you're running on. This isn't something we want for our kernel, because a kernel that runs on top of e.g. Windows does not make much sense. Instead, we want to compile for a clearly defined _target system_. + +### Installing Rust Nightly +Rust has three release channels: _stable_, _beta_, and _nightly_. The Rust Book explains the difference between these channels really well, so take a minute and [check it out](https://doc.rust-lang.org/book/appendix-07-nightly-rust.html#choo-choo-release-channels-and-riding-the-trains). For building an operating system we will need some experimental features that are only available on the nightly channel, so we need to install a nightly version of Rust. + +To manage Rust installations I highly recommend [rustup]. It allows you to install nightly, beta, and stable compilers side-by-side and makes it easy to update them. With rustup you can use a nightly compiler for the current directory by running `rustup override set nightly`. Alternatively, you can add a file called `rust-toolchain` with the content `nightly` to the project's root directory. You can check that you have a nightly version installed by running `rustc --version`: The version number should contain `-nightly` at the end. + +[rustup]: https://www.rustup.rs/ + +The nightly compiler allows us to opt-in to various experimental features by using so-called _feature flags_ at the top of our file. For example, we could enable the experimental [`asm!` macro] for inline assembly by adding `#![feature(asm)]` to the top of our `main.rs`. Note that such experimental features are completely unstable, which means that future Rust versions might change or remove them without prior warning. For this reason we will only use them if absolutely necessary. + +[`asm!` macro]: https://doc.rust-lang.org/unstable-book/library-features/asm.html + +### Target Specification +Cargo supports different target systems through the `--target` parameter. The target is described by a so-called _[target triple]_, which describes the CPU architecture, the vendor, the operating system, and the [ABI]. For example, the `x86_64-unknown-linux-gnu` target triple describes a system with a `x86_64` CPU, no clear vendor and a Linux operating system with the GNU ABI. Rust supports [many different target triples][platform-support], including `arm-linux-androideabi` for Android or [`wasm32-unknown-unknown` for WebAssembly](https://www.hellorust.com/setup/wasm-target/). + +[target triple]: https://clang.llvm.org/docs/CrossCompilation.html#target-triple +[ABI]: https://stackoverflow.com/a/2456882 +[platform-support]: https://forge.rust-lang.org/release/platform-support.html +[custom-targets]: https://doc.rust-lang.org/nightly/rustc/targets/custom.html + +For our target system, however, we require some special configuration parameters (e.g. no underlying OS), so none of the [existing target triples][platform-support] fits. Fortunately, Rust allows us to define [our own target][custom-targets] through a JSON file. For example, a JSON file that describes the `x86_64-unknown-linux-gnu` target looks like this: + +```json +{ + "llvm-target": "x86_64-unknown-linux-gnu", + "data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128", + "arch": "x86_64", + "target-endian": "little", + "target-pointer-width": "64", + "target-c-int-width": "32", + "os": "linux", + "executables": true, + "linker-flavor": "gcc", + "pre-link-args": ["-m64"], + "morestack": false +} +``` + +Most fields are required by LLVM to generate code for that platform. For example, the [`data-layout`] field defines the size of various integer, floating point, and pointer types. Then there are fields that Rust uses for conditional compilation, such as `target-pointer-width`. The third kind of fields define how the crate should be built. For example, the `pre-link-args` field specifies arguments passed to the [linker]. + +[`data-layout`]: https://llvm.org/docs/LangRef.html#data-layout +[linker]: https://en.wikipedia.org/wiki/Linker_(computing) + +We also target `x86_64` systems with our kernel, so our target specification will look very similar to the one above. Let's start by creating a `x86_64-blog_os.json` file (choose any name you like) with the common content: + +```json +{ + "llvm-target": "x86_64-unknown-none", + "data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128", + "arch": "x86_64", + "target-endian": "little", + "target-pointer-width": "64", + "target-c-int-width": "32", + "os": "none", + "executables": true +} +``` + +Note that we changed the OS in the `llvm-target` and the `os` field to `none`, because we will run on bare metal. + +We add the following build-related entries: + + +```json +"linker-flavor": "ld.lld", +"linker": "rust-lld", +``` + +Instead of using the platform's default linker (which might not support Linux targets), we use the cross platform [LLD] linker that is shipped with Rust for linking our kernel. + +[LLD]: https://lld.llvm.org/ + +```json +"panic-strategy": "abort", +``` + +This setting specifies that the target doesn't support [stack unwinding] on panic, so instead the program should abort directly. This has the same effect as the `panic = "abort"` option in our Cargo.toml, so we can remove it from there. (Note that in contrast to the Cargo.toml option, this target option also applies when we recompile the `core` library later in this post. So be sure to add this option, even if you prefer to keep the Cargo.toml option.) + +[stack unwinding]: https://www.bogotobogo.com/cplusplus/stackunwinding.php + +```json +"disable-redzone": true, +``` + +We're writing a kernel, so we'll need to handle interrupts at some point. To do that safely, we have to disable a certain stack pointer optimization called the _“red zone”_, because it would cause stack corruptions otherwise. For more information, see our separate post about [disabling the red zone]. + +[disabling the red zone]: @/edition-2/posts/02-minimal-rust-kernel/disable-red-zone/index.md + +```json +"features": "-mmx,-sse,+soft-float", +``` + +The `features` field enables/disables target features. We disable the `mmx` and `sse` features by prefixing them with a minus and enable the `soft-float` feature by prefixing it with a plus. Note that there must be no spaces between different flags, otherwise LLVM fails to interpret the features string. + +The `mmx` and `sse` features determine support for [Single Instruction Multiple Data (SIMD)] instructions, which can often speed up programs significantly. However, using the large SIMD registers in OS kernels leads to performance problems. The reason is that the kernel needs to restore all registers to their original state before continuing an interrupted program. This means that the kernel has to save the complete SIMD state to main memory on each system call or hardware interrupt. Since the SIMD state is very large (512–1600 bytes) and interrupts can occur very often, these additional save/restore operations considerably harm performance. To avoid this, we disable SIMD for our kernel (not for applications running on top!). + +[Single Instruction Multiple Data (SIMD)]: https://en.wikipedia.org/wiki/SIMD + +A problem with disabling SIMD is that floating point operations on `x86_64` require SIMD registers by default. To solve this problem, we add the `soft-float` feature, which emulates all floating point operations through software functions based on normal integers. + +For more information, see our post on [disabling SIMD](@/edition-2/posts/02-minimal-rust-kernel/disable-simd/index.md). + +#### Putting it Together +Our target specification file now looks like this: + +```json +{ + "llvm-target": "x86_64-unknown-none", + "data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128", + "arch": "x86_64", + "target-endian": "little", + "target-pointer-width": "64", + "target-c-int-width": "32", + "os": "none", + "executables": true, + "linker-flavor": "ld.lld", + "linker": "rust-lld", + "panic-strategy": "abort", + "disable-redzone": true, + "features": "-mmx,-sse,+soft-float" +} +``` + +### Building our Kernel +Compiling for our new target will use Linux conventions (I'm not quite sure why, I assume that it's just LLVM's default). This means that we need an entry point named `_start` as described in the [previous post]: + +[previous post]: @/edition-2/posts/01-freestanding-rust-binary/index.md + +```rust +// src/main.rs + +#![no_std] // don't link the Rust standard library +#![no_main] // disable all Rust-level entry points + +use core::panic::PanicInfo; + +/// This function is called on panic. +#[panic_handler] +fn panic(_info: &PanicInfo) -> ! { + loop {} +} + +#[no_mangle] // don't mangle the name of this function +pub extern "C" fn _start() -> ! { + // this function is the entry point, since the linker looks for a function + // named `_start` by default + loop {} +} +``` + +Note that the entry point needs to be called `_start` regardless of your host OS. + +We can now build the kernel for our new target by passing the name of the JSON file as `--target`: + +``` +> cargo build --target x86_64-blog_os.json + +error[E0463]: can't find crate for `core` +``` + +It fails! The error tells us that the Rust compiler no longer finds the [`core` library]. This library contains basic Rust types such as `Result`, `Option`, and iterators, and is implicitly linked to all `no_std` crates. + +[`core` library]: https://doc.rust-lang.org/nightly/core/index.html + +The problem is that the core library is distributed together with the Rust compiler as a _precompiled_ library. So it is only valid for supported host triples (e.g., `x86_64-unknown-linux-gnu`) but not for our custom target. If we want to compile code for other targets, we need to recompile `core` for these targets first. + +#### The `build-std` Option + +That's where the [`build-std` feature] of cargo comes in. It allows to recompile `core` and other standard library crates on demand, instead of using the precompiled versions shipped with the Rust installation. This feature is very new and still not finished, so it is marked as "unstable" and only available on [nightly Rust compilers]. + +[`build-std` feature]: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#build-std +[nightly Rust compilers]: #installing-rust-nightly + +To use the feature, we need to create a [cargo configuration] file at `.cargo/config.toml` with the following content: + +```toml +# in .cargo/config.toml + +[unstable] +build-std = ["core", "compiler_builtins"] +``` + +This tells cargo that it should recompile the `core` and `compiler_builtins` libraries. The latter is required because it is a dependency of `core`. In order to recompile these libraries, cargo needs access to the rust source code, which we can install with `rustup component add rust-src`. + +