From fdccda6b540f4f307e81c37bbfb3efec70a629e4 Mon Sep 17 00:00:00 2001 From: Philip Molloy Date: Fri, 27 Dec 2019 16:25:23 -0500 Subject: [PATCH 01/85] Add missing word to sentence --- .../second-edition/posts/01-freestanding-rust-binary/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md b/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md index 5564e899..8b2a7d47 100644 --- a/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md +++ b/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md @@ -145,7 +145,7 @@ Language items are special functions and types that are required internally by t [`Copy`]: https://doc.rust-lang.org/nightly/core/marker/trait.Copy.html [copy code]: https://github.com/rust-lang/rust/blob/485397e49a02a3b7ff77c17e4a3f16c653925cb3/src/libcore/marker.rs#L296-L299 -Providing own implementations of language items would be possible, but this should only be done as a last resort. The reason is that language items are highly unstable implementation details and not even type checked (so the compiler doesn't even check if a function has the right argument types). Fortunately, there is a more stable way to fix the above language item error. +Providing our own implementations of language items would be possible, but this should only be done as a last resort. The reason is that language items are highly unstable implementation details and not even type checked (so the compiler doesn't even check if a function has the right argument types). Fortunately, there is a more stable way to fix the above language item error. The `eh_personality` language item marks a function that is used for implementing [stack unwinding]. By default, Rust uses unwinding to run the destructors of all live stack variables in case of a [panic]. This ensures that all used memory is freed and allows the parent thread to catch the panic and continue execution. Unwinding, however, is a complicated process and requires some OS specific libraries (e.g. [libunwind] on Linux or [structured exception handling] on Windows), so we don't want to use it for our operating system. From d9d204a6cd500e069da42cdee24a1c6a6b9b5288 Mon Sep 17 00:00:00 2001 From: Philip Molloy Date: Sun, 29 Dec 2019 11:47:06 -0500 Subject: [PATCH 02/85] Remove superfluous adjective "own" may be used as an adjective, but must follow a possessive word. Rather than add "our" just remove the adjective entirely since ownership is not significant. Co-Authored-By: Philipp Oppermann --- .../second-edition/posts/01-freestanding-rust-binary/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md b/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md index 8b2a7d47..cfc1f87f 100644 --- a/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md +++ b/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md @@ -145,7 +145,7 @@ Language items are special functions and types that are required internally by t [`Copy`]: https://doc.rust-lang.org/nightly/core/marker/trait.Copy.html [copy code]: https://github.com/rust-lang/rust/blob/485397e49a02a3b7ff77c17e4a3f16c653925cb3/src/libcore/marker.rs#L296-L299 -Providing our own implementations of language items would be possible, but this should only be done as a last resort. The reason is that language items are highly unstable implementation details and not even type checked (so the compiler doesn't even check if a function has the right argument types). Fortunately, there is a more stable way to fix the above language item error. +While providing custom implementations of language items is possible, it should only be done as a last resort. The reason is that language items are highly unstable implementation details and not even type checked (so the compiler doesn't even check if a function has the right argument types). Fortunately, there is a more stable way to fix the above language item error. The `eh_personality` language item marks a function that is used for implementing [stack unwinding]. By default, Rust uses unwinding to run the destructors of all live stack variables in case of a [panic]. This ensures that all used memory is freed and allows the parent thread to catch the panic and continue execution. Unwinding, however, is a complicated process and requires some OS specific libraries (e.g. [libunwind] on Linux or [structured exception handling] on Windows), so we don't want to use it for our operating system. From f1b13d7ed363cda767dc51f0a509da1ccdedfb5c Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 5 Jan 2020 20:02:39 +0100 Subject: [PATCH 03/85] The LLVM error code bug was resolved (#711) We already updated the other posts in https://github.com/phil-opp/blog_os/pull/644 but forgot to update this post --- .../content/second-edition/posts/05-cpu-exceptions/index.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/blog/content/second-edition/posts/05-cpu-exceptions/index.md b/blog/content/second-edition/posts/05-cpu-exceptions/index.md index 840117a6..493c6bbb 100644 --- a/blog/content/second-edition/posts/05-cpu-exceptions/index.md +++ b/blog/content/second-edition/posts/05-cpu-exceptions/index.md @@ -193,12 +193,6 @@ In the `x86_64` crate, the interrupt stack frame is represented by the [`Interru [`InterruptStackFrame`]: https://docs.rs/x86_64/0.8.1/x86_64/structures/idt/struct.InterruptStackFrame.html -Note that there is currently [a bug in LLVM] that leads to wrong error code arguments. The cause of the issue is already known and a solution is [being worked on]. - -[a bug in LLVM]: https://github.com/rust-lang/rust/issues/57270 -[being worked on]: https://reviews.llvm.org/D56275 - - ### Behind the Scenes The `x86-interrupt` calling convention is a powerful abstraction that hides almost all of the messy details of the exception handling process. However, sometimes it's useful to know what's happening behind the curtain. Here is a short overview of the things that the `x86-interrupt` calling convention takes care of: From 05ac04d30dbb1d3a2ac4f5f817b5ad1c06e2f318 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Tue, 7 Jan 2020 12:23:32 +0100 Subject: [PATCH 04/85] Update copyright year in footer --- blog/templates/second-edition/base.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/templates/second-edition/base.html b/blog/templates/second-edition/base.html index 7319cb8a..e7df9a0f 100644 --- a/blog/templates/second-edition/base.html +++ b/blog/templates/second-edition/base.html @@ -39,7 +39,7 @@
- © . All rights reserved. + © . All rights reserved. Contact
From 1bde324162798d5d7269168d5fb2b8bdd58a1b11 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Tue, 7 Jan 2020 13:10:51 +0100 Subject: [PATCH 05/85] Remove azure pipelines and travis CI scripts --- .travis.yml | 13 ------------ azure-pipelines.yml | 48 --------------------------------------------- 2 files changed, 61 deletions(-) delete mode 100644 .travis.yml delete mode 100644 azure-pipelines.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index cab13b7b..00000000 --- a/.travis.yml +++ /dev/null @@ -1,13 +0,0 @@ -language: minimal - -branches: - only: - # This is where pull requests from "bors r+" are built. - - staging - # This is where pull requests from "bors try" are built. - - trying - # Build post braches - - /^post-.*$/ - -script: - - echo "Nothing to do for master branch" diff --git a/azure-pipelines.yml b/azure-pipelines.yml deleted file mode 100644 index 2098d3ca..00000000 --- a/azure-pipelines.yml +++ /dev/null @@ -1,48 +0,0 @@ -# Documentation: https://aka.ms/yaml - -trigger: - branches: - include: - - '*' - exclude: - - 'staging.tmp' - -pool: - vmImage: ubuntu-16.04 - -steps: -- bash: | - echo "Hello world from $AGENT_NAME running on $AGENT_OS" - echo "Reason: $BUILD_REASON" - echo "Requested for: $BUILD_REQUESTEDFOR" - displayName: 'Build Info' - continueOnError: true - -- bash: curl -sL https://github.com/getzola/zola/releases/download/v0.9.0/zola-v0.9.0-x86_64-unknown-linux-gnu.tar.gz | tar zxv - displayName: "Download Zola" - -- script: python -m pip install --upgrade pip setuptools wheel - displayName: 'Install Python Tools' - -- script: python -m pip install --user -r requirements.txt - displayName: 'Install Python Libraries' - workingDirectory: "blog" - -- script: python before_build.py - displayName: "Run before_build.py script" - workingDirectory: "blog" - -- script: ../zola build - displayName: "Build Site" - workingDirectory: "blog" - -- task: PublishPipelineArtifact@0 - inputs: - artifactName: 'generated_site' - targetPath: 'blog/public' - -- script: curl -L https://git.io/misspell | bash - displayName: "Install misspell" - -- script: bin/misspell -error blog/content - displayName: "Check for common typos" From dd5baca85d3449fb5ef5fe1238b06cf1fc47de9f Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Tue, 7 Jan 2020 13:11:07 +0100 Subject: [PATCH 06/85] Updates in December 2019 (#712) --- blog/content/status-update/2020-01-07.md | 58 ++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 blog/content/status-update/2020-01-07.md diff --git a/blog/content/status-update/2020-01-07.md b/blog/content/status-update/2020-01-07.md new file mode 100644 index 00000000..42893bcd --- /dev/null +++ b/blog/content/status-update/2020-01-07.md @@ -0,0 +1,58 @@ ++++ +title = "Updates in December 2019" +date = 2020-01-07 ++++ + +Happy New Year! + +This post gives an overview of the recent updates to the _Writing an OS in Rust_ blog and the corresponding libraries and tools. + +## `blog_os` + +The repository of the _Writing an OS in Rust_ blog received the following updates: + +- Update `x86_64` dependency to version 0.8.1. This included the [dependency update](https://github.com/phil-opp/blog_os/pull/701) itself, an [update of the frame allocation code](https://github.com/phil-opp/blog_os/pull/703), and an [update of the blog](https://github.com/phil-opp/blog_os/pull/704). +- [License the `blog/content` folder under CC BY-NC](https://github.com/phil-opp/blog_os/pull/705) +- [Reword sentence in first post](https://github.com/phil-opp/blog_os/pull/709) by [@pamolloy](https://github.com/pamolloy) + +Further, we're still working on adding [Experimental Support for Community Translations](https://github.com/phil-opp/blog_os/pull/692) to the blog, starting with [Simplified Chinese](https://github.com/phil-opp/blog_os/pull/694) and [Traditional Chinese](https://github.com/phil-opp/blog_os/pull/699). Any help is appreciated! + +## `bootloader` + +There were no updates to the bootloader this month. + +I'm currently working on rewriting the 16-bit/32-bit stages in Rust and making the bootloader more modular in the process. This should make it much easier to add support for UEFI and GRUB booting later. + +## `bootimage` + +There were no updates to the `bootimage` tool this month. + +## `x86_64` + +We landed a number of breaking changes this month: + +- [Replace `ux` dependency with custom wrapper structs](https://github.com/rust-osdev/x86_64/pull/91) +- [Add new UnusedPhysFrame type and use it in Mapper::map_to](https://github.com/rust-osdev/x86_64/pull/89) +- [Make Mapper trait object safe by adding `Self: Sized` bounds on generic functions](https://github.com/rust-osdev/x86_64/pull/84) +- [Rename divide_by_zero field of IDT to divide_error](https://github.com/rust-osdev/x86_64/pull/108) +- [Introduce new diverging handler functions for exceptions classified as "abort"](https://github.com/rust-osdev/x86_64/pull/109) + +These changes were released an version 0.8.0. Unfortunately, there was a missing re-export for the new `UnusedPhysFrame` type. We fixed it in [#110](https://github.com/rust-osdev/x86_64/pull/110) and released the fix as version 0.8.1. + +There was one more addition to the `x86_64` crate afterwards: + +- [Add support for cr4 control register (with complete documentation)](https://github.com/rust-osdev/x86_64/pull/111) by [@KarimAllah](https://github.com/KarimAllah) (released as version 0.8.2). + +There were also a few changes related to continuous integration: + +- [Remove bors from this repo](https://github.com/rust-osdev/x86_64/pull/103) +- [Run 'push' builds only for master branch](https://github.com/rust-osdev/x86_64/pull/104) +- [Remove Travis CI and Azure Pipelines scripts](https://github.com/rust-osdev/x86_64/pull/105) +- [Add caching of cargo crates to GitHub Actions CI](https://github.com/rust-osdev/x86_64/pull/100) + +## `cargo-xbuild` + +The `cargo-xbuild` crate, which cross-compiles the sysroot, received the following updates this month: + +- [Add `--quiet` flag that suppresses "waiting for file lock" message](https://github.com/rust-osdev/cargo-xbuild/pull/43) by [@Nils-TUD](https://github.com/Nils-TUD) (published as version 0.5.19) +- [Fix wrong feature name for memcpy=false](https://github.com/rust-osdev/cargo-xbuild/pull/50) (released as version 0.5.20) From 581849098f20df76ddb04d110eb387d5911464f9 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Wed, 8 Jan 2020 12:24:09 +0100 Subject: [PATCH 07/85] Fix typo --- blog/content/second-edition/posts/10-heap-allocation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/content/second-edition/posts/10-heap-allocation/index.md b/blog/content/second-edition/posts/10-heap-allocation/index.md index 51fa3932..0c517ec1 100644 --- a/blog/content/second-edition/posts/10-heap-allocation/index.md +++ b/blog/content/second-edition/posts/10-heap-allocation/index.md @@ -251,7 +251,7 @@ It defines the two required methods [`alloc`] and [`dealloc`], which correspond The trait additionally defines the two methods [`alloc_zeroed`] and [`realloc`] with default implementations: -- The [`alloc_zeroed`] method is equivalent to calling `alloc` and then setting the allocated memory block to zero, which is exactly what the provided default implementation does. An allocator implementations can override the default implementations with a more efficient custom implementation if possible. +- The [`alloc_zeroed`] method is equivalent to calling `alloc` and then setting the allocated memory block to zero, which is exactly what the provided default implementation does. An allocator implementation can override the default implementations with a more efficient custom implementation if possible. - The [`realloc`] method allows to grow or shrink an allocation. The default implementation allocates a new memory block with the desired size and copies over all the content from the previous allocation. Again, an allocator implementation can probably provide a more efficient implementation of this method, for example by growing/shrinking the allocation in-place if possible. [`alloc_zeroed`]: https://doc.rust-lang.org/alloc/alloc/trait.GlobalAlloc.html#method.alloc_zeroed From 4e5a7573961b84a5b72e80ec6868aad1cbbd6433 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Wed, 8 Jan 2020 12:39:11 +0100 Subject: [PATCH 08/85] It is now possible to define allocators in submodules (#715) --- .../second-edition/posts/10-heap-allocation/index.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/blog/content/second-edition/posts/10-heap-allocation/index.md b/blog/content/second-edition/posts/10-heap-allocation/index.md index 0c517ec1..85dcf28a 100644 --- a/blog/content/second-edition/posts/10-heap-allocation/index.md +++ b/blog/content/second-edition/posts/10-heap-allocation/index.md @@ -306,15 +306,13 @@ We now have a simple allocator, but we still have to tell the Rust compiler that The `#[global_allocator]` attribute tells the Rust compiler which allocator instance it should use as the global heap allocator. The attribute is only applicable to a `static` that implements the `GlobalAlloc` trait. Let's register an instance of our `Dummy` allocator as the global allocator: ```rust -// in src/lib.rs +// in src/allocator.rs #[global_allocator] static ALLOCATOR: allocator::Dummy = allocator::Dummy; ``` -Since the `Dummy` allocator is a [zero sized type], we don't need to specify any fields in the initialization expression. Note that the `#[global_allocator]` module [cannot be used in submodules][pr51335], so we need to put it into the `lib.rs`. - -[pr51335]: https://github.com/rust-lang/rust/pull/51335 +Since the `Dummy` allocator is a [zero sized type], we don't need to specify any fields in the initialization expression. When we now try to compile it, the first error should be gone. Let's fix the remaining second error: @@ -520,7 +518,7 @@ linked_list_allocator = "0.6.4" Then we can replace our dummy allocator with the allocator provided by the crate: ```rust -// in src/lib.rs +// in src/allocator.rs use linked_list_allocator::LockedHeap; @@ -547,7 +545,7 @@ pub fn init_heap( // new unsafe { - super::ALLOCATOR.lock().init(HEAP_START, HEAP_SIZE); + ALLOCATOR.lock().init(HEAP_START, HEAP_SIZE); } Ok(()) From 5937ec2e04665b02a185d842920b551d4e7a8b3e Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Wed, 8 Jan 2020 11:02:42 +0100 Subject: [PATCH 09/85] Reintroduce allocator designs post The post was split off the Heap Allocations post because it became too large. To keep the tree clean, it was then temporarily removed. This commit restores the post by reverting the removal commit. This reverts commit 029d77ef21ec54f3f16c4bbad260d6f46e7a413c. --- .../allocation-fragmentation.svg | 2 + .../posts/11-allocator-designs/index.md | 726 ++++++++++++++++++ .../linked-list-allocation.svg | 2 + .../linked-list-allocator-push.svg | 2 + .../linked-list-allocator-remove-region.svg | 2 + .../qemu-bump-allocator.png | Bin 0 -> 9526 bytes blog/templates/second-edition/index.html | 1 + 7 files changed, 735 insertions(+) create mode 100644 blog/content/second-edition/posts/11-allocator-designs/allocation-fragmentation.svg create mode 100644 blog/content/second-edition/posts/11-allocator-designs/index.md create mode 100644 blog/content/second-edition/posts/11-allocator-designs/linked-list-allocation.svg create mode 100644 blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-push.svg create mode 100644 blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-remove-region.svg create mode 100644 blog/content/second-edition/posts/11-allocator-designs/qemu-bump-allocator.png diff --git a/blog/content/second-edition/posts/11-allocator-designs/allocation-fragmentation.svg b/blog/content/second-edition/posts/11-allocator-designs/allocation-fragmentation.svg new file mode 100644 index 00000000..b9c74784 --- /dev/null +++ b/blog/content/second-edition/posts/11-allocator-designs/allocation-fragmentation.svg @@ -0,0 +1,2 @@ + +
allocated
allocated
next
next
time
time
heap end
heap end
heap start
heap start
1
1
2
2
3
3
4
4
5
5 \ No newline at end of file diff --git a/blog/content/second-edition/posts/11-allocator-designs/index.md b/blog/content/second-edition/posts/11-allocator-designs/index.md new file mode 100644 index 00000000..0ccefad1 --- /dev/null +++ b/blog/content/second-edition/posts/11-allocator-designs/index.md @@ -0,0 +1,726 @@ ++++ +title = "Allocator Designs" +weight = 11 +path = "allocator-designs" +date = 0000-01-01 ++++ + +TODO + + + +This blog is openly developed on [GitHub]. If you have any problems or questions, please open an issue there. You can also leave comments [at the bottom]. The complete source code for this post can be found in the [`post-10`][post branch] branch. + +[GitHub]: https://github.com/phil-opp/blog_os +[at the bottom]: #comments +[post branch]: https://github.com/phil-opp/blog_os/tree/post-10 + + + +TODO optional + +## Introduction + +TODO + +## Design Goals + +The responsibility of an allocator is to manage the available heap memory. It needs to return unused memory on `alloc` calls and keep track of memory freed by `dealloc` so that it can be reused again. Most importantly, it must never hand out memory that is already in use somewhere else because this would cause undefined behavior. + +Apart from correctness, there are many secondary design goals. For example, it should effectively utilize the available memory and keep [fragmentation] low. Furthermore, it should work well for concurrent applications and scale to any number of processors. For maximal performance, it could even optimize the memory layout with respect to the CPU caches to improve [cache locality] and avoid [false sharing]. + +[cache locality]: http://docs.cray.com/books/S-2315-50/html-S-2315-50/qmeblljm.html +[fragmentation]: https://en.wikipedia.org/wiki/Fragmentation_(computing) +[false sharing]: http://mechanical-sympathy.blogspot.de/2011/07/false-sharing.html + +These requirements can make good allocators very complex. For example, [jemalloc] has over 30.000 lines of code. This complexity often undesired in kernel code where a single bug can lead to severe security vulnerabilities. Fortunately, the allocation patterns of kernel code are often much simpler compared to userspace code, so that relatively simple allocator design often suffice. In the following we explain three possible kernel allocator designs and explain their advantages and drawbacks. + +[jemalloc]: http://jemalloc.net/ + +## Bump Allocator + +The most simple allocator design is a _bump allocator_. It allocates memory linearly and only keeps track of the number of allocated bytes and the number of allocations. It is only useful in very specific use cases because it has a severe limitation: it can only free all memory at once. + +The base type looks like this: + +```rust +// in src/allocator.rs + +pub struct BumpAllocator { + heap_start: usize, + heap_end: usize, + next: usize, + allocations: usize, +} + +impl BumpAllocator { + /// Creates a new bump allocator with the given heap bounds. + /// + /// This method is unsafe because the caller must ensure that the given + /// memory range is unused. + pub const unsafe fn new(heap_start: usize, heap_size: usize) -> Self { + BumpAllocator { + heap_start, + heap_end: heap_start + heap_size, + next: heap_start, + allocations: 0, + } + } +} +``` + +Instead of using the `HEAP_START` and `HEAP_SIZE` constants directly, we use separate `heap_start` and `heap_end` fields. This makes the type more flexible, for example it also works when we only want to assign a part of the heap region. The purpose of the `next` field is to always point to the first unused byte of the heap, i.e. the start address of the next allocation. The `allocations` field is a simple counter for the active allocations with the goal of resetting the allocator after the last allocation was freed. + +We provide a simple constructor function that creates a new `BumpAllocator`. It initializes the `heap_start` and `heap_end` fields using the given start address and size. The `allocations` counter is initialized with 0. The `next` field is set to `heap_start` since the whole heap should be unused at this point. Since this is something that the caller must guarantee, the function needs to be unsafe. Given an invalid memory range, the planned implementation of the `GlobalAlloc` trait would cause undefined behavior when it is used as global allocator. + +### A `Locked` Wrapper + +Implementing the [`GlobalAlloc`] trait directly for the `BumpAllocator` struct is not possible. The problem is that the `alloc` and `dealloc` methods of the trait only take an immutable `&self` reference, but we need to update the `next` and `allocations` fields for every allocation, which is only possible with an exclusive `&mut self` reference. The reason that the `GlobalAlloc` trait is specified this way is that the global allocator needs to be stored in an immutable `static` that only allows `&self` references. + +To be able to implement the trait for our `BumpAllocator` struct, we need to add synchronized [interior mutability] to get mutable field access through the `&self` reference. A type that adds the required synchronization and allows interior mutabilty is the [`spin::Mutex`] spinlock that we already used multiple times for our kernel, for example [for our VGA buffer writer][vga-mutex]. To use it, we create a `Locked` wrapper type: + +[interior mutability]: https://doc.rust-lang.org/book/ch15-05-interior-mutability.html +[`spin::Mutex`]: https://docs.rs/spin/0.5.0/spin/struct.Mutex.html +[vga-mutex]: ./second-edition/posts/03-vga-text-buffer/index.md#spinlocks + +```rust +// in src/allocator.rs + +pub struct Locked { + inner: spin::Mutex, +} + +impl Locked { + pub const fn new(inner: A) -> Self { + Locked { + inner: spin::Mutex::new(inner), + } + } +} +``` + +The type is a generic wrapper around a `spin::Mutex`. It imposes no restrictions on the wrapped type `A`, so it can be used to wrap all kinds of types, not just allocators. It provides a simple `new` constructor function that wraps a given value. + +### Implementing `GlobalAlloc` + +With the help of the `Locked` wrapper type we now can implement the `GlobalAlloc` trait for our bump allocator. The trick is to implement the trait not for the `BumpAllocator` directly, but for the wrapped `Locked` type. The implementation looks like this: + +```rust +// in src/allocator.rs + +unsafe impl GlobalAlloc for Locked { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + let mut bump = self.inner.lock(); + + let alloc_start = align_up(bump.next, layout.align()); + let alloc_end = alloc_start + layout.size(); + + if alloc_end > bump.heap_end { + null_mut() // out of memory + } else { + bump.next = alloc_end; + bump.allocations += 1; + alloc_start as *mut u8 + } + } + + unsafe fn dealloc(&self, _ptr: *mut u8, _layout: Layout) { + let mut bump = self.inner.lock(); + + bump.allocations -= 1; + if bump.allocations == 0 { + bump.next = bump.heap_start; + } + } +} +``` + +The first step for both `alloc` and `dealloc` is to call the [`Mutex::lock`] method to get a mutable reference to the wrapped allocator type. The instance remains locked until the end of the method, so that no data race can occur in multithreaded contexts (we will add threading support soon). + +[`Mutex::lock`]: https://docs.rs/spin/0.5.0/spin/struct.Mutex.html#method.lock + +The `alloc` implementation first performs the required alignment on the `next` address, as specified by the given [`Layout`]. This yields the start address of the allocation. The code for the `align_up` function is shown below. Next, we add the requested allocation size to `alloc_start` to get the end address of the allocation. If it is larger than the end address of the heap, we return a null pointer since there is not enough memory available. Otherwise, we update the `next` address (the next allocation should start after the current allocation), increase the `allocations` counter by 1, and return the `alloc_start` address converted to a `*mut u8` pointer. + +The `dealloc` function ignores the given pointer and `Layout` arguments. Instead, it just decreases the `allocations` counter. If the counter reaches `0` again, it means that all allocations were freed again. In this case, it resets the `next` address to the `heap_start` address to make the complete heap memory available again. + +The last remaining part of the implementation is the `align_up` function, which looks like this: + +```rust +// in src/allocator.rs + +fn align_up(addr: usize, align: usize) -> usize { + let remainder = addr % align; + if remainder == 0 { + addr // addr already aligned + } else { + addr - remainder + align + } +} +``` + +The function first computes the [remainder] of the division of `addr` by `align`. If the remainder is `0`, the address is already aligned with the given alignment. Otherwise, we align the address by subtracting the remainder (so that the new remainder is 0) and then adding the alignment (so that the address does not become smaller than the original address). + +[remainder]: https://en.wikipedia.org/wiki/Euclidean_division + +### Using It + +To use the bump allocator instead of the dummy allocator, we need to update the `ALLOCATOR` static in `lib.rs`: + +```rust +// in src/lib.rs + +use allocator::{Locked, BumpAllocator, HEAP_START, HEAP_SIZE}; + +#[global_allocator] +static ALLOCATOR: Locked = + Locked::new(BumpAllocator::new(HEAP_START, HEAP_SIZE)); +``` + +Here it becomes important that we declared both the `Locked::new` and the `BumpAllocator::new` as [`const` functions]. If they were normal functions, a compilation error would occur because the initialization expression of a `static` must evaluable at compile time. + +[`const` functions]: https://doc.rust-lang.org/reference/items/functions.html#const-functions + +Now we can use `Box` and `Vec` without runtime errors: + +```rust +// in src/main.rs + +use alloc::{boxed::Box, vec::Vec, collections::BTreeMap}; + +fn kernel_main(boot_info: &'static BootInfo) -> ! { + // […] initialize interrupts, mapper, frame_allocator, heap + + // allocate a number on the heap + let heap_value = Box::new(41); + println!("heap_value at {:p}", heap_value); + + // create a dynamically sized vector + let mut vec = Vec::new(); + for i in 0..500 { + vec.push(i); + } + println!("vec at {:p}", vec.as_slice()); + + // try to create one million boxes + for _ in 0..1_000_000 { + let _ = Box::new(1); + } + + // […] call `test_main` in test context + println!("It did not crash!"); + blog_os::hlt_loop(); +} +``` + +This code example only uses the `Box` and `Vec` types, but there are many more allocation and collection types in the `alloc` crate that we can now all use in our kernel, including: + +- the reference counted pointers [`Rc`] and [`Arc`] +- the owned string type [`String`] and the [`format!`] macro +- [`LinkedList`] +- the growable ring buffer [`VecDeque`] +- [`BinaryHeap`] +- [`BTreeMap`] and [`BTreeSet`] + +[`Rc`]: https://doc.rust-lang.org/alloc/rc/ +[`Arc`]: https://doc.rust-lang.org/alloc/arc/ +[`String`]: https://doc.rust-lang.org/collections/string/struct.String.html +[`format!`]: https://doc.rust-lang.org/alloc/macro.format.html +[`LinkedList`]: https://doc.rust-lang.org/collections/linked_list/struct.LinkedList.html +[`VecDeque`]: https://doc.rust-lang.org/collections/vec_deque/struct.VecDeque.html +[`BinaryHeap`]: https://doc.rust-lang.org/collections/binary_heap/struct.BinaryHeap.html +[`BTreeMap`]: https://doc.rust-lang.org/collections/btree_map/struct.BTreeMap.html +[`BTreeSet`]: https://doc.rust-lang.org/collections/btree_set/struct.BTreeSet.html + +When we run our project now, we see the following: + +![QEMU printing ` +heap_value at 0x444444440000 +vec at 0x4444444408000 +panicked at 'allocation error: Layout { size_: 4, align_: 4 }', src/lib.rs:91:5 +](qemu-bump-allocator.png) + +As expected, we see that the `Box` and `Vec` values live on the heap, as indicated by the pointer starting with `0x_4444_4444`. The reason that the vector starts at offset `0x800` is not that the boxed value is `0x800` bytes large, but the [reallocations] that occur when the vector needs to increase its capacity. For example, when the vector's capacity is 32 and we try to add the next element, the vector allocates a new backing array with capacity 64 behind the scenes and copies all elements over. Then it frees the old allocation, which in our case is equivalent to leaking it since our bump allocator doesn't reuse freed memory. + +[reallocations]: https://doc.rust-lang.org/alloc/vec/struct.Vec.html#capacity-and-reallocation + +While the basic `Box` and `Vec` examples work as expected, our loop that tries to create one million boxes causes a panic. The reason is that the bump allocator never reuses freed memory, so that for each created `Box` a few bytes are leaked. This makes the bump allocator unsuitable for many applications in practice, apart from some very specific use cases. + +### When to use a Bump Allocator + +The big advantage of bump allocation is that it's very fast. Compared to other allocator designs (see below) that need to actively look for a fitting memory block and perform various bookkeeping tasks on `alloc` and `dealloc`, a bump allocator can be optimized to just a few assembly instructions. This makes bump allocators useful for optimizing the allocation performance, for example when creating a [virtual DOM library]. + +[virtual DOM library]: https://hacks.mozilla.org/2019/03/fast-bump-allocated-virtual-doms-with-rust-and-wasm/ + +While a bump allocator is seldom used as the global allocator, the principle of bump allocation is often applied in form of [arena allocation], which basically batches individual allocations together to improve performance. An example for an arena allocator for Rust is the [`toolshed`] crate. + +[arena allocation]: https://mgravell.github.io/Pipelines.Sockets.Unofficial/docs/arenas.html +[`toolshed`]: https://docs.rs/toolshed/0.8.1/toolshed/index.html + +### Reusing Freed Memory? + +The main limitation of a bump allocator is that it never reuses deallocated memory. The question is: Can we extend our bump allocator somehow to remove this limitation? + +As we learned at the beginning of this post, allocations can live arbitarily long and can be freed in an arbitrary order. This means that we need to keep track of a potentially unbounded number of non-continuous, unused memory regions, as illustrated by the following example: + +![](allocation-fragmentation.svg) + +The graphic shows the heap over the course of time. At the beginning, the complete heap is unused and the `next` address is equal to `heap_start` (line 1). Then the first allocation occurs (line 2). In line 3, a second memory block is allocated and the first allocation is freed. Many more allocations are added in line 4. Half of them are very short-lived and already get freed in line 5, where also another new allocation is added. + +Line 5 shows the fundamental problem: We have five unused memory regions with different sizes in total, but the `next` pointer can only point to the beginning of the last region. While we could store the start addresses and sizes of the other unused memory regions in an array of size 4 for this example, this isn't a general solution since we could easily create an example with 8, 16, or 1000 unused memory regions. + +Normally when we have a potentially unbounded number of items, we can just use a heap allocated collection. This isn't really possible in our case, since the heap allocator can't depend on itself (it would cause endless recursion or deadlocks). So we need to find a different solution. + +## LinkedList Allocator + +A common trick to keep track of an arbitrary number of free memory areas is to use these areas itself as backing storage. This utilizes the fact that the regions are still mapped to a virtual address and backed by a physical frame, but the stored information is not needed anymore. By storing the information about the freed region in the region itself, we can keep track of an unbounded number of freed regions without needing additional memory. + +The most common implementation approach is to construct a single linked list in the freed memory, with each node being a freed memory region: + +![](linked-list-allocation.svg) + +Each list node contains two fields: The size of the memory region and a pointer to the next unused memory region. With this approach, we only need a pointer to the first unused region (called `head`), independent of the number of memory regions. + +In the following, we will create a simple `LinkedListAllocator` type that uses the above approach for keeping track of freed memory regions. Since the implementation is a bit longer, we will start with a simple placeholder type before we start to implement the `alloc` and `dealloc` operations. + +### The Allocator Type + +We start by creating a private `ListNode` struct: + +```rust +// in src/allocator.rs + +struct ListNode { + size: usize, + next: Option<&'static mut ListNode>, +} + +impl ListNode { + const fn new(size: usize) -> Self { + ListNode { + size, + next: None, + } + } + + fn start_addr(&self) -> usize { + self as *const Self as usize + } + + fn end_addr(&self) -> usize { + self.start_addr() + self.size + } +} +``` + +Like in the graphic, a list node has a `size` field and an optional pointer to the next node. The type has a simple constructor function and methods to calculate the start and end addresses of the represented region. + +With the `ListNode` struct as building block, we can now create the `LinkedListAllocator` struct: + +```rust +// in src/allocator.rs + +pub struct LinkedListAllocator { + head: ListNode, +} + +impl LinkedListAllocator { + pub const fn new() -> Self { + Self { + head: ListNode::new(0), + } + } + + /// Initialize the allocator with the given heap bounds. + /// + /// This function is unsafe because the caller must guarantee that the given + /// heap bounds are valid and that the heap is unused. This method must be + /// called only once. + pub unsafe fn init(&mut self, heap_start: usize, heap_size: usize) { + self.add_free_region(heap_start, heap_size); + } + + /// Adds the given memory region to the front of the list. + unsafe fn add_free_region(&mut self, addr: usize, size: usize) { + unimplemented!(); + } +} +``` + +The struct contains a `head` node that points to the first heap region. We are only interested in the value of the `next` pointer, so we set the `size` to 0 in the `new` function. Making `head` a `ListNode` instead of just a `&'static mut ListNode` has the advantage that the implementation of the `alloc` method will be simpler. + +In contrast to the bump allocator, the `new` function doesn't initialize the allocator with the heap bounds. The reason is that the initialization requires to write a node to the heap memory, which can only happen at runtime. The `new` function, however, needs to be a [`const` function] that can be evaluated at compile time, because it will be used for initializing the `ALLOCATOR` static. To work around this, we provide a separate `init` method that can be called at runtime. + +[`const` function]: https://doc.rust-lang.org/reference/items/functions.html#const-functions + +The `init` method uses a `add_free_region` method, whose implementation will be shown in a moment. For now, we use the [`unimplemented!`] macro to provide a placeholder implementation that always panics. + +[`unimplemented!`]: https://doc.rust-lang.org/core/macro.unimplemented.html + +Our first goal is to set a prototype of the `LinkedListAllocator` as the global allocator. In order to be able to do that, we need to provide a placeholder implementation of the `GlobalAlloc` trait: + +```rust +// in src/allocator.rs + +unsafe impl GlobalAlloc for Locked { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + unimplemented!(); + } + + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + unimplemented!(); + } +} +``` + +Like with the bump allocator, we don't implement the trait directly for the `LinkedListAllocator`, but only for a wrapped `Locked`. The [`Locked` wrapper] adds interior mutability through a spinlock, which allows us to modify the allocator instance even though the `alloc` and `dealloc` methods only take `&self` references. Instead of providing an implementation, we use the [`unimplemented!`] macro again to get a minimal prototype. + +[`Locked` wrapper]: ./second-edition/posts/10-heap-allocation/index.md#a-locked-wrapper + +With this placeholder implementation, we can now change the global allocator to a `LinkedListAllocator`: + +```rust +// in src/lib.rs + +use allocator::{Locked, LinkedListAllocator}; + +#[global_allocator] +static ALLOCATOR: Locked = + Locked::new(LinkedListAllocator::new()); +``` + +Since the `new` method creates an empty allocator, we also need to update our `allocator::init` function to call `LinkedListAllocator::init` with the heap bounds: + +```rust +// in src/allocator.rs + +pub fn init_heap( + mapper: &mut impl Mapper, + frame_allocator: &mut impl FrameAllocator, +) -> Result<(), MapToError> { + // […] map all heap pages + + // new + unsafe { + super::ALLOCATOR.inner.lock().init(HEAP_START, HEAP_SIZE); + } + + Ok(()) +} +``` + +It's important to call the `init` function after the mapping of the heap pages, because the function will already write to the heap (once we'll properly implement it). The `unsafe` block is safe here because we just mapped the heap region to unused frames, so that the passed heap region is valid. + +When we run our code now, it will of course panic since it runs into the `unimplemented!` in `add_free_region`. Let's fix that by providing a proper implementation for that method. + +### The `add_free_region` Method + +The `add_free_region` method provides the fundamental _push_ operation on the linked list. We currently only call this method from `init`, but it will also be the central method in our `dealloc` implementation. Remember, the `dealloc` method is called when an allocated memory region is freed again. To keep track of this freed memory region, we want to push it to the linked list. + +The implementation of the `add_free_region` method looks like this: + +```rust +// in src/allocator.rs + +impl LinkedListAllocator { + /// Adds the given memory region to the front of the list. + unsafe fn add_free_region(&mut self, addr: usize, size: usize) { + // ensure that the freed region is capable of holding ListNode + assert!(align_up(addr, mem::align_of::()) == addr); + assert!(size >= mem::size_of::()); + + // create a new list node and append it at the start of the list + let mut node = ListNode::new(size); + node.next = self.head.next.take(); + let node_ptr = addr as *mut ListNode; + node_ptr.write(node); + self.head.next = Some(&mut *node_ptr) + } +} +``` + +The method takes a memory region represented by an address and size as argument and adds it to the front of the list. First, it ensures that the given region has the neccessary size and alignment for storing a `ListNode`. Then it creates the node and inserts it to the list through the following steps: + +![](linked-list-allocator-push.svg) + +Step 0 shows the state of the heap before `add_free_region` is called. In step 1, the method is called with the memory region marked as `freed` in the graphic. After the initial checks, the method creates a new `node` on its stack with the size of the freed region. It then uses the [`Option::take`] method to set the `next` pointer of the node to the current `head` pointer, thereby resetting the `head` pointer to `None`. + +[`Option::take`]: https://doc.rust-lang.org/core/option/enum.Option.html#method.take + +In step 2, the method writes the newly created `node` to the beginning of the freed memory region through the [`write`] method. It then points the `head` pointer to the new node. The resulting pointer structure looks a bit chaotic because the freed region is always inserted at the beginning of the list, but if we follow the pointers we see that each free region is still reachable from the `head` pointer. + +[`write`]: https://doc.rust-lang.org/std/primitive.pointer.html#method.write + +### The `find_region` Method + +The second fundamental operation on a linked list is finding an entry and removing it from the list. This is the central operation needed for implementing the `alloc` method. We implement the operation as a `find_region` method in the following way: + +```rust +// in src/allocator.rs + +impl LinkedListAllocator { + /// Looks for a free region with the given size and alignment and removes + /// it from the list. + /// + /// Returns a tuple of the list node and the start address of the allocation. + fn find_region(&mut self, size: usize, align: usize) + -> Option<(&'static mut ListNode, usize)> + { + // reference to current list node, updated for each iteration + let mut current = &mut self.head; + // look for a large enough memory region in linked list + while let Some(ref mut region) = current.next { + if let Ok(alloc_start) = Self::alloc_from_region(®ion, size, align) { + // region suitable for allocation -> remove node from list + let next = region.next.take(); + let ret = Some((current.next.take().unwrap(), alloc_start)); + current.next = next; + return ret; + } else { + // region not suitable -> continue with next region + current = current.next.as_mut().unwrap(); + } + } + + // no suitable region found + None + } +} +``` + +The method uses a `current` variable and a [`while let` loop] to iterate over the list elements. At the beginning, `current` is set to the (dummy) `head` node. On each iteration, it is then updated to to the `next` field of the current node (in the `else` block). If the region is suitable for an allocation with the given size and alignment, the region is removed from the list and returned together with the `alloc_start` address. + +[`while let` loop]: https://doc.rust-lang.org/reference/expressions/loop-expr.html#predicate-pattern-loops + +When the `current.next` pointer becomes `None`, the loop exits. This means that we iterated over the whole list but found no region that is suitable for an allocation. In that case, we return `None`. The check whether a region is suitable is done by a `alloc_from_region` function, whose implementation will be shown in a moment. + +Let's take a more detailed look at how a suitable region is removed from the list: + +![](linked-list-allocator-remove-region.svg) + +Step 0 shows the situation before any pointer adjustments. The `region` and `current` regions and the `region.next` and `current.next` pointers are marked in the graphic. In step 1, both the `region.next` and `current.next` pointers are reset to `None` by using the [`Option::take`] method. The original pointers are stored in local variables called `next` and `ret`. + +In step 2, the `current.next` pointer is set to the local `next` pointer, which is the original `region.next` pointer. The effect is that `current` now directly points to the region after `region`, so that `region` is no longer element of the linked list. The function then returns the pointer to `region` stored in the local `ret` variable. + +### The `alloc_from_region` Function + +The `alloc_from_region` function returns whether a region is suitable for an allocation with given size and alignment. It is defined like this: + +```rust +// in src/allocator.rs + +impl LinkedListAllocator { + /// Try to use the given region for an allocation with given size and alignment. + /// + /// Returns the allocation start address on success. + fn alloc_from_region(region: &ListNode, size: usize, align: usize) + -> Result + { + let alloc_start = align_up(region.start_addr(), align); + let alloc_end = alloc_start + size; + + if alloc_end > region.end_addr() { + // region too small + return Err(()); + } + + let excess_size = region.end_addr() - alloc_end; + if excess_size > 0 && excess_size < mem::size_of::() { + // rest of region too small to hold a ListNode (required because the + // allocation splits the region in a used and a free part) + return Err(()); + } + + // region suitable for allocation + Ok(alloc_start) + } +} +``` + +First, the function calculates the start and end address of a potential allocation, using the `align_up` function we defined earlier. If the end address is behind the end address of the region, the allocation doesn't fit in the region and we return an error. + +The function performs a less obvious check after that. This check is neccessary because most of the time an allocation does not fit a suitable region perfectly, so that a part of the region remains usable after the allocation. This part of the region must store its own `ListNode` after the allocation, so it must be large enough to do so. The check verifies exactly that: either the allocation fits perfectly (`excess_size == 0`) or the excess size is large enough to store a `ListNode`. + +### Implementing `GlobalAlloc` + +With the fundamental operations provided by the `add_free_region` and `find_region` methods, we can now finally implement the `GlobalAlloc` trait: + +```rust +// in src/allocator.rs + +unsafe impl GlobalAlloc for Locked { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + // perform layout adjustments + let (size, align) = LinkedListAllocator::size_align(layout); + let mut allocator = self.inner.lock(); + + if let Some((region, alloc_start)) = allocator.find_region(size, align) { + let alloc_end = alloc_start + size; + let excess_size = region.end_addr() - alloc_end; + if excess_size > 0 { + allocator.add_free_region(alloc_end, excess_size); + } + alloc_start as *mut u8 + } else { + null_mut() + } + } + + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + // perform layout adjustments + let (size, _) = LinkedListAllocator::size_align(layout); + + self.inner.lock().add_free_region(ptr as usize, size) + } +} +``` + +Let's start with the `dealloc` method because it is simpler: First, it performs some layout adjustments, which we will explain in a moment, and retrieves a `&mut LinkedListAllocator` reference by calling the [`Mutex::lock`] function on the [`Locked` wrapper]. Then it calls the `add_free_region` function to add the deallocated region to the free list. + +The `alloc` method is a bit more complex. It starts with the same layout adjustments and also calls the [`Mutex::lock`] function to receive a mutable allocator reference. Then it uses the `find_region` method to find a suitable memory region for the allocation and remove it from the list. If this doesn't succeed and `None` is returned, it returns `null_mut` to signal an error as there is no suitable memory region. + +In the success case, the `find_region` method returns a tuple of the suitable region (no longer in the list) and the start address of the allocation. Using `alloc_start`, the allocation size, and the end address of the region, it calculates the end address of the allocation and the excess size again. If the excess size is not null, it calls `add_free_region` to add the excess size of the memory region back to the free list. Finally, it returns the `alloc_start` address casted as a `*mut u8` pointer. + +### Layout Adjustments + +```rust +// in src/allocator.rs + +impl LinkedListAllocator { + /// Adjust the given layout so that the resulting allocated memory + /// region is also capable of storing a `ListNode`. + /// + /// Returns the adjusted size and alignment as a (size, align) tuple. + fn size_align(layout: Layout) -> (usize, usize) { + let layout = layout.align_to(mem::align_of::()) + .and_then(|l| l.pad_to_align()) + .expect("adjusting alignment failed"); + let size = layout.size().max(mem::size_of::()); + (size, layout.align()) + } +} +``` + + + + + + + + + + +##### Allocation +In order to allocate a block of memory, we need to find a hole that satisfies the size and alignment requirements. If the found hole is larger than required, we split it into two smaller holes. For example, when we allocate a 24 byte block right after initialization, we split the single hole into a hole of size 24 and a hole with the remaining size: + +![split hole](split-hole.svg) + +Then we use the new 24 byte hole to perform the allocation: + +![24 bytes allocated](allocate.svg) + +To find a suitable hole, we can use several search strategies: + +- **best fit**: Search the whole list and choose the _smallest_ hole that satisfies the requirements. +- **worst fit**: Search the whole list and choose the _largest_ hole that satisfies the requirements. +- **first fit**: Search the list from the beginning and choose the _first_ hole that satisfies the requirements. + +Each strategy has its advantages and disadvantages. Best fit uses the smallest hole possible and leaves larger holes for large allocations. But splitting the smallest hole might create a tiny hole, which is too small for most allocations. In contrast, the worst fit strategy always chooses the largest hole. Thus, it does not create tiny holes, but it consumes the large block, which might be required for large allocations. + +For our use case, the best fit strategy is better than worst fit. The reason is that we have a minimal hole size of 16 bytes, since each hole needs to be able to store a size (8 bytes) and a pointer to the next hole (8 bytes). Thus, even the best fit strategy leads to holes of usable size. Furthermore, we will need to allocate very large blocks occasionally (e.g. for [DMA] buffers). + +[DMA]: https://en.wikipedia.org/wiki/Direct_memory_access + +However, both best fit and worst fit have a significant problem: They need to scan the whole list for each allocation in order to find the optimal block. This leads to long allocation times if the list is long. The first fit strategy does not have this problem, as it returns as soon as it finds a suitable hole. It is fairly fast for small allocations and might only need to scan the whole list for large allocations. + +### Deallocation +To deallocate a block of memory, we can just insert its corresponding hole somewhere into the list. However, we need to merge adjacent holes. Otherwise, we are unable to reuse the freed memory for larger allocations. For example: + +![deallocate memory, which leads to adjacent holes](deallocate.svg) + +In order to use these adjacent holes for a large allocation, we need to merge them to a single large hole first: + +![merge adjacent holes and allocate large block](merge-holes-and-allocate.svg) + +The easiest way to ensure that adjacent holes are always merged, is to keep the hole list sorted by address. Thus, we only need to check the predecessor and the successor in the list when we free a memory block. If they are adjacent to the freed block, we merge the corresponding holes. Else, we insert the freed block as a new hole at the correct position. + +## Implementation +The detailed implementation would go beyond the scope of this post, since it contains several hidden difficulties. For example: + +- Several merge cases: Merge with the previous hole, merge with the next hole, merge with both holes. +- We need to satisfy the alignment requirements, which requires additional splitting logic. +- The minimal hole size of 16 bytes: We must not create smaller holes when splitting a hole. + +I created the [linked_list_allocator] crate to handle all of these cases. It consists of a [Heap struct] that provides an `allocate_first_fit` and a `deallocate` method. It also contains a [LockedHeap] type that wraps `Heap` into spinlock so that it's usable as a static system allocator. If you are interested in the implementation details, check out the [source code][linked_list_allocator source]. + +[linked_list_allocator]: https://docs.rs/crate/linked_list_allocator/0.4.1 +[Heap struct]: https://docs.rs/linked_list_allocator/0.4.1/linked_list_allocator/struct.Heap.html +[LockedHeap]: https://docs.rs/linked_list_allocator/0.4.1/linked_list_allocator/struct.LockedHeap.html +[linked_list_allocator source]: https://github.com/phil-opp/linked-list-allocator + +We need to add the extern crate to our `Cargo.toml` and our `lib.rs`: + +``` shell +> cargo add linked_list_allocator +``` + +```rust +// in src/lib.rs +extern crate linked_list_allocator; +``` + +Now we can change our global allocator: + +```rust +use linked_list_allocator::LockedHeap; + +#[global_allocator] +static HEAP_ALLOCATOR: LockedHeap = LockedHeap::empty(); +``` + +We can't initialize the linked list allocator statically, since it needs to initialize the first hole (like described [above](#initialization)). This can't be done at compile time, so the function can't be a `const` function. Therefore we can only create an empty heap and initialize it later at runtime. For that, we add the following lines to our `rust_main` function: + +```rust +// in src/lib.rs + +#[no_mangle] +pub extern "C" fn rust_main(multiboot_information_address: usize) { + […] + + // set up guard page and map the heap pages + memory::init(boot_info); + + // initialize the heap allocator + unsafe { + HEAP_ALLOCATOR.lock().init(HEAP_START, HEAP_START + HEAP_SIZE); + } + […] +} +``` + +It is important that we initialize the heap _after_ mapping the heap pages, since the init function writes to the heap memory (the first hole). + +Our kernel uses the new allocator now, so we can deallocate memory without leaking it. The example from above should work now without causing an OOM situation: + +```rust +// in rust_main in src/lib.rs + +for i in 0..10000 { + format!("Some String"); +} +``` + +## Performance +The linked list based approach has some performance problems. Each allocation or deallocation might need to scan the complete list of holes in the worst case. However, I think it's good enough for now, since our heap will stay relatively small for the near future. When our allocator becomes a performance problem eventually, we can just replace it with a faster alternative. + +## Summary +Now we're able to use heap storage in our kernel without leaking memory. This allows us to effectively process dynamic data such as user supplied strings in the future. We can also use `Rc` and `Arc` to create types with shared ownership. And we have access to various data structures such as `Vec` or `Linked List`, which will make our lives much easier. We even have some well tested and optimized [binary heap] and [B-tree] implementations! + +[binary heap]:https://en.wikipedia.org/wiki/Binary_heap +[B-tree]: https://en.wikipedia.org/wiki/B-tree + + +--- + +TODO: update date + +--- diff --git a/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocation.svg b/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocation.svg new file mode 100644 index 00000000..a24ac160 --- /dev/null +++ b/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocation.svg @@ -0,0 +1,2 @@ + +
heap end
heap end
heap start
heap start
size
size
next pointer
<div>next pointer</div>
head
head \ No newline at end of file diff --git a/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-push.svg b/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-push.svg new file mode 100644 index 00000000..734bfaaf --- /dev/null +++ b/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-push.svg @@ -0,0 +1,2 @@ + +
Step
[Not supported by viewer]
1
[Not supported by viewer]
2
[Not supported by viewer]
0
[Not supported by viewer]
heap end
heap end
heap start
heap start
size
size
next pointer
<div>next pointer</div>
head
head
heap end
heap end
heap start
heap start
size
size
next pointer
<div>next pointer</div>
head
head
node:
node:
freed
freed
heap end
heap end
heap start
heap start
size
size
next pointer
<div>next pointer</div>
head
head
Operations
<b>Operations</b> \ No newline at end of file diff --git a/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-remove-region.svg b/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-remove-region.svg new file mode 100644 index 00000000..229130b9 --- /dev/null +++ b/blog/content/second-edition/posts/11-allocator-designs/linked-list-allocator-remove-region.svg @@ -0,0 +1,2 @@ + +
Step
[Not supported by viewer]
1
[Not supported by viewer]
2
[Not supported by viewer]
0
[Not supported by viewer]
heap end
heap end
heap start
heap start
size
size
next pointer
<div>next pointer</div>
head
head
Operations
<b>Operations</b>
region
region
current
<div>current</div>
current.next
current.next
region.next
region.next
heap end
heap end
heap start
heap start
size
size
next pointer
<div>next pointer</div>
region
region
current
<div>current</div>
head
head
ret
ret
next
[Not supported by viewer]
heap end
heap end
heap start
heap start
size
size
next pointer
<div>next pointer</div>
region
region
current
<div>current</div>
head
head
ret
ret \ No newline at end of file diff --git a/blog/content/second-edition/posts/11-allocator-designs/qemu-bump-allocator.png b/blog/content/second-edition/posts/11-allocator-designs/qemu-bump-allocator.png new file mode 100644 index 0000000000000000000000000000000000000000..71397c2970bb068afcf9a424d6094c46da573e73 GIT binary patch literal 9526 zcmeAS@N?(olHy`uVBq!ia0y~yV7kP>z_^x!je&u|`rBk>1_lO}VkgfK4h{~E8jh3> z1_lO+64!{5;QX|b^2DN4hTO!GRNdm_qSVy9;*9)~6VpC;F)%1Fc)B=-RLpsMw{k-2 z^+fq+_iLl`jhhcTG06mo_&5l%<|$~t+rZ$()~K-LY1iW<74M>@LjMvi`{ta`3V$>y z`Kaddkad&Rt_TVf+n5wNZOu_J-P0lmW-x_&PC0Y*jwh#xroe568MV89f31Fh@6X3w z)m2qhm9@LlmsLOey-Mmu<@-Cu=VR|x?=Sy#r+9VFgL~We@4We`D?D( zmtADmX8d)-J1_eCnWft*tyXT++xYgAxah7eU(QY2|E05Y+Wkd+S-Y8p_2a`THhl}< z9aHy~NnC&9=cm{I7auo`32_h1HoGr7`FiNvlH2+>KTOiyzhYf(T-5FBn~uKDdK8xP zH*C}2Ptkft4+XVq1jA?Bn+C@2j;?FI_U2QCn&+&m|DQa%Ulw)zR(yHHY1PyDpPsOk z=EPTZ3!U|gz8CZ9{HJ+Kp4x6dy>12f>FJ?c-j&=ajFr~6udKDs|EKmm@8C0YX_@nL zb@%^kjGgQIT29Aivw3U|`}!@M{pmkHz3$H4aDb&WCnn{Qc^ju$@oS;my8o{PpI6yw z5wQMW(fix?@7$?)eDUop1_p-fYbUGwuWfs&CE5OBv3vfGU2B;b8kiqkS$X;A4|Ti9 zt?SC185kZYta?}bd-iPUJd2{Ow;34@FkUn9UA4+9_WH`%R$o_mPOhrVD4M*)D|Btl z`{1RSX8YcyKmE8YGwk)=fQr{uZHF(qe!`MvzA`HReAr@%ggDPB6AfO z7(_B=srg<`3Aw(~eK8N);VX+T7Olv;v}zi2_U^fP+vkR?{rKaN+UkWBOXq2YUNXs? zWz~Ezqp9lH)77iIvKRa8zOydT)+;uikv_ zeP7plzdE@~!A7cg)0Qn$_Lp4WxaZ}!*+s`M9^U!&g3mnL?Gy6nSk1Pt`Q&o%|8f7N z=2yLUJx(nXTDH3S{=a9>K0I{ZeXDWtMUC$6zh}?j*^e7tS8Uz_ggx3$;8;?d_5>gi@#{wU|!>%6`@Hq&2b)!qzymS;1^GTxh^ z;q{5E4!Pb-@ls1}zkT-a_q)@+=CWrqLefoE1z$hBC2H=L+iSPnUOUI4P)q*b33nq2 zo=dOGrdKaHe%Nf*vXco09=q?Jx_!ScKX32XqbpzczOMZDQ~cA${#qOF>{lyabK728 z{LnyuUf%QN`=3v>EX&aHwe@;sg<-;Q-$s7 z|NjhsZk=a6=h~x_e{N2i<1D|3rTO_y1>}KTk6bHGMh%<@2repPzq9USF^G|Cjpz=X?Gi znPb4q5EI^Z+Szi~+`M}~&#_;67b2;{dufU1X0;n(YqdgGl~mvDY4eq<{n9Mi$M&*h zSLW7LPX%m!RxJ;%(nqyK3(I zz5m~xF=@Wy-93BT{;RXkIbA(_|HobZCpQx($8Db;mi^oF-}$-L`*$yt?e6Zjto!p* z#>!;+%$YMwsvlpt^Zxtg_uoI?`KFJjVWc@%r8B>wny@x7z< z+I5oGT-JB@Dx=c#yFab7|L*?tYx*STyGta$ZT_;U{Q3Gl9|KlpO<8PyZt2SDr*88x z6!2!xQaiiiWJ0Fq_uBJ&GiN1=J!@ZWdsW*$G<5Ahp7!RF-Fx@E&CNg6ZujM3Nz~(w z=SpMKSKdEwz3=A{W)azmFM!?vf}r9^CQVQL{rT_vzT5tM)4yMTeqP!!?%sK)-2WZbf4cZ@*)8pOjazrm_-;3z zwZ7=fGU-cMv#zX-{#;xC(f8-u`F}&)=K8UH4z9fX{r>#h(Y~{mOpFd7vZ5dZe_sTRJ-<%b4TAV8^ zbn3x`88c_zl-d0^VqW#PimJD9Zq0#tyWeh;nKe6loqFY)jr}PWGI=}RvCI9~$HUfq zwW?3ve%{J&A#H~f<@V+3>E}H&wkXx|EUKx~Jg?7F^!=>3TVl=ICpCPfw?TP#aqH%t zZ!cW@vs~YD%K!I2mwwdf=G0+-IrsgJqequs@tyqg<;*`@<$pfB^SA5$iJzaJPf}T5 z_q6{Pv`&r)KAScIC1Du(y6w7p3M>U?s65!y)Vc8@~V4WD}Szduk+4@b5(qQ z&Humk&!gk<)BnHxcl~+e_CHboUX)+gi%!2FpL$30?x`;mUoP={Wo21(_+iN6yT*H0SoB7+j zd|qybf`BzKDL*D9>z_Py$jNQ-#ZQmFX6b+G;@*4mG2_bvlKg^} zWpB3W==HshjQrBbzVG{_D!J-6izRv3PG5d`wv=KkqEzf$Y}+t=~aZZA8Y+V=eXlYio8pOnvw z^~gVDv;9u~-p^mmSGm6|wSHM*gu&1L+Sue`o>&&gwkRHT zd}`RMz`6Oe*9Fa=?;m{Kb5i);9~pIz?tXtN|MxckbN%~;Ci{Q(UjDhU{eMkd>|D>< zZz+?OtaGn?dB^8f;9O7R*pC+3c|Mb=LCTfR5%`CEs}D#5`s{VwUu zIsfNVa`wK-DFu0U-Y0)7IyP1PmH4Nf$L;2-hqalTKmR;+^V^bDR;7JUKEzC2R&V#b z@4sW+@9+PT{=M7F|70J(>`B}Fy^81S-(39j&)B%~tij4>e5va^&#jZqzqe+~!kY>5 z(zh5HV(e$BlwMW|4Qo#b37vZX_g??JU1x6J`}XYWy;mz=mc4qK6Rywx^kBk(Nc zr7hhrCLitM9-6e*C=|FKfPijz5uL z;kEjz-}1|vJTpRP6{i<(^IWCuUaaEl*V7+5Y01jjKC-rxR+?B==h>Y-CA@g~=g#=Q z(|?|LbV}l-*4>qHyVeGZ8BN}KQ|s)Xnsezhj?4e_*jIV~`*Z#OYi}2?_}u&F)Z+Nl z8-Dp4?Vk`*Ts(92tT~x&l036jF3+4R$q*GSegVcJH|1AKLpqZii9v zv0w9Eh?d6c`3uOaU3J;yYu3;8e{Kfz+Ax?&&vgV(fnZ)PFkv?<4(Xck`BBf1Q$+wkkt5bL%C$&tGfaR?NM&@%pn~{gs<7 zA}Z^q9Q~Jb@58a(mwfH(KV4PVQ+dPfp7V40xdr>DBwk)}(&t&)Did$-$W>7;C-s`1p9gygmDm#>B+@t*!f41w{Cvfa@&X7-X~=oKXb;XPj0W_d7IBARd@61-kR6h z{{MFW@9rg@&%~cye*HD*x^LdzuiL)-?zSv`cBb$8#E_Sz^G>wO{mhd4wLWg*`~MgJ z8~Oj49DiP3F5WB3{Zm)@U-O?2&Ce{We7Di&*UZ)}n{Be)e@DFgz1n`x?rZmEEVY|w z|Kn`^eEl>lmU-!L!TKL}+_fOl`oeuwe_H*&iZ@b^0SsUzK`DeS`jPqCLo;h-XkKw_u zz);WSmrs5^Z$EwK&Y3-Ki$g-^hOPG7eK%_Ffe(jzqhCFp_rz+hpZf8BdH?NqJ+;?* zzmhz!JNM5)`9BxiUd!>ZuY8@hb)Mhy*{d>_UVr;ES%1!1?f5Asrw{w4rKf$q|F<># z$?j===^yTJ@BIAX;-9zd_b07aQ|Gc|NrBk z-rxK5?f)5nfA*%+Sn}Mx8Pe&uGtbqS{{Qs+U+BJB_geX0Ui01gd*AQk@3*?z>gTA} z|GrqW(0cAnzo}^(rYv3Nb@kPfuSeQHeg5x0xorLPm1UFbHk-Vg_|7+d+0`X0mtCrS zwEffme}|2KHtsK~zgO^l^{2aU6VKWFe){J(|33TupKe^9IZIP^9}`1?!PU~qdE2K8 ztNT5B7M*u8Wc}=2>w>qg+T#B9slwc+j!eF8Up0Jf-yO9p{1oCh*JkF;o6eq7ORU7N z8_SsPd_M2`vb%Y@{GTQ6|J|_e_})vOxw$h#L(iQ*|Gf2i{E0Jfa)MNYF9-kpu=1x( zP0EDnVQa4(UMe$NQuVX{+P+S?xm9-MeQuuX{cP=hR^L9o{G92}6FbZL?paJd+M4@p zUR>DS-`~v5_r{g~o-}*L+_`7hZofCp`u#5L?RP#c+OjHaUhTJ;HvfLifAZtwQBe6?^oLHJNHC--D~sDuls9^>TkVI+h(nI-G2X!qZi#u zw;0uCXa!%@_?$l7D5Jc%V%_B4_sjML`CD&puYReg{w2-iOj7-fgBg{6N5Zu4#+hH< zdrft*7X!mO_O)Twiyvw%-gRzn`Mr&|nD>5rrYU|mWa(8e+pJGN9@kGQTRu6iTx;#M zAK&gZFa7xClz#os`QBGgRk-z=epQK-du6!qaN;bj`2WAIPY!wcb=OPZ-h8GT@o%1= zc(n4*m9@`{mq@PhYu=vt+xqUDmAiggpO-n8@XlkB+Tnx8io@4$-C`oa{xWa7jYqua z*Mb>!`%lfS|GNBBF#r82kN-a`|0FH{$7kQCBcEr=U%QzgW5vKQ-E*>v?_|I2vzIPi z+SB%U=YsU~bo06HRdscC{`P-;ZoLg#wrsint_d5~&DQz)TeA0+jCEObT3Xt)*Ac(d z|7TVH+7r0%>ss~T>!r(2e(w2xV&&=B z{EwP>^L8t*uReXd=)U#6iS2QFr}Wo<(6+pP(U^y!K-JWDa+S%ckoD6;PM#E2zx-Zb zHFR>=?x|{zEoVJ{{`~VHZuL)xxb>G@pX-&Mm1XJ@ezoG=lU1vx>D%QLJvwsoHSeV6!$v2UwZx3X#WO_k29X{T6f&`($ekk_f^f} z`E^)lYu<_Nd;g^F+*)Te|Nmy=ms^fzFf?fAE`7f9^4ld|*3AzU79T9IQLCz);lBH- z-|pL<>o3R7Tb8%y%`eSQ$Nu}zN}ICupuypT1(g==b5~`7Mqe&xF2DVHS?E?huE%%p zT~W3k<8mjj{Mg>ptAD<6x)D1wzH0N&h2e7Y=LP4c+>Kp! zd^?BacdP#NZ#x3cpOSc)dpim=W_0+V#^0s?UdalSCz(i1^{?L^AF@{Kt<0NT28JDC zYnE($C?@_r*6;J@&*3Gtb3guX31DEj(fO!y2_wUUs|Wef}lu3oS#`jKJx;l$$b z-Leb}1qn8PE;P@+bAz9O;l?BdUIqqMmP`f)4FwNo1_wq*DFy~Ej!O&-0S*(`7#f(G zOc)qM1eP!|ENB>2IvN6_Aut*OqaiRF0s|ca4(gt>p4oa%TJ`JdRdutfoFMJfd0YRe zq?ToG&b@JVoA0i3Z(rR%X}2rpH#pgd!!oUy{cl9v8?TwlT6W!OBSu-#^ zI8gCpA^Y74tPBiZSZnSBN6b?qtzB9?x@z5Vqsecc zYe%v%+_=;CeB!=&vPWt+h!=;f=dG(Y{QgztbXO^QZHMc;cf{HH+U+;zFH6@7jJi>r?@kA7CA&CDR9<7#_p_T76`f9H2^=UR2!c7K`e^xfNYc7J;_)${v# z6}7v^XaC)O$z3p#M*my)UnOOENropdoi_(lYfIZt@GU9vqt{^-a5%`W+&_RO-f~GcpJ)Z|0Usbqq|&eZNj9RYTI@- zbzf$sEAtTdAHnMvu>Ma&NkPIPFkINKd=66iGAGp)NjGj3=L;(|6FLE z&i3-J6Tc(_!*5VjS5)em?Gf50ZKlw`%urzY_O!y@x;M|<7DfcHF+9k+HQh1FgN-3! z|L$eW_cJpzoOT5_+JVzeAn-N)Q-rRo^MJx4m{n{c-rs#`CHb!d7;<8hOsd`h%zed|DsY~ zHvfzN=gBwL^=&<$`{lim@8@~C`#Ey9t(E>;z4_Pr>qf6n=Wm~XecH`6_s+h$nJLN8 zAY{GoM#1uhOHA&(+x}tmrOUgv#ZHcSupc02asxBNbZ_mMIWL%`-^(*Ftn*);yySYn zf9&rYY2R*3T%Ax*1g_)#&7=qJ8xMTG(LWC=8uzF!pP9D78GrHCzu!*R54rA z*Dbc6V0#et)VObT?(uT7J(tRi|G$s(=`L4&btfzKcj~Eap20TEvl$pz_IkQFhHU%o z^=`JO?rDm?b&^* zp6`15Dl$I&oC0*X#=~KX&JI?T*{UIqJ_p zmwii$UAq4ArK)vH=I+_<^XW!f>Du_TNmkzuvsaz`eM9B>QMc6ZulTlAr*8iIwP;=T z(Kyy~o_$xtoO)&^2yO&F`bHT(~EC=$DZ7u_U6U8w#hXIazghl zndkX>#*>YcQq|TjcR5)W@3-`L@gz_Q=GlLAN$#u?`FBq(w@v=~E-hBOfBKZ0r%T^X zoqKKX^y6u}ji&!D-6hVLv3&j9yL-&1@A1=LzPIA<+pD|EfBgwdRlB=x_0GSqW7lnq zd8^M^{dL~X>+=gI+w6~>x%ZUW^yJ&>k=LWD|H|2(tO6yiz4EsTRWJQdEBl;EpuoVCgR{<@liwXyrQ z%oE*Rd47-7we8DG;@_3n#$@UDAHDh~X+zTT_3P(u`C9Z)s3YUF-iH7xCM1>!!ST^-#7Q(+B^My;O~>Bw%Ny{?)`Z1Df0KT z9n(uU8HaBDJ(F+UvGzMpE$Y7=-o>{lyY&B?qh)W8zRni99(5rm`%-PignPnzoR`>t zx^9WRtzOIcW_#_1`?rc!-(KA&f26iT;P#$yzDH&8XCH5^ySr5W+w9(Cn~e7}H%$Nb z>Z_^e-QSij@1K>{?fmxe=|Ao(DsOi`NjmJwwlqHEpjdLKne-mJlL}vYubH3RF=H*; zzx-|M_TF0O`DV$y-dp>xtL2_AJDxPj$olU2$y08=wYziL?EN>U*>mTdpMBNdbLQTt z>v1!moPPhS==R?2K2hi0&in6LclOk~V|lM;rR+A-GC%om_L4hwKOWpanVqsdxh(wE z4gL2sch~;EnZ3J~|K{uS`}f+s)AqV&tiCe+l3Co!;+)!>lZtDXzrN*lvTXXg{M}~u ze=aodp1qYh!T$5SqfzWXYb;LW?!WcMT5tMkqipsAbq@}Ies%NR`Gw)HXUHrmPF#QP z{@*H{bxU3+O7E;Ht8tz-_5R!$2kJJ(w@b(LzxjSQ+WzptQ^~4d8K$2<;&`iD@W%N) z?uBa${SN;tzw1-a&cIOZ39bN0=($g4dAY38f7UbGXuTz8zDBY!G;n_bjRrKnde|

 -
-

First Edition

- You are viewing the second edition of “Writing an OS in Rust”, which is still in progress. The first edition has more content, but is no longer updated. We try our best to incorporate the missing content soon. -
-
@@ -75,6 +70,11 @@
+

First Edition

+

You are currently viewing the second edition of “Writing an OS in Rust”. The first edition is very different in many aspects, for example it builds upon the GRUB bootloader instead of using the `bootloader` crate. In case you're interested in it, it is still available. Note that the first edition is no longer updated and might contain outdated information. read the first edition »

+
+ +

Support Me

{% include "support.html" %}
From c7af79f4525e0cd0e776b65bbec2deb700204782 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Tue, 28 Jan 2020 17:06:47 +0100 Subject: [PATCH 62/85] Remove '(Second Edition)' from title --- blog/templates/second-edition/base.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/templates/second-edition/base.html b/blog/templates/second-edition/base.html index e7df9a0f..35247461 100644 --- a/blog/templates/second-edition/base.html +++ b/blog/templates/second-edition/base.html @@ -25,7 +25,7 @@

- {{ config.title | safe }} (Second Edition) + {{ config.title | safe }}

{{ config.extra.subtitle | replace(from=" ", to=" ") | safe }}

{% block header %}{% endblock header %} From 6a56ea242fde41e4f82d13be527cc2b7b02273e3 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Tue, 28 Jan 2020 17:10:46 +0100 Subject: [PATCH 63/85] Fix dummy allocator code example (#728) We forgot to remove the `allocator::` prefix when moving the ALLOCATOR declaration into the `allocator` module. --- blog/content/second-edition/posts/10-heap-allocation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/content/second-edition/posts/10-heap-allocation/index.md b/blog/content/second-edition/posts/10-heap-allocation/index.md index 0bc6eae7..b5fc949a 100644 --- a/blog/content/second-edition/posts/10-heap-allocation/index.md +++ b/blog/content/second-edition/posts/10-heap-allocation/index.md @@ -309,7 +309,7 @@ The `#[global_allocator]` attribute tells the Rust compiler which allocator inst // in src/allocator.rs #[global_allocator] -static ALLOCATOR: allocator::Dummy = allocator::Dummy; +static ALLOCATOR: Dummy = Dummy; ``` Since the `Dummy` allocator is a [zero sized type], we don't need to specify any fields in the initialization expression. From 409c6de44a2247ae657fb5660dc368d50474490b Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Wed, 29 Jan 2020 13:29:01 +0100 Subject: [PATCH 64/85] Add doc link for eh_personality language item --- .../second-edition/posts/01-freestanding-rust-binary/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md b/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md index 6bba4d3e..5e64784a 100644 --- a/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md +++ b/blog/content/second-edition/posts/01-freestanding-rust-binary/index.md @@ -147,8 +147,9 @@ Language items are special functions and types that are required internally by t While providing custom implementations of language items is possible, it should only be done as a last resort. The reason is that language items are highly unstable implementation details and not even type checked (so the compiler doesn't even check if a function has the right argument types). Fortunately, there is a more stable way to fix the above language item error. -The `eh_personality` language item marks a function that is used for implementing [stack unwinding]. By default, Rust uses unwinding to run the destructors of all live stack variables in case of a [panic]. This ensures that all used memory is freed and allows the parent thread to catch the panic and continue execution. Unwinding, however, is a complicated process and requires some OS specific libraries (e.g. [libunwind] on Linux or [structured exception handling] on Windows), so we don't want to use it for our operating system. +The [`eh_personality` language item] marks a function that is used for implementing [stack unwinding]. By default, Rust uses unwinding to run the destructors of all live stack variables in case of a [panic]. This ensures that all used memory is freed and allows the parent thread to catch the panic and continue execution. Unwinding, however, is a complicated process and requires some OS specific libraries (e.g. [libunwind] on Linux or [structured exception handling] on Windows), so we don't want to use it for our operating system. +[`eh_personality` language item]: https://github.com/rust-lang/rust/blob/edb368491551a77d77a48446d4ee88b35490c565/src/libpanic_unwind/gcc.rs#L11-L45 [stack unwinding]: http://www.bogotobogo.com/cplusplus/stackunwinding.php [libunwind]: http://www.nongnu.org/libunwind/ [structured exception handling]: https://msdn.microsoft.com/en-us/library/windows/desktop/ms680657(v=vs.85).aspx From 5f7437df8ddb31ff230c780aad099b7b34af3854 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Thu, 30 Jan 2020 12:05:04 +0100 Subject: [PATCH 65/85] Fix and improve generated HTML --- blog/templates/macros.html | 6 +++--- blog/templates/second-edition/index.html | 12 ++++++++---- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/blog/templates/macros.html b/blog/templates/macros.html index 061dac66..74090afd 100644 --- a/blog/templates/macros.html +++ b/blog/templates/macros.html @@ -10,9 +10,9 @@ {% macro utterances() %} diff --git a/blog/templates/second-edition/index.html b/blog/templates/second-edition/index.html index 38267733..e369aca1 100644 --- a/blog/templates/second-edition/index.html +++ b/blog/templates/second-edition/index.html @@ -8,6 +8,8 @@ {% set posts_section = get_section(path = "second-edition/posts/_index.md") %} {% set posts = posts_section.pages %} +

Posts

+

This blog series creates a small operating system in the @@ -48,11 +50,11 @@

Subscribe

Receive notifications about new posts and other major changes! You can either:

-

@@ -78,7 +80,9 @@

Support Me

{% include "support.html" %}
+{% endblock main %} +{% block after_main %} -{% endblock main %} +{% endblock after_main %} From 4f428263992da1944d20dc1dffadf907d2cafd3f Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Thu, 30 Jan 2020 12:42:13 +0100 Subject: [PATCH 66/85] Update address on contact page --- blog/content/pages/contact.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/content/pages/contact.md b/blog/content/pages/contact.md index fb0e007e..34c952f0 100644 --- a/blog/content/pages/contact.md +++ b/blog/content/pages/contact.md @@ -8,4 +8,4 @@ Philipp Oppermann contact@phil-opp.com -Dr.Gustav-Knodel-Str. 7b, 76344 Eggenstein, Germany +Gerwigstraße 17, 76131 Karlsruhe, Germany From 670ac60e1bf92e51e9b3667f78bf1299ea7ca17a Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Thu, 30 Jan 2020 13:04:56 +0100 Subject: [PATCH 67/85] Update Allocator Designs post to use LinkedListAllocator::lock method --- .../second-edition/posts/11-allocator-designs/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/blog/content/second-edition/posts/11-allocator-designs/index.md b/blog/content/second-edition/posts/11-allocator-designs/index.md index 0803b335..6b6284ba 100644 --- a/blog/content/second-edition/posts/11-allocator-designs/index.md +++ b/blog/content/second-edition/posts/11-allocator-designs/index.md @@ -701,7 +701,7 @@ unsafe impl GlobalAlloc for Locked { unsafe fn alloc(&self, layout: Layout) -> *mut u8 { // perform layout adjustments let (size, align) = LinkedListAllocator::size_align(layout); - let mut allocator = self.inner.lock(); + let mut allocator = self.lock(); if let Some((region, alloc_start)) = allocator.find_region(size, align) { let alloc_end = alloc_start.checked_add(size).expect("overflow"); @@ -719,7 +719,7 @@ unsafe impl GlobalAlloc for Locked { // perform layout adjustments let (size, _) = LinkedListAllocator::size_align(layout); - self.inner.lock().add_free_region(ptr as usize, size) + self.lock().add_free_region(ptr as usize, size) } } ``` From e2ea963d5d8dc653c7e67cb4bd4a17a65116aac8 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Fri, 31 Jan 2020 12:12:54 +0100 Subject: [PATCH 68/85] Make 'All Posts' link fixed Fixes #731 --- blog/static/css/main.css | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/blog/static/css/main.css b/blog/static/css/main.css index fd587608..b0f18a5c 100644 --- a/blog/static/css/main.css +++ b/blog/static/css/main.css @@ -188,7 +188,7 @@ aside#all-posts-link { width: 12rem; position: sticky; float: left; - top: 1rem; + top: 3.5rem; margin-top: -4rem; margin-left: -15rem; font-size: 90%; @@ -258,11 +258,10 @@ aside#all-posts-link { } aside#all-posts-link { - float: left; - position: absolute; - bottom: 0; - left: -15rem; + position: fixed; + top: 1.25rem; margin-top: 0; + margin-left: -15rem; } } From 728620b527dd4d8e308c9344e319e5e11bdad859 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Fri, 31 Jan 2020 14:17:22 +0100 Subject: [PATCH 69/85] Mark active item in table of contents (#733) --- blog/static/css/main.css | 11 +++++++++-- blog/static/js/main.js | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 2 deletions(-) diff --git a/blog/static/css/main.css b/blog/static/css/main.css index b0f18a5c..7f940cbb 100644 --- a/blog/static/css/main.css +++ b/blog/static/css/main.css @@ -193,14 +193,21 @@ aside#all-posts-link { margin-left: -15rem; font-size: 90%; line-height: 1.2; + } + + #toc-aside li:not(.active) > a, #toc-aside h2 { opacity: .5; transition: opacity .5s; } - #toc-aside:hover { + #toc-aside:hover li:not(.active) > a, #toc-aside:hover h2 { opacity: 1; } + #toc-aside li.active > a { + font-weight: bold; + } + #toc-aside h2 { font-size: 110%; margin-bottom: .2rem; @@ -212,7 +219,7 @@ aside#all-posts-link { list-style:none; } - #toc-aside ol li:before { + #toc-aside ol li a:before { content: ""; border-color: transparent #008eef; border-style: solid; diff --git a/blog/static/js/main.js b/blog/static/js/main.js index 96ed6181..ce6b080b 100644 --- a/blog/static/js/main.js +++ b/blog/static/js/main.js @@ -3,6 +3,8 @@ window.onload = function() { if (container != null) { resize_toc(container); + toc_scroll_position(container); + window.onscroll = function() { toc_scroll_position(container) }; } } @@ -24,3 +26,38 @@ function resize_toc(container) { resizeId = setTimeout(resize, 300); }; } + +function toc_scroll_position(container) { + if (container.offsetParent === null) { + // skip computation if ToC is not visible + return; + } + var items = container.querySelectorAll("li") + + // remove active class for all items + for (item of container.querySelectorAll("li")) { + item.classList.remove("active"); + } + + // look for active item + var site_offset = document.documentElement.scrollTop; + var current_toc_item = null; + for (item of container.querySelectorAll("li")) { + if (item.offsetParent === null) { + // skip items that are not visible + continue; + } + var anchor = item.firstElementChild.getAttribute("href"); + var heading = document.querySelector(anchor); + if (heading.offsetTop <= (site_offset + document.documentElement.clientHeight / 3)) { + current_toc_item = item; + } else { + break; + } + } + + // set active call for current ToC item + if (current_toc_item != null) { + current_toc_item.classList.add("active"); + } +} From b569fcbc25bbc9611d7e46579976ed5b1cd557bf Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Fri, 31 Jan 2020 14:19:52 +0100 Subject: [PATCH 70/85] Fix typo in Javascript comment --- blog/static/js/main.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/static/js/main.js b/blog/static/js/main.js index ce6b080b..dfa1dd4d 100644 --- a/blog/static/js/main.js +++ b/blog/static/js/main.js @@ -56,7 +56,7 @@ function toc_scroll_position(container) { } } - // set active call for current ToC item + // set active class for current ToC item if (current_toc_item != null) { current_toc_item.classList.add("active"); } From 423ab3fa686140ecaa2c776726a96a2adea01e1f Mon Sep 17 00:00:00 2001 From: Maxim Zholobak Date: Fri, 31 Jan 2020 17:20:13 +0200 Subject: [PATCH 71/85] Make active section link more discreet (#734) --- blog/static/css/main.css | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/blog/static/css/main.css b/blog/static/css/main.css index 7f940cbb..3eec6879 100644 --- a/blog/static/css/main.css +++ b/blog/static/css/main.css @@ -195,12 +195,12 @@ aside#all-posts-link { line-height: 1.2; } - #toc-aside li:not(.active) > a, #toc-aside h2 { + #toc-aside li > a, #toc-aside h2 { opacity: .5; transition: opacity .5s; } - #toc-aside:hover li:not(.active) > a, #toc-aside:hover h2 { + #toc-aside:hover li > a, #toc-aside:hover h2 { opacity: 1; } From e91567f637249974edc38857c930699b6f6ac803 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 1 Feb 2020 14:24:41 +0100 Subject: [PATCH 72/85] Reword 'thank you' section --- blog/templates/status-update-page.html | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/blog/templates/status-update-page.html b/blog/templates/status-update-page.html index 45105ad5..bc4dccad 100644 --- a/blog/templates/status-update-page.html +++ b/blog/templates/status-update-page.html @@ -14,7 +14,8 @@

Thank You!

-

I want to thank all the people who support me on GitHub, Patreon, and Donorbox. It means a lot to me!

+

Thanks to all the contributors this month!

+

I also want to thank all the people who support me on GitHub, Patreon, and Donorbox. It means a lot to me!

{% endblock main %} From e134914ef8f683fe0ff6958c6e0a3cee1d179951 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 1 Feb 2020 14:24:58 +0100 Subject: [PATCH 73/85] Updates in January 2020 --- blog/content/status-update/2020-02-01.md | 69 ++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 blog/content/status-update/2020-02-01.md diff --git a/blog/content/status-update/2020-02-01.md b/blog/content/status-update/2020-02-01.md new file mode 100644 index 00000000..818a08ed --- /dev/null +++ b/blog/content/status-update/2020-02-01.md @@ -0,0 +1,69 @@ ++++ +title = "Updates in January 2020" +date = 2020-02-01 ++++ + +This post gives an overview of the recent updates to the _Writing an OS in Rust_ blog and the corresponding libraries and tools. + +## `blog_os` + +The repository of the _Writing an OS in Rust_ blog received the following updates: + +- [Move #[global_allocator] into allocator module](https://github.com/phil-opp/blog_os/pull/714) +- [Update many_boxes test to scale with heap size](https://github.com/phil-opp/blog_os/pull/716) +- [New post about allocator designs](https://github.com/phil-opp/blog_os/pull/719) 🎉 +- [Provide multiple implementations of align_up and mention performance](https://github.com/phil-opp/blog_os/pull/721) +- [Refactor Simplified Chinese translation of post 3](https://github.com/phil-opp/blog_os/pull/725) by [@Rustin-Liu](https://github.com/Rustin-Liu) +- [Use checked addition for allocator implementations](https://github.com/phil-opp/blog_os/pull/726) +- [Fix dummy allocator code example](https://github.com/phil-opp/blog_os/pull/728) +- [Some style updates to the front page](https://github.com/phil-opp/blog_os/pull/729) +- [Mark active item in table of contents](https://github.com/phil-opp/blog_os/pull/733) +- [Make active section link more discreet](https://github.com/phil-opp/blog_os/pull/734) by [@Menschenkindlein](https://github.com/Menschenkindlein) + +I also started working on the upcoming post about threads. + +## `bootloader` + +The bootloader crate received two minor updates this month: + +- [Move architecture checks from build script into lib.rs](https://github.com/rust-osdev/bootloader/pull/91) +- [Update x86_64 dependency to version 0.8.3](https://github.com/rust-osdev/bootloader/pull/92) by [@vinaychandra](https://github.com/vinaychandra) + +Since I focused my time on the new _Allocator Designs_ post, I did not have the time to make more progress on my plan to rewrite the 16-bit/32-bit stages of the bootloader in Rust. I hope to get back to it soon. + +## `bootimage` + +There were no updates to the `bootimage` tool this month. + +## `x86_64` + +The following changes were merged this month: + +- [Allow immediate port version of in/out instructions](https://github.com/rust-osdev/x86_64/pull/115) by [@m-ou-se](https://github.com/m-ou-se) +- [Make more functions const](https://github.com/rust-osdev/x86_64/pull/116) by [@m-ou-se](https://github.com/m-ou-se) + - Released as version 0.8.3 +- [Return the UnusedPhysFrame on MapToError::PageAlreadyMapped](https://github.com/rust-osdev/x86_64/pull/118) by [@haraldh](https://github.com/haraldh) + - This is a **breaking change** since it changes the signature of a type. + - No new release was published yet to give us the option to bundle it with other breaking changes. + +There are also some pull requests that have some open design questions and are still being discussed: + +- [Add p23_insert_flag_mask argument to mapper.map_to()](https://github.com/rust-osdev/x86_64/pull/114) by [@haraldh](https://github.com/haraldh) + - Related proposal: [Page Table Visitors](https://github.com/rust-osdev/x86_64/issues/121) by [@mark-i-m](https://github.com/mark-i-m) +- [Add User Mode registers](https://github.com/rust-osdev/x86_64/pull/119) by [@vinaychandra](https://github.com/vinaychandra) + +Please feel free to join these discussions if you have opinions on the matter. + +## `cargo-xbuild` + +The `cargo-xbuild` crate, which cross-compiles the sysroot, received the following updates this month: + +- [Override target path for building sysroot](https://github.com/rust-osdev/cargo-xbuild/pull/52) by [@upsuper](https://github.com/upsuper) + - Published as version 0.5.21 + +## `uart_16550` + +The `uart_16550` crate, which provides basic support for uart_16550 serial output, received a small dependency update: + +- [Update dependency for x86_64](https://github.com/rust-osdev/uart_16550/pull/4) by [@haraldh](https://github.com/haraldh) + - Published as version 0.2.2 From 5276972c6757ac9b1f48d420f3b2481383494e78 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Mon, 3 Feb 2020 17:13:28 +0100 Subject: [PATCH 74/85] Set `meta.description` tag correctly --- blog/config.toml | 1 + .../content/second-edition/extra/building-on-android/index.md | 2 ++ .../posts/02-minimal-rust-kernel/disable-red-zone/index.md | 2 ++ .../posts/02-minimal-rust-kernel/disable-simd/index.md | 2 ++ blog/templates/second-edition/base.html | 2 +- blog/templates/second-edition/extra.html | 4 ++++ blog/templates/second-edition/page.html | 4 ++++ 7 files changed, 16 insertions(+), 1 deletion(-) diff --git a/blog/config.toml b/blog/config.toml index 7fc83ae9..28bae2c9 100644 --- a/blog/config.toml +++ b/blog/config.toml @@ -1,5 +1,6 @@ title = "Writing an OS in Rust" base_url = "https://os.phil-opp.com" +description = "This blog series creates a small operating system in the Rust programming language. Each post is a small tutorial and includes all needed code, so you can follow along if you like." highlight_code = true highlight_theme = "visual-studio-dark" diff --git a/blog/content/second-edition/extra/building-on-android/index.md b/blog/content/second-edition/extra/building-on-android/index.md index 7d0d362a..9bdf0ef5 100644 --- a/blog/content/second-edition/extra/building-on-android/index.md +++ b/blog/content/second-edition/extra/building-on-android/index.md @@ -6,6 +6,8 @@ weight = 3 I finally managed to get `blog_os` building on my Android phone using [termux](https://termux.com/). This post explains the necessary steps to set it up. + + Screenshot of the compilation output from android diff --git a/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-red-zone/index.md b/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-red-zone/index.md index 023e0af3..212ccb97 100644 --- a/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-red-zone/index.md +++ b/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-red-zone/index.md @@ -10,6 +10,8 @@ The [red zone] is an optimization of the [System V ABI] that allows functions to [red zone]: http://eli.thegreenplace.net/2011/09/06/stack-frame-layout-on-x86-64#the-red-zone [System V ABI]: http://wiki.osdev.org/System_V_ABI + + ![stack frame with red zone](red-zone.svg) The image shows the stack frame of a function with `n` local variables. On function entry, the stack pointer is adjusted to make room on the stack for the return address and the local variables. diff --git a/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-simd/index.md b/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-simd/index.md index da46183f..6fe2f8d8 100644 --- a/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-simd/index.md +++ b/blog/content/second-edition/posts/02-minimal-rust-kernel/disable-simd/index.md @@ -9,6 +9,8 @@ template = "second-edition/extra.html" [Single Instruction Multiple Data (SIMD)]: https://en.wikipedia.org/wiki/SIMD + + - [MMX]: The _Multi Media Extension_ instruction set was introduced in 1997 and defines eight 64 bit registers called `mm0` through `mm7`. These registers are just aliases for the registers of the [x87 floating point unit]. - [SSE]: The _Streaming SIMD Extensions_ instruction set was introduced in 1999. Instead of re-using the floating point registers, it adds a completely new register set. The sixteen new registers are called `xmm0` through `xmm15` and are 128 bits each. - [AVX]: The _Advanced Vector Extensions_ are extensions that further increase the size of the multimedia registers. The new registers are called `ymm0` through `ymm15` and are 256 bits each. They extend the `xmm` registers, so e.g. `xmm0` is the lower half of `ymm0`. diff --git a/blog/templates/second-edition/base.html b/blog/templates/second-edition/base.html index 35247461..5ff4e0b2 100644 --- a/blog/templates/second-edition/base.html +++ b/blog/templates/second-edition/base.html @@ -6,7 +6,7 @@ - + {% if current_url %} diff --git a/blog/templates/second-edition/extra.html b/blog/templates/second-edition/extra.html index 0926bcd2..105ff0be 100644 --- a/blog/templates/second-edition/extra.html +++ b/blog/templates/second-edition/extra.html @@ -4,6 +4,10 @@ {% block title %}{{ page.title }} | {{ config.title }}{% endblock title %} +{% block description -%} +{{ page.summary | safe | striptags }} +{%- endblock description %} + {% block main %}

{{ page.title }}

{{ page.content | safe }} diff --git a/blog/templates/second-edition/page.html b/blog/templates/second-edition/page.html index 94000b8f..4cc78136 100644 --- a/blog/templates/second-edition/page.html +++ b/blog/templates/second-edition/page.html @@ -7,6 +7,10 @@ {% endblock header %} +{% block description -%} +{{ page.summary | safe | striptags }} +{%- endblock description %} + {% block main %}

{{ page.title }}