From 4d59d7350c9b3a652c00495264061a8f2177d9c5 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 8 Nov 2015 14:49:53 +0100 Subject: [PATCH 01/30] Add skeleton for next post --- posts/DRAFT-multiboot-module.md | 49 +++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 posts/DRAFT-multiboot-module.md diff --git a/posts/DRAFT-multiboot-module.md b/posts/DRAFT-multiboot-module.md new file mode 100644 index 00000000..66399703 --- /dev/null +++ b/posts/DRAFT-multiboot-module.md @@ -0,0 +1,49 @@ +--- +layout: post +title: 'A Multiboot Module' +--- +TODO + +## The Multiboot Information Structure +When a multiboot compliant bootloader loads a kernel, it passes a pointer to a boot information struct in the `ebx` register. We can use it to get information about available memory and loaded kernel sections. So let's write a module for it! + +TODO + +## Start and End of Kernel +We can now use the ELF section tag to calculate the start and end address of our loaded kernel: + +TODO + +## A frame allocator +When we create a paging module in the next post, we will need to map virtual pages to free physical frames. So we will need some kind of allocator that keeps track of physical frames and gives us a free one when needed. We can use the memory tag to write such a frame allocator. + +The allocator struct looks like this: + +```rust +struct AreaFrameAllocator { + first_used_frame: Frame, + last_used_frame: Frame, + current_area: Option, + areas: MemoryAreaIter, +} +``` +TODO + +To allocate a frame we try to find one in the current area and update the first/last used bounds. If we can't find one, we look for the new area with the minimal start address, that still contains free frames. If the current area is `None`, there are no free frames left. + +TODO + +### Unit Tests +TODO + +## Remapping the Kernel Sections +We can use the ELF section tag to write a skeleton that remaps the kernel correctly: + +```rust +for section in multiboot.elf_tag().sections() { + for page in start_page..end_page { + // TODO identity_map(page, section.writable(), section.executable()) + } +} +``` +TODO From 1f074264c04e0bb6967a17b632d86f1b791c0563 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Wed, 11 Nov 2015 23:30:16 +0100 Subject: [PATCH 02/30] Rename and restructure multiboot info post --- posts/{DRAFT-multiboot-module.md => DRAFT-multiboot-info.md} | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) rename posts/{DRAFT-multiboot-module.md => DRAFT-multiboot-info.md} (95%) diff --git a/posts/DRAFT-multiboot-module.md b/posts/DRAFT-multiboot-info.md similarity index 95% rename from posts/DRAFT-multiboot-module.md rename to posts/DRAFT-multiboot-info.md index 66399703..5c0b689c 100644 --- a/posts/DRAFT-multiboot-module.md +++ b/posts/DRAFT-multiboot-info.md @@ -1,12 +1,11 @@ --- layout: post -title: 'A Multiboot Module' +title: 'The Multiboot Information Structure' --- -TODO -## The Multiboot Information Structure When a multiboot compliant bootloader loads a kernel, it passes a pointer to a boot information struct in the `ebx` register. We can use it to get information about available memory and loaded kernel sections. So let's write a module for it! +## The Structure TODO ## Start and End of Kernel From c064de51908c2972bd0fad021371b2acc69efdaf Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Wed, 11 Nov 2015 23:32:36 +0100 Subject: [PATCH 03/30] Add description of info struct and elf tag --- posts/DRAFT-multiboot-info.md | 67 ++++++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/posts/DRAFT-multiboot-info.md b/posts/DRAFT-multiboot-info.md index 5c0b689c..4cd4d936 100644 --- a/posts/DRAFT-multiboot-info.md +++ b/posts/DRAFT-multiboot-info.md @@ -3,9 +3,74 @@ layout: post title: 'The Multiboot Information Structure' --- -When a multiboot compliant bootloader loads a kernel, it passes a pointer to a boot information struct in the `ebx` register. We can use it to get information about available memory and loaded kernel sections. So let's write a module for it! +When a Multiboot compliant bootloader loads a kernel, it passes a pointer to a boot information structure in the `ebx` register. We can use it to get information about available memory and loaded kernel sections. + +TODO ## The Structure +The Multiboot information structure looks like this: + +Field | Type +---------------- | ----------- +total size | u32 +reserved | u32 +tags | variable +end tag = (0, 8) | (u32, u32) + +There are many different types of tags, but they all have the same beginning: + +Field | Type +------------- | ----------------- +type | u32 +size | u32 +other fields | variable + +All tags are 8-byte aligned. The last tag must be the _end tag_, which is a tag of type `0` and size `8`. + +## A Rust module + +TODO + +## Tags + +We are interested in two tags, the _Elf-symbols_ tag and the _memory map_ tag. For a full list of possible tags see section 3.4 in the Multiboot 2 specification ([PDF][Multiboot 2]). + +[Multiboot 2]: http://nongnu.askapache.com/grub/phcoder/multiboot.pdf + +### The Elf-Symbols Tag +The Elf-symbols tag contains a list of all sections of the loaded [ELF] kernel. It has the following format: + +[ELF]: https://en.wikipedia.org/wiki/Executable_and_Linkable_Format + +Field | Type +--------------------------- | ----------------- +type = 9 | u32 +size | u32 +number of entries | u16 +entry size | u16 +string table | u16 +reserved | u16 +section headers | variable + +The section headers are just copied from the ELF file, so we need to look at the ELF specification to find the corresponding structure definition. Our kernel is a 64-bit ELF file, so we need to look at the ELF-64 specification ([PDF][ELF specification]). According to section 4 and figure 3, a section header has the following format: + +[ELF specification]: http://www.uclibc.org/docs/elf-64-gen.pdf + +Field | Type | Value +--------------------------- | ---------------- | ----------- +name | u32 | string table index +type | u32 | `0` (unused), `1` (section of program), `3` (string table), `8` (uninitialized section), etc. +flags | u64 | `0x1` (writable), `0x2` (loaded), `0x4` (executable), etc. +address | u64 | virtual start address of section (0 if not loaded) +file offset | u64 | offset (in bytes) of section contents in the file +size | u64 | size of the section in bytes +link | u32 | associated section (only for some section types) +info | u32 | extra information (only for some section types) +address align | u64 | required alignment of section (power of 2) +entry size | u64 | contains the entry size for table sections (e.g. string table) + +### The Memory Map Tag + TODO ## Start and End of Kernel From a54b2d8f476a71d70563eb2b02a84479c207b5f2 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Thu, 12 Nov 2015 17:29:13 +0100 Subject: [PATCH 04/30] Remove detailed description of multiboot structures Instead, we will use the external crate `multiboot2-elf64` at https://github.com/phil-opp/multiboot2-elf64 . The removed parts were added to its README for documentation --- posts/DRAFT-multiboot-info.md | 71 ++--------------------------------- 1 file changed, 3 insertions(+), 68 deletions(-) diff --git a/posts/DRAFT-multiboot-info.md b/posts/DRAFT-multiboot-info.md index 4cd4d936..bf8b02c2 100644 --- a/posts/DRAFT-multiboot-info.md +++ b/posts/DRAFT-multiboot-info.md @@ -3,76 +3,11 @@ layout: post title: 'The Multiboot Information Structure' --- +TODO + +## The Multiboot Information Structure When a Multiboot compliant bootloader loads a kernel, it passes a pointer to a boot information structure in the `ebx` register. We can use it to get information about available memory and loaded kernel sections. -TODO - -## The Structure -The Multiboot information structure looks like this: - -Field | Type ----------------- | ----------- -total size | u32 -reserved | u32 -tags | variable -end tag = (0, 8) | (u32, u32) - -There are many different types of tags, but they all have the same beginning: - -Field | Type -------------- | ----------------- -type | u32 -size | u32 -other fields | variable - -All tags are 8-byte aligned. The last tag must be the _end tag_, which is a tag of type `0` and size `8`. - -## A Rust module - -TODO - -## Tags - -We are interested in two tags, the _Elf-symbols_ tag and the _memory map_ tag. For a full list of possible tags see section 3.4 in the Multiboot 2 specification ([PDF][Multiboot 2]). - -[Multiboot 2]: http://nongnu.askapache.com/grub/phcoder/multiboot.pdf - -### The Elf-Symbols Tag -The Elf-symbols tag contains a list of all sections of the loaded [ELF] kernel. It has the following format: - -[ELF]: https://en.wikipedia.org/wiki/Executable_and_Linkable_Format - -Field | Type ---------------------------- | ----------------- -type = 9 | u32 -size | u32 -number of entries | u16 -entry size | u16 -string table | u16 -reserved | u16 -section headers | variable - -The section headers are just copied from the ELF file, so we need to look at the ELF specification to find the corresponding structure definition. Our kernel is a 64-bit ELF file, so we need to look at the ELF-64 specification ([PDF][ELF specification]). According to section 4 and figure 3, a section header has the following format: - -[ELF specification]: http://www.uclibc.org/docs/elf-64-gen.pdf - -Field | Type | Value ---------------------------- | ---------------- | ----------- -name | u32 | string table index -type | u32 | `0` (unused), `1` (section of program), `3` (string table), `8` (uninitialized section), etc. -flags | u64 | `0x1` (writable), `0x2` (loaded), `0x4` (executable), etc. -address | u64 | virtual start address of section (0 if not loaded) -file offset | u64 | offset (in bytes) of section contents in the file -size | u64 | size of the section in bytes -link | u32 | associated section (only for some section types) -info | u32 | extra information (only for some section types) -address align | u64 | required alignment of section (power of 2) -entry size | u64 | contains the entry size for table sections (e.g. string table) - -### The Memory Map Tag - -TODO - ## Start and End of Kernel We can now use the ELF section tag to calculate the start and end address of our loaded kernel: From d07530d660e04e26c1aed1dbc3e55a3711eb76bb Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Thu, 12 Nov 2015 17:39:20 +0100 Subject: [PATCH 05/30] Rename new post to `Remapping the Kernel` --- .../{DRAFT-multiboot-info.md => DRAFT-remapping-the-kernel.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename posts/{DRAFT-multiboot-info.md => DRAFT-remapping-the-kernel.md} (97%) diff --git a/posts/DRAFT-multiboot-info.md b/posts/DRAFT-remapping-the-kernel.md similarity index 97% rename from posts/DRAFT-multiboot-info.md rename to posts/DRAFT-remapping-the-kernel.md index bf8b02c2..5b4c563c 100644 --- a/posts/DRAFT-multiboot-info.md +++ b/posts/DRAFT-remapping-the-kernel.md @@ -1,6 +1,6 @@ --- layout: post -title: 'The Multiboot Information Structure' +title: 'Remapping the Kernel' --- TODO From 73aa41e25e1b8891c11e607294a9029f60d4996a Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Thu, 12 Nov 2015 17:52:22 +0100 Subject: [PATCH 06/30] Pass multiboot pointer as argument to rust_main --- posts/DRAFT-remapping-the-kernel.md | 19 +++++++++++++++++++ src/arch/x86_64/boot.asm | 3 +++ src/arch/x86_64/long_mode_init.asm | 2 +- src/lib.rs | 2 +- 4 files changed, 24 insertions(+), 2 deletions(-) diff --git a/posts/DRAFT-remapping-the-kernel.md b/posts/DRAFT-remapping-the-kernel.md index 5b4c563c..72b8f9bd 100644 --- a/posts/DRAFT-remapping-the-kernel.md +++ b/posts/DRAFT-remapping-the-kernel.md @@ -8,6 +8,25 @@ TODO ## The Multiboot Information Structure When a Multiboot compliant bootloader loads a kernel, it passes a pointer to a boot information structure in the `ebx` register. We can use it to get information about available memory and loaded kernel sections. +First, we need to pass this pointer to our kernel as an argument to `rust_main`. To find out how arguments are passed to functions, we can look at the [calling convention of Linux]: + +[calling convention of Linux]: https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI + +> The first six integer or pointer arguments are passed in registers RDI, RSI, RDX, RCX, R8, and R9 + +So to pass the pointer to our kernel, we need to move it to `rdi` before calling the kernel. Since we're not using the `rdi`/`edi` register in our bootstrap code right now, we can simply set the `edi` register right after booting (in `boot.asm`): + +```nasm +start: + mov esp, stack_top + mov edi, ebx ; Move Multiboot info pointer to edi +``` +Now we can add the argument to our `rust_main`: + +```rust +pub extern fn rust_main(multiboot_information_address: usize) { ... } +``` + ## Start and End of Kernel We can now use the ELF section tag to calculate the start and end address of our loaded kernel: diff --git a/src/arch/x86_64/boot.asm b/src/arch/x86_64/boot.asm index c424e39a..c0fa155e 100644 --- a/src/arch/x86_64/boot.asm +++ b/src/arch/x86_64/boot.asm @@ -19,6 +19,9 @@ section .text bits 32 start: mov esp, stack_top + ; Move Multiboot info pointer to edi to pass it to the kernel. We must not + ; modify the `edi` register until the kernel it called. + mov edi, ebx call test_multiboot call test_cpuid diff --git a/src/arch/x86_64/long_mode_init.asm b/src/arch/x86_64/long_mode_init.asm index 1dfb0186..47927d58 100644 --- a/src/arch/x86_64/long_mode_init.asm +++ b/src/arch/x86_64/long_mode_init.asm @@ -20,7 +20,7 @@ bits 64 long_mode_start: call setup_SSE - ; call rust main + ; call rust main (with multiboot pointer in rdi) call rust_main .os_returned: ; rust main returned, print `OS returned!` diff --git a/src/lib.rs b/src/lib.rs index 5d469e52..741a5669 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -23,7 +23,7 @@ extern crate spin; mod vga_buffer; #[no_mangle] -pub extern fn rust_main() { +pub extern fn rust_main(multiboot_information_address: usize) { // ATTENTION: we have a very small stack and no guard page vga_buffer::clear_screen(); println!("Hello World{}", "!"); From 33fdbce5308640b9bfc733f11bd9fd75ff3f5035 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Fri, 13 Nov 2015 15:04:29 +0100 Subject: [PATCH 07/30] Print available memory areas using multiboot info --- Cargo.toml | 3 ++ posts/DRAFT-remapping-the-kernel.md | 46 +++++++++++++++++++++++++++++ src/lib.rs | 8 +++++ 3 files changed, 57 insertions(+) diff --git a/Cargo.toml b/Cargo.toml index 050c62a0..7eb63f79 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -9,3 +9,6 @@ crate-type = ["staticlib"] [dependencies] rlibc = "0.1.4" spin = "0.3.4" + +[dependencies.multiboot2] +git = "https://github.com/phil-opp/multiboot2-elf64" diff --git a/posts/DRAFT-remapping-the-kernel.md b/posts/DRAFT-remapping-the-kernel.md index 72b8f9bd..277289c7 100644 --- a/posts/DRAFT-remapping-the-kernel.md +++ b/posts/DRAFT-remapping-the-kernel.md @@ -27,6 +27,52 @@ Now we can add the argument to our `rust_main`: pub extern fn rust_main(multiboot_information_address: usize) { ... } ``` +Now we can use the [multiboot2-elf64] crate to query get some information about mapped kernel sections and available memory. I just wrote it for this blog post since I could not find any other Multiboot 2 crate. It's really ugly and incomplete, but it does its job. + +[multiboot2-elf64]: https://github.com/phil-opp/multiboot2-elf64 + +So let's add a dependency on the git repository in the `Cargo.toml`: + +```toml +... +[dependencies.multiboot2] +git = "https://github.com/phil-opp/multiboot2-elf64" +``` + +Now we can add `extern crate multiboot2` and use it to print available memory areas. + +### Available Memory +The boot information structure consists of various _tags_. The _memory map_ tag contains a list of all areas of available RAM. Special areas such as the VGA text buffer at `0xb8000` are not available. Note that some of the available memory is already used by our kernel and by the multiboot information structure itself. + +To print available memory areas, we can use the `multiboot2` crate in our `rust_main` as follows: + +```rust +let boot_info = unsafe{ multiboot2::load(multiboot_information_address) }; + +println!("memory areas:"); +for area in boot_info.memory_map_tag().unwrap().memory_areas() { + println!(" start: 0x{:x}, length: 0x{:x}", area.base_addr, area.length); +} +``` + +The `load` function is `unsafe` because it relies on a valid address. Since the memory tag is not required, the `memory_map_tag()` function returns an `Option`. The `memory_areas()` function returns the desired memory area iterator. + +The output looks like this: + +``` +Hello World! +memory areas: + start: 0x0, length: 0x9fc00 + start: 0x100000, length: 0x7ee0000 +``` +So we have one area from `0x0` to `0x9fc00`, which is a bit below the 1MiB mark. The second, bigger area starts at 1MiB and contains the rest of available memory. The area from `0x9fc00` to 1MiB is not available. For example the VGA text buffer at `0xb8000` is in that area. This is the reason for putting our kernel at 1MiB and not at e.g. `0x0`. + +If you give QEMU more than 4GiB of memory by passing `-m 5G`, you get another unusable area below the 4GiB mark. This memory is normally mapped to some hardware devices. See the [OSDev Wiki][Memory_map] for more information. + +[Memory_map]: http://wiki.osdev.org/Memory_Map_(x86) + +### Kernel ELF Sections + ## Start and End of Kernel We can now use the ELF section tag to calculate the start and end address of our loaded kernel: diff --git a/src/lib.rs b/src/lib.rs index 741a5669..12f2bb24 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -18,6 +18,7 @@ extern crate rlibc; extern crate spin; +extern crate multiboot2; #[macro_use] mod vga_buffer; @@ -28,6 +29,13 @@ pub extern fn rust_main(multiboot_information_address: usize) { vga_buffer::clear_screen(); println!("Hello World{}", "!"); + let boot_info = unsafe{ multiboot2::load(multiboot_information_address) }; + + println!("memory areas:"); + for area in boot_info.memory_map_tag().unwrap().memory_areas() { + println!(" start: 0x{:x}, length: 0x{:x}", area.base_addr, area.length); + } + loop{} } From 895d40cd13361798b15d45a3d1db8e86aa6e8c03 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Fri, 13 Nov 2015 19:56:27 +0100 Subject: [PATCH 08/30] Add section about kernel elf sections --- posts/DRAFT-remapping-the-kernel.md | 40 +++++++++++++++++++++++++++++ src/arch/x86_64/linker.ld | 10 +++++++- src/lib.rs | 6 +++++ 3 files changed, 55 insertions(+), 1 deletion(-) diff --git a/posts/DRAFT-remapping-the-kernel.md b/posts/DRAFT-remapping-the-kernel.md index 277289c7..3a397615 100644 --- a/posts/DRAFT-remapping-the-kernel.md +++ b/posts/DRAFT-remapping-the-kernel.md @@ -72,6 +72,46 @@ If you give QEMU more than 4GiB of memory by passing `-m 5G`, you get another un [Memory_map]: http://wiki.osdev.org/Memory_Map_(x86) ### Kernel ELF Sections +To read and print the sections of our kernel ELF file, we can use the _Elf-sections_ tag: + +```rust +println!("kernel sections:"); +for section in boot_info.elf_sections_tag().unwrap().sections() { + println!(" addr: 0x{:x}, size: 0x{:x}, flags: 0x{:x}", + section.addr, section.size, section.flags); +} +``` +This should print out the start address and size of all kernel sections. If the section is writable, the `0x1` is set in `flags`. The `0x4` bit marks an executable section and the `0x2` indicates that the section was loaded in memory. For example, the `.text` section is executable but not writable and the `.data` section just the opposite. + +But when we execute it, tons of really small sections are printed. We can use the `objdump -h build/kernel-x86_64.bin` command to list the sections with name. There seem to be over 200 sections and many of them start with `.text.*` or `.data.rel.ro.local.*`. The Rust compiler puts each function in an own `.text` subsection. To merge these subsections, we can update our linker script: + +``` +SECTIONS { + . = 1M; + + .boot : + { + KEEP(*(.multiboot_header)) + } + + .text : + { + *(.text .text.*) + } + + .rodata : { + *(.rodata .rodata.*) + } + + .data.rel.ro : { + *(.data.rel.ro.local*) *(.data.rel.ro .data.rel.ro.*) + } +} +``` + +These lines are taken from the default linker script of `ld`, which can be obtained through `ld ‑verbose`. Now there are only 12 sections left and we get a much more useful output: + +![qemu output](/images/qemu-memory-areas-and-kernel-sections.png) ## Start and End of Kernel We can now use the ELF section tag to calculate the start and end address of our loaded kernel: diff --git a/src/arch/x86_64/linker.ld b/src/arch/x86_64/linker.ld index 08222888..47be2d8f 100644 --- a/src/arch/x86_64/linker.ld +++ b/src/arch/x86_64/linker.ld @@ -27,6 +27,14 @@ SECTIONS { .text : { - *(.text) + *(.text .text.*) + } + + .rodata : { + *(.rodata .rodata.*) + } + + .data.rel.ro : { + *(.data.rel.ro.local*) *(.data.rel.ro .data.rel.ro.*) } } diff --git a/src/lib.rs b/src/lib.rs index 12f2bb24..210ad7f7 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -36,6 +36,12 @@ pub extern fn rust_main(multiboot_information_address: usize) { println!(" start: 0x{:x}, length: 0x{:x}", area.base_addr, area.length); } + println!("kernel sections:"); + for section in boot_info.elf_sections_tag().unwrap().sections() { + println!(" addr: 0x{:x}, size: 0x{:x}, flags: 0x{:x}", + section.addr, section.size, section.flags); + } + loop{} } From 2d1c801c7230e2f202640b0701c4dc89eebad835 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Fri, 13 Nov 2015 23:54:13 +0100 Subject: [PATCH 09/30] Rename post to `Allocating Frames` --- ...DRAFT-remapping-the-kernel.md => DRAFT-allocating-frames.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename posts/{DRAFT-remapping-the-kernel.md => DRAFT-allocating-frames.md} (99%) diff --git a/posts/DRAFT-remapping-the-kernel.md b/posts/DRAFT-allocating-frames.md similarity index 99% rename from posts/DRAFT-remapping-the-kernel.md rename to posts/DRAFT-allocating-frames.md index 3a397615..0e4a9484 100644 --- a/posts/DRAFT-remapping-the-kernel.md +++ b/posts/DRAFT-allocating-frames.md @@ -1,6 +1,6 @@ --- layout: post -title: 'Remapping the Kernel' +title: 'Allocating Frames' --- TODO From fc389c9e9e2ce7b4bea1168893631af1336cc1be Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 13:20:18 +0100 Subject: [PATCH 10/30] Increase stack size to 4096 --- posts/DRAFT-allocating-frames.md | 11 +++++++++++ src/arch/x86_64/boot.asm | 2 +- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 0e4a9484..dbe0d28c 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -5,6 +5,17 @@ title: 'Allocating Frames' TODO +## Preparation +We still have a really tiny stack of 64 bytes, which won't suffice for this post. So we will increase it to 4096 (one page) in `boot.asm`: + +```asm +section .bss +... +stack_bottom: + resb 4096 +stack_top: +``` + ## The Multiboot Information Structure When a Multiboot compliant bootloader loads a kernel, it passes a pointer to a boot information structure in the `ebx` register. We can use it to get information about available memory and loaded kernel sections. diff --git a/src/arch/x86_64/boot.asm b/src/arch/x86_64/boot.asm index c0fa155e..66cba279 100644 --- a/src/arch/x86_64/boot.asm +++ b/src/arch/x86_64/boot.asm @@ -152,7 +152,7 @@ p3_table: p2_table: resb 4096 stack_bottom: - resb 64 + resb 4096 stack_top: section .rodata From f2b91d3d617e4efda2d38cb00521f3acb3dcf228 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 10:59:48 +0100 Subject: [PATCH 11/30] Improve panic handler --- posts/DRAFT-allocating-frames.md | 24 ++++++++++++++++++++++++ src/lib.rs | 6 +++++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index dbe0d28c..edef87ab 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -82,6 +82,30 @@ If you give QEMU more than 4GiB of memory by passing `-m 5G`, you get another un [Memory_map]: http://wiki.osdev.org/Memory_Map_(x86) +### Handling Panics +We used `unwrap` in the code above, which will panic if there is no memory map tag. But our current panic handler just loops without printing any error message. Of course we could replace `unwrap` by a `match`, but we should fix the panic handler nonetheless: + +```rust +#[lang = "panic_fmt"] +extern fn panic_fmt() -> ! { + println!("PANIC"); + loop{} +} +``` +Now we get a `PANIC` message. But we can do even better. The `panic_fmt` function has actually some arguments: + +```rust +#[lang = "panic_fmt"] +extern fn panic_fmt(fmt: core::fmt::Arguments, file: &str, line: u32) -> ! { + println!("\n\nPANIC in {} at line {}:", file, line); + println!(" {}", fmt); + loop{} +} +``` +Be careful with these arguments as the compiler does not check arguments for `lang_items`. + +You can try our new panic handler by inserting a `panic` somewhere. Now we get the panic message and the causing source line. + ### Kernel ELF Sections To read and print the sections of our kernel ELF file, we can use the _Elf-sections_ tag: diff --git a/src/lib.rs b/src/lib.rs index 210ad7f7..91145492 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -51,4 +51,8 @@ extern fn eh_personality() {} #[cfg(not(test))] #[lang = "panic_fmt"] -extern fn panic_fmt() -> ! {loop{}} +extern fn panic_fmt(fmt: core::fmt::Arguments, file: &str, line: u32) -> ! { + println!("\n\nPANIC in {} at line {}:", file, line); + println!(" {}", fmt); + loop{} +} From 718db100cd3d628d6f078c494845575a14fb08d6 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 11:11:52 +0100 Subject: [PATCH 12/30] Add let bindings for tags --- posts/DRAFT-allocating-frames.md | 11 +++++++---- src/lib.rs | 6 ++++-- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index edef87ab..0fea30b3 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -59,13 +59,13 @@ To print available memory areas, we can use the `multiboot2` crate in our `rust_ ```rust let boot_info = unsafe{ multiboot2::load(multiboot_information_address) }; +let memory_map_tag = boot_info.memory_map_tag().expect("Memory map tag required"); println!("memory areas:"); -for area in boot_info.memory_map_tag().unwrap().memory_areas() { +for area in emory_map_tag.memory_areas() { println!(" start: 0x{:x}, length: 0x{:x}", area.base_addr, area.length); } ``` - The `load` function is `unsafe` because it relies on a valid address. Since the memory tag is not required, the `memory_map_tag()` function returns an `Option`. The `memory_areas()` function returns the desired memory area iterator. The output looks like this: @@ -83,7 +83,7 @@ If you give QEMU more than 4GiB of memory by passing `-m 5G`, you get another un [Memory_map]: http://wiki.osdev.org/Memory_Map_(x86) ### Handling Panics -We used `unwrap` in the code above, which will panic if there is no memory map tag. But our current panic handler just loops without printing any error message. Of course we could replace `unwrap` by a `match`, but we should fix the panic handler nonetheless: +We used `expect` in the code above, which will panic if there is no memory map tag. But our current panic handler just loops without printing any error message. Of course we could replace `expect` by a `match`, but we should fix the panic handler nonetheless: ```rust #[lang = "panic_fmt"] @@ -110,8 +110,11 @@ You can try our new panic handler by inserting a `panic` somewhere. Now we get t To read and print the sections of our kernel ELF file, we can use the _Elf-sections_ tag: ```rust +let elf_sections_tag = boot_info.elf_sections_tag() + .expect("Elf-sections tag required"); + println!("kernel sections:"); -for section in boot_info.elf_sections_tag().unwrap().sections() { +for section in elf_sections_tag.sections() { println!(" addr: 0x{:x}, size: 0x{:x}, flags: 0x{:x}", section.addr, section.size, section.flags); } diff --git a/src/lib.rs b/src/lib.rs index 91145492..ebd6a591 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -30,14 +30,16 @@ pub extern fn rust_main(multiboot_information_address: usize) { println!("Hello World{}", "!"); let boot_info = unsafe{ multiboot2::load(multiboot_information_address) }; + let memory_map_tag = boot_info.memory_map_tag().expect("Memory map tag required"); + let elf_sections_tag = boot_info.elf_sections_tag().expect("Memory map tag required"); println!("memory areas:"); - for area in boot_info.memory_map_tag().unwrap().memory_areas() { + for area in memory_map_tag.memory_areas() { println!(" start: 0x{:x}, length: 0x{:x}", area.base_addr, area.length); } println!("kernel sections:"); - for section in boot_info.elf_sections_tag().unwrap().sections() { + for section in elf_sections_tag.sections() { println!(" addr: 0x{:x}, size: 0x{:x}, flags: 0x{:x}", section.addr, section.size, section.flags); } From ccaa2ed6453483019180748cdbe941bd7dd83c0c Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 11:30:34 +0100 Subject: [PATCH 13/30] Add section about start and end of kernel/multiboot --- posts/DRAFT-allocating-frames.md | 22 ++++++++++++++++++++-- src/lib.rs | 9 +++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 0fea30b3..696d9355 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -151,10 +151,28 @@ These lines are taken from the default linker script of `ld`, which can be obtai ![qemu output](/images/qemu-memory-areas-and-kernel-sections.png) -## Start and End of Kernel +### Start and End of Kernel We can now use the ELF section tag to calculate the start and end address of our loaded kernel: -TODO +```rust +let kernel_start = elf_sections_tag.sections().map(|s| s.addr) + .min().unwrap(); +let kernel_end = elf_sections_tag.sections().map(|s| s.addr + s.size) + .max().unwrap(); +``` +The other used memory area is the Multiboot Information structure: + +```rust +let multiboot_start = multiboot_information_address; +let multiboot_end = multiboot_start + (boot_info.total_size as usize); +``` +Printing these numbers gives us: + +``` +kernel_start: 0x100000, kernel_end: 0x11a168 +multiboot_start: 0x11d400, multiboot_end: 0x11d9c8 +``` +So the kernel starts at 1MiB (like expected) and is about 105 KiB in size. The multiboot information structure was placed at `0x11d400` by GRUB and needs 1480 bytes. Of course your numbers could be a bit different due to different versions of Rust or GRUB. ## A frame allocator When we create a paging module in the next post, we will need to map virtual pages to free physical frames. So we will need some kind of allocator that keeps track of physical frames and gives us a free one when needed. We can use the memory tag to write such a frame allocator. diff --git a/src/lib.rs b/src/lib.rs index ebd6a591..68ef0203 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -44,6 +44,15 @@ pub extern fn rust_main(multiboot_information_address: usize) { section.addr, section.size, section.flags); } + let kernel_start = elf_sections_tag.sections().map(|s| s.addr).min().unwrap(); + let kernel_end = elf_sections_tag.sections().map(|s| s.addr + s.size).max().unwrap(); + + let multiboot_start = multiboot_information_address; + let multiboot_end = multiboot_start + (boot_info.total_size as usize); + + println!("kernel start: 0x{:x}, kernel end: 0x{:x}", kernel_start, kernel_end); + println!("multiboot start: 0x{:x}, multiboot end: 0x{:x}", multiboot_start, multiboot_end); + loop{} } From 6f8a21eba652ce11a95f96fb19f71e2742ca2f19 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 11:41:16 +0100 Subject: [PATCH 14/30] Create a memory module with a Frame struct --- posts/DRAFT-allocating-frames.md | 25 ++++++++++++++++++++++++- src/lib.rs | 1 + src/memory/mod.rs | 12 ++++++++++++ 3 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 src/memory/mod.rs diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 696d9355..de15f421 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -175,7 +175,30 @@ multiboot_start: 0x11d400, multiboot_end: 0x11d9c8 So the kernel starts at 1MiB (like expected) and is about 105 KiB in size. The multiboot information structure was placed at `0x11d400` by GRUB and needs 1480 bytes. Of course your numbers could be a bit different due to different versions of Rust or GRUB. ## A frame allocator -When we create a paging module in the next post, we will need to map virtual pages to free physical frames. So we will need some kind of allocator that keeps track of physical frames and gives us a free one when needed. We can use the memory tag to write such a frame allocator. +When we create a paging module in the next post, we will need to map virtual pages to free physical frames. So we will need some kind of allocator that keeps track of physical frames and gives us a free one when needed. We can use the information about memory areas to write such a frame allocator. + +### A Memory Module +First we create a memory module with a `Frame` type (`src/memory/mod.rs`): + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub struct Frame { + number: usize, +} +``` +We use `usize` here since the number of frames depends on the memory size. The long `derive` line makes frames printable and comparable. (Don't forget to add the `mod memory` line to `src/lib.rs`.) + +To make it easy to get the corresponding frame for a physical address, we add a `containing_address` method: + +```rust +pub const PAGE_SIZE: usize = 4096; + +impl Frame { + fn containing_address(address: usize) -> Frame { + Frame{ number: address / PAGE_SIZE } + } +} +``` The allocator struct looks like this: diff --git a/src/lib.rs b/src/lib.rs index 68ef0203..723510b2 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -22,6 +22,7 @@ extern crate multiboot2; #[macro_use] mod vga_buffer; +mod memory; #[no_mangle] pub extern fn rust_main(multiboot_information_address: usize) { diff --git a/src/memory/mod.rs b/src/memory/mod.rs new file mode 100644 index 00000000..b60d5022 --- /dev/null +++ b/src/memory/mod.rs @@ -0,0 +1,12 @@ +pub const PAGE_SIZE: usize = 4096; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub struct Frame { + number: usize, +} + +impl Frame { + fn containing_address(address: usize) -> Frame { + Frame{ number: address / PAGE_SIZE } + } +} From a2047bc70a0baeb32df5767564ccd96d6b36f123 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 15:41:11 +0100 Subject: [PATCH 15/30] Add an AreaFrameAllocator --- posts/DRAFT-allocating-frames.md | 83 ++++++++++++++++++++++++++---- src/lib.rs | 5 +- src/memory/area_frame_allocator.rs | 67 ++++++++++++++++++++++++ src/memory/mod.rs | 4 ++ 4 files changed, 148 insertions(+), 11 deletions(-) create mode 100644 src/memory/area_frame_allocator.rs diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index de15f421..addceb8c 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -200,24 +200,87 @@ impl Frame { } ``` -The allocator struct looks like this: +### The Allocator +Now we can put everything together and create the frame allocator. It looks like this: ```rust -struct AreaFrameAllocator { - first_used_frame: Frame, - last_used_frame: Frame, - current_area: Option, +use memory::Frame; +use multiboot2::{MemoryAreaIter, MemoryArea}; + +pub struct AreaFrameAllocator { + next_free_frame: Frame, + current_area: Option<&'static MemoryArea>, areas: MemoryAreaIter, + kernel_start: Frame, + kernel_end: Frame, + multiboot_start: Frame, + multiboot_end: Frame, } ``` -TODO +The `next_free_frame` field is a simple counter that is increased every time we return a frame. The `current_area` field holds the memory area that contains `next_free_frame`. If `next_free_frame` leaves this area, we will look for the next one in `areas`. The `{kernel, multiboot}_{start, end}` fields are used to avoid returning already used fields. -To allocate a frame we try to find one in the current area and update the first/last used bounds. If we can't find one, we look for the new area with the minimal start address, that still contains free frames. If the current area is `None`, there are no free frames left. +```rust +pub fn new(kernel_start: usize, kernel_end: usize, + multiboot_start: usize, multiboot_end: usize, + memory_areas: MemoryAreaIter) -> AreaFrameAllocator +{ + let mut allocator = AreaFrameAllocator { + next_free_frame: Frame::containing_address(0), + current_area: None, + areas: memory_areas, + kernel_start: Frame::containing_address(kernel_start), + kernel_end: Frame::containing_address(kernel_end), + multiboot_start: Frame::containing_address(multiboot_start), + multiboot_end: Frame::containing_address(multiboot_end), + }; + allocator.choose_next_area(); + allocator +} +``` -TODO +```rust +fn choose_next_area(&mut self) { + self.current_area = self.areas.clone().filter(|area| { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) >= self.next_free_frame + }).min_by(|area| area.base_addr); +} +``` -### Unit Tests -TODO +```rust +pub fn allocate_frame(&mut self) -> Option { + match self.current_area { + None => None, + Some(area) => { + let frame = self.next_free_frame; + let current_area_last_frame = { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) + }; + + if frame > current_area_last_frame { + self.choose_next_area() + } else if frame >= self.kernel_start && + frame <= self.kernel_end + { + self.next_free_frame = Frame { + number: self.kernel_end.number + 1 + } + } else if frame >= self.multiboot_start && + frame <= self.multiboot_end + { + self.next_free_frame = Frame { + number: self.multiboot_end.number + 1 + } + } else { + self.next_free_frame.number += 1; + return Some(frame); + } + self.allocate_frame() + } + } +} +``` ## Remapping the Kernel Sections We can use the ELF section tag to write a skeleton that remaps the kernel correctly: diff --git a/src/lib.rs b/src/lib.rs index 723510b2..4c0ee4c1 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -13,7 +13,7 @@ // limitations under the License. #![feature(no_std, lang_items)] -#![feature(const_fn, unique, core_str_ext)] +#![feature(const_fn, unique, core_str_ext, iter_cmp)] #![no_std] extern crate rlibc; @@ -54,6 +54,9 @@ pub extern fn rust_main(multiboot_information_address: usize) { println!("kernel start: 0x{:x}, kernel end: 0x{:x}", kernel_start, kernel_end); println!("multiboot start: 0x{:x}, multiboot end: 0x{:x}", multiboot_start, multiboot_end); + let mut frame_allocator = memory::AreaFrameAllocator::new(kernel_start as usize, + kernel_end as usize, multiboot_start, multiboot_end, memory_map_tag.memory_areas()); + loop{} } diff --git a/src/memory/area_frame_allocator.rs b/src/memory/area_frame_allocator.rs new file mode 100644 index 00000000..ade79217 --- /dev/null +++ b/src/memory/area_frame_allocator.rs @@ -0,0 +1,67 @@ +use memory::Frame; +use multiboot2::{MemoryAreaIter, MemoryArea}; + +/// A frame allocator that uses the memory areas from the multiboot information structure as +/// source. The {kernel, multiboot}_{start, end} fields are used to avoid returning memory that is +/// already in use. +/// +/// `kernel_end` and `multiboot_end` are _inclusive_ bounds. +pub struct AreaFrameAllocator { + next_free_frame: Frame, + current_area: Option<&'static MemoryArea>, + areas: MemoryAreaIter, + kernel_start: Frame, + kernel_end: Frame, + multiboot_start: Frame, + multiboot_end: Frame, +} + +impl AreaFrameAllocator { + pub fn new(kernel_start: usize, kernel_end: usize, multiboot_start: usize, multiboot_end: usize, + memory_areas: MemoryAreaIter) -> AreaFrameAllocator + { + let mut allocator = AreaFrameAllocator { + next_free_frame: Frame::containing_address(0), + current_area: None, + areas: memory_areas, + kernel_start: Frame::containing_address(kernel_start), + kernel_end: Frame::containing_address(kernel_end), + multiboot_start: Frame::containing_address(multiboot_start), + multiboot_end: Frame::containing_address(multiboot_end), + }; + allocator.choose_next_area(); + allocator + } + + pub fn allocate_frame(&mut self) -> Option { + match self.current_area { + None => None, + Some(area) => { + let frame = self.next_free_frame; + let current_area_last_frame = { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) + }; + + if frame > current_area_last_frame { + self.choose_next_area() + } else if frame >= self.kernel_start && frame <= self.kernel_end { + self.next_free_frame = Frame{ number: self.kernel_end.number + 1 } + } else if frame >= self.multiboot_start && frame <= self.multiboot_end { + self.next_free_frame = Frame{ number: self.multiboot_end.number + 1 } + } else { + self.next_free_frame.number += 1; + return Some(frame); + } + self.allocate_frame() + } + } + } + + fn choose_next_area(&mut self) { + self.current_area = self.areas.clone().filter(|area| { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) >= self.next_free_frame + }).min_by(|area| area.base_addr); + } +} diff --git a/src/memory/mod.rs b/src/memory/mod.rs index b60d5022..496244c4 100644 --- a/src/memory/mod.rs +++ b/src/memory/mod.rs @@ -1,3 +1,7 @@ +pub use self::area_frame_allocator::AreaFrameAllocator; + +mod area_frame_allocator; + pub const PAGE_SIZE: usize = 4096; #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] From 75988f13248aaacb66e25f88a47141a6cb8c5917 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 15:44:47 +0100 Subject: [PATCH 16/30] Test: allocate all frames --- src/lib.rs | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/lib.rs b/src/lib.rs index 4c0ee4c1..92dc1f03 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -57,6 +57,13 @@ pub extern fn rust_main(multiboot_information_address: usize) { let mut frame_allocator = memory::AreaFrameAllocator::new(kernel_start as usize, kernel_end as usize, multiboot_start, multiboot_end, memory_map_tag.memory_areas()); + for i in 0.. { + if let None = frame_allocator.allocate_frame() { + println!("allocated {} frames", i); + break; + } + } + loop{} } From 8304439c8249d4467d9f134fecc42b1863e06b2d Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 19:21:23 +0100 Subject: [PATCH 17/30] Add a FrameAllocator trait --- posts/DRAFT-allocating-frames.md | 12 +++++++++++- src/lib.rs | 1 + src/memory/area_frame_allocator.rs | 20 ++++++++++++-------- src/memory/mod.rs | 5 +++++ 4 files changed, 29 insertions(+), 9 deletions(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index addceb8c..393428c2 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -200,6 +200,16 @@ impl Frame { } ``` +We also add a `FrameAllocator` trait: + +```rust +pub trait FrameAllocator { + fn allocate_frame(&mut self) -> Option; + fn deallocate_frame(&mut self, frame: Frame); +} +``` +This allows us to create another, more advanced frame allocator in the future. + ### The Allocator Now we can put everything together and create the frame allocator. It looks like this: @@ -248,7 +258,7 @@ fn choose_next_area(&mut self) { ``` ```rust -pub fn allocate_frame(&mut self) -> Option { +fn allocate_frame(&mut self) -> Option { match self.current_area { None => None, Some(area) => { diff --git a/src/lib.rs b/src/lib.rs index 92dc1f03..e0ae38fc 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -58,6 +58,7 @@ pub extern fn rust_main(multiboot_information_address: usize) { kernel_end as usize, multiboot_start, multiboot_end, memory_map_tag.memory_areas()); for i in 0.. { + use memory::FrameAllocator; if let None = frame_allocator.allocate_frame() { println!("allocated {} frames", i); break; diff --git a/src/memory/area_frame_allocator.rs b/src/memory/area_frame_allocator.rs index ade79217..4a534f4d 100644 --- a/src/memory/area_frame_allocator.rs +++ b/src/memory/area_frame_allocator.rs @@ -1,4 +1,4 @@ -use memory::Frame; +use memory::{Frame, FrameAllocator}; use multiboot2::{MemoryAreaIter, MemoryArea}; /// A frame allocator that uses the memory areas from the multiboot information structure as @@ -33,7 +33,16 @@ impl AreaFrameAllocator { allocator } - pub fn allocate_frame(&mut self) -> Option { + fn choose_next_area(&mut self) { + self.current_area = self.areas.clone().filter(|area| { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) >= self.next_free_frame + }).min_by(|area| area.base_addr); + } +} + +impl FrameAllocator for AreaFrameAllocator { + fn allocate_frame(&mut self) -> Option { match self.current_area { None => None, Some(area) => { @@ -58,10 +67,5 @@ impl AreaFrameAllocator { } } - fn choose_next_area(&mut self) { - self.current_area = self.areas.clone().filter(|area| { - let address = area.base_addr + area.length - 1; - Frame::containing_address(address as usize) >= self.next_free_frame - }).min_by(|area| area.base_addr); - } + fn deallocate_frame(&mut self, _frame: Frame) {unimplemented!()} } diff --git a/src/memory/mod.rs b/src/memory/mod.rs index 496244c4..385c574f 100644 --- a/src/memory/mod.rs +++ b/src/memory/mod.rs @@ -14,3 +14,8 @@ impl Frame { Frame{ number: address / PAGE_SIZE } } } + +pub trait FrameAllocator { + fn allocate_frame(&mut self) -> Option; + fn deallocate_frame(&mut self, frame: Frame); +} From 16078431dbf9ec316d271d81261a73c82dcec0b6 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sat, 14 Nov 2015 20:03:03 +0100 Subject: [PATCH 18/30] Add basic description for frame allocator methods --- posts/DRAFT-allocating-frames.md | 111 +++++++++++++++++------------ src/memory/area_frame_allocator.rs | 44 +++++++----- 2 files changed, 89 insertions(+), 66 deletions(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 393428c2..62ff0458 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -211,10 +211,10 @@ pub trait FrameAllocator { This allows us to create another, more advanced frame allocator in the future. ### The Allocator -Now we can put everything together and create the frame allocator. It looks like this: +Now we can put everything together and create the actual frame allocator. It looks like this: ```rust -use memory::Frame; +use memory::{Frame, FrameAllocator}; use multiboot2::{MemoryAreaIter, MemoryArea}; pub struct AreaFrameAllocator { @@ -227,7 +227,67 @@ pub struct AreaFrameAllocator { multiboot_end: Frame, } ``` -The `next_free_frame` field is a simple counter that is increased every time we return a frame. The `current_area` field holds the memory area that contains `next_free_frame`. If `next_free_frame` leaves this area, we will look for the next one in `areas`. The `{kernel, multiboot}_{start, end}` fields are used to avoid returning already used fields. +The `next_free_frame` field is a simple counter that is increased every time we return a frame. The `current_area` field holds the memory area that contains `next_free_frame`. If `next_free_frame` leaves this area, we will look for the next one in `areas`. When there are no areas left, all frames are used and `current_area` becomes `None`. The `{kernel, multiboot}_{start, end}` fields are used to avoid returning already used fields. + +To implement the `FrameAllocator` trait, we need to implement the `allocate_frame` and the `deallocate_frame` methods. The former looks like this: + +```rust +fn allocate_frame(&mut self) -> Option { + if let Some(area) = self.current_area { + let frame = self.next_free_frame; + + // the last frame of the current area + let current_area_last_frame = { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) + }; + + if frame > current_area_last_frame { + // all frames of current area are used, switch to next area + self.choose_next_area(); + } else if frame >= self.kernel_start && frame <= self.kernel_end { + // `frame` is used by the kernel + self.next_free_frame = Frame { + number: self.kernel_end.number + 1 + }; + } else if frame >= self.multiboot_start && frame <= self.multiboot_end { + // `frame` is used by the multiboot information structure + self.next_free_frame = Frame { + number: self.multiboot_end.number + 1 + }; + } else { + // frame is unused, increment `next_free_frame` and return it + self.next_free_frame.number += 1; + return Some(frame); + } + // `frame` was not valid, try it again with the updated `next_free_frame` + self.allocate_frame() + } else { + None // no free frames left + } +} +``` +The `choose_next_area` method isn't part of the trait and thus goes into an `impl AreaFrameAllocator` block: + +```rust +fn choose_next_area(&mut self) { + self.current_area = self.areas.clone().filter(|area| { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) >= self.next_free_frame + }).min_by(|area| area.base_addr); +} +``` +This function chooses the area with the minimal base address that still has free frames, i.e. `next_free_frame` is smaller than its last frame. Note that we need to clone the iterator because the order of areas in the memory map isn't specified. + +We don't have a data structure to store free frames, so we can't implement `deallocate_frame` reasonably. Thus we use the `unimplemented` macro, which just panics when called: + +```rust +fn deallocate_frame(&mut self, _frame: Frame) { + unimplemented!() +} +``` + +Now we only need a constructor function: ```rust pub fn new(kernel_start: usize, kernel_end: usize, @@ -247,50 +307,7 @@ pub fn new(kernel_start: usize, kernel_end: usize, allocator } ``` - -```rust -fn choose_next_area(&mut self) { - self.current_area = self.areas.clone().filter(|area| { - let address = area.base_addr + area.length - 1; - Frame::containing_address(address as usize) >= self.next_free_frame - }).min_by(|area| area.base_addr); -} -``` - -```rust -fn allocate_frame(&mut self) -> Option { - match self.current_area { - None => None, - Some(area) => { - let frame = self.next_free_frame; - let current_area_last_frame = { - let address = area.base_addr + area.length - 1; - Frame::containing_address(address as usize) - }; - - if frame > current_area_last_frame { - self.choose_next_area() - } else if frame >= self.kernel_start && - frame <= self.kernel_end - { - self.next_free_frame = Frame { - number: self.kernel_end.number + 1 - } - } else if frame >= self.multiboot_start && - frame <= self.multiboot_end - { - self.next_free_frame = Frame { - number: self.multiboot_end.number + 1 - } - } else { - self.next_free_frame.number += 1; - return Some(frame); - } - self.allocate_frame() - } - } -} -``` +Note that we call `choose_next_area` manually here because `allocate_frame` returns `None` as soon as `current_area` is `None`. ## Remapping the Kernel Sections We can use the ELF section tag to write a skeleton that remaps the kernel correctly: diff --git a/src/memory/area_frame_allocator.rs b/src/memory/area_frame_allocator.rs index 4a534f4d..d36b3705 100644 --- a/src/memory/area_frame_allocator.rs +++ b/src/memory/area_frame_allocator.rs @@ -43,27 +43,33 @@ impl AreaFrameAllocator { impl FrameAllocator for AreaFrameAllocator { fn allocate_frame(&mut self) -> Option { - match self.current_area { - None => None, - Some(area) => { - let frame = self.next_free_frame; - let current_area_last_frame = { - let address = area.base_addr + area.length - 1; - Frame::containing_address(address as usize) - }; + if let Some(area) = self.current_area { + let frame = self.next_free_frame; - if frame > current_area_last_frame { - self.choose_next_area() - } else if frame >= self.kernel_start && frame <= self.kernel_end { - self.next_free_frame = Frame{ number: self.kernel_end.number + 1 } - } else if frame >= self.multiboot_start && frame <= self.multiboot_end { - self.next_free_frame = Frame{ number: self.multiboot_end.number + 1 } - } else { - self.next_free_frame.number += 1; - return Some(frame); - } - self.allocate_frame() + // the last frame of the current area + let current_area_last_frame = { + let address = area.base_addr + area.length - 1; + Frame::containing_address(address as usize) + }; + + if frame > current_area_last_frame { + // all frames of current area are used, switch to next area + self.choose_next_area(); + } else if frame >= self.kernel_start && frame <= self.kernel_end { + // `frame` is used by the kernel + self.next_free_frame = Frame{ number: self.kernel_end.number + 1 }; + } else if frame >= self.multiboot_start && frame <= self.multiboot_end { + // `frame` is used by the multiboot information structure + self.next_free_frame = Frame{ number: self.multiboot_end.number + 1 }; + } else { + // frame is unused, increment `next_free_frame` and return it + self.next_free_frame.number += 1; + return Some(frame); } + // `frame` was not valid, try it again with the updated `next_free_frame` + self.allocate_frame() + } else { + None // no free frames left } } From 939d33efce1ef90b519277a603b4d51af97ae600 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:00:08 +0100 Subject: [PATCH 19/30] Many improvements to `allocating frames` post --- posts/DRAFT-allocating-frames.md | 66 +++++++++++++++++++++++++------- 1 file changed, 52 insertions(+), 14 deletions(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 62ff0458..c859b503 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -66,7 +66,7 @@ for area in emory_map_tag.memory_areas() { println!(" start: 0x{:x}, length: 0x{:x}", area.base_addr, area.length); } ``` -The `load` function is `unsafe` because it relies on a valid address. Since the memory tag is not required, the `memory_map_tag()` function returns an `Option`. The `memory_areas()` function returns the desired memory area iterator. +The `load` function is `unsafe` because it relies on a valid address. Since the memory tag is not required by the Multiboot specification, the `memory_map_tag()` function returns an `Option`. The `memory_areas()` function returns the desired memory area iterator. The output looks like this: @@ -76,7 +76,7 @@ memory areas: start: 0x0, length: 0x9fc00 start: 0x100000, length: 0x7ee0000 ``` -So we have one area from `0x0` to `0x9fc00`, which is a bit below the 1MiB mark. The second, bigger area starts at 1MiB and contains the rest of available memory. The area from `0x9fc00` to 1MiB is not available. For example the VGA text buffer at `0xb8000` is in that area. This is the reason for putting our kernel at 1MiB and not at e.g. `0x0`. +So we have one area from `0x0` to `0x9fc00`, which is a bit below the 1MiB mark. The second, bigger area starts at 1MiB and contains the rest of available memory. The area from `0x9fc00` to 1MiB is not available since it contains for example the VGA text buffer at `0xb8000`. This is the reason for putting our kernel at 1MiB and not at e.g. `0x0`. If you give QEMU more than 4GiB of memory by passing `-m 5G`, you get another unusable area below the 4GiB mark. This memory is normally mapped to some hardware devices. See the [OSDev Wiki][Memory_map] for more information. @@ -102,7 +102,7 @@ extern fn panic_fmt(fmt: core::fmt::Arguments, file: &str, line: u32) -> ! { loop{} } ``` -Be careful with these arguments as the compiler does not check arguments for `lang_items`. +Be careful with these arguments as the compiler does not check the function signature for `lang_items`. You can try our new panic handler by inserting a `panic` somewhere. Now we get the panic message and the causing source line. @@ -119,9 +119,9 @@ for section in elf_sections_tag.sections() { section.addr, section.size, section.flags); } ``` -This should print out the start address and size of all kernel sections. If the section is writable, the `0x1` is set in `flags`. The `0x4` bit marks an executable section and the `0x2` indicates that the section was loaded in memory. For example, the `.text` section is executable but not writable and the `.data` section just the opposite. +This should print out the start address and size of all kernel sections. If the section is writable, the `0x1` bit is set in `flags`. The `0x4` bit marks an executable section and the `0x2` bit indicates that the section was loaded in memory. For example, the `.text` section is executable but not writable and the `.data` section just the opposite. -But when we execute it, tons of really small sections are printed. We can use the `objdump -h build/kernel-x86_64.bin` command to list the sections with name. There seem to be over 200 sections and many of them start with `.text.*` or `.data.rel.ro.local.*`. The Rust compiler puts each function in an own `.text` subsection. To merge these subsections, we can update our linker script: +But when we execute it, tons of really small sections are printed. We can use the `objdump -h build/kernel-x86_64.bin` command to list the sections with name. There seem to be over 200 sections and many of them start with `.text.*` or `.data.rel.ro.local.*`. This is because the Rust compiler puts e.g. each function in an own `.text` subsection. To merge these subsections, we can update our linker script: ``` SECTIONS { @@ -147,10 +147,14 @@ SECTIONS { } ``` -These lines are taken from the default linker script of `ld`, which can be obtained through `ld ‑verbose`. Now there are only 12 sections left and we get a much more useful output: +These lines are taken from the default linker script of `ld`, which can be obtained through `ld ‑verbose`. The `.text` _output_ section contains now all `.text.*` _input_ sections of the static library (and the same applies for the `.rodata` and `.data.rel.ro` sections). + +Now there are only 12 sections left and we get a much more useful output: ![qemu output](/images/qemu-memory-areas-and-kernel-sections.png) +If you like, you can compare this output to the `objdump -h build/kernel-x86_64.bin` output. You will see that the start addresses and sizes match exactly for each section. The sections with flags `0x0` are mostly debug sections, so they don't need to be loaded. And the last few sections of the QEMU output aren't in the `objdump` output because they are special sections such as string tables. + ### Start and End of Kernel We can now use the ELF section tag to calculate the start and end address of our loaded kernel: @@ -172,7 +176,7 @@ Printing these numbers gives us: kernel_start: 0x100000, kernel_end: 0x11a168 multiboot_start: 0x11d400, multiboot_end: 0x11d9c8 ``` -So the kernel starts at 1MiB (like expected) and is about 105 KiB in size. The multiboot information structure was placed at `0x11d400` by GRUB and needs 1480 bytes. Of course your numbers could be a bit different due to different versions of Rust or GRUB. +So the kernel starts at 1MiB (like expected) and is about 105 KiB in size. The multiboot information structure was placed at `0x11d400` by GRUB and needs 1480 bytes. Of course your numbers could be a bit different due to different versions of Rust or GRUB (or some differences in the source code). ## A frame allocator When we create a paging module in the next post, we will need to map virtual pages to free physical frames. So we will need some kind of allocator that keeps track of physical frames and gives us a free one when needed. We can use the information about memory areas to write such a frame allocator. @@ -211,7 +215,7 @@ pub trait FrameAllocator { This allows us to create another, more advanced frame allocator in the future. ### The Allocator -Now we can put everything together and create the actual frame allocator. It looks like this: +Now we can put everything together and create the actual frame allocator. Therefor we create a `src/memory/area_frame_allocator.rs` submodule. The allocator struct looks like this: ```rust use memory::{Frame, FrameAllocator}; @@ -227,7 +231,7 @@ pub struct AreaFrameAllocator { multiboot_end: Frame, } ``` -The `next_free_frame` field is a simple counter that is increased every time we return a frame. The `current_area` field holds the memory area that contains `next_free_frame`. If `next_free_frame` leaves this area, we will look for the next one in `areas`. When there are no areas left, all frames are used and `current_area` becomes `None`. The `{kernel, multiboot}_{start, end}` fields are used to avoid returning already used fields. +The `next_free_frame` field is a simple counter that is increased every time we return a frame. It's initialized to `0` and every frame below it counts as used. The `current_area` field holds the memory area that contains `next_free_frame`. If `next_free_frame` leaves this area, we will look for the next one in `areas`. When there are no areas left, all frames are used and `current_area` becomes `None`. The `{kernel, multiboot}_{start, end}` fields are used to avoid returning already used fields. To implement the `FrameAllocator` trait, we need to implement the `allocate_frame` and the `deallocate_frame` methods. The former looks like this: @@ -277,9 +281,9 @@ fn choose_next_area(&mut self) { }).min_by(|area| area.base_addr); } ``` -This function chooses the area with the minimal base address that still has free frames, i.e. `next_free_frame` is smaller than its last frame. Note that we need to clone the iterator because the order of areas in the memory map isn't specified. +This function chooses the area with the minimal base address that still has free frames, i.e. `next_free_frame` is smaller than its last frame. Note that we need to clone the iterator because the order of areas in the memory map isn't specified. If there are no areas with free frames left, `min_by` automatically returns the desired `None`. -We don't have a data structure to store free frames, so we can't implement `deallocate_frame` reasonably. Thus we use the `unimplemented` macro, which just panics when called: +We don't have a data structure to store free frames, so we can't implement `deallocate_frame` reasonably. Thus we use the `unimplemented` macro, which just panics when the method is called: ```rust fn deallocate_frame(&mut self, _frame: Frame) { @@ -287,7 +291,7 @@ fn deallocate_frame(&mut self, _frame: Frame) { } ``` -Now we only need a constructor function: +Now we only need a constructor function to make it usable: ```rust pub fn new(kernel_start: usize, kernel_end: usize, @@ -307,9 +311,43 @@ pub fn new(kernel_start: usize, kernel_end: usize, allocator } ``` -Note that we call `choose_next_area` manually here because `allocate_frame` returns `None` as soon as `current_area` is `None`. +Note that we call `choose_next_area` manually here because `allocate_frame` returns `None` as soon as `current_area` is `None`. So by calling `choose_next_area` we initialize it to the area with the minimal base address. -## Remapping the Kernel Sections +### Testing it +Now we can test it in main. Therefor we need to [re-export] the `AreaFrameAllocator` in the `memory` module. Then we can create a new allocator: + +[re-export]: https://doc.rust-lang.org/book/crates-and-modules.html#re-exporting-with-pub-use + +```rust +let mut frame_allocator = memory::AreaFrameAllocator::new( + kernel_start as usize, kernel_end as usize, multiboot_start, + multiboot_end, memory_map_tag.memory_areas()); +``` + +Now you can test it by adding some test allocations: + +```rust +println!("{:?}", frame_allocator.allocate_frame()) +``` +You will see that frame number starts at `0` and increases steadily, but the kernel and Multiboot frames are left out (you need to allocate many frames to see this since the kernel starts at frame 256). + +The following `for` loop allocates all frames and prints out the total number of allocated frames: + +```rust +for i in 0.. { + if let None = frame_allocator.allocate_frame() { + println!("allocated {} frames", i); + break; + } +} +``` +You can try different amounts of memory by passing e.g. `-m 500M` to QEMU. To compare these numbers, [WolframAlpha] can be very helpful. + +[WolframAlpha]: http://www.wolframalpha.com/input/?i=%2832698+*+4096%29+bytes+in+MiB + +## What's next? + +### Remapping the Kernel Sections We can use the ELF section tag to write a skeleton that remaps the kernel correctly: ```rust From cf835e19ae45289b1622bfedd34eb1c269e38cbb Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:01:36 +0100 Subject: [PATCH 20/30] Recommend Eric Kidd's Bare Metal Rust series --- posts/DRAFT-allocating-frames.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index c859b503..824afae7 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -358,3 +358,9 @@ for section in multiboot.elf_tag().sections() { } ``` TODO + +## Recommended Posts +Eric Kidd started the [Bare Metal Rust] series last week. Like this post, it builds upon the code from [Printing to Screen], but tries to support keyboard input instead of wrestling through memory management details ;). + +[Bare Metal Rust]: http://www.randomhacks.net/bare-metal-rust/ +[Printing to Screen]: {% post_url 2015-10-23-printing-to-screen %} From ca07527ede16381122470890d5a76f2716461f38 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:09:20 +0100 Subject: [PATCH 21/30] Add `What's next?` text --- posts/DRAFT-allocating-frames.md | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 824afae7..6c3dda6c 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -346,18 +346,7 @@ You can try different amounts of memory by passing e.g. `-m 500M` to QEMU. To co [WolframAlpha]: http://www.wolframalpha.com/input/?i=%2832698+*+4096%29+bytes+in+MiB ## What's next? - -### Remapping the Kernel Sections -We can use the ELF section tag to write a skeleton that remaps the kernel correctly: - -```rust -for section in multiboot.elf_tag().sections() { - for page in start_page..end_page { - // TODO identity_map(page, section.writable(), section.executable()) - } -} -``` -TODO +The next post will be about paging again. But this time we will use the frame allocator to create a safe rust module that allows us to switch page tables and map pages. Then we will use this module and the information from the ELF sections tag to remap the kernel correctly. ## Recommended Posts Eric Kidd started the [Bare Metal Rust] series last week. Like this post, it builds upon the code from [Printing to Screen], but tries to support keyboard input instead of wrestling through memory management details ;). From d0cf5e7c61f94b088240577e5e4b37ece6ab5f61 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:16:30 +0100 Subject: [PATCH 22/30] Link from `Printing to Screen` --- posts/2015-10-23-printing-to-screen.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/posts/2015-10-23-printing-to-screen.md b/posts/2015-10-23-printing-to-screen.md index 0989ebe2..8839fb09 100644 --- a/posts/2015-10-23-printing-to-screen.md +++ b/posts/2015-10-23-printing-to-screen.md @@ -413,7 +413,11 @@ pub extern fn rust_main() { Since we imported the macros at crate level, they are available in all modules and thus provide an easy and safe interface to the VGA buffer. ## What's next? -In the next posts we will map the kernel pages correctly so that accessing `0x0` or writing to `.rodata` is not possible anymore. To obtain the loaded kernel sections we will read the multiboot information structure. Then we will create a paging module and use it to switch to a new page table where the kernel sections are mapped correctly. +In the next posts we will map the kernel pages correctly so that accessing `0x0` or writing to `.rodata` is not possible anymore. To obtain the loaded kernel sections we will read the Multiboot information structure. Then we will create a paging module and use it to switch to a new page table where the kernel sections are mapped correctly. + +The [next post] describes the Multiboot information structure and creates a frame allocator using the information about memory areas. + +[next post]: {{ page.next.url }} ## Other Rust OS Projects Now that you know the very basics of OS development in Rust, you should also check out the following projects: From 6ce34f5e2b403f3a3899cc4c48b957c1804eb99e Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:28:15 +0100 Subject: [PATCH 23/30] Add introduction --- posts/DRAFT-allocating-frames.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 6c3dda6c..e1e180c5 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -3,7 +3,7 @@ layout: post title: 'Allocating Frames' --- -TODO +In this post we create an allocator that provides free physical frames for a future paging module. To get the required information about available and used memory we use the Multiboot information structure. Additionally, we improve the `panic` handler to print the corresponding message and source line. ## Preparation We still have a really tiny stack of 64 bytes, which won't suffice for this post. So we will increase it to 4096 (one page) in `boot.asm`: From a8580ad3a258c445a1ac3baa7d8e5f8a189b08dd Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:52:02 +0100 Subject: [PATCH 24/30] Fix: `next_free_frame` must be increased to the start of the current area --- posts/DRAFT-allocating-frames.md | 9 +++++++++ src/memory/area_frame_allocator.rs | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index e1e180c5..4c4583b9 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -279,10 +279,19 @@ fn choose_next_area(&mut self) { let address = area.base_addr + area.length - 1; Frame::containing_address(address as usize) >= self.next_free_frame }).min_by(|area| area.base_addr); + + if let Some(area) = self.current_area { + let start_frame = Frame::containing_address(area.base_addr as usize); + if self.next_free_frame < start_frame { + self.next_free_frame = start_frame; + } + } } ``` This function chooses the area with the minimal base address that still has free frames, i.e. `next_free_frame` is smaller than its last frame. Note that we need to clone the iterator because the order of areas in the memory map isn't specified. If there are no areas with free frames left, `min_by` automatically returns the desired `None`. +If the `next_free_frame` is below the new `current_area`, it needs to be updated to the area's start frame. Else, the `allocate_frame` call could return an unavailable frame. + We don't have a data structure to store free frames, so we can't implement `deallocate_frame` reasonably. Thus we use the `unimplemented` macro, which just panics when the method is called: ```rust diff --git a/src/memory/area_frame_allocator.rs b/src/memory/area_frame_allocator.rs index d36b3705..edc67651 100644 --- a/src/memory/area_frame_allocator.rs +++ b/src/memory/area_frame_allocator.rs @@ -38,6 +38,13 @@ impl AreaFrameAllocator { let address = area.base_addr + area.length - 1; Frame::containing_address(address as usize) >= self.next_free_frame }).min_by(|area| area.base_addr); + + if let Some(area) = self.current_area { + let start_frame = Frame::containing_address(area.base_addr as usize); + if self.next_free_frame < start_frame { + self.next_free_frame = start_frame; + } + } } } From 18d397b48d9a4f3dcee3e1832820be8fc86b7a03 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:53:59 +0100 Subject: [PATCH 25/30] Some improvements to the `allocating frames` post --- posts/DRAFT-allocating-frames.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/posts/DRAFT-allocating-frames.md b/posts/DRAFT-allocating-frames.md index 4c4583b9..ce6ca136 100644 --- a/posts/DRAFT-allocating-frames.md +++ b/posts/DRAFT-allocating-frames.md @@ -25,7 +25,7 @@ First, we need to pass this pointer to our kernel as an argument to `rust_main`. > The first six integer or pointer arguments are passed in registers RDI, RSI, RDX, RCX, R8, and R9 -So to pass the pointer to our kernel, we need to move it to `rdi` before calling the kernel. Since we're not using the `rdi`/`edi` register in our bootstrap code right now, we can simply set the `edi` register right after booting (in `boot.asm`): +So to pass the pointer to our kernel, we need to move it to `rdi` before calling the kernel. Since we're not using the `rdi`/`edi` register in our bootstrap code, we can simply set the `edi` register right after booting (in `boot.asm`): ```nasm start: @@ -38,9 +38,10 @@ Now we can add the argument to our `rust_main`: pub extern fn rust_main(multiboot_information_address: usize) { ... } ``` -Now we can use the [multiboot2-elf64] crate to query get some information about mapped kernel sections and available memory. I just wrote it for this blog post since I could not find any other Multiboot 2 crate. It's really ugly and incomplete, but it does its job. +Instead of writing an own Multiboot module, we use the [multiboot2-elf64] crate. It gives us some basic information about mapped kernel sections and available memory. I just wrote it for this blog post since I could not find any other Multiboot 2 crate. It's really ugly and incomplete, but it does its job[^fn-multiboot-crate]. [multiboot2-elf64]: https://github.com/phil-opp/multiboot2-elf64 +[^fn-multiboot-crate]: All contributions are welcome! If you want to maintain it, please contact me! So let's add a dependency on the git repository in the `Cargo.toml`: @@ -53,9 +54,11 @@ git = "https://github.com/phil-opp/multiboot2-elf64" Now we can add `extern crate multiboot2` and use it to print available memory areas. ### Available Memory -The boot information structure consists of various _tags_. The _memory map_ tag contains a list of all areas of available RAM. Special areas such as the VGA text buffer at `0xb8000` are not available. Note that some of the available memory is already used by our kernel and by the multiboot information structure itself. +The boot information structure consists of various _tags_. See section 3.4 of the Multiboot specification ([PDF][multiboot specification]) for a complete list. The _memory map_ tag contains a list of all areas of available RAM. Special areas such as the VGA text buffer at `0xb8000` are not available. Note that some of the available memory is already used by our kernel and by the multiboot information structure itself. -To print available memory areas, we can use the `multiboot2` crate in our `rust_main` as follows: +[multiboot specification]: http://nongnu.askapache.com/grub/phcoder/multiboot.pdf + +To print all available memory areas, we can use the `multiboot2` crate in our `rust_main` as follows: ```rust let boot_info = unsafe{ multiboot2::load(multiboot_information_address) }; @@ -190,7 +193,7 @@ pub struct Frame { number: usize, } ``` -We use `usize` here since the number of frames depends on the memory size. The long `derive` line makes frames printable and comparable. (Don't forget to add the `mod memory` line to `src/lib.rs`.) +(Don't forget to add the `mod memory` line to `src/lib.rs`.) We use `usize` here since the number of frames depends on the memory size. The long `derive` line makes frames printable, clonable, and comparable. To make it easy to get the corresponding frame for a physical address, we add a `containing_address` method: @@ -352,7 +355,7 @@ for i in 0.. { ``` You can try different amounts of memory by passing e.g. `-m 500M` to QEMU. To compare these numbers, [WolframAlpha] can be very helpful. -[WolframAlpha]: http://www.wolframalpha.com/input/?i=%2832698+*+4096%29+bytes+in+MiB +[WolframAlpha]: http://www.wolframalpha.com/input/?i=%2832605+*+4096%29+bytes+in+MiB ## What's next? The next post will be about paging again. But this time we will use the frame allocator to create a safe rust module that allows us to switch page tables and map pages. Then we will use this module and the information from the ELF sections tag to remap the kernel correctly. From 28c56f2c06676e4278ee15c6642e04d69f197c01 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 02:54:52 +0100 Subject: [PATCH 26/30] Publish `Allocating Frames` post --- ...DRAFT-allocating-frames.md => 2015-11-15-allocating-frames.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename posts/{DRAFT-allocating-frames.md => 2015-11-15-allocating-frames.md} (100%) diff --git a/posts/DRAFT-allocating-frames.md b/posts/2015-11-15-allocating-frames.md similarity index 100% rename from posts/DRAFT-allocating-frames.md rename to posts/2015-11-15-allocating-frames.md From 78c2fd6acc350adc0b87c781829421f6fbdeb0e7 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 11:48:43 +0100 Subject: [PATCH 27/30] Many small improvements --- posts/2015-11-15-allocating-frames.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/posts/2015-11-15-allocating-frames.md b/posts/2015-11-15-allocating-frames.md index ce6ca136..00255ed1 100644 --- a/posts/2015-11-15-allocating-frames.md +++ b/posts/2015-11-15-allocating-frames.md @@ -54,7 +54,7 @@ git = "https://github.com/phil-opp/multiboot2-elf64" Now we can add `extern crate multiboot2` and use it to print available memory areas. ### Available Memory -The boot information structure consists of various _tags_. See section 3.4 of the Multiboot specification ([PDF][multiboot specification]) for a complete list. The _memory map_ tag contains a list of all areas of available RAM. Special areas such as the VGA text buffer at `0xb8000` are not available. Note that some of the available memory is already used by our kernel and by the multiboot information structure itself. +The boot information structure consists of various _tags_. See section 3.4 of the Multiboot specification ([PDF][multiboot specification]) for a complete list. The _memory map_ tag contains a list of all available RAM areas. Special areas such as the VGA text buffer at `0xb8000` are not available. Note that some of the available memory is already used by our kernel and by the multiboot information structure itself. [multiboot specification]: http://nongnu.askapache.com/grub/phcoder/multiboot.pdf @@ -79,7 +79,7 @@ memory areas: start: 0x0, length: 0x9fc00 start: 0x100000, length: 0x7ee0000 ``` -So we have one area from `0x0` to `0x9fc00`, which is a bit below the 1MiB mark. The second, bigger area starts at 1MiB and contains the rest of available memory. The area from `0x9fc00` to 1MiB is not available since it contains for example the VGA text buffer at `0xb8000`. This is the reason for putting our kernel at 1MiB and not at e.g. `0x0`. +So we have one area from `0x0` to `0x9fc00`, which is a bit below the 1MiB mark. The second, bigger area starts at 1MiB and contains the rest of available memory. The area from `0x9fc00` to 1MiB is not available since it contains for example the VGA text buffer at `0xb8000`. This is the reason for putting our kernel at 1MiB and not somewhere below. If you give QEMU more than 4GiB of memory by passing `-m 5G`, you get another unusable area below the 4GiB mark. This memory is normally mapped to some hardware devices. See the [OSDev Wiki][Memory_map] for more information. @@ -107,7 +107,7 @@ extern fn panic_fmt(fmt: core::fmt::Arguments, file: &str, line: u32) -> ! { ``` Be careful with these arguments as the compiler does not check the function signature for `lang_items`. -You can try our new panic handler by inserting a `panic` somewhere. Now we get the panic message and the causing source line. +Now we get the panic message and the causing source line. You can try it by inserting a `panic` somewhere. ### Kernel ELF Sections To read and print the sections of our kernel ELF file, we can use the _Elf-sections_ tag: @@ -124,7 +124,7 @@ for section in elf_sections_tag.sections() { ``` This should print out the start address and size of all kernel sections. If the section is writable, the `0x1` bit is set in `flags`. The `0x4` bit marks an executable section and the `0x2` bit indicates that the section was loaded in memory. For example, the `.text` section is executable but not writable and the `.data` section just the opposite. -But when we execute it, tons of really small sections are printed. We can use the `objdump -h build/kernel-x86_64.bin` command to list the sections with name. There seem to be over 200 sections and many of them start with `.text.*` or `.data.rel.ro.local.*`. This is because the Rust compiler puts e.g. each function in an own `.text` subsection. To merge these subsections, we can update our linker script: +But when we execute it, tons of really small sections are printed. We can use the `objdump -h build/kernel-x86_64.bin` command to list the sections with name. There seem to be over 200 sections and many of them start with `.text.*` or `.data.rel.ro.local.*`. This is because the Rust compiler puts e.g. each function in its own `.text` subsection. To merge these subsections, we need to update our linker script: ``` SECTIONS { @@ -182,7 +182,7 @@ multiboot_start: 0x11d400, multiboot_end: 0x11d9c8 So the kernel starts at 1MiB (like expected) and is about 105 KiB in size. The multiboot information structure was placed at `0x11d400` by GRUB and needs 1480 bytes. Of course your numbers could be a bit different due to different versions of Rust or GRUB (or some differences in the source code). ## A frame allocator -When we create a paging module in the next post, we will need to map virtual pages to free physical frames. So we will need some kind of allocator that keeps track of physical frames and gives us a free one when needed. We can use the information about memory areas to write such a frame allocator. +When we create a paging module in the next post, we will need free physical frames to create new page tables. So we need some kind of allocator that keeps track of physical frames and gives us a free one when needed. We can use the information about memory areas to write such a frame allocator. ### A Memory Module First we create a memory module with a `Frame` type (`src/memory/mod.rs`): @@ -193,7 +193,7 @@ pub struct Frame { number: usize, } ``` -(Don't forget to add the `mod memory` line to `src/lib.rs`.) We use `usize` here since the number of frames depends on the memory size. The long `derive` line makes frames printable, clonable, and comparable. +(Don't forget to add the `mod memory` line to `src/lib.rs`.) Instead of e.g. the start address, we just store the frame number. We use `usize` here since the number of frames depends on the memory size. The long `derive` line makes frames printable, clonable, and comparable. To make it easy to get the corresponding frame for a physical address, we add a `containing_address` method: @@ -303,7 +303,7 @@ fn deallocate_frame(&mut self, _frame: Frame) { } ``` -Now we only need a constructor function to make it usable: +Now we only need a constructor function to make the allocator usable: ```rust pub fn new(kernel_start: usize, kernel_end: usize, @@ -326,7 +326,7 @@ pub fn new(kernel_start: usize, kernel_end: usize, Note that we call `choose_next_area` manually here because `allocate_frame` returns `None` as soon as `current_area` is `None`. So by calling `choose_next_area` we initialize it to the area with the minimal base address. ### Testing it -Now we can test it in main. Therefor we need to [re-export] the `AreaFrameAllocator` in the `memory` module. Then we can create a new allocator: +In order to test it in main, we need to [re-export] the `AreaFrameAllocator` in the `memory` module. Then we can create a new allocator: [re-export]: https://doc.rust-lang.org/book/crates-and-modules.html#re-exporting-with-pub-use @@ -336,12 +336,12 @@ let mut frame_allocator = memory::AreaFrameAllocator::new( multiboot_end, memory_map_tag.memory_areas()); ``` -Now you can test it by adding some test allocations: +Now we can test it by adding some frame allocations: ```rust println!("{:?}", frame_allocator.allocate_frame()) ``` -You will see that frame number starts at `0` and increases steadily, but the kernel and Multiboot frames are left out (you need to allocate many frames to see this since the kernel starts at frame 256). +You will see that the frame number starts at `0` and increases steadily, but the kernel and Multiboot frames are left out (you need to allocate many frames to see this since the kernel starts at frame 256). The following `for` loop allocates all frames and prints out the total number of allocated frames: @@ -358,7 +358,7 @@ You can try different amounts of memory by passing e.g. `-m 500M` to QEMU. To co [WolframAlpha]: http://www.wolframalpha.com/input/?i=%2832605+*+4096%29+bytes+in+MiB ## What's next? -The next post will be about paging again. But this time we will use the frame allocator to create a safe rust module that allows us to switch page tables and map pages. Then we will use this module and the information from the ELF sections tag to remap the kernel correctly. +The next post will be about paging again. We will use the frame allocator to create a safe module that allows us to switch page tables and map pages. Then we will use this module and the information from the Elf-sections tag to remap the kernel correctly. ## Recommended Posts Eric Kidd started the [Bare Metal Rust] series last week. Like this post, it builds upon the code from [Printing to Screen], but tries to support keyboard input instead of wrestling through memory management details ;). From 5de4255fcaeb28bf83e7696d25a889d4b4067ac8 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 11:49:08 +0100 Subject: [PATCH 28/30] Add a conclusion to the `allocating frames` post --- posts/2015-11-15-allocating-frames.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/posts/2015-11-15-allocating-frames.md b/posts/2015-11-15-allocating-frames.md index 00255ed1..09743945 100644 --- a/posts/2015-11-15-allocating-frames.md +++ b/posts/2015-11-15-allocating-frames.md @@ -357,6 +357,10 @@ You can try different amounts of memory by passing e.g. `-m 500M` to QEMU. To co [WolframAlpha]: http://www.wolframalpha.com/input/?i=%2832605+*+4096%29+bytes+in+MiB +## Conclusion + +Now we have a working frame allocator. It is a bit rudimentary and cannot free frames, but it also is very fast since it reuses the Multiboot memory map and does not need any costly initialization. A future post will build upon this allocator and add a stack-like data structure for freed frames. + ## What's next? The next post will be about paging again. We will use the frame allocator to create a safe module that allows us to switch page tables and map pages. Then we will use this module and the information from the Elf-sections tag to remap the kernel correctly. From 96cee9ab2b1b5c9ae458ae7fc4600af1438aa069 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 11:58:34 +0100 Subject: [PATCH 29/30] Link github repository --- posts/2015-11-15-allocating-frames.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/posts/2015-11-15-allocating-frames.md b/posts/2015-11-15-allocating-frames.md index 09743945..17e0256d 100644 --- a/posts/2015-11-15-allocating-frames.md +++ b/posts/2015-11-15-allocating-frames.md @@ -5,6 +5,10 @@ title: 'Allocating Frames' In this post we create an allocator that provides free physical frames for a future paging module. To get the required information about available and used memory we use the Multiboot information structure. Additionally, we improve the `panic` handler to print the corresponding message and source line. +The full source code is available on [Github][source repo]. Feel free to open issues there if you have any problems or improvements. You can also leave a comment at the bottom. + +[source repo]: https://github.com/phil-opp/blog_os/tree/allocating_frames + ## Preparation We still have a really tiny stack of 64 bytes, which won't suffice for this post. So we will increase it to 4096 (one page) in `boot.asm`: From f997f1f0f00b8f69e5bf057a8cf5f674232af450 Mon Sep 17 00:00:00 2001 From: Philipp Oppermann Date: Sun, 15 Nov 2015 11:58:47 +0100 Subject: [PATCH 30/30] Small wording improvement --- posts/2015-11-15-allocating-frames.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/posts/2015-11-15-allocating-frames.md b/posts/2015-11-15-allocating-frames.md index 17e0256d..0fe36ec9 100644 --- a/posts/2015-11-15-allocating-frames.md +++ b/posts/2015-11-15-allocating-frames.md @@ -10,7 +10,7 @@ The full source code is available on [Github][source repo]. Feel free to open is [source repo]: https://github.com/phil-opp/blog_os/tree/allocating_frames ## Preparation -We still have a really tiny stack of 64 bytes, which won't suffice for this post. So we will increase it to 4096 (one page) in `boot.asm`: +We still have a really tiny stack of 64 bytes, which won't suffice for this post. So we increase it to 4096 bytes (one page) in `boot.asm`: ```asm section .bss