Booting kernel

Sat Dec 09, 2023 8:44 am

Dear KolibriOS developers!

First, sorry, I don't speak Russian. Second, I really like what you have done here. So much so, that I've decided to support it in my current project.

I'm working on a FOSS, GPL'd boot manager Easyboot, which supports both BIOS and UEFI, and can already load several kernels, including KolibriOS on BIOS machines (here is the source). Now I wish to add support for booting KolibriOS on UEFI machines too, and that's where my headache starts.

I've read through these forums, studied uefi4kos.asm, and as far as I can tell, now KOLIBRI.KRN and KERNEL.MNT are two different files. This is very problematic for several reasons: a) makes shipping KolibiriOS more complicated b) there's no magic bytes in KOLIBRI.KRN, so nobody can tell that's actually a KolibriOS kernel.

My understanding is, this separation was done by dunkaist, because his UEFI loader didn't know where the BIOS code ends.

If you don't mind, my I suggest a different solution? It only increases the kernel's size by 7 bytes, but makes the file separation unnecessary.

In biosboot.asm, right after the identifier string and before including bootstr.inc:

Code: Select all

        jmp     start_of_code

if lang eq sp
include "kernelsp.inc"  ; spanish kernel messages
else if lang eq et
version db    'Kolibri OS  versioon 0.7.7.0+    ',13,10,13,10,0
else
version db    'Kolibri OS  version 0.7.7.0+     ',13,10,13,10,0
end if

+           db          2Ch-($-$$) dup 0          ; place the BIOS initialization's code size at 2C
+           dd          B32-10000h

include "boot/bootstr.inc"     ; language-independent boot messages

This would add a dword with the size at a fixed location, so then uefi4kos.asm in kernel_trampoline could do something like

Code: Select all

        mov     eax, cr4
        and     eax, not CR4_PAE
        mov     cr4, eax
                
-        push    KERNEL_BASE
+        mov     eax, KERNEL_BASE
+        ; is it the new unified KERNEL.MNT or the old separated KOLIBRI.KRN?
+        cmp     'Koli', dword [eax + 3]
+        jne     @f
+        add     eax, dword [eax + 2Ch]
+@@:     push    eax
        retn

This would keep backward compatibility with the current solution, but also works with the very same kernel file like BIOS, rendering the two separate kernels unnecessary.

What are your thoughts?
bzt

Mon Dec 11, 2023 12:48 am

So what do you think? I'd love to have a simple way of locating the protected mode entry point, and then BIOS and UEFI could use the same kernel file.

Cheers,
bzt

Mon Dec 11, 2023 10:07 am

Hi bzt,

Thank you for your patch.

bzt wrote: ↑Sat Dec 09, 2023 8:44 am My understanding is, this separation was done by dunkaist, because his UEFI loader didn't know where the BIOS code ends.

If you don't mind, my I suggest a different solution? It only increases the kernel's size by 7 bytes, but makes the file separation unnecessary.

Indeed, jumping directly to the 32-bit code was one of reasons to separate BIOS code to bootbios.asm. Another reason was to avoid dead 16-bit code when booting on modern UEFI systems. In principle, I agree that getting the only universal kernel image is worth having some dead code. Unfortunately, things have got more complicated since separation of bootbios.asm. If you grep /kernel/trunk, you can see that UEFI macro is used in a number of files:

Code: Select all

$ grep 'defined UEFI' -r . --include="*.inc" --include="*.asm"
./core/apic.inc:if defined UEFI
./video/cursors.inc:if ~defined UEFI
./init.inc:if defined UEFI
./kernel.asm:if ~defined UEFI

This is because some BOOT_LO fields are only set by our UEFI loader, while most or all BIOS loaders leave them uninitialized. We don't have proper versioning of the boot protocol and we have quite a few of exotic loaderes that e.g. boot KolibriOS from DOS or Windows. With that in mind, I just wrapped new UEFI logic in macro here and there. To remove these macros and build the universal BIOS/UEFI kernel image we should first go through all the BIOS loaders and make them to initialize the new BOOT_LO fields with some sensible values. This requires non-trivial testing too. I.e. this is the effort yet to be done. Your patch is a move to the right direction, but more moves are still required.

In short, let's keep two separate images so far. There is a technical issue preventing unification that can be resolved in the future. I'm not ready to edit and test all the BIOS loaders at the moment, sorry.

Mon Dec 11, 2023 8:02 pm

dunkaist wrote: ↑Mon Dec 11, 2023 10:07 amIndeed, jumping directly to the 32-bit code was one of reasons to separate BIOS code to bootbios.asm. Another reason was to avoid dead 16-bit code when booting on modern UEFI systems.

Yes, but I think you shouldn't worry about that dead code. It's just a few kilobytes, and it provides backward compatibility out-of-the-box with all the non-UEFI loaders. (Those wouldn't mind this protmode entry pointer either.)

dunkaist wrote: ↑Mon Dec 11, 2023 10:07 amUnfortunately, things have got more complicated since separation of bootbios.asm. If you grep /kernel/trunk, you can see that UEFI macro is used in a number of files

I did. I took the libery to assess the situation before I made this suggestion. I'm fairly certain they won't do anything (about 99.99999% sure).

Code: Select all

$ grep UEFI `find . -name '*.asm'` `find . -name '*.inc'`
./kernel.asm:if ~ defined UEFI
./init.inc:if defined UEFI
./init.inc:        ; UEFI loader knows where RSDP is
./video/cursors.inc:if ~defined UEFI
./blkdev/disk.inc:; GUID Partition Table Header, UEFI 2.6, Table 18
./blkdev/disk.inc:; GPT Partition Entry, UEFI 2.6, Table 19
./core/apic.inc:; non-UEFI loaders don't load DEVICES.DAT and don't initialize [acpi_dev_size]
./core/apic.inc:if defined UEFI
./bootloader/uefi4kos/uefi.inc:;; Based on UEFI library for fasm by bzt, Public Domain.        ;;
./bootloader/uefi4kos/uefi64.inc:;; Based on UEFI library for fasm by bzt, Public Domain.        ;;
./bootloader/uefi4kos/uefi32.inc:;; Based on UEFI library for fasm by bzt, Public Domain.        ;;
$

Now one by one:

kernel.asm: this is just the one that prefixes the binary with the biosboot code.

init.inc:

Code: Select all

if defined UEFI
        ; UEFI loader knows where RSDP is
        mov     ebx, [BOOT_LO.acpi_rsdp]
        test    ebx, ebx
        jz      .done
        call    .check
else
        movzx   ebx, word [0x40E]
        shl     ebx, 4
        lea     ecx, [ebx+1024]
        call    .check               
...

Considering that like you said, biosboot leaves the BOOT_LO zeroed out, this "if" could be removed. If acpi_rsdp is set, we can use it, if not, then the code could go on as usual. (However if I were you, I would consider moving the word [0x40E] part into bioscode and set BOOT_LO.acpi_rsdp there, so init.inc wouldn't have to care where that pointer come from.)

video/cursors.inc:

Code: Select all

if ~defined UEFI
        cmp     [SCR_MODE], word 0x12
        jne     .not_vga
        ; TODO

Similar, only this time it's the UEFI loader that leaves the mode field zero. (We must check if SCR_MODE is indeed initialized from BOOT_LO.mode, and then this "if" can be removed.)

blkdev/disk.inc: just a comment about GPT

core/apic.inc:

Code: Select all

; non-UEFI loaders don't load DEVICES.DAT and don't initialize [acpi_dev_size]
if defined UEFI
        cmp     [acpi_dev_size], 0
        jz      @f
        stdcall map_io_mem, [acpi_dev_data], [acpi_dev_size], PG_SWR
        mov     [acpi_dev_data], eax
        jmp     .loaded
@@:
end if

This again won't cause trouble, just need to check if acpi_dev_size is indeed initialized from BOOT_LO.devicesdat_size (which biosboot zeroes out), and then it's safe to remove this "if".

dunkaist wrote: ↑Mon Dec 11, 2023 10:07 amWe don't have proper versioning of the boot protocol

No need, there's no difference. It's just a few checks if some fields are filled it or not, neither the workflow, nor the BOOT_LO structure changed in any way.

dunkaist wrote: ↑Mon Dec 11, 2023 10:07 amwe have quite a few of exotic loaderes that e.g. boot KolibriOS from DOS or Windows.

Again, not a problem because these just jump to biosboot code in the kernel, and they never jump to the protmode code.

dunkaist wrote: ↑Mon Dec 11, 2023 10:07 amTo remove these macros and build the universal BIOS/UEFI kernel image we should first go through all the BIOS loaders and make them to initialize the new BOOT_LO fields with some sensible values.

You don't have to do any of this if you keep biosboot in the kernel (for now at least).

I agree that removing biosboot and rewriting all loaders to come to a common ground with BOOT_LO is the way to go on the long term, but that's not what I suggesting here. I'm only suggesting a small step: keep biosboot, just add protmode entry to it at a fixed location. This way no matter what all those loaders do, because it's biosboot that initializes BOOT_LO on BIOS systems.

dunkaist wrote: ↑Mon Dec 11, 2023 10:07 amIn short, let's keep two separate images so far.

That's problematic, because KOLIBRI.KRN (the UEFI variant) has no header, so there's no way of telling that it is actually a KolibriOS kernel.

dunkaist wrote: ↑Mon Dec 11, 2023 10:07 amThere is a technical issue preventing unification that can be resolved in the future. I'm not ready to edit and test all the BIOS loaders at the moment, sorry.

Only if you about to remove biosboot, which I think you shouldn't. Maybe in the future when you're ready, but not now.

I believe having a unified kernel (with well detectable magic bytes) is very very important, and I'm offering my help to achieve that. Anything you need, I can provide you patches and I can test, just let me know what you need! I strongly believe this worth the effort, and I'm ready to sacrifice me spare time for it.

Cheers,
bzt

ps.: I'm glad to see that my uefi.inc was useful to you!

Mon Dec 11, 2023 9:50 pm

Okay, here's a patch to help you with the unification. (Again, this does not change boot flow, it just removes the need for "if" macros.)

First, this patch moves the BIOS based RSDP search from init.inc to bootcode.inc. This way init.inc can be firmware neutral, it shouldn't care where that BOOT_LO.acpi_rsdp come from.

Code: Select all

diff -r -U 3 aaa/kernel/boot/bootcode.inc bbb/kernel/boot/bootcode.inc
--- aaa/kernel/boot/bootcode.inc	2023-10-23 14:53:06.000000000 +0200
+++ bbb/kernel/boot/bootcode.inc	2023-12-11 19:46:40.072978847 +0100
@@ -480,6 +480,35 @@
 apm_end:
         _setcursor d80x25_top_num, 0
 
+; --------------- ACPI ---------------------
+ACPI_LO_RSDP_WINDOW_END    = 0x000A0000
+ACPI_HI_RSDP_WINDOW_START  = 0x000E0000
+
+acpi:   push    es
+        xor     eax, eax
+        mov     [BOOT_LO.devicesdat_data], eax
+        mov     [BOOT_LO.devicesdat_size], eax
+        mov     [BOOT_LO.acpi_rsdp], eax
+        mov     ax, word [40Eh]
+        add     ax, 64
+.check: mov     es, ax
+        cmp     [es:0], dword 'RSD '
+        jne     .next
+        cmp     [es:4], dword 'PTR '
+        jne     .next
+        shl     eax, 4
+        mov     [BOOT_LO.acpi_rsdp], eax
+        jmp     .end
+.next:  inc     ax
+        ; skip VRAM and ROM area A0000h to E0000h
+        cmp     ax, (ACPI_LO_RSDP_WINDOW_END shr 4)
+        jne     @f
+        add     ax, ((ACPI_HI_RSDP_WINDOW_START - ACPI_LO_RSDP_WINDOW_END) shr 4)
+@@:     ; end of 1Mb?
+        or      ax, ax
+        jnz     .check
+.end:   pop     es
+
 if ~ defined extended_primary_loader
 ;CHECK current of code
         cmp     [cfgmanager.loader_block], -1
diff -r -U 3 aaa/kernel/init.inc bbb/kernel/init.inc
--- aaa/kernel/init.inc	2023-10-23 14:53:06.000000000 +0200
+++ bbb/kernel/init.inc	2023-12-11 19:25:11.392907707 +0100
@@ -410,8 +410,6 @@
         ret
 endp
 
-ACPI_HI_RSDP_WINDOW_START  = 0x000E0000
-ACPI_HI_RSDP_WINDOW_END    = 0x00100000
 ACPI_RSDP_CHECKSUM_LENGTH  = 20
 
 proc acpi_locate_tables uses ebx esi edi
@@ -475,27 +473,11 @@
         push    ebx
         push    edi
 
-if defined UEFI
-        ; UEFI loader knows where RSDP is
         mov     ebx, [BOOT_LO.acpi_rsdp]
         test    ebx, ebx
         jz      .done
+        mov     edi, ebx
         call    .check
-else
-        movzx   ebx, word [0x40E]
-        shl     ebx, 4
-        lea     ecx, [ebx+1024]
-        call    .check
-
-        test    ebx, ebx
-        jz      @F
-        jmp     .done
-
-@@:
-        mov     ebx, ACPI_HI_RSDP_WINDOW_START
-        mov     edi, ACPI_HI_RSDP_WINDOW_END
-        call    .check
-end if
 .done:
         mov     [acpi_rsdp_base - OS_BASE], ebx
         test    ebx, ebx

Second, it's safe to remove "if ~defined UEFI" in video/cursors.inc, because in video/framebuffer.inc:

Code: Select all

        movzx   eax, word [BOOT.vesa_mode]    ; screen mode
        mov     dword [SCR_MODE], eax

So SCR_MODE comes from BOOT_LO.vesa_mode, which is always zero (so not 0x12 and not 0x4000) on UEFI.

Lastly, "if defined UEFI" in core/apic.inc can be removed, because in kernel.asm

Code: Select all

        mov     eax, [BOOT.devicesdat_size]
        mov     [acpi_dev_size], eax
        mov     eax, [BOOT.devicesdat_data]
        mov     [acpi_dev_data], eax

So acpi_dev_size comes from BOOT_LO.devicesdat_size, and the above patch makes sure of it that this is zeroed out on BIOS.

I've attached all my modifications in one zip. I've compiled and successfully booted this on BIOS as well as on UEFI machines (but obviously more tests are needed, I just performed the basic tests). I've also included the modified uefi4kos, which now supports this unified kernel, but at the same time is backward compatible with the current KOLIBRI.KRN kernel.

Hope this helps!
bzt

Wed Dec 13, 2023 10:33 am

Hi bzt,

Since you're so interested in having this done, I agree to put some effort too.

1.
I tried your unified_kernel.zip patch and it doesn't compile.

Code: Select all

flat assembler  version 1.73.31  (16384 kilobytes memory, x64)
uefi64kos.asm [1015]:
        cmp     'Koli', dword [eax + 3]
processed: cmp 'Koli',dword[eax+3]
error: invalid operand.

Not a big deal but still. Same for uefi32kos.

2.
I'm afraid that bootbios doesn't zero acpi_rsdp, devicesdat_* and maybe other fields of BOOT_LO. At least, my GDB reports this.

3.
I don't think it's worth supporting booting the old (i.e. current) non-unified UEFI kernel in uefi*kos loaders. Just switch to the new scheme and that's it.

4.
Maybe we should put the BIOS init code offset just after 'jmp start_of_code', i.e. just before 'Kolibri OS version ...'. This way we can avoid one magic number (2Ch) and a few padding bytes.

5.
I can create an SVN account for you. Send me your desired password in a private message in case you don't want a randomly generated one.

bzt wrote: ↑Mon Dec 11, 2023 8:02 pm ps.: I'm glad to see that my uefi.inc was useful to you!

I really appreciate your work, it was of great help to me when I developed uefi4kos.

Thank you,
Ivan

Wed Dec 13, 2023 2:44 pm

dunkaist wrote: ↑Wed Dec 13, 2023 10:33 amI tried your unified_kernel.zip patch and it doesn't compile.

That's strange, maybe different fasm versions? I could compile it, see kernel.mnt in the zip. Anyway, you're right this isn't needed at all, enough to support the latest.

dunkaist wrote: ↑Wed Dec 13, 2023 10:33 amI'm afraid that bootbios doesn't zero acpi_rsdp, devicesdat_*

It does for sure with my patch. In bootcode.inc:

Code: Select all

+        xor     eax, eax
+        mov     [BOOT_LO.devicesdat_data], eax
+        mov     [BOOT_LO.devicesdat_size], eax
+        mov     [BOOT_LO.acpi_rsdp], eax

As for the other fields, they are not affected (except BOOT_LO.vesa_mode, but that has to be cleared in the uefi loaders).

dunkaist wrote: ↑Wed Dec 13, 2023 10:33 amI don't think it's worth supporting booting the old (i.e. current) non-unified UEFI kernel in uefi*kos loaders. Just switch to the new scheme and that's it.

Yeah, you're right. I might have gone overprotective about backward compatibility here. Also supporting only the latest allows to remove the code that doesn't compile for you.

dunkaist wrote: ↑Wed Dec 13, 2023 10:33 amMaybe we should put the BIOS init code offset just after 'jmp start_of_code', i.e. just before 'Kolibri OS version ...'. This way we can avoid one magic number (2Ch) and a few padding bytes.

Good for me! I was afraid to move the 'Kolibri OS version ...' string because I wasn't sure if something is using it as magic bytes or not, so I've put after it just to be on the safe side (and at 2Ch because it's 4 bytes long, so this gives a header of 30h bytes in total: jmp, magic, protmode address. Anything after this header is guaranteed to be 16 bytes aligned, and also the current kernel has CDCDCDCDh here which is easy to detect for not being an unified kernel). But if you say that we are free to move the string, then good for me, any fixed address will do.

dunkaist wrote: ↑Wed Dec 13, 2023 10:33 amI can create an SVN account for you. Send me your desired password in a private message in case you don't want a randomly generated one.

Ok!

Cheers,
bzt

Thu Dec 14, 2023 6:32 am

You were right, I focused on kernel.mnt only, uefi4kos indeed did not compile! I wrote too much AT&T syntax lately, I accidently swapped the parameters. But I agree that check is unnecessary if we're only about to support the latest kernel.

I give it a little more thought, and here's a third alternative:

Code: Select all

        jmp     start_of_code
+version db     'Kolibri OS v0.7.7.0+    ',13,10,13,10,0
+        dd     B32-10000h ; place the BIOS initialization's code size at 20h

-if lang eq sp
-include "kernelsp.inc"  ; spanish kernel messages
-else if lang eq et
-version db    'Kolibri OS  versioon 0.7.7.0+    ',13,10,13,10,0
-else
-version db    'Kolibri OS  version 0.7.7.0+     ',13,10,13,10,0
-end if

include "boot/bootstr.inc"     ; language-independent boot messages

My reasoning:
- I don't think we need translated magic, just use "v" instead (translated "version" makes updating the actual version number in the string more difficult, as that's now scattered in multiple files)
- properly translated version message appears a little bit later anyway (from boot/bootstr.inc), so why duplicate it?
- this does not move the "Kolibri OS" part of the magic, so in case some programs already checking this magic, it won't break
- there are 4 additional spaces in the magic string, one for each number, so that the version can grow into more bytes (like "0.7.10.0+")
- no positioning required
- the protmode code's address is now at 20h
- the current kernel has 20202020h here, so still easy to detect not-the-latest, non-uniform kernels

Pick whichever alternative you'd like! It's equal to me, and you are the main developer, the choice should be yours.
bzt