Booting kernel

Internal structure and you change requests/suggestions
  • So what do you think? I'd love to have a simple way of locating the protected mode entry point, and then BIOS and UEFI could use the same kernel file.

    Cheers,
    bzt
  • Hi bzt,

    Thank you for your patch.
    bzt wrote: Sat Dec 09, 2023 8:44 am My understanding is, this separation was done by dunkaist, because his UEFI loader didn't know where the BIOS code ends.

    If you don't mind, my I suggest a different solution? It only increases the kernel's size by 7 bytes, but makes the file separation unnecessary.
    Indeed, jumping directly to the 32-bit code was one of reasons to separate BIOS code to bootbios.asm. Another reason was to avoid dead 16-bit code when booting on modern UEFI systems. In principle, I agree that getting the only universal kernel image is worth having some dead code. Unfortunately, things have got more complicated since separation of bootbios.asm. If you grep /kernel/trunk, you can see that UEFI macro is used in a number of files:

    Code: Select all

    $ grep 'defined UEFI' -r . --include="*.inc" --include="*.asm"
    ./core/apic.inc:if defined UEFI
    ./video/cursors.inc:if ~defined UEFI
    ./init.inc:if defined UEFI
    ./kernel.asm:if ~defined UEFI
    This is because some BOOT_LO fields are only set by our UEFI loader, while most or all BIOS loaders leave them uninitialized. We don't have proper versioning of the boot protocol and we have quite a few of exotic loaderes that e.g. boot KolibriOS from DOS or Windows. With that in mind, I just wrapped new UEFI logic in macro here and there. To remove these macros and build the universal BIOS/UEFI kernel image we should first go through all the BIOS loaders and make them to initialize the new BOOT_LO fields with some sensible values. This requires non-trivial testing too. I.e. this is the effort yet to be done. Your patch is a move to the right direction, but more moves are still required.

    In short, let's keep two separate images so far. There is a technical issue preventing unification that can be resolved in the future. I'm not ready to edit and test all the BIOS loaders at the moment, sorry.
  • dunkaist wrote: Mon Dec 11, 2023 10:07 amIndeed, jumping directly to the 32-bit code was one of reasons to separate BIOS code to bootbios.asm. Another reason was to avoid dead 16-bit code when booting on modern UEFI systems.
    Yes, but I think you shouldn't worry about that dead code. It's just a few kilobytes, and it provides backward compatibility out-of-the-box with all the non-UEFI loaders. (Those wouldn't mind this protmode entry pointer either.)
    dunkaist wrote: Mon Dec 11, 2023 10:07 amUnfortunately, things have got more complicated since separation of bootbios.asm. If you grep /kernel/trunk, you can see that UEFI macro is used in a number of files
    I did. I took the libery to assess the situation before I made this suggestion. I'm fairly certain they won't do anything (about 99.99999% sure).

    Code: Select all

    $ grep UEFI `find . -name '*.asm'` `find . -name '*.inc'`
    ./kernel.asm:if ~ defined UEFI
    ./init.inc:if defined UEFI
    ./init.inc:        ; UEFI loader knows where RSDP is
    ./video/cursors.inc:if ~defined UEFI
    ./blkdev/disk.inc:; GUID Partition Table Header, UEFI 2.6, Table 18
    ./blkdev/disk.inc:; GPT Partition Entry, UEFI 2.6, Table 19
    ./core/apic.inc:; non-UEFI loaders don't load DEVICES.DAT and don't initialize [acpi_dev_size]
    ./core/apic.inc:if defined UEFI
    ./bootloader/uefi4kos/uefi.inc:;; Based on UEFI library for fasm by bzt, Public Domain.        ;;
    ./bootloader/uefi4kos/uefi64.inc:;; Based on UEFI library for fasm by bzt, Public Domain.        ;;
    ./bootloader/uefi4kos/uefi32.inc:;; Based on UEFI library for fasm by bzt, Public Domain.        ;;
    $
    
    Now one by one:

    kernel.asm: this is just the one that prefixes the binary with the biosboot code.

    init.inc:

    Code: Select all

    if defined UEFI
            ; UEFI loader knows where RSDP is
            mov     ebx, [BOOT_LO.acpi_rsdp]
            test    ebx, ebx
            jz      .done
            call    .check
    else
            movzx   ebx, word [0x40E]
            shl     ebx, 4
            lea     ecx, [ebx+1024]
            call    .check               
    ...
    
    Considering that like you said, biosboot leaves the BOOT_LO zeroed out, this "if" could be removed. If acpi_rsdp is set, we can use it, if not, then the code could go on as usual. (However if I were you, I would consider moving the word [0x40E] part into bioscode and set BOOT_LO.acpi_rsdp there, so init.inc wouldn't have to care where that pointer come from.)

    video/cursors.inc:

    Code: Select all

    if ~defined UEFI
            cmp     [SCR_MODE], word 0x12
            jne     .not_vga
            ; TODO
    
    Similar, only this time it's the UEFI loader that leaves the mode field zero. (We must check if SCR_MODE is indeed initialized from BOOT_LO.mode, and then this "if" can be removed.)

    blkdev/disk.inc: just a comment about GPT

    core/apic.inc:

    Code: Select all

    ; non-UEFI loaders don't load DEVICES.DAT and don't initialize [acpi_dev_size]
    if defined UEFI
            cmp     [acpi_dev_size], 0
            jz      @f
            stdcall map_io_mem, [acpi_dev_data], [acpi_dev_size], PG_SWR
            mov     [acpi_dev_data], eax
            jmp     .loaded
    @@:
    end if
    
    This again won't cause trouble, just need to check if acpi_dev_size is indeed initialized from BOOT_LO.devicesdat_size (which biosboot zeroes out), and then it's safe to remove this "if".
    dunkaist wrote: Mon Dec 11, 2023 10:07 amWe don't have proper versioning of the boot protocol
    No need, there's no difference. It's just a few checks if some fields are filled it or not, neither the workflow, nor the BOOT_LO structure changed in any way.
    dunkaist wrote: Mon Dec 11, 2023 10:07 amwe have quite a few of exotic loaderes that e.g. boot KolibriOS from DOS or Windows.
    Again, not a problem because these just jump to biosboot code in the kernel, and they never jump to the protmode code.
    dunkaist wrote: Mon Dec 11, 2023 10:07 amTo remove these macros and build the universal BIOS/UEFI kernel image we should first go through all the BIOS loaders and make them to initialize the new BOOT_LO fields with some sensible values.
    You don't have to do any of this if you keep biosboot in the kernel (for now at least).

    I agree that removing biosboot and rewriting all loaders to come to a common ground with BOOT_LO is the way to go on the long term, but that's not what I suggesting here. I'm only suggesting a small step: keep biosboot, just add protmode entry to it at a fixed location. This way no matter what all those loaders do, because it's biosboot that initializes BOOT_LO on BIOS systems.
    dunkaist wrote: Mon Dec 11, 2023 10:07 amIn short, let's keep two separate images so far.
    That's problematic, because KOLIBRI.KRN (the UEFI variant) has no header, so there's no way of telling that it is actually a KolibriOS kernel.
    dunkaist wrote: Mon Dec 11, 2023 10:07 amThere is a technical issue preventing unification that can be resolved in the future. I'm not ready to edit and test all the BIOS loaders at the moment, sorry.
    Only if you about to remove biosboot, which I think you shouldn't. Maybe in the future when you're ready, but not now.

    I believe having a unified kernel (with well detectable magic bytes) is very very important, and I'm offering my help to achieve that. Anything you need, I can provide you patches and I can test, just let me know what you need! I strongly believe this worth the effort, and I'm ready to sacrifice me spare time for it.

    Cheers,
    bzt

    ps.: I'm glad to see that my uefi.inc was useful to you! :-D
  • Okay, here's a patch to help you with the unification. (Again, this does not change boot flow, it just removes the need for "if" macros.)

    First, this patch moves the BIOS based RSDP search from init.inc to bootcode.inc. This way init.inc can be firmware neutral, it shouldn't care where that BOOT_LO.acpi_rsdp come from.

    Code: Select all

    diff -r -U 3 aaa/kernel/boot/bootcode.inc bbb/kernel/boot/bootcode.inc
    --- aaa/kernel/boot/bootcode.inc	2023-10-23 14:53:06.000000000 +0200
    +++ bbb/kernel/boot/bootcode.inc	2023-12-11 19:46:40.072978847 +0100
    @@ -480,6 +480,35 @@
     apm_end:
             _setcursor d80x25_top_num, 0
     
    +; --------------- ACPI ---------------------
    +ACPI_LO_RSDP_WINDOW_END    = 0x000A0000
    +ACPI_HI_RSDP_WINDOW_START  = 0x000E0000
    +
    +acpi:   push    es
    +        xor     eax, eax
    +        mov     [BOOT_LO.devicesdat_data], eax
    +        mov     [BOOT_LO.devicesdat_size], eax
    +        mov     [BOOT_LO.acpi_rsdp], eax
    +        mov     ax, word [40Eh]
    +        add     ax, 64
    +.check: mov     es, ax
    +        cmp     [es:0], dword 'RSD '
    +        jne     .next
    +        cmp     [es:4], dword 'PTR '
    +        jne     .next
    +        shl     eax, 4
    +        mov     [BOOT_LO.acpi_rsdp], eax
    +        jmp     .end
    +.next:  inc     ax
    +        ; skip VRAM and ROM area A0000h to E0000h
    +        cmp     ax, (ACPI_LO_RSDP_WINDOW_END shr 4)
    +        jne     @f
    +        add     ax, ((ACPI_HI_RSDP_WINDOW_START - ACPI_LO_RSDP_WINDOW_END) shr 4)
    +@@:     ; end of 1Mb?
    +        or      ax, ax
    +        jnz     .check
    +.end:   pop     es
    +
     if ~ defined extended_primary_loader
     ;CHECK current of code
             cmp     [cfgmanager.loader_block], -1
    diff -r -U 3 aaa/kernel/init.inc bbb/kernel/init.inc
    --- aaa/kernel/init.inc	2023-10-23 14:53:06.000000000 +0200
    +++ bbb/kernel/init.inc	2023-12-11 19:25:11.392907707 +0100
    @@ -410,8 +410,6 @@
             ret
     endp
     
    -ACPI_HI_RSDP_WINDOW_START  = 0x000E0000
    -ACPI_HI_RSDP_WINDOW_END    = 0x00100000
     ACPI_RSDP_CHECKSUM_LENGTH  = 20
     
     proc acpi_locate_tables uses ebx esi edi
    @@ -475,27 +473,11 @@
             push    ebx
             push    edi
     
    -if defined UEFI
    -        ; UEFI loader knows where RSDP is
             mov     ebx, [BOOT_LO.acpi_rsdp]
             test    ebx, ebx
             jz      .done
    +        mov     edi, ebx
             call    .check
    -else
    -        movzx   ebx, word [0x40E]
    -        shl     ebx, 4
    -        lea     ecx, [ebx+1024]
    -        call    .check
    -
    -        test    ebx, ebx
    -        jz      @F
    -        jmp     .done
    -
    -@@:
    -        mov     ebx, ACPI_HI_RSDP_WINDOW_START
    -        mov     edi, ACPI_HI_RSDP_WINDOW_END
    -        call    .check
    -end if
     .done:
             mov     [acpi_rsdp_base - OS_BASE], ebx
             test    ebx, ebx
    


    Second, it's safe to remove "if ~defined UEFI" in video/cursors.inc, because in video/framebuffer.inc:

    Code: Select all

            movzx   eax, word [BOOT.vesa_mode]    ; screen mode
            mov     dword [SCR_MODE], eax
    
    So SCR_MODE comes from BOOT_LO.vesa_mode, which is always zero (so not 0x12 and not 0x4000) on UEFI.



    Lastly, "if defined UEFI" in core/apic.inc can be removed, because in kernel.asm

    Code: Select all

            mov     eax, [BOOT.devicesdat_size]
            mov     [acpi_dev_size], eax
            mov     eax, [BOOT.devicesdat_data]
            mov     [acpi_dev_data], eax
    
    So acpi_dev_size comes from BOOT_LO.devicesdat_size, and the above patch makes sure of it that this is zeroed out on BIOS.

    I've attached all my modifications in one zip. I've compiled and successfully booted this on BIOS as well as on UEFI machines (but obviously more tests are needed, I just performed the basic tests). I've also included the modified uefi4kos, which now supports this unified kernel, but at the same time is backward compatible with the current KOLIBRI.KRN kernel.

    Hope this helps!
    bzt
    Attachments
    unified_kernel.zip (175.51 KiB)
    Unified kernel
    Downloaded 144 times
  • Hi bzt,

    Since you're so interested in having this done, I agree to put some effort too.

    1.
    I tried your unified_kernel.zip patch and it doesn't compile.

    Code: Select all

    flat assembler  version 1.73.31  (16384 kilobytes memory, x64)
    uefi64kos.asm [1015]:
            cmp     'Koli', dword [eax + 3]
    processed: cmp 'Koli',dword[eax+3]
    error: invalid operand.
    Not a big deal but still. Same for uefi32kos.

    2.
    I'm afraid that bootbios doesn't zero acpi_rsdp, devicesdat_* and maybe other fields of BOOT_LO. At least, my GDB reports this.

    3.
    I don't think it's worth supporting booting the old (i.e. current) non-unified UEFI kernel in uefi*kos loaders. Just switch to the new scheme and that's it.

    4.
    Maybe we should put the BIOS init code offset just after 'jmp start_of_code', i.e. just before 'Kolibri OS version ...'. This way we can avoid one magic number (2Ch) and a few padding bytes.

    5.
    I can create an SVN account for you. Send me your desired password in a private message in case you don't want a randomly generated one.

    bzt wrote: Mon Dec 11, 2023 8:02 pm ps.: I'm glad to see that my uefi.inc was useful to you! :-D
    I really appreciate your work, it was of great help to me when I developed uefi4kos.

    Thank you,
    Ivan
  • dunkaist wrote: Wed Dec 13, 2023 10:33 amI tried your unified_kernel.zip patch and it doesn't compile.
    That's strange, maybe different fasm versions? I could compile it, see kernel.mnt in the zip. Anyway, you're right this isn't needed at all, enough to support the latest.
    dunkaist wrote: Wed Dec 13, 2023 10:33 amI'm afraid that bootbios doesn't zero acpi_rsdp, devicesdat_*
    It does for sure with my patch. In bootcode.inc:

    Code: Select all

    +        xor     eax, eax
    +        mov     [BOOT_LO.devicesdat_data], eax
    +        mov     [BOOT_LO.devicesdat_size], eax
    +        mov     [BOOT_LO.acpi_rsdp], eax
    As for the other fields, they are not affected (except BOOT_LO.vesa_mode, but that has to be cleared in the uefi loaders).
    dunkaist wrote: Wed Dec 13, 2023 10:33 amI don't think it's worth supporting booting the old (i.e. current) non-unified UEFI kernel in uefi*kos loaders. Just switch to the new scheme and that's it.
    Yeah, you're right. I might have gone overprotective about backward compatibility here. Also supporting only the latest allows to remove the code that doesn't compile for you.
    dunkaist wrote: Wed Dec 13, 2023 10:33 amMaybe we should put the BIOS init code offset just after 'jmp start_of_code', i.e. just before 'Kolibri OS version ...'. This way we can avoid one magic number (2Ch) and a few padding bytes.
    Good for me! I was afraid to move the 'Kolibri OS version ...' string because I wasn't sure if something is using it as magic bytes or not, so I've put after it just to be on the safe side (and at 2Ch because it's 4 bytes long, so this gives a header of 30h bytes in total: jmp, magic, protmode address. Anything after this header is guaranteed to be 16 bytes aligned, and also the current kernel has CDCDCDCDh here which is easy to detect for not being an unified kernel). But if you say that we are free to move the string, then good for me, any fixed address will do.
    dunkaist wrote: Wed Dec 13, 2023 10:33 amI can create an SVN account for you. Send me your desired password in a private message in case you don't want a randomly generated one.
    Ok!

    Cheers,
    bzt
  • You were right, I focused on kernel.mnt only, uefi4kos indeed did not compile! I wrote too much AT&T syntax lately, I accidently swapped the parameters. But I agree that check is unnecessary if we're only about to support the latest kernel.

    I give it a little more thought, and here's a third alternative:

    Code: Select all

            jmp     start_of_code
    +version db     'Kolibri OS v0.7.7.0+    ',13,10,13,10,0
    +        dd     B32-10000h ; place the BIOS initialization's code size at 20h
    
    -if lang eq sp
    -include "kernelsp.inc"  ; spanish kernel messages
    -else if lang eq et
    -version db    'Kolibri OS  versioon 0.7.7.0+    ',13,10,13,10,0
    -else
    -version db    'Kolibri OS  version 0.7.7.0+     ',13,10,13,10,0
    -end if
    
    include "boot/bootstr.inc"     ; language-independent boot messages
    
    My reasoning:
    - I don't think we need translated magic, just use "v" instead (translated "version" makes updating the actual version number in the string more difficult, as that's now scattered in multiple files)
    - properly translated version message appears a little bit later anyway (from boot/bootstr.inc), so why duplicate it?
    - this does not move the "Kolibri OS" part of the magic, so in case some programs already checking this magic, it won't break
    - there are 4 additional spaces in the magic string, one for each number, so that the version can grow into more bytes (like "0.7.10.0+")
    - no positioning required
    - the protmode code's address is now at 20h
    - the current kernel has 20202020h here, so still easy to detect not-the-latest, non-uniform kernels

    Pick whichever alternative you'd like! It's equal to me, and you are the main developer, the choice should be yours.
    bzt
  • I agree to use the only short form of the version as the kernel's magic bytes.

    I'd also ask you to change 'Kolibri OS' to 'KolibriOS' in that string and add another padding byte. KolibriOS seems to be our official spelling. It's not a formal thing but still.

    bzt wrote: Wed Dec 13, 2023 2:44 pm It does for sure with my patch. In bootcode.inc:

    Code: Select all

    +        xor     eax, eax
    +        mov     [BOOT_LO.devicesdat_data], eax
    +        mov     [BOOT_LO.devicesdat_size], eax
    +        mov     [BOOT_LO.acpi_rsdp], eax
    Sorry, but I don't see these lines in your .zip patch.
    bzt wrote: Wed Dec 13, 2023 2:44 pm Good for me! I was afraid to move the 'Kolibri OS version ...' string because I wasn't sure if something is using it as magic bytes or not
    I'm pretty sure you're the only person here who cares about kernel signature. :)
  • Okay, thanks for the input, here's the final patch then for the unified kernel!

    This includes everything we've talked about, has no "if defined UEFI", and the header is the small one with "KolibriOS" as you've asked. Hope it's going to be useful for others as well! Tested on BIOS and UEFI too, works like a charm.

    Please drop a post here when it's merged, and I'll update the Easyboot plugin as well.

    Cheers,
    bzt

    ps: I honestly don't think I'm the only one who likes to identify executable files by their headers ;-)
    Attachments
    unified_kernel.zip (159.16 KiB)
    as a zip (with compiled kernel.mnt)
    Downloaded 152 times
    as a diff patch
    Downloaded 127 times
  • bzt,

    Sorry, I still can't get your patch working. Maybe I'm doing something wrong but the kernel doesn't boot from floppy or hdd. If you're sure it works, feel free to commit, I sent your svn login/pass to you in a private message. I'm going to be afk for a week around Christmas.
  • dunkaist wrote: Tue Dec 19, 2023 10:56 pmSorry, I still can't get your patch working.
    What seems to be the problem? Any messages, screenshot?
    dunkaist wrote: Tue Dec 19, 2023 10:56 pmMaybe I'm doing something wrong but the kernel doesn't boot from floppy or hdd.
    This patch changed nothing about anything related to those. In fact, even the biosboot.bin should be unchanged for that (the ACPI pointer search is the only thing that's been added, and that should run long after the kernel booted and you finished with the configuration menu).

    If you can't even boot the kernel, could it be that you have issues with the boot sector or with the image and not the kernel maybe? I haven't touched bootloader/boot_fat12.asm at all.
    I had no troubles with booting the kernel.mnt I've attached, neither on BIOS, nor on UEFI.

    Cheers,
    bzt
  • bzt wrote: Tue Dec 19, 2023 11:20 pm What seems to be the problem? Any messages, screenshot?
    Just a blank black screen after the blue boot screen.
    bzt wrote: Tue Dec 19, 2023 11:20 pm the ACPI pointer search is the only thing that's been added
    Can it be that ds register isn't set when accessing BOOT_LO variables?

    I test with qemu as follows:

    Code: Select all

    qemu-system-i386 -nodefaults -vga std -display sdl -drive if=ide,media=disk,format=raw,file=kolibri.raw
  • dunkaist wrote: Wed Dec 20, 2023 12:57 amJust a blank black screen after the blue boot screen.
    So after the blue screen, then this must be the ACPI code.
    dunkaist wrote: Wed Dec 20, 2023 12:57 amCan it be that ds register isn't set when accessing BOOT_LO variables?
    Yes. Please try this:

    Code: Select all

    diff -r -U 3 aaa/kernel/boot/bootcode.inc bbb/kernel/boot/bootcode.inc
    --- aaa/kernel/boot/bootcode.inc	2023-12-11 19:46:40.072978847 +0100
    +++ bbb/kernel/boot/bootcode.inc	2023-12-20 15:48:54.091519602 +0100
    @@ -484,20 +484,22 @@
     ACPI_LO_RSDP_WINDOW_END    = 0x000A0000
     ACPI_HI_RSDP_WINDOW_START  = 0x000E0000
     
    -acpi:   push    es
    -        xor     eax, eax
    -        mov     [BOOT_LO.devicesdat_data], eax
    -        mov     [BOOT_LO.devicesdat_size], eax
    -        mov     [BOOT_LO.acpi_rsdp], eax
    +acpi:   xor     eax, eax
    +        mov     [es:BOOT_LO.devicesdat_data], eax
    +        mov     [es:BOOT_LO.devicesdat_size], eax
    +        mov     [es:BOOT_LO.acpi_rsdp], eax
    +        push    ds
    +        mov     ds, ax
             mov     ax, word [40Eh]
             add     ax, 64
    -.check: mov     es, ax
    -        cmp     [es:0], dword 'RSD '
    +.check: mov     ds, ax
    +        cmp     [ds:0], dword 'RSD '
             jne     .next
    -        cmp     [es:4], dword 'PTR '
    +        cmp     [ds:4], dword 'PTR '
             jne     .next
             shl     eax, 4
    -        mov     [BOOT_LO.acpi_rsdp], eax
    +        mov     [es:BOOT_LO.acpi_rsdp], eax
             jmp     .end
     .next:  inc     ax
             ; skip VRAM and ROM area A0000h to E0000h
    @@ -507,7 +509,7 @@
     @@:     ; end of 1Mb?
             or      ax, ax
             jnz     .check
    -.end:   pop     es
    +.end:   pop     ds
     
     if ~ defined extended_primary_loader
     ;CHECK current of code
    
    This explicitly sets ds, leaves es as-is to access BOOT_LO and uses ds to scan for the RSDPTR.

    As a whole, not as a diff (easier to review):

    Code: Select all

    acpi:   xor     eax, eax
            mov     [es:BOOT_LO.devicesdat_data], eax
            mov     [es:BOOT_LO.devicesdat_size], eax
            mov     [es:BOOT_LO.acpi_rsdp], eax
            push    ds
            mov     ds, ax
            mov     ax, word [40Eh]
            add     ax, 64
    .check: mov     ds, ax
            cmp     [ds:0], dword 'RSD '
            jne     .next
            cmp     [ds:4], dword 'PTR '
            jne     .next
            shl     eax, 4
            mov     [es:BOOT_LO.acpi_rsdp], eax
            jmp     .end
    .next:  inc     ax
            ; skip VRAM and ROM area A0000h to E0000h
            cmp     ax, (ACPI_LO_RSDP_WINDOW_END shr 4)
            jne     @f
            add     ax, ((ACPI_HI_RSDP_WINDOW_START - ACPI_LO_RSDP_WINDOW_END) shr 4)
    @@:     ; end of 1Mb?
            or      ax, ax
            jnz     .check
    .end:   pop     ds
    
    Hope this helps,
    bzt
  • bzt,

    I committed your patch in #9958 with my minor edits. Thank you.
  • Who is online

    Users browsing this forum: No registered users and 0 guests