Board.KolibriOS.org

Official KolibriOS board
https://board.kolibrios.org/

UMKa -- User-Mode KolibriOS developer tools

https://board.kolibrios.org/viewtopic.php?t=4195

Page 1 of 1

UMKa -- User-Mode KolibriOS developer tools

Posted: Wed May 06, 2020 2:37 am
by dunkaist
Hi there!

I've been developing umka for quite some time. It's not finished yet but can be useful already.

What is UMKa?

The idea is to make userspace UNIX tools that use as much unchanged
KolibriOS source as possible to test architecture-independent parts of the
kernel in your favorite developer environment. It is a common project for a
set of KolibriOS developer tools which are based on original KolibriOS code
wrapped and hacked as to run in the UNIX programming environment.

I.e. umka is not actually an emulator. It embeds some kernel code of KolibriOS and runs
it natively in the userspace.

The idea is not new.
The most similar project I'm aware of is LKL.

What I Can Do with UMKa
  • Run some parts of KolibriOS kernel code in Linux userspace, without emulators like QEMU or Bochs.
  • Debug kernel code conveniently (e.g. with gdb).
  • Write and run reproducible automatic tests of kolibri kernel code, including unit tests.
  • Collect line and branch coverage of kolibri kernel code transparently, i.e. without additional instrumentation code.
What I Can Not Do with UMKa

You can't run KolibriOS userspace code (programs, libs) and hardware dependent kernel code (e.g. device drivers).

Which Parts of the Kernel Can Be Compiled with UMKa?
  • Block layer
    • Underlying devices are replaced with regular files -- disk images.
    • Since media is accessed via uniform callback interface (thanks CleverMouse), block layer code doesn't care if it reads a floppy drive, ATA disk or a file.
    • Examples: disk cache, MBR and GPT scanning, adding and deleting disks.
  • FS
    • All FATs, NTFS. EXTs and XFS are supported.
    • ISO9660 (Joliet?) code accesses hardware directly and thus isn't compiled in umka.
  • LFB graphics
    • Instead of the linear framebuffer UMKa uses a simple memory buffer.
    • You can dump the buffer to a PNG image to e.g. compare against the reference image.
  • A number of small self-contained functions.
  • May be some other code later.
What Is Already Done
  • A couple of tests for XFS. Can be used to write tests for other file systems.
  • An example test for graphics (test/016_#f01_#draw_all.t): pixel, line, rectangle, image, text (including smoothing), button, skin, related internal structures (win_stack, win_pos).
  • A tool to collect line and branch coverage on test runs.
How It Works
  • No more changes to KolibriOS code are required to use UMKa! Preprocessor magic.
  • umka.asm includes those kernel files that can run in userspace, adds stdcall wrappers, provides a few stub functions, does a number of fasm preprocessor hacks to compile it all to umka.o.
  • Tools are built around interfaces of umka.o.
  • Coverage is collected via LBR. Slowdown is about 100x.(
What Tools Are Available

umka_shell
A shell that reads commands from stdin or a file, can run interactively.

Example session:

Code: Select all

$ ../umka_shell
/> disk_add ../img/xfs_v4_ftype1_s05k_b2k_n8k.img hd0 -c 0
/hd0/1: xfs
/> disk_add ../img/xfs_short_dir_i8.img hd1 -c 0
/hd1/1: xfs

/> ls70 /hd0/1/sf_empty
status = 6 end_of_file, count = 2
total = 2
----f .
----f ..
/> ls70 /hd0/1/sf
status = 6 end_of_file, count = 5
total = 5
----f .
----f ..
----f d0000000000_
----f d0000000001_x
----f d0000000002_xx

/> disk_del hd0
/> disk_del hd1
^C
umka_fuse

FUSE based file system, operates on disk images using KolibriOS kernel FS code.

The difference with umka_shell is that requests to KolibriOS code are not written by hand. Instead, they are translated from FUSE calls when you walk mounted FS using cd, ls, stat and other standard UNIX utilities.

Example session:

Code: Select all

$ mkdir /tmp/m

$ ./umka_fuse /tmp/m kolibri.img

$ ls /tmp/m
hd0

$ ls /tmp/m/hd0/
1

$ ls /tmp/m/hd0/1
 3D         CALENDAR      DEMOS      END              GAMES        ICONS16.PNG   KUZKINA.MID   MADMOUSE   MYKEY         RTFREAD    SKINCFG      TEST          VMODE
 ALLGAMES   COLRDIAL      DEVELOP    ESKIN            GMON         ICONS32.PNG   LANG.INC      MAGNIFY    NETWORK       RUN        @SS          TINFO         @VOLUME
 APM        CONFIG.INC    DISPTEST   EXAMPLE.ASM      HACONFIG     INDEX.HTM     LAUNCHER      MEDIA      @NOTIFY       SCRSHOOT   STRUCT.INC   TINYPAD       ZKEY
 APP_PLUS   CPU           @DOCKY     FB2READ          HDD_INFO     KBD           LIB           @MENU      NOTIFY3.PNG   SEARCHAP   SYSMON       TMPDISK
 ASCIIVJU   CPUID         DOCPACK   'File Managers'   HOME.PNG     KERNEL.MNT    LOADDRV       MGB        @OPEN         SETTINGS   SYSPANEL     TOOLBAR.PNG
 BARSCFG    CROPFLAT      DRIVERS    FONTS            @HOTANGLES   KF_VIEW       LOD           MOUSECFG   PCIDEV        SETUP      @TASKBAR     TXTREAD
 CALC       DEFAULT.SKN   EASYSHOT   FSPEED           @ICON        KPACK         MACROS.INC    MOUSEMUL   RDSAVE        SHELL      TERMINAL     UNZ

$ ls /tmp/m/hd0/1/LIB
ARCHIVER.OBJ  BUF2D.OBJ    CONSOLE.OBJ  ICONV.OBJ     KMENU.OBJ   LIBIMG.OBJ  LIBIO.OBJ   NETCODE.OBJ  PIXLIB.OBJ    RASTERWORKS.OBJ  TINYGL.OBJ
BOX_LIB.OBJ   CNV_PNG.OBJ  HTTP.OBJ     INPUTBOX.OBJ  LIBGFX.OBJ  LIBINI.OBJ  MSGBOX.OBJ  NETWORK.OBJ  PROC_LIB.OBJ  SORT.OBJ

$ fusermount -u /tmp/m
Unlike umka_shell, umka_fuse is more like a PoC at the moment.

How to Build

Code: Select all

$ KOLIBRI=/path/to/kolibrios make
/path/to/kolibrios is where you checked out `svn co svn://kolibrios.org`.

How to Run Tests

First, generate disk images.

Code: Select all

$ cd img

$ make
Then run tests.

Code: Select all

$ cd test

$ make -B
../umka_shell < 001_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype1.t > 001_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype1.out.log
../umka_shell < 002_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype0.t > 002_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype0.out.log
../umka_shell < 003_#f70_#f70s0_#xfs_#s05k_read_without_holes.t > 003_#f70_#f70s0_#xfs_#s05k_read_without_holes.out.log
../umka_shell < 004_#f70_#f70s5_#xfs_stat.t > 004_#f70_#f70s5_#xfs_stat.out.log
../umka_shell < 005_#f30_#xfs_cur_dir.t > 005_#f30_#xfs_cur_dir.out.log
../umka_shell < 006_#f70_#f70s0_#xfs_#s05k_read_with_holes.t > 006_#f70_#f70s0_#xfs_#s05k_read_with_holes.out.log
../umka_shell < 007_#f70_#xfs_#lookup_all.t > 007_#f70_#xfs_#lookup_all.out.log
../umka_shell < 008_#f70_#f70s1_#xfs_#s4k_ls_all_dir_types_ftype0.t > 008_#f70_#f70s1_#xfs_#s4k_ls_all_dir_types_ftype0.out.log
../umka_shell < 009_#f70_#f70s0_#f70s1_#xfs_#s05k_#xattr.t > 009_#f70_#f70s0_#f70s1_#xfs_#s05k_#xattr.out.log
../umka_shell < 010_#f70_#xfs_#lookup_#xattr_all.t > 010_#f70_#xfs_#lookup_#xattr_all.out.log
../umka_shell < 011_#f70_#f70s1_#xfs_#lookup_#unicode_all.t > 011_#f70_#f70s1_#xfs_#lookup_#unicode_all.out.log
../umka_shell < 012_#f70_#f70s1_#xfs_#v5_#s05k_ls_all_dir_types_ftype1.t > 012_#f70_#f70s1_#xfs_#v5_#s05k_ls_all_dir_types_ftype1.out.log
../umka_shell < 013_#f70_#xfs_#v5_#lookup_all.t > 013_#f70_#xfs_#v5_#lookup_all.out.log
../umka_shell < 014_#f70_#f70s0_#xfs_#v5_#s05k_read_with_holes.t > 014_#f70_#f70s0_#xfs_#v5_#s05k_read_with_holes.out.log
../umka_shell < 015_#f70_#f70s0_#xfs_#v5_#s05k_read_without_holes.t > 015_#f70_#f70s0_#xfs_#v5_#s05k_read_without_holes.out.log
../umka_shell < 017_#f70_#f70s0_#xfs_#s05k_read_above_4GiB.t > 017_#f70_#f70s0_#xfs_#s05k_read_above_4GiB.out.log
../umka_shell < 018_#xfs_#coverage_all.t > 018_#xfs_#coverage_all.out.log
../umka_shell < 019_#xfs_deep_btree.t > 019_#xfs_deep_btree.out.log
../umka_shell < 016_#f01_#draw_all.t > 016_#f01_#draw_all.out.log
all tests passed
How to Collect Coverage

Code: Select all

$ cd test

$ sudo modprobe msr

$ make -B COVERAGE=pew
...
$ ../covpreproc ../umka.lst 0x34 coverage.* > ../umka.cov
Coverage file looks as follows.

Code: Select all

 3716 -        844/0               844 :         jz      @f
 3717  -                             0 :         test    eax, eax
 3718 --         0/0                 0 :         jnz     .dyndisk_cleanup
 3719  -                             0 :         dec     esi
 3720                                  : @@:
 3721                              844 :         cmp     byte [esi], 0
 3722          835/9               844 :         jnz     @f
line 3716: covered, branch (844 times taken, 'not taken' uncovered)
line 3717: uncovered, not a branch
line 3718: uncovered, branch (both uncovered)
line 3722: covered, branch (835 times taken, 9 not taken)

Troubleshooting
  • umka_* tools are 32-bit. On 64-bit platforms they need 32-bit libraries. Just in case, here is libfuse3.
  • There is an issue with documentation in our project, umka is not an exception. Feel free to ping me.
Source

Code: Select all

$ git clone https://baravy.by/r/umka.git
UMKa runs on Linux, therefore I don't think it should be committed to KolibriOS repository.
I didn't test it under Windows. In theory, it should be possible. Patches are welcome.

Re: UMKa -- User-Mode KolibriOS developer tools

Posted: Wed May 06, 2020 10:52 pm
by CleverMouse
For FASM, taken/not-taken statistics for branches can be collected just by overriding jcc/jcxz/jecxz, something like this:

Code: Select all

macro redeclare_jcc [cc]
{
    macro j#cc arg \{
        push    eax
        set#cc  al
        movzx   eax, al
        inc     [stat_table+stat_id*2+eax]
        pop     eax
        stat_id = stat_id + 1
        j#cc    arg
    \}
}
redeclare_jcc a,b,c,e,g,l,o,p,s,z,ae,be,ge,le,na,nb,nc,ne,ng,nl,no,np,ns,nz,pe,po,nae,nbe,nge,nle

Re: UMKa -- User-Mode KolibriOS developer tools

Posted: Tue Dec 07, 2021 11:03 am
by Boppan
Hi, the clang-conpatibility patch is in the main branch. All available to me tests are passed.

Dunkaist, please, verify if the patch didnt't break any of your tests.

Re: UMKa -- User-Mode KolibriOS developer tools

Posted: Sat Dec 11, 2021 1:24 pm
by Boppan
Hi!

UMKa is ported to Windows, changes are available in windows branch.

Please verify if the changes didn't break anything for you (all tests I could execute are successful and no workflow change is required).

Also please note that it makes really big ammount of changes in umka.asm in order to make it compileable both in win32 and linux (pubsym macro is added to mangle names as windows requires).

Re: UMKa -- User-Mode KolibriOS developer tools

Posted: Sat Dec 11, 2021 11:34 pm
by Boppan
Hi!

Just added a new pretty complicated test for FAT32. About to merge it once you approve changes with clang migration in the main branch.

Once the test is in the new branch we'll be able to be more brave making changes in FAT filesystem implementation.

The test is on fat32_test0 branch.

Re: UMKa -- User-Mode KolibriOS developer tools

Posted: Sun Dec 12, 2021 6:49 pm
by Boppan
Hi!

UMKa is integrated into kernel test system. Since it hasn't been confirmed that the changes made in the main branch was safe for your tests I've created trunk branch that is used in out test system.

Now all the tests added to the branch will be available on svn.

Now it should be created some system to run specific tests instead of just going wild and executing whatever is in the test directory.

Re: UMKa -- User-Mode KolibriOS developer tools

Posted: Tue Dec 28, 2021 11:34 am
by Boppan
Forgot to mention: now UMKa's main branch works on Windows, thanks to dunkaist for support.
All times are UTC+03:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited