Board.KolibriOS.org

Hi there!

I've been developing umka for quite some time. It's not finished yet but can be useful already.

What is UMKa?

The idea is to make userspace UNIX tools that use as much unchanged
KolibriOS source as possible to test architecture-independent parts of the
kernel in your favorite developer environment. It is a common project for a
set of KolibriOS developer tools which are based on original KolibriOS code
wrapped and hacked as to run in the UNIX programming environment.

I.e. umka is not actually an emulator. It embeds some kernel code of KolibriOS and runs
it natively in the userspace.

The idea is not new.
The most similar project I'm aware of is LKL.

What I Can Do with UMKa

• Run some parts of KolibriOS kernel code in Linux userspace, without emulators like QEMU or Bochs.
• Debug kernel code conveniently (e.g. with gdb).
• Write and run reproducible automatic tests of kolibri kernel code, including unit tests.
• Collect line and branch coverage of kolibri kernel code transparently, i.e. without additional instrumentation code.

What I Can Not Do with UMKa

You can't run KolibriOS userspace code (programs, libs) and hardware dependent kernel code (e.g. device drivers).

Which Parts of the Kernel Can Be Compiled with UMKa?

• Block layer
  
  • Underlying devices are replaced with regular files -- disk images.
  • Since media is accessed via uniform callback interface (thanks CleverMouse), block layer code doesn't care if it reads a floppy drive, ATA disk or a file.
  • Examples: disk cache, MBR and GPT scanning, adding and deleting disks.
• FS
  
  • All FATs, NTFS. EXTs and XFS are supported.
  • ISO9660 (Joliet?) code accesses hardware directly and thus isn't compiled in umka.
• LFB graphics
  
  • Instead of the linear framebuffer UMKa uses a simple memory buffer.
  • You can dump the buffer to a PNG image to e.g. compare against the reference image.
• A number of small self-contained functions.
• May be some other code later.

What Is Already Done

• A couple of tests for XFS. Can be used to write tests for other file systems.
• An example test for graphics (test/016_#f01_#draw_all.t): pixel, line, rectangle, image, text (including smoothing), button, skin, related internal structures (win_stack, win_pos).
• A tool to collect line and branch coverage on test runs.

How It Works

• No more changes to KolibriOS code are required to use UMKa! Preprocessor magic.
• umka.asm includes those kernel files that can run in userspace, adds stdcall wrappers, provides a few stub functions, does a number of fasm preprocessor hacks to compile it all to umka.o.
• Tools are built around interfaces of umka.o.
• Coverage is collected via LBR. Slowdown is about 100x.(

What Tools Are Available

umka_shell
A shell that reads commands from stdin or a file, can run interactively.

Example session:

$ ../umka_shell
/> disk_add ../img/xfs_v4_ftype1_s05k_b2k_n8k.img hd0 -c 0
/hd0/1: xfs
/> disk_add ../img/xfs_short_dir_i8.img hd1 -c 0
/hd1/1: xfs

/> ls70 /hd0/1/sf_empty
status = 6 end_of_file, count = 2
total = 2
----f .
----f ..
/> ls70 /hd0/1/sf
status = 6 end_of_file, count = 5
total = 5
----f .
----f ..
----f d0000000000_
----f d0000000001_x
----f d0000000002_xx

/> disk_del hd0
/> disk_del hd1
^C

umka_fuse

FUSE based file system, operates on disk images using KolibriOS kernel FS code.

The difference with umka_shell is that requests to KolibriOS code are not written by hand. Instead, they are translated from FUSE calls when you walk mounted FS using cd, ls, stat and other standard UNIX utilities.

Example session:

$ mkdir /tmp/m

$ ./umka_fuse /tmp/m kolibri.img

$ ls /tmp/m
hd0

$ ls /tmp/m/hd0/
1

$ ls /tmp/m/hd0/1
 3D         CALENDAR      DEMOS      END              GAMES        ICONS16.PNG   KUZKINA.MID   MADMOUSE   MYKEY         RTFREAD    SKINCFG      TEST          VMODE
 ALLGAMES   COLRDIAL      DEVELOP    ESKIN            GMON         ICONS32.PNG   LANG.INC      MAGNIFY    NETWORK       RUN        @SS          TINFO         @VOLUME
 APM        CONFIG.INC    DISPTEST   EXAMPLE.ASM      HACONFIG     INDEX.HTM     LAUNCHER      MEDIA      @NOTIFY       SCRSHOOT   STRUCT.INC   TINYPAD       ZKEY
 APP_PLUS   CPU           @DOCKY     FB2READ          HDD_INFO     KBD           LIB           @MENU      NOTIFY3.PNG   SEARCHAP   SYSMON       TMPDISK
 ASCIIVJU   CPUID         DOCPACK   'File Managers'   HOME.PNG     KERNEL.MNT    LOADDRV       MGB        @OPEN         SETTINGS   SYSPANEL     TOOLBAR.PNG
 BARSCFG    CROPFLAT      DRIVERS    FONTS            @HOTANGLES   KF_VIEW       LOD           MOUSECFG   PCIDEV        SETUP      @TASKBAR     TXTREAD
 CALC       DEFAULT.SKN   EASYSHOT   FSPEED           @ICON        KPACK         MACROS.INC    MOUSEMUL   RDSAVE        SHELL      TERMINAL     UNZ

$ ls /tmp/m/hd0/1/LIB
ARCHIVER.OBJ  BUF2D.OBJ    CONSOLE.OBJ  ICONV.OBJ     KMENU.OBJ   LIBIMG.OBJ  LIBIO.OBJ   NETCODE.OBJ  PIXLIB.OBJ    RASTERWORKS.OBJ  TINYGL.OBJ
BOX_LIB.OBJ   CNV_PNG.OBJ  HTTP.OBJ     INPUTBOX.OBJ  LIBGFX.OBJ  LIBINI.OBJ  MSGBOX.OBJ  NETWORK.OBJ  PROC_LIB.OBJ  SORT.OBJ

$ fusermount -u /tmp/m

Unlike umka_shell, umka_fuse is more like a PoC at the moment.

How to Build

$ KOLIBRI=/path/to/kolibrios make

/path/to/kolibrios is where you checked out `svn co svn://kolibrios.org`.

How to Run Tests

First, generate disk images.

$ cd img

$ make

Then run tests.

$ cd test

$ make -B
../umka_shell < 001_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype1.t > 001_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype1.out.log
../umka_shell < 002_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype0.t > 002_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype0.out.log
../umka_shell < 003_#f70_#f70s0_#xfs_#s05k_read_without_holes.t > 003_#f70_#f70s0_#xfs_#s05k_read_without_holes.out.log
../umka_shell < 004_#f70_#f70s5_#xfs_stat.t > 004_#f70_#f70s5_#xfs_stat.out.log
../umka_shell < 005_#f30_#xfs_cur_dir.t > 005_#f30_#xfs_cur_dir.out.log
../umka_shell < 006_#f70_#f70s0_#xfs_#s05k_read_with_holes.t > 006_#f70_#f70s0_#xfs_#s05k_read_with_holes.out.log
../umka_shell < 007_#f70_#xfs_#lookup_all.t > 007_#f70_#xfs_#lookup_all.out.log
../umka_shell < 008_#f70_#f70s1_#xfs_#s4k_ls_all_dir_types_ftype0.t > 008_#f70_#f70s1_#xfs_#s4k_ls_all_dir_types_ftype0.out.log
../umka_shell < 009_#f70_#f70s0_#f70s1_#xfs_#s05k_#xattr.t > 009_#f70_#f70s0_#f70s1_#xfs_#s05k_#xattr.out.log
../umka_shell < 010_#f70_#xfs_#lookup_#xattr_all.t > 010_#f70_#xfs_#lookup_#xattr_all.out.log
../umka_shell < 011_#f70_#f70s1_#xfs_#lookup_#unicode_all.t > 011_#f70_#f70s1_#xfs_#lookup_#unicode_all.out.log
../umka_shell < 012_#f70_#f70s1_#xfs_#v5_#s05k_ls_all_dir_types_ftype1.t > 012_#f70_#f70s1_#xfs_#v5_#s05k_ls_all_dir_types_ftype1.out.log
../umka_shell < 013_#f70_#xfs_#v5_#lookup_all.t > 013_#f70_#xfs_#v5_#lookup_all.out.log
../umka_shell < 014_#f70_#f70s0_#xfs_#v5_#s05k_read_with_holes.t > 014_#f70_#f70s0_#xfs_#v5_#s05k_read_with_holes.out.log
../umka_shell < 015_#f70_#f70s0_#xfs_#v5_#s05k_read_without_holes.t > 015_#f70_#f70s0_#xfs_#v5_#s05k_read_without_holes.out.log
../umka_shell < 017_#f70_#f70s0_#xfs_#s05k_read_above_4GiB.t > 017_#f70_#f70s0_#xfs_#s05k_read_above_4GiB.out.log
../umka_shell < 018_#xfs_#coverage_all.t > 018_#xfs_#coverage_all.out.log
../umka_shell < 019_#xfs_deep_btree.t > 019_#xfs_deep_btree.out.log
../umka_shell < 016_#f01_#draw_all.t > 016_#f01_#draw_all.out.log
all tests passed

How to Collect Coverage

$ cd test

$ sudo modprobe msr

$ make -B COVERAGE=pew
...
$ ../covpreproc ../umka.lst 0x34 coverage.* > ../umka.cov

Coverage file looks as follows.

 3716 -        844/0               844 :         jz      @f
 3717  -                             0 :         test    eax, eax
 3718 --         0/0                 0 :         jnz     .dyndisk_cleanup
 3719  -                             0 :         dec     esi
 3720                                  : @@:
 3721                              844 :         cmp     byte [esi], 0
 3722          835/9               844 :         jnz     @f

line 3716: covered, branch (844 times taken, 'not taken' uncovered)
line 3717: uncovered, not a branch
line 3718: uncovered, branch (both uncovered)
line 3722: covered, branch (835 times taken, 9 not taken)

Troubleshooting

• umka_* tools are 32-bit. On 64-bit platforms they need 32-bit libraries. Just in case, here is libfuse3.
• There is an issue with documentation in our project, umka is not an exception. Feel free to ping me.

Source

$ git clone https://baravy.by/r/umka.git

UMKa runs on Linux, therefore I don't think it should be committed to KolibriOS repository.
I didn't test it under Windows. In theory, it should be possible. Patches are welcome.

For FASM, taken/not-taken statistics for branches can be collected just by overriding jcc/jcxz/jecxz, something like this:

macro redeclare_jcc [cc]
{
    macro j#cc arg \{
        push    eax
        set#cc  al
        movzx   eax, al
        inc     [stat_table+stat_id*2+eax]
        pop     eax
        stat_id = stat_id + 1
        j#cc    arg
    \}
}
redeclare_jcc a,b,c,e,g,l,o,p,s,z,ae,be,ge,le,na,nb,nc,ne,ng,nl,no,np,ns,nz,pe,po,nae,nbe,nge,nle

_________________
Сделаем мир лучше!