UMKa -- User-Mode KolibriOS developer tools

• 

  dunkaist Mentor

  Posts 814

  Joined: Mon Oct 19, 2009 10:58 am

  Wed May 06, 2020 2:37 am

  Hi there!
  
  I've been developing umka for quite some time. It's not finished yet but can be useful already.
  
  What is UMKa?
  
  The idea is to make userspace UNIX tools that use as much unchanged
  KolibriOS source as possible to test architecture-independent parts of the
  kernel in your favorite developer environment. It is a common project for a
  set of KolibriOS developer tools which are based on original KolibriOS code
  wrapped and hacked as to run in the UNIX programming environment.
  
  I.e. umka is not actually an emulator. It embeds some kernel code of KolibriOS and runs
  it natively in the userspace.
  
  The idea is not new.
  The most similar project I'm aware of is LKL.
  
  What I Can Do with UMKa
  

  • Run some parts of KolibriOS kernel code in Linux userspace, without emulators like QEMU or Bochs.
  • Debug kernel code conveniently (e.g. with gdb).
  • Write and run reproducible automatic tests of kolibri kernel code, including unit tests.
  • Collect line and branch coverage of kolibri kernel code transparently, i.e. without additional instrumentation code.

  What I Can Not Do with UMKa
  
  You can't run KolibriOS userspace code (programs, libs) and hardware dependent kernel code (e.g. device drivers).
  
  Which Parts of the Kernel Can Be Compiled with UMKa?
  

  • Block layer
    • Underlying devices are replaced with regular files -- disk images.
    • Since media is accessed via uniform callback interface (thanks CleverMouse), block layer code doesn't care if it reads a floppy drive, ATA disk or a file.
    • Examples: disk cache, MBR and GPT scanning, adding and deleting disks.
  • FS
    • All FATs, NTFS. EXTs and XFS are supported.
    • ISO9660 (Joliet?) code accesses hardware directly and thus isn't compiled in umka.
  • LFB graphics
    • Instead of the linear framebuffer UMKa uses a simple memory buffer.
    • You can dump the buffer to a PNG image to e.g. compare against the reference image.
  • A number of small self-contained functions.
  • May be some other code later.

  What Is Already Done
  

  • A couple of tests for XFS. Can be used to write tests for other file systems.
  • An example test for graphics (test/016_#f01_#draw_all.t): pixel, line, rectangle, image, text (including smoothing), button, skin, related internal structures (win_stack, win_pos).
  • A tool to collect line and branch coverage on test runs.

  How It Works

  • No more changes to KolibriOS code are required to use UMKa! Preprocessor magic.
  • umka.asm includes those kernel files that can run in userspace, adds stdcall wrappers, provides a few stub functions, does a number of fasm preprocessor hacks to compile it all to umka.o.
  • Tools are built around interfaces of umka.o.
  • Coverage is collected via LBR. Slowdown is about 100x.(

  What Tools Are Available
  
  umka_shell
  A shell that reads commands from stdin or a file, can run interactively.
  
  Example session:

  Code: Select all

  $ ../umka_shell
  /> disk_add ../img/xfs_v4_ftype1_s05k_b2k_n8k.img hd0 -c 0
  /hd0/1: xfs
  /> disk_add ../img/xfs_short_dir_i8.img hd1 -c 0
  /hd1/1: xfs
  
  /> ls70 /hd0/1/sf_empty
  status = 6 end_of_file, count = 2
  total = 2
  ----f .
  ----f ..
  /> ls70 /hd0/1/sf
  status = 6 end_of_file, count = 5
  total = 5
  ----f .
  ----f ..
  ----f d0000000000_
  ----f d0000000001_x
  ----f d0000000002_xx
  
  /> disk_del hd0
  /> disk_del hd1
  ^C

  umka_fuse
  
  FUSE based file system, operates on disk images using KolibriOS kernel FS code.
  
  The difference with umka_shell is that requests to KolibriOS code are not written by hand. Instead, they are translated from FUSE calls when you walk mounted FS using cd, ls, stat and other standard UNIX utilities.
  
  Example session:

  Code: Select all

  $ mkdir /tmp/m
  
  $ ./umka_fuse /tmp/m kolibri.img
  
  $ ls /tmp/m
  hd0
  
  $ ls /tmp/m/hd0/
  1
  
  $ ls /tmp/m/hd0/1
   3D         CALENDAR      DEMOS      END              GAMES        ICONS16.PNG   KUZKINA.MID   MADMOUSE   MYKEY         RTFREAD    SKINCFG      TEST          VMODE
   ALLGAMES   COLRDIAL      DEVELOP    ESKIN            GMON         ICONS32.PNG   LANG.INC      MAGNIFY    NETWORK       RUN        @SS          TINFO         @VOLUME
   APM        CONFIG.INC    DISPTEST   EXAMPLE.ASM      HACONFIG     INDEX.HTM     LAUNCHER      MEDIA      @NOTIFY       SCRSHOOT   STRUCT.INC   TINYPAD       ZKEY
   APP_PLUS   CPU           @DOCKY     FB2READ          HDD_INFO     KBD           LIB           @MENU      NOTIFY3.PNG   SEARCHAP   SYSMON       TMPDISK
   ASCIIVJU   CPUID         DOCPACK   'File Managers'   HOME.PNG     KERNEL.MNT    LOADDRV       MGB        @OPEN         SETTINGS   SYSPANEL     TOOLBAR.PNG
   BARSCFG    CROPFLAT      DRIVERS    FONTS            @HOTANGLES   KF_VIEW       LOD           MOUSECFG   PCIDEV        SETUP      @TASKBAR     TXTREAD
   CALC       DEFAULT.SKN   EASYSHOT   FSPEED           @ICON        KPACK         MACROS.INC    MOUSEMUL   RDSAVE        SHELL      TERMINAL     UNZ
  
  $ ls /tmp/m/hd0/1/LIB
  ARCHIVER.OBJ  BUF2D.OBJ    CONSOLE.OBJ  ICONV.OBJ     KMENU.OBJ   LIBIMG.OBJ  LIBIO.OBJ   NETCODE.OBJ  PIXLIB.OBJ    RASTERWORKS.OBJ  TINYGL.OBJ
  BOX_LIB.OBJ   CNV_PNG.OBJ  HTTP.OBJ     INPUTBOX.OBJ  LIBGFX.OBJ  LIBINI.OBJ  MSGBOX.OBJ  NETWORK.OBJ  PROC_LIB.OBJ  SORT.OBJ
  
  $ fusermount -u /tmp/m

  Unlike umka_shell, umka_fuse is more like a PoC at the moment.
  
  How to Build
  

  Code: Select all

  $ KOLIBRI=/path/to/kolibrios make

  /path/to/kolibrios is where you checked out `svn co svn://kolibrios.org`.
  
  How to Run Tests
  
  First, generate disk images.

  Code: Select all

  $ cd img
  
  $ make

  Then run tests.

  Code: Select all

  $ cd test
  
  $ make -B
  ../umka_shell < 001_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype1.t > 001_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype1.out.log
  ../umka_shell < 002_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype0.t > 002_#f70_#f70s1_#xfs_#s05k_ls_all_dir_types_ftype0.out.log
  ../umka_shell < 003_#f70_#f70s0_#xfs_#s05k_read_without_holes.t > 003_#f70_#f70s0_#xfs_#s05k_read_without_holes.out.log
  ../umka_shell < 004_#f70_#f70s5_#xfs_stat.t > 004_#f70_#f70s5_#xfs_stat.out.log
  ../umka_shell < 005_#f30_#xfs_cur_dir.t > 005_#f30_#xfs_cur_dir.out.log
  ../umka_shell < 006_#f70_#f70s0_#xfs_#s05k_read_with_holes.t > 006_#f70_#f70s0_#xfs_#s05k_read_with_holes.out.log
  ../umka_shell < 007_#f70_#xfs_#lookup_all.t > 007_#f70_#xfs_#lookup_all.out.log
  ../umka_shell < 008_#f70_#f70s1_#xfs_#s4k_ls_all_dir_types_ftype0.t > 008_#f70_#f70s1_#xfs_#s4k_ls_all_dir_types_ftype0.out.log
  ../umka_shell < 009_#f70_#f70s0_#f70s1_#xfs_#s05k_#xattr.t > 009_#f70_#f70s0_#f70s1_#xfs_#s05k_#xattr.out.log
  ../umka_shell < 010_#f70_#xfs_#lookup_#xattr_all.t > 010_#f70_#xfs_#lookup_#xattr_all.out.log
  ../umka_shell < 011_#f70_#f70s1_#xfs_#lookup_#unicode_all.t > 011_#f70_#f70s1_#xfs_#lookup_#unicode_all.out.log
  ../umka_shell < 012_#f70_#f70s1_#xfs_#v5_#s05k_ls_all_dir_types_ftype1.t > 012_#f70_#f70s1_#xfs_#v5_#s05k_ls_all_dir_types_ftype1.out.log
  ../umka_shell < 013_#f70_#xfs_#v5_#lookup_all.t > 013_#f70_#xfs_#v5_#lookup_all.out.log
  ../umka_shell < 014_#f70_#f70s0_#xfs_#v5_#s05k_read_with_holes.t > 014_#f70_#f70s0_#xfs_#v5_#s05k_read_with_holes.out.log
  ../umka_shell < 015_#f70_#f70s0_#xfs_#v5_#s05k_read_without_holes.t > 015_#f70_#f70s0_#xfs_#v5_#s05k_read_without_holes.out.log
  ../umka_shell < 017_#f70_#f70s0_#xfs_#s05k_read_above_4GiB.t > 017_#f70_#f70s0_#xfs_#s05k_read_above_4GiB.out.log
  ../umka_shell < 018_#xfs_#coverage_all.t > 018_#xfs_#coverage_all.out.log
  ../umka_shell < 019_#xfs_deep_btree.t > 019_#xfs_deep_btree.out.log
  ../umka_shell < 016_#f01_#draw_all.t > 016_#f01_#draw_all.out.log
  all tests passed

  How to Collect Coverage
  

  Code: Select all

  $ cd test
  
  $ sudo modprobe msr
  
  $ make -B COVERAGE=pew
  ...
  $ ../covpreproc ../umka.lst 0x34 coverage.* > ../umka.cov

  Coverage file looks as follows.

  Code: Select all

   3716 -        844/0               844 :         jz      @f
   3717  -                             0 :         test    eax, eax
   3718 --         0/0                 0 :         jnz     .dyndisk_cleanup
   3719  -                             0 :         dec     esi
   3720                                  : @@:
   3721                              844 :         cmp     byte [esi], 0
   3722          835/9               844 :         jnz     @f
  

  line 3716: covered, branch (844 times taken, 'not taken' uncovered)
  line 3717: uncovered, not a branch
  line 3718: uncovered, branch (both uncovered)
  line 3722: covered, branch (835 times taken, 9 not taken)
  
  Troubleshooting
  

  • umka_* tools are 32-bit. On 64-bit platforms they need 32-bit libraries. Just in case, here is libfuse3.
  • There is an issue with documentation in our project, umka is not an exception. Feel free to ping me.

  Source
  

  Code: Select all

  $ git clone https://baravy.by/r/umka.git

  UMKa runs on Linux, therefore I don't think it should be committed to KolibriOS repository.
  I didn't test it under Windows. In theory, it should be possible. Patches are welcome.

  • Quote

macro redeclare_jcc [cc]
{
    macro j#cc arg \{
        push    eax
        set#cc  al
        movzx   eax, al
        inc     [stat_table+stat_id*2+eax]
        pop     eax
        stat_id = stat_id + 1
        j#cc    arg
    \}
}
redeclare_jcc a,b,c,e,g,l,o,p,s,z,ae,be,ge,le,na,nb,nc,ne,ng,nl,no,np,ns,nz,pe,po,nae,nbe,nge,nle