I hoped someone else replies, I'm not an expert in VGA related stuff.
I fired up Kolibri OS 0.7.7 from a CD
Did you really try 0.7.7.0 release
from 2009 or it was a fresh nightbuild aka 0.7.7.0+?
When I booted, Kolibri gave me only 3 video options: one 320x200 and two 640x480 - and it auto-selected the better 640x480 with plenty of colours. I tried re-booting several times, with the same result. Then I went into the system screen and used the "Video options" icon. The panel displayed was blank - no options whatsoever.
KolibriOS can't change VESA modes on the fly. AFAIK, this application works together with VGA card specific drivers which don't exist for your card.
I found a debug screen that said my VESA was recognized - I hope that info might help.
If you mean PCIDEV program, what are VendorID/DeviceID columns for you card. Also, you can find a VESA version supported by hardware on the blue boot screen, is it 2.0?
The only other problem was the DEBUG panel. It shakes, is unstable and difficult to read.
Could you describe in more detail, please? It may be hard to read because of a number of logs at system start. What do you mean by unstable, does it crash?