Fully supporting the Xbox video hardware could mean a lot of work (or at least a lot of copying from the Xbox kernel). But there is no need to implement a complete video driver. Both bootloaders provided by the Xbox Linux project set up a 640x480 screen (32 bit colour, RGBX), which is useful for many setups. This linear framebuffer lies at the top of memory (around 60 MB) and can be used by your operating system without ever touching any video registers. This just works like the VESA modes supported by most operating systems, which get set up by the boot loader (using BIOS functions) and never get switched afterwards. Have a look at the bootloader section to learn how to find out where this framebuffer is located.
If you want full support, you can reuse driver code for nVidia video hardware, as the Xbox video hardware is fully nVidia compatible. The problem is just that the Xbox does not have a VGA RAMDAC, but a PAL/NTSC video encoder (which is connected through the SMBus), so you always need to make sure when you set up a new mode that the video encoder is also correctly set up, or else you will get garbage on the screen.
Unfortunately, there is one of three incompatible video encoders in every Xbox: The first Xboxes used Connexant chips (1.0 to 1.3), later ones used Focus components (1.4 and 1.5), and 1.6 Xboxes include an integrated Xcalibur unit. That is why you might have to copy a lot of code from the Xbox Linux kernel and have others with different Xboxes test your code. You can find all this code in drivers/video/xbox/*.