You can quickly "hint" the window to a specific size with the following command:

VBoxManage controlvm <uuid>|<name> setvideomodehint <xres> <yres> <bpp>
For example, if you have a guest called "Windows XP Pro", and you wish to set the resolution to 1024x600 at 32 bits per pixel (aka 32 bit) you would use the following command:

VBoxManage controlvm "Windows XP Pro" setvideomodehint 1024 600 32
There are some prerequisites for this however:

This only works if you've launched the Guest OS from a windowed environment on the Host OS (ie. Windows / X Windows / Mac OS Desktop).

Guest Additions must be installed in the Guest OS.