My understanding is that most OS X doesn't allow applications to run fullscreen unless they match the desktop's color depth, meaning it's simply not going to work. There's a color depth setting in ag.cfg you can try, but I expect that'll cause worse problems.