I am not sure what you mean. Could you provide a close up of a portion of the screen that captures an example of the poor quality? That screen capture doe not attract my attention to anything in particular.
Jagged lines can be improved by going into Preferences of Toon Boom under Display and checking all Rendering Options and sliding the Texture & Images Resolution to the right towards Better Quality then restarting the program.
You might wonder whether Render options have anything to do with pre-render quality. In 3D software these options control display quality in working/creation mode. It is more productive having a truer representation of the end result available while building a project so video cards have been designed to do a lot more work to generate a better quality representation of the data.
I am on a Mac too and using the full (paid) version of TBS 7. I didn’t expect any improvement from the Open GL Anti-Aliasing because I just have a standard MacMini video card however there could be Open GL enhancements in all Macs nowadays because it improved the quality and it did not prompt me saying anything was wrong as I would expect if I did not have the hardware to support the option.
I am quite impressed with the speed in creation modes. I saw nothing that slowed down performance after selecting everything I suspected could improve display quality.