Terrible quality when output to DVD


I am trying to create an SD DVD of a film project that contains a mix of live footage and animation (inter-cut, not in a Roger Rabbit way). The quality of the SD DVD, when watched on an HD TV (1080p) is really pretty poor, but not uniformly for the different footage. The film will be mastered in HD and the DVDs are just for showing work in progress, but ultimately I want to produce festival screeners etc, so need the quality to be vastly better. I’m hoping that someone might be able to suggest where the problem is and what can be done to improve the overall result.

For the animation, when there is little or no camera movement, the main problem is jagged lines on the characters’ black outlines and a bit of noise. Pans and pull backs cause significant noise and a pronounced shimmering, especially on finer lines. The live footage is ok-ish (darker, with a loss of vibrancy in the colours). Text graphics, particularly when placed over live footage are poor (very jagged). Simple white text on black background is ok.

My workflow is as follows:

Hardware for all processes:
Macbook Pro 2.6 GHz with Intel Core 2 Duo,
OS Mac OS X v 10.5.8.
Memory: 4GB 667 MHz

Animation produced start to finish on TBS 5.0 (Build 5.0.13834).
Live footage shot on XDCAM EX at 1080p25
Editing is on FCP (v7.0.3). Text graphics are generated on FCP. The FCP file is exported directly to Compressor 3.5.3 to create the MPEG2 and then brought into DVD Studio Pro (v 4.2.2) to create the SD DVD. SD video format is PAL.

The TBS settings are:
Aspect ratio: 16 x 9
Resolution: 1920 x 1080
Frame rate: 25 fps
TBS Export settings: to QuickTime movie
Compression Type: Apple Pro Res 422 (HQ); Quality High; Automatic Gamma Correction; Progressive; Dimensions 1920 x 1080; frame rate current
All filter settings as per default

The timeline settings for FCP are as follows:
Sequence settings: XDCAM EX 1080p25
General: Frame size 1920 x 1080; Aspect ratio HDTV 1080i 16:9; Pixel Aspect Ratio ‘square’; Field dominance ‘none’
Video Processing: Render all YUV material in high precision YUV; Process Maximum White as ‘white’; Motion Filtering Quality ‘best’
Render Control: Filters & frame blending for speed – both checked
Render: frame rate 100%; Resolution 100%; Codec Apple ProRes 422 (HDV, XDCAM HD/EX only)
Master Templates and Motion Projects: best; ‘Always use Best when Rendering Movies’ – checked.
I’ve pretty much tried all variations of encoder settings in Compressor for DVD Best Quality – presets and tweaking the various video format options (e.g. field dominance, quality settings, GOP and frame control and filter options). None of the variations made any observable difference. The only thing that improved the animation slightly was applying a noise removal filter in the chroma channel at 4 iterations. Although it reduced the noise, it didn’t improve jagging significantly of animation or text and made the rest of the footage soft.

If someone out there is routinely doing this and can recommend some appropriate settings and where the problem may be coming from, I’d be really grateful to hear from you!