antialiasing settings

i feel like i am missing some obvious information here, but i’ve never quite understood the variables for these settings. i am attempting to use full scene antialiasing instead of real-time. harmony will render the output files without compromise, even if antialiasing is off in the authoring environment, but i don’t know how to maximize the sample value itself. i know that part of this is just using my ol’ eyeballs, but i’m hoping a good samaritan or toonboom could elaborate on controlling this intelligently.

- with full scene antialiasing enabled, how might you determine a necessary sample amount? should you have a single optimal setting, or do you need greater amounts for more demanding shapes/situations?

- is there a way to know, based on one’s installed video card, what the maximum sample value should be? the stated aa specs for my radeon hd 4870 are:
- Multi-sample anti-aliasing (2, 4, or 8 samples per pixel)
- Up to 24x Custom Filter Anti-Aliasing (CFAA) for improved quality
- Adaptive super-sampling and multi-sampling
so, does that mean that harmony “locks in” at those values of 2, 4, 8, or does it have anything to gain from that “CFAA” spec? would going higher than 8 be useful, or is that the threshold where further increments would not be visibly superior in the authoring environment?

- if going higher than 8 is useful, should i stick to single digits, or go further, perhaps in “stepped” increments like multiples of 2 or 4 (say, after 8, try 16)?

i made an animated gif to demonstrate a test i made. it’s a series of full-scene aa screen captures, sample values noted, followed by an RT one:

it looks like going beyond 8 in full scene for me isn’t an improvement – capped at 8. switching to real time, however, looks significantly better than the best full scene. i need to experiment more with an actual animated scene as far as performance, but is that quality gap to be expected? the documentation steers you away from real-time aa usage, but it sure does look better while drawing.

does either aa mode have issues when used with effects? does full-scene aa interpret all the networked effects correctly? i have actually used real-time aa mostly in the past, using animate pro, but i’m looking at switching over to full-scene.

You should use full-scene aa in most cases. This aa is done by your graphic card on the fly in the OpenGL views.

The Real-time AA is actually doing a mini render every time you draw something. It does a render at the scene’s resolution taking the 12 field area as it’s reference box. It will not take into account camera zoom ins or scaling the layer with a peg for example. This AA was made to give the animator an approximate preview of the quality the line will have at the output so that he would not overwork and loose time trying to fix minute imperfections. Some clean up artists would zoom in very close and fiddle trying to remove bumps which in the final render would be represented by only one or two pixels. This AA has the drawback that is will slow down a lot your drawing because of the constant re-rendering it does.

Neither AAs have any impact on final rendered images. The final rendered images aa is controlled in the layer’s properties.

The value for the Full-scene AA should be below 8 in most cases. It will depend on the smoothness you want to see when drawing, the complexity of your scene and the power of your graphic card. The higher the setting the slower your playback will become. It was chosen originaly to not do any AA so that the playback, while doing scene planning (camera and layer movements), would be faster. Draftsmen-women often find they prefer some softer lines so that is why you can add AA.

So bottom line put it as high as you want until you see the performance degrade. I find a 2 or 4 is sufficient for my liking.

stevemasson, thanks for your detailed response. what do you mean by “the 12 field area”? can you explain?

is there a global way to set all layers to render w/ “high” aa, perhaps a preference to set all new layers to have this max. rendered aa value, or a network node process to force this for an entire scene render? it defaults to medium on all layers, and i can only seem to change it on one layer at a time. also, any guidelines on the “aa exponent” in these settings? it defaults to 1, wonder how high it can/should go without getting silly. i see the field can be set between -1000 to 1000. negative values?!

i noticed that in RT aa mode, setting the first value to match my working canvas-zoom level seemed to provide the optimal smoothing. for example, setting it to 1 when working at 100%, 2 when zoomed in 200%, 3 for 300%, so on. zooming in to the workspace beyond that “matched-up” value causes shapes to get artifacted/fuzzy, and working at, say, 100% while the factor value is set to 2 or 3 seemed to also cause a lack of smoothness compared to the “matched-up” value. on my system, regardless of the first RT aa factor value, 64 tiles for the second value field seems to be a sweet spot – it draws pretty quickly (though overall it doesn’t match FS aa speed), though setting it to a higher value (say, 128) or even lower (?!) caused noticeable slow-down.

also, RT aa can be re-set on the fly whilst working, whereas full-scene required an application restart each time you changed it (which isn’t terrible, though on my dual-monitor setup i end up having to re-arrange menus and stuff after a restart).

a-ha – in app preferences, network tab, you can set the default rendered aa value and exponent, which will apply to all NEW elements/layers created after this setting is made. it doesn’t seem to update the value in existing elements when changed, though. i’d still like to know more about that exponent value.

documentation states:
“Controls the amount of area around the Line and Colour Art edges that is used in the antialiasing process. A higher value uses less area, resulting in sharper edges, while a lower value uses more area, resulting in softer edges”.

what about negative settings, and what are reasonable values? it’s a little counter-intuitive that higher values would render less smooth. i guess further tests are in order…

From what I remember the usable range is 0 to 1 and negative doesn’t work.

If you want to change the existing layer modules you would have to create a QSA script. I’m not a pro at scripts but I modified an existing one to do it. Here it is. You just have to change the value of High and the exponent to what you want. Select the modules in the Network you want to change and run the script.

function Change_layer_AntiAliasing()
for (var sel_index = 0 ; sel_index < selection.numberOfNodesSelected(); sel_index++)
var node_name = selection.selectedNode( sel_index );

MessageLog.trace( node.getTextAttr( node_name, 1, “antialiasingQuality” ) );
node.setTextAttr( node_name, “antialiasingQuality”, 1, “High” ); //The choices for the Anti-Aliasing are Low, Medium Low, Medium and High.
MessageLog.trace( node.getTextAttr( node_name, 1, “antialiasingExponent” ) );
node.setTextAttr( node_name, “antialiasingExponent”, 1, “0.3” ); //The choices for the Anti-Aliasing Exponent are between 0 and 1.

yes, thanks very much for the info steve and lilly. this is already a very useful discussion thread for me.

i really appreciate your time making this great script, and this is the first time i’ve looked at one! it looks like javascript or actionscript to me so it seems quite approachable.

lilly, what is the purpose then of having those numeric fields go higher than 1, or negative? is there a process where that would be necessary? it’s cool that it is possible, but i’m wondering if that is deliberate flexibility or maybe it was just overlooked to restrict this.

The purpose for that factor (real-time AA) is if you intend to zoom in eventually it would give you a preview closer to the final result.

For example, lets say you are drawing a small character and you are doing a zoom from 12 field to 6 field. To see what the pixels will look on your character when the camera has zoom all the way in you would set the factor to 2 because you would want to have double the resolution on the preview.

If your camera (or scale of the layer) stayed at 12 field you would need a resolution equal to the scene. If you zoom in to 6 you would want the preview to be equal to the scene resolution at 6 field (resolution x 2).

Just to expand on the question about what does he mean by the 12-field area, he means the drawing grid when you turn on the grid in the drawing view. Each drawing that you do is mapped onto this drawing grid, so when you turn on real-time antialiasing, it actually does a render of this grid, which is why it’s slow.

And when you push the values up higher than 1, then you’re rendering it out at a higher resolution than your scene will render in the final render, so you’ll see more details than you need to in fact.

Hope that helps!