There is a big difference in behavior depending on the fact whether the thumbnail already exists or not.
If it does not exist then it will always be created according to the logic described in module_imagefunctions function auto_thumbnail.
Also have a look at the timthumb-config.php where you can specify the zoom crop behavior.
So the behavior setting has no influence on that.
If it does exist and behavior 3 is selected then the dimensions of the found thumbnail are compared to the dimensions specified in the upload settings. If it not matches then a recreation is forced.
So much for the creation of the thumbnails.
But this setting also influences the HTML dimensions of the img tag. For setting 1 and 3 the upload settings are used and for setting 2 the dimensions that are found for the thumbnail itself. This can give peculiar effects if you create your own thumbnails.
Then there is an additional setting called specthumbmax which offers the possibility to define a maximum width or height. This is useful when you have images next to each other with different orientations. So for a portrait oriented image this value will be used to maximize its height and for a landscape oriented image for its width.
So suppose your upload settings are 300x200 and the found thumbnail has dimensions 600x400 then the img tag will have dimensions 300x200 for setting 1 and 3 (the latter also resulting in recreation of the thumbnail). Setting 2 gives you an img tag of 600x400.
Remember that you can specify fancybox_thumbnail for the behavior installation wide and thumbbehav parm on the fancybox template tag to overrule this.
You have to select your own complex of settings depending on the situation. There is no simple process. If you want to be absolutely safe always create your own thumbnails and use setting 2 (so you are sure they won't be recreated and are always shown in the dimensions you specified when creating them). If you want something else you'll have to think...................
Quality is in the detail of things............