Some of these are over a decade old and though the tools used have changed since then, the techniques still remain the same for the most part. The things that have changed are generally ways we can track these roto shapes onto moving objects in the frame (examples of which are using a planar tracker like mocha, or smart vectors, or projection on a 3d tracked card).

The Art Of Rotocoping

This is a great fxguide article outlining the history, applications, software and basic tips.

Scott Squires Rotoscoping

Great videos showing a good rundown of rotoscoping techniques for beginners.

0:41 Rotoscoping Basics and Examples

2:50 Tracing for animation

6:42 Matting with GreenScreen

8:02 Garbage Matting

9:55 Mid Ground Roto

12:08 Compositing Mid Ground

14:54 Creating Twins

19:10 Colorizing

20:36 Animated Wipe

Here is his article on rotoing hair.

Silhouette Roto tutorial video

Mocha Pro Rotoscoping Basics

Nuke Rotoscoping Basics

If you aren’t already using something to automatically create shot structures for you (i use the wonderful smartlib withing nuke), you should look at this mac/win freebie. It will save you time and mean all your shots are in clear structures. Here’s a video on how it works (and click here to go to their page):

https://m.youtube.com/watch?feature=youtu.be&v=4jdj_lE_er8

The proEXR package has been around for a decade and bundles plugins for Photoshop, premiere pro, & after effects. Indeed adobe has bundled 3 of the packs plugins in after effects for many years. Having access to all the layers from a CG or comp render in Photoshop is handy, and now you can pick it all up for free it is worth checking out (click to visit page).

The alternative which is worth a peak also is exr-io (click here to go to site). As both plugins are free now and offer similar features, check out both to see which appeals most.

They have rolled it into a Nuke node but as of yet it is purely an in house tool (quite possibly will never be released to the public but they have released the research paper so odds are some boffin might cobble together their own version).

Check out this video for an overview.

They have published the paper in PDF form which you can get here.

Another tip is if you’re using a decent Digi green or blue screen and you want markers you can quickly throw on the screen or move with little hassle, make up some markers using velcro (hooked side). On most modern screens the markers can write literally be thrown on the screen and will stick. Even handy if you have pre-marked the screen using coloured gaffer tape markers but they go for a close up and you see no markers are in shot. In this case you can quickly throw a couple of trackers into frame (faster than tape).

If you want to matchmove a shot you really want to have things at different depths for the software to use that parallax to work out the camera movement. You can either add green sticks (or c-stands) just out from the screen with markers on or make sure you have props with trackable textures. In that regard, work with the art Dept to make props, set pieces, that have some features that can be tracked built in (so the audience just sees a normal prop, the matchmove artist has decent textures to track in the set, & the comper doesn’t get angry having to paint out coloured dots on the set). Again, camera tests before the shoot will tell you what you need for your particular situation. 

Shapes of markers is a subject that’s debated. Crosses are favored by some as one cross can provide multiple points to track which can help in frames where you don’t have many markers in view. The down side is when they get out of focus their shape becomes a bit like a blobby  cloud which can change frame to frame and upset the tracker. Because of this, some match movers prefer round dots (of course that means you need to make sure you have at least 7 in shot at any one time, at different parallax distances for a matchmove (unless the set has trackable features). Heres an article and video about that.

For those not on the shoot but handed coloured markers on screens that need removing you can try having a look at this  (click here) Nukepedia written article for inspiration.
At the end of the day you’ll most likely have to endure some paint work if the above doesn’t work but creating a garbage matte so you only deal with stuff crossing close to the edge of your subject will limit this pain.

Here are a couple of videos HollywoodCamerawork did as an intro to matchmove marker placement (from their vfx course).

For more info on matchmoving in general I recommend this book:

* note that the image at the top of this article is also from HollywoodCameraworks site (click here). If you want free vfx plates to practice your keying or tracking check them out. Also their camera work course is awesome and I’ve found the vfx course good to help explain certain techniques to directors in the planning stages. 

You could just attach a tracker directly to the stmap it generates but even with a small target box size it may get confused by pixel values in your target box, distorting locally. What you need is a way to pick one pixel colour in the stmap and track it around the image as it moves. Magno Borgo has come up with a way to do this and has rolled it up into a simple nuke group you can use (note you can always swap out the second node, which auto tracks the max luma, for a tracker node if you prefer what it offers you, but I kind of like the way he does it. Here is the link to the nuke group and below is his video discussing it.