Do you ever edge up with hideous edges in your images like this when you blend two layers?  These sorts of edge issues are common when placing a person on a new background, replacing a sky, or manually blending exposures to bring back a beautiful sunset.  Thankfully, Photoshop has a secret weapon called “Refine Edge” (aka “Refine Mask”) to help clean up these nasty edges.  It’s an amazing tool that is actually pretty simple to use once you understand it.  But it can be intimidating and confusing at first, so I created this tutorial to help demystify the magic of Refine Edge.

There are several scenarios where I find I find Refine Edge to be absolutely indispensable:

• Manual exposure blending with hard edges.  This is common when blending a darker sunset exposure into a lighter building/skyline exposure.  Not only is this a very hard edged selection (even with a luminosity mask), but the pixels often won’t match up perfectly.  If you compare the highlight pixels on the lighter exposure to the same pixels on the darker exposure, you’ll see that they are often just a little bit different.  The reason is that some of the highlights spill over by a pixel or two in the brighter exposure.  So a mask that is perfectly built on one layer won’t be perfectly aligned with the edge on the other layer.  Refine Edge offers a very simple and effective way to create a mask that naturally transitions from one to the other.  Without it, you’ll see a small halo at this high contrast edges.
• Cutting a person/object out to put on a new background.  Even if you have an ideal background (such as a medium gray seamless backdrop) that you can easily select with the magic wand or quick select tool, the edges of the selection shouldn’t be masked perfectly black or white.  There are typically fine details such as hair/fur, you might have translucent objects that mix with the background, or an edge pixel may be a combination of the subject and background.  Refine Edge is perfect for this job.
• To be honest, pretty much any hard mask created with the built-in Photoshop selection tools is a candidate for enhancement with Refine Edge.  Using Refine Edge can add that element of subtlety that makes all the difference between a soft-edged selection/mask that looks great and a hard-edged one that looks obviously “Photoshopped”.

 

So that covers why we should use Refine Edge, but how do we actually use it?   There are so many options in the tool that it can feel overwhelming.  But it isn’t so bad once you get the hang of it.  In the following video, I go into detail on what the various refine options do and how to use them effectively.  While the video shows how to correct a landscape image, the general techniques are very applicable to portraits and other work.

 

 

View mode

You need to see what you’re doing, and this is where you get to determine how to visualize the changes that are being made.  These settings only help you visualize the refinements, they don’t actually do anything to change the output of the Refine Edge tool.  There are many options here, I only use a few of them:

• “On Layers“.  This is what your layered image will look like after applying the refinements, and is therefore the most important view.  Press <L> for a shortcut to this view.
• “Black and White“.  This is your mask, plain and simple.  Press <K> for a shortcut to this view.
• If you are trying to cut an object out from the background, then the “On black” and “On white” visualizations are helpful to see if you’ve accurately selected your subject.
• If you are trying to brush in areas that weren’t originally in the mask (such as strands of hair), “Reveal Layer” can be helpful to see where to brush.
• “Show Original” checkbox.  This shows your layered image before any refinements.  Press <P> to turn this on and off to check how much your refinements have improved the image.
• “Show Radius” checkbox:  This shows where the edge/smart radius are refining the mask.  If you don’t see important edges here, it is likely that they are low contrast (which is common with luminosity masks).  This is your queue to use the brush tool below to make sure these edges get cleaned up as well.   Press <J> to turn this view on and off to check that you are refining important edges.

And, just like elsewhere in Photoshop, you can use <space> to get the hand tool to move around the view and <ctrl/cmd> +/- to quickly zoom in and out.

 

Edge Detection

This is the “magic” in Refine Edge.  By magic, I mean both the incredible power to make stunning masks that you could never make by hand, as well as a black box full of confusion.  I’ll do my best to demystify the options here, as using them correctly will allow you to make some funny looking masks look amazing.  Photoshop will first look for edges that need refinement and, within a certain pixel radius of that edge, use the layer’s pixel data to refine the mask.  It will generally try to create a gradient from black to white, and will try to include more of the types of pixels that are dominant in the mask.   My general approach is to tweak the radius for the best overall look, try the smart radius, and then brush edges in or out to refine little details if needed.

• Radius.  This setting determines how far from the detected edges Photoshop should refine the mask.  Within this zone (which you can view by turning on “show radius”), Refine Edge will try to add and remove areas of the mask automatically.  For manual exposure blending of high contrast edges, I tend to keep this value pretty low, around 1-3 pixels, depending mostly on the resolution of the image.  For hair selections and other messiness, a larger radius may be needed.  Radius will typically do a pretty good job finding and fixing high contrast edges, but will tend to ignore more subtle edges, which can mean that it misses edges in luminosity masks if the mask edge is more gray than white.  You can quickly spot missed edges by turning on “show radius”.  When radius isn’t finding these edges, the brush tool comes to the rescue…
• Smart radius.  This option is intended to deal with masks that include both simple and complex edges (such as a skyline that includes both straight-edged buildings and messy tree details).  In a nutshell, when smart radius is on, Refine Edge will use a smaller radius in areas with simple/clean edge, and a larger radius in areas with complex/messy edges.  In reality, it’s it’s a bit more complicated, and smart radius has no effect if the radius is set to a value less than 10.0.  Rather than trying to understand the rocket science behind it, the best way to understand it is to turn “show radius” on and off to see if it improves the result for mixed edges.  While smart radius can be helpful, I often find that a better strategy is to use a modest radius and then use the “refine radius” brush described next to effectively increase the radius in the more complex areas.
• Refine Radius Tool (the brush).  This allows you to selectively tell Refine Edge to target areas outside the radius (rather than increasing the radius globally).  This will include more edge details (lightens stuff that was missed).  This is a critical tool for smoothing edges in luminosity masks if the mask isn’t white enough at the edge.  This is also a great option for extraneous details such as a railing or antenna on top of a building or random strands of hair in a portrait.  Use a brush width large enough to brush over the target edges, but don’t make it so large that you start adjusting non-target areas.  If you make a brush stroke and don’t like the result, you can always press -Z to undo the last brush stroke.  If you want to go back multiple steps, you have to start over.
• Erase Refinements tool (the eraser).   This is the opposite of the brush and will cause Refine Edge to leave these targeted areas alone (ie, this tells Refine Edge to keep the original mask edge in the erased areas).  Note that not only will the erase cause Refine Edge to ignore anything you brushed in, it will also cause the tool to ignore anything detected with the radius.  If you erase the wrong area, <cmd/ctrl>-Z is your friend, or you might consider cancelling and starting over.  While you can click and hold on the brush to switch to the eraser, a much easier option is to always leave the brush on and just hold  <alt/option> when you need to use the eraser.

 

Adjust Edge

These tools are a little more straight-forward, as they take the mask (as adjusted by edge detection) and just apply simple changes to it.  Most of the time, I just use a tiny bit of feathering and leaving the rest of these settings alone.

• Feather:  Blurs (softens) the mask.  Unlike the radius tool (which analyzes the edges to try to include just the right detail), feather is a dumb tool that affects everything – so you typically don’t want to use much feathering on photographs (a little goes a long way).  I typically use 0-1 pixels when manually blending exposures at a hard edge (such as a bright sunset sky at the edge of a building/trees).  This helps produce an anti-aliasing effect to soften the edges.
• Shift edge:  Positive values expand the mask, and negative values cause it to shrink.  This can be helpful in a few scenarios: when masking translucent objects, when trying to remove fringing from colored backgrounds, or when blending two exposures at a high contrast edge.  I generally leave this in the 0-10% range for my landscape work.  This is especially helpful to overcome issues where highlights spill over by a few pixels in the brighter exposure, causing a mismatch of edges.  Note that this tool is mostly moving gray parts of the mask to black or white.  So the same amount of shifting can cause a much more dramatic change on a luminosity mask than a hard-edged mask.
• Smooth:  Reduces sharp areas of the mask.  I never use this and can’t think of a good reason to use this on a photograph (maybe to tweak a “sloppy border” effect?)
• Contrast:  Creates a sharper edge to the mask.  I never use this with my photographs, but you might find a scenario where you want to tighten up a loose/blurry edged mask.

 

Output

• Output to:  I leave this on “layer mask” when I’m adjusting a mask or “selection” when refining a selection, unless I use the next option…
• Decontaminate Colors:  Turn this on to get rid of color cast when cutting something out from a colored background, such as a green screen behind hair.  Adjust the amount as needed.  If you use this, you will be forced to output to a new layer or document, as Photoshop is going to change the pixel data, not just the mask.  If you use decontaminate, output to “new layer with layer mask” is probably the most useful option (and you can then delete or hide the source layer).

 

General workflow

There’s definitely no right way to use Refine Edge, but I find the following workflow keeps things simple and easy for me.

1. Adjust the radius until you find what looks best.
2. Use brush to include/protect more of the edge as needed.  Press <ctrl/cmd>-Z to undo the last stroke.  Note that you can only undo a single step, so take some care or you’ll need to start over.  You cannot simply use the eraser to undo the brush.
3. Use the eraser to protect any areas that should not be refined.  Again, take care, as you can only undo one stroke before you need to cancel and start over.
4. Adjust feather.  A little goes a long way.
5. Adjust shift edge.  A little goes a long way.
6. Try using “decontaminate colors” if you see fringing at the edges (even if the fringing is gray).  Note that this will create a new layer, as it makes destructive changes to the pixels.  Alternatively, instead us using the built-in decontaminate option, you can achieve the same result by creating a new blank layer, clipping it to the masked layer, and then painting in adjustments in the edge areas.  Toggle this new layer’s blend mode (or your paint brush) between color and luminosity blend mode as needed.  Using a layer to decontaminate has the advantage of being a more non-destructive workflow (you are likely still changing your mask in the process, but you are preserving the option to change it later without having destroyed the underlying image pixels).
7. Throughout the process, try using the view options suggested above to help visualize the refinement.  I typically use several of these views before I finally hit the “ok” button.

 

A simpler workflow for Lumenzia users

If you prefer to remove guesswork and speed up the process, Lumenzia is designed to simplify the process of refining edges of both masks and selections.  It also offers a simple way to “decontaminate colors” in a non-destructive manner.  Lumenzia will analyze the image and load the Refine Mask/Selection tool (it supports both) with recommended settings.  Simply click “Edge” to use the Refine tool with settings optimized for landscape exposure blending or -click “Edge” to use settings optimized for cutting out portraits from a backdrop.  You can accept the defaults or tweak the settings.  I generally use the following workflow:

1. Click “Edge” in Lumenzia to start with recommended settings, and adjust if desired.
2. Use the brush tool to include more of the edge as needed.  I don’t typically use the eraser because Lumenzia’s “Restore” tool offers a much more simple and controlled way to protect parts of the mask.  Just click “Restore” before and after “Edge” to paint any edges back to their original state.  This is also a very powerful way easily use the Refine Edge multiple times if you want to use a different radius in different parts of the image.
3. Click “ok” and then use the non-destructive decontaminate if you have fringing.  Use the eye-dropper tool to sample the color you want at the edges (such as some other part of a person’s hair), and use a brush to paint it onto the edges of the newly created decontaminate layer.  Considering using both the color and luminosity blend modes on this non-destructive layer, depending on the type of fringe color/tone you need to remove.

 

Troubleshooting

Sometimes you just can’t seem to get the result you want with the refine edge tool.  Here are a couple issues to be aware of and potential solutions:

• Sharpening.  If you apply sharpening to your RAW file, this increases edge contrast, and can create edge artifacts that become more pronounced as you start to create a mask around this area.  To deal with this, try a lower sharpening amount and add the desired sharpening later in the process.
• Chromatic aberration:  I recommend turning on chromatic aberration reduction on all RAW files (I have Lightroom setup to apply this to every file I import).  If you don’t address chromatic aberration before you create your mask, you may find it difficult to refine these colored edges.
• Poor initial selections.   Refine Edge can sometimes work magic, but if you give it a very sloppy starting point, you significantly reduce the chances of getting a nice result.  If I’m selecting a an edge that is not contiguous (for example, the sky gaps between tree branches), I like to make the selection with a few steps.  First, I use the magic wand (with contiguous turned off) to select the desired tonal range.  This often selects random areas elsewhere in the image, so I use the lasso masking tool in subtracted mode (hold or choose the “subtract from selection” option in the toolbar) to select the areas I want to remove from the selection.  And then I finally add in other contiguous areas (such as the rest of the sky) by using the magic wand or quick mask tools in additive mode (hold or choose the “add to selection” option in the toolbar).

 

If you enjoyed this article, please be sure to sign up for my free newsletter to get more Photoshop tips & tricks for your images.