How to Mask a Subject in Photoshop Using the Subject Select Tool

Photoshop’s Subject Select tool lets you create detailed masks, and we’re going to show you how.

The ability to create masks in Photoshop is one of the core features that allow for layer-based photo editing.

In this tutorial, we’re going to show you how to use the Select Subject tool to create a mask in Photoshop. We’ll also show you a trick to create a more complicated mask. Let’s get started!

Tóm Tắt

Where to Find the Select Subject Tool

To access the Select Subject tool in Photoshop, go to Select > Subject at the top of the menu bar.

You can also access it if you’re already using the Object Selection, Quick Selection, or Magic Wand tools. With these tools active, you will see Subject Select and Select and Mask as options at the top.

Getting Familiar With the Layout

Before we dive into the fun part, let’s have a look at the tools, modes, features, and settings that will come in handy when you’re creating masks.

When you enter masking mode after choosing Select and Mask, there are three menus to work with. The one on the left contains the selection and navigation tools.

They include the Quick Selection tool, Refine Edge Tool Brush, Brush tool, Object Selection tool, Lasso tool, Hand tool, and Zoom tool. Each of these has a specific application and works in tandem with the other menus.

The Properties Panel

The properties panel is where the localized and global adjustments are located. Most of these features are located on the left under Properties. There are two options on the top bar as well, namely Select Subject and Refine Hair.

Related: How to Select All of the Same Color in Photoshop

View Mode

In View Mode, you determine the best view for creating your mask. There are preview options such as Show Edge, Show Original, Real-Time Refinement, and High Quality Preview. The color and opacity can also be adjusted here.

Refine Mode

In Refine Mode, there are two options: Color Aware and Object Aware. Either one can be used to refine your mask.

Edge Detection

Edge Detection tells Photoshop how to process the edges around the masked subject. For example, you can extend the edge by turning the slider to 250 pixels. Checking the Smart Radius box tells Photoshop to shape the edge according to the precise contours of the subject.

Global Refinements may be needed for some masks and not at all for others. The four sliders include Smooth, Feather, Contrast, and Shift Edge. There are two additional options to Clear Selection or to Invert the mask.

Output Settings

The Output Settings are important because it determines how the new mask is applied to your image. Your options are Layer Mask, New Layer, New Layer along with Layer Mask, New Document, and New Document along with Layer Mask. There is also a box option to Decontaminate Colors.

How to Create a Mask With Select Subject

For this example, we’re going to mask a tennis ball. You can download the image from Pexels if you’d like to follow along.

Masking a round tennis ball may seem like an easy task at first glance. But most tennis balls aren’t perfectly round. Plus, they have bits of protruding fuzz. This makes it near-impossible to create a realistic-looking mask in just one click.

Let’s look at how the Select Subject tool will help us mask this tennis ball:

With the image loaded in Photoshop, press Ctrl + J twice to create two layer copies.
Select Layer 1 in the middle of the layer stack. Go to Select > Subject.
A selection will be made around the tennis ball. Click on Select and Mask.
In the Properties panel, choose the red overlay from the View Mode.
Go to Output To at the bottom and select Layer Mask from the dropdown menu. Click OK.
Select Layer 1 Copy at the top of the layer stack. Go to Select > Subject.
With the selection of the tennis ball made, click on Select and Mask at the top.
Enter the following values in the Properties menu on the right: Radius – 1 and check the Smart Redius box, Smooth – 19, Feather – 24.4, Contrast – 8, Shift Edge – +2, and check Output To Layer Mask. Press OK.

We now have two layer masks sitting on top of the original image. The Layer 1 Copy represents Photoshop’s automatic selection using the Subject Select feature. The top layer represents the custom values we’ve entered to create the best possible selection of the tennis ball.

Related: How to Fix Bad Lighting in Your Portraits Using Photoshop

Inspect the Layer Masks

In order to inspect each layer mask, uncheck any layers underneath the one you’re looking at. We’re going to examine the Layer 1 Copy. Uncheck both layers underneath to see the mask.

Layer 1 Copy Mask:

Notice how we’ve created a soft selection here, where a little bit of the fuzz appears. It’s not what the tennis ball looks like in the original image, but we’ll fix that later.

Next, let’s inspect Layer 1 by unchecking the Layer 1 Copy above and checking Layer 1 as shown below.

Layer 1 Mask:

This mask was selected by Photoshop. Notice the edges around the tennis ball are dark, solid, and jagged? This copy is also far from perfect. But we’ll show you how to combine these masks to reap the best of both for the final mask.

Related: How to Use Layers and Masks in Photoshop: A Beginner’s Guide

Refine the Layer Masks

Now, we’re going to take the best of both layer masks to create the perfect mask for the tennis ball. We’re also going to create a solid black adjustment layer to replicate the black background of the original image, adjust the opacity, and perform a simple trick to perfect the final result.

Follow these steps:

Create a Solid Color adjustment by clicking on the adjustments icon at the bottom right of the screen.
Change the color to black by dragging the circle down to the bottom-right corner of the box as shown below. All the values should be at 0. Then click OK.
Select and drag the Color Fill 1 layer down below Layer 1.
Change the Layer 1 Opacity to 50.
Now for the trick; activate the Layer 1 Copy and press Ctrl + J three times.

Notice how some of the fuzz re-appears? We now have a more natural-looking tennis ball. This is because duplicating a layer will double the effect and create a more accurate mask.

Masking Made Easy With Subject Select Tool

Photoshop is always improving its selection and masking tools. It’s now easier and quicker than ever to create more refined masks.

Give this tutorial a shot; you’ll be amazed at the detailed results you can get with the Subject Select tool.