Dear Adobe,
I’ve decided to start a new category on this blog called “Dear Adobe” for the purpose of writing open letters to you. I don’t really expect you to listen, I just want to put some ideas and open-ended questions out there for people to consider. This will never be meant to bash or criticize, just some friendly suggestions from someone who’s used your software for a while.
My first letter concerns one of my favorite features in Photoshop: the layer mask. I chose this feature mainly because it’s central to my process, but also because I feel it could be so much more powerful with only a little tweaking. So here goes, my suggestion for additional layer mask functionality in Photoshop CS5 (or 6, or 7, or just someday).
I propose a feature called “masking groups”. Why do layer masks have to be flat? Why can’t a layer mask be a composite of multiple layers in a group or even the result of a smart object which contains many layers? For instance, a masking group could contain many layers which are freely editable and independent of one another. The layer to be masked would see this group as a single, flattened mask and change it’s opacity according to the sum total of all layers in this masking group. At any time I would be able to go back into the masking group and move elements around, scale them, delete them, etc.
My workaround for this has been to nest the image to be masked within successive groups, each with their own mask attached. But due to the group depth limitation in Photoshop, this system only works for up to 5 layers. Also, it does not truly behave in the same way as the “masking group” I envision. With the current workaround, each successive group and mask can only be subtractive (black). If you try to put an additive layer mask (white) over the same space as a subtractive mask on a group nested anywhere below it, it will not reveal the underlying image. This is because the groups and their respective masks are not seen as a single, flattened image by the underlying layer to be masked. In the proposed masking groups, the sum total of all masks would be treated as one. The best analogy for this would be the Smart Object. While a smart object may have many layers within it, only the layers that are actually showing will be visible in the document in which they are placed.
I’ve included a mock-up of how these masking groups might work. Masking groups are just like any other group except that they are linked as a mask to another layer or normal group, sort of like a layered version of the clipping group. In fact, this concept could be accomplished simply by allowing multiple layers to be used as the base for a clipping group. Notice in the example below how the Square, Triangle, and Circle are all separate layers in the masking group, but that their flattened state is reflected in the composite thumbnail image at the top of the group. This is the true layer mask that will effect the “layer to be masked” below it. This composite is constantly updated as layers are edited within the masking group. Also note how the white (additive) square sits above the black (subtractive) triangle, effectively masking part of the triangle and subsequently revealing part of the underlying “layer to be masked”. This is the key to the whole concept, and it’s something that is not possible using the workaround I described earlier. A masking group could be applied to a single layer, or to another regular group containing many layers to be masked at once.
An example of how a masking group might work (click to enlarge)
I hope this all makes sense. I think a feature like this could unlock all sorts of new possibilities. If this is, in fact, already possible, please let me know how to accomplish it. When I worked for you I walked over to XD one day and asked a guy about this and he said it wasn’t possible, so for now, I’m going with that assumption.
Sincerely,
-Scott
P.S. The shower head upstairs has been leaking, could I get some help with that too?
If anyone else has any suggestions concerning masking functionality in Photoshop or just want to second the notion, sound off in the comments.
I think the easiest way would be to allow Smart Objects as masking objects. But then you dont have the direct feedback of the result, since its nested in another document window.
They also could extend the Channels with Grouping for alpha channels, allowing you to mask a channel group.
I usually work with vector masks as much as possible since you can combine multible shapes into a single path layer and still have it easily selectable and editable.
Smart objects as layer masks would be a great addition indeed when vector shapes just don’t cut it …
I’ve had to do something like this once and ended up finding a way-of-sorts to have multi-layer masks but it’s pretty convoluted, so it’s only for emergencies I guess. Maybe there’s some way to simplify it though?
1) Put all mask layers and all artwork layers in two separate smart objects and create a Clipping Mask between them, using the mask as a base.
2) Open up the mask smart object and create yet another smart object of all the layers inside.
3) Open the layer styles dialog for this newly created, third smart object, and head to the “This Layer” slider in the “Blend If” section of the Blending Options. Hold Alt/Opt and drag the left part of the white slider on the right all the way to the left.
This should make light parts in the mask transparent while keeping the dark ones opaque. And I hope all that made sense 😀
Let’s hope Adobe is listening. I’ve ran in to this problem a few times before
Haleluia amen I second that.
Oh yes indeed. A limitation that causes too much manual labour. And it would be a natural progression to the layer mask feature if it is ever implemented.
Yep I think this would be a great addition to the software.
I work a lot in Adobe After Effects where I am so used to pre-composing a bunch of layers in order to use it as a layer mask. Then I can freely travel back into that pre-composition and change stuff as i need, whether it is vector, pixels or a mixture of both, and my changes are reflected in the layer mask.
Then when I use Photoshop I feel a little hampered by the constraints you mention.
Any other after effects users ever used after effects for a still image composite? I have, but it never feels right! and of course my PS knowledge is lesser than my AE knowledge.
I think you can do this already with smart objects and clipping masks…
http://stephenknowlton.com/clippyclip.png
is this not quite it?
Have you tried using the knockout parameter in the layer property panel ? If you set it to shallow, than setting the fill parameter to 0% it will mask every layers beneath it. To prevent this, you group the mask layers and the layer to be masked. That way you can have multiple layers masking. Here’s a little example : http://www.glockenspiel.ca/photoshop_knockout_shallow.png and http://www.glockenspiel.ca/photoshop_knockout_shallow_2.png
Masking in Photoshop has rashed my nutsack. It’s lack of nesting is far behind all the other core tools. Being able to mask groups would be great.
Masks in Photoshop should always have its own layer I think, lemme explain. Layer masks kinda irrate me in how they opperate, the mask not being represented as it’s own layer. It should work like clipping masks. Where you have one layer and you can set any amount layers to be masked by it. And it shows its nesting heirarchy nicely, just like how a clipping mask does. The only problem with clipping masks is you can’t have more than one. You have to use to vector masks and layer masks thereafter for a the clipping masks children / contents. You can’t nest clipping masks them.
Why not just make a simple masking heirarchy. That’s what I want, every mask item on it’s own layer, that obeys it’s heirarchy. And you can nest them for a few levels. That I think, would be an easy system that amalgamates all the masking functionalities. Making them easier to understand because they behave the same, regardless of whether you’re using a vector shape, raster brushwork, etc… Basically an improved clipping mask is what I am asking for.
I’d think masking groups would be better fit under the Channels palette, as that’s where the actual raw masks are stored.
Dude, I’ve craved this same functionality for a while too. You’ve explained it well, and provided strong rationale. I’d hope that this is reason enough for them to at least explore the idea (if they haven’t already). I’m going to start using a “#dearadobe” hashtag on twitter. Let’s see if we can spread this article a little further.
As usual, great work. Cheers!
In a way this is sort of possible? If you use a lower layer as a clipping mask (that option click line thingy), you can use that as your additive mask function, which you can view by toggling the visibility off of the layer you’re clipping. both of these layers can be masked as well, so essentially you have, 3 layer masks affecting one layer, without even nesting yet. It’s not infinitely but you add the group nesting you speak of, you have about 8 separate editable layers of masking.
One suggestion I would have for Adobe would be to allow groups to be “option-clicked” onto a clipping mask – which again would add 2 more levels of masking (the actual layer and it’s layer mask) or better yet allow a group to be the clipping mask – and everything in the group folder would be evaluated on its grayscale values to be a composite mask, additive and subtractive, complete with blending modes.
While they’re at it, they should include the option to “Apply layer mask to group” – and each item within the group gets it applied.
I was thinking of a similar concept for when I get my blog back up. I have a lot of ideas of things to change/unify between the programs.
I definitely agree with this one. I also wish Illustrator’s layer masks were easier to use with complex shapes, instead of requiring it to basically be one path/compound path.
Honestly, if Photoshop were more like After Effects, it’d be a lot easier and quicker. AE effects work much better, faster, and always function like Photoshop’s do on smart objects. Sometimes I work in AE and bring it into Photoshop!
@John I didn’t see your comment before posting. I agree wholeheartedly. AE is better to work in, but it doesn’t feel natural because that’s not what it’s designed for per se.
Adobe needs to unify a lot of the features. For instance, why do sliders exist in Photoshop’s toolbar to change text size, while it isn’t present in Illustrator? Why can Photoshop preview fonts using the arrow keys and Illustrator can’t? Why does kerning act slightly differently between the two (or three if you count AE, which does it best).
Yeah, something like AE would be nice where you could have multiple vector and bitmap masks and set them to add, subtract, intersect, etc.
One thing that bothers me (maybe I’m dumb and am doing something wrong) but if I want to blur a layer without blurring the mask I have to unlink and then relink the mask. It’s kind of a pain. I don’t think I’ve ever wanted to blur both at the same time so I don’t understand how this could be a useful feature.
Illustrator Layer Masks do allow layering.
Again the point that @Joe Clay is making is that they cannot be complex shapes but meshes & gradients can be applied to shapes within the layer mask.
i often use a soft brush with Layer Masks in Photoshop. Obviously this can’t be done in Illustrator. Although I did discover that raster images can be placed in the the Illustrator Layer Mask which opens up a world of possibilities for me.
Oh, god yes – I’ve wanted this functionality for a long time. I’m actually hoping that one day Photoshop will have the option to use a node-based system a-la Nuke or Shake, or at least take a few cues from After Effects – it would allow for so much more flexibility…
I am with you on this one. The glory of photoshop is its layering functionality. Why this doesn’t extend to layer masks is beyond me. It would definitely open a whole new world of functionality and creativity within the program!
I think it would be great if there was an option for layers to keep their names being dragged from one document to another instead of renaming it “LAYER 27”
Or if files themselves kept their names as they are dragged into a PSD.
Right?
A node based system like Nuke , would be cool!
i love this new column!
something i have wanted from adobe and can’t figure out why it never works is the distribution tool in photoshop. it seems to work fine in illustrator but whenever i try and distribute in PS it never evenly distributes. perhaps i’m doing something wrong but no one in my office can figure it out. we have all resigned to the idea that the tool doesnt work right. and i’d like to think we are all competent designers. can someone either answer this or can adobe fix this please?!
Totally agree there. This is a much needed feature.
To tell you the truth I’m surprised this feature hasn’t already been added by adobe. I thought i would have come with CS4.
Great simple idea…I’ve tried accomplishing this method of designing for so long only to be forced to search for an inefficient way around it. I’d be surprised if they weren’t already considering this feature.
Great idea!
@Steve – I see what you mean about smart objects and clipping masks. This is basically doing what Scott asks for in his original post. But there is a problem for me.. I have to enter the smart object and make my changes, and I can’t see the effect ‘live’ on my masked out layer. Unless I am doing something wrong? On a normal layer mask I like to paint my mask while looking at the result.
@jeff clermont – Hey Jeff, I just had a play with the knockout groups. They are pretty cool, and do help. I still find it a bit limiting though because I can only seem to have the bottom, or ‘background’ layer being revealed. What if I want more layers underneath? I’ve only had a 5 min play with this so maybe I don’t get it.
Thanks for the insights though, they will defo help me out.
Oh and for all those compositors out there that are saying they would like a node based workflow – and I know this is off topic, but have you seen a plugin called filter forge for PS? It’s for making your own PS filters. I got the demo the other week, it’s pure node goodness like nuke or eyeon fusion.. mmm
@jeff clermont – Just figured it, I miss-read what you said about putting it all into a group. Also, setting the group’s layer mode to ‘pass through’ made it work.
@John – Yeah, if you want to have other layers underneath unaffected by the knockout layers, put them outside of the group. The only thing is you can’t have subtractive or additive mask.
Here’s a link to the feature-request page:
http://www.adobe.com/go/wish
How about using clipping masks on a group? I can’t count the number of times I’ve wanted to apply an adjustment layer to only one group, only to be frustratingly denied.
@Nate G – I’m pretty sure you could achieve the effect you speak of by selecting the outline of the group layer mask, then selecting the layers in the layers palette and hit the layer mask button.
@Frank – Blurring a layer without blurring the mask, in my experience, leads to a very unnatural looking layer. For example, if you’re trying to use a motion blur, the contents would be blurred, but then be defined by a hard edge, which is visually confusing. I’m curious, in what context would you not want to blur both?
Smart thought and smart responses. Thanks for sharing.
this is needed. I think I’m going to enjoy this new category, and I hope Adobe starts listening/commenting. Keep ’em coming Scott.
I second this idea. And i hope it’s gonna work more like in your concept then like Smart Object.
I’m also quite happy you pointed the nested group limitations. There isn’t a single project where i don’t encounter this limitation. It’s the first thing i check in all new PS releases.
Just use After effects!
…yes for stills images
oh well: have Adobe incorporate “Pre composition”.
For who don’t know PRE COMP are like gruping but more powerfoul and much editable.
What about node based compositing? Which applies to everything, not just masks.
Surely this a much more flexible approach to the whole process. The presentation and methods of interaction could still be managed as per some previous suggestions, but really a powerful node based compositing work flow is is sorely missing in a modern graphics application.
I’d much rather drag layers around in Photoshop than unlink and relink in/outputs on nodes. It just doesn’t feel right for Photoshop.
I just want to beable to have longer film strips that i can edit. i do most of my masking in AE not PS since i am mostly a motion designer that comes from a print background. very interesting thought scott. i think the smart object idea someone said here might be a simple solution in the future.
This would be great! I like the “dear” adobe idea as well. If you tweet new posts directly to adobe (@adobe) their pr department will pick it up.
There’s a little hack/workaround for what you’ve described that involves using a smart object as a clipping mask. Others have already mentioned it. It does everything except capture the exclusion ability you’re looking for – but there’s a somewhat messy way to do that which involves making smart objects out of your image/smart object mask pairs, and clipping them with other smart objects.
It only actually gets complicated because you can’t expand the tree all at once like you can in illustrator. You can do this easily with Illustrator, but of course the masks are hard lined, and if you use a gradient mask in illustrator you’re right back to the hard to use “why is this an exception to the model I like?” thing.
Thanks for starting the conversation.
I used to do a lot of work in Shake, and that really showed me how powerful masking (or rather layering) can be. It gives you complete (and non-destructive) freedom to add, subtract or otherwise combine as many masks as you please, and in an intuitive way.
I would *love* a node based photoshop. Not gonna happen though…
Haha, leaky faucet. Maybe in CS5.
Oh, did you try to put a mask over it and fill it white? That might stop the leak!
The one simple addition I would love to see in Photoshop is the ability to add bleed when setting up a document. It’s available on Illustrator and InDesign, and almost every project I work on has me adding 6mm, working out 3mm from each edge for guides, then a further 10mm for spacing….. life would be so much easier with this simple bit stck on.
Yes, a quite good variant