Is there a way to smooth a mask

the same as with facegroups ( relax option)?

Do you mean to blur ?

You cannot smooth mask locally, but you can blur it globally.

I want the “crisp” edge you get with facegroups.

Blur helps a bit I guess…