Is there a word for when the actor acts in a play as part of the play?

I don't think it has a name. "Recursion" is a word that springs to mind. An actor playing an actor playing an actor playing ... is recursive.