[Incognito]

Whats the rule if I'm at Hierophant just camping him and a group wants to take the crypt?
Do I have to give up the camp since they are a full group or do they get everything but Hierophant?