[Jimjam]

Enchanters need follow no gods - they are gods!

There is your agnostic justification.

Otherwise I agree with Tadkins - the spice the religions add to the character is worth the drawbacks.