Does faction matter for me?

[Zonais]

I'm an agnostic enchanter. I can basically move around the world as I see fit with illusions and faction changing spells etc.

My question is as I go around leveling, should I avoid any camps that may harm certain factions I can't raise back up easily or trick the people into loving me anyways? Like any guard camps in certain cities or anything like this?