Enchanter Illusions
 

How do they work? Are they a +X amount of faction and -Y amount of that races 'enemies' or does it just move your factions to the initial factions of the race the turn into? (And religions just trump it all)

faction info here, unless it differs from live~

each faction has a basic relation to each race, class and deity (this is before any earned faction points)

let say that Druids in Sure Fall Glide have following standing:

at 0 to Humans
at +100 to Rangers and Druids
at +100 to followers of Tunare or Karana
at -500 to dark Elves
at 0 to Enchanters
at -500 to followers of Innoruk

the list is actually very extensive covering all possible race/class/deity combos.

So if you are Human Druid of Tunare your starting faction with this group is: 0 + 100 + 100 = 200 (which is Amiable)

If you are Dark Elf Enchanter of Innoruk your starting faction with this group is: -500 + 0 + - 500 = -1000 (Which is Threateningly)

When you cast Illusion Human, you changing your -500 as DE to 0 as a Human, so your total faction changes to 0 + 0 + -500, placing you at Dubious

note that number given are for general understanding only, they are not exact

So with guards that start out as like dubious or something out of illusion and amiably in illusion, and then you kill them down to scowls out of illusion they should be like threateningly in illusion? And that's normal? feels like they should like me in illusion if i've killed them or not...
(This is the case with my gnome enchanter and the trolls guards of grobb)

You have a set value @ a particular faction, and then illusion modifies that. Therefore if an illusion will only give you say, 500 points, and you're starting at max kos, it's quite possible you go from scowls to threateningly.

It used to be a little buggy though, not sure if it's been altered. For example, my Agnostic Erudite Magician killed Ogre guards down to max kos, but with guise (DE illusion) I'd be at least apprehensive and indifferent in Oggok, I could buy/sell and bank.

No, any additional faction you have gained also counts in on top whatever illusion gives you.
You may have been Indifferent in Illusion, but if you gain another -1000 from killing something, you will be KOS even with illusion

If you use an illusion, you are calculated as if you were that race. Nothing more, nothing less.

Here is an example of a faction table:
http://i56.tinypic.com/255q25d.jpg
If you were a High Elf, and you cast Illusion: Gnome, you would use the Gnome portion of the table, instead of High Elf. That's really all there is to it. The total impact of faction is composite of the base, your class, your race, your deity, faction modifiers (alliance spells), and your individual faction level (the tally of all +/- hits you've received).

Illusions are a bit buggy. I grinded like five levels off of Grobb guards and they would still be dubious in troll illusion despite my faction with them being as low as it can get. They wouldn't aggro and I could use the bank (and the vendors stay indifferent due to being on a different faction; not sure if that's working as intended, either) and I would guess the same goes for other cities and illusions of their respective races.

Illusions aren't buggy, your understanding of what's happening is incorrect. It works exactly like I described.