As long as the FCC id are the same it should work fine. Its printed on the back of the fob. You will still have to do the programming though. Takes all of 5 minutes. 06 Im pretty sure has Memory 1 and Memory 2 fobs, not sure if it really matters what one you get, as the 04 doesnt support memory seats.
I got a used fob on ebay for $20 before I got my switchblade, worked with memory seat fine, then I had to swap my 06 driver door for what's apparently an 04/05 with no memory seat function, fob still works fine, it just said Welcome Driver 1 in the info center with that fob, and then surprisingly also did the first time I put my switchblade in afterwards.