The gearing is shorter in first of not second as well. Basically, the first batch of STi's were meant to be driven off the lot and make it to the rally course the same day. They were shipped without rear seats, apparently they were installed after they got unloaded because they knew most people would want a rear seat (which is why the fronts and rears are different materials.) They were also shipped without a stereo. The wheels lug pattern is different, which I say is a good thing because the 04 STi is the only one with a WRX bolt pattern so your BBS wheels are more valuable. The suspension for the 04 is also different than the other years. The 05+ have a much more aesthetically pleasing center console in most people's opinion. In 05 they started in with more creature comforts. But you still have all the other year's STi goodies, IC sprayer, adjustable HID's, Driver Controled Center Differential, beefy six speed.
I don't really think there is any reason to steer away from an 04 though, I owned 2, both great cars.
355 awhp fp green six speed wagon in Corvette Atomic Orange