If the 2004 is prior to the facelift, the headlights will fit. If the 2004 was built after the facelift the headlights will not fit. You can tell the difference in the headlights per the following thread:
E53 X5 Face Lift? - PistonHeads
In 2002, while the same configuration, depending on when your 2002 was produced, it could have headlights that are fairly easy or very difficult to open. That only matters if you would like to upgrade your headlights with angel eyes etc. in the future.
I don't know what the cost of the 2004 headlights will be but you might want to take a look at Depo replacement headlight instead of investing in the 2004s.
If the problem is your headlight lens are not clear there are restoration kits that can take care of that issue.