I don't really meet the criteria that you set forth but I'll give my ¢.02 as well...
The E70 X5 is amazing. If you enjoy spirited driving the 4.8 is extra amazing. But as other members have expressed and as you have found out, the maintenance on these vehicles is a little over the top. It's a trade off, you have an amazing car but you will definitely be paying for it. Whether or not it is worth it is entirely up to you. For me, hell yeah it is! But in the amount of time I have owned mine, I have had it in my garage or at my Indie more times than I care to think about.... I think one of the key things with E70 ownership is being a DIY-er. most of the repairs aren't too difficult. If you can at least do some of the major repairs yourself then the affordability of the vehicle is much better.
As far as "pitfalls I wish I knew about before" ALL of my woes have been engine related. The 4.8L is a "special" motor and loves to leak all over the place. The rest of the vehicle, Chassis, drive train, etc... I cant complain about.
If the 4.8L is what you are looking for then this thread may shed some more light on things to look out for when doing an inspection.>>
https://xoutpost.com/bmw-sav-forums/...icks-info.html
If the 3.0 is what you are looking for, they are much less problematic and easier to work on. The diesel's are also very popular.