When we were shopping for our Z4 M, the boxter was ofcourse also in our minds, but the one thing that kept us from seriously considering it (we didnt even bother with a test drive) was the way it looks and the image it comes with. Don't get me wrong - I am sure it is a brilliant car

In this market segment all the cars are going to be better than average, but they will all have their individual styles and stigmas.
With a car like this you have to consider what
image you want to reinforce about your self.
In my mind the Boxter is the 'safe' choice. No one will think twice about a boxter: "Nice car" and people will move on. It's motoring without drawing too much attention to your self. The Z4 on the other hand is more of a statement. Some might argue that the Boxter is more of a gentleman's car rather than the yuppie pony-tail car that the Z4 is. You can attribute all the cars in the segment with these kinda attributes, but in the end all of this is superflous because ...
... get the car that transforms you the most into Homer Simpson confronted with a box of donuts

That is how you make the choice - I suspect that is the boxter in your case
And my views of the boxter: Good car, that could be better (no limited slip dif), that looks like it is confused: is it going backwards or forward? The biggest problem with the boxter? The 911 - porsche will never let it be better than the 911.