Yes, you are measuring voltage drop instead of voltage potential. Connect the negative test lead to ground for measurements. Sounds like your problem is not at the relay though, likely the fuel pump or wiring to it which you need to check next. Try to find some videos on using a multimeter.

__________________
1988 325is (purchased new) sold
2004 X5 3.0 2005 X3 2.5
2008 X5 3.0 (new to me)