Low/high speed grinding sound

[Aidzer0]

Hello fellow enthusiast, please bare with me, I will be including lots of details for the issue I am experiencing;

About a week ago, I noticed a fairly large puddle of oil, at the time, I thought it was engine oil, it did not have the fuel smell of used engine oil, but I had done an oil change recently so that made sense, anyway, I checked the engine oil and it was at the mark I left at when I changed the oil, weird, okay.

So fast forward to 3 days ago, my wife is on her way to work when she calls in a panic that there is this loud grinding noise, I drive there, investigate the car and notice that the front driver side axle is about 1/2 inch out of the diff... so, in the parking lot I lift the car and tried to snap it back in, that didn't work, I then pulled fuse #46 and went for a slow drive, noise was gone. I drove it back home, removed the axle, apart from diff fluid being everywhere, I did not see any issues with the axle, circlip was good, splines were good, nothing wrong with it, even the CV joints were good. I put the axle back, seated it back into the diff (really complicated when you're working in -32 degree weather), refilled the diff with 75w90 oil. Put fuse #46 back and the sound was still there. Remove fuse 46 sound is gone again.

Details about the sound, it is really loud when driving low speeds, I can feel the steering wheel shake at about 2mph. It seems to be coming from the front/mid of the car. I remove fuse 46, sound is gone, this tells me its not trans, brakes or wheel bearings, its either transfer case, front drive shaft, diff or axles(?) It sounds like straight cut gears in a race transmission mixed with metallic grinding. I have not driven the car higher than about 5-10 mph with the grinding sound because I don't want to cause more damage. When my wife was driving she was on the highway at 60mph and I could hear the sound over the phone. It is loudest at lower speeds and steering does not influence it.

Details about the car; 2006 X5 4.4l canada spec car, got the car a couple of months ago, no lights on the dash (anymore), changed a lot of engine related things, VANOS solenoid seals, oil, filter, fuel pump and filter, spark plugs, expansion tank, water pump, water transfer pipe to water pump the short rubber/metal one (this actually caused a coolant leak which I need to investigate and fix too, it's not the usual n62 transfer pipe as that was fixed by previous owners mech), thermostat, all the coolant temp sensors (they were aftermarket, temp gauge was inaccurate and causing no start issues, I used VNO).

There are no codes to speak of until I remove fuse 46, then the dash lights up with traction control, ABS warnings.
From my understanding, I have been reading tons about this, the facelift model does not suffer the same front driveshaft issue as the post-face lift model, it could still be the case? How would I troubleshoot this?
I checked both front axles, they seem to be fine, wheel bearings would not be the issue otherwise the sound would remain after pull fuse 46, it's not brakes for the same reason.
So (to me) the only culprits would be the t-case, front diff, front drive shaft and maybe the axles, maybe axles make the sound only under load(?)

How would I go about troubleshooting this quickly/easily, it's bloody cold here at the moment and I need to fix this outside in the elements.

Side note, I have no cooking clue how the front driver axle popped out the diff... I did notice that when I removed the 12 point 36mm nut, the splines were free of the wheel hub and I did not need a puller or anything to remove the axle hub side... So someone was in there recently...

Any advice is welcome, I am fairly proficient with cars, I rebuilt an e46 ground up.