Yes - it matters very much on xDrive.
It depends on the wear on the remaining two tyres and the overall rolling circumference.
Ideally, to ensure you don't wear out your transfer case, you need the rolling circumference of all four wheels to be with about 1.5% (or less) of each other.
Many tyres will change by more than 1.5% over the life time of the tyre, due to tread wear. Depends on the tread depth when new etc.
You can calculate the variation using a calculator like this one.
https://www.omnicalculator.com/math/circumference
In most cases, you should be OK just replacing two tyres (same size and spec etc.) , as long as the other two are very similar in rolling circumference. YMMV... literally....