The blind bearing puller worked!
Was still a hassle, because the puller only had 2 posts, and they kept spreading and moving, would not stay straight. Also tried the sliding hammer, which would not work.
Finally got it out.
I guess the rust is the indicator how far to push in the new part