So I let the car sit for a week, and came back to tackle it again. I lifted the valve cover up and tried forcing it sideways, and all 4 of the the spark plug tubes popped out all at once! I think sitting in my cold garage for a week caused everything to shrink up slightly and release the bonds that had been created. Problem solved! At least until I get to the second valve cover...