Peltier’s are very fussy when it comes to their power supply and control thereof.
If you buy a Peltier device make sure it comes with its own controller/power supply.
Many laptops used to have Peltier’s in them to cool the processor; they basically had the cold end mechanically attached to the processor while the hot end was usually connected to a heat sink of some description. Some of them also had a fan cooling the heatsink.
Many of the older IBM and Toshiba models had this type of setup but have since abandoned this technology for more power efficient designs.

While USB is currently rated up to 5 amps, this is using the larger style connectors, not the smaller variants.
Assuming your phone has a micro USB or similar connector, the max power this connector is rated at is 10W @ 5V. Even this is a big ask for such a small connector and if you pass much more through it you’re likely to encounter issues with the connector itself.