Sure. For mp3s when meta data support v2 was introduced it allowed album art for the track to be embedded directly in the file. You can use any id3v2 tag editor with mp3s to do that or simply use iTunes. Embedding album art in the mp3 is no different in principal from adding information about song name, album, artist, etc

Not sure about windows media. Got a feelin' that behaves differently.