When you lose your hat at either Tall Mountain or Snowman's Land, get a star. After that, the toad next to the door will have your hat.