Archeologist Suzi Gage suggests the answer in an article at The Guardian:
Quite astonishingly, there is physical evidence that Neanderthals more than 100,000 years ago were tanning animal skins – a stone tool from the site of Neumark-Nord in Germany has preserved scraps of organic material stuck to it that were soaked in tannin, the substance in oak bark used to make leather. It was probably part of the tool handle that got wet while the hides were being worked.
DNA from lice indicate that our ancestors wore clothes as long as 170,000 years ago.