Well, fucking with the body's internal clock does reduce neurogenesis, but saying smaller brains is a pretty large jump from neurogenesis. :p

And no, 5-10 thousand years is not enough time to adjust, because the 24-hour cycle is hardwired from when life started. There is a direct connection between DNA transcription and the biological cycles, so unless a mutation popped up or some special epigenetics are happening, I don't see any adaptation within that frame of time.