LM317-based adj power supply output takes a while to stablize

I recently built-up a quick little LM317-based adjustable power supply schematic:

However I've noticed the output can take several seconds (on the order of 10 or so) to "settle down" after turning the trimmer to desired output voltage. What happens is if I set it to 5.000V (as an example), my DMM will show the output slowly tick down in 1mV increments or so until the voltage finally settles; sometimes as much as 100mV lower than what I initially set.

Doesn't seem to matter if I turn the trimmer fast or slow. Once I let go, my DMM will show the voltage slowly "tick down" until it finally stabilizes. The same issue occurs with or without load. To provide stable load for testing (100mA or so) I'm using a constant current dummy load similar to the EEVblog one. Also, originally I didn't have the 10uF adj cap, but adding it doesn't seem to make a difference.

Any idea why this is happening or is this just normal? Seems like I have enough smoothing capacitance so I don't think it's a ripple issue. Could it be too much capacitance on the output? I've seen other people demo their LM317 circuits on youtube and their DMM doesn't seem to "tick down" after setting the output voltage like mine does.

Maybe my DMMs are just too accurate, heh. =)

EDIT: I tried replacing the 'dummy load' with a simple 1k resistor (also tried a 100-ohm). Didn't really seem to make a difference. Also, seems like after I set the voltage my meter 'ticks down' pretty fast at first then slower and slower. "Feels" almost logarithmic:

I just remembered my agilent meter does data logging (duh). Here's graph of ~90 data points showing the voltage drop (after setting to around 5.25V or so) over ~90 seconds:

Here's another over 5 minutes (I was trying to dial-in 15V):

UPDATE: Replaced the 5K pot (which was a crappy single-turn rated at 50mW) with a 5K 10-turn precision pot (rated at 2W). Now I can crank it up to, say 5V or 12V or whatever and my DMM shows a stable reading right away.

Answer

I think that you are dissipating too much power into the potentiometer. My calculation works out to 161mW when the total resistance is used. If it is a low power 200mW version, it might be too much. Swap R5 for a 1K or 910 ohm and do the experiment with the 5V again and see how it varies.