I think I know exactly what that is, because I've been experiencing that for a couple of years now. Obviously it's a condition that's not going to blow anything up.
The tranny shaft that your clutch release bearing (throw-out bearing) sits on gets worn, and what your hearing on deceleration, is the bearing vibrating on that shaft. When your on power and the clutch is loaded, it doesn't do it.
Subaru or some aftermarket parts place make a sleeve-ing kit to remedy the problem, but its more expensive than the noise is annoying so I haven't done it. Maybe when I do the clutch next time. I get the noise in 3rd and 4th.
The rest of your post regarding the shifting is a little confusing, but yeah cruising in 4th at 40mph, there's not much power. Downshift..... simple
Hope that helps