its time for a boost leak test, check all the couplers and fittings/compression rings from your turbo to your intercooler and from your intercooler to your intake manifold. also check for split or broken vaccum hoses.
If you cant find anything out of the ordinary you can have it fogged, they hook up a machine that makes smoke and they pressurize your intake path, that way they can see where the leak is.
Ive got one but they're out here in SoCal, for a virginia driver i would just run a quick google search for Subaru Tuners / Import Tuners. Boost leak detection is kinda over the heads of most chain automotive places. It shouldnt cost you more than a hundred dollars to have them diagnose and repair the issue so long as the leak isnt coming from something major like an inter-cooler or a compressor.
Some folks make their own boost leak testers by putting a tire valve stem on a PVC cap, attaching it to their intake plumbing and pressurizing the system with a compressor set to low pressure. If your going the DIY route dont go to fast and dont go over your cars max boost pressure (14-18PSI). then its just a matter of using soapy water to find the leak. If you handle it yourself also make sure you pop the oil cap just to make sure you aren't pressurizing any oil systems.
Either way DIY or professional good luck finding that leak, Im crossing my fingers for you that its something silly like a cracked vaccum hose
Sounds like what happened to me a few months back. Everyone kept saying "boost leak test"over and over. As it turned out the C-clip on the wastegate actuator arm was missing and the arm had come completely off. Give it a look-see. It's right under your heat shield for the turbo.
The sensation is as hard to forget as it is to ignore!!!