When you check your engine oil, use a bright flashlight and twist the dipstick so the light hits it at different angles, this should allow you to see tiny metal specks in the oil, regardless if it was the engine or turbo causing this. You can also take a magnet and run it near the dipstick to pick up any metal that may be in the oil, just don't get too close or you may attract the dipstick first and need to start over. One last test would be to get an oil sample to BlackStone Labs, their test kit is free to ship to you, and once you fill the container and send it out you should get a response within a week or so. The cost for the actual sample testing is less than $30 so it could save you $$$ before you buy more parts, only downside is that you might not want to keep driving the car in its current condition as depending on what is actually wrong..you could be causing more damage as you go. I did their sample testing after my turbo went bad to get a baseline of the oil before I changed it out to know what the contaminant levels shouldn't be and to see what changes after I run it again with new oil and a replacement turbo.
Also if you pull the turbo off again (I know I know) you can grab the shaft from both ends and push/pull in/out and towards the housings, you should have just the slightest jiggle but barely any due to not having oil pressure..but if it moves in/out, towards the housings more than 1 or 2mm the turbo may be shot (test your old one like this too).
Also can you post more info on your 04 wrx? Things like mileage, mods, services done, etc can help tremendously with diagnosing issues. Ex, a high mileage oil pump/pickup from 3 prior abusive owners could have seen blockage at least once due to improper maintenance and using dirty oil for extended periods of time, causing the bearings to not receive efficient flow and slowly overtime spinning the bearing. <- This is a more extreme example but still very possible.