When filling up our 2005 Impreza Outback Sport (2.5 liter N/A engine, 5 speed manual) a couple of weeks ago I noticed the car has to crank a long before it started. I recall having a propblems on our '98 Legacy like that and it was the evap purge valve (mounted on the intake manifold). 

 

I thought this might be the same issue here, but after the next fill-up, the car started right up. My wife drove it about 80 miles today and told me she got another CEL (we just had to replace the upstream O2 sensor a few weeks ago). This time it was a P0457 (EVAP System Leak Detected Fuel Cap Loose/Off). I tightened the fuel cap and reset the ECU. Code gone for now, but when I started it it was hard to start again. Restarted again a few minutes later, and it it started fine.

 

I also noticed when I let off the gas (in or out of gear) the idle sometimes drops to 500 RPM or so, then goes back up to around 750 where it normally is (idle is smooth).

 

With the hard starting and this code, where would be a good place to start looking?

Check the purge valve to see if its stuck open. Just need to find the vacuum hose where it goes in the manifold and try to blow through the solenoid. No air should flow. If it does,replace the solenoid.

You can get an evap code from overfilling the gas tank, but it is not usually this code.  Always stop on the first click.  You may need a new gas cap, but I'd suggest waiting until the CEL comes on again and see if you get the same code again.