Thanks for the replies guys. I took the car to a Subaru dealer and the mecanic diagnosed it right away. The sound was from the muffler, and it's nothing to worry about. He said it's a very common problem. If the muffler is replaced, the sound will go away, but may come back later on if a stock muffler is used.
PS I bought the car