I got an '84 GL, and I had one long ago too, but in both, I have noticed, while driving, and shifting down, after letting the clutch back out, some back and forth movement of the shifter lever, sometimes seems extreme to me.  Now IDK much at all about how these engines and trannies are mounted, having never looked into it or dealt with it personally, but on my last little subie, it developed an accompanying "clunk" sound. 

 

I do most of my own work, but at the time I had a guy would do spoob for me cheap, and he had just done the clutch on that rig when the clunk started, so turned out there was a loose mounting bolt, prolly one he forgot to tighten, but anyway, that solved the clunking, but as I recall (if I truly do, as the last subie was years ago), but not the back and forth jerking movement of the shifter lever whenever the clutch went back in and maybe it was going a bit faster than is ideal for a downshift, or even under hard acceleration, say if I am trying to zip out into traffic cuz I don't wanna slow anyone down, but I wanna get into a busy roadway without waiting a week.

 

Anyway, I suspect there is some flexibility in the tranny mountings, maybe rubber bushings or something that explains this, because far as I know it's not an actual "problem" it's just that 84, Subaru GL wagons, are the only cars I've really noticed this thing happening, or at least happening as severely as it is on my subies.  I also figure, Subaru being a completely unique drivertrain setup, it's not so surprising if it is normal.  Just hope I ain't gotta crawl under, cuz I hate any work that involves that.

Needs tranny mounts. The EA81 mounts suck and are small. They are likely ripped in half.

play in cv joints andsteering rack ends/tie rods cn also give similar symptoms