Hanging on to that nice finish?

[WNC Goater]

I've got some nice ScotchBrite finishes on blades that are fine. I actually use a SB wheel on a buffing motor. So the finish runs perpendicular to the blade, as opposed to longitudinally. I suppose a ScothBrite belt would do the same.
But I typically hand sand after about a 400-600 grit belt to get an even finer finish before the SB finish. I back my emery paper with a piece of flat steel, working it longitudinally. Taking this up to about 2000 grit makes a nice finish.

Here's the problem.

After attaching the handle scales, wiping epoxy, peening pins, sanding and shaping the handle, applying finish, wiping, sanding, polishing and the hundred assorted steps, I always end up with some very light scratch or other "cootie" around the ricasso area, in spite of covering the blade with blue painters tape. I've found it virtually impossible to remove these, trying to work longitudinally right up to the forward edge of the handle as you get the little swirls and "j" patterns when reversing the emery paper. Even starting against the handle and drawing in one direction leaves very light swirls and various obvious blemishes. And so, I end up with that ScotchBrite buffing wheel, because it CAN be run right up against the front of the handle scale. But the end result is reducing my nice hand finish to a light satin finish. Again, nothing wrong with that but I question how to maintain that nice hand sanded finish.

So, any tips? Am I too obsessive compulsive? Is it simply a need to be extra careful? Am I missing a "trick of the trade"?