Only thing I would suggest for the moment is to put the dremel aside and don't worry about polishing anything. When you are done with the easy jig, the only finish work will be to break the sharp edges left by the router, and deburr the holes left by the drill bits. Nothing in the lower to polish.
I would also hold off on any attempt to polish, grind, stone, or otherwise alter any component in the fire control group. Most of these parts are just surface hardened. When you attempt to smooth the surfaces, you run the very real risk of going though that hard surface and exposing the underlying softer steel to wear. This sort of work is best left for well after you have a functional lower receiver.
The other advice... take your time. The router can do the work, but can also ruin the work in you try to cut too fast. Trying to rush the work will lead to damage. I use a milling machine to do my lowers, but I am just as susceptible to trying to rush the work... and a mill can do exponentially more damage than a router