I concur with Eric.
A drill press is made to plunge. It is not designed for lateral loads on the spindle. If you are determined to use it, make very light cuts.
The temptation is there to rationalize it by thinking you are only going to do one or two receivers and the drill press will do a job that is "good enough". If the cost of the jig and router is an issue, you could deal with that by:
Go in together with some friends to spread the cost. One of them might even already own a suitable router, eliminating that cost.
Places that rent tools may have a suitable router that could be rented.
Borrow it if you know someone that owns one.
Search for a used jig. Now that 80% is going away, or is at least on hold pending legal wrangling, peeps that have finished their lowers will be selling them.
And then there is the finished results.
This is what you can expect when using a drill press using drill bits. While it may be fully functional, and you can't see it when the rifle is assembled, my OCD would not permit me to make something to this level of quality.
You said you were using an end mill, so naturally yours won't look like this.
This is the results you get from a 5D jig and router. If the trigger pocket was anodized, it could pass for factory. I blacken the white metal with Birchwood Casey Aluminum Black, so unless you look close, you can't tell.