The 4-40 is for the hole at the back of the receiver, holds in the spring and plunger for the take down pin. Makes it much easier to change stocks. The 8-32 is for the hole under the pistol grip, secures the spring and plunger for the safety. It's not as much of a pain as the take down pin.
We've done the .05 trigger job as well, what I found is there seems to be some inconsistency on the thread pitch of the grip screw, so it's always good to check your screw before you buy the set screw, and tap if needed.
The only reason I would consider tapping the selector detent hole would be if I used the grip screw / set screw method of trigger adjustment. If you use one of these it's not necessary to remove the grip to adjust the trigger.