I find some m4's are very finicky about how they are assembled/disassembled. Try removing other parts first like the pistol grip/motor, trigger pin, mag catch, stock/buffer tube screw and the body pins might come out easier. You need to remove those anyways to get at the mechbox.
I need to use a punch to get out a buddy's rear pin and on some guns they'll fall out! Luck of the draw sometimes on the fitting of things together.
If it ain't broke and shooting fine don't tinker is my recomendation though!
