Should I break up with him...?

I think you should tell him that you're considering breaking up with him and see what he says, if he doesn't care, end it.. if he apologizes and gives you a good explanation for what's been going on, give him another chance.