Should we think that punishment, because it involves intentionally inflicting harm without consent, is defeasibly impermissible (much like it is defeasibly impermissible, at least in private life, to initiate coercion against other people)? I suspect not, for a very simple reason: not intentionally inflicting harm on others (without their consent) might well not be the relevant sort of basic, defeasible duty that would play this role. Here, then, is another, better candidate: people having a basic moral right not to be gratuitously harmed by others. Why is this candidate better? That is a complicated question, but I’ll say two things. First, for me — and maybe for you — it is more intuitive (brutely, but also because it covers more ground, more cases). Second, it better comports with the following principle (based loosely on what Thomas Scanlon says in the first two or three chapters of his MORAL DIMENSIONS): the essential basis for the moral-permissibility-status of actions does not include the intentions or motivations of the agent (though in special cases such features are included).
If my suggestion here is right, then the Boonin-style sweeping argument against punishment does not work. One would have to argue something like this: the harms of punishment are — or perhaps usually are and should be presumed to be — gratuitous. That is a tall order. My conclusion is that there is no “problem of punishment” (or, if there is, it is really not much of a problem, at least not until someone demonstrates either a strong-enough thesis about punishment and gratuitous harm or that the fundamental duty here is definitely one of not intentionally harming others without their consent).