That’s the name of an exemption from the Freedom of Information Act: government agencies don’t have to reveal records if they are part of “the deliberative process”. Congressional subpoenas are limited by the same or a similar exemption, under the name “executive privilege”. But the reasons for these things are rather mysterious: when government officials are conspiring with each other (“deliberating”), isn’t that exactly the sort of thing the public should know? Or that Congress should know, in the case of “executive privilege”?
For that matter, what exactly these exemptions consist of is not entirely clear. Lawyers would probably tell me that it’s clear, but they would also probably advise me that to determine whether they apply in any given case I should avail myself of the services of a legal specialist who is familiar with the precedents set by a long series of court cases and who would charge upwards of $500/hour.
The general idea, admittedly, is clear enough: the exemptions protect records which are made by a government agency as it tries to decide what to do, and which would be rather embarrassing if released in public. To divulge them would reveal all sorts of floundering around: half-baked schemes, wrong ideas that got shot down, and abuse received by ideas that ended up successful. But pretty much any document an agency creates could be said to feed into its decision-making in one way or another, which yields endless opportunities for lawyers to squabble over the details.
Wikipedia’s stub article says that these privileges are “common-law” privileges, which is lawyer-speak for something that was invented by lawyers and judges rather than something that Congress passed into law (or, in the case of “executive privilege”, something found in the Constitution). “The theory”, it says, “is that by guaranteeing confidentiality, the government will receive better or more candid advice, recommendations and opinions”. Well, certainly there are people who would refrain from telling the government something if they knew it would become public. But that’s often because it isn’t actually true. Being known for saying an unpopular truth can be bad, but isn’t as bad as being known for saying an unpopular lie. (Popular ideas, whether true or false, are going to get heard by the government anyway.) For that matter, in practice these privileges mostly seem to be used to protect government employees, not outsiders offering advice. These people have chosen to be in government, and are being paid to be in government; they can stand a little embarrassment, even if undeserved.
Other arguments might be imagined. But these exemptions aren’t about national security secrets, where it is important that the public should know them but even more important that foreigners should not; national security has its own exemptions. Nor are they about protecting government officials from harrassment; in the case of FOIA that’s done by fees, and in the case of executive privilege it doesn’t really need to be done in the first place, since Congress has a lot less manpower with which to investigate than the executive branch has to respond to its investigations.
So what’s going on here seems to be that in response to lawsuits demanding documents, executive branch lawyers have argued that the Freedom of Information Act couldn’t possibly mean what it actually says, because that would be much too intrusive; likewise for Congress’s oversight powers. And the courts have been weak enough to buy those arguments.