My approach to load-bearing terminology is to define what I mean when I use it. Furthermore, in my definitions I take care to avoid a tone of prescriptivity. Who am I, after all, to be asserting what these words ought to mean to other people? Rather, I prefer to take a non-exhaustive, phenomenological tack, and then successively try to poke holes in it.

Here are my non-exhaustive criteria, then, I consider necessary for a phenomenon to be called a protocol:

Procedural
A protocol has to take place in time. A natural anti-example, then, would be something like a file format specification. This is an appropriately tricky example because file formats are specified the same way as protocols, indeed the processes of parsing and serializing a file of a given format looks very much like a network protocol. The publication of said specification, or rather an agreement among peers to use said published specification absolutely can be construed as a protocol. Further formalization of a spec into a standard, is similarly not a protocol, but the standards body could very well have a protocol for producing standards.
Describable
A protocol must be amenable to description, even if the description is not exhaustive. For example, a protocol can have a meta-rule to ignore any operation it doesn't recognize. A protocol need not be strictly repeatable in content, but it does need to have something like a grammar.
Artificial
There is a question that processes like DNA transcription constitute a protocol, though I am inclined to emphasize the conscious creation and/or negotiation by and among sapient (or at least sentient) agents: A protocol is something you consciously try to establish, rather than just muddle toward an equilibrium.
Social/Political
If other people (or at least other beings, e.g. animals) aren't involved, then it isn't meaningful to call the phenomenon a protocol. Without the social component what we have is a procedure, or perhaps a ritual (which is distinct from ceremony, which can involve other people but doesn't have to), or even a policy, which can even be aggregated into a rubric, framework, or regimen.
Participatory
If you aren't participating in a protocol then the most you can be is merely subject to it. This doesn't mean protocols are entirely permissionless though; there are plenty of examples of protocols that require permission to participate. This brings about the notion of a protocol scope: the participants experience a protocol but others see something else, such as a platform or framework. (Or, take for instance going to a hospital: the various staff members engage in all sorts of protocols, to which you are quite literally a patient.)
Emergent Epiphenomena
The function of a protocol is to systematize a social process that would otherwise require ad-hoc negotiation. In other words, a protocol need not itself directly subsume emergent phenomena (e.g., shaking hands), but by eliminating the overhead of whatever ad-hoc negotiation it replaces, it clears the way for resources (particularly time but not limited to it) to be used for other things.

The definition these criteria attempt to fence in is not intended to be authoritative, but useful.