应用说明：Field 存在¶

本应用说明解释了 protobuf 字段的各种存在跟踪规则。它还解释了具有基本类型的 singular proto3 字段的显式存在跟踪的行为。

背景¶

Field presence 是指一个 protobuf 字段是否有一个值的概念。对于 protobufs 来说，有两种不同的存在表现形式。no presence，即生成的消息 API 存储字段值（仅），以及 explicit presence，即 API 也存储一个字段是否已被设置。

从历史上看，proto2 大多遵循 explicit presence，而 proto3 只暴露了 no presence 语义。基本类型（数字、字符串、字节和枚举）的 singular proto3 字段，如果用 optional 标签定义，就会像 proto2 一样，具有 explicit presence （这个功能在 3.15 版本中默认启用）。

存在法则¶

Presence disciplines 定义了在 API 表示法 和 序列化表示法 之间转换的语义。no presence 法则依靠字段值本身在（去）序列化时做出决定，而 explicit presence 法则则依靠明确的跟踪状态

存在于 tag-value stream （wire 格式）的序列化中¶

The wire format is a stream of tagged, self-delimiting values. By definition, the wire format represents a sequence of present values. In other words, every value found within a serialization represents a present field; furthermore, the serialization contains no information about not-present values.

The generated API for a proto message includes (de)serialization definitions which translate between API types and a stream of definitionally present (tag, value) pairs. This translation is designed to be forward- and backward-compatibile across changes to the message definition; however, this compatibility introduces some (perhaps surprising) considerations when deserializing wire-formatted messages:

When serializing, fields with no presence are not serialized if they contain their default value.
- For numeric types, the default is 0.
- For enums, the default is the zero-valued enumerator.
- For strings, bytes, and repeated fields, the default is the zero-length value.
- For messages, the default is the language-specific null value.
“Empty” length-delimited values (such as empty strings) can be validly represented in serialized values: the field is “present,” in the sense that it appears in the wire format. However, if the generated API does not track presence, then these values may not be re-serialized; i.e., the empty field may be “not present” after a serialization round-trip.
When deserializing, duplicate field values may be handled in different ways depending on the field definition.
- Duplicate repeated fields are typically appended to the field’s API representation. (Note that serializing a packed repeated field produces only one, length-delimited value in the tag stream.)
- Duplicate optional field values follow the rule that “the last one wins.”
oneof fields expose the API-level invariant that only one field is set at a time. However, the wire format may include multiple (tag, value) pairs which notionally belong to the oneof. Similar to optional fields, the generated API follows the “last one wins” rule.
Out-of-range values are not returned for enum fields in generated proto2 APIs. However, out-of-range values may be stored as unknown fields in the API, even though the wire-format tag was recognized.

Presence in named-field mapping formats¶

Protobufs can be represented in human-readable, textual forms. Two notable formats are TextFormat (the output format produced by generated message DebugString methods) and JSON.

These formats have correctness requirements of their own, and are generally stricter than tagged-value stream formats. However, TextFormat more closely mimics the semantics of the wire format, and does, in certain cases, provide similar semantics (for example, appending repeated name-value mappings to a repeated field). In particular, similar to the wire format, TextFormat only includes fields which are present.

JSON is a much stricter format, however, and cannot validly represent some semantics of the wire format or TextFormat.

Notably, JSON elements are semantically unordered, and each member must have a unique name. This is different from TextFormat rules for repeated fields.
JSON may include fields that are “not present,” unlike the no presence discipline for other formats:
- JSON defines a null value, which may be used to represent a defined but not-present field.
- Repeated field values may be included in the formatted output, even if they are equal to the default (an empty list).
Because JSON elements are unordered, there is no way to unambiguously interpret the “last one wins” rule.
- In most cases, this is fine: JSON elements must have unique names: repeated field values are not valid JSON, so they do not need to be resolved as they are for TextFormat.
- However, this means that it may not be possible to interpret oneof fields unambiguously: if multiple cases are present, they are unordered.

In theory, JSON can represent presence in a semantic-preserving fashion. In practice, however, presence correctness can vary depending upon implementation choices, especially if JSON was chosen as a means to interoperate with clients not using protobufs.

Presence in proto2 APIs¶

This table outlines whether presence is tracked for fields in proto2 APIs (both for generated APIs and using dynamic reflection):

Field type	Explicit Presence
Singular numeric (integer or floating point)	✔️
Singular enum	✔️
Singular string or bytes	✔️
Singular message	✔️
Repeated
Oneofs	✔️
Maps

Singular fields (of all types) track presence explicitly in the generated API. The generated message interface includes methods to query presence of fields. For example, the field foo has a corresponding has_foo method. (The specific name follows the same language-specific naming convention as the field accessors.) These methods are sometimes referred to as “hazzers” within the protobuf implementation.

Similar to singular fields, oneof fields explicitly track which one of the members, if any, contains a value. For example, consider this example oneof:

oneof foo {
  int32 a = 1;
  float b = 2;
}

Depending on the target language, the generated API would generally include several methods:

A hazzer for the oneof: has_foo
A oneof case method: foo
Hazzers for the members: has_a, has_b
Getters for the members: a, b

Repeated fields and maps do not track presence: there is no distinction between an empty and a not-present repeated field.

Presence in proto3 APIs¶

This table outlines whether presence is tracked for fields in proto3 APIs (both for generated APIs and using dynamic reflection):

Field type	`optional`	Explicit Presence
Singular numeric (integer or floating point)	No
Singular enum	No
Singular string or bytes	No
Singular numeric (integer or floating point)	Yes	✔️
Singular enum	Yes	✔️
Singular string or bytes	Yes	✔️
Singular message	Yes	✔️
Singular message	No	✔️
Repeated	N/A
Oneofs	N/A	✔️
Maps	N/A

Similar to proto2 APIs, proto3 does not track presence explicitly for repeated fields. Without the optional label, proto3 APIs do not track presence for basic types (numeric, string, bytes, and enums), either. Oneof fields affirmatively expose presence, although the same set of hazzer methods may not generated as in proto2 APIs.

Under the no presence discipline, the default value is synonymous with “not present” for purposes of serialization. To notionally “clear” a field (so it won’t be serialized), an API user would set it to the default value.

The default value for enum-typed fields under no presence is the corresponding 0-valued enumerator. Under proto3 syntax rules, all enum types are required to have an enumerator value which maps to 0. By convention, this is an UNKNOWN or similarly-named enumerator. If the zero value is notionally outside the domain of valid values for the application, this behavior can be thought of as tantamount to explicit presence.

Semantic differences¶

The no presence serialization discipline results in visible differences from the explicit presence tracking discipline, when the default value is set. For a singular field with numeric, enum, or string type:

No presence discipline:
- Default values are not serialized.
- Default values are not merged-from.
- To “clear” a field, it is set to its default value.
- The default value may mean:
  - the field was explicitly set to its default value, which is valid in the application-specific domain of values;
  - the field was notionally “cleared” by setting its default; or
  - the field was never set.
Explicit presence discipline:
- Explicitly set values are always serialized, including default values.
- Un-set fields are never merged-from.
- Explicitly set fields – including default values – are merged-from.
- A generated has_foo method indicates whether or not the field foo has been set (and not cleared).
- A generated clear_foo method must be used to clear (i.e., un-set) the value.

Considerations for merging¶

Under the no presence rules, it is effectively impossible for a target field to merge-from its default value (using the protobuf’s API merging functions). This is because default values are skipped, similar to the no presence serialization discipline. Merging only updates the target (merged-to) message using the non-skipped values from the update (merged-from) message.

The difference in merging behavior has further implications for protocols which rely on partial “patch” updates. If field presence is not tracked, then an update patch alone cannot represent an update to the default value, because only non-default values are merged-from.

Updating to set a default value in this case requires some external mechanism, such as FieldMask. However, if presence is tracked, then all explicitly-set values – even default values – will be merged into the target.

Considerations for change-compatibility¶

Changing a field between explicit presence and no presence is a binary-compatible change for serialized values in wire format. However, the serialized representation of the message may differ, depending on which version of the message definition was used for serialization. Specifically, when a “sender” explicitly sets a field to its default value:

The serialized value following no presence discipline does not contain the default value, even though it was explicitly set.
The serialized value following explicit presence discipline contains every “present” field, even if it contains the default value.

This change may or may not be safe, depending on the application’s semantics. For example, consider two clients with different versions of a message definition.

Client A uses this definition of the message, which follows the explicit presence serialization discipline for field foo:

syntax = "proto3";
message Msg {
  optional int32 foo = 1;
}

Client B uses a definition of the same message, except that it follows the no presence discipline:

syntax = "proto3";
message Msg {
  int32 foo = 1;
}

Now, consider a scenario where client A observes foo’s presence as the clients repeatedly exchange the “same” message by deserializing and reserializing:

// Client A:
Msg m_a;
m_a.set_foo(1);                  // non-default value
assert(m_a.has_foo());           // OK
Send(m_a.SerializeAsString());   // to client B

// Client B:
Msg m_b;
m_b.ParseFromString(Receive());  // from client A
assert(m_b.foo() == 1);          // OK
Send(m_b.SerializeAsString());   // to client A

// Client A:
m_a.ParseFromString(Receive());  // from client B
assert(m_a.foo() == 1);          // OK
assert(m_a.has_foo());           // OK
m_a.set_foo(0);                  // default value
Send(m_a.SerializeAsString());   // to client B

// Client B:
Msg m_b;
m_b.ParseFromString(Receive());  // from client A
assert(m_b.foo() == 0);          // OK
Send(m_b.SerializeAsString());   // to client A

// Client A:
m_a.ParseFromString(Receive());  // from client B
assert(m_a.foo() == 0);          // OK
assert(m_a.has_foo());           // FAIL

If client A depends on explicit presence for foo, then a “round trip” through client B will be lossy from the perspective of client A. In the example, this is not a safe change: client A requires (by assert) that the field is present; even without any modifications through the API, that requirement fails in a value- and peer-dependent case.

How to enable explicit presence in proto3¶

These are the general steps to use field tracking support for proto3:

Add an optional field to a .proto file.
Run protoc (at least v3.15, or v3.12 using --experimental_allow_proto3_optional flag).
Use the generated “hazzer” methods and “clear” methods in application code, instead of comparing or setting default values.

`.proto` file changes¶

This is an example of a proto3 message with fields which follow both no presence and explicit presence semantics:

syntax = "proto3";
package example;

message MyMessage {
  // No presence:
  int32 not_tracked = 1;

  // Explicit presence:
  optional int32 tracked = 2;
}

`protoc` invocation¶

Presence tracking for proto3 messages is enabled by default since v3.15.0 release, formerly up until v3.12.0 the --experimental_allow_proto3_optional flag was required when using presence tracking with protoc.

Using the generated code¶

The generated code for proto3 fields with explicit presence (the optional label) will be the same as it would be in a proto2 file.

This is the definition used in the “no presence” examples below:

syntax = "proto3";
package example;
message Msg {  
  int32 foo = 1;
}

This is the definition used in the “explicit presence” examples below:

syntax = "proto3";
package example;
message Msg {
  optional int32 foo = 1;
}

In the examples, a function GetProto constructs and returns a message of type Msg with unspecified contents.

C++ example¶

No presence:

Msg m = GetProto();
if (m.foo() != 0) {
  // "Clear" the field:
  m.set_foo(0);
} else {
  // Default value: field may not have been present.
  m.set_foo(1);
}

Explicit presence:

Msg m = GetProto();
if (m.has_foo()) {
  // Clear the field:
  m.clear_foo();
} else {
  // Field is not present, so set it.
  m.set_foo(1);
}

C# example¶

No presence:

var m = GetProto();
if (m.Foo != 0) {
  // "Clear" the field:
  m.Foo = 0;
} else {
  // Default value: field may not have been present.
  m.Foo = 1;
}

Explicit presence:

var m = GetProto();
if (m.HasFoo) {
  // Clear the field:
  m.ClearFoo();
} else {
  // Field is not present, so set it.
  m.Foo = 1;
}

Go example¶

No presence:

m := GetProto()
if m.Foo != 0 {
  // "Clear" the field:
  m.Foo = 0
} else {
  // Default value: field may not have been present.
  m.Foo = 1
}

Explicit presence:

m := GetProto()
if m.Foo != nil {
  // Clear the field:
  m.Foo = nil
} else {
  // Field is not present, so set it.
  m.Foo = proto.Int32(1)
}

Java example¶

These examples use a Builder to demonstrate clearing. Simply checking presence and getting values from a Builder follows the same API as the message type.

No presence:

Msg.Builder m = GetProto().toBuilder();
if (m.getFoo() != 0) {
  // "Clear" the field:
  m.setFoo(0);
} else {
  // Default value: field may not have been present.
  m.setFoo(1);
}

Explicit presence:

Msg.Builder m = GetProto().toBuilder();
if (m.hasFoo()) {
  // Clear the field:
  m.clearFoo()
} else {
  // Field is not present, so set it.
  m.setFoo(1);
}

Python example¶

No presence:

m = example.Msg()
if m.foo != 0:
  # "Clear" the field:
  m.foo = 0
else:
  # Default value: field may not have been present.
  m.foo = 1

Explicit presence:

m = example.Msg()
if m.HasField('foo'):
  # Clear the field:
  m.ClearField('foo')
else:
  # Field is not present, so set it.
  m.foo = 1

Ruby example¶

No presence:

m = Msg.new
if m.foo != 0
  # "Clear" the field:
  m.foo = 0
else
  # Default value: field may not have been present.
  m.foo = 1
end

Explicit presence:

m = Msg.new
if m.has_foo?
  # Clear the field:
  m.clear_foo
else
  # Field is not present, so set it.
  m.foo = 1
end

Javascript example¶

No presence:

var m = new Msg();
if (m.getFoo() != 0) {
  // "Clear" the field:
  m.setFoo(0);
} else {
  // Default value: field may not have been present.
  m.setFoo(1);
}

Explicit presence:

var m = new Msg();
if (m.hasFoo()) {
  // Clear the field:
  m.clearFoo()
} else {
  // Field is not present, so set it.
  m.setFoo(1);
}

Objective C example¶

No presence:

Msg *m = [[Msg alloc] init];
if (m.foo != 0) {
  // "Clear" the field:
  m.foo = 0;
} else {
  // Default value: field may not have been present.
  m.foo = 1;
}

Explicit presence:

Msg *m = [[Msg alloc] init];
if (m.hasFoo()) {
  // Clear the field:
  [m clearFoo];
} else {
  // Field is not present, so set it.
  [m setFoo:1];
}

protobuf 0.1 文档

应用说明：Field 存在¶

背景¶

存在法则¶

存在于 tag-value stream （wire 格式）的序列化中¶

Presence in named-field mapping formats¶

Presence in proto2 APIs¶

Presence in proto3 APIs¶

Semantic differences¶

Considerations for merging¶

Considerations for change-compatibility¶

How to enable explicit presence in proto3¶

.proto file changes¶

protoc invocation¶

Using the generated code¶

C++ example¶

C# example¶

Go example¶

Java example¶

Python example¶

Ruby example¶

Javascript example¶

Objective C example¶

`.proto` file changes¶

`protoc` invocation¶