ProtoBuf Utilities

GitLab Docs Pipeline Coverage License
This utility package contains tools for making protocol buffers truly a dynamic type, with various extensions that build upon the core. See morimekta.net/utils for procedures on releases.

Getting Started

To add to maven: Add this line to pom.xml under dependencies:

<dependency>
    <dependency>
        <groupId>net.morimekta.utils</groupId>
        <artifactId>proto-core</artifactId>
        <version>1.0.4</version>
    </dependency>
    <dependency>
        <groupId>net.morimekta.utils</groupId>
        <artifactId>proto-gson</artifactId>
        <version>1.0.4</version>
    </dependency>
    <dependency>
        <groupId>net.morimekta.utils</groupId>
        <artifactId>proto-jackson</artifactId>
        <version>1.0.4</version>
    </dependency>
    <dependency>
        <groupId>net.morimekta.utils</groupId>
        <artifactId>proto-testing</artifactId>
        <version>1.0.4</version>
        <scope>test</scope>
    </dependency>
</dependency>

To add to gradle: Add this line to the dependencies group in build.gradle:

implementation 'net.morimekta.utils:proto-core:1.0.4'
implementation 'net.morimekta.utils:proto-gson:1.0.4'
implementation 'net.morimekta.utils:proto-jackson:1.0.4'
testImplementation 'net.morimekta.utils:proto-testing:1.0.4'

Proto : Core

Contains utilities to manage message and enum types, and convert values and fields.

  • ProtoMessage Wrapper around a message (and message type) to help with accessing values and modifying message content.
  • ProtoEnum: Wrapper for an enum type to help with value lookup and conversion.
  • ProtoList: Wrapper for a repeated (list) field to behave like a java list. Includes a mutable variant that will mutate the underlying data to match with updated map entries.
  • ProtoMap: Wrapper for a map field to behave like a java map. Includes a mutable variant that will mutate the underlying data to match with updated map entries.

And various utilities for managing fields, message content and types, reflection for accessing static message types and methods, and converting values.

Proto : Gson

Serialize and deserialize proto messages and enums in any structure with various extra modes using the Gson Json reader and writer.

  • Write field keys using field number or name.
  • Write enum values using its number or name.
  • Write compact messages using the JSON array syntax with fields in numeric order.
  • Write Any messages with the unpacked message and a @type marker.
  • Read back with all the options above.
  • Read strictly (only allowing know fields and values), or leniently (ignoring unknown fields and enum values).

Uses ProtoTypeOptions.Option to modify reading and writing features.

  • FAIL_ON_UNKNOWN_ENUM: Fail parsing when encountering an unknown enum value (name or number). Otherwise return null on unknown enum values.
  • FAIL_ON_UNKNOWN_FIELD: Fail parsing when encountering an unknown message field. Otherwise, ignore field and it's value.
  • FAIL_ON_NULL_VALUE: When parsing a message and encountering an null value (either explicit null, or an unparseable field value of some kind), fail parsing.
  • IGNORE_UNKNOWN_ANY_TYPE: When parsing an unpacked Any message, skip the value if the type is not known.
  • LENIENT_READER: Allow uncommon field values to be parsed leniently. E.g. string to boolean or number.
  • WRITE_FIELD_AS_NUMBER: Write fields as number instead of the field name.
  • WRITE_ENUM_AS_NUMBER: Write enum values as the enum number instead of the value name.
  • WRITE_UNPACKED_ANY: Write any messages with known types as unpacked messages. This will write the message as the JSON content of itself with a @type or __type field as the first field entry.
  • WRITE_COMPACT_MESSAGE: Write messages with the option morimekta.proto.compact set to true as array instead of as a map. The array is simply the numerically ordered field values as an array. Trailing nulls are skipped.
  • WRITE_TIMESTAMP_AS_ISO: Write instances of google.protobuf.Timestamp as an ISO instant formatted date time string, using UTC as timezone, e.g. 2009-02-13T23:31:30Z.
  • WRITE_DURATION_AS_STRING: Write instances of google.protobuf.Duration as a simple duration string of seconds, e.g. 3.7s for a 3.7 second duration.

And ProtoTypeOptions.Value to modify some specific values options.

  • ANY_TYPE_FIELD_NAME: The name of the 'type' field when writing unpacked Any. Defaults to @type.
  • ANY_TYPE_PREFIX: The type prefix before the prototype in the type field value. Defaults to type.googleapis.com/

Note that to detect types of unwrapped Any and extensions, the type registry needs to be set with the appropriate types registered.

Proto : Jackson

Same features as Gson, but using Jackson.

Uses ProtoFeature to modify reading and writing boolean features:Contains the same options with the same defaults as ProtoTypeOptions.Option in proto-gson.

Uses ProtoStringFeature to modify reading and writing string (value) based features: Contains the same options with the same defaults as ProtoTypeOptions.Value in proto-gson.

Proto : JDBI

Support easy mapping of proto messages (and field types) to JDBI columns and rows.

  • BindMessage: Bind a message as a single value, either encoding as binary (VARBINARY using its normal binary encoding) and as strings (VARCHAR).
  • BindEnumName: Bind a proto enum type as a string value.
  • BindEnumNumber: Bind a proto enum type as a numeric value.
  • RegisterEnumMapper: Register column mappers for reading enum values from JDBI result sets.
  • RegisterMessageMapper: Register row and column mappers for proto Message types.
  • MessageUpserter: Helper class to handle inserting content from messages into a table. The helper will only select values form the message itself, not using nested structure or anything like that.

And also some helpers for handling other fields / columns of ByteString.

  • BindByteString: Annotation that can be put on a bind that also ensures the field is handled correctly as a ByteString data type. Also supports changing the encoding of the type (for VARCHAR and similar string-like fields).
  • RegisterByteString: Register byte string mappers for column result and argument mappers

Proto : Testing

  • ProtoMatchers: Hamcrest matchers for matching two proto messages. Will ignore fields using the syntax of proto field masks. The output will show differences between the two messages in a diff-like format.

Possible Future

  • Proto : JDBI Map proto <-> JDBI, both v2 and v3.
  • Proto : Config Port providence - config to use proto?