Android programming has a small but very peculiar problem. And when you want to introduce auto-generated code like thrift models into it you tend to see it clearly. It has a 64k method limit per .dex file. And thrift generates a lot of methods. Larger RPC systems can easily generate 20000 methods or more. And it was such a hassle that even the folks at Microsoft decided they wanted to solve the problem, which they called thrifty. Well, when I saw that announcement I had already made providence, and it was only a few days ago.

But (and probably for this reason alone) the protobuf serializer model system has a separate nano format, which is designed to have as few methods as possible. And it goes to extremes to address the limitation. It has (per default):

  • All fields are public, writable, no getter or setter methods.
  • No descriptors for reflective uses.
  • Repeated fields (similar to thrift list<*> types) are placed in arrays instead of list objects (no dependent classes).
  • The bytes field type (similar to binary) is placed in byte arrays, not ByteBuffers (ditto).
  • Enums are integer constants only. Validity is only checked during (de)serialization (no enum, and can be optimized away by ProGuard).
  • Only the read (mergeFrom) and write (writeTo) methods for serialization remain.

And everything else is optional. All of these are ways to avoid generating classes (all of which has at least some methods), enums (which on older android systems could slow down the app considerably), dependencies (which may require more classes to be kept by ProGuard), and otherwise avoid generating methods in general.

The Intermediate Solution

But removing all of that will still put some limitations of what the system can do. E.g. protobuf nano-java can only serialize to and from it’s main binary format and has no RPC service support. And thrifty is still limited by the TProtocol design flaw.

I have decided to have a look at this too, and made the tiny java format for providence, which does quite a bit of what thrifty did, but using the main structure from providence, though no longer generating the reflection helpers and thus rendering the default serializers useless, and cutting out most dependencies.

The goal is to:

  • Generate few methods (but not to avoid generating methods to the extreme).
  • Requiring no java-reflection, which has a tendency of confusing ProGuard.
  • Keep dependencies to the minimum (requiring few referenced classes).

And everything else that may drag in more dependencies or generate extra methods should be optional.

The generated classes

Not everything is supported yet, and some ways of using the classes are altogether removed. So the remaining part of the generated java is for enums:

  • The value and name is wrapped in asInteger and asString methods, which are inherited from io-util interfaces, mainly used for serialization.
  • The static forValue and forName methods. These are needed for de-serialization.

And for messages:

  • A single getter method for each field. Required and default primitive type fields are always present. Otherwise only null is used as a presence outside the builder.
  • The _Builder class with a setter method per field. The builder keeps using a union field reference for presence checking during message building (unions only).
  • The standard Object, Comparable and Stringable overridden methods: equals, hashCode, compareTo, toString and asString.
  • The standard builder-related methods: MyMessage.builder() and message.mutate() are still there.
  • Unions still has the unionField method, but the fields are declared as int constants in place of the _Field enum.

The messages are still true immutable objects, so having messages in constants still make true constants. The whole has two dependencies:

  • net.morimekta.utils:io-util:0.2.3+ for binary field support and some interfaces and helpers used in toString / asString methods, and for the LinkedHash*Builder helper classes for ordered collection support.
  • com.google.guava:guava:* for the other immutable collections support.

Serialization with tiny-java

Since the new tiny java classes no longer have the reflection tools in place, serialization needs to be handled in a different way. I have not settled on a method yet, but will probably try out something like this.

The interface would be two methods on each message class, one to write and one to read the desired binary format, probably decided at compile time:

MyMessage myMessage = MyMessage.read(istream);
myMessage.write(ostream);

Then the methods themselves would use the BinaryWriter and BinaryReader classes from net.morimekta.utils:io-util to do the actual serialization and deserialization, similar to how the default serializers work. The dependency is already there for binary field support (and more).

JSON

Json can be handled in an entirely different manner, by using jackson-2.7. I prefer Jackson 2 as it supports both using builders and specifying custom serializers and deserializers. Many android apps these days already rely on jackson to do some JSON, as it is a pretty fast, well tested and not too include-intensive JSON library.

By adding a custom deserializer class for each message type, we can ensure that we can deserialize all the various nits and tricks that comes from the default format’s JsonSerializer. For unions, a deserializer class is also added, to avoid serializing all the fields. The net result is as if we were using the JsonSerializer(fields = NAME, enums = ID) variant that also has disabled serializing to compact.

And if JSON serialization is no longer supported, it can be removed by simply not compiling with the jackson option. Everything else should still work as before.

Further Works

  • Remove jackson integration from the main java generator. The JsonSerializer format generates json that is compatible with the tiny jackson integration, so it should not be needed. And since the code has diverged so much it needs a serious update anyway.
  • Generate read and write methods (as described above). Can add helper classes in a new providence-tiny module, but mainly try to keep with the no extra deps policy. And rename the jackson module to providence-tiny-jackson. It is no longer ‘core’.
  • Generate pretty printing toString / asString variants on the tiny messages, since the PrettyPrinter is also useless.
  • Make a release with all these updates. The current 0.1.0 version does not have much of the updates I did during the long weekend for the tiny java format.

And finally:

  • Figure out how services would be done with the new format. It took me quite a while to figure out services for the main providence java format, so this is probably not trivial either.