Integration Testing
The providence project contains integration tests. The integration tests can be run separately, and is not a dependency for running the main test suites.
There are currently two IT test suites.
- [it-serialization]: Thorough testing of serialization focusing on
serialization speed. The thrift compatibility is for the most part
already tested in
providence-thrift. - [it-hazelcast]: Proper testing of generated hazelcast generated code by using actual generated code against a local hazelcast instance including how thrift IDL updates are handled both forward and backward.
Serialization Speed
To run the serialization integration test suite, after installing the library, run:
mvn package -Pit-serialization
java -jar it-serialization/target/it-serialization.jar --help
java -jar it-serialization/target/it-serialization.jar
It will take a while, and print quite a lot to std out.
This is a comparative serialization speed test. It is used to see the progress of serialization optimization, and to compare with the native thrift libraries.
- The
pvdcolumns are the native providence serialization or in the case of the*_protocolthe thrift protocol wrapper performance. - The
thrcolumns is for the native thrift protocol read / write. The “content” is exactly same as the providence version. But compatibility tests are not done here, but in theprovidence-testingmodule.
Latest output from the serialization speed IT, sorted by the SUM of the providence serialization time. Lower is better. There are a number tests, each with a structure focusing on some aspect or style of structures to test. Exceptions are not tested as part of this, as the way the stack traces are handled quickly deteriorates the results for reading providence, though not affecting the others much.
java -jar it-serialization/target/it-serialization.jar --runs 50 --generate 1000
Last updated using 2.0.0-SNAPSHOT release and compared against apache thrift 0.12.0.
Many containers:
READ WRITE SUM SIZE
name : pvd thr -- pvd thr = pvd thr -- (ratio / size)
binary: 1,80 1,00 -- 1,41 1,00 = 1,60 1,00 -- ( 1,00 / 8,6 MiB)
fast_binary: 1,92 -- 2,08 = 2,00 -- ( 0,85 / 7,3 MiB)
tuple_protocol: 2,64 1,10 -- 2,27 0,75 = 2,45 0,92 -- ( 0,81 / 6,9 MiB)
binary_protocol: 2,58 -- 2,65 = 2,62 -- ( 1,00 / 8,6 MiB)
compact_protocol: 3,03 1,32 -- 2,61 0,93 = 2,82 1,13 -- ( 0,85 / 7,3 MiB)
json: 8,63 -- 7,92 = 8,28 -- ( 1,18 / 10,2 MiB)
json_named: 9,75 -- 8,35 = 9,05 -- ( 1,43 / 12,3 MiB)
pretty: 15,73 -- 9,72 = 12,72 -- ( 1,46 / 12,5 MiB)
json_pretty: 14,45 -- 11,43 = 12,94 -- ( 2,31 / 19,9 MiB)
config: 19,51 -- 11,13 = 15,32 -- ( 1,94 / 16,7 MiB)
Many Optional Fields:
READ WRITE SUM SIZE
name : pvd thr -- pvd thr = pvd thr -- (ratio / size)
binary: 1,02 1,00 -- 0,86 1,00 = 0,94 1,00 -- ( 1,00 / 2,5 MiB)
fast_binary: 1,28 -- 1,35 = 1,31 -- ( 0,89 / 2,2 MiB)
tuple_protocol: 1,72 1,10 -- 1,47 0,74 = 1,60 0,92 -- ( 0,83 / 2,0 MiB)
binary_protocol: 1,78 -- 1,72 = 1,75 -- ( 1,00 / 2,5 MiB)
compact_protocol: 2,12 1,34 -- 1,72 0,94 = 1,92 1,14 -- ( 0,87 / 2,1 MiB)
json: 6,19 -- 5,57 = 5,88 -- ( 1,27 / 3,1 MiB)
json_named: 7,15 -- 5,88 = 6,51 -- ( 1,53 / 3,8 MiB)
json_pretty: 8,75 -- 7,12 = 7,94 -- ( 2,02 / 5,0 MiB)
pretty: 10,79 -- 6,47 = 8,63 -- ( 1,48 / 3,6 MiB)
config: 12,44 -- 7,13 = 9,78 -- ( 1,77 / 4,4 MiB)
Many Required Fields:
READ WRITE SUM SIZE
name : pvd thr -- pvd thr = pvd thr -- (ratio / size)
binary: 0,95 1,00 -- 0,90 1,00 = 0,92 1,00 -- ( 1,00 / 2,4 MiB)
fast_binary: 1,21 -- 1,43 = 1,32 -- ( 0,89 / 2,2 MiB)
tuple_protocol: 1,50 1,00 -- 1,49 0,70 = 1,50 0,85 -- ( 0,83 / 2,0 MiB)
binary_protocol: 1,67 -- 1,80 = 1,74 -- ( 1,00 / 2,4 MiB)
compact_protocol: 2,01 1,38 -- 1,79 0,93 = 1,90 1,16 -- ( 0,87 / 2,1 MiB)
json: 5,85 -- 5,91 = 5,88 -- ( 1,27 / 3,1 MiB)
json_named: 6,70 -- 6,17 = 6,44 -- ( 1,53 / 3,7 MiB)
json_pretty: 8,16 -- 7,49 = 7,83 -- ( 2,03 / 4,9 MiB)
pretty: 10,46 -- 6,78 = 8,62 -- ( 1,49 / 3,6 MiB)
config: 12,15 -- 7,47 = 9,81 -- ( 1,78 / 4,3 MiB)
Deep Structure:
READ WRITE SUM SIZE
name : pvd thr -- pvd thr = pvd thr -- (ratio / size)
binary: 0,99 1,00 -- 0,79 1,00 = 0,89 1,00 -- ( 1,00 / 7,4 MiB)
fast_binary: 1,30 -- 1,50 = 1,40 -- ( 0,82 / 6,0 MiB)
tuple_protocol: 1,90 0,97 -- 1,77 0,60 = 1,83 0,79 -- ( 0,75 / 5,5 MiB)
binary_protocol: 2,07 -- 2,23 = 2,15 -- ( 1,00 / 7,4 MiB)
compact_protocol: 2,43 1,32 -- 2,23 0,90 = 2,33 1,11 -- ( 0,81 / 6,0 MiB)
json: 6,18 -- 5,87 = 6,03 -- ( 1,22 / 9,0 MiB)
json_named: 7,85 -- 6,44 = 7,14 -- ( 1,74 / 12,8 MiB)
pretty: 13,68 -- 7,13 = 10,40 -- ( 1,71 / 12,6 MiB)
json_pretty: 13,48 -- 9,85 = 11,66 -- ( 3,41 / 25,0 MiB)
config: 18,61 -- 8,90 = 13,75 -- ( 2,64 / 19,4 MiB)
NOTE: The thrift JSON protocol was removed because some weird bug keeps messing up deserialization and stopping the test. I will put it back once the bug is fixed or can be bypassed.
NOTE: Since the test is for the speed of the serialization, we are only interested in the comparison between the serializers, not the absolute values. All numbers have been normalised to be relative to the native thrift binary protocol implementation (that’s why it’s 1.00 for thrift binary).
NOTE It seems to me like providence fares better the larger the test is and the more contentious the threads and memory are. E.g. this is a comparison over 100 runs of 5000 random generated structures each for the binary protocol only:
READ WRITE SUM SIZE
name : pvd thr -- pvd thr = pvd thr -- (ratio / size)
containers: 1,45 1,00 -- 1,43 1,00 = 1,44 1,00 -- ( 1,00 / 43,2 MiB)
many_fields: 0,91 1,00 -- 0,97 1,00 = 0,94 1,00 -- ( 1,00 / 12,2 MiB)
many_required_fields: 0,86 1,00 -- 0,97 1,00 = 0,92 1,00 -- ( 1,00 / 12,2 MiB)
deep_structure: 0,82 1,00 -- 0,85 1,00 = 0,84 1,00 -- ( 1,00 / 36,8 MiB)
But I have not been able to verify the cause. Sometimes I get the same change in timing with the same tests as above, still without me knowing why.