Dictionary Benchmarks

Introduction

As I'm re-writing AWS.jl I want to improve on the existing code from AWSCore.jl. I had noticed the usage of SymDict.jl throughout the codebase. I wanted to see how much quicker SymbolDict was in comparison to other data structures and if we should continue to use it in the next iteration of AWS interfacing.

The main usages of SymDict are for things such as containing information about an HTTP request, storing AWSCredentials, etc. These structures are holding <100 values in them and this is what we will primarily be testing. For the fun of it I decided to include higher data points (see below).

In my benchmarking I am comparing the following data structures; Typed Dict, Untyped Dict, Symbol Dict, Typed Little Dict, Untyped Little Dict, and NamedTuple. I wanted to compare the insert, access and delete functions for each of these data structures, with various amounts of data points (10, 100, 1,000, 10,000). I also ran 100 iterations of each of these tests and gathered the min, max, mean, and median values for the above combinations.

Benchmarks -- Median Times

10 Items

	Typed Dict	Untyped Dict	Symbol Dict	Typed Little Dict	Untyped Little Dict
Insert	0.0000124500	0.0000059440	0.0000148030	0.0000168695	0.0000167185
Access	0.0000002070	0.0000004865	0.0000002210	0.0000000600	0.0000035455
Delete	0.0000002100	0.0000004290	0.0000002185	0.0000001440	0.0000016225

100 Items

	Typed Dict	Untyped Dict	Symbol Dict	Typed Little Dict	Untyped Little Dict
Insert	0.0001015510	0.0000500395	0.0001185475	0.0001395585	0.0002099295
Access	0.0000021420	0.0000046095	0.0000021790	0.0000032600	0.0001083420
Delete	0.0000021510	0.0000040855	0.0000021940	0.0000025960	0.0000368785

1,000 Items

	Typed Dict	Untyped Dict	Symbol Dict	Typed Little Dict	Untyped Little Dict
Insert	0.0012016840	0.0005139545	0.0013987445	0.0018717950	0.0074831120
Access	0.0000315860	0.0000377065	0.0000315710	0.0003486135	0.0066067190
Delete	0.0000316205	0.0000337010	0.0000327680	0.0001253675	0.0017595730

10,000 Items

	Typed Dict	Untyped Dict	Symbol Dict	Typed Little Dict	Untyped Little Dict
Insert	0.0134638940	0.0114962655	0.0145114960	0.0360271060	0.6680125280
Access	0.0003432605	0.0006121685	0.0003335305	0.0206947345	0.6456753780
Delete	0.0003279035	0.0004877520	0.0003291930	0.0078071745	0.1694028450

Conclusion

With the above data point in our use case of ~10 items in the data structure, SymDict is 2.07E-06 quicker at insertions, however LittleDict is 1.61E-07 and 7.45E-08 quicker as accessing and deleting data.

The benefits of this at such low levels is negligible at best, and really either of these data structures will work. For the AWS.jl re-write I will continue forward using LittleDict on the sole basis that the package is more maintained. SymDict.jl has not been updated since 2018-09-20, while OrderedCollections is actively being worked on.

References

• AWSCore.jl
• SymDict.jl
• OrderedCollections.jl
• Julia Dictionary Benchmarking