Jay

Jay aims to be the fastest production safe, serialization package written in Go as an alternative to JSON, Protocol Buffers, FlatBuffers, gob, MessagePack, Bebop, and mus-go with less setup required and no extra languages to learn.

Jay doesn't determine any types during runtime. Instead, the marshalling and unmarshalling functions are easily generated using the jay commandline tool.

TLDR;

Pros:

• The fastest Go serialization and deserialization message format (some types use zero memory allocations).
• Encoded variables use less network overhead because no schema is added to the output when marshalling (unlike JSON).
• No custom language to learn. Jay uses Go's built in ast to find exported structs in your existing codebase.
• No hassles between client and server. Generate the code once and share (via a Go module or copy-paste).
• Options to generate code optimized for:
  • Processing more requests per second (higher CPU throughput) OR,
  • Least network bandwidth used (10/100 networks).
• Doesn't introduce extra dependencies.
• Output could be compressed with gzip, brotli, zstd or others.
• Greater support for Go's built-in types.

Cons:

• Need to regenerate methods when Go structs are modified using the jay commandline tool.
• Marshalled output is not human-readable.
• Only written for the Go language.

Speed

Unlike JSON, Jay's marshal and unmarshal functions are generated by a command line tool. This significantly increases execution speed during runtime by removing type reflection.

Most small structs with 10 fields can be serialized within 175 nanoseconds on old hardware (Intel T6400 @ 2.0 GHz with GM45 GPU).

Install

Install the jay command line tool to generate marshal and unmarshal code.

go install github.com/speedyhoon/jay/generate/cmd/jay

Command line tool.

1. Execute the command line tool and specify which .go file contains the exported structs. For example, type Car struct is located within main.go.

   cd my_project_path
   jay main.go

2. jay will then generate methods MarshalJ() and UnmarshalJ([]byte) in jay.go (the default output file).
3. The new methods can then be used. For example,

package main

import "fmt"

type Car struct {
	ID            uint
	Make, Model   string
	Auto, HasFuel bool
}

func main() {
	car := Car{ID: 42, Make: "Ford", Model: "Escort", Auto: true, HasFuel: true}
	var src []byte
	src = car.MarshalJ()
	fmt.Println(src)

	var car2 Car
	err := car2.UnmarshalJ(src)
	fmt.Printf("err: %v, %+v", err, car2)
}

Message format

Exported struct fields are concatenated together in binary format, delimited by each field's length.

High-throughput mode on a 64-bit system:

Auto & HasFuel, ID,               Make,      Model         = 21 bytes
[192,           42,0,0,0,0,0,0,0, 4,F,o,r,d, 6,E,s,c,o,r,t]

Low-bandwidth mode: (unfinished)

Auto & HasFuel, ID,   Make,      Model         = 15 bytes
[192,           1,42, 4,F,o,r,d, 6,E,s,c,o,r,t]

Supported types:

• bool, []bool
• byte, []byte
• complex64, complex128
• []complex64, []complex128
• float32, float64
• []float32, []float64
• int, int8, int16, int32, int64
• []int, []int8, []int16, []int32, []int64
• rune, []rune
• string (Currently limited to 255 byte lengths.)
• struct and anonymous structs
• time.Time, time.Duration
• []time.Time, []time.Duration
• uint, uint8, uint16, uint32, uint64
• []uint, []uint8, []uint16, []uint32, []uint64
• Arrays for all the above types with lengths from 1 to 255, for example: [5]string, [9]time.Duration and [33]time.Time (no fuzz testing yet)

Jay also supports imported types. If the exported type is in the same package or a subpackage then jay will automatically find it with:

cd my_project_directory
jay .

To include an external type via the commandline, either:

• Specify the imported file:
  jay myFile.go ../other_project/myImportedFile.go
• Or include that directory:
  jay myFile.go my_imported_directory

TODO

In order of priority:

• Expand fuzz testing and test coverage.
• Field tag options and documentation.
• Increase supported slice length from 255 items to 2²⁴ (16,777,215)
• Add tests for array types with more than 255 items.
• Performance benchmarks.
• Low-bandwidth mode.
• Aliased definition slice types like []float where float is defined as type float float32.
  (Already supported: type float = float32, type floats = []float32 & type floats []float32).

Undecided

• Pointers (*uint64)
  R&D -- have a working prototype for *bool and all integer types, undecided on *string, slices, arrays and struct slices.
• Multi-dimensional arrays & slices? ([][]string)
• Maps? (map[string]uint)

Done

• Specify which struct types to process in a large project directory. E.g.: -y Animal,settings.Config,engine.Specs
• Aliased types like type float = float32 and type floats = []float32.
• Simple definition types for built-ins only like type floats []float32.
• Slices with lengths 0–255 ([]string).

Not Supported

• interface{} requires adding reflection and/or type switches at a significant performance and complexity cost. Either use a different package or write your own function to convert the interface{} to one of the supported types (a []byte might be the fastest option).
• Double nested pointers (**string).
• Generics (any).
• Functions.

Why

Conceived in 2011 as an interesting way to improve network communication. It wasn't until 2023 when I was using JSON (wasn't my choice by the way) to marshal and unmarshal data between microcontrollers that it posed a significant bottleneck. The aim was to process external messages within a dozen microseconds to restore performance without upgrading the processor.

Name

Clarification: The serialization format is Jay, whereas jay is the command line tool.

Jay (pronounced as just J) is a wordplay on JSON without the SON, since the schema information is chopped off 🪚 and it's not human-readable.

The name Jay also gives tribute to a 17-year-old netbook with a stuck j key on the keyboard. 🔁 😆 Every boot looks like:

jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj
jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj
jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj