JSON support #345

rooooooooob · 2022-01-26T06:02:23Z

This also auto-generates .ts (and thus flow) definitions for all types and inserts them into the to_js_value(): JsValue .ts/flow defs in place of any.

This is a multi-phase process:

Generate (using schemars traits) .json schemas for all types. This
is done by running the program in rust/json-gen which will export these
to rust/json-gen/schemas
Running scripts/run-json2ts.js to convert these to .ts files. This also does a lot of fixing up like appending JSON, and removing weird warts generated by the 2 tools (schemars/json-schemas->ts)
Running scripts/json-ts-types.js to replace any any return types
with the correct *JSON type generated above and merging these into the main .ts def file

so that we can get better type information than simply `any` in the .ts/flow type annotations. This is a multi-phase process: 1) Generate (using `schemars` traits) .json schemas for all types. This is done by running the program in rust/json-gen which will export these to rust/json-gen/schemas 2) Running scripts/run-json2ts.js to convert these to .ts files 3) We will need to convert the names to add in the JSON suffix to these (not implemented yet) 4) Running scripts/json-ts-types.js to replace any `any` return types with the correct *JSON type generated above some files are hardcoded so they don't line up right now and some steps aren't working as the schemas generated in 1) inline ALL referenced schemas within a single schema so in step 2) we end up with a lot of duplicate type definitions. We might be able to just have a stage to remove duplicates from the schemas, or maybe there's a way to change the json to ts tool to not do that. Some types also throw runtime errors when serializing e.g. non-string keys despite compiling fine since serde != serde_json

scripts/run-json2ts.js

rooooooooob · 2022-01-26T06:04:33Z

More byte types should probably be manually declared as hex strings too since schemars/serde default to a number array.

rooooooooob · 2022-01-26T06:06:53Z

rust/Cargo.toml

 rand_os = "0.1"
 rand_chacha = "0.1"
+
+[build-dependencies]


We can probably get rid of these. I was experimenting with rust build scripts but decided to just have a subcrate to run the script instead since I couldn't access the main crate from it (but I've never worked with rust build scripts so maybe it's possible). It makes more sense to only run these when we do the wasm builds/flowgen I think though anyway.

-get rid of [k: string]: unknown in structs -add js:ts-json-gen command that runs all stages of rust->schemas->ts gen -fixes for some runtime errors (maps) -adding some missing json impls

All definitions are wiped from schemas now and all references to those local definitions are transformed into external definitions which we pass in a param to the schema compiler to not generate definitions for.

Fixes for: -Strings/empty MapOf_ defs -to_js_value() to actually be a JsValue instead of to_json() -swap out type defs in .ts with the JSON ones -get rid of infinite recursion in certain types (no longer using external $refs) -finalize npm build commands -probably some other fixes

rooooooooob · 2022-02-17T23:56:11Z

rust/Cargo.toml

 # wasm
 [target.'cfg(all(target_arch = "wasm32", not(target_os = "emscripten")))'.dependencies]
-wasm-bindgen = "=0.2.78"
+wasm-bindgen = { version = "=0.2.78", features = ["serde-serialize"] }


@SebastienGllmt I had to use this to construct JsValue from a serde-serialization value (I didn't see a way from JSON strings either without making the object a string itself using JsValue::from_str()). I had to do this as otherwise you just end up with a string return type. I'm still confused why we were manually specifying types like ErgoBoxJSON in our flow types when we used sigma-rust from Yoroi since those serde_json calls are the same as I used here in to_json() which definitely return a string (tested from wasm). That's why I put in both to_json() and to_js_value().

Did we have some magic conversion while ignoring the true type going on for sigma-rust that ran JSON.parse()?

To answer your question, no we didn't do anything special.

rooooooooob · 2022-02-18T00:01:02Z

rust/src/metadata.rs

+
+// just for now we'll do json-in-json until I can figure this out better
+// TODO: maybe not generate this? or how do we do this?
+impl JsonSchema for TransactionMetadatum {


This is not the best solution but the options are either:

Just make our own (incompatible with cardano-node) JSON rep using schemars/serde

Manually write .ts defs for the cardano-node JSON schema we follow (I picked DetailedSchema since it supports everything (besides edge-cases with i64 in serde-json) and merge those in as a final step. We'd want to do the same with plutus datums I guess if we go this route.

I'm leaning towards 2 but wanted to get opinions on it. We might have to write them as JSON schemas and generate from there and hope we can get it to match the .ts we want as otherwise other schemas referring to metadata (e.g. tx) might break.

rooooooooob · 2022-02-18T00:01:37Z

rust/src/utils.rs

+
+            #[cfg(all(target_arch = "wasm32", not(target_os = "emscripten")))]
+            pub fn to_js_value(&self) -> Result<JsValue, JsError> {
+                JsValue::from_serde(&self)


See comment above in the cargo.toml

rooooooooob · 2022-02-18T00:02:30Z

rust/src/utils.rs

  }
 }

+impl serde::Serialize for BigNum {


Do we want to_json()/to_js_value()/from_json() here (and BigInt, etc) for consistency even though to_str()/from_str() do the same thing as this would do?

rooooooooob · 2022-02-18T00:03:28Z

scripts/json-ts-types.js

+  // TODO: we might want to make sure we don't have other cases where this would replace
+  // things it shouldn't. We'd have to do some go-back-a-few-lines replace to only do this
+  // for to_json() comments.
+  inputFile[i] = line.replace(/(\s?\*\s?\@returns\s\{)(any)(\})/, `$1${currentClass}JSON$3`);


None of the types return any in our library for now but this could be a problem in the future.

SebastienGllmt · 2022-02-24T12:49:55Z

rust/Cargo.toml

 # wasm
 [target.'cfg(all(target_arch = "wasm32", not(target_os = "emscripten")))'.dependencies]
-wasm-bindgen = "=0.2.78"
+wasm-bindgen = { version = "=0.2.78", features = ["serde-serialize"] }


To answer your question, no we didn't do anything special.

lisicky · 2022-05-17T00:01:57Z

@rooooooooob could you update cardano_serialization_lib.js.flow ? I'm afraid I do smth wrong because I got different result then in the branch.

SebastienGllmt · 2022-05-17T03:47:21Z

@lisicky dcSpark will no longer be contributing to this repo because we are instead managing our fork (cardano-multiplatform-lib) and it would be too much work to back-port all our changes back to here.

rooooooooob · 2022-05-17T04:00:10Z

@lisicky it's a generated file you shouldn't need to manually edit it

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow # rust/src/utils.rs

lisicky · 2022-05-20T12:39:29Z

~~/check~~

lisicky

/check

lisicky

/check

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow # rust/src/crypto.rs # rust/src/lib.rs # rust/src/plutus.rs # rust/src/tx_builder.rs # rust/src/utils.rs

lisicky

/check

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow # rust/src/lib.rs # rust/src/metadata.rs

lisicky

/check

# Conflicts: # rust/src/plutus.rs

lisicky

/check

lisicky

/check

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow

rooooooooob marked this pull request as draft January 26, 2022 06:02

rooooooooob commented Jan 26, 2022

View reviewed changes

scripts/run-json2ts.js Outdated Show resolved Hide resolved

rooooooooob commented Jan 26, 2022

View reviewed changes

rooooooooob and others added 4 commits February 9, 2022 14:28

Various JSON schema fixes:

fe09abc

-get rid of [k: string]: unknown in structs -add js:ts-json-gen command that runs all stages of rust->schemas->ts gen -fixes for some runtime errors (maps) -adding some missing json impls

Misc fixes + remove duplicate definitions

301fcee

All definitions are wiped from schemas now and all references to those local definitions are transformed into external definitions which we pass in a param to the schema compiler to not generate definitions for.

Merge branch 'master' into auto-json-to-ts

f44022f

rooooooooob marked this pull request as ready for review February 17, 2022 23:08

rooooooooob changed the title ~~[WIP] initial work on generating JSON (de)serialization + .ts types from it~~ JSON support Feb 17, 2022

rooooooooob requested review from SebastienGllmt and vsubhuman February 17, 2022 23:52

rooooooooob commented Feb 17, 2022

View reviewed changes

rooooooooob commented Feb 18, 2022

View reviewed changes

vsubhuman added this to the 10.1.0 milestone Feb 18, 2022

vsubhuman mentioned this pull request Feb 18, 2022

How can i parse the signed tx ? #378

Closed

SebastienGllmt approved these changes Feb 27, 2022

View reviewed changes

rooooooooob mentioned this pull request Mar 16, 2022

Recreate transaction body from CBOR hex of transaction? #406

Closed

SebastienGllmt mentioned this pull request Mar 31, 2022

Dcspark/auto json to ts dcSpark/cardano-multiplatform-lib#35

Merged

vsubhuman requested a review from lisicky May 13, 2022 12:23

vsubhuman mentioned this pull request May 13, 2022

Add to_json on transactions #34

Closed

vsubhuman modified the milestones: 10.1.0, 10.2.0 May 19, 2022

Merge branch 'master' into auto-json-to-ts

98fc96a

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow # rust/src/utils.rs

lisicky reviewed May 20, 2022

View reviewed changes

lisicky added 3 commits May 23, 2022 12:41

update gitignore

daf2a6a

add json support for PlutusWitness

5b72af1

update gitignore

02fa329

lisicky reviewed May 23, 2022

View reviewed changes

auto update Cargo.lock package-lock.json

c23248e

lisicky approved these changes May 25, 2022

View reviewed changes

lisicky added 3 commits June 20, 2022 18:04

Merge branch 'babbage' into auto-json-to-ts

94d8b84

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow # rust/src/crypto.rs # rust/src/lib.rs # rust/src/plutus.rs # rust/src/tx_builder.rs # rust/src/utils.rs

merge fix

cee4be0

add json support for babbage structs

183155e

lisicky reviewed Jun 21, 2022

View reviewed changes

lisicky changed the base branch from master to babbage June 21, 2022 15:19

lisicky and others added 4 commits June 23, 2022 13:34

add test

c56be38

Merge branch 'babbage' into auto-json-to-ts

c11a6cb

Merge branch 'babbage' into auto-json-to-ts

c9fbbde

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow # rust/src/lib.rs # rust/src/metadata.rs

merge fix

d3ae33d

lisicky reviewed Jul 12, 2022

View reviewed changes

Base automatically changed from babbage to master July 27, 2022 13:15

Merge branch 'babbage' into auto-json-to-ts

348a9aa

# Conflicts: # rust/src/plutus.rs

lisicky reviewed Jul 29, 2022

View reviewed changes

Merge branch 'master' into auto-json-to-ts

3782f80

lisicky reviewed Jul 29, 2022

View reviewed changes

vsubhuman added 2 commits July 29, 2022 21:20

Merge remote-tracking branch 'origin/master' into auto-json-to-ts

6869580

# Conflicts: # rust/pkg/cardano_serialization_lib.js.flow

flowgen update

3ea0c16

vsubhuman approved these changes Jul 29, 2022

View reviewed changes

vsubhuman merged commit ef546b4 into master Jul 29, 2022

vsubhuman deleted the auto-json-to-ts branch July 29, 2022 18:55

vsubhuman mentioned this pull request Sep 29, 2022

Release / 11.1.0 (Dancing Babbage) #529

Merged

14 tasks

JSON support #345

JSON support #345

Uh oh!

Conversation

rooooooooob commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rooooooooob commented Jan 26, 2022

Uh oh!

rooooooooob Jan 26, 2022

Choose a reason for hiding this comment

Uh oh!

rooooooooob Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

SebastienGllmt Feb 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rooooooooob Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

rooooooooob Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

rooooooooob Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

rooooooooob Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

SebastienGllmt Feb 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lisicky commented May 17, 2022

Uh oh!

SebastienGllmt commented May 17, 2022

Uh oh!

rooooooooob commented May 17, 2022

Uh oh!

lisicky commented May 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lisicky left a comment

Choose a reason for hiding this comment

Uh oh!

lisicky left a comment

Choose a reason for hiding this comment

Uh oh!

lisicky left a comment

Choose a reason for hiding this comment

Uh oh!

lisicky left a comment

Choose a reason for hiding this comment

Uh oh!

lisicky left a comment

Choose a reason for hiding this comment

Uh oh!

lisicky left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rooooooooob commented Jan 26, 2022 •

edited

Loading

SebastienGllmt Feb 24, 2022 •

edited

Loading

SebastienGllmt Feb 24, 2022 •

edited

Loading

lisicky commented May 20, 2022 •

edited

Loading