Client-side MRU signing -- step 5/5

[jpeletier]

Note:

This PR is being reviewed in several steps. In this PR you can view the entire change. For a step by step process, please look at the following:

• Step 1 : (PR client-side mru signatures -- step 1/5 #732) Minimum set of changes for client-side MRU signatures to work + system-time based MRUs. This was the status roughly 8 days ago.
• Step 2: (PR client-side mru signatures -- step 2/5 #733) Documentation, refactors and new, more concrete unit tests for the new structures
• Step 3: (PR client-side mru signatures -- step 3/5 #734) Refactor LookupParams so that the different combinations are documented. Moved LookupParams to its own file. Reorder code and create separate files for a few structures
• Step 4 (PR client-side mru signatures -- step 4/5 #748): Final refactors. Refactor serialization for extensibility. Refactor Timestamp to a "Timestamp" struct rather than a lone uint64. More comments throughout.
• Step 5: (PR Client-side MRU signing -- step 5/5 #704) Rewrite CLI and simplify client and internal object structure according to comments to decouple resource creation from updating.

Abstract

The current implementation of MRU uses the running node's private key to sign all MRUs that are requested to that node. This is a limitation for environments in which a node is shared among different users (for example, a company-wide node) and also does not allow people using public nodes to test MRUs, Swarm's Power of the Dark Side!

This PR moves MRU signatures to the client, so anyone can create and update resources with their private/public key through any node, including support for Web3 browsers such as Mist and plug-ins like Metamask. Indeed, this would allow to update content directly from your browser!

This PR's continues @nolash's work that detaches ENS for authorization. WARNING: merging this PR will automatically merge @nolash's #743 !!

Additionally, this PR adds the following:

• Fully removes ENS dependencies, such as having to name resources after domain names. The resource name is now a completely independent convenience user field.
• Command-line tool (swarm resource) to manage creating, updating resources and reading their metadata easily
• Code clean-up and refactoring. Segregated the metadata from updates in easily upgradeable structures
• Refactored chunk serialization/deserialization for extensibility
• Strong verification that the user updating a resource indeed owns the private key associated to it via a 2-step hash algorithm. There was a bug in the current implementation that allowed anyone to update your resource!!
• Removed blockchain dependency for timing. Update frequencies are expressed in seconds. This was implemented as an abstraction with a timestampProvider interface, so really any source of timing can be used, or time can be "frozen" for testing.
• Added support for MRUs in the client (client.go), so MRUs can be used from other golang applications, including Swarm's cli.
• Allow the user to set a start time in the future. This allows to publish a resource that won't resolve until the clock hits that time.
• Documentation update

Support for automatically signing MRUs by the node's bzzaccount private key has been removed from the server side. However, the API easily allows the node itself to use MRUs with its own private key. The swarm command line tool (client) can pick your local account from the same place, so regular users wouldn't notice where signing is actually taking place.

Code refactoring and terminology standarization

Throughout the package, the code has been refactored with the following reusable and extensible structures. To avoid confusion, the use of ambiguous terms such as addror key has been removed.

Main structures:

• resourceUpdate: encapsulates the information sent as part of a resource update: period, version, multihash flag, the data payload and a reference to the Mutable Resource's metadata chunk.
• resourceMetadata: carries the immutable information about a mutable resource: start time, frequency, name and owner.
• resource: resourceUpdate + resourceMetadata. Caches resource data. When synced it contains the most recent version of the resource data and the metadata of its root chunk
• SignedResourceUpdate: resourceUpdate + signature
• UpdateRequest message: SignedResourceUpdate + resourceMetadata. Serializable structure used to issue Mutable Resource creation and update messages.

Terms:

• ownerAddr: Address of the resource owner. Type: storage.Address
• metaHash is the SHA3 hash of the encoded resourceMetadata, excluding ownerAddr
• rootAddr: this is the key of the chunk that contains the Mutable Resource metadata. Calculated as the SHA3 hash of ownerAddr and metaHash. Type: storage.Address
• updateAddr: this is the key of the chunk that contains the Mutable Resource update. Calculated as the SHA3 hash of period, version and rootAddr. Type: storage.Address

Chunk Validator refactor

The chunk validator (Validate method) has been rewritten to:

• Validate medatata chunks, whose key is content-addressed not as a direct Swarm hash of the content, but rather as SHA3(ownerAddr, metaHash) where metaHash = SHA3(resource name, frequency, startTime).
• Validate update chunks verifying signatures and proof of ownership of the resource to be updated without having to lookup/download the referenced metadata chunk.
  • Ownership is proven if the following holds true rootAddr == SHA3(signature_address, metaHash), where signature_address is the recovered address from the signature of the update chunk digest; digest = SHA3(updateAddr, metaHash, data), where updateAddr is the update chunk address, updateAddr = SHA3(period, version, rootAddr).

HTTP API

The HTTP API for resources (/bzz-resource:/) now works as follows:

Resource creation:

POST /bzz-resource:/ with the following JSON as payload:

{
  "name": string,
  "frequency": number,
  "startTime": number,
  "rootAddr" : hex string,
  "data": hex string,
  "multihash": boolean,
  "period": number,
  "version": number,
  "signature": hex string
}

where:

• name Resource name. This is a user field. You can use any name
• frequency Time interval the resource is expected to update at, in seconds.
• startTime Time the resource is valid from, in Unix time (seconds).
• ownerAddr is the address derived from your public key. Hex encoded.
• multihash is a flag indicating whether the data field should be interpreted as raw data or a multihash
• data contains hex-encoded raw data or a multihash of the content the mutable resource will be initialized with
• period indicates for what period we are signing. Set to 1 for creation.
• version indicates what resource version of the period we are signing. Must be set to 1 for creation.
• signature Signature of the digest calculated as follows digest = H(H(period, version, rootAddr), metaHash, data). Hex encoded.

Returns: MRU Manifest Key, as a quoted string. As before, this key can be put in an ENS Resolver contract with setContent.
The MRU Manifest Key resolves to a manifest that points to the metadata root chunk (rootAddr)

To update a resource:

Step 1:

either
GET /bzz-resource:/<MRU_MANIFEST_KEY>/meta
-or-
GET /bzz-resource:/<DOMAIN THAT POINTS TO MRU_MANIFEST_KEY>/meta

This produces a response like this:

{
  "name": "myresource",
  "frequency": 300,
  "startTime": 1528722352,
  "ownerAddr": "0x7a2e393025c567ec4089d34f393ae6b5c234536a",
  "rootAddr": "0x0c5acf8e3176900bc5b7b732261b5ebafc05a56da3b01c044214408e841f4ded",
  "metaHash": "0x0d6968416844245d605b5e4205858e9a4779936fc3a279dce2d38415de91d79f",
  "version": 1,
  "period": 8,
  "multiHash": true
}

where:

• Metadata Fields
  • name: filled with the original value used to create the resource
  • frequency: filled with the original value used to create the resource
  • startTime: filled with the original value used to create the resource
  • ownerAddr: filled with the original value used to create the resource
  • metaHash Metadata hash, calculated as metaHash=H(len(metadata), startTime, frequency,name). Hex encoded
  • rootAddr Metadata Root Chunk Address, calculated as rootAddr = H(metaHash, ownerAddr). This scheme effectively locks the root chunk so that only the owner of the private key that ownerAddr was derived from can sign updates.
• Information about current update
  • version: filled with the version number of the MRU that must be used for the next update to be successful.
  • period: filled with the period number of the MRU that must be used for the next update to be successful
  • multihash: whether the current data pointed to by the resource is multihash

Step 2:

To update the resource, create a new flat JSON with the following fields:

• data: new data you want to set
• multihash: whether the new data should be considered a multihash
• period: use the suggested period from Step 1.
• version: use the suggested version from Step 1.
• signature, calculated in the same way as explained for resource creation.

Then, POST the resulting JSON to:
POST /bzz-resource:/

do not change version/period values, since this will cause your update to be rejected.

Phew! That looks like a lot of stuff! But don't worry, we got you covered with API and tools. See below.

Reading a resource

This is unchanged, you can still use as before:

• GET /bzz-resource:/<MRU_MANIFEST_KEY>/
• GET /bzz-resource://<MRU_MANIFEST_KEY> - get latest update
• GET /bzz-resource://<MRU_MANIFEST_KEY>/<n> - get latest update on period n
• GET /bzz-resource://<MRU_MANIFEST_KEY>/<n>/<m> - get update version m of period n

Go API

Swarm client (package swarm/api/client) has been updated with the following new methods:

• CreateResource(name string, frequency, startTime uint64, data []byte, multihash bool, signer mru.Signer)
  • CreateResource creates a Mutable Resource with the given name and frequency, initializing it with the provided data. Data is interpreted as multihash or not depending on the multihash parameter.
    • name: human-readable name for your resource.
    • startTime: when the resource starts to be valid. 0 means "now". Unix time in seconds.
    • data: initial data the resource will contain.
    • multihash: whether to interpret data as multihash
    • signer: Signer object containing the Sign callback function
    • Returns the resulting Mutable Resource manifest address that you can use to include in an ENS Resolver (setContent) or reference future updates (Client.UpdateResource)
• UpdateResource(manifestAddressOrDomain string, data []byte, signer mru.Signer)
  • UpdateResource allows you to send a new version of your content
    • manifestAddressOrDomain is the address you obtained in CreateResource or an ENS domain whose Resolver points to that address.
    • data: new data to update the resource to.
    • signer: Signer object containing the Sign callback function
• GetResource(manifestAddressOrDomain string) (io.ReadCloser, error)
  • Returns the latest data currently contained in the resource as an octect stream. TODO: allow to get specific period/version
• GetResourceMetadata(manifestAddressOrDomain string) (*mru.UpdateRequest, error)
  • Returns an UpdateRequest object that describes the resource and can be used to construct an update.

Command line tools:

In this first version, the swarm cli interface has been updated with a new (advanced) command resource to manage resources, with this syntax:

Create:

swarm --bzzaccount="<account>" resource create [--raw] <name> <frequency> <0x hex data>
If successful, this command will output a Manifest Address that can be used in ENS's setContent or directly in your browser via http://localhost:8500/bzz:/

Update:

swarm --bzzaccount="<account>" resource [--raw] update <Manifest Address or ENS domain> <0x Hexdata>

the --raw flag sets multihash to false. This means that by default the data is considered to be a multihash. If you want to use the output of swarm up, prefix it with 0x1b20 to indicate a keccak256 hash.

Quick and dirty test:

$ MH=$(swarm up file.jpg) && swarm --bzzaccount "your public key" resource create myresource 300 0x1b20${MH}

This uploads file.jpg and creates a resource that is expected to update every 5 minutes.

Final notes:

Please let me know your feedback, questions and test issues while I document and clean up the code. I hope you like this feature. I am available on Gitter (@jpeletier). Enjoy!!

PR Review guide:

Client-side MRU signing.pdf

[zelig]

@jpeletier LGTM but lets wait for thorough review and have further discussion

[holisticode]

This flag is a bit too little descriptive for my taste. In the context of swarm, there's a lot which could be raw. Maybe something which better connects it to MRU?

[jpeletier]

I agree.

Any ideas?
maybe --verbatim

[holisticode]

and/or prefix with MRU (whatever case feels better appropriate)?

[jpeletier]

I'll go for --rawmru

[jpeletier]

fixed 464798

[holisticode]

this should probably be removed

[jpeletier]

do you mean the error or the fmt.Println? the latter is so that the user gets the manifest address and can take it to ENS or use it in the browser

[holisticode]

yeah I meant the fmt.Println. I see the reason now. I interpreted it as it would be just some debugging output. Please maybe add a comment describing this. Also, consider adding some info for the user in the output (only if necessary).

[jpeletier]

fixed 464798

[holisticode]

It's difficult for me to review this PR as I am not clear what is being added to @nolash's #668. Shouldn't this be a PR on that one instead of swarm-network-rewrite?

[holisticode]

Please also check, there are conflicts in this PR

[nolash]

I seem to have lost all comments on the previous review I made. Submitting to see if they've been invisibly registered.

[nolash]

should be newAccountManager

[nolash]

It strikes me as confusing to have all params for all subcommands on the same flag help page.

openssl for example has contextual help for every subcommand; openssl asn1parse --help, openssl s_client --help etc. Maybe we should do the same.

It would be out of scope for this PR, but I feel this addition highlights the problem, given MRU's relative obscurity to the swarm core functionality.

[nolash]

Should also have --rawmru flag?

[nolash]

Should be proper "usage" output?

[nolash]

should this autodetect hex, and if it is not then treat as string?

[nolash]

we can't use the pointer?

[jpeletier]

If you don't copy the data, then the caller who passed r could alter your index contents from outside.

[nolash]

Right so we assume SignedResourceUpdates can be reused?

[jpeletier]

Maybe not SignedResourceUpdate, but the data slice put there could come from anywhere within the node.

[nolash]

Ok. Better safe than sorry, I guess.

[nolash]

Maybe the 1 should be flagsLength?

[jpeletier]

That 1 actually refers to the uint8 that helps encode the length of the resource name. I have improved the comments around it.

[nolash]

Maybe we don't need mru.Timestamp anymore now?

[jpeletier]

I'd rather leave it. I know it just contains an uint64 inside, but it makes things very easy to extend. For example, I was able to add/remove the proof thing, including serialization in under 10 mins with all test passing on the first attempt, without errors.

[nolash]

ok

[nolash]

setDirty?

[jpeletier]

I inlined this method since it was only used by setData.

[nolash]

Please see further up for comments on this (my review comments got hidden since I intermittently went offline..)

[nolash]

...fortunately they were, so I'll continue.

[nolash]

MarshalJSON?

[jpeletier]

I already renamed this to fomJSON in a prior review. In any case, this is a private method.

[nolash]

don't need anymore, right?

[jpeletier]

I'd rather keep the structure for the reasons stated above, makes it very easy to extend or modify in the future + a place for the serializers to be.