[PREVIEW] Adaptive frequency MRUs

[jpeletier]

Abstract

The current MRU implementation requires users to agree upon a predefined frequency and start time to start publishing updates about a certain topic. This causes lots of problems if that update frequency is not honored and requires users to know other user's update frequencies / start times in order to look up their updates on common topics

This PR removes this limitation via a novel adaptive frequency resource lookup algorithm. This algorithm automatically adjusts to the publisher's actual update frequency and converges quickly whether an update is found or not.

Users "following" a publisher automatically "tune" to the perceived frequency and can guess easily where the next update ought to be, meaning that subsequent lookups to get a newer update run faster or can be prefetched. This also allows to monitor a resource easily.

The algorithm is described below.

As a result, interactions with Swarm's MRUs are greatly simplified since the user doesn't have to come up with a start time and frequency upfront, but rather start publishing updates about the topic they want.

API changes

HTTP API

To publish an update:

1.- Get resource metainformation

• GET /bzz-resource:/?topic=<TOPIC>&user=<USER>&meta=1
• GET /bzz-resource:/<MANIFEST OR ENS NAME>/?meta=1

Where:

• user: Ethereum address of the user who publishes the resource
• topic: Resource topic, encoded as a hex string.

Note:

• If topic is omitted, it is assumed to be zero, 0x000...
• if name=<name> is provided, a subtopic is composed with that name

A common use is to omit topic and just use name, allowing for human-readable topics.

You will receive a JSON like the below:

{
  "view": {
    "topic": "0x6a61766900000000000000000000000000000000000000000000000000000000",
    "user": "0xdfa2db618eacbfe84e94a71dda2492240993c45b"
  },
  "epoch": {
    "level": 16,
    "time": 1534237239
  }
}

2.- Post the update

POST /bzz-resource:/?topic=<TOPIC>&user=<USER>&level=<LEVEL>&time=<TIME>&signature=<SIGNATURE>

body: binary stream with the update data.

To get the last update:

• GET /bzz-resource:/?topic=<TOPIC>&user=<USER>
• GET /bzz-resource:/<MANIFEST OR ENS NAME>

Note:

• Again, if topic is omitted, it is assumed to be zero, 0x000...
• if name=<name> is provided, a subtopic is composed with that name

A common use is to omit topic and just use name, allowing for human-readable topics.
Thus, this is also valid: GET /bzz-resource:/?name=profile-picture&user=<USER>

To get a previous update:

• GET /bzz-resource:/?topic=<TOPIC>&user=<USER>&time=<T>
• GET /bzz-resource:/<MANIFEST OR ENS NAME>?time=<T>

Advanced search:

If you have an idea of when the last update happened, you can also hint the lookup algorithm by adding the following extra parameters:

• hint.time: Time at when you think the last update happened
• hint.level: Approximate period you think the updates where happening at, expressed as log2(T), rounded down. For example, a resource updating every 300 seconds, level should be set to 8. log2(300) = 8.22. See the Adaptive Frequency algorithm below for details on this.

Note that this would only affect first lookups. Your swarm node will keep track of last updates and automatically use the last seen update as a hint. Using these parameters would override that automatic hint.

To publish a manifest:

POST /bzz-resource:/?topic=<TOPIC>&user=<USER>&manifest=1 with an empty body.

Note: this functionality could be moved to the client and removed from the node, since this just creates a JSON and publishes it to bzz-raw, so the client could actually create this itself and call client.UploadRaw().

CLI

Creating a resource manifest:

swarm resource create is redefined as a command line to create and publish a MRU manifest only.

swarm resource create [command options]

creates and publishes a new Mutable Resource manifest pointing to a specified user's updates about a particular topic.
          The topic can be specified directly with the --topic flag as an hex string
          If no topic is specified, the default topic (zero) will be used
          The --name flag can be used to specify subtopics with a specific name
          The --user flag allows to have this manifest refer to a user other than yourself. If not specified,
          it will then default to your local account (--bzzaccount)

OPTIONS:
--name value   User-defined name for the new resource, limited to 32 characters. If combined with topic, the resource will be a subtopic with this name
--topic value  User-defined topic this resource is tracking, hex encoded. Limited to 64 hexadecimal characters
--user value   Indicates the user who updates the resource

Update a resource

swarm resource update [command options] <0x Hex data>

creates a new update on the specified topic
          The topic can be specified directly with the --topic flag as an hex string
          If no topic is specified, the default topic (zero) will be used
          The --name flag can be used to specify subtopics with a specific name.
          If you have a manifest, you can specify it with --manifest instead of --topic / --name
          to refer to the resource

OPTIONS:
--manifest value  Refers to the resource through a manifest
--name value      User-defined name for the new resource, limited to 32 characters. If combined with topic, the resource will be a subtopic with this name
--topic value     User-defined topic this resource is tracking, hex encoded. Limited to 64 hexadecimal characters

Quick and dirty test:

In the example, the user wants to publish his/her profile picture so it can be found by anyone who knows his/her Ethereum address.

# (OPTIONAL) Publish a manifest:
swarm --bzzaccount "your public key" resource create --name "profile-picture"

# the above command will output a manifest hash. We will refer to it as $MH later on

# Publish the first update:
$ IMAGE=$(swarm up myprofilepicture.jpg) && swarm --bzzaccount "<YOUR ADDRESS>" resource update --name "profile-picture" "0x1b20$IMAGE"

# To retrieve the latest update:
$ curl 'http://localhost:8500/bzz-resource:/?name=profile-picture&user=<YOUR ADDRESS>'

# Alternatively, if we created a manifest, we can use it:
$ IMAGE=$(swarm up myprofilepicture.jpg) && swarm --bzzaccount "your public key" resource update --manifest "$MH" "0x1b20$IMAGE"

# Your last profile picture can be viewed in:
# http://localhost:8500/bzz:/$MH
# (Only if a manifest was published)

Adaptive frequency lookup algorithm

At the core of this PR is a new lookup algorithm with the following properties:

• Does not require the user to commit to an update frequency without affecting lookup time linearly with the time difference since now and when that last update happened.
• The algorithm finishes quickly if no updates found
• Once the last update is found, subsequent lookups take less time.
• If we have a rough idea of where the last update was, we can hint the algorithm to get a faster lookup time.
• It allows time-based lookups.
• The lookup key of the next update can be predicted and monitored.

Revamping the resource frequency concept

Note: Starting with this PR, in the documentation and this text, we use the strict definition of frequency as f = 1 / T. Thus, higher frequencies mean shorter periods of time.

In this new implementation, period lengths are expressed as powers of 2. The highest frequency (shortest period, update every second) is expressed as 2⁰ = 1 second. The lowest update frequency is currently set to 2²⁵ = 33554432 seconds which equals to roughly one year.

Therefore, the frequency can be encoded as just the exponent. We call this exponent frequency level, or level for short. A higher level means a longer period and thus a smaller frequency.

Introducing Epochs

Now that we have determined a set of finite possible frequencies, we can divide time in a grid of epochs. One epoch is a concrete time range at a specific frequency level, starting at a specific point in time, called the epoch base time. Level 0 epochs have a maximum length of 2⁰ = 1 seconds. Level 3 epochs have a maximum length of 2³ = 8 seconds, etc.

To refer to a specific epoch, or epoch ID we need to know the epoch base time and the epoch level

We will use this epoch addressing scheme to derive a chunk address in which to store a particular update.

Epoch base time

To caclculate the epoch base time of any given instant in time at a particular level, we use the simple formula:

baseTime(t, level) = t & ( 0xFFFFFFFFFFFFFFFF << level )

In other words, we are dropping the level lowest significant bits of t.

Seeding algorithm

The seeding algorithm describes the process followed by the update publisher to determine in what epoch "plant" the content so it can be found (harvested) by users. The algorithm works as follows:

First updates

Any first resource update will have a level of 25.

Note: We have chosen 25 as the highest level. This is a constant in code and can be changed.

Thus, if as of writing this it is August 12th 2018 at 16:51 UTC, Unix Time is 1534092715. Therefore, the epoch base time is 1534092715 & 0xFFFFFFFFFE000000 = 1509949440

The epoch id for a first update now is therefore (1509949440, 25)

Subsequent updates

To determine the epoch in which to store a subsequent update, the publisher needs to know where they stored the previous update. This should be straightforward. However, if the publisher can't or does not want to keep track of this, it can always use the harvesting algorithm (see below) to find their last update.

The selected epoch for a subsequent update must be the epoch with the highest possible level that is not already occupied by a previous update.

Let's say that we want to update our resource 5 minutes later. The Unix Time is now 1534093015.

We calculate getBaseTime(1534093015, 25) = 1509949440.
This results in the same epoch as before (1534093015, 25). Therefore, we decrease the level and calculate again:
getBaseTime(1534093015, 24) = 1526726656

Thus, the next update will be located at (1526726656, 24)

If the publisher keeps updating the resource exactly every 5 minutes, the epoch grid will look like this:

update #1,  t=1534092715, epoch=(1509949440, 25)
update #2,  t=1534093015, epoch=(1526726656, 24)
update #3,  t=1534093315, epoch=(1526726656, 23)
update #4,  t=1534093615, epoch=(1530920960, 22)
update #5,  t=1534093915, epoch=(1533018112, 21)
update #6,  t=1534094215, epoch=(1534066688, 20)
update #7,  t=1534094515, epoch=(1534066688, 19)
update #8,  t=1534094815, epoch=(1534066688, 18)
update #9,  t=1534095115, epoch=(1534066688, 17)
update #10, t=1534095415, epoch=(1534066688, 16)
update #11, t=1534095715, epoch=(1534066688, 15)
update #12, t=1534096015, epoch=(1534083072, 14)
update #13, t=1534096315, epoch=(1534091264, 13)
update #14, t=1534096615, epoch=(1534095360, 12)
update #15, t=1534096915, epoch=(1534095360, 11)
update #16, t=1534097215, epoch=(1534096384, 10)
update #17, t=1534097515, epoch=(1534096896, 9)
update #18, t=1534097815, epoch=(1534097408, 11)
update #19, t=1534098115, epoch=(1534097408, 10)
update #20, t=1534098415, epoch=(1534097920, 9)
update #21, t=1534098715, epoch=(1534098176, 8)
update #22, t=1534099015, epoch=(1534098432, 10)
update #23, t=1534099315, epoch=(1534098944, 9)
update #24, t=1534099615, epoch=(1534099200, 8)
update #25, t=1534099915, epoch=(1534099456, 15)
update #26, t=1534100215, epoch=(1534099456, 14)
update #27, t=1534100515, epoch=(1534099456, 13)
update #28, t=1534100815, epoch=(1534099456, 12)
update #29, t=1534101115, epoch=(1534099456, 11)
update #30, t=1534101415, epoch=(1534100480, 10)

If the publisher keeps updating every 5 minutes (300s), we can expect the updates to stay around level 8-9 (2⁸ = 256 seconds, 2⁹ = 512 seconds). The publisher can however, at any time vary this update frequency or just update randomly. This does not affect the algorithm.

Here is a closer look at the converging levels further down:

Harvesting algorithm

The harvesting algorithm describes how to find the latest update of a resource. This involves looking up an epoch and walking back in time until we find the last update.

Start Epoch

To select the best starting epoch to walk our grid, we have to assume the worst case, which is that the resource was never updated after we last saw it.

If we don't know when the resource was last updated, we asume 0 as the "last time" it was updated.

We can guess a start level as the position of the nonzero bit of XOR(last base time, now) counting from the left. The bigger the difference among the two times (last update time and now), the higher the level will be as the update frequency we are estimating is lower.

If the resulting level is higher than 25, we use 25.

Walking the grid - a simple algorithm

Consider the following grid. In it we have marked in yellow where the updates have happened in the past.

All the above is unknown to the harvester. All we know is the last seen update happened at (20,2), marked in light orange. We call this the "hint". The algorithm will consider this hint but will discard it if it proves it really did not contain an update. An invalid hint can pontentially slow down the algorithm but won't stop it from finding the last update.

Now it is t=26 and we want to look for the last update. Our guess at a start level is:

XOR(20, 26) = 14 = 1110b, in which the first nonzero bit counting from the left is bit #3. Thus, our first lookup will happen at (baseTime(26,3), 3) = (24,3), shown in dark blue below:

If a lookup at (24,3) fails, we consider that there are no updates at lower levels either, since the seeding algorithm would've filled (24,3) before going down. This means there are no updates on or after t=24. Thus, our search area would be reduced to the area directly below (20,2) (green area). We restart the algorithm as if now was 23

If however, a lookup at (24,3) succeeds, then we know the last update could either be the one at (24,3) itself or may be there is a later one in the epochs below (blue area). At this point we consider that the update is in 24 <= t <= 26. We restart the algorithm with the hint set to (24,3) instead of the original one. If the lookup then fails, then the last update was indeed in (24,3)

This is how the algorithm would play out if now t=26 if the last update is in (22,1)

Lookups are, in this order:

#1 (24,3) (fails)
#2 (22,1) (succeeds, but we don't know if this would be the latest update). Go down one level to confirm.
#3 (23,0) (fails)
#4 (22,0) (fails, so we return the last found value at (22,1)

Locking on / following a resource

Once we have found the last update of a resource, we can easily calculate in what epoch will the next update appear, if the publisher actually makes it.

In figure 9 above, if the last update was found at (22,1) and now it is t=26, the next update must happen exactly at (24,3). This holds true until t=32. Beyond that point, the next update can be expected at (32,4) until t=48

Therefore, the node following the resource could sample the expected epoch, keeping in sync with the publisher.

Final notes:

Please let me know your feedback, questions and test issues while I document and clean up the code. I hope you like this feature. I am available on Gitter (@jpeletier). Enjoy!!

[zelig]

extract this in a constructor please

[jpeletier]

Done. Thanks. The constructor was already there (LookupLatest), just not used.

[zelig]

this comment is no longer correct right?

[jpeletier]

Corrected. Thanks.

[zelig]

why not use pointer to LookupParams?

[jpeletier]

lp gets modified so don't want to touch the original.
Corrected.

[zelig]

metainformationA

[jpeletier]

Fixed. Thanks.

[zelig]

ok got it

[nagydani]

I like the overall design, but I have a few questions:

1. Is the order of query string parameters fixed or they can be switched places?
2. Is the API extensible by new parameters and how the persence of unknown parameters should be handled? Are they ignored or is it an error?
3. Same about responses.
   How about adding protocol version numbers for avoiding compatibility issues? I feel that this is going to change a lot, when people actually start using them.

[jpeletier]

@nagydani, thank you for your comments.

Is the order of query string parameters fixed or they can be switched places?

The order is not fixed. Parameters work in any order.

Is the API extensible by new parameters and how the persence of unknown parameters should be handled? Are they ignored or is it an error?

Unknown parameters are ignored. No error will be thrown if you put an unknown parameter.

Same about responses.

Responses are either:

• binary content (the resource you asked for)
• a HTTP error
• a JSON. The client ignores fields in the JSON it does not recognize.

How about adding protocol version numbers for avoiding compatibility issues? I feel that this is going to change a lot, when people actually start using them.

Yes, we should have API versioning, but I guess not only in bzz-resource, but throughout. The easiest right now would be to have it as another query string parameter, e.g., ?version=1. The cleanest would be to have it as part of the path: /bzz-resource:/v1/?x=1&y=2. I think we should have a more holistic API design to have API version scheme be coherent across services.

[acud]

closing due to merge to upstream