State Expiry + Online Pruning #16

patrick-ogrady · 2022-01-06T04:59:57Z

https://docs.google.com/document/d/1kK6J0BkogmUuwr47pseNPN5YPWtA-WSx40LhLQRa27k/edit

t=2022-01-06T00:09:20-0800 lvl=dbug msg="prefix expired" prefix=foo
t=2022-01-06T00:09:20-0800 lvl=dbug msg="prefix expired" prefix=foo
t=2022-01-06T00:09:20-0800 lvl=dbug msg="prefix expired" prefix=foo
t=2022-01-06T00:09:20-0800 lvl=dbug msg="rprefix pruned" rprefix=e6f0014586d469e2a11f494a93cabdab0f69f612
t=2022-01-06T00:09:20-0800 lvl=dbug msg="rprefix pruned" rprefix=1900832420acc0fed9673c86e45a3ce325b280a8
t=2022-01-06T00:09:20-0800 lvl=dbug msg="rprefix pruned" rprefix=c13e3f783aec2c16df329fc6780871d2e27aed43

chain/storage_test.go

patrick-ogrady · 2022-01-06T05:41:28Z

chain/storage.go

+	}
+	if !exists {
+		return ErrPrefixMissing
+	}


patrick-ogrady · 2022-01-06T05:41:38Z

chain/storage.go

+	}
+	if !exists {
+		return ErrPrefixMissing
+	}


patrick-ogrady · 2022-01-06T05:41:57Z

chain/storage.go

+	}
+	if !exists {
+		return nil, ErrPrefixMissing
+	}


Signed-off-by: Gyuho Lee <[email protected]>

gyuho

lgtm, added some docs

gyuho · 2022-01-06T17:25:34Z

chain/storage.go

+}

-	k = make([]byte, n)
+// Assumes [prefix] and [key] do not contain delimiter


nice simplification :)

gyuho · 2022-01-06T18:05:13Z

chain/storage_test.go

 			key:  []byte("hello"),
 			opts: []OpOption{WithPrefix()},
 			kvs: []KeyValue{
-				{Key: []byte("foo/hello00000"), Value: []byte("bar00000")},


gyuho · 2022-01-06T18:09:05Z

vm/vm.go

 	vm.stopc = make(chan struct{})
 	vm.donecRun = make(chan struct{})
 	vm.donecRegossip = make(chan struct{})
+	vm.donecPrune = make(chan struct{})


Is this being used for wait for prune done?

Added wait on this channel in shutdown.

gyuho · 2022-01-06T18:22:11Z

tests/integration/integration_test.go

 			kvs, err := instances[0].cli.Range(pfx, k)
 			gomega.Ω(err).To(gomega.BeNil())
-			gomega.Ω(kvs[0].Key).To(gomega.Equal(append(append(pfx, parser.Delimiter), k...)))
+			gomega.Ω(kvs[0].Key).To(gomega.Equal(k))


+1, much cleaner to expect key with prefix removed.

gyuho · 2022-01-06T18:34:40Z

tests/e2e/e2e_test.go

 				kvs, err := inst.cli.Range(pfx, k)
 				gomega.Ω(err).To(gomega.BeNil())
-				gomega.Ω(kvs[0].Key).To(gomega.Equal(append(append(pfx, parser.Delimiter), k...)))
+				gomega.Ω(kvs[0].Key).To(gomega.Equal(k))


Signed-off-by: Gyuho Lee <[email protected]>

patrick-ogrady · 2022-01-06T19:34:24Z

chain/storage.go


 func PutPrefixInfo(db database.KeyValueWriter, prefix []byte, i *PrefixInfo, lastExpiry int64) error {
-	if i.RawPrefix == (ids.ShortID{}) {
+	if i.RawPrefix == ids.ShortEmpty {


patrick-ogrady · 2022-01-06T20:16:09Z

chain/storage.go

 			return err
 		}
-		expiry := int64(binary.LittleEndian.Uint64(curKey[2 : 2+8]))
+		expired := int64(binary.LittleEndian.Uint64(curKey[2 : 2+8]))


I think it may make sense to extract these into a key parsing function

Signed-off-by: Gyuho Lee <[email protected]>

patrick-ogrady added 7 commits January 2, 2022 16:34

first strokes

7fdae38

more scoping

8653b7f

add additional todo

e4e6b3e

cleanup underway

2450c1a

more progress

6e0a24c

more progress

eab928c

everything compiles

5679700

patrick-ogrady marked this pull request as draft January 6, 2022 05:00

patrick-ogrady commented Jan 6, 2022

View reviewed changes

chain/storage_test.go Outdated Show resolved Hide resolved

patrick-ogrady commented Jan 6, 2022

View reviewed changes

chain/storage_test.go Outdated Show resolved Hide resolved

patrick-ogrady commented Jan 6, 2022

View reviewed changes

chain/storage_test.go Outdated Show resolved Hide resolved

fix integration test

a788ec3

patrick-ogrady force-pushed the state-expiry branch from eefbc42 to 1c57ea9 Compare January 6, 2022 05:12

cleanup

be531ae

patrick-ogrady force-pushed the state-expiry branch from 1c57ea9 to be531ae Compare January 6, 2022 05:15

patrick-ogrady added 3 commits January 5, 2022 21:24

fix hash issue

9a54fbe

automatically generate raw prefix in storage

25006bf

revert auto hash generation

4b78fa1

patrick-ogrady force-pushed the state-expiry branch from d08083f to 4b78fa1 Compare January 6, 2022 05:39

patrick-ogrady commented Jan 6, 2022

View reviewed changes

chain/storage.go

}

if !exists {

return ErrPrefixMissing

}

Copy link

Contributor Author

patrick-ogrady Jan 6, 2022 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: \n

patrick-ogrady commented Jan 6, 2022

View reviewed changes

chain/storage.go

}

if !exists {

return ErrPrefixMissing

}

Copy link

Contributor Author

patrick-ogrady Jan 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: \n

patrick-ogrady commented Jan 6, 2022

View reviewed changes

chain/storage.go

}

if !exists {

return nil, ErrPrefixMissing

}

Copy link

Contributor Author

patrick-ogrady Jan 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: \n

patrick-ogrady added 8 commits January 5, 2022 22:33

first pass on expiry keys

84fc802

jank pruning

0e34537

expiring tests

efbb435

fix tests

e98e50c

add context locking

aeb1821

add logging

b8e57b4

cleanup db at end of test

14ee27e

fix rprefix gen

085be1e

fix init issue

ada6ee8

patrick-ogrady force-pushed the state-expiry branch 2 times, most recently from fb506fe to ada6ee8 Compare January 6, 2022 08:32

patrick-ogrady added 4 commits January 6, 2022 00:40

fix pruning correctness

dd2169d

fix difficulty estimate

81a81e7

working again

1296705

fix underflow

2dad05e

patrick-ogrady changed the title ~~State expiry~~ State expiry + Pruning Jan 6, 2022

patrick-ogrady changed the title ~~State expiry + Pruning~~ State Expiry + Online Pruning Jan 6, 2022

patrick-ogrady requested a review from gyuho January 6, 2022 09:16

gyuho added 4 commits January 6, 2022 10:22

vm: missing "<-donecPrune"

ad5a558

Signed-off-by: Gyuho Lee <[email protected]>

chain: use "ids.ShortEmpty"

ecfb929

Signed-off-by: Gyuho Lee <[email protected]>

chain: add "PutPrefixInfo/Key" tests

bf70964

Signed-off-by: Gyuho Lee <[email protected]>

chain: document new storage functions

a16f187

Signed-off-by: Gyuho Lee <[email protected]>

gyuho reviewed Jan 6, 2022

View reviewed changes

chain: fix error checks

d0cfc38

Signed-off-by: Gyuho Lee <[email protected]>

gyuho marked this pull request as ready for review January 6, 2022 19:28

patrick-ogrady commented Jan 6, 2022

View reviewed changes

chain: define "extractSpecificTimeKey"

f2e68b4

Signed-off-by: Gyuho Lee <[email protected]>

gyuho force-pushed the state-expiry branch from 4718939 to f2e68b4 Compare January 6, 2022 20:59

gyuho approved these changes Jan 6, 2022

View reviewed changes

patrick-ogrady merged commit abf66bb into master Jan 6, 2022

gyuho deleted the state-expiry branch January 7, 2022 00:43

State Expiry + Online Pruning #16

State Expiry + Online Pruning #16

Uh oh!

Conversation

patrick-ogrady commented Jan 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrick-ogrady Jan 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gyuho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

patrick-ogrady commented Jan 6, 2022 •

edited

Loading

patrick-ogrady Jan 6, 2022 •

edited

Loading