Secret Store, Intermittent hung up on Key generation session

[zean00]

• Parity Ethereum version: 2.5.7
• Operating system: Linux (docker)
• Installation: built from source (with secret store)
• Fully synchronized: no
• Network: private / aura / secret store
• Restarted: yes
• Confguration: 3 nodes with aura consensus (docker compose)

Secret store frequently failed to generate server key session and just hung up (resulting in request timeout), need to restart all secret store nodes to recover

Expected Behaviour
Key generation success, if error occurred should return error message immediately and should not causing request timeout.

Actual
I rarely have successful request of key generation in 10 request sequentially, one hang up, I need to restart all secret store node to be able to get expected response from secret store. This issue is not occurred in ocean protocol fork (I'm using their docker images of parity fork with secret store - oceanprotocol/parity-ethereum:master)

Step to Reproduce
Invoke POST /shadow/[DOCUMENT_ID]/[SIGNED_DOCUMENT_ID]/1

Notes
This is not specific on key generation session, sometimes this happening on document key store request or document key retrieval. I suspect this is related to cluster communication / coordination

Configuration

parity]
chain = "/home/parity/chain.json"

[account]
password = ["/home/parity/password"]

[network]
reserved_peers = "/home/parity/reserved_peers"

[rpc]
interface = "0.0.0.0"
cors = ["all"]
hosts = ["all"]
apis = ["all"]
server_threads = 8
processing_threads = 0

[mining]
reseal_on_txs = "none"
engine_signer = "0x59f6566306843fb7d18f782733e19223ca5f07ab"
force_sealing = true
#reseal_max_period = 300000
tx_queue_no_early_reject = true
tx_queue_mem_limit = 0
tx_queue_per_sender = 8192
tx_queue_size = 32768
gas_floor_target = "40000000"

[websockets]
disable = false
port = 8546
interface = "all"
origins = ["http://127.0.0.1:8180"]
apis = ["all"]
hosts = ["all"]

[secretstore]
self_secret = "59f6566306843fb7d18f782733e19223ca5f07ab" 
disable = false
disable_http = false      # This node will expose a Secret Store HTTP API
http_interface = "all"  # The HTTP API is available locally only
http_port = 8010          # The Secret Store HTTP API is available at this port
acl_contract = "289631594ec63a592aed277Ef6Dce46937959A7e"     # Insecure, will be changed later on
#acl_contract = "none"
server_set_contract = "none"
nodes = ["40eadf91ffa4cae66491316f157604fd4ff68589143b74a7879f83b135e35ebceddb383adb62f0f2a29aab39cc9bd496f9fbbeda737a054e8c90d371d98689fd@10.1.0.101:8011","1fef86537f848e8fce58596aa24511f3a4b9df68919b4a2155e607132a8eb10c0c199d050faa36194325c000ed97ecd2c433e00712ca44e2b21393fb216e19be@10.1.0.102:8011","7aead779d8027d73a3f23378712945a01a88bd23bc32679017e7dcd2c670895e6f1710881b7c658b8d1af76c4d4cbc9fcdf86f32e9db062ea1799de16aa2bc8a@10.1.0.103:8011"]
interface = "all"
port = 8011               # port used to communicated with other Secret Store n0des
path = "/parity/config/secret/db.ss1"

[misc]
logging = "secretstore=trace,secretstore_net=trace"

Log

Node1 40eadf91ffa4cae66491316f157604fd4ff68589143b74a7879f83b135e35ebceddb383adb62f0f2a29aab39cc9bd496f9fbbeda737a054e8c90d371d98689fd

2019-10-08 06:01:32 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: sent message Generation.InitializeSession to 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Generation.ConfirmInitialization from 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: sent message Generation.InitializeSession to 0x7aea…bc8a
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Generation.ConfirmInitialization from 0x7aea…bc8a
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.CompleteInitialization to 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.CompleteInitialization to 0x7aea…bc8a
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.KeysDissemination to 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.KeysDissemination to 0x7aea…bc8a
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Generation.KeysDissemination from 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Generation.PublicKeyShare from 0x1fef…19be
2019-10-08 06:01:42 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-08 06:01:52 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-08 06:01:57 UTC IO Worker #3 INFO import     2/25 peers     51 KiB chain   21 KiB db  0 bytes queue    3 KiB sync  RPC:  3 conn,    0 req/s,  533 µs
2019-10-08 06:02:02 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-08 06:02:12 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-08 06:02:12 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAliveResponse from 0x1fef…19be

Node2 1fef86537f848e8fce58596aa24511f3a4b9df68919b4a2155e607132a8eb10c0c199d050faa36194325c000ed97ecd2c433e00712ca44e2b21393fb216e19be

2019-10-08 06:01:32 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Generation.InitializeSession from 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: sent message Generation.ConfirmInitialization to 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Generation.CompleteInitialization from 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: sent message Generation.KeysDissemination to 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: sent message Generation.KeysDissemination to 0x7aea…bc8a
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Generation.KeysDissemination from 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Generation.KeysDissemination from 0x7aea…bc8a
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: sent message Generation.PublicKeyShare to 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: sent message Generation.PublicKeyShare to 0x7aea…bc8a
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Generation.PublicKeyShare from 0x7aea…bc8a
2019-10-08 06:01:42 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-08 06:01:52 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-08 06:01:52 UTC IO Worker #2 INFO import     2/25 peers     17 KiB chain   18 KiB db  0 bytes queue    2 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-08 06:02:02 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-08 06:02:12 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x7aea…bc8a
2019-10-08 06:02:12 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x40ea…89fd
2019-10-08 06:02:12 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-08 06:02:12 UTC tokio-runtime-worker-1 WARN secretstore_net  0x1fef…19be: network error 'early eof' when reading message from node 0x7aea…bc8a
2019-10-08 06:02:12 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: removing connection to 0x7aea…bc8a at 10.1.0.103:8011
2019-10-08 06:02:18 UTC IO Worker #1 INFO import  Imported #9 0x9a64…19c9 (0 txs, 0.00 Mgas, 1 ms, 1.30 KiB)
2019-10-08 06:02:22 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-08 06:02:27 UTC tokio-runtime-worker-2 WARN secretstore_net  0x1fef…19be: timeout when establishing outbound connection with 10.1.0.103:8011
2019-10-08 06:02:27 UTC IO Worker #2 INFO import     1/25 peers     41 KiB chain   18 KiB db  0 bytes queue    2 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-08 06:02:32 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-08 06:02:37 UTC tokio-runtime-worker-1 WARN secretstore_net  0x1fef…19be: timeout when establishing outbound connection with 10.1.0.103:8011

Node3 7aead779d8027d73a3f23378712945a01a88bd23bc32679017e7dcd2c670895e6f1710881b7c658b8d1af76c4d4cbc9fcdf86f32e9db062ea1799de16aa2bc8a

2019-10-08 06:01:32 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-08 06:01:32 UTC IO Worker #1 INFO import     2/25 peers     51 KiB chain   22 KiB db  0 bytes queue    2 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.InitializeSession from 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.ConfirmInitialization to 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.CompleteInitialization from 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x40ea…89fd
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.PublicKeyShare from 0x1fef…19be
2019-10-08 06:01:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x40ea…89fd
2019-10-08 06:01:42 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-08 06:01:52 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-08 06:02:02 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-08 06:02:02 UTC IO Worker #2 INFO import     2/25 peers     51 KiB chain   22 KiB db  0 bytes queue    2 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-08 06:02:12 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-08 06:02:12 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAlive from 0x40ea…89fd


====================

stack backtrace:
   0:     0x557cc09e2abd - <no info>
   1:     0x557cc09dfdd2 - <no info>
   2:     0x557cc0c89387 - <no info>
   3:     0x557cc0a3d674 - <no info>
   4:     0x557cc0a3d3ee - <no info>
   5:     0x557cc0a3e755 - <no info>
   6:     0x557cc0a4c43b - <no info>
   7:     0x557cc0a4dd85 - <no info>
   8:     0x557cbfa8b7f8 - <no info>
   9:     0x557cc08d244f - <no info>
  10:     0x557cc08d07d2 - <no info>
  11:     0x557cc08cf160 - <no info>
  12:     0x557cc08ba041 - <no info>
  13:     0x557cc08d3b8b - <no info>
  14:     0x557cc08d16c2 - <no info>
  15:     0x557cc0a3baae - <no info>
  16:     0x557cc0a3b98b - <no info>
  17:     0x7f41818acbf0 - <no info>

Thread 'tokio-runtime-worker-2' panicked at 'supplied instant is later than self', src/libcore/option.rs:1034

This is a bug. Please report it at:

    https://github.com/paritytech/parity-ethereum/issues/new


2019-10-08 06:02:12 UTC main INFO parity_ethereum::run  Finishing work, please wait...
2019-10-08 06:02:12 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-08 06:02:12 UTC main WARN shutdown  unable to get mut ref for engine for shutdown.

[niklasad1]

Hey,

That error is related to std::time::Instant::duration_since which is may because of a faulty clock or bug in devp2p I guess.

/cc @svyatonik @grbIzl any clues?

[svyatonik]

@zean00 Could you, please:

1. try with try_ss_instant_fix branch which I have forked from master just now or cherry-pick f1055ead9fcdf151c14f749195212ace6b1b8aad into your branch?
2. if issue remains, could you try to reproduce it with debug build? Specifically - I'm interested in that panic backtrace.

[zean00]

Here is what I have after applying try_ss_instant_fix patch. I didn't get panic backtrace this time

2019-10-10 02:23:54 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: sent message Generation.InitializeSession to 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Generation.ConfirmInitialization from 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.InitializeSession to 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Generation.ConfirmInitialization from 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.CompleteInitialization to 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.CompleteInitialization to 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.KeysDissemination to 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: sent message Generation.KeysDissemination to 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Generation.KeysDissemination from 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: received message Generation.KeysDissemination from 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: sent message Generation.PublicKeyShare to 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: sent message Generation.PublicKeyShare to 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: received message Generation.PublicKeyShare from 0x7aea…bc8a
2019-10-10 02:23:54 UTC IO Worker #2 INFO import     2/25 peers     53 KiB chain   32 KiB db  0 bytes queue    9 KiB sync  RPC:  3 conn,    7 req/s, 1843 µs
2019-10-10 02:23:59 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:24:09 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:24:19 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:24:24 UTC IO Worker #2 INFO import     2/25 peers     53 KiB chain   32 KiB db  0 bytes queue    9 KiB sync  RPC:  3 conn,    0 req/s, 1843 µs
2019-10-10 02:24:26 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAlive from 0x7aea…bc8a
2019-10-10 02:24:29 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:24:39 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:24:49 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:24:59 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:24:59 UTC tokio-runtime-worker-2 WARN ethcore_secretstore::key_server_cluster::client_sessions::generation_session  0x40ea…89fd: generation session failed with timeout
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 WARN secretstore  GenerateServerKey request /shadow/b0c42c1c1fc30aa0cbd445d07dbaabd4b6360c506ef345bf52b84a1f6908c0a4/d7b5350c5af2447eb199f7552ef61fd26b0681c998e441d5cda2d7fd1776bfbb04228cbdb82375d0c0b2ad976d8834a9cafe9bfd297108a06e000de0c72d50c000/1 has failed with: node required for this operation is currently disconnected
2019-10-10 02:24:59 UTC tokio-runtime-worker-2 WARN secretstore_net  0x40ea…89fd: keep alive timeout for node 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Generation.PublicKeyShare from 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 WARN secretstore_net  0x40ea…89fd: generation session read error 'no active session with given id' when requested for session from node 0x1fef…19be
2019-10-10 02:24:59 UTC IO Worker #0 INFO import     2/25 peers     53 KiB chain   32 KiB db  0 bytes queue    9 KiB sync  RPC:  3 conn,    0 req/s, 1843 µs
2019-10-10 02:24:59 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAliveResponse from 0x7aea…bc8a
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAlive from 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:25:09 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:25:09 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:25:09 UTC tokio-runtime-worker-0 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:25:09 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:25:19 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:25:19 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:25:19 UTC tokio-runtime-worker-0 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:25:19 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:25:29 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:25:29 UTC IO Worker #1 INFO import     2/25 peers     53 KiB chain   32 KiB db  0 bytes queue    9 KiB sync  RPC:  3 conn,    0 req/s, 1843 µs
2019-10-10 02:25:29 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:25:29 UTC tokio-runtime-worker-0 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:25:29 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:25:30 UTC IO Worker #2 INFO import  Imported #14 0x05a7…f082 (0 txs, 0.00 Mgas, 250 ms, 1.30 KiB)
2019-10-10 02:25:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAlive from 0x7aea…bc8a
2019-10-10 02:25:37 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAlive from 0x1fef…19be
2019-10-10 02:25:39 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:25:39 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:25:39 UTC tokio-runtime-worker-2 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:25:39 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:25:49 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:25:49 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:25:49 UTC tokio-runtime-worker-0 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:25:49 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:25:59 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:25:59 UTC IO Worker #2 INFO import     2/25 peers     54 KiB chain   32 KiB db  0 bytes queue    9 KiB sync  RPC:  3 conn,    0 req/s, 1843 µs
2019-10-10 02:25:59 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:25:59 UTC tokio-runtime-worker-1 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:25:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:26:09 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:26:09 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAliveResponse from 0x7aea…bc8a
2019-10-10 02:26:09 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:26:09 UTC tokio-runtime-worker-2 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:26:09 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:26:17 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAlive from 0x1fef…19be
2019-10-10 02:26:19 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:26:19 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:26:19 UTC tokio-runtime-worker-1 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:26:19 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:26:29 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:26:29 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:26:29 UTC tokio-runtime-worker-0 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:26:29 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:26:29 UTC IO Worker #2 INFO import     2/25 peers     54 KiB chain   32 KiB db  0 bytes queue    9 KiB sync  RPC:  3 conn,    0 req/s, 1843 µs
2019-10-10 02:26:39 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: executing maintain procedures
2019-10-10 02:26:39 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: inserting connection to 0x1fef…19be at 10.1.0.102:8011. Connected to 2 of 2 nodes
2019-10-10 02:26:39 UTC tokio-runtime-worker-1 WARN secretstore_net  0x40ea…89fd: network error 'early eof' when reading message from node 0x1fef…19be
2019-10-10 02:26:39 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x40ea…89fd: removing connection to 0x1fef…19be at 10.1.0.102:8011
2019-10-10 02:26:46 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x40ea…89fd: received message Cluster.KeepAlive from 0x7aea…bc8a


2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Generation.InitializeSession from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: sent message Generation.ConfirmInitialization to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Generation.CompleteInitialization from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: sent message Generation.KeysDissemination to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Generation.KeysDissemination from 0x7aea…bc8a
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: sent message Generation.KeysDissemination to 0x7aea…bc8a
2019-10-10 02:23:47 UTC IO Worker #3 INFO import  Syncing      #13 0xfd4d…2fac     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed       #13    2/25 peers     34 KiB chain   33 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Generation.KeysDissemination from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Generation.PublicKeyShare from 0x7aea…bc8a
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: sent message Generation.PublicKeyShare to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: sent message Generation.PublicKeyShare to 0x7aea…bc8a
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Generation.PublicKeyShare from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Generation.SessionCompleted from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: sent message Generation.SessionCompleted to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 INFO secretstore_net  0x1fef…19be: generation session completed
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Encryption.InitializeEncryptionSession from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: sent message Encryption.ConfirmEncryptionInitialization to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 INFO secretstore_net  0x1fef…19be: encryption session completed
2019-10-10 02:23:54 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Generation.InitializeSession from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: sent message Generation.ConfirmInitialization to 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Generation.KeysDissemination from 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Generation.CompleteInitialization from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: sent message Generation.KeysDissemination to 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: sent message Generation.KeysDissemination to 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Generation.KeysDissemination from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: sent message Generation.PublicKeyShare to 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: sent message Generation.PublicKeyShare to 0x7aea…bc8a
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Generation.PublicKeyShare from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Generation.PublicKeyShare from 0x7aea…bc8a
2019-10-10 02:23:54 UTC IO Worker #1 INFO import  Imported #14 0xc276…2666 (1 txs, 0.18 Mgas, 85 ms, 0.88 KiB)
2019-10-10 02:23:57 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:24:07 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:24:17 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:24:17 UTC IO Worker #0 INFO import     2/25 peers     36 KiB chain   35 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:24:26 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x7aea…bc8a
2019-10-10 02:24:27 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:24:29 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x40ea…89fd
2019-10-10 02:24:37 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:24:39 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x40ea…89fd
2019-10-10 02:24:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:24:47 UTC IO Worker #2 INFO import     2/25 peers     36 KiB chain   35 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:24:49 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x40ea…89fd
2019-10-10 02:24:57 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:24:57 UTC tokio-runtime-worker-0 WARN ethcore_secretstore::key_server_cluster::client_sessions::generation_session  0x1fef…19be: generation session failed with timeout
2019-10-10 02:24:57 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAliveResponse from 0x7aea…bc8a
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Generation.SessionError(no active session with given id) from 0x40ea…89fd
2019-10-10 02:24:59 UTC tokio-runtime-worker-1 WARN secretstore_net  0x1fef…19be: generation session read error 'no active session with given id' when requested for session from node 0x40ea…89fd
2019-10-10 02:24:59 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:25:07 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:25:17 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:25:17 UTC IO Worker #2 INFO import     2/25 peers     36 KiB chain   35 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:25:27 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:25:33 UTC IO Worker #2 INFO import  Imported #15 0xa20f…89cb (0 txs, 0.00 Mgas, 77 ms, 1.30 KiB)
2019-10-10 02:25:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x7aea…bc8a
2019-10-10 02:25:37 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:25:37 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:25:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:25:52 UTC IO Worker #0 INFO import     2/25 peers     37 KiB chain   35 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:25:57 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:26:07 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:26:07 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAliveResponse from 0x7aea…bc8a
2019-10-10 02:26:17 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:26:17 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:26:27 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:26:27 UTC IO Worker #3 INFO import     2/25 peers     37 KiB chain   35 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:26:37 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:26:46 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAlive from 0x7aea…bc8a
2019-10-10 02:26:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:26:57 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:26:57 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:26:57 UTC IO Worker #3 INFO import     2/25 peers     37 KiB chain   35 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:27:07 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:27:12 UTC IO Worker #0 INFO import  Imported #16 0xdc24…c74d (0 txs, 0.00 Mgas, 51 ms, 1.30 KiB)
2019-10-10 02:27:17 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: executing maintain procedures
2019-10-10 02:27:17 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x1fef…19be: received message Cluster.KeepAliveResponse from 0x7aea…bc8a



2019-10-10 02:23:26 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:23:26 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-10 02:23:26 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.InitializeSession from 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.ConfirmInitialization to 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.CompleteInitialization from 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x1fef…19be
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x1fef…19be
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Generation.PublicKeyShare from 0x1fef…19be
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x1fef…19be
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.PublicKeyShare from 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Generation.SessionCompleted from 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.SessionCompleted to 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-0 INFO secretstore_net  0x7aea…bc8a: generation session completed
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Encryption.InitializeEncryptionSession from 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Encryption.ConfirmEncryptionInitialization to 0x40ea…89fd
2019-10-10 02:23:33 UTC tokio-runtime-worker-2 INFO secretstore_net  0x7aea…bc8a: encryption session completed
2019-10-10 02:23:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:23:39 UTC IO Worker #2 INFO import  Imported #13 0xe0b0…da26 (1 txs, 0.20 Mgas, 168 ms, 1.17 KiB)
2019-10-10 02:23:46 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:23:46 UTC IO Worker #3 INFO import     2/25 peers     33 KiB chain   27 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.InitializeSession from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.ConfirmInitialization to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.CompleteInitialization from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x1fef…19be
2019-10-10 02:23:47 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.PublicKeyShare from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x1fef…19be
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x1fef…19be
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.PublicKeyShare from 0x1fef…19be
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Generation.SessionCompleted from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.SessionCompleted to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-0 INFO secretstore_net  0x7aea…bc8a: generation session completed
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Encryption.InitializeEncryptionSession from 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: sent message Encryption.ConfirmEncryptionInitialization to 0x40ea…89fd
2019-10-10 02:23:47 UTC tokio-runtime-worker-1 INFO secretstore_net  0x7aea…bc8a: encryption session completed
2019-10-10 02:23:51 UTC IO Worker #1 INFO import  Syncing      #13 0xe0b0…da26     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed       #11    2/25 peers     33 KiB chain   27 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Generation.InitializeSession from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.ConfirmInitialization to 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.CompleteInitialization from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.KeysDissemination to 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Generation.KeysDissemination from 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Generation.PublicKeyShare from 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x1fef…19be
2019-10-10 02:23:54 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: sent message Generation.PublicKeyShare to 0x40ea…89fd
2019-10-10 02:23:54 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Generation.PublicKeyShare from 0x1fef…19be
2019-10-10 02:23:56 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:23:57 UTC IO Worker #0 INFO import  Imported #14 0x48b1…b28f (1 txs, 0.18 Mgas, 49 ms, 0.88 KiB)
2019-10-10 02:24:06 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:24:16 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:24:21 UTC IO Worker #1 INFO import     2/25 peers     36 KiB chain   30 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:24:26 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:24:26 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:24:26 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-10 02:24:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:24:46 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:24:51 UTC IO Worker #0 INFO import     2/25 peers     36 KiB chain   30 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:24:56 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:24:56 UTC tokio-runtime-worker-2 WARN ethcore_secretstore::key_server_cluster::client_sessions::generation_session  0x7aea…bc8a: generation session failed with timeout
2019-10-10 02:24:57 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAlive from 0x1fef…19be
2019-10-10 02:24:59 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAlive from 0x40ea…89fd
2019-10-10 02:25:06 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:25:16 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:25:21 UTC IO Worker #3 INFO import     2/25 peers     36 KiB chain   30 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:25:26 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:25:36 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:25:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-10 02:25:36 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:25:36 UTC IO Worker #2 INFO import  Imported #15 0x6eaa…7a1c (0 txs, 0.00 Mgas, 86 ms, 1.30 KiB)
2019-10-10 02:25:46 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:25:51 UTC IO Worker #2 INFO import     2/25 peers     36 KiB chain   30 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:25:56 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:26:06 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:26:07 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAlive from 0x1fef…19be
2019-10-10 02:26:09 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAlive from 0x40ea…89fd
2019-10-10 02:26:16 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:26:21 UTC IO Worker #2 INFO import     2/25 peers     36 KiB chain   30 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:26:26 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:26:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:26:46 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:26:46 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x1fef…19be
2019-10-10 02:26:46 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAliveResponse from 0x40ea…89fd
2019-10-10 02:26:56 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:26:56 UTC IO Worker #0 INFO import     2/25 peers     36 KiB chain   30 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:27:06 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:27:15 UTC IO Worker #0 INFO import  Imported #16 0xf93e…3351 (0 txs, 0.00 Mgas, 30 ms, 1.30 KiB)
2019-10-10 02:27:16 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:27:17 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAlive from 0x1fef…19be
2019-10-10 02:27:19 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: received message Cluster.KeepAlive from 0x40ea…89fd
2019-10-10 02:27:26 UTC tokio-runtime-worker-0 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:27:26 UTC IO Worker #1 INFO import     2/25 peers     36 KiB chain   30 KiB db  0 bytes queue    9 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
2019-10-10 02:27:36 UTC tokio-runtime-worker-1 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures
2019-10-10 02:27:46 UTC tokio-runtime-worker-2 TRACE secretstore_net  0x7aea…bc8a: executing maintain procedures

[svyatonik]

OK, thanks for your report and testing! I'll open PR fixing this panic today.

Re session hang in last logs - that reminded me of #10266. I've checked - the fix for #10266 (which is #10303) isn't in any 2.5.* releases, unfortunately. The first official release that has a fix for this, is 2.6.0.