Archive node loses peers (stop syncing blocks)

[APshenkin]

• Parity Ethereum version: 2.4.6
• Operating system: Linux
• Installation: docker
• Fully synchronized: yes
• Network: ethereum mainnet
• Restarted: yes

Machine spec: 8 CPU 64GB ram, parity cache-size= 32GB

We have launched archive node and complete its sync. Today we update the node from 2.4.5. to 2.4.6 and run some software that fetch blocks information.

It's working, but peers connection is unstable and after some time node stop syncing blocks (it just displays Syncing 0.00 0.00 blk/s )

Below is examples of peers pump and down logs and last blocks (reversed order)

peers info

2019-06-05 20:32:52 UTC   12/25 peers     60 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 34380 µs
--
2019-06-05 20:32:12 UTC   10/25 peers     54 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 32475 µs
2019-06-05 20:31:40 UTC   10/25 peers     53 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 32475 µs
2019-06-05 20:31:05 UTC    9/25 peers     53 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 24494 µs
2019-06-05 20:30:34 UTC    9/25 peers     53 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 17362 µs
2019-06-05 20:29:55 UTC    9/25 peers     53 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 18661 µs
2019-06-05 20:29:20 UTC    9/25 peers     53 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    7 req/s, 17362 µs
2019-06-05 20:28:49 UTC    9/25 peers     52 MiB chain   16 MiB db  0 bytes queue   82 KiB sync  RPC:  1 conn,    1 req/s, 32642 µs
2019-06-05 20:28:15 UTC   10/25 peers     49 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 31118 µs
2019-06-05 20:27:41 UTC    6/25 peers     48 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    2 req/s, 25000 µs
2019-06-05 20:27:05 UTC    5/25 peers     48 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 25000 µs
2019-06-05 20:26:33 UTC    5/25 peers     48 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 20223 µs
2019-06-05 20:26:00 UTC    5/25 peers     48 MiB chain   16 MiB db  0 bytes queue   80 KiB sync  RPC:  1 conn,    1 req/s, 33003 µs
2019-06-05 20:26:00 UTC    4/25 peers     48 MiB chain   16 MiB db  0 bytes queue   80 KiB sync  RPC:  1 conn,    1 req/s, 33003 µs
2019-06-05 20:25:11 UTC    5/25 peers     48 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 33558 µs
2019-06-05 20:24:33 UTC    6/25 peers     47 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 33558 µs
2019-06-05 20:23:53 UTC    8/25 peers     47 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 30569 µs
2019-06-05 20:23:53 UTC    8/25 peers     47 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 30569 µs
2019-06-05 20:23:11 UTC    8/25 peers     47 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 26161 µs
2019-06-05 20:22:43 UTC Updated conversion rate to Ξ1 = US$248.89 (19132568 wei/gas)
2019-06-05 20:22:40 UTC    7/25 peers     47 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 17618 µs
2019-06-05 20:22:16 UTC    8/25 peers     47 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 15189 µs
2019-06-05 20:21:37 UTC    8/25 peers     46 MiB chain   16 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    4 req/s, 21464 µs
2019-06-05 20:21:02 UTC    7/25 peers     44 MiB chain   15 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 25221 µs
2019-06-05 20:21:02 UTC    7/25 peers     44 MiB chain   15 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 25221 µs
2019-06-05 20:20:31 UTC    7/25 peers     43 MiB chain   15 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 25221 µs
2019-06-05 20:19:50 UTC    5/25 peers     43 MiB chain   15 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    3 req/s, 25221 µs

stuck on block

2019-06-05 20:36:00 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7895893   21/25 peers     62 MiB chain   17 MiB db  0 bytes queue  443 MiB sync  RPC:  1 conn,    0 req/s, 46705 µs
2019-06-05 20:35:55 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7893988   21/25 peers     62 MiB chain   17 MiB db  0 bytes queue  355 MiB sync  RPC:  1 conn,    1 req/s, 46705 µs
2019-06-05 20:35:50 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7893988   21/25 peers     62 MiB chain   17 MiB db  0 bytes queue  176 MiB sync  RPC:  1 conn,    1 req/s, 46705 µs
2019-06-05 20:35:45 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7892083   20/25 peers     62 MiB chain   17 MiB db  0 bytes queue  147 MiB sync  RPC:  1 conn,    1 req/s, 38701 µs
2019-06-05 20:35:40 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7885225   21/25 peers     62 MiB chain   17 MiB db  0 bytes queue  551 MiB sync  RPC:  1 conn,    0 req/s, 38701 µs
2019-06-05 20:35:35 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7885225   19/25 peers     62 MiB chain   17 MiB db  0 bytes queue  401 MiB sync  RPC:  1 conn,    1 req/s, 38701 µs
2019-06-05 20:35:30 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7885225   19/25 peers     62 MiB chain   17 MiB db  0 bytes queue  318 MiB sync  RPC:  1 conn,    1 req/s, 38701 µs
2019-06-05 20:35:25 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7885225   19/25 peers     62 MiB chain   17 MiB db  0 bytes queue  212 MiB sync  RPC:  1 conn,    0 req/s, 33990 µs
2019-06-05 20:35:20 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7884971   17/25 peers     62 MiB chain   17 MiB db  0 bytes queue  140 MiB sync  RPC:  1 conn,    1 req/s, 33990 µs
2019-06-05 20:35:15 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7884971   16/25 peers     61 MiB chain   17 MiB db  0 bytes queue   60 MiB sync  RPC:  1 conn,    2 req/s, 33990 µs
2019-06-05 20:35:10 UTC Syncing #7901361 0xd3df…ad5c     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7884971   18/25 peers     61 MiB chain   17 MiB db  0 bytes queue   87 KiB sync  RPC:  1 conn,    1 req/s, 33990 µs
2019-06-05 20:35:08 UTC Syncing #7901361 0xd3df…ad5c     0.60 blk/s   59.2 tx/s    4.8 Mgas/s      0+    0 Qed  #7884971   17/25 peers     61 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    2 req/s, 32341 µs
2019-06-05 20:35:01 UTC Reorg to #7901360 0xba42…c2a0 (0x8d58…2732 #7901358 0x9dc9…e404 0x3b25…123b)
2019-06-05 20:35:00 UTC Syncing #7901359 0x8d58…2732     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    1 Qed  #7901360   13/25 peers     61 MiB chain   17 MiB db   98 KiB queue   80 KiB sync  RPC:  1 conn,    0 req/s, 33990 µs
2019-06-05 20:34:55 UTC Syncing #7901359 0x8d58…2732     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7901358   11/25 peers     61 MiB chain   17 MiB db  0 bytes queue   81 KiB sync  RPC:  1 conn,    1 req/s, 29244 µs
2019-06-05 20:34:51 UTC Syncing #7901359 0x8d58…2732     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7901358   11/25 peers     61 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 29244 µs
2019-06-05 20:34:46 UTC Syncing #7901359 0x8d58…2732     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7901358   11/25 peers     61 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 29244 µs
2019-06-05 20:34:45 UTC Syncing #7901359 0x8d58…2732     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7901358   11/25 peers     61 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 29244 µs
2019-06-05 20:34:45 UTC Syncing #7901359 0x8d58…2732     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7901358   10/25 peers     61 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 29244 µs
2019-06-05 20:34:44 UTC Syncing #7901359 0x8d58…2732     0.06 blk/s    6.6 tx/s    0.4 Mgas/s      0+    0 Qed  #7901358   11/25 peers     61 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 29244 µs
2019-06-05 20:34:30 UTC Imported #7901359 0x8d58…2732 (102 txs, 7.99 Mgas, 709 ms, 29.14 KiB)
2019-06-05 20:34:09 UTC Imported #7901358 0x9dc9…e404 (136 txs, 7.99 Mgas, 632 ms, 34.54 KiB)
2019-06-05 20:34:09 UTC   10/25 peers     61 MiB chain   17 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 23838 µs
2019-06-05 20:33:43 UTC Imported #7901357 0xec21…a3fd (118 txs, 8.00 Mgas, 563 ms, 28.45 KiB)

Also we have usual full node and it's not faced this issue (it's just continue syncing normaly)
After archive node restart it continuous syncing

Is it a bug? Or maybe we can do something to prevent restarting every two hours?

Thanks in advance

[APshenkin]

So found another strange behaviour. It looks like archive node take all 32GB cache. And when it uses all cache then it stucks

[jam10o-new]

@APshenkin if you specify a cache_size, parity will try to use all of that as cache, that is expected. Some memory will also be used beyond that during normal operation.

Things I am concerned about -
The node getting stuck after its cache fills
The large reorg that happened right before your node got stuck

[APshenkin]

Yes I'm agreed, but it strange, that it stucks each time when cache fills (this was several times today already)

Feel free to ask any other questions

Will post some info here if this will occur again

[APshenkin]

@joshua-mir So this occurs again

2019-06-06 02:10:45 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #49400   24/25 peers    410 MiB chain   49 MiB db  0 bytes queue   50 MiB sync  RPC:  1 conn,    2 req/s, 44139 µs
--
2019-06-06 02:10:40 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #49400   24/25 peers    402 MiB chain   49 MiB db  0 bytes queue   45 MiB sync  RPC:  1 conn,    1 req/s, 44139 µs
2019-06-06 02:10:35 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #49400   23/25 peers    381 MiB chain   49 MiB db  0 bytes queue   28 MiB sync  RPC:  1 conn,    0 req/s, 44139 µs
2019-06-06 02:10:30 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #46733   23/25 peers    366 MiB chain   49 MiB db  0 bytes queue   18 MiB sync  RPC:  1 conn,    1 req/s, 44139 µs
2019-06-06 02:10:25 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #32382   20/25 peers    351 MiB chain   49 MiB db  0 bytes queue   16 MiB sync  RPC:  1 conn,    0 req/s, 44139 µs
2019-06-06 02:10:20 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #57149   22/25 peers    341 MiB chain   49 MiB db  0 bytes queue   24 MiB sync  RPC:  1 conn,    2 req/s, 44139 µs
2019-06-06 02:10:15 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #55119   17/25 peers    340 MiB chain   49 MiB db  0 bytes queue   18 MiB sync  RPC:  1 conn,    1 req/s, 44139 µs
2019-06-06 02:10:10 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #47370   18/25 peers    337 MiB chain   49 MiB db  0 bytes queue   17 MiB sync  RPC:  1 conn,    0 req/s, 44139 µs
2019-06-06 02:10:05 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #36835   18/25 peers    334 MiB chain   49 MiB db  0 bytes queue   18 MiB sync  RPC:  1 conn,    1 req/s, 44139 µs
2019-06-06 02:10:00 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #54610   18/25 peers    333 MiB chain   49 MiB db  0 bytes queue   29 MiB sync  RPC:  1 conn,    0 req/s, 44139 µs
2019-06-06 02:09:55 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #54610   12/25 peers    328 MiB chain   49 MiB db  0 bytes queue   27 MiB sync  RPC:  1 conn,    1 req/s, 44139 µs
2019-06-06 02:09:50 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #54610   11/25 peers    328 MiB chain   49 MiB db  0 bytes queue   13 MiB sync  RPC:  1 conn,    1 req/s, 44139 µs
2019-06-06 02:09:45 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #50805   10/25 peers    327 MiB chain   49 MiB db  0 bytes queue   11 MiB sync  RPC:  1 conn,    0 req/s, 44102 µs
2019-06-06 02:09:40 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #45214    9/25 peers    327 MiB chain   49 MiB db  0 bytes queue   12 MiB sync  RPC:  1 conn,    1 req/s, 44102 µs
2019-06-06 02:09:35 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #38751    9/25 peers    326 MiB chain   49 MiB db  0 bytes queue    9 MiB sync  RPC:  1 conn,    0 req/s, 43872 µs
2019-06-06 02:09:30 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #35941    8/25 peers    326 MiB chain   49 MiB db  0 bytes queue    7 MiB sync  RPC:  1 conn,    1 req/s, 43872 µs
2019-06-06 02:09:25 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #29726    6/25 peers    325 MiB chain   49 MiB db  0 bytes queue    8 MiB sync  RPC:  1 conn,    0 req/s, 43800 µs
2019-06-06 02:09:20 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #25527    6/25 peers    324 MiB chain   49 MiB db  0 bytes queue    5 MiB sync  RPC:  1 conn,    2 req/s, 43800 µs
2019-06-06 02:09:15 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #19685    4/25 peers    324 MiB chain   49 MiB db  0 bytes queue    4 MiB sync  RPC:  1 conn,    2 req/s, 43800 µs
2019-06-06 02:09:10 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #16025    4/25 peers    324 MiB chain   49 MiB db  0 bytes queue    3 MiB sync  RPC:  1 conn,    1 req/s, 43800 µs
2019-06-06 02:09:05 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed    #11811    4/25 peers    324 MiB chain   49 MiB db  0 bytes queue    2 MiB sync  RPC:  1 conn,    1 req/s, 43800 µs
2019-06-06 02:09:00 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed     #9525    4/25 peers    324 MiB chain   49 MiB db  0 bytes queue    2 MiB sync  RPC:  1 conn,    1 req/s, 43328 µs
2019-06-06 02:08:55 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed     #6731    3/25 peers    324 MiB chain   49 MiB db  0 bytes queue    1 MiB sync  RPC:  1 conn,    1 req/s, 43328 µs
2019-06-06 02:08:50 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed     #4462    3/25 peers    324 MiB chain   49 MiB db  0 bytes queue    1 MiB sync  RPC:  1 conn,    2 req/s, 40224 µs
2019-06-06 02:08:45 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed      #896    5/25 peers    320 MiB chain   49 MiB db  0 bytes queue    1 MiB sync  RPC:  1 conn,    1 req/s, 40224 µs
2019-06-06 02:08:40 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed        #0    3/25 peers    320 MiB chain   49 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 40224 µs
2019-06-06 02:08:35 UTC Syncing #7902878 0x4a3b…110b     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #7902859    3/25 peers    319 MiB chain   49 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 33841 µs
2019-06-06 02:08:10 UTC    0/25 peers    319 MiB chain   49 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 33841 µs
2019-06-06 02:08:00 UTC Imported #7902878 0x4a3b…110b (184 txs, 6.98 Mgas, 688 ms, 24.98 KiB)
2019-06-06 02:07:46 UTC Imported #7902877 0xb5de…c514 (35 txs, 3.82 Mgas, 224 ms, 7.07 KiB) + another 1 block(s) containing 207 tx(s)
2019-06-06 02:07:31 UTC    1/25 peers    318 MiB chain   49 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 33841 µs
2019-06-06 02:07:20 UTC Imported #7902874 0xc496…7548 (129 txs, 7.99 Mgas, 3102 ms, 20.17 KiB) + another 6 block(s) containing 765 tx(s)
2019-06-06 02:07:07 UTC Reorg to #7902867 0xd356…d615 (0x8082…84ab #7902866 0x794a…7153 )
2019-06-06 02:06:55 UTC    0/25 peers    318 MiB chain   48 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    0 req/s, 38258 µs
2019-06-06 02:06:24 UTC    1/25 peers    318 MiB chain   48 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 38258 µs
2019-06-06 02:05:32 UTC Imported #7902867 0x8082…84ab (62 txs, 7.99 Mgas, 415 ms, 7.37 KiB)
2019-06-06 02:05:31 UTC    4/25 peers    318 MiB chain   48 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    3 req/s, 30119 µs
2019-06-06 02:05:31 UTC    4/25 peers    318 MiB chain   48 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    3 req/s, 30119 µs
2019-06-06 02:05:31 UTC Imported #7902866 0x794a…7153 (138 txs, 7.99 Mgas, 582 ms, 20.54 KiB)
2019-06-06 02:05:19 UTC Imported #7902862 0x91d9…c746 (93 txs, 8.01 Mgas, 894 ms, 15.32 KiB) + another 1 block(s) containing 139 tx(s)
2019-06-06 02:05:04 UTC Imported #7902860 0x34ce…49cc (203 txs, 8.00 Mgas, 564 ms, 34.38 KiB)
2019-06-06 02:04:40 UTC    4/25 peers    317 MiB chain   48 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    2 req/s, 21974 µs
2019-06-06 02:04:40 UTC    4/25 peers    317 MiB chain   48 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    2 req/s, 21974 µs
2019-06-06 02:04:40 UTC Imported #7902859 0x66cb…b7a0 (150 txs, 8.00 Mgas, 572 ms, 18.83 KiB)
2019-06-06 02:04:26 UTC Imported #7902858 0xbaed…6107 (242 txs, 7.92 Mgas, 504 ms, 36.30 KiB)
2019-06-06 02:04:11 UTC    4/25 peers    317 MiB chain   48 MiB db  0 bytes queue   79 KiB sync  RPC:  1 conn,    1 req/s, 18733 µs

And in this case parity wasn't use all memory

[APshenkin]

Also maybe it will help somehow: We use trace_replayBlockTransactions API for each new generated block.

[jam10o-new]

that's an extremely heavy rpc method - it could very well be one of the causes of the problem here if you see it taking a while to return a response.

[APshenkin]

No, It's completes very fast. E.g. after syncing node after restart it completes ~ 150 success request per minute

[APshenkin]

It's strange, but looks like increase disk space solve the issue.
There were ~120 GB free space. After increasing to ~ 500 GB issue was solved.
Our parity node doesn't stuck 4 hours already (previosly it stucks in 1-2 hours)

Also it's strange, but after increating disk space Active memory value was droped down 🤔

Will continue monitoring

[jam10o-new]

Just to warn you, 500gb will not be sufficient for an archive node (once you are in sync) - you will need upwards of 2tb. You don't want to hit a full disk either because it's a known issue that full disks can cause database corruption.

[APshenkin]

Our node is already synced up 😄
Current disk size 3.328Tb

parity folder size 2.6Tb