add reconnection logic [CPP-371]

[notoriaga]

No description provided.

[silverjam]

Changes look good but we have quite a lot of thread::spawn calls now (which may be unavoidable given our architecture)-- can we create a diagram with draw.io that details how these threads interact? We should detail if threads are meant to be ephemeral or if they're long lived too.

[silverjam]

I'm not seeing this timeout trigger, at least doesn't seem to work when tearing down the VPN:

[notoriaga]

Hmm strange, for me when I disconnect the vpn it triggers an error from the underlying tcp connection saying Resource temporarily unavailable (os error 11). Similar thing happens if I just disable wifi, haven't actually figured out a way to simulate no messages for 30 seconds without the io resource just returning an error itself

[silverjam]

Should be able to chain two socat instances together, one connected to the Piksi, connect another socat to the Piksi socat, console connects to this one, then kill the one connected to the Piksi.

[silverjam]

Any luck on reproducing this?

[notoriaga]

No I just got another io error, BrokenPipe. I'm trying to shorten the timeout and then throttle my internet but that doesn't appear to work either

[notoriaga]

Actually that does trigger the timeout for me, then after un-throttling it reconnects

[notoriaga]

@silverjam Added a diagram of the backend threads. Not really sure how one typically draws something like this but for what I have each bar represents a thread. The left -> right layout represents parent/child relationships between threads.

@john-michaelburke does this seem accurate to you?

[silverjam]

@silverjam Added a diagram of the backend threads. Not really sure how one typically draws something like this but for what I have each bar represents a thread. The left -> right layout represents parent/child relationships between threads.

Looks great, thanks