additional fix for 504 errors

[kartiksubbarao]

Describe the bug
@justinpolygon I found another place in the code that needs to be updated in order for the new #525 retry logic to work properly. The following line (a remnant of the old code) needs to be removed:

https://github.com/polygon-io/client-python/blob/master/polygon/rest/base.py#L113

This line effectively overrides the new retry_strategy object specified in the urllib3.PoolManager initialization line, so it prevents that logic from taking effect. I discovered this when I backed out my tenacity workaround and got a 504 error last night.

To Reproduce
I found a way to reproduce this problem by monkey-patching the _make_request method in urllib3 to randomly return 504 instead of the actual error code, which forces an exercise of the retry logic. To implement this test, make the following change to polygon/rest/base.py. Right after the import urllib3 line, insert the following code:

original_make_request = urllib3.connectionpool.HTTPConnectionPool._make_request
def monkey_patched_make_request(self, conn, method, url, **kwargs):
    response = original_make_request(self, conn, method, url, **kwargs)
    import random
    if random.random() < 0.90: response.status = 504
    return response
urllib3.connectionpool.HTTPConnectionPool._make_request = monkey_patched_make_request

Here is a sample script to test the library with the above change:

import polygon
from pprint import pprint

polygon_api = polygon.RESTClient(POLYGON_KEY_REDACTED, retries=2, trace=True, verbose=True)

trades = [*polygon_api.list_trades('AAPL', timestamp_gt=1695944980000000000, timestamp_lte=1695945000000000000)]
pprint(trades)

With the current code, running the sample script results in the following kind of intermittent error, which indicates that retries aren't actually happening:

polygon.exceptions.BadResponse: {"results"': [...]}

When I delete line 113 from polygon/rest/base.api, I get a totally different intermittent error:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.polygon.io', port=443): Max retries exceeded with url: /v3/trades/AAPL?timestamp.lte=1695945000000000000&timestamp.gt=1695944980000000000 (Caused by ResponseError('too many 504 error responses'))

This indicates that the retries are actually happening.

[justinpolygon]

Thanks for tracking this down and submitted the patch!

[justinpolygon]

We just released v1.12.8 with your patch. Thank you!

[kartiksubbarao]

Thanks for the super fast response Justin!