StatusCode.UNAVAILABLE OS Error: Error received from peer

[modenl]

I am experiencing this error in the new version(0.3.3) which used the same grpc channel over longer period of time:
grpc/grpc#16515
No repro with 0.2.5 which I used new grpc channel for every call.

[willcl-ark]

Hello @modenl, sorry to hear that. I am not able to reproduce my side, do you have any logs or steps that I could take to try and see what might be happening?

Reading through that issue looks like many commenters actually had other network issues, do you suspect that something like this was not contributing to your problem?

Failing that I could also try a branch which increased the channel connectivity polling rate as per the other commenter in that thread, and see if that helps solve your issue.

The channel is monitored by grpc package every 5 seconds, the channel monitoring introduced in lnd_grpc is actually to stimulate regeneration of the stub (not strictly necessary) when the channel goes stale, and is then (hopefully) reconnected. This is to help during WalletUnlock stage, after which LightningServicer becomes available, but seems to trip up if the stub is created before the service is online.

[modenl]

Another related issue:
grpc/grpc#11043
My observation is that it happens after idle for couple hours. I don't know why that 5 seconds grpc connectivity check doesn't work.
My workaround is to open and close channel for every rpc call.

[modenl]

More related issue: grpc/grpc#10684 (comment)
Example of how to handle retry: https://www.programcreek.com/python/example/95429/grpc._channel

[modenl]

I've confirmed that the grpc client will be disconnected by grpc server after idling for couple minutes. Adding retry once logic worked around for me.

[willcl-ark]

Thanks @modenl. Is is something that you can make a PR out of?