Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pool connect ignores DNS ttl #254

Open
alexmac opened this issue Feb 5, 2018 · 4 comments
Open

Pool connect ignores DNS ttl #254

alexmac opened this issue Feb 5, 2018 · 4 comments

Comments

@alexmac
Copy link

alexmac commented Feb 5, 2018

  • asyncpg version: 0.14
  • PostgreSQL version: 9.5/9.6
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : Using AWS RDS
  • Python version: 3.6
  • Platform: Linux in docker
  • Do you use pgbouncer?: Not in this instances
  • Did you install asyncpg with pip?: yes
  • If you built asyncpg locally, which version of Cython did you use?:
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : We only use asyncio

After performing a failover/reboot of an AWS RDS instance asyncpg does not appear to reconnect gracefully - it looks like the code in Pool caches the IP of the PG instance after dns resolution and does not respect whatever TTL we might have set on the cname that points at our master: https://github.com/MagicStack/asyncpg/blob/master/asyncpg/pool.py#L133

@egalpin
Copy link

egalpin commented Feb 7, 2018

See #253 (which supersedes #249), which attempts to address this exact issue by tracking the consecutive number of exceptions that a connection has incurred. After the max is reached, the connection is closed so that a new one can be added to the pool. We've found this to have effectively mitigated our failover issues (also using RDS).

You can install it by adding the following to your requirements.txt or as part of a build process (also requires Cython):
(requirements.txt)

git+https://github.com/egalpin/asyncpg.git@add_max_exceptions_to_pool#egg=asyncpg

(plain pip)

pip install git+https://github.com/egalpin/asyncpg.git@add_max_exceptions_to_pool#egg=asyncpg

@egalpin
Copy link

egalpin commented Feb 7, 2018

Though, after reading the code you pointed to more closely, it may be better handled by updating the connection logic to not cache (or, optionally so) the resolved address

@egalpin
Copy link

egalpin commented Feb 7, 2018

See specifically this commit which will cause a pool to cycle through your provided hosts again should a connection end up being closed due to too many consecutive exceptions

@elprans
Copy link
Member

elprans commented Mar 7, 2018

asyncpg does not perform any name resolution itself, Connection._addr and Pool._working_addr contain the host string as it was passed to connect().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants