This library injects logic for post-condition blocking checks in boto3 to eliminate the majority of retry code that everyone has been forced to add (and develop tests for) in their own code. This is especially useful in serial scripts that manipulate AWS, where different commands in the script depend on changes from the previous command to be fully realized. Hopefully this sort of logic can make its way into botocore in the future so that serialized scripts can take advantage of it and become vastle simplified.
boto3-post-conditions hooks into botocore in a manner similar to the simulation library moto such that when a call completes that modifies something, the client verifies that the change has been realized. This does require a little bit of stack inspection, as the client and original call parameters are not passed through the handle_event mechanism.
Every developer who has worked with the AWS API has encountered eventual consistency issues. This behavior has led virtually every reliable product that leverages the AWS API to add their own code to defeat the effects of eventual consistency. The best way to deal with eventual consistency in a pipeline is to add post-condition checks. For example, if you delete a secret in AWS Secrets Manager, it is not immediately deleted. It is scheduled for deletion and then it is eventually deleted. If you attempt to insert a new secret with the same name during this transition you get an error.
Tag management falls into the same category, as some AWS subsystems rely on the Resource and Tag Manager to handle tagging, and that means more eventual consistency. If you modify or remove a tag on a resource, it is often not realized immediately (causing a situation where you cannot read what you wrote), so follow-up code that depends on that tag being modified or deleted can fail.
pip install boto3-post-conditions
or
poetry add boto3-post-conditions
Create a boto3 client like you normally would. To enforce post-conditions
on calls to that client, register the client with the PostConditionEnforcer
:
import boto3 from boto3_post_conditions import PostConditionEnforcer
client = boto3.client("ssm") PostConditionEnforcer.register(client)
The enforcer will inject event handlers into the client definition to block returning from your API calls until changes are actually realized in the AWS service you are modifying.
It's pre-alpha, so there are going to be some major gaps...
- Only supports some of Secrets Manager and SSM initially.
- Post-conditions are always checked serially after each modification. There is no batch post-condition processing yet, but certainly being able to wrap boto calls in a context manager and then finalize their changes all at once (allowing AWS to process all the requests in parallel) is going to be useful.
Each boto3 call that makes a modification can be verified for liveness with a subsequent read call. The "after-call" boto3 event occurs after every call allowing code to plug into the request stream. When the response indicates a change is successful, post-conditions can be enforced by reading back the resource and ensuring the expected change is there.
Take the Systems Manager (SSM) Parameter Store, for example. As an eventually consistent system, extra care must be taken to avoid some common pitfalls, such as:
- Calling put_parameter immediately followed by get_parameter has no guarantee of success. In fact, it can raise a ParameterNotFound.
- Calling delete_parameter immediately followed by get_parameter can still return the parameter.
boto3_post_conditions adds logic to ensure that the modifications you are making are realized by that subsystem's control plane before returning control to the caller. For example with put_parameter, boto3_post_conditions will ensure that:
- The parameter can be read (with GetParameter) to handle the case of a novel parameter being added.
- The parameter's tags can be read, should the Tags be set during creation. The resource and tag manager is a separate subsystem which adds even more delays to realization.
Every modification or deletion has a post-condition remedy that can ensure
the vast majority of these cases are eliminated, and virtually removing retry
logic from your code and tests! As proof, look at the recorded test for
test_ssm_integration
.