jmdrapeau that is quite twisted requirement, however, nothing is impossible.
When it comes to manage multi-roles access model, typically, permissions are combined, so when Role A has ability to manage Term A and Role B - Term B, then user with both roles can manage both terms. That is the logical way of thinking.
Obviously, my first recommendation would be to reconsider the access management strategy and choose single role model. However, if that is not possible, then my next recommendation would be as following:
Disregarding all other requirements, it appears, that the main concern here is the scenario when user changes post's term from one to another to gain additional permissions. In this case, you would need to write a few lines of code that prevents user from doing that. Basically, if post is in
auto-draft state (typically this is the initial state of any post when you click "Add New") or has no terms assigned, then allow to assign terms, otherwise deny.
Your statement would have to invoke callback function that would do the proper assessment:
In the statement above, you are restricting access to assign any term of
your-custom-taxonomy to posts if
MyCustomerHandler::assess static method returns
true. So the
assess method actually does the magic here.
I hope that would give you a good starting point.