My own father (now deceased) could be a very difficult person.
He had very poor relational skills, resorted to passive aggressive behaviors, and couldn't communicate his feelings well.
If the difficult family shows some willingness or ability to improve the relationship and you are motivated to try to improve it, then initiate a conversation or series of conversations to discuss your own boundaries, listen to theirs, and to try to negotiate for better behavior.
This can be a tricky conversation when someone is defensive, sensitive, or angry.
We have far too much emotional investment and history with these people to be able to disengage without being deeply wounded in the process.