When you are behind Closed Doors you can’t simply walk away, because you are privy to the secrets. This does not mean you remain until death.
You can’t quit, BUT! You can be made unnecessary. You can be supplanted by someone else.
In every organisation are the Power Hungry. They want your post. You could be deputy treasurer or vice secretary or some warm body that was appointed for whatever reason. No matter what your post, there is someone who wants it and getting you out is how it is done.
It can’t be obvious, of course, you can’t appear to want to leave, you have to be ‘forced’ out. This is not that difficult.
You can miss meetings, have to leave early. You make a point, they dismiss it, cool. You can use the help.
All you have to do is stop grasping power, stop committing your soul, take a step back.