I agree — CYA by getting legal advice about the possibilities of lawsuits. However, as Leanna said — it is YOUR story told from YOUR perspective and if they don’t like it they can write their own book!
The Constitution (assuming you are from the U.S.) gives us a very important right — Freedom of Speech.
Change names and places but leave all the other details. It’s creative non-fiction right? The creative part is that the names and places and dates have been changed to protect identities and feelings. The non-fiction part is the story itself and detailing what actually happened in that slice of time.
If the folks who are part of Your-story (your history) are going to get mad about what they read, then they shouldn’t have done the things that you are writing about…