Stories and memories in your family space are visible to the members of that family, based on their role. See Family roles and permissions.
Perspective contributors see the one loved one's story they were invited to, not your whole family space.
Share links (portraits and collections) can be viewed by anyone who has the link, until it expires.
Cora conversations are private to you.
Your controls
In Settings > Privacy you can:
turn off Show my photos to family members,
choose whether people can invite you to new families,
turn people grouping and photo location on or off.