It's currently not possible. I'll add an option to their dialogue for you to deny them.
Please send me your save folder so I can check it out. Instructions are here:
https://steamcommunity.com/app/688060/discussions/0/1741106440033049878/