[2020] how to remove a "perk" ?

You can use the forgetting elixir you get from the trader to refund the points.

You can also mod the game to make sure that this perk has zero effect if you determine that everyone else also has to agree.