Four councillors who voted to allow a rapist taxi driver to keep his operator's licence have resigned from a Scottish council's committee.