A failure by the state in Northern Ireland to regulate mother and baby homes and Magdalene Laundries was a serious human rights issue that should be examined in an upcoming public inquiry, a report has found.