Education Minister Paul Givan has been criticised for meeting "an unrepresentative group, an unelected group" of loyalists.