Applying Ethical AI Frameworks in practice: Evaluating conversational AI chatbot solutions
| Suzanne Atkins |
Suzanne Atkins, Ishwarradj Badrie and Sieuwert van Otterloo
Computers and Society Research Journal (2021), 1
Ethical AI frameworks are designed to encourage the accountability, responsibility and transparency of AI applications. They provide principles for ethical design. To be truly transparent, it should be clear to the user of the AI application that the designers followed responsible AI principles. In order to test how easy it is for a user to assess the responsibility of an AI system and to understand the differences between ethical AI frameworks, we evaluated four commercial chatbots against four responsible AI frameworks. We found that the ethical frameworks produced quite different assessment scores. Many ethical AI frameworks contain requirements/principles that are difficult to evaluate for anyone except the chatbot developer. Our results also show that domain-specific ethical AI guidelines are easier to use and yield more practical insights than domain-independent frameworks. We conclude that ethical AI researchers should focus on studying specific domains and not AI as a whole, and that ethical AI guidelines should focus more on creating measurable standards and less on stating high level principles.
Accompanying data: Framework_chatbot_scores
Suzanne Atkins is an information security consultant, supporting clients to set up information security management systems. She has a background as a research scientist and currently does research in ethical AI and project management in the tech sector.