While the P&F data science team needs to value each expert's opinion equally, it's impossible to please everyone. Instead of focusing on the subjective opinions of experts, they decided to evaluate chatbots based on questions from past customers. Now, experts don't have to think up questions to test the chatbot with, making the evaluation closer to real-world situations. After all, the original reason for involving experts was because they have a better understanding of real customer questions compared to the P&F data science team.
We found that the most common questions that P&F receives are about the technical description of the paper clip. P&F customers want to know the detailed technical specifications of the paper clip. P&F has thousands of types of paper clips, and it takes a long time for customer support to answer the questions.
The data science team, with their understanding of test-driven development, will create a dataset from the conversation history, which will include: Customer Questions and Customer Support Response:
With a dataset of questions and answers, P&F can retrospectively test and evaluate the chatbot’s performance. They create a new column called “Chatbot Responses” to store example responses of the chatbot to questions.
We can have experts and GPT-4 evaluate the quality of the chatbot's responses. Our ultimate goal is to use GPT-4 to automate the accuracy evaluation of chatbots. This is possible. if Experts and GPT-4 evaluate the answers alike.
The experts will create a new Excel sheet with each expert’s rating, and the data science team will add the GPT-4 rating.
There is Conflict Methodology Evaluated by various experts of Same chatbot replySince GPT-4 evaluates similarly to the majority vote of experts, it is conceivable that automatic evaluation can be performed with GPT-4. However, the opinion of each expert is valuable, and it is important to address the conflicting evaluation preferences among experts.
P&F will hold workshops with experts to The gold standard Answers to past question datasets
and evaluation Best Practice GuidelinesAll experts agree on this.
Armed with insights from the workshop, the data science team can craft more detailed evaluation prompts for GPT-4 that cover edge cases (e.g., “The chatbot should not ask to open a support ticket”). Experts can improve their paperclip documentation over time and Define best practices Instead of tedious chatbot evaluation.
By measuring the percentage of correct replies of the chatbot, P&F can decide whether to deploy the chatbot to a support channel. Approve the accuracy and deploy the chatbot.
Finally, we store all the chatbot responses and calculate how well the chatbot can solve real customer queries. Because customers can respond directly to the chatbot, it is also important to record responses from customers to understand customer sentiment.
You can use the same evaluation workflow to factually measure the success of your chatbot, even without the actual answer. However, your customers have received the first answer from your chatbot, and you don't know if they liked it or not. You need to investigate customer reactions to your chatbot's answers. You can automatically detect negative sentiment from customer answers and assign customer support specialists to handle angry customers.
