diff --git a/docs/content/C2 Building Systems with the ChatGPT API/10.评估（下）Evaluation-part2.ipynb b/docs/content/C2 Building Systems with the ChatGPT API/10.评估（下）Evaluation-part2.ipynb new file mode 100644 index 0000000..970cb9d --- /dev/null +++ b/docs/content/C2 Building Systems with the ChatGPT API/10.评估（下）Evaluation-part2.ipynb @@ -0,0 +1 @@ +{"cells":[{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# 第十章 评估（下）——当不存在一个简单的正确答案时\n","\n"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["在上一章中，了解了如何评估 LLM 模型在 **有明确正确答案** 的情况下的输出，我们可以编写一个函数来判断 LLM 输出是否正确地分类并列出产品。\n","\n","然而，如果 LLM 用于生成文本，而不仅仅是分类问题的答案呢？接下来，我们将探讨如何评估这种类型的 LLM 输出的方法。"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## 一、运行问答系统获得一个复杂回答"]},{"cell_type":"code","execution_count":3,"metadata":{},"outputs":[],"source":["import utils_zh\n","\n","'''\n","注意：限于模型对中文理解能力较弱，中文 Prompt 可能会随机出现不成功，可以多次运行；也非常欢迎同学探究更稳定的中文 Prompt\n","'''\n","# 用户消息\n","customer_msg = f\"\"\"\n","告诉我有关 the smartx pro phone 和 the fotosnap camera, the dslr one 的信息。\n","另外，你们这有什么 TVs ？\"\"\"\n","\n","# 从问题中抽取商品名\n","products_by_category = utils_zh.get_products_from_query(customer_msg)\n","# 将商品名转化为列表\n","category_and_product_list = utils_zh.read_string_to_list(products_by_category)\n","# 查找商品对应的信息\n","product_info = utils_zh.get_mentioned_product_info(category_and_product_list)\n","# 由信息生成回答\n","assistant_answer = utils_zh.answer_user_msg(user_msg=customer_msg, product_info=product_info)"]},{"cell_type":"code","execution_count":4,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["关于SmartX Pro手机和FotoSnap DSLR相机的信息：\n","\n","1. SmartX Pro手机（型号：SX-PP10）是一款功能强大的智能手机，拥有6.1英寸显示屏、128GB存储空间、12MP双摄像头和5G网络支持。价格为899.99美元，保修期为1年。\n","\n","2. FotoSnap DSLR相机（型号：FS-DSLR200）是一款多功能的单反相机，拥有24.2MP传感器、1080p视频拍摄、3英寸液晶屏和可更换镜头。价格为599.99美元，保修期为1年。\n","\n","关于电视的信息：\n","\n","我们有以下电视可供选择：\n","1. CineView 4K电视（型号：CV-4K55）- 55英寸显示屏，4K分辨率，支持HDR和智能电视功能。价格为599.99美元，保修期为2年。\n","2. CineView 8K电视（型号：CV-8K65）- 65英寸显示屏，8K分辨率，支持HDR和智能电视功能。价格为2999.99美元，保修期为2年。\n","3. CineView OLED电视（型号：CV-OLED55）- 55英寸OLED显示屏，4K分辨率，支持HDR和智能电视功能。价格为1499.99美元，保修期为2年。\n","\n","请问您对以上产品有任何进一步的问题或者需要了解其他产品吗？\n"]}],"source":["print(assistant_answer) "]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## 二、使用 GPT 评估回答是否正确"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["我们希望您能从中学到一个设计模式，即当您可以指定一个评估 LLM 输出的标准列表时，您实际上可以使用另一个 API 调用来评估您的第一个 LLM 输出。"]},{"cell_type":"code","execution_count":5,"metadata":{},"outputs":[],"source":["# 问题、上下文\n","cust_prod_info = {\n"," 'customer_msg': customer_msg,\n"," 'context': product_info\n","}"]},{"cell_type":"code","execution_count":6,"metadata":{},"outputs":[],"source":["from tool import get_completion_from_messages\n","\n","def eval_with_rubric(test_set, assistant_answer):\n"," \"\"\"\n"," 使用 GPT API 评估生成的回答\n","\n"," 参数：\n"," test_set: 测试集\n"," assistant_answer: 助手的回复\n"," \"\"\"\n"," \n"," cust_msg = test_set['customer_msg']\n"," context = test_set['context']\n"," completion = assistant_answer\n"," \n"," # 人设\n"," system_message = \"\"\"\\\n"," 你是一位助理，通过查看客户服务代理使用的上下文来评估客户服务代理回答用户问题的情况。\n"," \"\"\"\n","\n"," # 具体指令\n"," user_message = f\"\"\"\\\n"," 你正在根据代理使用的上下文评估对问题的提交答案。以下是数据：\n"," [开始]\n"," ************\n"," [用户问题]: {cust_msg}\n"," ************\n"," [使用的上下文]: {context}\n"," ************\n"," [客户代理的回答]: {completion}\n"," ************\n"," [结束]\n","\n"," 请将提交的答案的事实内容与上下文进行比较，忽略样式、语法或标点符号上的差异。\n"," 回答以下问题：\n"," 助手的回应是否只基于所提供的上下文？（是或否）\n"," 回答中是否包含上下文中未提供的信息？（是或否）\n"," 回应与上下文之间是否存在任何不一致之处？（是或否）\n"," 计算用户提出了多少个问题。（输出一个数字）\n"," 对于用户提出的每个问题，是否有相应的回答？\n"," 问题1：（是或否）\n"," 问题2：（是或否）\n"," ...\n"," 问题N：（是或否）\n"," 在提出的问题数量中，有多少个问题在回答中得到了回应？（输出一个数字）\n","\"\"\"\n","\n"," messages = [\n"," {'role': 'system', 'content': system_message},\n"," {'role': 'user', 'content': user_message}\n"," ]\n","\n"," response = get_completion_from_messages(messages)\n"," return response"]},{"cell_type":"code","execution_count":8,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["助手的回应只基于所提供的上下文。是\n","回答中不包含上下文中未提供的信息。是\n","回应与上下文之间不存在任何不一致之处。是\n","用户提出了2个问题。2\n","对于用户提出的每个问题，都有相应的回答。\n","问题1：是\n","问题2：是\n","在提出的问题数量中，有2个问题在回答中得到了回应。2\n"]}],"source":["evaluation_output = eval_with_rubric(cust_prod_info, assistant_answer)\n","print(evaluation_output)"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## 三、评估生成回答与标准回答的差距"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["在经典的自然语言处理技术中，有一些传统的度量标准用于衡量 LLM 输出与人类专家编写的输出的相似度。例如，BLUE 分数可用于衡量两段文本的相似程度。\n","\n","实际上有一种更好的方法，即使用 Prompt。您可以指定 Prompt，使用 Prompt 来比较由 LLM 自动生成的客户服务代理响应与人工理想响应的匹配程度。"]},{"cell_type":"code","execution_count":34,"metadata":{},"outputs":[],"source":["'''基于中文Prompt的验证集'''\n","test_set_ideal = {\n"," 'customer_msg': \"\"\"\\\n","告诉我有关 the Smartx Pro 手机 和 FotoSnap DSLR相机, the dslr one 的信息。\\n另外，你们这有什么电视 ？\"\"\",\n"," 'ideal_answer':\"\"\"\\\n","SmartX Pro手机是一款功能强大的智能手机，拥有6.1英寸显示屏、128GB存储空间、12MP双摄像头和5G网络支持。价格为899.99美元，保修期为1年。\n","FotoSnap DSLR相机是一款多功能的单反相机，拥有24.2MP传感器、1080p视频拍摄、3英寸液晶屏和可更换镜头。价格为599.99美元，保修期为1年。\n","\n","我们有以下电视可供选择：\n","1. CineView 4K电视（型号：CV-4K55）- 55英寸显示屏，4K分辨率，支持HDR和智能电视功能。价格为599.99美元，保修期为2年。\n","2. CineView 8K电视（型号：CV-8K65）- 65英寸显示屏，8K分辨率，支持HDR和智能电视功能。价格为2999.99美元，保修期为2年。\n","3. CineView OLED电视（型号：CV-OLED55）- 55英寸OLED显示屏，4K分辨率，支持HDR和智能电视功能。价格为1499.99美元，保修期为2年。\n"," \"\"\"\n","}"]},{"cell_type":"code","execution_count":37,"metadata":{},"outputs":[],"source":["def eval_vs_ideal(test_set, assistant_answer):\n"," \"\"\"\n"," 评估回复是否与理想答案匹配\n","\n"," 参数：\n"," test_set: 测试集\n"," assistant_answer: 助手的回复\n"," \"\"\"\n"," cust_msg = test_set['customer_msg']\n"," ideal = test_set['ideal_answer']\n"," completion = assistant_answer\n"," \n"," system_message = \"\"\"\\\n"," 您是一位助理，通过将客户服务代理的回答与理想（专家）回答进行比较，评估客户服务代理对用户问题的回答质量。\n"," 请输出一个单独的字母（A 、B、C、D、E），不要包含其他内容。 \n"," \"\"\"\n","\n"," user_message = f\"\"\"\\\n"," 您正在比较一个给定问题的提交答案和专家答案。数据如下:\n"," [开始]\n"," ************\n"," [问题]: {cust_msg}\n"," ************\n"," [专家答案]: {ideal}\n"," ************\n"," [提交答案]: {completion}\n"," ************\n"," [结束]\n","\n"," 比较提交答案的事实内容与专家答案，关注在内容上，忽略样式、语法或标点符号上的差异。\n"," 你的关注核心应该是答案的内容是否正确，内容的细微差异是可以接受的。\n"," 提交的答案可能是专家答案的子集、超集，或者与之冲突。确定适用的情况，并通过选择以下选项之一回答问题：\n"," （A）提交的答案是专家答案的子集，并且与之完全一致。\n"," （B）提交的答案是专家答案的超集，并且与之完全一致。\n"," （C）提交的答案包含与专家答案完全相同的细节。\n"," （D）提交的答案与专家答案存在分歧。\n"," （E）答案存在差异，但从事实的角度来看这些差异并不重要。\n"," 选项：ABCDE\n","\"\"\"\n","\n"," messages = [\n"," {'role': 'system', 'content': system_message},\n"," {'role': 'user', 'content': user_message}\n"," ]\n","\n"," response = get_completion_from_messages(messages)\n"," return response"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["这个评分标准来自于 OpenAI 开源评估框架，这是一个非常棒的框架，其中包含了许多评估方法，既有 OpenAI 开发人员的贡献，也有更广泛的开源社区的贡献。\n","\n","在这个评分标准中，我们要求 LLM 针对提交答案与专家答案进行信息内容的比较，并忽略其风格、语法和标点符号等方面的差异，但关键是我们要求它进行比较，并输出从A到E的分数，具体取决于提交的答案是否是专家答案的子集、超集或完全一致，这可能意味着它虚构或编造了一些额外的事实。\n","\n","LLM 将选择其中最合适的描述。\n"]},{"cell_type":"code","execution_count":33,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["关于SmartX Pro手机和FotoSnap DSLR相机的信息：\n","\n","1. SmartX Pro手机（型号：SX-PP10）是一款功能强大的智能手机，拥有6.1英寸显示屏、128GB存储空间、12MP双摄像头和5G网络支持。价格为899.99美元，保修期为1年。\n","\n","2. FotoSnap DSLR相机（型号：FS-DSLR200）是一款多功能的单反相机，拥有24.2MP传感器、1080p视频拍摄、3英寸液晶屏和可更换镜头。价格为599.99美元，保修期为1年。\n","\n","关于电视的信息：\n","\n","我们有以下电视可供选择：\n","1. CineView 4K电视（型号：CV-4K55）- 55英寸显示屏，4K分辨率，支持HDR和智能电视功能。价格为599.99美元，保修期为2年。\n","2. CineView 8K电视（型号：CV-8K65）- 65英寸显示屏，8K分辨率，支持HDR和智能电视功能。价格为2999.99美元，保修期为2年。\n","3. CineView OLED电视（型号：CV-OLED55）- 55英寸OLED显示屏，4K分辨率，支持HDR和智能电视功能。价格为1499.99美元，保修期为2年。\n","\n","请问您对以上产品有任何进一步的问题或者需要了解其他产品吗？\n"]}],"source":["print(assistant_answer)"]},{"cell_type":"code","execution_count":38,"metadata":{},"outputs":[{"data":{"text/plain":["'C'"]},"execution_count":38,"metadata":{},"output_type":"execute_result"}],"source":["eval_vs_ideal(test_set_ideal, assistant_answer)\n","# 对于该生成回答，GPT 判断生成内容与标准答案一致"]},{"cell_type":"code","execution_count":39,"metadata":{},"outputs":[],"source":["assistant_answer_2 = \"life is like a box of chocolates\""]},{"cell_type":"code","execution_count":40,"metadata":{},"outputs":[{"data":{"text/plain":["'D'"]},"execution_count":40,"metadata":{},"output_type":"execute_result"}],"source":["eval_vs_ideal(test_set_ideal, assistant_answer_2)\n","# 对于明显异常答案，GPT 判断为不一致"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["希望您从本章中学到两个设计模式。\n","\n","1. 即使没有专家提供的理想答案，只要能制定一个评估标准，就可以使用一个 LLM 来评估另一个 LLM 的输出。\n","\n","2. 如果您可以提供一个专家提供的理想答案，那么可以帮助您的 LLM 更好地比较特定助手输出是否与专家提供的理想答案相似。\n","\n","希望这可以帮助您评估 LLM 系统的输出，以便在开发期间持续监测系统的性能，并使用这些工具不断评估和改进系统的性能。"]},{"cell_type":"markdown","metadata":{},"source":["## 四、英文版"]},{"cell_type":"markdown","metadata":{},"source":["**1. 对问答系统提问**"]},{"cell_type":"code","execution_count":42,"metadata":{},"outputs":[],"source":["import utils_en\n","\n","# 用户消息\n","customer_msg = f\"\"\"\n","tell me about the smartx pro phone and the fotosnap camera, the dslr one.\n","Also, what TVs or TV related products do you have?\"\"\"\n","\n","# 从问题中抽取商品名\n","products_by_category = utils_en.get_products_from_query(customer_msg)\n","# 将商品名转化为列表\n","category_and_product_list = utils_en.read_string_to_list(products_by_category)\n","# 查找商品对应的信息\n","product_info = utils_en.get_mentioned_product_info(category_and_product_list)\n","# 由信息生成回答\n","assistant_answer = utils_en.answer_user_msg(user_msg=customer_msg, product_info=product_info)"]},{"cell_type":"code","execution_count":43,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Sure! Let me provide you with some information about the SmartX ProPhone and the FotoSnap DSLR Camera.\n","\n","The SmartX ProPhone is a powerful smartphone with advanced camera features. It has a 6.1-inch display, 128GB storage, a 12MP dual camera, and supports 5G connectivity. The SmartX ProPhone is priced at $899.99 and comes with a 1-year warranty.\n","\n","The FotoSnap DSLR Camera is a versatile camera that allows you to capture stunning photos and videos. It features a 24.2MP sensor, 1080p video recording, a 3-inch LCD screen, and supports interchangeable lenses. The FotoSnap DSLR Camera is priced at $599.99 and also comes with a 1-year warranty.\n","\n","As for TVs and TV-related products, we have a range of options available. Some of our popular TV models include the CineView 4K TV, CineView 8K TV, and CineView OLED TV. We also have home theater systems like the SoundMax Home Theater and SoundMax Soundbar. Could you please let me know your specific requirements or preferences so that I can assist you better?\n"]}],"source":["print(assistant_answer) "]},{"cell_type":"markdown","metadata":{},"source":["**2. 使用GPT评估**"]},{"cell_type":"code","execution_count":44,"metadata":{},"outputs":[],"source":["# 问题、上下文\n","cust_prod_info = {\n"," 'customer_msg': customer_msg,\n"," 'context': product_info\n","}"]},{"cell_type":"code","execution_count":45,"metadata":{},"outputs":[],"source":["def eval_with_rubric(test_set, assistant_answer):\n"," \"\"\"\n"," 使用 GPT API 评估生成的回答\n","\n"," 参数：\n"," test_set: 测试集\n"," assistant_answer: 助手的回复\n"," \"\"\"\n","\n"," cust_msg = test_set['customer_msg']\n"," context = test_set['context']\n"," completion = assistant_answer\n"," \n"," # 要求 GPT 作为一个助手评估回答正确性\n"," system_message = \"\"\"\\\n"," You are an assistant that evaluates how well the customer service agent \\\n"," answers a user question by looking at the context that the customer service \\\n"," agent is using to generate its response. \n"," \"\"\"\n","\n"," # 具体指令\n"," user_message = f\"\"\"\\\n","You are evaluating a submitted answer to a question based on the context \\\n","that the agent uses to answer the question.\n","Here is the data:\n"," [BEGIN DATA]\n"," ************\n"," [Question]: {cust_msg}\n"," ************\n"," [Context]: {context}\n"," ************\n"," [Submission]: {completion}\n"," ************\n"," [END DATA]\n","\n","Compare the factual content of the submitted answer with the context. \\\n","Ignore any differences in style, grammar, or punctuation.\n","Answer the following questions:\n"," - Is the Assistant response based only on the context provided? (Y or N)\n"," - Does the answer include information that is not provided in the context? (Y or N)\n"," - Is there any disagreement between the response and the context? (Y or N)\n"," - Count how many questions the user asked. (output a number)\n"," - For each question that the user asked, is there a corresponding answer to it?\n"," Question 1: (Y or N)\n"," Question 2: (Y or N)\n"," ...\n"," Question N: (Y or N)\n"," - Of the number of questions asked, how many of these questions were addressed by the answer? (output a number)\n","\"\"\"\n","\n"," messages = [\n"," {'role': 'system', 'content': system_message},\n"," {'role': 'user', 'content': user_message}\n"," ]\n","\n"," response = get_completion_from_messages(messages)\n"," return response"]},{"cell_type":"code","execution_count":46,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["- Is the Assistant response based only on the context provided? (Y or N)\n","Y\n","\n","- Does the answer include information that is not provided in the context? (Y or N)\n","N\n","\n","- Is there any disagreement between the response and the context? (Y or N)\n","N\n","\n","- Count how many questions the user asked. (output a number)\n","2\n","\n","- For each question that the user asked, is there a corresponding answer to it?\n","Question 1: Y\n","Question 2: Y\n","\n","- Of the number of questions asked, how many of these questions were addressed by the answer? (output a number)\n","2\n"]}],"source":["evaluation_output = eval_with_rubric(cust_prod_info, assistant_answer)\n","print(evaluation_output)"]},{"cell_type":"markdown","metadata":{},"source":["**3. 评估生成回答与标准回答的差距**"]},{"cell_type":"code","execution_count":47,"metadata":{},"outputs":[],"source":["test_set_ideal = {\n"," 'customer_msg': \"\"\"\\\n","tell me about the smartx pro phone and the fotosnap camera, the dslr one.\n","Also, what TVs or TV related products do you have?\"\"\",\n"," 'ideal_answer':\"\"\"\\\n","Of course! The SmartX ProPhone is a powerful \\\n","smartphone with advanced camera features. \\\n","For instance, it has a 12MP dual camera. \\\n","Other features include 5G wireless and 128GB storage. \\\n","It also has a 6.1-inch display. The price is $899.99.\n","\n","The FotoSnap DSLR Camera is great for \\\n","capturing stunning photos and videos. \\\n","Some features include 1080p video, \\\n","3-inch LCD, a 24.2MP sensor, \\\n","and interchangeable lenses. \\\n","The price is 599.99.\n","\n","For TVs and TV related products, we offer 3 TVs \\\n","\n","\n","All TVs offer HDR and Smart TV.\n","\n","The CineView 4K TV has vibrant colors and smart features. \\\n","Some of these features include a 55-inch display, \\\n","'4K resolution. It's priced at 599.\n","\n","The CineView 8K TV is a stunning 8K TV. \\\n","Some features include a 65-inch display and \\\n","8K resolution. It's priced at 2999.99\n","\n","The CineView OLED TV lets you experience vibrant colors. \\\n","Some features include a 55-inch display and 4K resolution. \\\n","It's priced at 1499.99.\n","\n","We also offer 2 home theater products, both which include bluetooth.\\\n","The SoundMax Home Theater is a powerful home theater system for \\\n","an immmersive audio experience.\n","Its features include 5.1 channel, 1000W output, and wireless subwoofer.\n","It's priced at 399.99.\n","\n","The SoundMax Soundbar is a sleek and powerful soundbar.\n","It's features include 2.1 channel, 300W output, and wireless subwoofer.\n","It's priced at 199.99\n","\n","Are there any questions additional you may have about these products \\\n","that you mentioned here?\n","Or may do you have other questions I can help you with?\n"," \"\"\"\n","}"]},{"cell_type":"code","execution_count":56,"metadata":{},"outputs":[],"source":["def eval_vs_ideal(test_set, assistant_answer):\n"," \"\"\"\n"," 评估回复是否与理想答案匹配\n","\n"," 参数：\n"," test_set: 测试集\n"," assistant_answer: 助手的回复\n"," \"\"\"\n"," cust_msg = test_set['customer_msg']\n"," ideal = test_set['ideal_answer']\n"," completion = assistant_answer\n"," \n"," system_message = \"\"\"\\\n"," You are an assistant that evaluates how well the customer service agent \\\n"," answers a user question by comparing the response to the ideal (expert) response\n"," Output a single letter and nothing else. \n"," \"\"\"\n","\n"," user_message = f\"\"\"\\\n","You are comparing a submitted answer to an expert answer on a given question. Here is the data:\n"," [BEGIN DATA]\n"," ************\n"," [Question]: {cust_msg}\n"," ************\n"," [Expert]: {ideal}\n"," ************\n"," [Submission]: {completion}\n"," ************\n"," [END DATA]\n","\n","Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.\n"," The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. \n"," Answer the question by selecting one of the following options:\n"," (A) The submitted answer is a subset of the expert answer and is fully consistent with it.\n"," (B) The submitted answer is a superset of the expert answer and is fully consistent with it.\n"," (C) The submitted answer contains all the same details as the expert answer.\n"," (D) There is a disagreement between the submitted answer and the expert answer.\n"," (E) The answers differ, but these differences don't matter from the perspective of factuality.\n"," choice_strings: ABCDE\n","\"\"\"\n","\n"," messages = [\n"," {'role': 'system', 'content': system_message},\n"," {'role': 'user', 'content': user_message}\n"," ]\n","\n"," response = get_completion_from_messages(messages)\n"," return response"]},{"cell_type":"code","execution_count":54,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Sure! Let me provide you with some information about the SmartX ProPhone and the FotoSnap DSLR Camera.\n","\n","The SmartX ProPhone is a powerful smartphone with advanced camera features. It has a 6.1-inch display, 128GB storage, a 12MP dual camera, and supports 5G connectivity. The SmartX ProPhone is priced at $899.99 and comes with a 1-year warranty.\n","\n","The FotoSnap DSLR Camera is a versatile camera that allows you to capture stunning photos and videos. It features a 24.2MP sensor, 1080p video recording, a 3-inch LCD screen, and supports interchangeable lenses. The FotoSnap DSLR Camera is priced at $599.99 and also comes with a 1-year warranty.\n","\n","As for TVs and TV-related products, we have a range of options available. Some of our popular TV models include the CineView 4K TV, CineView 8K TV, and CineView OLED TV. We also have home theater systems like the SoundMax Home Theater and SoundMax Soundbar. Could you please let me know your specific requirements or preferences so that I can assist you better?\n"]}],"source":["print(assistant_answer)"]},{"cell_type":"code","execution_count":57,"metadata":{},"outputs":[{"data":{"text/plain":["'D'"]},"execution_count":57,"metadata":{},"output_type":"execute_result"}],"source":["# 由于模型的更新，目前在原有 Prompt 上不再能够正确判断\n","eval_vs_ideal(test_set_ideal, assistant_answer)"]},{"cell_type":"code","execution_count":58,"metadata":{},"outputs":[],"source":["assistant_answer_2 = \"life is like a box of chocolates\""]},{"cell_type":"code","execution_count":59,"metadata":{},"outputs":[{"data":{"text/plain":["'D'"]},"execution_count":59,"metadata":{},"output_type":"execute_result"}],"source":["eval_vs_ideal(test_set_ideal, assistant_answer_2)\n","# 对于明显异常答案，GPT 判断为不一致"]}],"metadata":{"kernelspec":{"display_name":"zyh_gpt","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.11"},"orig_nbformat":4},"nbformat":4,"nbformat_minor":2} diff --git a/docs/content/C2 Building Systems with the ChatGPT API/9.评估（上） Evaluation-part1.ipynb b/docs/content/C2 Building Systems with the ChatGPT API/9.评估（上） Evaluation-part1.ipynb new file mode 100644 index 0000000..1a98d52 --- /dev/null +++ b/docs/content/C2 Building Systems with the ChatGPT API/9.评估（上） Evaluation-part1.ipynb @@ -0,0 +1,1663 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "aa3de8c6", + "metadata": { + "height": 30 + }, + "source": [ + "# 第九章 评估（上）——存在一个简单的正确答案时\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "c768620b", + "metadata": {}, + "source": [ + "在之前的章节中，我们展示了如何使用 LLM 构建应用程序，包括评估输入、处理输入以及在向用户显示输出之前进行最终输出检查。\n", + "\n", + "构建这样的系统后，如何知道它的工作情况？甚至在部署后并让用户使用它时，如何跟踪它的运行情况，发现任何缺陷，并持续改进系统的答案质量？\n", + "\n", + "在本章中，我们想与您分享一些最佳实践，用于评估 LLM 的输出。\n", + "\n", + "构建基于 LLM 的应用程序与传统的监督学习应用程序有所不同。由于可以快速构建基于 LLM 的应用程序，因此评估方法通常不从测试集开始。相反，通常会逐渐建立一组测试示例。\n", + "\n", + "在传统的监督学习环境中，需要收集训练集、开发集或保留交叉验证集，然后在整个开发过程中使用它们。\n", + "\n", + "然而，如果能够在几分钟内指定 Prompt，并在几个小时内得到相应结果，那么暂停很长时间去收集一千个测试样本将是一件极其痛苦的事情。因为现在，可以在零个训练样本的情况下获得这个成果。\n", + "\n", + "因此，在使用 LLM 构建应用程序时，您将体会到如下的过程：\n", + "\n", + "首先，您会在只有一到三个样本的小样本中调整 Prompt，并尝试让 Prompt 在它们身上起作用。\n", + "\n", + "然后，当系统进行进一步的测试时，您可能会遇到一些棘手的例子。Prompt 在它们身上不起作用，或者算法在它们身上不起作用。\n", + "\n", + "这就是使用 ChatGPT API 构建应用程序的开发者所经历的挑战。\n", + "\n", + "在这种情况下，您可以将这些额外的几个示例添加到您正在测试的集合中，以机会主义地添加其他棘手的示例。\n", + "\n", + "最终，您已经添加了足够的这些示例到您缓慢增长的开发集中，以至于通过手动运行每个示例来测试 Prompt 变得有些不方便。\n", + "\n", + "然后，您开始开发在这些小示例集上用于衡量性能的指标，例如平均准确性。\n", + "\n", + "这个过程的一个有趣方面是，如果您觉得您的系统已经足够好了，您可以随时停在那里，不再改进它。事实上，许多已部署的应用程序停在第一或第二个步骤，并且运行得非常好。\n", + "\n", + "需要注意的是，有很多大模型的应用程序没有实质性的风险，即使它没有给出完全正确的答案。\n", + "\n", + "但是，对于部分高风险应用，如果存在偏见或不适当的输出可能对某人造成伤害，那么收集测试集、严格评估系统的性能、确保在使用之前它能够做正确的事情，就变得更加重要。\n", + "\n", + "但是，如果您只是使用它来总结文章供自己阅读，而不是给别人看，那么可能造成的危害风险更小，您可以在这个过程中早早停止，而不必去花费更大的代价去收集更大的数据集。" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "6f4062ea", + "metadata": { + "height": 47 + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'电脑和笔记本': ['TechPro 超极本',\n", + " 'BlueWave 游戏本',\n", + " 'PowerLite Convertible',\n", + " 'TechPro Desktop',\n", + " 'BlueWave Chromebook'],\n", + " '智能手机和配件': ['SmartX ProPhone'],\n", + " '专业手机': ['MobiTech PowerCase',\n", + " 'SmartX MiniPhone',\n", + " 'MobiTech Wireless Charger',\n", + " 'SmartX EarBuds'],\n", + " '电视和家庭影院系统': ['CineView 4K TV',\n", + " 'SoundMax Home Theater',\n", + " 'CineView 8K TV',\n", + " 'SoundMax Soundbar',\n", + " 'CineView OLED TV'],\n", + " '游戏机和配件': ['GameSphere X',\n", + " 'ProGamer Controller',\n", + " 'GameSphere Y',\n", + " 'ProGamer Racing Wheel',\n", + " 'GameSphere VR Headset'],\n", + " '音频设备': ['AudioPhonic Noise-Canceling Headphones',\n", + " 'WaveSound Bluetooth Speaker',\n", + " 'AudioPhonic True Wireless Earbuds',\n", + " 'WaveSound Soundbar',\n", + " 'AudioPhonic Turntable'],\n", + " '相机和摄像机': ['FotoSnap DSLR Camera',\n", + " 'ActionCam 4K',\n", + " 'FotoSnap Mirrorless Camera',\n", + " 'ZoomMaster Camcorder',\n", + " 'FotoSnap Instant Camera']}" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import utils_zh\n", + "\n", + "products_and_category = utils_zh.get_products_and_category()\n", + "products_and_category" + ] + }, + { + "cell_type": "markdown", + "id": "d91f5384", + "metadata": { + "height": 30 + }, + "source": [ + "## 一、找出相关产品和类别名称" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "ac683bfb", + "metadata": {}, + "outputs": [], + "source": [ + "def find_category_and_product_v1(user_input,products_and_category):\n", + " \"\"\"\n", + " 从用户输入中获取到产品和类别\n", + "\n", + " 参数：\n", + " user_input：用户的查询\n", + " products_and_category：产品类型和对应产品的字典\n", + " \"\"\"\n", + " \n", + " delimiter = \"####\"\n", + " system_message = f\"\"\"\n", + " 您将提供客户服务查询。\\\n", + " 客户服务查询将用{delimiter}字符分隔。\n", + " 输出一个 Python 列表，列表中的每个对象都是 Json 对象，每个对象的格式如下：\n", + " '类别': <电脑和笔记本, 智能手机和配件, 电视和家庭影院系统, \\\n", + " 游戏机和配件, 音频设备, 相机和摄像机中的一个>,\n", + " 以及\n", + " '名称': <必须在下面允许的产品中找到的产品列表>\n", + " \n", + " 其中类别和产品必须在客户服务查询中找到。\n", + " 如果提到了一个产品，它必须与下面允许的产品列表中的正确类别关联。\n", + " 如果没有找到产品或类别，输出一个空列表。\n", + " \n", + " 根据产品名称和产品类别与客户服务查询的相关性，列出所有相关的产品。\n", + " 不要从产品的名称中假设任何特性或属性，如相对质量或价格。\n", + " \n", + " 允许的产品以 JSON 格式提供。\n", + " 每个项目的键代表类别。\n", + " 每个项目的值是该类别中的产品列表。\n", + " 允许的产品：{products_and_category}\n", + " \n", + " \"\"\"\n", + " \n", + " few_shot_user_1 = \"\"\"我想要最贵的电脑。\"\"\"\n", + " few_shot_assistant_1 = \"\"\" \n", + " [{'category': '电脑和笔记本', \\\n", + "'products': ['TechPro 超极本', 'BlueWave 游戏本', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n", + " \"\"\"\n", + " \n", + " messages = [ \n", + " {'role':'system', 'content': system_message}, \n", + " {'role':'user', 'content': f\"{delimiter}{few_shot_user_1}{delimiter}\"}, \n", + " {'role':'assistant', 'content': few_shot_assistant_1 },\n", + " {'role':'user', 'content': f\"{delimiter}{user_input}{delimiter}\"}, \n", + " ] \n", + " return get_completion_from_messages(messages)" + ] + }, + { + "cell_type": "markdown", + "id": "aca82030", + "metadata": { + "height": 30 + }, + "source": [ + "## 二、在一些查询上进行评估" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "cacb96b2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': '电视和家庭影院系统', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n" + ] + } + ], + "source": [ + "# 第一个评估的查询\n", + "customer_msg_0 = f\"\"\"如果我预算有限，我可以买哪款电视？\"\"\"\n", + "\n", + "products_by_category_0 = find_category_and_product_v1(customer_msg_0,\n", + " products_and_category)\n", + "print(products_by_category_0)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "04364405", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': '智能手机和配件', 'products': ['MobiTech PowerCase', 'SmartX MiniPhone', 'MobiTech Wireless Charger', 'SmartX EarBuds']}]\n" + ] + } + ], + "source": [ + "customer_msg_1 = f\"\"\"我需要一个智能手机的充电器\"\"\"\n", + "\n", + "products_by_category_1 = find_category_and_product_v1(customer_msg_1,\n", + " products_and_category)\n", + "print(products_by_category_1)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "66e9ecd0", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\" \\n [{'category': '电脑和笔记本', 'products': ['TechPro 超极本', 'BlueWave 游戏本', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\"" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "customer_msg_2 = f\"\"\"\n", + "你们有哪些电脑？\"\"\"\n", + "\n", + "products_by_category_2 = find_category_and_product_v1(customer_msg_2,\n", + " products_and_category)\n", + "products_by_category_2" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "112cfd5f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': '智能手机和配件', 'products': ['SmartX ProPhone']}, {'category': '相机和摄像机', 'products': ['FotoSnap DSLR Camera']}]\n", + " \n", + " [{'category': '电视和家庭影院系统', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n" + ] + } + ], + "source": [ + "customer_msg_3 = f\"\"\"\n", + "告诉我关于smartx pro手机和fotosnap相机的信息，那款DSLR的。\n", + "我预算有限，你们有哪些性价比高的电视推荐？\"\"\"\n", + "\n", + "products_by_category_3 = find_category_and_product_v1(customer_msg_3,\n", + " products_and_category)\n", + "print(products_by_category_3)" + ] + }, + { + "cell_type": "markdown", + "id": "d58f15be", + "metadata": {}, + "source": [ + "它看起来像是输出了正确的数据，但没有按照要求的格式输出。这使得将其解析为 Python 字典列表更加困难。" + ] + }, + { + "cell_type": "markdown", + "id": "ff2af235", + "metadata": { + "height": 30 + }, + "source": [ + "## 三、更难的测试用例\n", + "\n", + "找出一些在实际使用中，模型表现不如预期的查询。" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "5b11172f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': '电视和家庭影院系统', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n", + " [{'category': '游戏机和配件', 'products': ['GameSphere X', 'ProGamer Controller', 'GameSphere Y', 'ProGamer Racing Wheel', 'GameSphere VR Headset']}]\n", + " [{'category': '电脑和笔记本', 'products': ['TechPro 超极本', 'BlueWave 游戏本', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n" + ] + } + ], + "source": [ + "customer_msg_4 = f\"\"\"\n", + "告诉我关于CineView电视的信息，那款8K的，还有Gamesphere游戏机，X款的。\n", + "我预算有限，你们有哪些电脑？\"\"\"\n", + "\n", + "products_by_category_4 = find_category_and_product_v1(customer_msg_4,products_and_category)\n", + "print(products_by_category_4)" + ] + }, + { + "cell_type": "markdown", + "id": "92b63d8b", + "metadata": { + "height": 30 + }, + "source": [ + "## 四、修改指令以处理难测试用例" + ] + }, + { + "cell_type": "markdown", + "id": "ddcee6a5", + "metadata": {}, + "source": [ + "我们在提示中添加了以下内容，不要输出任何不在 JSON 格式中的附加文本，并添加了第二个示例，使用用户和助手消息进行 few-shot 提示。" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "d3b183bf", + "metadata": {}, + "outputs": [], + "source": [ + "def find_category_and_product_v2(user_input,products_and_category):\n", + " \"\"\"\n", + " 从用户输入中获取到产品和类别\n", + "\n", + " 添加：不要输出任何不符合 JSON 格式的额外文本。\n", + " 添加了第二个示例（用于 few-shot 提示），用户询问最便宜的计算机。\n", + " 在这两个 few-shot 示例中，显示的响应只是 JSON 格式的完整产品列表。\n", + "\n", + " 参数：\n", + " user_input：用户的查询\n", + " products_and_category：产品类型和对应产品的字典 \n", + " \"\"\"\n", + " delimiter = \"####\"\n", + " system_message = f\"\"\"\n", + " 您将提供客户服务查询。\\\n", + " 客户服务查询将用{delimiter}字符分隔。\n", + " 输出一个 Python列表，列表中的每个对象都是 JSON 对象，每个对象的格式如下：\n", + " '类别': <电脑和笔记本, 智能手机和配件, 电视和家庭影院系统, \\\n", + " 游戏机和配件, 音频设备, 相机和摄像机中的一个>,\n", + " 以及\n", + " '名称': <必须在下面允许的产品中找到的产品列表>\n", + " 不要输出任何不是 JSON 格式的额外文本。\n", + " 输出请求的 JSON 后，不要写任何解释性的文本。\n", + " \n", + " 其中类别和产品必须在客户服务查询中找到。\n", + " 如果提到了一个产品，它必须与下面允许的产品列表中的正确类别关联。\n", + " 如果没有找到产品或类别，输出一个空列表。\n", + " \n", + " 根据产品名称和产品类别与客户服务查询的相关性，列出所有相关的产品。\n", + " 不要从产品的名称中假设任何特性或属性，如相对质量或价格。\n", + " \n", + " 允许的产品以 JSON 格式提供。\n", + " 每个项目的键代表类别。\n", + " 每个项目的值是该类别中的产品列表。\n", + " 允许的产品：{products_and_category}\n", + " \n", + " \"\"\"\n", + " \n", + " few_shot_user_1 = \"\"\"我想要最贵的电脑。你推荐哪款？\"\"\"\n", + " few_shot_assistant_1 = \"\"\" \n", + " [{'category': '电脑和笔记本', \\\n", + "'products': ['TechPro 超极本', 'BlueWave 游戏本', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n", + " \"\"\"\n", + " \n", + " few_shot_user_2 = \"\"\"我想要最便宜的电脑。你推荐哪款？\"\"\"\n", + " few_shot_assistant_2 = \"\"\" \n", + " [{'category': '电脑和笔记本', \\\n", + "'products': ['TechPro 超极本', 'BlueWave 游戏本', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n", + " \"\"\"\n", + " \n", + " messages = [ \n", + " {'role':'system', 'content': system_message}, \n", + " {'role':'user', 'content': f\"{delimiter}{few_shot_user_1}{delimiter}\"}, \n", + " {'role':'assistant', 'content': few_shot_assistant_1 },\n", + " {'role':'user', 'content': f\"{delimiter}{few_shot_user_2}{delimiter}\"}, \n", + " {'role':'assistant', 'content': few_shot_assistant_2 },\n", + " {'role':'user', 'content': f\"{delimiter}{user_input}{delimiter}\"}, \n", + " ] \n", + " return get_completion_from_messages(messages)" + ] + }, + { + "cell_type": "markdown", + "id": "83e8ab86", + "metadata": { + "height": 30 + }, + "source": [ + "## 五、在难测试用例上评估修改后的指令" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "4a547b34", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': '智能手机和配件', 'products': ['SmartX ProPhone']}, {'category': '相机和摄像机', 'products': ['FotoSnap DSLR Camera', 'ActionCam 4K', 'FotoSnap Mirrorless Camera', 'ZoomMaster Camcorder', 'FotoSnap Instant Camera']}, {'category': '电视和家庭影院系统', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n", + " \n" + ] + } + ], + "source": [ + "customer_msg_3 = f\"\"\"\n", + "告诉我关于smartx pro手机和fotosnap相机的信息，那款DSLR的。\n", + "另外，你们有哪些电视？\"\"\"\n", + "\n", + "products_by_category_3 = find_category_and_product_v2(customer_msg_3,\n", + " products_and_category)\n", + "print(products_by_category_3)" + ] + }, + { + "cell_type": "markdown", + "id": "22a0a17b", + "metadata": { + "height": 30 + }, + "source": [ + "## 六、回归测试：验证模型在以前的测试用例上仍然有效\n", + "\n", + "检查并修复模型以提高难以测试的用例效果，同时确保此修正不会对先前的测试用例性能造成负面影响。" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "b5ba773b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': '电视和家庭影院系统', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n" + ] + } + ], + "source": [ + "customer_msg_0 = f\"\"\"如果我预算有限，我可以买哪款电视？\"\"\"\n", + "\n", + "products_by_category_0 = find_category_and_product_v2(customer_msg_0,\n", + " products_and_category)\n", + "print(products_by_category_0)" + ] + }, + { + "cell_type": "markdown", + "id": "4440ce1f", + "metadata": { + "height": 30 + }, + "source": [ + "## 七、收集开发集进行自动化测试" + ] + }, + { + "cell_type": "markdown", + "id": "2af63218", + "metadata": {}, + "source": [ + "当您要调整的开发集不仅仅是一小部分示例时，开始自动化测试过程就变得有用了。" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "8a0b751f", + "metadata": { + "height": 207 + }, + "outputs": [], + "source": [ + "msg_ideal_pairs_set = [\n", + " \n", + " # eg 0\n", + " {'customer_msg':\"\"\"如果我预算有限，我可以买哪种电视？\"\"\",\n", + " 'ideal_answer':{\n", + " '电视和家庭影院系统':set(\n", + " ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']\n", + " )}\n", + " },\n", + "\n", + " # eg 1\n", + " {'customer_msg':\"\"\"我需要一个智能手机的充电器\"\"\",\n", + " 'ideal_answer':{\n", + " '智能手机和配件':set(\n", + " ['MobiTech PowerCase', 'MobiTech Wireless Charger', 'SmartX EarBuds']\n", + " )}\n", + " },\n", + " # eg 2\n", + " {'customer_msg':f\"\"\"你有什么样的电脑\"\"\",\n", + " 'ideal_answer':{\n", + " '电脑和笔记本':set(\n", + " ['TechPro 超极本', 'BlueWave 游戏本', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook'\n", + " ])\n", + " }\n", + " },\n", + "\n", + " # eg 3\n", + " {'customer_msg':f\"\"\"告诉我关于smartx pro手机和fotosnap相机的信息，那款DSLR的。\\\n", + "另外，你们有哪些电视？\"\"\",\n", + " 'ideal_answer':{\n", + " '智能手机和配件':set(\n", + " ['SmartX ProPhone']),\n", + " '相机和摄像机':set(\n", + " ['FotoSnap DSLR Camera']),\n", + " '电视和家庭影院系统':set(\n", + " ['CineView 4K TV', 'SoundMax Home Theater','CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV'])\n", + " }\n", + " }, \n", + " \n", + " # eg 4\n", + " {'customer_msg':\"\"\"告诉我关于CineView电视，那款8K电视、\\\n", + " Gamesphere游戏机和X游戏机的信息。我的预算有限，你们有哪些电脑？\"\"\",\n", + " 'ideal_answer':{\n", + " '电视和家庭影院系统':set(\n", + " ['CineView 8K TV']),\n", + " '游戏机和配件':set(\n", + " ['GameSphere X']),\n", + " '电脑和笔记本':set(\n", + " ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook'])\n", + " }\n", + " },\n", + " \n", + " # eg 5\n", + " {'customer_msg':f\"\"\"你们有哪些智能手机\"\"\",\n", + " 'ideal_answer':{\n", + " '智能手机和配件':set(\n", + " ['SmartX ProPhone', 'MobiTech PowerCase', 'SmartX MiniPhone', 'MobiTech Wireless Charger', 'SmartX EarBuds'\n", + " ])\n", + " }\n", + " },\n", + " # eg 6\n", + " {'customer_msg':f\"\"\"我预算有限。你能向我推荐一些智能手机吗？\"\"\",\n", + " 'ideal_answer':{\n", + " '智能手机和配件':set(\n", + " ['SmartX EarBuds', 'SmartX MiniPhone', 'MobiTech PowerCase', 'SmartX ProPhone', 'MobiTech Wireless Charger']\n", + " )}\n", + " },\n", + "\n", + " # eg 7 # this will output a subset of the ideal answer\n", + " {'customer_msg':f\"\"\"有哪些游戏机适合我喜欢赛车游戏的朋友？\"\"\",\n", + " 'ideal_answer':{\n", + " '游戏机和配件':set([\n", + " 'GameSphere X',\n", + " 'ProGamer Controller',\n", + " 'GameSphere Y',\n", + " 'ProGamer Racing Wheel',\n", + " 'GameSphere VR Headset'\n", + " ])}\n", + " },\n", + " # eg 8\n", + " {'customer_msg':f\"\"\"送给我摄像师朋友什么礼物合适？\"\"\",\n", + " 'ideal_answer': {\n", + " '相机和摄像机':set([\n", + " 'FotoSnap DSLR Camera', 'ActionCam 4K', 'FotoSnap Mirrorless Camera', 'ZoomMaster Camcorder', 'FotoSnap Instant Camera'\n", + " ])}\n", + " },\n", + " \n", + " # eg 9\n", + " {'customer_msg':f\"\"\"我想要一台热水浴缸时光机\"\"\",\n", + " 'ideal_answer': []\n", + " }\n", + " \n", + "]\n" + ] + }, + { + "cell_type": "markdown", + "id": "6e0f1db4", + "metadata": { + "height": 30 + }, + "source": [ + "## 八、通过与理想答案比较来评估测试用例" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "d9530285", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "def eval_response_with_ideal(response,\n", + " ideal,\n", + " debug=False):\n", + " \"\"\"\n", + " 评估回复是否与理想答案匹配\n", + " \n", + " 参数：\n", + " response: 回复的内容\n", + " ideal: 理想的答案\n", + " debug: 是否打印调试信息\n", + " \"\"\"\n", + " if debug:\n", + " print(\"回复：\")\n", + " print(response)\n", + " \n", + " # json.loads() 只能解析双引号，因此此处将单引号替换为双引号\n", + " json_like_str = response.replace(\"'\",'\"')\n", + " \n", + " # 解析为一系列的字典\n", + " l_of_d = json.loads(json_like_str)\n", + " \n", + " # 当响应为空，即没有找到任何商品时\n", + " if l_of_d == [] and ideal == []:\n", + " return 1\n", + " \n", + " # 另外一种异常情况是，标准答案数量与回复答案数量不匹配\n", + " elif l_of_d == [] or ideal == []:\n", + " return 0\n", + " \n", + " # 统计正确答案数量\n", + " correct = 0 \n", + " \n", + " if debug:\n", + " print(\"l_of_d is\")\n", + " print(l_of_d)\n", + "\n", + " # 对每一个问答对 \n", + " for d in l_of_d:\n", + "\n", + " # 获取产品和目录\n", + " cat = d.get('category')\n", + " prod_l = d.get('products')\n", + " # 有获取到产品和目录\n", + " if cat and prod_l:\n", + " # convert list to set for comparison\n", + " prod_set = set(prod_l)\n", + " # get ideal set of products\n", + " ideal_cat = ideal.get(cat)\n", + " if ideal_cat:\n", + " prod_set_ideal = set(ideal.get(cat))\n", + " else:\n", + " if debug:\n", + " print(f\"没有在标准答案中找到目录 {cat}\")\n", + " print(f\"标准答案: {ideal}\")\n", + " continue\n", + " \n", + " if debug:\n", + " print(\"产品集合：\\n\",prod_set)\n", + " print()\n", + " print(\"标准答案的产品集合：\\n\",prod_set_ideal)\n", + "\n", + " # 查找到的产品集合和标准的产品集合一致\n", + " if prod_set == prod_set_ideal:\n", + " if debug:\n", + " print(\"正确\")\n", + " correct +=1\n", + " else:\n", + " print(\"错误\")\n", + " print(f\"产品集合: {prod_set}\")\n", + " print(f\"标准的产品集合: {prod_set_ideal}\")\n", + " if prod_set <= prod_set_ideal:\n", + " print(\"回答是标准答案的一个子集\")\n", + " elif prod_set >= prod_set_ideal:\n", + " print(\"回答是标准答案的一个超集\")\n", + "\n", + " # 计算正确答案数\n", + " pc_correct = correct / len(l_of_d)\n", + " \n", + " return pc_correct" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "e06d9fe3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "用户提问: 有哪些游戏机适合我喜欢赛车游戏的朋友？\n", + "标准答案: {'游戏机和配件': {'ProGamer Racing Wheel', 'ProGamer Controller', 'GameSphere Y', 'GameSphere VR Headset', 'GameSphere X'}}\n" + ] + } + ], + "source": [ + "print(f'用户提问: {msg_ideal_pairs_set[7][\"customer_msg\"]}')\n", + "print(f'标准答案: {msg_ideal_pairs_set[7][\"ideal_answer\"]}')" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "2ff332b4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "回答: \n", + " [{'category': '游戏机和配件', 'products': ['GameSphere X', 'ProGamer Controller', 'GameSphere Y', 'ProGamer Racing Wheel', 'GameSphere VR Headset']}]\n", + " \n" + ] + }, + { + "data": { + "text/plain": [ + "1.0" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = find_category_and_product_v2(msg_ideal_pairs_set[7][\"customer_msg\"],\n", + " products_and_category)\n", + "print(f'回答: {response}')\n", + "\n", + "eval_response_with_ideal(response,\n", + " msg_ideal_pairs_set[7][\"ideal_answer\"])" + ] + }, + { + "cell_type": "markdown", + "id": "d1313b17", + "metadata": { + "height": 30 + }, + "source": [ + "## 九、在所有测试用例上运行评估，并计算正确的用例比例\n", + "\n", + "注意：如果任何 API 调用超时，将无法运行" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "d39407c0", + "metadata": { + "height": 30 + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "示例 0\n", + "0: 1.0\n", + "示例 1\n", + "错误\n", + "产品集合: {'SmartX ProPhone'}\n", + "标准的产品集合: {'MobiTech Wireless Charger', 'SmartX EarBuds', 'MobiTech PowerCase'}\n", + "1: 0.0\n", + "示例 2\n", + "2: 1.0\n", + "示例 3\n", + "3: 1.0\n", + "示例 4\n", + "错误\n", + "产品集合: {'SoundMax Home Theater', 'CineView 8K TV', 'CineView 4K TV', 'CineView OLED TV', 'SoundMax Soundbar'}\n", + "标准的产品集合: {'CineView 8K TV'}\n", + "回答是标准答案的一个超集\n", + "错误\n", + "产品集合: {'ProGamer Racing Wheel', 'ProGamer Controller', 'GameSphere Y', 'GameSphere VR Headset', 'GameSphere X'}\n", + "标准的产品集合: {'GameSphere X'}\n", + "回答是标准答案的一个超集\n", + "错误\n", + "产品集合: {'TechPro 超极本', 'TechPro Desktop', 'BlueWave Chromebook', 'PowerLite Convertible', 'BlueWave 游戏本'}\n", + "标准的产品集合: {'TechPro Desktop', 'BlueWave Chromebook', 'TechPro Ultrabook', 'PowerLite Convertible', 'BlueWave Gaming Laptop'}\n", + "4: 0.0\n", + "示例 5\n", + "错误\n", + "产品集合: {'SmartX ProPhone'}\n", + "标准的产品集合: {'MobiTech Wireless Charger', 'SmartX EarBuds', 'SmartX MiniPhone', 'SmartX ProPhone', 'MobiTech PowerCase'}\n", + "回答是标准答案的一个子集\n", + "5: 0.0\n", + "示例 6\n", + "错误\n", + "产品集合: {'SmartX ProPhone'}\n", + "标准的产品集合: {'MobiTech Wireless Charger', 'SmartX EarBuds', 'SmartX MiniPhone', 'SmartX ProPhone', 'MobiTech PowerCase'}\n", + "回答是标准答案的一个子集\n", + "6: 0.0\n", + "示例 7\n", + "7: 1.0\n", + "示例 8\n", + "8: 1.0\n", + "示例 9\n", + "9: 1\n", + "正确比例为 10: 0.6\n" + ] + } + ], + "source": [ + "import time\n", + "\n", + "score_accum = 0\n", + "for i, pair in enumerate(msg_ideal_pairs_set):\n", + " time.sleep(20)\n", + " print(f\"示例 {i}\")\n", + " \n", + " customer_msg = pair['customer_msg']\n", + " ideal = pair['ideal_answer']\n", + " \n", + " # print(\"Customer message\",customer_msg)\n", + " # print(\"ideal:\",ideal)\n", + " response = find_category_and_product_v2(customer_msg,\n", + " products_and_category)\n", + "\n", + " \n", + " # print(\"products_by_category\",products_by_category)\n", + " score = eval_response_with_ideal(response,ideal,debug=False)\n", + " print(f\"{i}: {score}\")\n", + " score_accum += score\n", + " \n", + "\n", + "n_examples = len(msg_ideal_pairs_set)\n", + "fraction_correct = score_accum / n_examples\n", + "print(f\"正确比例为 {n_examples}: {fraction_correct}\")" + ] + }, + { + "cell_type": "markdown", + "id": "5d885db6", + "metadata": {}, + "source": [ + "使用 Prompt 构建应用程序的工作流程与使用监督学习构建应用程序的工作流程非常不同。\n", + "\n", + "因此，我们认为这是需要记住的一件好事，当您正在构建监督学习模型时，会感觉到迭代速度快了很多。\n", + "\n", + "如果您并未亲身体验，可能会惊叹于仅有手动构建的极少样本，就可以产生高效的评估方法。您可能会认为，仅有 10 个样本是不具备统计意义的。但当您真正运用这种方式时，您可能会对向开发集中添加一些复杂样本所带来的效果提升感到惊讶。\n", + "\n", + "这对于帮助您和您的团队找到有效的 Prompt 和有效的系统非常有帮助。\n", + "\n", + "在本章中，输出可以被定量评估，就像有一个期望的输出一样，您可以判断它是否给出了这个期望的输出。\n", + "\n", + "在下一章中，我们将探讨如何在更加模糊的情况下评估我们的输出。即正确答案可能不那么明确的情况。" + ] + }, + { + "cell_type": "markdown", + "id": "7637f00a", + "metadata": {}, + "source": [ + "## 十、英文版" + ] + }, + { + "cell_type": "markdown", + "id": "9a238fac", + "metadata": {}, + "source": [ + "**1. 找出产品和类别名称**" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "86965e88", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'Computers and Laptops': ['TechPro Ultrabook',\n", + " 'BlueWave Gaming Laptop',\n", + " 'PowerLite Convertible',\n", + " 'TechPro Desktop',\n", + " 'BlueWave Chromebook'],\n", + " 'Smartphones and Accessories': ['SmartX ProPhone',\n", + " 'MobiTech PowerCase',\n", + " 'SmartX MiniPhone',\n", + " 'MobiTech Wireless Charger',\n", + " 'SmartX EarBuds'],\n", + " 'Televisions and Home Theater Systems': ['CineView 4K TV',\n", + " 'SoundMax Home Theater',\n", + " 'CineView 8K TV',\n", + " 'SoundMax Soundbar',\n", + " 'CineView OLED TV'],\n", + " 'Gaming Consoles and Accessories': ['GameSphere X',\n", + " 'ProGamer Controller',\n", + " 'GameSphere Y',\n", + " 'ProGamer Racing Wheel',\n", + " 'GameSphere VR Headset'],\n", + " 'Audio Equipment': ['AudioPhonic Noise-Canceling Headphones',\n", + " 'WaveSound Bluetooth Speaker',\n", + " 'AudioPhonic True Wireless Earbuds',\n", + " 'WaveSound Soundbar',\n", + " 'AudioPhonic Turntable'],\n", + " 'Cameras and Camcorders': ['FotoSnap DSLR Camera',\n", + " 'ActionCam 4K',\n", + " 'FotoSnap Mirrorless Camera',\n", + " 'ZoomMaster Camcorder',\n", + " 'FotoSnap Instant Camera']}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import utils_en\n", + "\n", + "products_and_category = utils_en.get_products_and_category()\n", + "products_and_category" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "34ebb288", + "metadata": {}, + "outputs": [], + "source": [ + "def find_category_and_product_v1(user_input, products_and_category):\n", + " \"\"\"\n", + " 从用户输入中获取到产品和类别\n", + "\n", + " 参数：\n", + " user_input：用户的查询\n", + " products_and_category：产品类型和对应产品的字典\n", + " \"\"\"\n", + "\n", + " # 分隔符\n", + " delimiter = \"####\"\n", + " # 定义的系统信息，陈述了需要 GPT 完成的工作\n", + " system_message = f\"\"\"\n", + " You will be provided with customer service queries. \\\n", + " The customer service query will be delimited with {delimiter} characters.\n", + " Output a Python list of json objects, where each object has the following format:\n", + " 'category': ,\n", + " AND\n", + " 'products': \n", + "\n", + "\n", + " Where the categories and products must be found in the customer service query.\n", + " If a product is mentioned, it must be associated with the correct category in the allowed products list below.\n", + " If no products or categories are found, output an empty list.\n", + " \n", + "\n", + " List out all products that are relevant to the customer service query based on how closely it relates\n", + " to the product name and product category.\n", + " Do not assume, from the name of the product, any features or attributes such as relative quality or price.\n", + "\n", + " The allowed products are provided in JSON format.\n", + " The keys of each item represent the category.\n", + " The values of each item is a list of products that are within that category.\n", + " Allowed products: {products_and_category}\n", + " \n", + "\n", + " \"\"\"\n", + " # 给出几个示例\n", + " few_shot_user_1 = \"\"\"I want the most expensive computer.\"\"\"\n", + " few_shot_assistant_1 = \"\"\" \n", + " [{'category': 'Computers and Laptops', \\\n", + "'products': ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n", + " \"\"\"\n", + " \n", + " messages = [ \n", + " {'role':'system', 'content': system_message}, \n", + " {'role':'user', 'content': f\"{delimiter}{few_shot_user_1}{delimiter}\"}, \n", + " {'role':'assistant', 'content': few_shot_assistant_1 },\n", + " {'role':'user', 'content': f\"{delimiter}{user_input}{delimiter}\"}, \n", + " ] \n", + " return get_completion_from_messages(messages)\n" + ] + }, + { + "cell_type": "markdown", + "id": "acb7d1a0", + "metadata": {}, + "source": [ + "**2. 在一些查询上进行评估**" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "b0d7c381", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': 'Televisions and Home Theater Systems', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n" + ] + } + ], + "source": [ + "# 第一个评估的查询\n", + "customer_msg_0 = f\"\"\"Which TV can I buy if I'm on a budget?\"\"\"\n", + "\n", + "products_by_category_0 = find_category_and_product_v1(customer_msg_0,\n", + " products_and_category)\n", + "print(products_by_category_0)" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "562693ab", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': 'Smartphones and Accessories', 'products': ['MobiTech PowerCase', 'MobiTech Wireless Charger', 'SmartX EarBuds']}]\n", + " \n" + ] + } + ], + "source": [ + "# 第二个评估的查询\n", + "customer_msg_1 = f\"\"\"I need a charger for my smartphone\"\"\"\n", + "\n", + "products_by_category_1 = find_category_and_product_v1(customer_msg_1,\n", + " products_and_category)\n", + "print(products_by_category_1)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "64b81024", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\" \\n [{'category': 'Computers and Laptops', 'products': ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\"" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# 第三个评估查询\n", + "customer_msg_2 = f\"\"\"\n", + "What computers do you have?\"\"\"\n", + "\n", + "products_by_category_2 = find_category_and_product_v1(customer_msg_2,\n", + " products_and_category)\n", + "products_by_category_2" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "4d5dcf7d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': 'Smartphones and Accessories', 'products': ['SmartX ProPhone']}, {'category': 'Cameras and Camcorders', 'products': ['FotoSnap DSLR Camera']}, {'category': 'Televisions and Home Theater Systems', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n" + ] + } + ], + "source": [ + "# 第四个查询，更复杂\n", + "customer_msg_3 = f\"\"\"\n", + "tell me about the smartx pro phone and the fotosnap camera, the dslr one.\n", + "Also, what TVs do you have?\"\"\"\n", + "\n", + "products_by_category_3 = find_category_and_product_v1(customer_msg_3,\n", + " products_and_category)\n", + "print(products_by_category_3)" + ] + }, + { + "cell_type": "markdown", + "id": "c5d18197", + "metadata": {}, + "source": [ + "**3. 更难的测试用例**" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "a3d42e0e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': 'Televisions and Home Theater Systems', 'products': ['CineView 8K TV']}, {'category': 'Gaming Consoles and Accessories', 'products': ['GameSphere X']}, {'category': 'Computers and Laptops', 'products': ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n" + ] + } + ], + "source": [ + "customer_msg_4 = f\"\"\"\n", + "tell me about the CineView TV, the 8K one, Gamesphere console, the X one.\n", + "I'm on a budget, what computers do you have?\"\"\"\n", + "\n", + "products_by_category_4 = find_category_and_product_v1(customer_msg_4,\n", + " products_and_category)\n", + "print(products_by_category_4)" + ] + }, + { + "cell_type": "markdown", + "id": "302afee1", + "metadata": {}, + "source": [ + "**4. 修改指令**" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "33b649e4", + "metadata": {}, + "outputs": [], + "source": [ + "def find_category_and_product_v2(user_input, products_and_category):\n", + " \"\"\"\n", + " 从用户输入中获取到产品和类别\n", + " 添加：不要输出任何不符合 JSON 格式的额外文本。\n", + " 添加了第二个示例（用于 few-shot 提示），用户询问最便宜的计算机。\n", + " 在这两个 few-shot 示例中，显示的响应只是 JSON 格式的完整产品列表。\n", + "\n", + " 参数：\n", + " user_input：用户的查询\n", + " products_and_category：产品类型和对应产品的字典\n", + " \"\"\"\n", + " delimiter = \"####\"\n", + " system_message = f\"\"\"\n", + " You will be provided with customer service queries. \\\n", + " The customer service query will be delimited with {delimiter} characters.\n", + " Output a Python list of JSON objects, where each object has the following format:\n", + " 'category': ,\n", + " AND\n", + " 'products': \n", + " Do not output any additional text that is not in JSON format.\n", + " Do not write any explanatory text after outputting the requested JSON.\n", + "\n", + "\n", + " Where the categories and products must be found in the customer service query.\n", + " If a product is mentioned, it must be associated with the correct category in the allowed products list below.\n", + " If no products or categories are found, output an empty list.\n", + " \n", + "\n", + " List out all products that are relevant to the customer service query based on how closely it relates\n", + " to the product name and product category.\n", + " Do not assume, from the name of the product, any features or attributes such as relative quality or price.\n", + "\n", + " The allowed products are provided in JSON format.\n", + " The keys of each item represent the category.\n", + " The values of each item is a list of products that are within that category.\n", + " Allowed products: {products_and_category}\n", + " \n", + "\n", + " \"\"\"\n", + " \n", + " few_shot_user_1 = \"\"\"I want the most expensive computer. What do you recommend?\"\"\"\n", + " few_shot_assistant_1 = \"\"\" \n", + " [{'category': 'Computers and Laptops', \\\n", + "'products': ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n", + " \"\"\"\n", + " \n", + " few_shot_user_2 = \"\"\"I want the most cheapest computer. What do you recommend?\"\"\"\n", + " few_shot_assistant_2 = \"\"\" \n", + " [{'category': 'Computers and Laptops', \\\n", + "'products': ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook']}]\n", + " \"\"\"\n", + " \n", + " messages = [ \n", + " {'role':'system', 'content': system_message}, \n", + " {'role':'user', 'content': f\"{delimiter}{few_shot_user_1}{delimiter}\"}, \n", + " {'role':'assistant', 'content': few_shot_assistant_1 },\n", + " {'role':'user', 'content': f\"{delimiter}{few_shot_user_2}{delimiter}\"}, \n", + " {'role':'assistant', 'content': few_shot_assistant_2 },\n", + " {'role':'user', 'content': f\"{delimiter}{user_input}{delimiter}\"}, \n", + " ] \n", + " return get_completion_from_messages(messages)\n" + ] + }, + { + "cell_type": "markdown", + "id": "273db18f", + "metadata": {}, + "source": [ + "**5. 进一步评估**" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "f319f19f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': 'Smartphones and Accessories', 'products': ['SmartX ProPhone']}, {'category': 'Cameras and Camcorders', 'products': ['FotoSnap DSLR Camera']}, {'category': 'Televisions and Home Theater Systems', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n" + ] + } + ], + "source": [ + "customer_msg_3 = f\"\"\"\n", + "tell me about the smartx pro phone and the fotosnap camera, the dslr one.\n", + "Also, what TVs do you have?\"\"\"\n", + "\n", + "products_by_category_3 = find_category_and_product_v2(customer_msg_3,\n", + " products_and_category)\n", + "print(products_by_category_3)" + ] + }, + { + "cell_type": "markdown", + "id": "19430988", + "metadata": {}, + "source": [ + "**6. 回归测试**" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "1202f122", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " \n", + " [{'category': 'Televisions and Home Theater Systems', 'products': ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']}]\n", + " \n" + ] + } + ], + "source": [ + "customer_msg_0 = f\"\"\"Which TV can I buy if I'm on a budget?\"\"\"\n", + "\n", + "products_by_category_0 = find_category_and_product_v2(customer_msg_0,\n", + " products_and_category)\n", + "print(products_by_category_0)" + ] + }, + { + "cell_type": "markdown", + "id": "9c594222", + "metadata": {}, + "source": [ + "**7. 自动化测试**" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "24bd6ab4", + "metadata": {}, + "outputs": [], + "source": [ + "msg_ideal_pairs_set = [\n", + " \n", + " # eg 0\n", + " {'customer_msg':\"\"\"Which TV can I buy if I'm on a budget?\"\"\",\n", + " 'ideal_answer':{\n", + " 'Televisions and Home Theater Systems':set(\n", + " ['CineView 4K TV', 'SoundMax Home Theater', 'CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV']\n", + " )}\n", + " },\n", + "\n", + " # eg 1\n", + " {'customer_msg':\"\"\"I need a charger for my smartphone\"\"\",\n", + " 'ideal_answer':{\n", + " 'Smartphones and Accessories':set(\n", + " ['MobiTech PowerCase', 'MobiTech Wireless Charger', 'SmartX EarBuds']\n", + " )}\n", + " },\n", + " # eg 2\n", + " {'customer_msg':f\"\"\"What computers do you have?\"\"\",\n", + " 'ideal_answer':{\n", + " 'Computers and Laptops':set(\n", + " ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook'\n", + " ])\n", + " }\n", + " },\n", + "\n", + " # eg 3\n", + " {'customer_msg':f\"\"\"tell me about the smartx pro phone and \\\n", + " the fotosnap camera, the dslr one.\\\n", + " Also, what TVs do you have?\"\"\",\n", + " 'ideal_answer':{\n", + " 'Smartphones and Accessories':set(\n", + " ['SmartX ProPhone']),\n", + " 'Cameras and Camcorders':set(\n", + " ['FotoSnap DSLR Camera']),\n", + " 'Televisions and Home Theater Systems':set(\n", + " ['CineView 4K TV', 'SoundMax Home Theater','CineView 8K TV', 'SoundMax Soundbar', 'CineView OLED TV'])\n", + " }\n", + " }, \n", + " \n", + " # eg 4\n", + " {'customer_msg':\"\"\"tell me about the CineView TV, the 8K one, Gamesphere console, the X one.\n", + "I'm on a budget, what computers do you have?\"\"\",\n", + " 'ideal_answer':{\n", + " 'Televisions and Home Theater Systems':set(\n", + " ['CineView 8K TV']),\n", + " 'Gaming Consoles and Accessories':set(\n", + " ['GameSphere X']),\n", + " 'Computers and Laptops':set(\n", + " ['TechPro Ultrabook', 'BlueWave Gaming Laptop', 'PowerLite Convertible', 'TechPro Desktop', 'BlueWave Chromebook'])\n", + " }\n", + " },\n", + " \n", + " # eg 5\n", + " {'customer_msg':f\"\"\"What smartphones do you have?\"\"\",\n", + " 'ideal_answer':{\n", + " 'Smartphones and Accessories':set(\n", + " ['SmartX ProPhone', 'MobiTech PowerCase', 'SmartX MiniPhone', 'MobiTech Wireless Charger', 'SmartX EarBuds'\n", + " ])\n", + " }\n", + " },\n", + " # eg 6\n", + " {'customer_msg':f\"\"\"I'm on a budget. Can you recommend some smartphones to me?\"\"\",\n", + " 'ideal_answer':{\n", + " 'Smartphones and Accessories':set(\n", + " ['SmartX EarBuds', 'SmartX MiniPhone', 'MobiTech PowerCase', 'SmartX ProPhone', 'MobiTech Wireless Charger']\n", + " )}\n", + " },\n", + "\n", + " # eg 7 # this will output a subset of the ideal answer\n", + " {'customer_msg':f\"\"\"What Gaming consoles would be good for my friend who is into racing games?\"\"\",\n", + " 'ideal_answer':{\n", + " 'Gaming Consoles and Accessories':set([\n", + " 'GameSphere X',\n", + " 'ProGamer Controller',\n", + " 'GameSphere Y',\n", + " 'ProGamer Racing Wheel',\n", + " 'GameSphere VR Headset'\n", + " ])}\n", + " },\n", + " # eg 8\n", + " {'customer_msg':f\"\"\"What could be a good present for my videographer friend?\"\"\",\n", + " 'ideal_answer': {\n", + " 'Cameras and Camcorders':set([\n", + " 'FotoSnap DSLR Camera', 'ActionCam 4K', 'FotoSnap Mirrorless Camera', 'ZoomMaster Camcorder', 'FotoSnap Instant Camera'\n", + " ])}\n", + " },\n", + " \n", + " # eg 9\n", + " {'customer_msg':f\"\"\"I would like a hot tub time machine.\"\"\",\n", + " 'ideal_answer': []\n", + " }\n", + " \n", + "]\n" + ] + }, + { + "cell_type": "markdown", + "id": "5e8fb19d", + "metadata": {}, + "source": [ + "**8. 与理想答案对比**" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "08432b46", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "def eval_response_with_ideal(response,\n", + " ideal,\n", + " debug=False):\n", + " \"\"\"\n", + " 评估回复是否与理想答案匹配\n", + " \n", + " 参数：\n", + " response: 回复的内容\n", + " ideal: 理想的答案\n", + " debug: 是否打印调试信息\n", + " \"\"\"\n", + " if debug:\n", + " print(\"回复：\")\n", + " print(response)\n", + " \n", + " # json.loads() 只能解析双引号，因此此处将单引号替换为双引号\n", + " json_like_str = response.replace(\"'\",'\"')\n", + " \n", + " # 解析为一系列的字典\n", + " l_of_d = json.loads(json_like_str)\n", + " \n", + " # 当响应为空，即没有找到任何商品时\n", + " if l_of_d == [] and ideal == []:\n", + " return 1\n", + " \n", + " # 另外一种异常情况是，标准答案数量与回复答案数量不匹配\n", + " elif l_of_d == [] or ideal == []:\n", + " return 0\n", + " \n", + " # 统计正确答案数量\n", + " correct = 0 \n", + " \n", + " if debug:\n", + " print(\"l_of_d is\")\n", + " print(l_of_d)\n", + "\n", + " # 对每一个问答对 \n", + " for d in l_of_d:\n", + "\n", + " # 获取产品和目录\n", + " cat = d.get('category')\n", + " prod_l = d.get('products')\n", + " # 有获取到产品和目录\n", + " if cat and prod_l:\n", + " # convert list to set for comparison\n", + " prod_set = set(prod_l)\n", + " # get ideal set of products\n", + " ideal_cat = ideal.get(cat)\n", + " if ideal_cat:\n", + " prod_set_ideal = set(ideal.get(cat))\n", + " else:\n", + " if debug:\n", + " print(f\"没有在标准答案中找到目录 {cat}\")\n", + " print(f\"标准答案: {ideal}\")\n", + " continue\n", + " \n", + " if debug:\n", + " print(\"产品集合：\\n\",prod_set)\n", + " print()\n", + " print(\"标准答案的产品集合：\\n\",prod_set_ideal)\n", + "\n", + " # 查找到的产品集合和标准的产品集合一致\n", + " if prod_set == prod_set_ideal:\n", + " if debug:\n", + " print(\"正确\")\n", + " correct +=1\n", + " else:\n", + " print(\"错误\")\n", + " print(f\"产品集合: {prod_set}\")\n", + " print(f\"标准的产品集合: {prod_set_ideal}\")\n", + " if prod_set <= prod_set_ideal:\n", + " print(\"回答是标准答案的一个子集\")\n", + " elif prod_set >= prod_set_ideal:\n", + " print(\"回答是标准答案的一个超集\")\n", + "\n", + " # 计算正确答案数\n", + " pc_correct = correct / len(l_of_d)\n", + " \n", + " return pc_correct" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "c3a632fc", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "用户提问: What Gaming consoles would be good for my friend who is into racing games?\n", + "标准答案: {'Gaming Consoles and Accessories': {'ProGamer Racing Wheel', 'ProGamer Controller', 'GameSphere Y', 'GameSphere VR Headset', 'GameSphere X'}}\n" + ] + } + ], + "source": [ + "print(f'用户提问: {msg_ideal_pairs_set[7][\"customer_msg\"]}')\n", + "print(f'标准答案: {msg_ideal_pairs_set[7][\"ideal_answer\"]}')" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "0df36913", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "回答: \n", + " [{'category': 'Gaming Consoles and Accessories', 'products': ['GameSphere X', 'ProGamer Controller', 'GameSphere Y', 'ProGamer Racing Wheel', 'GameSphere VR Headset']}]\n", + " \n" + ] + }, + { + "data": { + "text/plain": [ + "1.0" + ] + }, + "execution_count": 37, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response = find_category_and_product_v2(msg_ideal_pairs_set[7][\"customer_msg\"],\n", + " products_and_category)\n", + "print(f'回答: {response}')\n", + "\n", + "eval_response_with_ideal(response,\n", + " msg_ideal_pairs_set[7][\"ideal_answer\"])" + ] + }, + { + "cell_type": "markdown", + "id": "94c0ffd4", + "metadata": {}, + "source": [ + "**9. 计算正确比例**" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "c87f9cfb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "示例 0\n", + "0: 1.0\n", + "示例 1\n", + "错误\n", + "产品集合: {'MobiTech Wireless Charger', 'SmartX EarBuds', 'SmartX MiniPhone', 'SmartX ProPhone', 'MobiTech PowerCase'}\n", + "标准的产品集合: {'MobiTech Wireless Charger', 'SmartX EarBuds', 'MobiTech PowerCase'}\n", + "回答是标准答案的一个超集\n", + "1: 0.0\n", + "示例 2\n", + "2: 1.0\n", + "示例 3\n", + "3: 1.0\n", + "示例 4\n", + "错误\n", + "产品集合: {'SoundMax Home Theater', 'CineView 8K TV', 'CineView 4K TV', 'CineView OLED TV', 'SoundMax Soundbar'}\n", + "标准的产品集合: {'CineView 8K TV'}\n", + "回答是标准答案的一个超集\n", + "错误\n", + "产品集合: {'ProGamer Racing Wheel', 'ProGamer Controller', 'GameSphere Y', 'GameSphere VR Headset', 'GameSphere X'}\n", + "标准的产品集合: {'GameSphere X'}\n", + "回答是标准答案的一个超集\n", + "4: 0.3333333333333333\n", + "示例 5\n", + "5: 1.0\n", + "示例 6\n", + "6: 1.0\n", + "示例 7\n", + "7: 1.0\n", + "示例 8\n", + "8: 1.0\n", + "示例 9\n", + "9: 1\n", + "正确比例为 10: 0.8333333333333334\n" + ] + } + ], + "source": [ + "import time\n", + "\n", + "score_accum = 0\n", + "for i, pair in enumerate(msg_ideal_pairs_set):\n", + " time.sleep(20)\n", + " print(f\"示例 {i}\")\n", + " \n", + " customer_msg = pair['customer_msg']\n", + " ideal = pair['ideal_answer']\n", + " \n", + " # print(\"Customer message\",customer_msg)\n", + " # print(\"ideal:\",ideal)\n", + " response = find_category_and_product_v2(customer_msg,\n", + " products_and_category)\n", + "\n", + " \n", + " # print(\"products_by_category\",products_by_category)\n", + " score = eval_response_with_ideal(response,ideal,debug=False)\n", + " print(f\"{i}: {score}\")\n", + " score_accum += score\n", + " \n", + "\n", + "n_examples = len(msg_ideal_pairs_set)\n", + "fraction_correct = score_accum / n_examples\n", + "print(f\"正确比例为 {n_examples}: {fraction_correct}\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/content/C2 Building Systems with the ChatGPT API/categories_zh.json b/docs/content/C2 Building Systems with the ChatGPT API/categories_zh.json new file mode 100644 index 0000000..8d4fcb7 --- /dev/null +++ b/docs/content/C2 Building Systems with the ChatGPT API/categories_zh.json @@ -0,0 +1,20 @@ +{"计费": + ["取消订阅或升级", + "添加付款方式", + "收费解释", + "争议费用"], + "技术支持": + ["常规故障排除", + "设备兼容性", + "软件更新"], + "账户管理": + ["重置密码", + "更新个人信息", + "关闭账户", + "账户安全"], + "一般咨询": + ["产品信息", + "定价", + "反馈", + "与人工对话"] +} \ No newline at end of file diff --git a/docs/content/C2 Building Systems with the ChatGPT API/products.json b/docs/content/C2 Building Systems with the ChatGPT API/products.json new file mode 100644 index 0000000..64d46cf --- /dev/null +++ b/docs/content/C2 Building Systems with the ChatGPT API/products.json @@ -0,0 +1 @@ +{"TechPro Ultrabook": {"name": "TechPro Ultrabook", "category": "Computers and Laptops", "brand": "TechPro", "model_number": "TP-UB100", "warranty": "1 year", "rating": 4.5, "features": ["13.3-inch display", "8GB RAM", "256GB SSD", "Intel Core i5 processor"], "description": "A sleek and lightweight ultrabook for everyday use.", "price": 799.99}, "BlueWave Gaming Laptop": {"name": "BlueWave Gaming Laptop", "category": "Computers and Laptops", "brand": "BlueWave", "model_number": "BW-GL200", "warranty": "2 years", "rating": 4.7, "features": ["15.6-inch display", "16GB RAM", "512GB SSD", "NVIDIA GeForce RTX 3060"], "description": "A high-performance gaming laptop for an immersive experience.", "price": 1199.99}, "PowerLite Convertible": {"name": "PowerLite Convertible", "category": "Computers and Laptops", "brand": "PowerLite", "model_number": "PL-CV300", "warranty": "1 year", "rating": 4.3, "features": ["14-inch touchscreen", "8GB RAM", "256GB SSD", "360-degree hinge"], "description": "A versatile convertible laptop with a responsive touchscreen.", "price": 699.99}, "TechPro Desktop": {"name": "TechPro Desktop", "category": "Computers and Laptops", "brand": "TechPro", "model_number": "TP-DT500", "warranty": "1 year", "rating": 4.4, "features": ["Intel Core i7 processor", "16GB RAM", "1TB HDD", "NVIDIA GeForce GTX 1660"], "description": "A powerful desktop computer for work and play.", "price": 999.99}, "BlueWave Chromebook": {"name": "BlueWave Chromebook", "category": "Computers and Laptops", "brand": "BlueWave", "model_number": "BW-CB100", "warranty": "1 year", "rating": 4.1, "features": ["11.6-inch display", "4GB RAM", "32GB eMMC", "Chrome OS"], "description": "A compact and affordable Chromebook for everyday tasks.", "price": 249.99}, "SmartX ProPhone": {"name": "SmartX ProPhone", "category": "Smartphones and Accessories", "brand": "SmartX", "model_number": "SX-PP10", "warranty": "1 year", "rating": 4.6, "features": ["6.1-inch display", "128GB storage", "12MP dual camera", "5G"], "description": "A powerful smartphone with advanced camera features.", "price": 899.99}, "MobiTech PowerCase": {"name": "MobiTech PowerCase", "category": "Smartphones and Accessories", "brand": "MobiTech", "model_number": "MT-PC20", "warranty": "1 year", "rating": 4.3, "features": ["5000mAh battery", "Wireless charging", "Compatible with SmartX ProPhone"], "description": "A protective case with built-in battery for extended usage.", "price": 59.99}, "SmartX MiniPhone": {"name": "SmartX MiniPhone", "category": "Smartphones and Accessories", "brand": "SmartX", "model_number": "SX-MP5", "warranty": "1 year", "rating": 4.2, "features": ["4.7-inch display", "64GB storage", "8MP camera", "4G"], "description": "A compact and affordable smartphone for basic tasks.", "price": 399.99}, "MobiTech Wireless Charger": {"name": "MobiTech Wireless Charger", "category": "Smartphones and Accessories", "brand": "MobiTech", "model_number": "MT-WC10", "warranty": "1 year", "rating": 4.5, "features": ["10W fast charging", "Qi-compatible", "LED indicator", "Compact design"], "description": "A convenient wireless charger for a clutter-free workspace.", "price": 29.99}, "SmartX EarBuds": {"name": "SmartX EarBuds", "category": "Smartphones and Accessories", "brand": "SmartX", "model_number": "SX-EB20", "warranty": "1 year", "rating": 4.4, "features": ["True wireless", "Bluetooth 5.0", "Touch controls", "24-hour battery life"], "description": "Experience true wireless freedom with these comfortable earbuds.", "price": 99.99}, "CineView 4K TV": {"name": "CineView 4K TV", "category": "Televisions and Home Theater Systems", "brand": "CineView", "model_number": "CV-4K55", "warranty": "2 years", "rating": 4.8, "features": ["55-inch display", "4K resolution", "HDR", "Smart TV"], "description": "A stunning 4K TV with vibrant colors and smart features.", "price": 599.99}, "SoundMax Home Theater": {"name": "SoundMax Home Theater", "category": "Televisions and Home Theater Systems", "brand": "SoundMax", "model_number": "SM-HT100", "warranty": "1 year", "rating": 4.4, "features": ["5.1 channel", "1000W output", "Wireless subwoofer", "Bluetooth"], "description": "A powerful home theater system for an immersive audio experience.", "price": 399.99}, "CineView 8K TV": {"name": "CineView 8K TV", "category": "Televisions and Home Theater Systems", "brand": "CineView", "model_number": "CV-8K65", "warranty": "2 years", "rating": 4.9, "features": ["65-inch display", "8K resolution", "HDR", "Smart TV"], "description": "Experience the future of television with this stunning 8K TV.", "price": 2999.99}, "SoundMax Soundbar": {"name": "SoundMax Soundbar", "category": "Televisions and Home Theater Systems", "brand": "SoundMax", "model_number": "SM-SB50", "warranty": "1 year", "rating": 4.3, "features": ["2.1 channel", "300W output", "Wireless subwoofer", "Bluetooth"], "description": "Upgrade your TV's audio with this sleek and powerful soundbar.", "price": 199.99}, "CineView OLED TV": {"name": "CineView OLED TV", "category": "Televisions and Home Theater Systems", "brand": "CineView", "model_number": "CV-OLED55", "warranty": "2 years", "rating": 4.7, "features": ["55-inch display", "4K resolution", "HDR", "Smart TV"], "description": "Experience true blacks and vibrant colors with this OLED TV.", "price": 1499.99}, "GameSphere X": {"name": "GameSphere X", "category": "Gaming Consoles and Accessories", "brand": "GameSphere", "model_number": "GS-X", "warranty": "1 year", "rating": 4.9, "features": ["4K gaming", "1TB storage", "Backward compatibility", "Online multiplayer"], "description": "A next-generation gaming console for the ultimate gaming experience.", "price": 499.99}, "ProGamer Controller": {"name": "ProGamer Controller", "category": "Gaming Consoles and Accessories", "brand": "ProGamer", "model_number": "PG-C100", "warranty": "1 year", "rating": 4.2, "features": ["Ergonomic design", "Customizable buttons", "Wireless", "Rechargeable battery"], "description": "A high-quality gaming controller for precision and comfort.", "price": 59.99}, "GameSphere Y": {"name": "GameSphere Y", "category": "Gaming Consoles and Accessories", "brand": "GameSphere", "model_number": "GS-Y", "warranty": "1 year", "rating": 4.8, "features": ["4K gaming", "500GB storage", "Backward compatibility", "Online multiplayer"], "description": "A compact gaming console with powerful performance.", "price": 399.99}, "ProGamer Racing Wheel": {"name": "ProGamer Racing Wheel", "category": "Gaming Consoles and Accessories", "brand": "ProGamer", "model_number": "PG-RW200", "warranty": "1 year", "rating": 4.5, "features": ["Force feedback", "Adjustable pedals", "Paddle shifters", "Compatible with GameSphere X"], "description": "Enhance your racing games with this realistic racing wheel.", "price": 249.99}, "GameSphere VR Headset": {"name": "GameSphere VR Headset", "category": "Gaming Consoles and Accessories", "brand": "GameSphere", "model_number": "GS-VR", "warranty": "1 year", "rating": 4.6, "features": ["Immersive VR experience", "Built-in headphones", "Adjustable headband", "Compatible with GameSphere X"], "description": "Step into the world of virtual reality with this comfortable VR headset.", "price": 299.99}, "AudioPhonic Noise-Canceling Headphones": {"name": "AudioPhonic Noise-Canceling Headphones", "category": "Audio Equipment", "brand": "AudioPhonic", "model_number": "AP-NC100", "warranty": "1 year", "rating": 4.6, "features": ["Active noise-canceling", "Bluetooth", "20-hour battery life", "Comfortable fit"], "description": "Experience immersive sound with these noise-canceling headphones.", "price": 199.99}, "WaveSound Bluetooth Speaker": {"name": "WaveSound Bluetooth Speaker", "category": "Audio Equipment", "brand": "WaveSound", "model_number": "WS-BS50", "warranty": "1 year", "rating": 4.5, "features": ["Portable", "10-hour battery life", "Water-resistant", "Built-in microphone"], "description": "A compact and versatile Bluetooth speaker for music on the go.", "price": 49.99}, "AudioPhonic True Wireless Earbuds": {"name": "AudioPhonic True Wireless Earbuds", "category": "Audio Equipment", "brand": "AudioPhonic", "model_number": "AP-TW20", "warranty": "1 year", "rating": 4.4, "features": ["True wireless", "Bluetooth 5.0", "Touch controls", "18-hour battery life"], "description": "Enjoy music without wires with these comfortable true wireless earbuds.", "price": 79.99}, "WaveSound Soundbar": {"name": "WaveSound Soundbar", "category": "Audio Equipment", "brand": "WaveSound", "model_number": "WS-SB40", "warranty": "1 year", "rating": 4.3, "features": ["2.0 channel", "80W output", "Bluetooth", "Wall-mountable"], "description": "Upgrade your TV's audio with this slim and powerful soundbar.", "price": 99.99}, "AudioPhonic Turntable": {"name": "AudioPhonic Turntable", "category": "Audio Equipment", "brand": "AudioPhonic", "model_number": "AP-TT10", "warranty": "1 year", "rating": 4.2, "features": ["3-speed", "Built-in speakers", "Bluetooth", "USB recording"], "description": "Rediscover your vinyl collection with this modern turntable.", "price": 149.99}, "FotoSnap DSLR Camera": {"name": "FotoSnap DSLR Camera", "category": "Cameras and Camcorders", "brand": "FotoSnap", "model_number": "FS-DSLR200", "warranty": "1 year", "rating": 4.7, "features": ["24.2MP sensor", "1080p video", "3-inch LCD", "Interchangeable lenses"], "description": "Capture stunning photos and videos with this versatile DSLR camera.", "price": 599.99}, "ActionCam 4K": {"name": "ActionCam 4K", "category": "Cameras and Camcorders", "brand": "ActionCam", "model_number": "AC-4K", "warranty": "1 year", "rating": 4.4, "features": ["4K video", "Waterproof", "Image stabilization", "Wi-Fi"], "description": "Record your adventures with this rugged and compact 4K action camera.", "price": 299.99}, "FotoSnap Mirrorless Camera": {"name": "FotoSnap Mirrorless Camera", "category": "Cameras and Camcorders", "brand": "FotoSnap", "model_number": "FS-ML100", "warranty": "1 year", "rating": 4.6, "features": ["20.1MP sensor", "4K video", "3-inch touchscreen", "Interchangeable lenses"], "description": "A compact and lightweight mirrorless camera with advanced features.", "price": 799.99}, "ZoomMaster Camcorder": {"name": "ZoomMaster Camcorder", "category": "Cameras and Camcorders", "brand": "ZoomMaster", "model_number": "ZM-CM50", "warranty": "1 year", "rating": 4.3, "features": ["1080p video", "30x optical zoom", "3-inch LCD", "Image stabilization"], "description": "Capture life's moments with this easy-to-use camcorder.", "price": 249.99}, "FotoSnap Instant Camera": {"name": "FotoSnap Instant Camera", "category": "Cameras and Camcorders", "brand": "FotoSnap", "model_number": "FS-IC10", "warranty": "1 year", "rating": 4.1, "features": ["Instant prints", "Built-in flash", "Selfie mirror", "Battery-powered"], "description": "Create instant memories with this fun and portable instant camera.", "price": 69.99}} \ No newline at end of file diff --git a/docs/content/C2 Building Systems with the ChatGPT API/products_zh.json b/docs/content/C2 Building Systems with the ChatGPT API/products_zh.json new file mode 100644 index 0000000..1c2c544 --- /dev/null +++ b/docs/content/C2 Building Systems with the ChatGPT API/products_zh.json @@ -0,0 +1,336 @@ +{ + "TechPro Ultrabook": { + "名称": "TechPro 超极本", + "类别": "电脑和笔记本", + "品牌": "TechPro", + "型号": "TP-UB100", + "保修期": "1 year", + "评分": 4.5, + "特色": ["13.3-inch display", "8GB RAM", "256GB SSD", "Intel Core i5 处理器"], + "描述": "一款时尚轻便的超极本，适合日常使用。", + "价格": 799.99 + }, + "BlueWave Gaming Laptop": { + "名称": "BlueWave 游戏本", + "类别": "电脑和笔记本", + "品牌": "BlueWave", + "型号": "BW-GL200", + "保修期": "2 years", + "评分": 4.7, + "特色": ["15.6-inch display", "16GB RAM", "512GB SSD", "NVIDIA GeForce RTX 3060"], + "描述": "一款高性能的游戏笔记本电脑，提供沉浸式体验。", + "价格": 1199.99 + }, + "PowerLite Convertible": { + "名称": "PowerLite Convertible", + "类别": "电脑和笔记本", + "品牌": "PowerLite", + "型号": "PL-CV300", + "保修期": "1 year", + "评分": 4.3, + "特色": ["14-inch touchscreen", "8GB RAM", "256GB SSD", "360-degree hinge"], + "描述": "一款多功能的可转换笔记本电脑，具有灵敏的触摸屏。", + "价格": 699.99 + }, + "TechPro Desktop": { + "名称": "TechPro Desktop", + "类别": "电脑和笔记本", + "品牌": "TechPro", + "型号": "TP-DT500", + "保修期": "1 year", + "评分": 4.4, + "特色": ["Intel Core i7 processor", "16GB RAM", "1TB HDD", "NVIDIA GeForce GTX 1660"], + "描述": "一款功能强大的台式电脑，适用于工作和娱乐。", + "价格": 999.99 + }, + "BlueWave Chromebook": { + "名称": "BlueWave Chromebook", + "类别": "电脑和笔记本", + "品牌": "BlueWave", + "型号": "BW-CB100", + "保修期": "1 year", + "评分": 4.1, + "特色": ["11.6-inch display", "4GB RAM", "32GB eMMC", "Chrome OS"], + "描述": "一款紧凑而价格实惠的Chromebook，适用于日常任务。", + "价格": 249.99 + }, + "SmartX ProPhone": { + "名称": "SmartX ProPhone", + "类别": "智能手机和配件", + "品牌": "SmartX", + "型号": "SX-PP10", + "保修期": "1 year", + "评分": 4.6, + "特色": ["6.1-inch display", "128GB storage", "12MP dual camera", "5G"], + "描述": "一款拥有先进摄像功能的强大智能手机。", + "价格": 899.99 + }, + "MobiTech PowerCase": { + "名称": "MobiTech PowerCase", + "类别": "智能手机和配件", + "品牌": "MobiTech", + "型号": "MT-PC20", + "保修期": "1 year", + "评分": 4.3, + "特色": ["5000mAh battery", "Wireless charging", "Compatible with SmartX ProPhone"], + "描述": "一款带有内置电池的保护手机壳，可延长使用时间。", + "价格": 59.99 + }, + "SmartX MiniPhone": { + "名称": "SmartX MiniPhone", + "类别": "智能手机和配件", + "品牌": "SmartX", + "型号": "SX-MP5", + "保修期": "1 year", + "评分": 4.2, + "特色": ["4.7-inch display", "64GB storage", "8MP camera", "4G"], + "描述": "一款紧凑而价格实惠的智能手机，适用于基本任务。", + "价格": 399.99 + }, + "MobiTech Wireless Charger": { + "名称": "MobiTech Wireless Charger", + "类别": "智能手机和配件", + "品牌": "MobiTech", + "型号": "MT-WC10", + "保修期": "1 year", + "评分": 4.5, + "特色": ["10W fast charging", "Qi-compatible", "LED indicator", "Compact design"], + "描述": "一款方便的无线充电器，使工作区域整洁无杂物。", + "价格": 29.99 + }, + "SmartX EarBuds": { + "名称": "SmartX EarBuds", + "类别": "智能手机和配件", + "品牌": "SmartX", + "型号": "SX-EB20", + "保修期": "1 year", + "评分": 4.4, + "特色": ["True wireless", "Bluetooth 5.0", "Touch controls", "24-hour battery life"], + "描述": "通过这些舒适的耳塞体验真正的无线自由。", + "价格": 99.99 + }, + + "CineView 4K TV": { + "名称": "CineView 4K TV", + "类别": "电视和家庭影院系统", + "品牌": "CineView", + "型号": "CV-4K55", + "保修期": "2 years", + "评分": 4.8, + "特色": ["55-inch display", "4K resolution", "HDR", "Smart TV"], + "描述": "一款色彩鲜艳、智能功能丰富的惊艳4K电视。", + "价格": 599.99 + }, + "SoundMax Home Theater": { + "名称": "SoundMax Home Theater", + "类别": "电视和家庭影院系统", + "品牌": "SoundMax", + "型号": "SM-HT100", + "保修期": "1 year", + "评分": 4.4, + "特色": ["5.1 channel", "1000W output", "Wireless subwoofer", "Bluetooth"], + "描述": "一款强大的家庭影院系统，提供沉浸式音频体验。", + "价格": 399.99 + }, + "CineView 8K TV": { + "名称": "CineView 8K TV", + "类别": "电视和家庭影院系统", + "品牌": "CineView", + "型号": "CV-8K65", + "保修期": "2 years", + "评分": 4.9, + "特色": ["65-inch display", "8K resolution", "HDR", "Smart TV"], + "描述": "通过这款惊艳的8K电视，体验未来。", + "价格": 2999.99 + }, + "SoundMax Soundbar": { + "名称": "SoundMax Soundbar", + "类别": "电视和家庭影院系统", + "品牌": "SoundMax", + "型号": "SM-SB50", + "保修期": "1 year", + "评分": 4.3, + "特色": ["2.1 channel", "300W output", "Wireless subwoofer", "Bluetooth"], + "描述": "使用这款时尚而功能强大的声音，升级您电视的音频体验。", + "价格": 199.99 + }, + "CineView OLED TV": { + "名称": "CineView OLED TV", + "类别": "电视和家庭影院系统", + "品牌": "CineView", + "型号": "CV-OLED55", + "保修期": "2 years", + "评分": 4.7, + "特色": ["55-inch display", "4K resolution", "HDR", "Smart TV"], + "描述": "通过这款OLED电视，体验真正的五彩斑斓。", + "价格": 1499.99 + }, + + "GameSphere X": { + "名称": "GameSphere X", + "类别": "游戏机和配件", + "品牌": "GameSphere", + "型号": "GS-X", + "保修期": "1 year", + "评分": 4.9, + "特色": ["4K gaming", "1TB storage", "Backward compatibility", "Online multiplayer"], + "描述": "一款下一代游戏机，提供终极游戏体验。", + "价格": 499.99 + }, + "ProGamer Controller": { + "名称": "ProGamer Controller", + "类别": "游戏机和配件", + "品牌": "ProGamer", + "型号": "PG-C100", + "保修期": "1 year", + "评分": 4.2, + "特色": ["Ergonomic design", "Customizable buttons", "Wireless", "Rechargeable battery"], + "描述": "一款高品质的游戏手柄，提供精准和舒适的操作。", + "价格": 59.99 + }, + "GameSphere Y": { + "名称": "GameSphere Y", + "类别": "游戏机和配件", + "品牌": "GameSphere", + "型号": "GS-Y", + "保修期": "1 year", + "评分": 4.8, + "特色": ["4K gaming", "500GB storage", "Backward compatibility", "Online multiplayer"], + "描述": "一款体积紧凑、性能强劲的游戏机。", + "价格": 399.99 + }, + "ProGamer Racing Wheel": { + "名称": "ProGamer Racing Wheel", + "类别": "游戏机和配件", + "品牌": "ProGamer", + "型号": "PG-RW200", + "保修期": "1 year", + "评分": 4.5, + "特色": ["Force feedback", "Adjustable pedals", "Paddle shifters", "Compatible with GameSphere X"], + "描述": "使用这款逼真的赛车方向盘，提升您的赛车游戏体验。", + "价格": 249.99 + }, + "GameSphere VR Headset": { + "名称": "GameSphere VR Headset", + "类别": "游戏机和配件", + "品牌": "GameSphere", + "型号": "GS-VR", + "保修期": "1 year", + "评分": 4.6, + "特色": ["Immersive VR experience", "Built-in headphones", "Adjustable headband", "Compatible with GameSphere X"], + "描述": "通过这款舒适的VR头戴设备，进入虚拟现实的世界。", + "价格": 299.99 + }, + + "AudioPhonic Noise-Canceling Headphones": { + "名称": "AudioPhonic Noise-Canceling Headphones", + "类别": "音频设备", + "品牌": "AudioPhonic", + "型号": "AP-NC100", + "保修期": "1 year", + "评分": 4.6, + "特色": ["Active noise-canceling", "Bluetooth", "20-hour battery life", "Comfortable fit"], + "描述": "通过这款降噪耳机，体验沉浸式的音效。", + "价格": 199.99 + }, + "WaveSound Bluetooth Speaker": { + "名称": "WaveSound Bluetooth Speaker", + "类别": "音频设备", + "品牌": "WaveSound", + "型号": "WS-BS50", + "保修期": "1 year", + "评分": 4.5, + "特色": ["Portable", "10-hour battery life", "Water-resistant", "Built-in microphone"], + "描述": "一款紧凑而多用途的蓝牙音箱，适用于随时随地收听音乐。", + "价格": 49.99 + }, + "AudioPhonic True Wireless Earbuds": { + "名称": "AudioPhonic True Wireless Earbuds", + "类别": "音频设备", + "品牌": "AudioPhonic", + "型号": "AP-TW20", + "保修期": "1 year", + "评分": 4.4, + "特色": ["True wireless", "Bluetooth 5.0", "Touch controls", "18-hour battery life"], + "描述": "通过这款舒适的真无线耳塞，无需线缆即可享受音乐。", + "价格": 79.99 + }, + "WaveSound Soundbar": { + "名称": "WaveSound Soundbar", + "类别": "音频设备", + "品牌": "WaveSound", + "型号": "WS-SB40", + "保修期": "1 year", + "评分": 4.3, + "特色": ["2.0 channel", "80W output", "Bluetooth", "Wall-mountable"], + "描述": "使用这款纤薄而功能强大的声音吧，升级您电视的音频体验。", + "价格": 99.99 + }, + "AudioPhonic Turntable": { + "名称": "AudioPhonic Turntable", + "类别": "音频设备", + "品牌": "AudioPhonic", + "型号": "AP-TT10", + "保修期": "1 year", + "评分": 4.2, + "特色": ["3-speed", "Built-in speakers", "Bluetooth", "USB recording"], + "描述": "通过这款现代化的唱片机，重拾您的黑胶唱片收藏。", + "价格": 149.99 + }, + + "FotoSnap DSLR Camera": { + "名称": "FotoSnap DSLR Camera", + "类别": "相机和摄像机", + "品牌": "FotoSnap", + "型号": "FS-DSLR200", + "保修期": "1 year", + "评分": 4.7, + "特色": ["24.2MP sensor", "1080p video", "3-inch LCD", "Interchangeable lenses"], + "描述": "使用这款多功能的单反相机，捕捉惊艳的照片和视频。", + "价格": 599.99 + }, + "ActionCam 4K": { + "名称": "ActionCam 4K", + "类别": "相机和摄像机", + "品牌": "ActionCam", + "型号": "AC-4K", + "保修期": "1 year", + "评分": 4.4, + "特色": ["4K video", "Waterproof", "Image stabilization", "Wi-Fi"], + "描述": "使用这款坚固而紧凑的4K运动相机，记录您的冒险旅程。", + "价格": 299.99 + }, + "FotoSnap Mirrorless Camera": { + "名称": "FotoSnap Mirrorless Camera", + "类别": "相机和摄像机", + "品牌": "FotoSnap", + "型号": "FS-ML100", + "保修期": "1 year", + "评分": 4.6, + "特色": ["20.1MP sensor", "4K video", "3-inch touchscreen", "Interchangeable lenses"], + "描述": "一款具有先进功能的小巧轻便的无反相机。", + "价格": 799.99 + }, + "ZoomMaster Camcorder": { + "名称": "ZoomMaster Camcorder", + "类别": "相机和摄像机", + "品牌": "ZoomMaster", + "型号": "ZM-CM50", + "保修期": "1 year", + "评分": 4.3, + "特色": ["1080p video", "30x optical zoom", "3-inch LCD", "Image stabilization"], + "描述": "使用这款易于使用的摄像机，捕捉生活的瞬间。", + "价格": 249.99 + }, + "FotoSnap Instant Camera": { + "名称": "FotoSnap Instant Camera", + "类别": "相机和摄像机", + "品牌": "FotoSnap", + "型号": "FS-IC10", + "保修期": "1 year", + "评分": 4.1, + "特色": ["Instant prints", "Built-in flash", "Selfie mirror", "Battery-powered"], + "描述": "使用这款有趣且便携的即时相机，创造瞬间回忆。", + "价格": 69.99 + } +} \ No newline at end of file diff --git a/docs/content/C2 Building Systems with the ChatGPT API/utils_en.py b/docs/content/C2 Building Systems with the ChatGPT API/utils_en.py new file mode 100644 index 0000000..babed92 --- /dev/null +++ b/docs/content/C2 Building Systems with the ChatGPT API/utils_en.py @@ -0,0 +1,778 @@ + +import json +import openai +from collections import defaultdict + +# 商品和目录的数据文件 +products_file = 'products.json' +categories_file = 'categories.json' + +# 分隔符 +delimiter = "####" +# 第二步（抽取商品）系统信息文本 +step_2_system_message_content = f""" +You will be provided with customer service a conversation. \ +The most recent user query will be delimited with \ +{delimiter} characters. +Output a python list of objects, where each object has \ +the following format: + 'category': , +OR + 'products': + +Where the categories and products must be found in \ +the customer service query. +If a product is mentioned, it must be associated with \ +the correct category in the allowed products list below. +If no products or categories are found, output an \ +empty list. +Only list products and categories that have not already \ +been mentioned and discussed in the earlier parts of \ +the conversation. + +Allowed products: + +Computers and Laptops category: +TechPro Ultrabook +BlueWave Gaming Laptop +PowerLite Convertible +TechPro Desktop +BlueWave Chromebook + +Smartphones and Accessories category: +SmartX ProPhone +MobiTech PowerCase +SmartX MiniPhone +MobiTech Wireless Charger +SmartX EarBuds + +Televisions and Home Theater Systems category: +CineView 4K TV +SoundMax Home Theater +CineView 8K TV +SoundMax Soundbar +CineView OLED TV + +Gaming Consoles and Accessories category: +GameSphere X +ProGamer Controller +GameSphere Y +ProGamer Racing Wheel +GameSphere VR Headset + +Audio Equipment category: +AudioPhonic Noise-Canceling Headphones +WaveSound Bluetooth Speaker +AudioPhonic True Wireless Earbuds +WaveSound Soundbar +AudioPhonic Turntable + +Cameras and Camcorders category: +FotoSnap DSLR Camera +ActionCam 4K +FotoSnap Mirrorless Camera +ZoomMaster Camcorder +FotoSnap Instant Camera + +Only output the list of objects, with nothing else. +""" + +step_2_system_message = {'role':'system', 'content': step_2_system_message_content} + +# 第四步（生成用户回答）的系统信息 +step_4_system_message_content = f""" + You are a customer service assistant for a large electronic store. \ + Respond in a friendly and helpful tone, with VERY concise answers. \ + Make sure to ask the user relevant follow-up questions. +""" + +step_4_system_message = {'role':'system', 'content': step_4_system_message_content} + +# 第六步（验证模型回答）的系统信息 +step_6_system_message_content = f""" + You are an assistant that evaluates whether \ + customer service agent responses sufficiently \ + answer customer questions, and also validates that \ + all the facts the assistant cites from the product \ + information are correct. + The conversation history, product information, user and customer \ + service agent messages will be delimited by \ + 3 backticks, i.e. ```. + Respond with a Y or N character, with no punctuation: + Y - if the output sufficiently answers the question \ + AND the response correctly uses product information + N - otherwise + + Output a single letter only. +""" + +step_6_system_message = {'role':'system', 'content': step_6_system_message_content} + +# 使用 ChatCompletion 接口 +def get_completion_from_messages(messages, model="gpt-3.5-turbo", temperature=0, max_tokens=500): + response = openai.ChatCompletion.create( + model=model, + messages=messages, + temperature=temperature, + max_tokens=max_tokens, + ) + return response.choices[0].message["content"] + +# 创建目录（如果没有本地目录文件，需要创建一份） +def create_categories(): + categories_dict = { + 'Billing': [ + 'Unsubscribe or upgrade', + 'Add a payment method', + 'Explanation for charge', + 'Dispute a charge'], + 'Technical Support':[ + 'General troubleshooting' + 'Device compatibility', + 'Software updates'], + 'Account Management':[ + 'Password reset' + 'Update personal information', + 'Close account', + 'Account security'], + 'General Inquiry':[ + 'Product information' + 'Pricing', + 'Feedback', + 'Speak to a human'] + } + + with open(categories_file, 'w') as file: + json.dump(categories_dict, file) + + return categories_dict + +# 获取目录数据 +def get_categories(): + with open(categories_file, 'r') as file: + categories = json.load(file) + return categories + +# 获取商品列表 +def get_product_list(): + """ + 具体原理参见第四节课 + """ + products = get_products() + product_list = [] + for product in products.keys(): + product_list.append(product) + + return product_list + +# 获取商品和目录 +def get_products_and_category(): + """ + 具体原理参见第五节课 + """ + products = get_products() + products_by_category = defaultdict(list) + for product_name, product_info in products.items(): + category = product_info.get('category') + if category: + products_by_category[category].append(product_info.get('name')) + + return dict(products_by_category) + +# 从商品数据中获取 +def get_products(): + with open(products_file, 'r') as file: + products = json.load(file) + return products + +# 从用户问题中抽取商品和类别 +def find_category_and_product(user_input,products_and_category): + delimiter = "####" + system_message = f""" + You will be provided with customer service queries. \ + The customer service query will be delimited with {delimiter} characters. + Output a python list of json objects, where each object has the following format: + 'category': , + OR + 'products': + + Where the categories and products must be found in the customer service query. + If a product is mentioned, it must be associated with the correct category in the allowed products list below. + If no products or categories are found, output an empty list. + + The allowed products are provided in JSON format. + The keys of each item represent the category. + The values of each item is a list of products that are within that category. + Allowed products: {products_and_category} + + """ + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_input}{delimiter}"}, + ] + return get_completion_from_messages(messages) + +# 相比上一个函数，限制了可获取的商品 +def find_category_and_product_only(user_input,products_and_category): + delimiter = "####" + system_message = f""" + You will be provided with customer service queries. \ + The customer service query will be delimited with {delimiter} characters. + Output a python list of objects, where each object has the following format: + 'category': , + OR + 'products': + + Where the categories and products must be found in the customer service query. + If a product is mentioned, it must be associated with the correct category in the allowed products list below. + If no products or categories are found, output an empty list. + + Allowed products: + Computers and Laptops category: +TechPro Ultrabook +BlueWave Gaming Laptop +PowerLite Convertible +TechPro Desktop +BlueWave Chromebook + +Smartphones and Accessories category: +SmartX ProPhone +MobiTech PowerCase +SmartX MiniPhone +MobiTech Wireless Charger +SmartX EarBuds + +Televisions and Home Theater Systems category: +CineView 4K TV +SoundMax Home Theater +CineView 8K TV +SoundMax Soundbar +CineView OLED TV + +Gaming Consoles and Accessories category: +GameSphere X +ProGamer Controller +GameSphere Y +ProGamer Racing Wheel +GameSphere VR Headset + +Audio Equipment category: +AudioPhonic Noise-Canceling Headphones +WaveSound Bluetooth Speaker +AudioPhonic True Wireless Earbuds +WaveSound Soundbar +AudioPhonic Turntable + +Cameras and Camcorders category: +FotoSnap DSLR Camera +ActionCam 4K +FotoSnap Mirrorless Camera +ZoomMaster Camcorder +FotoSnap Instant Camera + + Only output the list of objects, nothing else. + """ + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_input}{delimiter}"}, + ] + return get_completion_from_messages(messages) + +# 从问题中抽取商品 +def get_products_from_query(user_msg): + """ + 代码来自于第五节课 + """ + products_and_category = get_products_and_category() + delimiter = "####" + system_message = f""" + You will be provided with customer service queries. \ + The customer service query will be delimited with {delimiter} characters. + Output a python list of json objects, where each object has the following format: + 'category': , + OR + 'products': + + Where the categories and products must be found in the customer service query. + If a product is mentioned, it must be associated with the correct category in the allowed products list below. + If no products or categories are found, output an empty list. + + The allowed products are provided in JSON format. + The keys of each item represent the category. + The values of each item is a list of products that are within that category. + Allowed products: {products_and_category} + + """ + + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_msg}{delimiter}"}, + ] + category_and_product_response = get_completion_from_messages(messages) + + return category_and_product_response + + +# 商品信息的搜索 +def get_product_by_name(name): + products = get_products() + return products.get(name, None) + +def get_products_by_category(category): + products = get_products() + return [product for product in products.values() if product["category"] == category] + +def get_mentioned_product_info(data_list): + """ + 具体原理参见第五、六节课 + """ + product_info_l = [] + + if data_list is None: + return product_info_l + + for data in data_list: + try: + if "products" in data: + products_list = data["products"] + for product_name in products_list: + product = get_product_by_name(product_name) + if product: + product_info_l.append(product) + else: + print(f"Error: Product '{product_name}' not found") + elif "category" in data: + category_name = data["category"] + category_products = get_products_by_category(category_name) + for product in category_products: + product_info_l.append(product) + else: + print("Error: Invalid object format") + except Exception as e: + print(f"Error: {e}") + + return product_info_l + + +# 以下函数原理参见第五节课 +def read_string_to_list(input_string): + if input_string is None: + return None + + try: + input_string = input_string.replace("'", "\"") # Replace single quotes with double quotes for valid JSON + data = json.loads(input_string) + return data + except json.JSONDecodeError: + print("Error: Invalid JSON string") + return None + +def generate_output_string(data_list): + output_string = "" + + if data_list is None: + return output_string + + for data in data_list: + try: + if "products" in data: + products_list = data["products"] + for product_name in products_list: + product = get_product_by_name(product_name) + if product: + output_string += json.dumps(product, indent=4) + "\n" + else: + print(f"Error: Product '{product_name}' not found") + elif "category" in data: + category_name = data["category"] + category_products = get_products_by_category(category_name) + for product in category_products: + output_string += json.dumps(product, indent=4) + "\n" + else: + print("Error: Invalid object format") + except Exception as e: + print(f"Error: {e}") + + return output_string + +# Example usage: +#product_information_for_user_message_1 = generate_output_string(category_and_product_list) +#print(product_information_for_user_message_1) +# 回答用户问题 +def answer_user_msg(user_msg,product_info): + """ + 代码参见第五节课 + """ + delimiter = "####" + system_message = f""" + You are a customer service assistant for a large electronic store. \ + Respond in a friendly and helpful tone, with concise answers. \ + Make sure to ask the user relevant follow up questions. + """ + # user_msg = f""" + # tell me about the smartx pro phone and the fotosnap camera, the dslr one. Also what tell me about your tvs""" + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_msg}{delimiter}"}, + {'role':'assistant', 'content': f"Relevant product information:\n{product_info}"}, + ] + response = get_completion_from_messages(messages) + return response + +# 创建并存入商品数据 +def create_products(): + """ + Create products dictionary and save it to a file named products.json + """ + # product information + # fun fact: all these products are fake and were generated by a language model + products = { + "TechPro Ultrabook": { + "name": "TechPro Ultrabook", + "category": "Computers and Laptops", + "brand": "TechPro", + "model_number": "TP-UB100", + "warranty": "1 year", + "rating": 4.5, + "features": ["13.3-inch display", "8GB RAM", "256GB SSD", "Intel Core i5 processor"], + "description": "A sleek and lightweight ultrabook for everyday use.", + "price": 799.99 + }, + "BlueWave Gaming Laptop": { + "name": "BlueWave Gaming Laptop", + "category": "Computers and Laptops", + "brand": "BlueWave", + "model_number": "BW-GL200", + "warranty": "2 years", + "rating": 4.7, + "features": ["15.6-inch display", "16GB RAM", "512GB SSD", "NVIDIA GeForce RTX 3060"], + "description": "A high-performance gaming laptop for an immersive experience.", + "price": 1199.99 + }, + "PowerLite Convertible": { + "name": "PowerLite Convertible", + "category": "Computers and Laptops", + "brand": "PowerLite", + "model_number": "PL-CV300", + "warranty": "1 year", + "rating": 4.3, + "features": ["14-inch touchscreen", "8GB RAM", "256GB SSD", "360-degree hinge"], + "description": "A versatile convertible laptop with a responsive touchscreen.", + "price": 699.99 + }, + "TechPro Desktop": { + "name": "TechPro Desktop", + "category": "Computers and Laptops", + "brand": "TechPro", + "model_number": "TP-DT500", + "warranty": "1 year", + "rating": 4.4, + "features": ["Intel Core i7 processor", "16GB RAM", "1TB HDD", "NVIDIA GeForce GTX 1660"], + "description": "A powerful desktop computer for work and play.", + "price": 999.99 + }, + "BlueWave Chromebook": { + "name": "BlueWave Chromebook", + "category": "Computers and Laptops", + "brand": "BlueWave", + "model_number": "BW-CB100", + "warranty": "1 year", + "rating": 4.1, + "features": ["11.6-inch display", "4GB RAM", "32GB eMMC", "Chrome OS"], + "description": "A compact and affordable Chromebook for everyday tasks.", + "price": 249.99 + }, + "SmartX ProPhone": { + "name": "SmartX ProPhone", + "category": "Smartphones and Accessories", + "brand": "SmartX", + "model_number": "SX-PP10", + "warranty": "1 year", + "rating": 4.6, + "features": ["6.1-inch display", "128GB storage", "12MP dual camera", "5G"], + "description": "A powerful smartphone with advanced camera features.", + "price": 899.99 + }, + "MobiTech PowerCase": { + "name": "MobiTech PowerCase", + "category": "Smartphones and Accessories", + "brand": "MobiTech", + "model_number": "MT-PC20", + "warranty": "1 year", + "rating": 4.3, + "features": ["5000mAh battery", "Wireless charging", "Compatible with SmartX ProPhone"], + "description": "A protective case with built-in battery for extended usage.", + "price": 59.99 + }, + "SmartX MiniPhone": { + "name": "SmartX MiniPhone", + "category": "Smartphones and Accessories", + "brand": "SmartX", + "model_number": "SX-MP5", + "warranty": "1 year", + "rating": 4.2, + "features": ["4.7-inch display", "64GB storage", "8MP camera", "4G"], + "description": "A compact and affordable smartphone for basic tasks.", + "price": 399.99 + }, + "MobiTech Wireless Charger": { + "name": "MobiTech Wireless Charger", + "category": "Smartphones and Accessories", + "brand": "MobiTech", + "model_number": "MT-WC10", + "warranty": "1 year", + "rating": 4.5, + "features": ["10W fast charging", "Qi-compatible", "LED indicator", "Compact design"], + "description": "A convenient wireless charger for a clutter-free workspace.", + "price": 29.99 + }, + "SmartX EarBuds": { + "name": "SmartX EarBuds", + "category": "Smartphones and Accessories", + "brand": "SmartX", + "model_number": "SX-EB20", + "warranty": "1 year", + "rating": 4.4, + "features": ["True wireless", "Bluetooth 5.0", "Touch controls", "24-hour battery life"], + "description": "Experience true wireless freedom with these comfortable earbuds.", + "price": 99.99 + }, + + "CineView 4K TV": { + "name": "CineView 4K TV", + "category": "Televisions and Home Theater Systems", + "brand": "CineView", + "model_number": "CV-4K55", + "warranty": "2 years", + "rating": 4.8, + "features": ["55-inch display", "4K resolution", "HDR", "Smart TV"], + "description": "A stunning 4K TV with vibrant colors and smart features.", + "price": 599.99 + }, + "SoundMax Home Theater": { + "name": "SoundMax Home Theater", + "category": "Televisions and Home Theater Systems", + "brand": "SoundMax", + "model_number": "SM-HT100", + "warranty": "1 year", + "rating": 4.4, + "features": ["5.1 channel", "1000W output", "Wireless subwoofer", "Bluetooth"], + "description": "A powerful home theater system for an immersive audio experience.", + "price": 399.99 + }, + "CineView 8K TV": { + "name": "CineView 8K TV", + "category": "Televisions and Home Theater Systems", + "brand": "CineView", + "model_number": "CV-8K65", + "warranty": "2 years", + "rating": 4.9, + "features": ["65-inch display", "8K resolution", "HDR", "Smart TV"], + "description": "Experience the future of television with this stunning 8K TV.", + "price": 2999.99 + }, + "SoundMax Soundbar": { + "name": "SoundMax Soundbar", + "category": "Televisions and Home Theater Systems", + "brand": "SoundMax", + "model_number": "SM-SB50", + "warranty": "1 year", + "rating": 4.3, + "features": ["2.1 channel", "300W output", "Wireless subwoofer", "Bluetooth"], + "description": "Upgrade your TV's audio with this sleek and powerful soundbar.", + "price": 199.99 + }, + "CineView OLED TV": { + "name": "CineView OLED TV", + "category": "Televisions and Home Theater Systems", + "brand": "CineView", + "model_number": "CV-OLED55", + "warranty": "2 years", + "rating": 4.7, + "features": ["55-inch display", "4K resolution", "HDR", "Smart TV"], + "description": "Experience true blacks and vibrant colors with this OLED TV.", + "price": 1499.99 + }, + + "GameSphere X": { + "name": "GameSphere X", + "category": "Gaming Consoles and Accessories", + "brand": "GameSphere", + "model_number": "GS-X", + "warranty": "1 year", + "rating": 4.9, + "features": ["4K gaming", "1TB storage", "Backward compatibility", "Online multiplayer"], + "description": "A next-generation gaming console for the ultimate gaming experience.", + "price": 499.99 + }, + "ProGamer Controller": { + "name": "ProGamer Controller", + "category": "Gaming Consoles and Accessories", + "brand": "ProGamer", + "model_number": "PG-C100", + "warranty": "1 year", + "rating": 4.2, + "features": ["Ergonomic design", "Customizable buttons", "Wireless", "Rechargeable battery"], + "description": "A high-quality gaming controller for precision and comfort.", + "price": 59.99 + }, + "GameSphere Y": { + "name": "GameSphere Y", + "category": "Gaming Consoles and Accessories", + "brand": "GameSphere", + "model_number": "GS-Y", + "warranty": "1 year", + "rating": 4.8, + "features": ["4K gaming", "500GB storage", "Backward compatibility", "Online multiplayer"], + "description": "A compact gaming console with powerful performance.", + "price": 399.99 + }, + "ProGamer Racing Wheel": { + "name": "ProGamer Racing Wheel", + "category": "Gaming Consoles and Accessories", + "brand": "ProGamer", + "model_number": "PG-RW200", + "warranty": "1 year", + "rating": 4.5, + "features": ["Force feedback", "Adjustable pedals", "Paddle shifters", "Compatible with GameSphere X"], + "description": "Enhance your racing games with this realistic racing wheel.", + "price": 249.99 + }, + "GameSphere VR Headset": { + "name": "GameSphere VR Headset", + "category": "Gaming Consoles and Accessories", + "brand": "GameSphere", + "model_number": "GS-VR", + "warranty": "1 year", + "rating": 4.6, + "features": ["Immersive VR experience", "Built-in headphones", "Adjustable headband", "Compatible with GameSphere X"], + "description": "Step into the world of virtual reality with this comfortable VR headset.", + "price": 299.99 + }, + + "AudioPhonic Noise-Canceling Headphones": { + "name": "AudioPhonic Noise-Canceling Headphones", + "category": "Audio Equipment", + "brand": "AudioPhonic", + "model_number": "AP-NC100", + "warranty": "1 year", + "rating": 4.6, + "features": ["Active noise-canceling", "Bluetooth", "20-hour battery life", "Comfortable fit"], + "description": "Experience immersive sound with these noise-canceling headphones.", + "price": 199.99 + }, + "WaveSound Bluetooth Speaker": { + "name": "WaveSound Bluetooth Speaker", + "category": "Audio Equipment", + "brand": "WaveSound", + "model_number": "WS-BS50", + "warranty": "1 year", + "rating": 4.5, + "features": ["Portable", "10-hour battery life", "Water-resistant", "Built-in microphone"], + "description": "A compact and versatile Bluetooth speaker for music on the go.", + "price": 49.99 + }, + "AudioPhonic True Wireless Earbuds": { + "name": "AudioPhonic True Wireless Earbuds", + "category": "Audio Equipment", + "brand": "AudioPhonic", + "model_number": "AP-TW20", + "warranty": "1 year", + "rating": 4.4, + "features": ["True wireless", "Bluetooth 5.0", "Touch controls", "18-hour battery life"], + "description": "Enjoy music without wires with these comfortable true wireless earbuds.", + "price": 79.99 + }, + "WaveSound Soundbar": { + "name": "WaveSound Soundbar", + "category": "Audio Equipment", + "brand": "WaveSound", + "model_number": "WS-SB40", + "warranty": "1 year", + "rating": 4.3, + "features": ["2.0 channel", "80W output", "Bluetooth", "Wall-mountable"], + "description": "Upgrade your TV's audio with this slim and powerful soundbar.", + "price": 99.99 + }, + "AudioPhonic Turntable": { + "name": "AudioPhonic Turntable", + "category": "Audio Equipment", + "brand": "AudioPhonic", + "model_number": "AP-TT10", + "warranty": "1 year", + "rating": 4.2, + "features": ["3-speed", "Built-in speakers", "Bluetooth", "USB recording"], + "description": "Rediscover your vinyl collection with this modern turntable.", + "price": 149.99 + }, + + "FotoSnap DSLR Camera": { + "name": "FotoSnap DSLR Camera", + "category": "Cameras and Camcorders", + "brand": "FotoSnap", + "model_number": "FS-DSLR200", + "warranty": "1 year", + "rating": 4.7, + "features": ["24.2MP sensor", "1080p video", "3-inch LCD", "Interchangeable lenses"], + "description": "Capture stunning photos and videos with this versatile DSLR camera.", + "price": 599.99 + }, + "ActionCam 4K": { + "name": "ActionCam 4K", + "category": "Cameras and Camcorders", + "brand": "ActionCam", + "model_number": "AC-4K", + "warranty": "1 year", + "rating": 4.4, + "features": ["4K video", "Waterproof", "Image stabilization", "Wi-Fi"], + "description": "Record your adventures with this rugged and compact 4K action camera.", + "price": 299.99 + }, + "FotoSnap Mirrorless Camera": { + "name": "FotoSnap Mirrorless Camera", + "category": "Cameras and Camcorders", + "brand": "FotoSnap", + "model_number": "FS-ML100", + "warranty": "1 year", + "rating": 4.6, + "features": ["20.1MP sensor", "4K video", "3-inch touchscreen", "Interchangeable lenses"], + "description": "A compact and lightweight mirrorless camera with advanced features.", + "price": 799.99 + }, + "ZoomMaster Camcorder": { + "name": "ZoomMaster Camcorder", + "category": "Cameras and Camcorders", + "brand": "ZoomMaster", + "model_number": "ZM-CM50", + "warranty": "1 year", + "rating": 4.3, + "features": ["1080p video", "30x optical zoom", "3-inch LCD", "Image stabilization"], + "description": "Capture life's moments with this easy-to-use camcorder.", + "price": 249.99 + }, + "FotoSnap Instant Camera": { + "name": "FotoSnap Instant Camera", + "category": "Cameras and Camcorders", + "brand": "FotoSnap", + "model_number": "FS-IC10", + "warranty": "1 year", + "rating": 4.1, + "features": ["Instant prints", "Built-in flash", "Selfie mirror", "Battery-powered"], + "description": "Create instant memories with this fun and portable instant camera.", + "price": 69.99 + } + } + + products_file = 'products.json' + with open(products_file, 'w') as file: + json.dump(products, file) + + return products \ No newline at end of file diff --git a/docs/content/C2 Building Systems with the ChatGPT API/utils_zh.py b/docs/content/C2 Building Systems with the ChatGPT API/utils_zh.py new file mode 100644 index 0000000..19ddc0a --- /dev/null +++ b/docs/content/C2 Building Systems with the ChatGPT API/utils_zh.py @@ -0,0 +1,760 @@ + +import json +import openai +from collections import defaultdict + +# 商品和目录的数据文件 +products_file = 'products_zh.json' +categories_file = 'categories_zh.json' + +# 分隔符 +delimiter = "####" +# 第二步（抽取商品）系统信息文本 +step_2_system_message_content = f""" +您将获得一次客户服务对话。最近的用户查询将使用{delimiter}字符进行分隔。 + +输出一个Python对象列表，其中每个对象具有以下格式： +'category': <包括以下几个类别：Computers and Laptops、martphones and Accessories、elevisions and Home Theater Systems、elevisions and Home Theater Systems、elevisions and Home Theater Systems、'category': <包括以下几个类别：Computers and Laptops、martphones and Accessories、elevisions and Home Theater Systems、elevisions and Home Theater Systems、elevisions and Home Theater Systems、相机和摄像机>, +或者 +'products': <必须是下面的允许产品列表中找到的产品> + +类别和产品必须在客户服务查询中找到。 +如果提到了产品，它必须与下面的允许产品列表中的正确类别相关联。 +如果未找到任何产品或类别，请输出一个空列表。 +只列出之前对话的早期部分未提及和讨论的产品和类别。 + +允许的产品： + +Computers and Laptops类别： +TechPro Ultrabook +BlueWave Gaming Laptop +PowerLite Convertible +TechPro Desktop +BlueWave Chromebook + +Smartphones and Accessories类别： +SmartX ProPhone +MobiTech PowerCase +SmartX MiniPhone +MobiTech Wireless Charger +SmartX EarBuds + +Televisions and Home Theater Systems类别： +CineView 4K TV +SoundMax Home Theater +CineView 8K TV +SoundMax Soundbar +CineView OLED TV + +Gaming Consoles and Accessories类别： +GameSphere X +ProGamer Controller +GameSphere Y +ProGamer Racing Wheel +GameSphere VR Headset + +Audio Equipment类别： +AudioPhonic Noise-Canceling Headphones +WaveSound Bluetooth Speaker +AudioPhonic True Wireless Earbuds +WaveSound Soundbar +AudioPhonic Turntable + +Cameras and Camcorders类别： +FotoSnap DSLR Camera +ActionCam 4K +FotoSnap Mirrorless Camera +ZoomMaster Camcorder +FotoSnap Instant Camera + +只输出对象列表，不包含其他内容。 +""" + +step_2_system_message = {'role':'system', 'content': step_2_system_message_content} + +# 第四步（生成用户回答）的系统信息 +step_4_system_message_content = f""" + 你是一家大型电子商店的客户服务助理。 + 以友好和乐于助人的语气回答，回答保持简洁明了。 + 确保让用户提出相关的后续问题。 +""" + +step_4_system_message = {'role':'system', 'content': step_4_system_message_content} + +# 第六步（验证模型回答）的系统信息 +step_6_system_message_content = f""" + 你是一个助手，评估客户服务代理的回答是否足够回答客户的问题，并验证助手从产品信息中引用的所有事实是否正确。 + 对话历史、产品信息、用户和客户服务代理的消息将用```进行分隔。 + 请用一个字母回答，不带标点符号： + Y - 如果输出足够回答问题，并且回答正确使用了产品信息 + N - 输出不足够回答问题，或者没有正确使用产品信息 + + 只输出一个字母。 +""" + +step_6_system_message = {'role':'system', 'content': step_6_system_message_content} + +# 使用 ChatCompletion 接口 +def get_completion_from_messages(messages, model="gpt-3.5-turbo", temperature=0, max_tokens=500): + response = openai.ChatCompletion.create( + model=model, + messages=messages, + temperature=temperature, + max_tokens=max_tokens, + ) + return response.choices[0].message["content"] + +# 创建目录（如果没有本地目录文件，需要创建一份） +def create_categories(): + categories_dict = { + 'Billing': [ + 'Unsubscribe or upgrade', + 'Add a payment method', + 'Explanation for charge', + 'Dispute a charge'], + 'Technical Support':[ + 'General troubleshooting', + 'Device compatibility', + 'Software updates'], + 'Account Management':[ + 'Password reset', + 'Update personal information', + 'Close account', + 'Account security'], + 'General Inquiry':[ + 'Product information', + 'Pricing', + 'Feedback', + 'Speak to a human'] + } + + with open(categories_file, 'w') as file: + json.dump(categories_dict, file) + + return categories_dict + +# 获取目录数据 +def get_categories(): + with open(categories_file, 'r') as file: + categories = json.load(file) + return categories + +# 获取商品列表 +def get_product_list(): + """ + 具体原理参见第四节课 + """ + products = get_products() + product_list = [] + for product in products.keys(): + product_list.append(product) + + return product_list + +# 获取商品和目录 +def get_products_and_category(): + """ + 具体原理参见第五节课 + """ + products = get_products() + products_by_category = defaultdict(list) + for product_name, product_info in products.items(): + category = product_info.get('类别') + if category: + products_by_category[category].append(product_info.get('名称')) + + return dict(products_by_category) + +# 从商品数据中获取 +def get_products(): + with open(products_file, 'r') as file: + products = json.load(file) + return products + +# 从用户问题中抽取商品和类别 +def find_category_and_product(user_input,products_and_category): + delimiter = "####" + system_message = f""" + 您将获得客户服务查询。 + 客户服务查询将使用{delimiter}字符分隔。 + 输出一个可解析的Python列表，列表每一个元素是一个JSON对象，每个对象具有以下格式： + 'category': <包括以下几个类别：Computers and Laptops，Smartphones and Accessories，Televisions and Home Theater Systems，Gaming Consoles and Accessories，Audio Equipment，Cameras and Camcorders> + 以及 + 'products': <必须是下面的允许产品列表中找到的产品列表> + + 其中类别和产品必须在客户服务查询中找到。 + 如果提到了产品，则必须将其与允许产品列表中的正确类别关联。 + 如果未找到任何产品或类别，则输出一个空列表。 + 除了列表外，不要输出其他任何信息！ + + 允许的产品以JSON格式提供。 + 每个项的键表示类别。 + 每个项的值是该类别中的产品列表。 + 允许的产品：{products_and_category} + + """ + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_input}{delimiter}"}, + ] + return get_completion_from_messages(messages) + +# 相比上一个函数，可获取的商品直接在 template 中限定 +def find_category_and_product_only(user_input,products_and_category): + delimiter = "####" + system_message = f""" +您将获得客户服务查询。 +客户服务查询将使用{delimiter}字符作为分隔符。 +请仅输出一个可解析的Python列表，列表每一个元素是一个JSON对象，每个对象具有以下格式： +'category': <包括以下几个类别：Computers and Laptops、Smartphones and Accessories、Televisions and Home Theater Systems、Gaming Consoles and Accessories、Audio Equipment、Cameras and Camcorders>, +以及 +'products': <必须是下面的允许产品列表中找到的产品列表> + +类别和产品必须在客户服务查询中找到。 +如果提到了某个产品，它必须与允许产品列表中的正确类别关联。 +如果未找到任何产品或类别，则输出一个空列表。 +除了列表外，不要输出其他任何信息！ + +允许的产品： + +Computers and Laptops category: +TechPro Ultrabook +BlueWave Gaming Laptop +PowerLite Convertible +TechPro Desktop +BlueWave Chromebook + +Smartphones and Accessories category: +SmartX ProPhone +MobiTech PowerCase +SmartX MiniPhone +MobiTech Wireless Charger +SmartX EarBuds + +Televisions and Home Theater Systems category: +CineView 4K TV +SoundMax Home Theater +CineView 8K TV +SoundMax Soundbar +CineView OLED TV + +Gaming Consoles and Accessories category: +GameSphere X +ProGamer Controller +GameSphere Y +ProGamer Racing Wheel +GameSphere VR Headset + +Audio Equipment category: +AudioPhonic Noise-Canceling Headphones +WaveSound Bluetooth Speaker +AudioPhonic True Wireless Earbuds +WaveSound Soundbar +AudioPhonic Turntable + +Cameras and Camcorders category: +FotoSnap DSLR Camera +ActionCam 4K +FotoSnap Mirrorless Camera +ZoomMaster Camcorder +FotoSnap Instant Camera + +只输出对象列表，不包含其他内容。 + """ + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_input}{delimiter}"}, + ] + return get_completion_from_messages(messages) + +# 从问题中抽取商品 +def get_products_from_query(user_msg): + """ + 代码来自于第五节课 + """ + products_and_category = get_products_and_category() + delimiter = "####" + system_message = f""" + 您将获得客户服务查询。 + 客户服务查询将使用{delimiter}字符作为分隔符。 + 请仅输出一个可解析的Python列表，列表每一个元素是一个JSON对象，每个对象具有以下格式： + 'category': <包括以下几个类别：Computers and Laptops、Smartphones and Accessories、Televisions and Home Theater Systems、Gaming Consoles and Accessories、Audio Equipment、Cameras and Camcorders>, + 以及 + 'products': <必须是下面的允许产品列表中找到的产品列表> + + 类别和产品必须在客户服务查询中找到。 + 如果提到了某个产品，它必须与允许产品列表中的正确类别关联。 + 如果未找到任何产品或类别，则输出一个空列表。 + 除了列表外，不要输出其他任何信息！ + + 允许的产品以JSON格式提供。 + 每个项目的键表示类别。 + 每个项目的值是该类别中的产品列表。 + + 以下是允许的产品：{products_and_category} + + """ + + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_msg}{delimiter}"}, + ] + category_and_product_response = get_completion_from_messages(messages) + + return category_and_product_response + + +# 商品信息的搜索 +def get_product_by_name(name): + products = get_products() + return products.get(name, None) + +def get_products_by_category(category): + products = get_products() + return [product for product in products.values() if product["category"] == category] + +def get_mentioned_product_info(data_list): + """ + 具体原理参见第五、六节课 + """ + product_info_l = [] + + if data_list is None: + return product_info_l + + for data in data_list: + try: + if "products" in data: + products_list = data["products"] + for product_name in products_list: + product = get_product_by_name(product_name) + if product: + product_info_l.append(product) + else: + print(f"错误: 商品 '{product_name}' 未找到") + elif "category" in data: + category_name = data["category"] + category_products = get_products_by_category(category_name) + for product in category_products: + product_info_l.append(product) + else: + print("错误：非法的商品格式") + except Exception as e: + print(f"Error: {e}") + + return product_info_l + + +# 以下函数原理参见第五节课 +def read_string_to_list(input_string): + if input_string is None: + return None + + try: + input_string = input_string.replace("'", "\"") # Replace single quotes with double quotes for valid JSON + data = json.loads(input_string) + return data + except json.JSONDecodeError: + print(input_string) + print("错误：非法的 Json 格式") + return None + +def generate_output_string(data_list): + output_string = "" + + if data_list is None: + return output_string + # print(data_list) + for data in data_list: + try: + if "products" in data: + # print(data) + products_list = data["products"] + for product_name in products_list: + product = get_product_by_name(product_name) + if product: + output_string += json.dumps(product, indent=4) + "\n" + else: + print(f"错误: 商品 '{product_name}' 没有找到") + elif "category" in data: + category_name = data["category"] + category_products = get_products_by_category(category_name) + for product in category_products: + output_string += json.dumps(product, indent=4) + "\n" + else: + print("错误：非法的商品格式") + except Exception as e: + print(f"Error: {e}") + + return output_string + +# Example usage: +#product_information_for_user_message_1 = generate_output_string(category_and_product_list) +#print(product_information_for_user_message_1) +# 回答用户问题 +def answer_user_msg(user_msg,product_info): + """ + 代码参见第五节课 + """ + delimiter = "####" + system_message = f""" + 您是一家大型电子商店的客户服务助理。\ + 请用友好和乐于助人的口吻回答问题，提供简洁明了的答案。\ + 确保向用户提出相关的后续问题。 + """ + # user_msg = f""" + # tell me about the smartx pro phone and the fotosnap camera, the dslr one. Also what tell me about your tvs""" + messages = [ + {'role':'system', 'content': system_message}, + {'role':'user', 'content': f"{delimiter}{user_msg}{delimiter}"}, + {'role':'assistant', 'content': f"相关产品信息:\n{product_info}"}, + ] + response = get_completion_from_messages(messages) + return response + +# 创建并存入商品数据 +def create_products(): + # product information + # fun fact: all these products are fake and were generated by a language model + products = { + "TechPro Ultrabook": { + "name": "TechPro Ultrabook", + "category": "Computers and Laptops", + "brand": "TechPro", + "model_number": "TP-UB100", + "warranty": "1 year", + "rating": 4.5, + "features": ["13.3-inch display", "8GB RAM", "256GB SSD", "Intel Core i5 processor"], + "description": "A sleek and lightweight ultrabook for everyday use.", + "price": 799.99 + }, + "BlueWave Gaming Laptop": { + "name": "BlueWave Gaming Laptop", + "category": "Computers and Laptops", + "brand": "BlueWave", + "model_number": "BW-GL200", + "warranty": "2 years", + "rating": 4.7, + "features": ["15.6-inch display", "16GB RAM", "512GB SSD", "NVIDIA GeForce RTX 3060"], + "description": "A high-performance gaming laptop for an immersive experience.", + "price": 1199.99 + }, + "PowerLite Convertible": { + "name": "PowerLite Convertible", + "category": "Computers and Laptops", + "brand": "PowerLite", + "model_number": "PL-CV300", + "warranty": "1 year", + "rating": 4.3, + "features": ["14-inch touchscreen", "8GB RAM", "256GB SSD", "360-degree hinge"], + "description": "A versatile convertible laptop with a responsive touchscreen.", + "price": 699.99 + }, + "TechPro Desktop": { + "name": "TechPro Desktop", + "category": "Computers and Laptops", + "brand": "TechPro", + "model_number": "TP-DT500", + "warranty": "1 year", + "rating": 4.4, + "features": ["Intel Core i7 processor", "16GB RAM", "1TB HDD", "NVIDIA GeForce GTX 1660"], + "description": "A powerful desktop computer for work and play.", + "price": 999.99 + }, + "BlueWave Chromebook": { + "name": "BlueWave Chromebook", + "category": "Computers and Laptops", + "brand": "BlueWave", + "model_number": "BW-CB100", + "warranty": "1 year", + "rating": 4.1, + "features": ["11.6-inch display", "4GB RAM", "32GB eMMC", "Chrome OS"], + "description": "A compact and affordable Chromebook for everyday tasks.", + "price": 249.99 + }, + "SmartX ProPhone": { + "name": "SmartX ProPhone", + "category": "Smartphones and Accessories", + "brand": "SmartX", + "model_number": "SX-PP10", + "warranty": "1 year", + "rating": 4.6, + "features": ["6.1-inch display", "128GB storage", "12MP dual camera", "5G"], + "description": "A powerful smartphone with advanced camera features.", + "price": 899.99 + }, + "MobiTech PowerCase": { + "name": "MobiTech PowerCase", + "category": "Smartphones and Accessories", + "brand": "MobiTech", + "model_number": "MT-PC20", + "warranty": "1 year", + "rating": 4.3, + "features": ["5000mAh battery", "Wireless charging", "Compatible with SmartX ProPhone"], + "description": "A protective case with built-in battery for extended usage.", + "price": 59.99 + }, + "SmartX MiniPhone": { + "name": "SmartX MiniPhone", + "category": "Smartphones and Accessories", + "brand": "SmartX", + "model_number": "SX-MP5", + "warranty": "1 year", + "rating": 4.2, + "features": ["4.7-inch display", "64GB storage", "8MP camera", "4G"], + "description": "A compact and affordable smartphone for basic tasks.", + "price": 399.99 + }, + "MobiTech Wireless Charger": { + "name": "MobiTech Wireless Charger", + "category": "Smartphones and Accessories", + "brand": "MobiTech", + "model_number": "MT-WC10", + "warranty": "1 year", + "rating": 4.5, + "features": ["10W fast charging", "Qi-compatible", "LED indicator", "Compact design"], + "description": "A convenient wireless charger for a clutter-free workspace.", + "price": 29.99 + }, + "SmartX EarBuds": { + "name": "SmartX EarBuds", + "category": "Smartphones and Accessories", + "brand": "SmartX", + "model_number": "SX-EB20", + "warranty": "1 year", + "rating": 4.4, + "features": ["True wireless", "Bluetooth 5.0", "Touch controls", "24-hour battery life"], + "description": "Experience true wireless freedom with these comfortable earbuds.", + "price": 99.99 + }, + + "CineView 4K TV": { + "name": "CineView 4K TV", + "category": "Televisions and Home Theater Systems", + "brand": "CineView", + "model_number": "CV-4K55", + "warranty": "2 years", + "rating": 4.8, + "features": ["55-inch display", "4K resolution", "HDR", "Smart TV"], + "description": "A stunning 4K TV with vibrant colors and smart features.", + "price": 599.99 + }, + "SoundMax Home Theater": { + "name": "SoundMax Home Theater", + "category": "Televisions and Home Theater Systems", + "brand": "SoundMax", + "model_number": "SM-HT100", + "warranty": "1 year", + "rating": 4.4, + "features": ["5.1 channel", "1000W output", "Wireless subwoofer", "Bluetooth"], + "description": "A powerful home theater system for an immersive audio experience.", + "price": 399.99 + }, + "CineView 8K TV": { + "name": "CineView 8K TV", + "category": "Televisions and Home Theater Systems", + "brand": "CineView", + "model_number": "CV-8K65", + "warranty": "2 years", + "rating": 4.9, + "features": ["65-inch display", "8K resolution", "HDR", "Smart TV"], + "description": "Experience the future of television with this stunning 8K TV.", + "price": 2999.99 + }, + "SoundMax Soundbar": { + "name": "SoundMax Soundbar", + "category": "Televisions and Home Theater Systems", + "brand": "SoundMax", + "model_number": "SM-SB50", + "warranty": "1 year", + "rating": 4.3, + "features": ["2.1 channel", "300W output", "Wireless subwoofer", "Bluetooth"], + "description": "Upgrade your TV's audio with this sleek and powerful soundbar.", + "price": 199.99 + }, + "CineView OLED TV": { + "name": "CineView OLED TV", + "category": "Televisions and Home Theater Systems", + "brand": "CineView", + "model_number": "CV-OLED55", + "warranty": "2 years", + "rating": 4.7, + "features": ["55-inch display", "4K resolution", "HDR", "Smart TV"], + "description": "Experience true blacks and vibrant colors with this OLED TV.", + "price": 1499.99 + }, + + "GameSphere X": { + "name": "GameSphere X", + "category": "Gaming Consoles and Accessories", + "brand": "GameSphere", + "model_number": "GS-X", + "warranty": "1 year", + "rating": 4.9, + "features": ["4K gaming", "1TB storage", "Backward compatibility", "Online multiplayer"], + "description": "A next-generation gaming console for the ultimate gaming experience.", + "price": 499.99 + }, + "ProGamer Controller": { + "name": "ProGamer Controller", + "category": "Gaming Consoles and Accessories", + "brand": "ProGamer", + "model_number": "PG-C100", + "warranty": "1 year", + "rating": 4.2, + "features": ["Ergonomic design", "Customizable buttons", "Wireless", "Rechargeable battery"], + "description": "A high-quality gaming controller for precision and comfort.", + "price": 59.99 + }, + "GameSphere Y": { + "name": "GameSphere Y", + "category": "Gaming Consoles and Accessories", + "brand": "GameSphere", + "model_number": "GS-Y", + "warranty": "1 year", + "rating": 4.8, + "features": ["4K gaming", "500GB storage", "Backward compatibility", "Online multiplayer"], + "description": "A compact gaming console with powerful performance.", + "price": 399.99 + }, + "ProGamer Racing Wheel": { + "name": "ProGamer Racing Wheel", + "category": "Gaming Consoles and Accessories", + "brand": "ProGamer", + "model_number": "PG-RW200", + "warranty": "1 year", + "rating": 4.5, + "features": ["Force feedback", "Adjustable pedals", "Paddle shifters", "Compatible with GameSphere X"], + "description": "Enhance your racing games with this realistic racing wheel.", + "price": 249.99 + }, + "GameSphere VR Headset": { + "name": "GameSphere VR Headset", + "category": "Gaming Consoles and Accessories", + "brand": "GameSphere", + "model_number": "GS-VR", + "warranty": "1 year", + "rating": 4.6, + "features": ["Immersive VR experience", "Built-in headphones", "Adjustable headband", "Compatible with GameSphere X"], + "description": "Step into the world of virtual reality with this comfortable VR headset.", + "price": 299.99 + }, + + "AudioPhonic Noise-Canceling Headphones": { + "name": "AudioPhonic Noise-Canceling Headphones", + "category": "Audio Equipment", + "brand": "AudioPhonic", + "model_number": "AP-NC100", + "warranty": "1 year", + "rating": 4.6, + "features": ["Active noise-canceling", "Bluetooth", "20-hour battery life", "Comfortable fit"], + "description": "Experience immersive sound with these noise-canceling headphones.", + "price": 199.99 + }, + "WaveSound Bluetooth Speaker": { + "name": "WaveSound Bluetooth Speaker", + "category": "Audio Equipment", + "brand": "WaveSound", + "model_number": "WS-BS50", + "warranty": "1 year", + "rating": 4.5, + "features": ["Portable", "10-hour battery life", "Water-resistant", "Built-in microphone"], + "description": "A compact and versatile Bluetooth speaker for music on the go.", + "price": 49.99 + }, + "AudioPhonic True Wireless Earbuds": { + "name": "AudioPhonic True Wireless Earbuds", + "category": "Audio Equipment", + "brand": "AudioPhonic", + "model_number": "AP-TW20", + "warranty": "1 year", + "rating": 4.4, + "features": ["True wireless", "Bluetooth 5.0", "Touch controls", "18-hour battery life"], + "description": "Enjoy music without wires with these comfortable true wireless earbuds.", + "price": 79.99 + }, + "WaveSound Soundbar": { + "name": "WaveSound Soundbar", + "category": "Audio Equipment", + "brand": "WaveSound", + "model_number": "WS-SB40", + "warranty": "1 year", + "rating": 4.3, + "features": ["2.0 channel", "80W output", "Bluetooth", "Wall-mountable"], + "description": "Upgrade your TV's audio with this slim and powerful soundbar.", + "price": 99.99 + }, + "AudioPhonic Turntable": { + "name": "AudioPhonic Turntable", + "category": "Audio Equipment", + "brand": "AudioPhonic", + "model_number": "AP-TT10", + "warranty": "1 year", + "rating": 4.2, + "features": ["3-speed", "Built-in speakers", "Bluetooth", "USB recording"], + "description": "Rediscover your vinyl collection with this modern turntable.", + "price": 149.99 + }, + + "FotoSnap DSLR Camera": { + "name": "FotoSnap DSLR Camera", + "category": "Cameras and Camcorders", + "brand": "FotoSnap", + "model_number": "FS-DSLR200", + "warranty": "1 year", + "rating": 4.7, + "features": ["24.2MP sensor", "1080p video", "3-inch LCD", "Interchangeable lenses"], + "description": "Capture stunning photos and videos with this versatile DSLR camera.", + "price": 599.99 + }, + "ActionCam 4K": { + "name": "ActionCam 4K", + "category": "Cameras and Camcorders", + "brand": "ActionCam", + "model_number": "AC-4K", + "warranty": "1 year", + "rating": 4.4, + "features": ["4K video", "Waterproof", "Image stabilization", "Wi-Fi"], + "description": "Record your adventures with this rugged and compact 4K action camera.", + "price": 299.99 + }, + "FotoSnap Mirrorless Camera": { + "name": "FotoSnap Mirrorless Camera", + "category": "Cameras and Camcorders", + "brand": "FotoSnap", + "model_number": "FS-ML100", + "warranty": "1 year", + "rating": 4.6, + "features": ["20.1MP sensor", "4K video", "3-inch touchscreen", "Interchangeable lenses"], + "description": "A compact and lightweight mirrorless camera with advanced features.", + "price": 799.99 + }, + "ZoomMaster Camcorder": { + "name": "ZoomMaster Camcorder", + "category": "Cameras and Camcorders", + "brand": "ZoomMaster", + "model_number": "ZM-CM50", + "warranty": "1 year", + "rating": 4.3, + "features": ["1080p video", "30x optical zoom", "3-inch LCD", "Image stabilization"], + "description": "Capture life's moments with this easy-to-use camcorder.", + "price": 249.99 + }, + "FotoSnap Instant Camera": { + "name": "FotoSnap Instant Camera", + "category": "Cameras and Camcorders", + "brand": "FotoSnap", + "model_number": "FS-IC10", + "warranty": "1 year", + "rating": 4.1, + "features": ["Instant prints", "Built-in flash", "Selfie mirror", "Battery-powered"], + "description": "Create instant memories with this fun and portable instant camera.", + "price": 69.99 + } + } + + products_file = 'products.json' + with open(products_file, 'w') as file: + json.dump(products, file) + + return products