From d90003e07a2c7add097f421c6643722ad0f75356 Mon Sep 17 00:00:00 2001 From: nowadays0421 Date: Mon, 31 Jul 2023 13:42:24 +0800 Subject: [PATCH] Finish Chat-5 --- .../C4 LangChain Chat with Your Data/5.检索 retrieval.ipynb | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/content/C4 LangChain Chat with Your Data/5.检索 retrieval.ipynb diff --git a/docs/content/C4 LangChain Chat with Your Data/5.检索 retrieval.ipynb b/docs/content/C4 LangChain Chat with Your Data/5.检索 retrieval.ipynb new file mode 100644 index 0000000..ca2fb88 --- /dev/null +++ b/docs/content/C4 LangChain Chat with Your Data/5.检索 retrieval.ipynb @@ -0,0 +1 @@ +{"cells":[{"attachments":{},"cell_type":"markdown","id":"0689733d","metadata":{},"source":["# 第五章 检索(Retrieval)\n","\n"]},{"attachments":{},"cell_type":"markdown","id":"d12fbd74","metadata":{},"source":["\n","检索是我们的检索增强生成(RAG)流程的核心。\n","\n","让我们获得前面课程存储的向量数据库(`VectorDB`)。"]},{"attachments":{},"cell_type":"markdown","id":"ed2569c6","metadata":{},"source":["## 一、向量数据库检索"]},{"attachments":{},"cell_type":"markdown","id":"651a10db","metadata":{},"source":["本章节需要使用`lark`包，运行以下命令安装"]},{"cell_type":"code","execution_count":1,"id":"c18f8a7b-62af-403e-9877-18d1c2265b4f","metadata":{"tags":[]},"outputs":[],"source":["!pip install -Uq lark"]},{"attachments":{},"cell_type":"markdown","id":"c2d552e1","metadata":{},"source":["### 1.1 相似性检索"]},{"cell_type":"markdown","id":"932872b1","metadata":{},"source":["首先将上节课所保存的向量数据库(`VectorDB`)加载进来："]},{"cell_type":"code","execution_count":3,"id":"fe368042","metadata":{"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["27\n"]}],"source":["from langchain.vectorstores import Chroma\n","from langchain.embeddings.openai import OpenAIEmbeddings\n","\n","persist_directory_chinese = 'docs/chroma/matplotlib/'\n","\n","embedding = OpenAIEmbeddings()\n","\n","vectordb_chinese = Chroma(\n"," persist_directory=persist_directory_chinese,\n"," embedding_function=embedding\n",")\n","\n","print(vectordb_chinese._collection.count())"]},{"attachments":{},"cell_type":"markdown","id":"9ae4fdd8","metadata":{},"source":["让我们现在来看看最大边际相关性的例子。因此，我们将从下面示例中加载有关蘑菇的信息。\n","\n","让我们现在运行它与MMR。让我们传入k等于2。我们仍然希望返回两个文档，但让我们设置获取k等于3，其中我们最初获取所有三个文档。我们现在可以看到，我们检索的文档中返回了有毒的信息。"]},{"cell_type":"code","execution_count":4,"id":"b110cceb","metadata":{},"outputs":[],"source":["texts_chinese = [\n"," \"\"\"毒鹅膏菌（Amanita phalloides）具有大型且引人注目的地上（epigeous）子实体（basidiocarp）\"\"\",\n"," \"\"\"一种具有大型子实体的蘑菇是毒鹅膏菌（Amanita phalloides）。某些品种全白。\"\"\",\n"," \"\"\"A. phalloides，又名死亡帽，是已知所有蘑菇中最有毒的一种。\"\"\",\n","]"]},{"attachments":{},"cell_type":"markdown","id":"84cd5f1c","metadata":{},"source":["对于这个例子，我们将创建一个小数据库，我们可以作为一个示例来使用。"]},{"cell_type":"code","execution_count":5,"id":"305e1714","metadata":{},"outputs":[{"name":"stderr","output_type":"stream","text":["100%|██████████| 1/1 [00:00<00:00, 1.51it/s]\n"]}],"source":["smalldb_chinese = Chroma.from_texts(texts_chinese, embedding=embedding)"]},{"attachments":{},"cell_type":"markdown","id":"239a8d95","metadata":{},"source":["下面是我们对于这个示例所提出的问题"]},{"cell_type":"code","execution_count":6,"id":"92312e57","metadata":{},"outputs":[],"source":["question_chinese = \"告诉我关于具有大型子实体的全白色蘑菇的信息\""]},{"attachments":{},"cell_type":"markdown","id":"d3224a6d","metadata":{},"source":["现在，我们可以运行一个相似性搜索，设置k=2，只返回两个最相关的文档。\n","\n","我们可以看到，没有提到它是有毒的事实。"]},{"cell_type":"code","execution_count":17,"id":"d4c5a47d","metadata":{},"outputs":[{"data":{"text/plain":["[Document(page_content='一种具有大型子实体的蘑菇是毒鹅膏菌（Amanita phalloides）。某些品种全白。', metadata={}),\n"," Document(page_content='毒鹅膏菌（Amanita phalloides）具有大型且引人注目的地上（epigeous）子实体（basidiocarp）', metadata={})]"]},"execution_count":17,"metadata":{},"output_type":"execute_result"}],"source":["smalldb_chinese.similarity_search(question_chinese, k=2)"]},{"attachments":{},"cell_type":"markdown","id":"bbb0ea94","metadata":{},"source":["现在，让我们运行最大边际相关性(MMR)。\n","\n","设置k=2，因为我们仍然希望返回两个文档。设置fetch_k=3，fetch_k是我们最初获取的所有文档(3个)。"]},{"cell_type":"code","execution_count":8,"id":"e15521d2","metadata":{},"outputs":[{"data":{"text/plain":["[Document(page_content='一种具有大型子实体的蘑菇是毒鹅膏菌（Amanita phalloides）。某些品种全白。', metadata={}),\n"," Document(page_content='A. phalloides，又名死亡帽，是已知所有蘑菇中最有毒的一种。', metadata={})]"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["smalldb_chinese.max_marginal_relevance_search(question_chinese,k=2, fetch_k=3)"]},{"attachments":{},"cell_type":"markdown","id":"e87c5f91","metadata":{},"source":["我们现在可以看到，我们检索的文档中返回了有毒的信息。"]},{"attachments":{},"cell_type":"markdown","id":"5a29e8c9","metadata":{},"source":["### 1.2 解决多样性：最大边际相关性(MMR)"]},{"attachments":{},"cell_type":"markdown","id":"a2b5c4ae","metadata":{},"source":["\n","我们刚刚通过一个示例引出了一个问题：如何加强搜索结果的多样性。\n"," \n","最大边际相关性(`Maximum marginal relevance`)试图在查询的相关性和结果的多样性之间实现两全其美。"]},{"attachments":{},"cell_type":"markdown","id":"2360545c","metadata":{},"source":["让我们回到上一节课的一个例子，当我们通过问题对向量数据库进行相似性搜索后\n","\n","我们可以看看前两个文档，只看前几个字符，可以看到它们是相同的。"]},{"cell_type":"code","execution_count":10,"id":"e8e142eb","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["docs[0]: \n","第⼀回：Matplotlib 初相识\n","⼀、认识matplotlib\n","Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种\n","\n","docs[1]: \n","第⼀回：Matplotlib 初相识\n","⼀、认识matplotlib\n","Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种\n"]}],"source":["question_chinese = \"Matplotlib是什么？\"\n","docs_ss_chinese = vectordb_chinese.similarity_search(question_chinese,k=3)\n","\n","print(\"docs[0]: \")\n","print(docs_ss_chinese[0].page_content[:100])\n","print()\n","print(\"docs[1]: \")\n","print(docs_ss_chinese[1].page_content[:100])"]},{"attachments":{},"cell_type":"markdown","id":"4c4ca1b6","metadata":{},"source":["我们可以使用`MMR`得到不一样的结果。"]},{"cell_type":"code","execution_count":11,"id":"408935bc","metadata":{},"outputs":[],"source":["docs_mmr_chinese = vectordb_chinese.max_marginal_relevance_search(question_chinese,k=3)"]},{"attachments":{},"cell_type":"markdown","id":"9076db81","metadata":{},"source":["当我们运行MMR后得到结果时，我们可以看到第一个与之前的相同，因为那是最相似的。"]},{"cell_type":"code","execution_count":12,"id":"d0acfaab","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["第⼀回：Matplotlib 初相识\n","⼀、认识matplotlib\n","Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种\n"]}],"source":["print(docs_mmr_chinese[0].page_content[:100])"]},{"attachments":{},"cell_type":"markdown","id":"7a93743f","metadata":{},"source":["但是当我们进行到第二个时，我们可以看到它是不同的。\n","\n","它在回应中获得了一些多样性。"]},{"cell_type":"code","execution_count":13,"id":"93d3206c","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["By Datawhale 数据可视化开源⼩组\n","© Copyright © Copyright 2021.y轴分为左右两个，因此 tick1 对应左侧的轴； tick2 对应右侧的轴。\n","x轴分为上下两个\n"]}],"source":["print(docs_mmr_chinese[1].page_content[:100])"]},{"attachments":{},"cell_type":"markdown","id":"b2b909bc","metadata":{},"source":["### 1.3 解决特殊性：使用元数据"]},{"attachments":{},"cell_type":"markdown","id":"7b63c5ee","metadata":{},"source":["\n","在上一节课中，我们展示了一个问题，是询问了关于文档中某一讲的问题，但得到的结果中也包括了来自其他讲的结果。\n","\n","为了解决这一问题，很多向量数据库都支持对`metadata`的操作。\n","\n","`metadata`为每个嵌入的块(embedded chunk)提供上下文。"]},{"cell_type":"code","execution_count":16,"id":"ba98df3c","metadata":{},"outputs":[],"source":["question_chinese = \"他们在第二讲中对Figure说了些什么？\" "]},{"attachments":{},"cell_type":"markdown","id":"3873525e","metadata":{},"source":["现在，我们以手动的方式来解决这个问题，我们会指定一个元数据过滤器`filter`"]},{"cell_type":"code","execution_count":17,"id":"b46c7e76","metadata":{},"outputs":[],"source":["docs_chinese = vectordb_chinese.similarity_search(\n"," question_chinese,\n"," k=3,\n"," filter={\"source\":\"docs/matplotlib/第二回：艺术画笔见乾坤.pdf\"}\n",")"]},{"attachments":{},"cell_type":"markdown","id":"869aee28","metadata":{},"source":["接下来，我们可以看到结果都来自对应的章节"]},{"cell_type":"code","execution_count":18,"id":"2708f6ae","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 9}\n","{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 10}\n","{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 0}\n"]}],"source":["for d in docs_chinese:\n"," print(d.metadata)\n"," "]},{"attachments":{},"cell_type":"markdown","id":"5e299f8e","metadata":{},"source":["当然，我们不能每次都采用手动的方式来解决这个问题，这会显得不够智能\n","\n","下一小节将要展示通过LLM来解决这个问题"]},{"attachments":{},"cell_type":"markdown","id":"ccc2d784","metadata":{},"source":["### 1.4 解决特殊性：在元数据中使用自查询检索器"]},{"attachments":{},"cell_type":"markdown","id":"82ef44b6","metadata":{},"source":["我们有一个有趣的挑战：我们通常希望从查询本身来推断元数据。\n","\n","为了解决这个问题，我们可以使用SelfQueryRetriever，它使用LLM来提取：\n"," \n","1. 用于向量搜索的查询(`query`)字符串，即：问题\n","2. 要一起传入的元数据过滤器\n","\n","大多数向量数据库支持元数据过滤器，因此不需要任何新的数据库及索引。"]},{"cell_type":"code","execution_count":20,"id":"b1d06084","metadata":{"tags":[]},"outputs":[],"source":["from langchain.llms import OpenAI\n","from langchain.retrievers.self_query.base import SelfQueryRetriever\n","from langchain.chains.query_constructor.base import AttributeInfo\n","\n","llm = OpenAI(temperature=0)"]},{"attachments":{},"cell_type":"markdown","id":"acd194c5","metadata":{},"source":["`AttributeInfo`是我们可以指定元数据中的不同字段以及它们对应的位置。\n","\n","在元数据中，我们只有两个字段，`source`和`page`。\n","\n","我们将填写每个属性的名称、描述和类型的描述。\n","\n","这些信息实际上将被传递给LLM，所以需要尽可能详细地描述。"]},{"cell_type":"code","execution_count":32,"id":"544ad7c1","metadata":{},"outputs":[],"source":["metadata_field_info_chinese = [\n"," AttributeInfo(\n"," name=\"source\",\n"," description=\"The lecture the chunk is from, should be one of `docs/matplotlib/第一回：Matplotlib初相识.pdf`, `docs/matplotlib/第二回：艺术画笔见乾坤.pdf`, or `docs/matplotlib/第三回：布局格式定方圆.pdf`\",\n"," type=\"string\",\n"," ),\n"," AttributeInfo(\n"," name=\"page\",\n"," description=\"The page from the lecture\",\n"," type=\"integer\",\n"," ),\n","]\n","\n","document_content_description_chinese = \"Matplotlib 课堂讲义\"\n","retriever_chinese = SelfQueryRetriever.from_llm(\n"," llm,\n"," vectordb_chinese,\n"," document_content_description_chinese,\n"," metadata_field_info_chinese,\n"," verbose=True\n",")\n","\n","question_chinese = \"他们在第二讲中对Figure做了些什么？\" "]},{"attachments":{},"cell_type":"markdown","id":"c51778b0-1fcd-40a4-bd6b-0f13fec8acb1","metadata":{},"source":["当你第一次执行下一行时，你会收到关于predict_and_parse已被弃用的**警告**。 这可以安全地忽略。"]},{"cell_type":"code","execution_count":33,"id":"ea39a97e","metadata":{},"outputs":[{"name":"stderr","output_type":"stream","text":["/root/autodl-tmp/env/gpt/lib/python3.10/site-packages/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n"," warnings.warn(\n"]},{"name":"stdout","output_type":"stream","text":["query='Figure' filter=Comparison(comparator=, attribute='source', value='docs/matplotlib/第二回：艺术画笔见乾坤.pdf') limit=None\n"]}],"source":["docs_chinese = retriever_chinese.get_relevant_documents(question_chinese)"]},{"cell_type":"markdown","id":"2200ceda","metadata":{},"source":["打印可以看到查询结果，基于子查询检索器，我们检索到的结果都是在第二回的文档中："]},{"cell_type":"code","execution_count":34,"id":"143061f5","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 9}\n","{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 10}\n","{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 0}\n","{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 6}\n"]}],"source":["for d in docs_chinese:\n"," print(d.metadata)"]},{"attachments":{},"cell_type":"markdown","id":"297b8168","metadata":{},"source":["### 1.5 其他技巧：压缩"]},{"attachments":{},"cell_type":"markdown","id":"564144da","metadata":{},"source":["另一种提高检索到的文档质量的方法是压缩。\n","\n","与查询最相关的信息可能隐藏在具有大量不相关文本的文档中。\n","\n","在应用程序中传递完整的文档可能会导致更昂贵的LLM调用和更差的响应。\n","\n","上下文压缩就是为了解决这个问题。"]},{"cell_type":"code","execution_count":35,"id":"a060cf74","metadata":{"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Document 1:\n","\n","Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种静态，动态，交互式的图表。\n","----------------------------------------------------------------------------------------------------\n","Document 2:\n","\n","Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种静态，动态，交互式的图表。\n"]}],"source":["from langchain.retrievers import ContextualCompressionRetriever\n","from langchain.retrievers.document_compressors import LLMChainExtractor\n","\n","def pretty_print_docs(docs):\n"," print(f\"\\n{'-' * 100}\\n\".join([f\"Document {i+1}:\\n\\n\" + d.page_content for i, d in enumerate(docs)]))\n","\n","llm = OpenAI(temperature=0)\n","compressor = LLMChainExtractor.from_llm(llm) # 压缩器\n","\n","compression_retriever_chinese = ContextualCompressionRetriever(\n"," base_compressor=compressor,\n"," base_retriever=vectordb_chinese.as_retriever()\n",")\n","# 对源文档进行压缩\n","\n","question_chinese = \"Matplotlib是什么？\"\n","compressed_docs_chinese = compression_retriever_chinese.get_relevant_documents(question_chinese)\n","pretty_print_docs(compressed_docs_chinese)"]},{"attachments":{},"cell_type":"markdown","id":"049b2601","metadata":{},"source":["现在当我们提出问题后，查看结果文档\n","\n","我们可以看到两件事。\n","\n","1. 它们比正常文档短很多\n","2. 仍然有一些重复的东西，这是因为在底层我们使用的是语义搜索算法。\n","\n","这就是我们在本课程前面使用MMR解决的问题。\n","\n","这是一个很好的例子，你可以结合各种技术得到最好的可能结果。"]},{"attachments":{},"cell_type":"markdown","id":"82c4fc4d","metadata":{},"source":["## 二、结合各种技术"]},{"attachments":{},"cell_type":"markdown","id":"54432975","metadata":{},"source":["为了做到这一点，我们在从向量数据库创建检索器时，可以将搜索类型设置为MMR。\n","\n","然后我们可以重新运行这个过程，看到我们返回的是一个过滤过的结果集，其中不包含任何重复的信息。"]},{"cell_type":"code","execution_count":37,"id":"cd6396bb","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Document 1:\n","\n","Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种静态，动态，交互式的图表。\n"]}],"source":["compression_retriever_chinese = ContextualCompressionRetriever(\n"," base_compressor=compressor,\n"," base_retriever=vectordb_chinese.as_retriever(search_type = \"mmr\")\n",")\n","\n","question_chinese = \"Matplotlib是什么？\"\n","compressed_docs_chinese = compression_retriever_chinese.get_relevant_documents(question_chinese)\n","pretty_print_docs(compressed_docs_chinese)"]},{"attachments":{},"cell_type":"markdown","id":"6c2b63e1","metadata":{},"source":["## 三、其他类型的检索"]},{"attachments":{},"cell_type":"markdown","id":"3e777a7b","metadata":{},"source":["值得注意的是，vetordb并不是唯一一种检索文档的工具。\n","\n","`LangChain`检索器抽象包括其他检索文档的方式，如：`TF-IDF` 或 `SVM`。"]},{"cell_type":"code","execution_count":40,"id":"83d2e808","metadata":{"tags":[]},"outputs":[],"source":["from langchain.retrievers import SVMRetriever\n","from langchain.retrievers import TFIDFRetriever\n","from langchain.document_loaders import PyPDFLoader\n","from langchain.text_splitter import RecursiveCharacterTextSplitter\n","\n","# 加载PDF\n","loader_chinese = PyPDFLoader(\"docs/matplotlib/第一回：Matplotlib初相识.pdf\")\n","pages_chinese = loader_chinese.load()\n","all_page_text_chinese = [p.page_content for p in pages_chinese]\n","joined_page_text_chinese = \" \".join(all_page_text_chinese)\n","\n","# 分割文本\n","text_splitter_chinese = RecursiveCharacterTextSplitter(chunk_size = 1500,chunk_overlap = 150)\n","splits_chinese = text_splitter_chinese.split_text(joined_page_text_chinese)\n","\n","# 检索\n","svm_retriever = SVMRetriever.from_texts(splits_chinese, embedding)\n","tfidf_retriever = TFIDFRetriever.from_texts(splits_chinese)"]},{"cell_type":"markdown","id":"59abbaff","metadata":{},"source":["接下来我们分别测试 TF-IDF 检索以及 SVM 检索的效果。"]},{"cell_type":"code","execution_count":42,"id":"cc823bea","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["page_content='fig, ax = plt.subplots() \\n# step4 绘制图像， 这⼀模块的扩展参考第⼆章进⼀步学习\\nax.plot(x, y, label=\\'linear\\') \\n# step5 添加标签，⽂字和图例，这⼀模块的扩展参考第四章进⼀步学习\\nax.set_xlabel(\\'x label\\') \\nax.set_ylabel(\\'y label\\') \\nax.set_title(\"Simple Plot\") \\nax.legend() ;\\n思考题\\n请思考两种绘图模式的优缺点和各⾃适合的使⽤场景\\n在第五节绘图模板中我们是以 OO 模式作为例⼦展示的，请思考并写⼀个 pyplot 绘图模式的简单模板' metadata={}\n"]}],"source":["question_chinese = \"这门课的主要主题是什么？\" \n","docs_svm_chinese = svm_retriever.get_relevant_documents(question_chinese)\n","print(docs_svm_chinese[0])"]},{"cell_type":"markdown","id":"e1372df3","metadata":{},"source":["可以看出，SVM 检索的效果要差于 VectorDB。"]},{"cell_type":"code","execution_count":43,"id":"01eb9d43","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["page_content='fig, ax = plt.subplots() \\n# step4 绘制图像， 这⼀模块的扩展参考第⼆章进⼀步学习\\nax.plot(x, y, label=\\'linear\\') \\n# step5 添加标签，⽂字和图例，这⼀模块的扩展参考第四章进⼀步学习\\nax.set_xlabel(\\'x label\\') \\nax.set_ylabel(\\'y label\\') \\nax.set_title(\"Simple Plot\") \\nax.legend() ;\\n思考题\\n请思考两种绘图模式的优缺点和各⾃适合的使⽤场景\\n在第五节绘图模板中我们是以 OO 模式作为例⼦展示的，请思考并写⼀个 pyplot 绘图模式的简单模板' metadata={}\n"]}],"source":["question_chinese = \"Matplotlib是什么？\"\n","docs_tfidf_chinese = tfidf_retriever.get_relevant_documents(question_chinese)\n","print(docs_tfidf_chinese[0])"]},{"cell_type":"markdown","id":"4ae14af5","metadata":{},"source":["同样，TF-IDF 检索的效果也不尽如人意。"]},{"cell_type":"markdown","id":"2bfb3661","metadata":{},"source":["## 四、英文版"]},{"cell_type":"markdown","id":"4a9b06d6","metadata":{},"source":["**1.1 相似性检索**"]},{"cell_type":"code","execution_count":44,"id":"0731edf4","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["209\n"]}],"source":["from langchain.vectorstores import Chroma\n","from langchain.embeddings.openai import OpenAIEmbeddings\n","\n","persist_directory = 'docs/chroma/cs229_lectures/'\n","\n","embedding = OpenAIEmbeddings()\n","vectordb = Chroma(\n"," persist_directory=persist_directory,\n"," embedding_function=embedding\n",")\n","print(vectordb._collection.count())"]},{"cell_type":"markdown","id":"484d895d","metadata":{},"source":["简单示例"]},{"cell_type":"code","execution_count":46,"id":"5e8873c5","metadata":{},"outputs":[{"name":"stderr","output_type":"stream","text":[" 0%| | 0/1 [00:00, attribute='source', value='docs/cs229_lectures/MachineLearning-Lecture03.pdf') limit=None\n","{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 14}\n","{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 0}\n","{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10}\n","{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10}\n"]}],"source":["from langchain.llms import OpenAI\n","from langchain.retrievers.self_query.base import SelfQueryRetriever\n","from langchain.chains.query_constructor.base import AttributeInfo\n","\n","llm = OpenAI(temperature=0)\n","\n","metadata_field_info = [\n"," AttributeInfo(\n"," name=\"source\",\n"," description=\"The lecture the chunk is from, should be one of `docs/cs229_lectures/MachineLearning-Lecture01.pdf`, `docs/cs229_lectures/MachineLearning-Lecture02.pdf`, or `docs/cs229_lectures/MachineLearning-Lecture03.pdf`\",\n"," type=\"string\",\n"," ),\n"," AttributeInfo(\n"," name=\"page\",\n"," description=\"The page from the lecture\",\n"," type=\"integer\",\n"," ),\n","]\n","\n","document_content_description = \"Lecture notes\"\n","retriever = SelfQueryRetriever.from_llm(\n"," llm,\n"," vectordb,\n"," document_content_description,\n"," metadata_field_info,\n"," verbose=True\n",")\n","\n","question = \"what did they say about regression in the third lecture?\"\n","\n","docs = retriever.get_relevant_documents(question)\n","\n","for d in docs:\n"," print(d.metadata)"]},{"cell_type":"markdown","id":"ffea27b2","metadata":{},"source":["**1.5 压缩**"]},{"cell_type":"code","execution_count":50,"id":"ac148c58","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Document 1:\n","\n","\"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms.\"\n","----------------------------------------------------------------------------------------------------\n","Document 2:\n","\n","\"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms.\"\n","----------------------------------------------------------------------------------------------------\n","Document 3:\n","\n","\"And the student said, \"Oh, it was the MATLAB.\" So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it.\"\n","----------------------------------------------------------------------------------------------------\n","Document 4:\n","\n","\"And the student said, \"Oh, it was the MATLAB.\" So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it.\"\n"]}],"source":["from langchain.retrievers import ContextualCompressionRetriever\n","from langchain.retrievers.document_compressors import LLMChainExtractor\n","\n","def pretty_print_docs(docs):\n"," print(f\"\\n{'-' * 100}\\n\".join([f\"Document {i+1}:\\n\\n\" + d.page_content for i, d in enumerate(docs)]))\n","\n","llm = OpenAI(temperature=0)\n","compressor = LLMChainExtractor.from_llm(llm) # 压缩器\n","\n","compression_retriever = ContextualCompressionRetriever(\n"," base_compressor=compressor,\n"," base_retriever=vectordb.as_retriever()\n",")\n","\n","question = \"what did they say about matlab?\"\n","compressed_docs = compression_retriever.get_relevant_documents(question)\n","pretty_print_docs(compressed_docs)"]},{"cell_type":"markdown","id":"b9c49b95","metadata":{},"source":["**2.1 结合各种技术**"]},{"cell_type":"code","execution_count":51,"id":"2f5e10be","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Document 1:\n","\n","\"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms.\"\n","----------------------------------------------------------------------------------------------------\n","Document 2:\n","\n","\"And the student said, \"Oh, it was the MATLAB.\" So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it.\"\n"]}],"source":["compression_retriever = ContextualCompressionRetriever(\n"," base_compressor=compressor,\n"," base_retriever=vectordb.as_retriever(search_type = \"mmr\")\n",")\n","\n","question = \"what did they say about matlab?\"\n","compressed_docs = compression_retriever.get_relevant_documents(question)\n","pretty_print_docs(compressed_docs)"]},{"cell_type":"markdown","id":"ca099bad","metadata":{},"source":["**3.1 其他类型的检索**"]},{"cell_type":"code","execution_count":52,"id":"e8330dc6","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["SVM:\n","page_content=\"let me just check what questions you have righ t now. So if there are no questions, I'll just \\nclose with two reminders, which are after class today or as you start to talk with other \\npeople in this class, I just encourage you again to start to form project partners, to try to \\nfind project partners to do your project with. And also, this is a good time to start forming \\nstudy groups, so either talk to your friends or post in the newsgroup, but we just \\nencourage you to try to star t to do both of those today, okay? Form study groups, and try \\nto find two other project partners. \\nSo thank you. I'm looking forward to teaching this class, and I'll see you in a couple of \\ndays. [End of Audio] \\nDuration: 69 minutes\" metadata={}\n","TF-IDF:\n","page_content=\"Saxena and Min Sun here did, wh ich is given an image like this, right? This is actually a \\npicture taken of the Stanford campus. You can apply that sort of cl ustering algorithm and \\ngroup the picture into regions. Let me actually blow that up so that you can see it more \\nclearly. Okay. So in the middle, you see the lines sort of groupi ng the image together, \\ngrouping the image into [inaudible] regions. \\nAnd what Ashutosh and Min did was they then applied the learning algorithm to say can \\nwe take this clustering and us e it to build a 3D model of the world? And so using the \\nclustering, they then had a lear ning algorithm try to learn what the 3D structure of the \\nworld looks like so that they could come up with a 3D model that you can sort of fly \\nthrough, okay? Although many people used to th ink it's not possible to take a single \\nimage and build a 3D model, but using a lear ning algorithm and that sort of clustering \\nalgorithm is the first step. They were able to. \\nI'll just show you one more example. I like this because it's a picture of Stanford with our \\nbeautiful Stanford campus. So again, taking th e same sort of clustering algorithms, taking \\nthe same sort of unsupervised learning algor ithm, you can group the pixels into different \\nregions. And using that as a pre-processing step, they eventually built this sort of 3D model of Stanford campus in a single picture. You can sort of walk into the ceiling, look\" metadata={}\n"]}],"source":["from langchain.retrievers import SVMRetriever\n","from langchain.retrievers import TFIDFRetriever\n","from langchain.document_loaders import PyPDFLoader\n","from langchain.text_splitter import RecursiveCharacterTextSplitter\n","\n","# 加载PDF\n","loader = PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture01.pdf\")\n","pages = loader.load()\n","all_page_text = [p.page_content for p in pages]\n","joined_page_text = \" \".join(all_page_text)\n","\n","# 分割文本\n","text_splitter = RecursiveCharacterTextSplitter(chunk_size = 1500,chunk_overlap = 150)\n","splits = text_splitter.split_text(joined_page_text)\n","\n","# 检索\n","svm_retriever = SVMRetriever.from_texts(splits, embedding)\n","tfidf_retriever = TFIDFRetriever.from_texts(splits)\n","\n","question = \"What are major topics for this class?\" # 这门课的主要主题是什么？\n","print(\"SVM:\")\n","docs_svm = svm_retriever.get_relevant_documents(question)\n","print(docs_svm[0])\n","\n","question = \"what did they say about matlab?\"\n","print(\"TF-IDF:\")\n","docs_tfidf = tfidf_retriever.get_relevant_documents(question)\n","print(docs_tfidf[0])\n","\n"]}],"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.11"}},"nbformat":4,"nbformat_minor":5}