1656 lines
64 KiB
Plaintext
1656 lines
64 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "0689733d",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 第五章 检索(Retrieval)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "d12fbd74",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"检索是我们的检索增强生成(RAG)流程的核心。\n",
|
||
"\n",
|
||
"让我们获得前面课程存储的向量数据库(`VectorDB`)。"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "ed2569c6",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 一、向量数据库检索"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "651a10db",
|
||
"metadata": {},
|
||
"source": [
|
||
"在当前文件夹下新建`.env`文件，内容为`OPENAI_API_KEY = \"sk-...\"`\n",
|
||
"\n",
|
||
"本章节需要使用`lark`包，运行以下命令安装"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "c18f8a7b-62af-403e-9877-18d1c2265b4f",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install -Uq lark"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "51b15e5c-9b92-4d40-a149-b56335d330d9",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import os\n",
|
||
"import openai\n",
|
||
"import sys\n",
|
||
"sys.path.append('../..')\n",
|
||
"\n",
|
||
"from dotenv import load_dotenv, find_dotenv\n",
|
||
"_ = load_dotenv(find_dotenv()) # read local .env file\n",
|
||
"\n",
|
||
"openai.api_key = os.environ['OPENAI_API_KEY']"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "c2d552e1",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1.1 相似性搜索"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "fe368042",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain.vectorstores import Chroma\n",
|
||
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
|
||
"persist_directory = 'docs/chroma/cs229_lectures/'\n",
|
||
"persist_directory_chinese = 'docs/chroma/matplotlib/'"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "5ba63e21",
|
||
"metadata": {},
|
||
"source": [
|
||
"将上节课所保存的向量数据库(`VectorDB`)加载进来"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "a0189dc5",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"embedding = OpenAIEmbeddings()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "2be10170",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"vectordb = Chroma(\n",
|
||
" persist_directory=persist_directory,\n",
|
||
" embedding_function=embedding\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "3659e0f7",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"209\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(vectordb._collection.count())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "a01ab000",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"vectordb_chinese = Chroma(\n",
|
||
" persist_directory=persist_directory_chinese,\n",
|
||
" embedding_function=embedding\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "a6998a03",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"27\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(vectordb_chinese._collection.count())"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "9ae4fdd8",
|
||
"metadata": {},
|
||
"source": [
|
||
"让我们现在来看看最大边际相关性的例子。因此，我们将从下面示例中加载有关蘑菇的信息。\n",
|
||
"\n",
|
||
"让我们现在运行它与MMR。让我们传入k等于2。我们仍然希望返回两个文档，但让我们设置获取k等于3，其中我们最初获取所有三个文档。我们现在可以看到，我们检索的文档中返回了有毒的信息。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "a807c758",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"texts = [\n",
|
||
" \"\"\"The Amanita phalloides has a large and imposing epigeous (aboveground) fruiting body (basidiocarp).\"\"\",\n",
|
||
" \"\"\"A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.\"\"\",\n",
|
||
" \"\"\"A. phalloides, a.k.a Death Cap, is one of the most poisonous of all known mushrooms.\"\"\",\n",
|
||
"]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "b110cceb",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"texts_chinese = [\n",
|
||
" \"\"\"毒鹅膏菌（Amanita phalloides）具有大型且引人注目的地上（epigeous）子实体（basidiocarp）\"\"\",\n",
|
||
" \"\"\"一种具有大型子实体的蘑菇是毒鹅膏菌（Amanita phalloides）。某些品种全白。\"\"\",\n",
|
||
" \"\"\"A. phalloides，又名死亡帽，是已知所有蘑菇中最有毒的一种。\"\"\",\n",
|
||
"]"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "84cd5f1c",
|
||
"metadata": {},
|
||
"source": [
|
||
"对于这个例子，我们将创建一个小数据库，我们可以作为一个示例来使用。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "715d54f3",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"smalldb = Chroma.from_texts(texts, embedding=embedding)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "305e1714",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"smalldb_chinese = Chroma.from_texts(texts_chinese, embedding=embedding)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "239a8d95",
|
||
"metadata": {},
|
||
"source": [
|
||
"下面是我们对于这个示例所提出的问题"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "9a37b5a5",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"question = \"Tell me about all-white mushrooms with large fruiting bodies\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "92312e57",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"question_chinese = \"告诉我关于具有大型子实体的全白色蘑菇的信息\""
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "d3224a6d",
|
||
"metadata": {},
|
||
"source": [
|
||
"现在，我们可以运行一个相似性搜索，设置k=2，只返回两个最相关的文档。\n",
|
||
"\n",
|
||
"我们可以看到，没有提到它是有毒的事实。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "24e3b025",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[Document(page_content='A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.', metadata={}),\n",
|
||
" Document(page_content='The Amanita phalloides has a large and imposing epigeous (aboveground) fruiting body (basidiocarp).', metadata={})]"
|
||
]
|
||
},
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"smalldb.similarity_search(question, k=2)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "d4c5a47d",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[Document(page_content='一种具有大型子实体的蘑菇是毒鹅膏菌（Amanita phalloides）。某些品种全白。', metadata={}),\n",
|
||
" Document(page_content='毒鹅膏菌（Amanita phalloides）具有大型且引人注目的地上（epigeous）子实体（basidiocarp）', metadata={})]"
|
||
]
|
||
},
|
||
"execution_count": 16,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"smalldb_chinese.similarity_search(question_chinese, k=2)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "bbb0ea94",
|
||
"metadata": {},
|
||
"source": [
|
||
"现在，让我们运行最大边际相关性(MMR)。\n",
|
||
"\n",
|
||
"设置k=2，因为我们仍然希望返回两个文档。设置fetch_k=3，fetch_k是我们最初获取的所有文档(3个)。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "4daa6c0d",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[Document(page_content='A mushroom with a large fruiting body is the Amanita phalloides. Some varieties are all-white.', metadata={}),\n",
|
||
" Document(page_content='A. phalloides, a.k.a Death Cap, is one of the most poisonous of all known mushrooms.', metadata={})]"
|
||
]
|
||
},
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"smalldb.max_marginal_relevance_search(question,k=2, fetch_k=3)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "e15521d2",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"[Document(page_content='一种具有大型子实体的蘑菇是毒鹅膏菌（Amanita phalloides）。某些品种全白。', metadata={}),\n",
|
||
" Document(page_content='A. phalloides，又名死亡帽，是已知所有蘑菇中最有毒的一种。', metadata={})]"
|
||
]
|
||
},
|
||
"execution_count": 18,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"smalldb_chinese.max_marginal_relevance_search(question,k=2, fetch_k=3)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "e87c5f91",
|
||
"metadata": {},
|
||
"source": [
|
||
"我们现在可以看到，我们检索的文档中返回了有毒的信息。"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "5a29e8c9",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1.2 解决多样性：最大边际相关性(MMR)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "a2b5c4ae",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"我们刚刚通过一个示例引出了一个问题：如何加强搜索结果的多样性。\n",
|
||
" \n",
|
||
"最大边际相关性(`Maximum marginal relevance`)试图在查询的相关性和结果的多样性之间实现两全其美。"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "2360545c",
|
||
"metadata": {},
|
||
"source": [
|
||
"让我们回到上一节课的一个例子，当我们通过问题对向量数据库进行相似性搜索后\n",
|
||
"\n",
|
||
"我们可以看看前两个文档，只看前几个字符，可以看到它们是相同的。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"id": "9bb2c0a9",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"question = \"what did they say about matlab?\"\n",
|
||
"docs_ss = vectordb.similarity_search(question,k=3)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"id": "f07f8793",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people '"
|
||
]
|
||
},
|
||
"execution_count": 20,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_ss[0].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"id": "e9f7e165",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people '"
|
||
]
|
||
},
|
||
"execution_count": 21,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_ss[1].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 22,
|
||
"id": "e8e142eb",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"question_chinese = \"Matplotlib是什么？\"\n",
|
||
"docs_ss_chinese = vectordb_chinese.similarity_search(question_chinese,k=3)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 23,
|
||
"id": "cf642f66",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'第⼀回：Matplotlib 初相识\\n⼀、认识matplotlib\\nMatplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种'"
|
||
]
|
||
},
|
||
"execution_count": 23,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_ss_chinese[0].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 24,
|
||
"id": "1e9f5cfe",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'第⼀回：Matplotlib 初相识\\n⼀、认识matplotlib\\nMatplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种'"
|
||
]
|
||
},
|
||
"execution_count": 24,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_ss_chinese[1].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "4c4ca1b6",
|
||
"metadata": {},
|
||
"source": [
|
||
"注意：使用`MMR`所得出结果的差异。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 25,
|
||
"id": "222234c5",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_mmr = vectordb.max_marginal_relevance_search(question,k=3)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 26,
|
||
"id": "408935bc",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_mmr_chinese = vectordb_chinese.max_marginal_relevance_search(question_chinese,k=3)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "9076db81",
|
||
"metadata": {},
|
||
"source": [
|
||
"当我们运行MMR后得到结果时，我们可以看到第一个与之前的相同，因为那是最相似的。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 27,
|
||
"id": "93b20226",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people '"
|
||
]
|
||
},
|
||
"execution_count": 27,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_mmr[0].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 28,
|
||
"id": "d0acfaab",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'第⼀回：Matplotlib 初相识\\n⼀、认识matplotlib\\nMatplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种'"
|
||
]
|
||
},
|
||
"execution_count": 28,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_mmr_chinese[0].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "7a93743f",
|
||
"metadata": {},
|
||
"source": [
|
||
"但是当我们进行到第二个时，我们可以看到它是不同的。\n",
|
||
"\n",
|
||
"它在回应中获得了一些多样性。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 29,
|
||
"id": "17d39762",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"\"mathematical work, he feels like he's disc overing truth and beauty in the universe. And \\nhe says it\""
|
||
]
|
||
},
|
||
"execution_count": 29,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_mmr[1].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 30,
|
||
"id": "93d3206c",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"'By Datawhale 数据可视化开源⼩组\\n© Copyright © Copyright 2021.y轴分为左右两个，因此 tick1 对应左侧的轴； tick2 对应右侧的轴。\\nx轴分为上下两个'"
|
||
]
|
||
},
|
||
"execution_count": 30,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_mmr_chinese[1].page_content[:100]"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "b2b909bc",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1.3 解决特殊性：使用元数据"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "7b63c5ee",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"在上一节课中，我们展示了一个问题，是询问了关于文档中某一讲的问题，但得到的结果中也包括了来自其他讲的结果。\n",
|
||
"\n",
|
||
"为了解决这一问题，很多向量数据库都支持对`metadata`的操作。\n",
|
||
"\n",
|
||
"`metadata`为每个嵌入的块(embedded chunk)提供上下文。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 31,
|
||
"id": "3c1a60b2",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"question = \"what did they say about regression in the third lecture?\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 32,
|
||
"id": "ba98df3c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"question_chinese = \"他们在第二讲中对Figure说了些什么？\" "
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "3873525e",
|
||
"metadata": {},
|
||
"source": [
|
||
"现在，我们以手动的方式来解决这个问题，我们会指定一个元数据过滤器`filter`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 33,
|
||
"id": "a8612840",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"docs = vectordb.similarity_search(\n",
|
||
" question,\n",
|
||
" k=3,\n",
|
||
" filter={\"source\":\"docs/cs229_lectures/MachineLearning-Lecture03.pdf\"}\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 34,
|
||
"id": "b46c7e76",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_chinese = vectordb_chinese.similarity_search(\n",
|
||
" question_chinese,\n",
|
||
" k=3,\n",
|
||
" filter={\"source\":\"docs/matplotlib/第二回：艺术画笔见乾坤.pdf\"}\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "869aee28",
|
||
"metadata": {},
|
||
"source": [
|
||
"接下来，我们可以看到结果都来自对应的章节"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 35,
|
||
"id": "97031876",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 0}\n",
|
||
"{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 14}\n",
|
||
"{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 4}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"for d in docs:\n",
|
||
" print(d.metadata)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 36,
|
||
"id": "2708f6ae",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 9}\n",
|
||
"{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 10}\n",
|
||
"{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 0}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"for d in docs_chinese:\n",
|
||
" print(d.metadata)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "5e299f8e",
|
||
"metadata": {},
|
||
"source": [
|
||
"当然，我们不能每次都采用手动的方式来解决这个问题，这会显得不够智能\n",
|
||
"\n",
|
||
"下一小节将要展示通过LLM来解决这个问题"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "ccc2d784",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1.4 解决特殊性：在元数据中使用自查询检索器"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "82ef44b6",
|
||
"metadata": {},
|
||
"source": [
|
||
"我们有一个有趣的挑战：我们通常希望从查询本身来推断元数据。\n",
|
||
"\n",
|
||
"为了解决这个问题，我们可以使用SelfQueryRetriever，它使用LLM来提取：\n",
|
||
" \n",
|
||
"1. 用于向量搜索的查询(`query`)字符串，即：问题\n",
|
||
"2. 要一起传入的元数据过滤器\n",
|
||
"\n",
|
||
"大多数向量数据库支持元数据过滤器，因此不需要任何新的数据库及索引。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 37,
|
||
"id": "b1d06084",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain.llms import OpenAI\n",
|
||
"from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
|
||
"from langchain.chains.query_constructor.base import AttributeInfo"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 48,
|
||
"id": "869c27c0",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"llm = OpenAI(temperature=0)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "acd194c5",
|
||
"metadata": {},
|
||
"source": [
|
||
"`AttributeInfo`是我们可以指定元数据中的不同字段以及它们对应的位置。\n",
|
||
"\n",
|
||
"在元数据中，我们只有两个字段，`source`和`page`。\n",
|
||
"\n",
|
||
"我们将填写每个属性的名称、描述和类型的描述。\n",
|
||
"\n",
|
||
"这些信息实际上将被传递给LLM，所以需要尽可能详细地描述。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 38,
|
||
"id": "0aa5e698",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"metadata_field_info = [\n",
|
||
" AttributeInfo(\n",
|
||
" name=\"source\",\n",
|
||
" description=\"The lecture the chunk is from, should be one of `docs/cs229_lectures/MachineLearning-Lecture01.pdf`, `docs/cs229_lectures/MachineLearning-Lecture02.pdf`, or `docs/cs229_lectures/MachineLearning-Lecture03.pdf`\",\n",
|
||
" type=\"string\",\n",
|
||
" ),\n",
|
||
" AttributeInfo(\n",
|
||
" name=\"page\",\n",
|
||
" description=\"The page from the lecture\",\n",
|
||
" type=\"integer\",\n",
|
||
" ),\n",
|
||
"]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 47,
|
||
"id": "544ad7c1",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"metadata_field_info_chinese = [\n",
|
||
" AttributeInfo(\n",
|
||
" name=\"source\",\n",
|
||
" description=\"讲义来源于 `docs/matplotlib/第一回：Matplotlib初相识.pdf`, `docs/matplotlib/第二回：艺术画笔见乾坤.pdf`, or `docs/matplotlib/第三回：布局格式定方圆.pdf` 的其中之一\",\n",
|
||
" type=\"string\",\n",
|
||
" ),\n",
|
||
" AttributeInfo(\n",
|
||
" name=\"page\",\n",
|
||
" description=\"讲义的那一页\",\n",
|
||
" type=\"integer\",\n",
|
||
" ),\n",
|
||
"]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 42,
|
||
"id": "e7906c15",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"document_content_description = \"Lecture notes\"\n",
|
||
"retriever = SelfQueryRetriever.from_llm(\n",
|
||
" llm,\n",
|
||
" vectordb,\n",
|
||
" document_content_description,\n",
|
||
" metadata_field_info,\n",
|
||
" verbose=True\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 49,
|
||
"id": "d5b99571",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"document_content_description_chinese = \"课堂讲义\"\n",
|
||
"retriever_chinese = SelfQueryRetriever.from_llm(\n",
|
||
" llm,\n",
|
||
" vectordb_chinese,\n",
|
||
" document_content_description_chinese,\n",
|
||
" metadata_field_info_chinese,\n",
|
||
" verbose=True\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 50,
|
||
"id": "79d781b9",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"question = \"what did they say about regression in the third lecture?\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 51,
|
||
"id": "8d9b7e18",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"question_chinese = \"他们在第二讲中对Figure说了些什么？\" "
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "c51778b0-1fcd-40a4-bd6b-0f13fec8acb1",
|
||
"metadata": {},
|
||
"source": [
|
||
"当你第一次执行下一行时，你会收到关于predict_and_parse已被弃用的**警告**。 这可以安全地忽略。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 55,
|
||
"id": "1d4f9f7d",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"query='regression' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='source', value='docs/cs229_lectures/MachineLearning-Lecture03.pdf') limit=None\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"docs = retriever.get_relevant_documents(question)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 52,
|
||
"id": "ea39a97e",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"query='Figure' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='source', value='docs/matplotlib/第二回：艺术画笔见乾坤.pdf') limit=None\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"docs_chinese = retriever_chinese.get_relevant_documents(question_chinese)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 45,
|
||
"id": "db04374e",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 14}\n",
|
||
"{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 0}\n",
|
||
"{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10}\n",
|
||
"{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 10}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"for d in docs:\n",
|
||
" print(d.metadata)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 53,
|
||
"id": "143061f5",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 9}\n",
|
||
"{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 10}\n",
|
||
"{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 0}\n",
|
||
"{'source': 'docs/matplotlib/第二回：艺术画笔见乾坤.pdf', 'page': 6}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"for d in docs_chinese:\n",
|
||
" print(d.metadata)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "297b8168",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1.5 其他技巧：压缩"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "564144da",
|
||
"metadata": {},
|
||
"source": [
|
||
"另一种提高检索到的文档质量的方法是压缩。\n",
|
||
"\n",
|
||
"与查询最相关的信息可能隐藏在具有大量不相关文本的文档中。\n",
|
||
"\n",
|
||
"在应用程序中传递完整的文档可能会导致更昂贵的LLM调用和更差的响应。\n",
|
||
"\n",
|
||
"上下文压缩就是为了解决这个问题。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 56,
|
||
"id": "a060cf74",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain.retrievers import ContextualCompressionRetriever\n",
|
||
"from langchain.retrievers.document_compressors import LLMChainExtractor"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 62,
|
||
"id": "038649c8",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def pretty_print_docs(docs):\n",
|
||
" print(f\"\\n{'-' * 100}\\n\".join([f\"Document {i+1}:\\n\\n\" + d.page_content for i, d in enumerate(docs)]))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 61,
|
||
"id": "fc686cf2",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"llm = OpenAI(temperature=0)\n",
|
||
"compressor = LLMChainExtractor.from_llm(llm) # 压缩器"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 59,
|
||
"id": "82794397",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"compression_retriever = ContextualCompressionRetriever(\n",
|
||
" base_compressor=compressor,\n",
|
||
" base_retriever=vectordb.as_retriever()\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 63,
|
||
"id": "915598f8",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"compression_retriever_chinese = ContextualCompressionRetriever(\n",
|
||
" base_compressor=compressor,\n",
|
||
" base_retriever=vectordb_chinese.as_retriever()\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 60,
|
||
"id": "cde86848",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Document 1:\n",
|
||
"\n",
|
||
"\"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms.\"\n",
|
||
"----------------------------------------------------------------------------------------------------\n",
|
||
"Document 2:\n",
|
||
"\n",
|
||
"\"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms.\"\n",
|
||
"----------------------------------------------------------------------------------------------------\n",
|
||
"Document 3:\n",
|
||
"\n",
|
||
"\"And the student said, \"Oh, it was the MATLAB.\" So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it.\"\n",
|
||
"----------------------------------------------------------------------------------------------------\n",
|
||
"Document 4:\n",
|
||
"\n",
|
||
"\"And the student said, \"Oh, it was the MATLAB.\" So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it.\"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"question = \"what did they say about matlab?\"\n",
|
||
"compressed_docs = compression_retriever.get_relevant_documents(question)\n",
|
||
"pretty_print_docs(compressed_docs)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 64,
|
||
"id": "39726b24",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Document 1:\n",
|
||
"\n",
|
||
"Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种静态，动态，交互式的图表。\n",
|
||
"----------------------------------------------------------------------------------------------------\n",
|
||
"Document 2:\n",
|
||
"\n",
|
||
"Matplotlib 是⼀个 Python 2D 绘图库，能够以多种硬拷⻉格式和跨平台的交互式环境⽣成出版物质量的图形，⽤来绘制各种静态，动态，交互式的图表。\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"question_chinese = \"Matplotlib是什么？\"\n",
|
||
"compressed_docs_chinese = compression_retriever_chinese.get_relevant_documents(question_chinese)\n",
|
||
"pretty_print_docs(compressed_docs_chinese)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "049b2601",
|
||
"metadata": {},
|
||
"source": [
|
||
"现在当我们提出问题后，查看结果文档\n",
|
||
"\n",
|
||
"我们可以看到两件事。\n",
|
||
"\n",
|
||
"1. 它们比正常文档短很多\n",
|
||
"2. 仍然有一些重复的东西，这是因为在底层我们使用的是语义搜索算法。\n",
|
||
"\n",
|
||
"这就是我们在本课程前面使用MMR解决的问题。\n",
|
||
"\n",
|
||
"这是一个很好的例子，你可以结合各种技术得到最好的可能结果。"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "82c4fc4d",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 二、结合各种技术"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "54432975",
|
||
"metadata": {},
|
||
"source": [
|
||
"为了做到这一点，我们在从向量数据库创建检索器时，可以将搜索类型设置为MMR。\n",
|
||
"\n",
|
||
"然后我们可以重新运行这个过程，看到我们返回的是一个过滤过的结果集，其中不包含任何重复的信息。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 67,
|
||
"id": "161ae1ad",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"compression_retriever = ContextualCompressionRetriever(\n",
|
||
" base_compressor=compressor,\n",
|
||
" base_retriever=vectordb.as_retriever(search_type = \"mmr\")\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 66,
|
||
"id": "cd6396bb",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"compression_retriever_chinese = ContextualCompressionRetriever(\n",
|
||
" base_compressor=compressor,\n",
|
||
" base_retriever=vectordb_chinese.as_retriever(search_type = \"mmr\")\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 68,
|
||
"id": "e77ccae1",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n",
|
||
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 8.0 seconds as it raised RateLimitError: Rate limit reached for default-text-davinci-003 in organization org-6XQxEC6cDJisHUyyNTit088C on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Document 1:\n",
|
||
"\n",
|
||
"\"MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to plot data. And it's sort of an extremely easy to learn tool to use for implementing a lot of learning algorithms.\"\n",
|
||
"----------------------------------------------------------------------------------------------------\n",
|
||
"Document 2:\n",
|
||
"\n",
|
||
"\"And the student said, \"Oh, it was the MATLAB.\" So for those of you that don't know MATLAB yet, I hope you do learn it. It's not hard, and we'll actually have a short MATLAB tutorial in one of the discussion sections for those of you that don't know it.\"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"question = \"what did they say about matlab?\"\n",
|
||
"compressed_docs = compression_retriever.get_relevant_documents(question)\n",
|
||
"pretty_print_docs(compressed_docs)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "fe68a14b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"question_chinese = \"Matplotlib是什么？\"\n",
|
||
"compressed_docs_chinese = compression_retriever_chinese.get_relevant_documents(question_chinese)\n",
|
||
"pretty_print_docs(compressed_docs_chinese)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "6c2b63e1",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 三、其他类型的检索"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "3e777a7b",
|
||
"metadata": {},
|
||
"source": [
|
||
"值得注意的是，vetordb并不是唯一一种检索文档的工具。\n",
|
||
"\n",
|
||
"`LangChain`检索器抽象包括其他检索文档的方式，如：`TF-IDF` 或 `SVM`。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 69,
|
||
"id": "83d2e808",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain.retrievers import SVMRetriever\n",
|
||
"from langchain.retrievers import TFIDFRetriever\n",
|
||
"from langchain.document_loaders import PyPDFLoader\n",
|
||
"from langchain.text_splitter import RecursiveCharacterTextSplitter"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 71,
|
||
"id": "bcf5b760",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# 加载PDF\n",
|
||
"loader = PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture01.pdf\")\n",
|
||
"pages = loader.load()\n",
|
||
"all_page_text = [p.page_content for p in pages]\n",
|
||
"joined_page_text = \" \".join(all_page_text)\n",
|
||
"\n",
|
||
"# 分割文本\n",
|
||
"text_splitter = RecursiveCharacterTextSplitter(chunk_size = 1500,chunk_overlap = 150)\n",
|
||
"splits = text_splitter.split_text(joined_page_text)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "1657e768",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# 加载PDF\n",
|
||
"loader_chinese = PyPDFLoader(\"docs/matplotlib/第一回：Matplotlib初相识.pdf\")\n",
|
||
"pages_chinese = loader_chinese.load()\n",
|
||
"all_page_text_chinese = [p.page_content for p in pages]\n",
|
||
"joined_page_text_chinese = \" \".join(all_page_text)\n",
|
||
"\n",
|
||
"# 分割文本\n",
|
||
"text_splitter_chinese = RecursiveCharacterTextSplitter(chunk_size = 1500,chunk_overlap = 150)\n",
|
||
"splits_chinese = text_splitter_chinese.split_text(joined_page_text_chinese)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "59abbaff",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# 检索\n",
|
||
"svm_retriever = SVMRetriever.from_texts(splits, embedding)\n",
|
||
"tfidf_retriever = TFIDFRetriever.from_texts(splits)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "9bb0d781",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# 检索\n",
|
||
"svm_retriever_splitter = SVMRetriever.from_texts(splits_chinese, embedding)\n",
|
||
"tfidf_retriever_splitter = TFIDFRetriever.from_texts(splits_chinese)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7885389e",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"question = \"What are major topics for this class?\" # 这门课的主要主题是什么？\n",
|
||
"docs_svm = svm_retriever.get_relevant_documents(question)\n",
|
||
"docs_svm[0]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2a1659c0",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"question = \"what did they say about matlab?\" # 他们关于Matlab说了些什么？\n",
|
||
"docs_tfidf = tfidf_retriever.get_relevant_documents(question)\n",
|
||
"docs_tfidf[0]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cc823bea",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"question_chinese = \"这门课的主要主题是什么？\" \n",
|
||
"docs_svm_chinese = svm_retriever_chinese.get_relevant_documents(question_chinese)\n",
|
||
"docs_svm_chinese[0]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "01eb9d43",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"question_chinese = \"Matplotlib是什么？\"\n",
|
||
"docs_tfidf_chinese = tfidf_retriever_chinese.get_relevant_documents(question_chinese)\n",
|
||
"docs_tfidf_chinese[0]"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.9.16"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|