andre/prompt-engineering-for-developers
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update 3.存储 .ipynb

Browse Source 
HuggingFace -> Hugging Face
This commit is contained in:
Ikko Eltociear Ashimine
2023-06-09 00:43:20 +09:00
committed by GitHub
parent b39c4766f3
commit 6186bee0ef
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
content/LangChain for LLM Application Development/3.存储 .ipynb
View File

@ -1198,7 +1198,7 @@
"ChatGPT使用一种基于字节对编码（Byte Pair Encoding，BPE）的方法来进行tokenization（将输入文本拆分为token）。 \n",
"BPE是一种常见的tokenization技术，它将输入文本分割成较小的子词单元。 \n",
"\n",
"OpenAI在其官方GitHub上公开了一个最新的开源Python库：tiktoken，这个库主要是用来计算tokens数量的。相比较HuggingFace的tokenizer，其速度提升了好几倍 <https://github.com/openai/tiktoken>\n",
"OpenAI在其官方GitHub上公开了一个最新的开源Python库：tiktoken，这个库主要是用来计算tokens数量的。相比较Hugging Face的tokenizer，其速度提升了好几倍 <https://github.com/openai/tiktoken>\n",
"\n",
"具体token计算方式,特别是汉字和英文单词的token区别，参考 <https://www.zhihu.com/question/594159910>\n"
]