{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 机器学习100天——第1天：数据预处理（Data Preprocessing）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "搭建anaconda环境，参考 https://zhuanlan.zhihu.com/p/33358809\n", "\n", "## 第一步：导入需要的库\n", "这两个是我们每次都需要导入的库。NumPy包含数学计算函数。Pandas用于导入和管理数据集。" ] }, { "cell_type": "code", <<<<<<< Updated upstream "execution_count": 4, ======= "execution_count": 2, >>>>>>> Stashed changes "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第二步：导入数据集\n", "数据集通常是.csv格式。CSV文件以文本形式保存表格数据。文件的每一行是一条数据记录。我们使用Pandas的read_csv方法读取本地csv文件为一个数据帧。然后，从数据帧中制作自变量和因变量的矩阵和向量。" ] }, { "cell_type": "code", <<<<<<< Updated upstream "execution_count": 7, ======= "execution_count": 6, >>>>>>> Stashed changes "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ <<<<<<< Updated upstream "Step 2: Importing dataset\nX\n[['France' 44.0 72000.0]\n ['Spain' 27.0 48000.0]\n ['Germany' 30.0 54000.0]\n ['Spain' 38.0 61000.0]\n ['Germany' 40.0 nan]\n ['France' 35.0 58000.0]\n ['Spain' nan 52000.0]\n ['France' 48.0 79000.0]\n ['Germany' 50.0 83000.0]\n ['France' 37.0 67000.0]]\nY\n['No' 'Yes' 'No' 'No' 'Yes' 'Yes' 'No' 'Yes' 'No' 'Yes']\n" ======= "Step 2: Importing dataset\nX:\n[['France' 44.0 72000.0]\n ['Spain' 27.0 48000.0]\n ['Germany' 30.0 54000.0]\n ['Spain' 38.0 61000.0]\n ['Germany' 40.0 nan]\n ['France' 35.0 58000.0]\n ['Spain' nan 52000.0]\n ['France' 48.0 79000.0]\n ['Germany' 50.0 83000.0]\n ['France' 37.0 67000.0]]\nY:\n['No' 'Yes' 'No' 'No' 'Yes' 'Yes' 'No' 'Yes' 'No' 'Yes']\n" >>>>>>> Stashed changes ] }, { "output_type": "execute_result", "data": { "text/plain": [ " Country Age Salary Purchased\n", "0 France 44.0 72000.0 No\n", "1 Spain 27.0 48000.0 Yes\n", "2 Germany 30.0 54000.0 No\n", "3 Spain 38.0 61000.0 No\n", "4 Germany 40.0 NaN Yes\n", "5 France 35.0 58000.0 Yes\n", "6 Spain NaN 52000.0 No\n", "7 France 48.0 79000.0 Yes\n", "8 Germany 50.0 83000.0 No\n", "9 France 37.0 67000.0 Yes" ], "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
CountryAgeSalaryPurchased
0France44.072000.0No
1Spain27.048000.0Yes
2Germany30.054000.0No
3Spain38.061000.0No
4Germany40.0NaNYes
5France35.058000.0Yes
6SpainNaN52000.0No
7France48.079000.0Yes
8Germany50.083000.0No
9France37.067000.0Yes
\n
" }, "metadata": {}, "execution_count": 6 } ], "source": [ "dataset = pd.read_csv('../datasets/Data.csv')\n", "# 不包括最后一列的所有列\n", "X = dataset.iloc[ : , :-1].values\n", "#取最后一列\n", "Y = dataset.iloc[ : , 3].values\n", "print(\"Step 2: Importing dataset\")\n", "print(\"X:\")\n", "print(X)\n", "print(\"Y:\")\n", "print(Y)\n", "dataset.head(100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第三步：处理丢失数据\n", "我们得到的数据很少是完整的。数据可能因为各种原因丢失，为了不降低机器学习模型的性能，需要处理数据。我们可以用整列的平均值或中间值替换丢失的数据。我们用sklearn.preprocessing库中的Imputer类完成这项任务。" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[[44.0 72000.0]\n [27.0 48000.0]\n [30.0 54000.0]\n [38.0 61000.0]]\n" ] }, { "output_type": "error", "ename": "ValueError", "evalue": "'X' and 'missing_values' types are expected to be both numerical. Got X.dtype=float64 and type(missing_values)=.", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mValueError\u001b[0m Traceback (most recent call last)", "\u001b[1;32m\u001b[0m in \u001b[0;36m\u001b[1;34m\u001b[0m\n\u001b[0;32m 6\u001b[0m \u001b[1;31m#imputer = Imputer(missing_values = \"NaN\", strategy = \"mean\", axis = 0)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 7\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m[\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m4\u001b[0m \u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m \u001b[0mimputer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mimputer\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m[\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m4\u001b[0m \u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 9\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m[\u001b[0m \u001b[1;33m:\u001b[0m \u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mimputer\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m[\u001b[0m \u001b[1;33m:\u001b[0m \u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 10\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"---------------------\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\sklearn\\impute\\_base.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, X, y)\u001b[0m\n\u001b[0;32m 275\u001b[0m \u001b[0mself\u001b[0m \u001b[1;33m:\u001b[0m \u001b[0mSimpleImputer\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 276\u001b[0m \"\"\"\n\u001b[1;32m--> 277\u001b[1;33m \u001b[0mX\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_validate_input\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0min_fit\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 278\u001b[0m \u001b[0msuper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_fit_indicator\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 279\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n", "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\sklearn\\impute\\_base.py\u001b[0m in \u001b[0;36m_validate_input\u001b[1;34m(self, X, in_fit)\u001b[0m\n\u001b[0;32m 251\u001b[0m \u001b[1;32mraise\u001b[0m \u001b[0mve\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 252\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 253\u001b[1;33m \u001b[0m_check_inputs_dtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmissing_values\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 254\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkind\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32min\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;34m\"i\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"u\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"f\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"O\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 255\u001b[0m raise ValueError(\"SimpleImputer does not support data with dtype \"\n", "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\sklearn\\impute\\_base.py\u001b[0m in \u001b[0;36m_check_inputs_dtype\u001b[1;34m(X, missing_values)\u001b[0m\n\u001b[0;32m 23\u001b[0m if (X.dtype.kind in (\"f\", \"i\", \"u\") and\n\u001b[0;32m 24\u001b[0m not isinstance(missing_values, numbers.Real)):\n\u001b[1;32m---> 25\u001b[1;33m raise ValueError(\"'X' and 'missing_values' types are expected to be\"\n\u001b[0m\u001b[0;32m 26\u001b[0m \u001b[1;34m\" both numerical. Got X.dtype={} and \"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 27\u001b[0m \u001b[1;34m\" type(missing_values)={}.\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", "\u001b[1;31mValueError\u001b[0m: 'X' and 'missing_values' types are expected to be both numerical. Got X.dtype=float64 and type(missing_values)=." ] } ], "source": [ "# If you use the newest version of sklearn, use the lines of code commented out\n", <<<<<<< Updated upstream "from sklearn.impute import SimpleImputer\n", "imputer = SimpleImputer(missing_values=np.nan, strategy=\"mean\")\n", "#from sklearn.preprocessing import Imputer\n", "# axis=0表示按列进行\n", "#imputer = Imputer(missing_values = \"NaN\", strategy = \"mean\", axis = 0)\n", "print(X[ 0:4 , 1:3])\n", "imputer = imputer.fit(X[ 0:4 , 1:3])\n", ======= "# from sklearn.impute import SimpleImputer\n", "# imputer = SimpleImputer(missing_values=\"NaN\", strategy=\"mean\")\n", "from sklearn.preprocessing import Imputer\n", "# axis=0表示按列进行\n", "imputer = Imputer(missing_values = \"NaN\", strategy = \"mean\", axis = 0)\n", "print(imputer)\n", "imputer = imputer.fit(X[ : , 1:3])\n", >>>>>>> Stashed changes "X[ : , 1:3] = imputer.transform(X[ : , 1:3])\n", "print(\"---------------------\")\n", "print(\"Step 3: Handling the missing data\")\n", "print(\"step2\")\n", "print(\"X\")\n", "print(X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第四步：解析分类数据\n", "分类数据指的是含有标签值而不是数字值的变量。取值范围通常是固定的。例如\"Yes\"和\"No\"不能用于模型的数学计算，所以需要解析成数字。为实现这一功能，我们从sklearn.preprocessing库导入LabelEncoder类。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------------------\n", "Step 4: Encoding categorical data\n", "X\n", "[[1.00000000e+00 0.00000000e+00 0.00000000e+00 4.40000000e+01\n", " 7.20000000e+04]\n", " [0.00000000e+00 0.00000000e+00 1.00000000e+00 2.70000000e+01\n", " 4.80000000e+04]\n", " [0.00000000e+00 1.00000000e+00 0.00000000e+00 3.00000000e+01\n", " 5.40000000e+04]\n", " [0.00000000e+00 0.00000000e+00 1.00000000e+00 3.80000000e+01\n", " 6.10000000e+04]\n", " [0.00000000e+00 1.00000000e+00 0.00000000e+00 4.00000000e+01\n", " 6.37777778e+04]\n", " [1.00000000e+00 0.00000000e+00 0.00000000e+00 3.50000000e+01\n", " 5.80000000e+04]\n", " [0.00000000e+00 0.00000000e+00 1.00000000e+00 3.87777778e+01\n", " 5.20000000e+04]\n", " [1.00000000e+00 0.00000000e+00 0.00000000e+00 4.80000000e+01\n", " 7.90000000e+04]\n", " [0.00000000e+00 1.00000000e+00 0.00000000e+00 5.00000000e+01\n", " 8.30000000e+04]\n", " [1.00000000e+00 0.00000000e+00 0.00000000e+00 3.70000000e+01\n", " 6.70000000e+04]]\n", "Y\n", "[0 1 0 0 1 1 0 1 0 1]\n" ] } ], "source": [ "from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n", "labelencoder_X = LabelEncoder()\n", "X[ : , 0] = labelencoder_X.fit_transform(X[ : , 0])\n", "#Creating a dummy variable\n", "onehotencoder = OneHotEncoder(categorical_features = [0])\n", "X = onehotencoder.fit_transform(X).toarray()\n", "labelencoder_Y = LabelEncoder()\n", "Y = labelencoder_Y.fit_transform(Y)\n", "print(\"---------------------\")\n", "print(\"Step 4: Encoding categorical data\")\n", "print(\"X\")\n", "print(X)\n", "print(\"Y\")\n", "print(Y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第五步：拆分数据集为测试集合和训练集合\n", "把数据集拆分成两个：一个是用来训练模型的训练集合，另一个是用来验证模型的测试集合。两者比例一般是80:20。我们导入sklearn.model_selection库中的train_test_split()方法。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------------------\n", "Step 5: Splitting the datasets into training sets and Test sets\n", "X_train\n", "[[ 0.00000000e+00 1.00000000e+00 0.00000000e+00 4.00000000e+01\n", " 6.37777778e+04]\n", " [ 1.00000000e+00 0.00000000e+00 0.00000000e+00 3.70000000e+01\n", " 6.70000000e+04]\n", " [ 0.00000000e+00 0.00000000e+00 1.00000000e+00 2.70000000e+01\n", " 4.80000000e+04]\n", " [ 0.00000000e+00 0.00000000e+00 1.00000000e+00 3.87777778e+01\n", " 5.20000000e+04]\n", " [ 1.00000000e+00 0.00000000e+00 0.00000000e+00 4.80000000e+01\n", " 7.90000000e+04]\n", " [ 0.00000000e+00 0.00000000e+00 1.00000000e+00 3.80000000e+01\n", " 6.10000000e+04]\n", " [ 1.00000000e+00 0.00000000e+00 0.00000000e+00 4.40000000e+01\n", " 7.20000000e+04]\n", " [ 1.00000000e+00 0.00000000e+00 0.00000000e+00 3.50000000e+01\n", " 5.80000000e+04]]\n", "X_test\n", "[[ 0.00000000e+00 1.00000000e+00 0.00000000e+00 3.00000000e+01\n", " 5.40000000e+04]\n", " [ 0.00000000e+00 1.00000000e+00 0.00000000e+00 5.00000000e+01\n", " 8.30000000e+04]]\n", "Y_train\n", "[1 1 1 0 1 0 0 1]\n", "Y_test\n", "[0 0]\n" ] } ], "source": [ "from sklearn.model_selection import train_test_split\n", "X_train, X_test, Y_train, Y_test = train_test_split( X , Y , test_size = 0.2, random_state = 0)\n", "print(\"---------------------\")\n", "print(\"Step 5: Splitting the datasets into training sets and Test sets\")\n", "print(\"X_train\")\n", "print(X_train)\n", "print(\"X_test\")\n", "print(X_test)\n", "print(\"Y_train\")\n", "print(Y_train)\n", "print(\"Y_test\")\n", "print(Y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第六步：特征量化\n", "大部分模型算法使用两点间的欧氏距离表示，但此特征在幅度、单位和范围姿态问题上变化很大。在距离计算中，高幅度的特征比低幅度特征权重更大。可用特征标准化或Z值归一化解决。导入sklearn.preprocessing库的StandardScalar类。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------------------\n", "Step 6: Feature Scaling\n", "X_train\n", "[[-1. 2.64575131 -0.77459667 0.26306757 0.12381479]\n", " [ 1. -0.37796447 -0.77459667 -0.25350148 0.46175632]\n", " [-1. -0.37796447 1.29099445 -1.97539832 -1.53093341]\n", " [-1. -0.37796447 1.29099445 0.05261351 -1.11141978]\n", " [ 1. -0.37796447 -0.77459667 1.64058505 1.7202972 ]\n", " [-1. -0.37796447 1.29099445 -0.0813118 -0.16751412]\n", " [ 1. -0.37796447 -0.77459667 0.95182631 0.98614835]\n", " [ 1. -0.37796447 -0.77459667 -0.59788085 -0.48214934]]\n", "X_test\n", "[[ 0. 0. 0. -1. -1.]\n", " [ 0. 0. 0. 1. 1.]]\n" ] } ], "source": [ "from sklearn.preprocessing import StandardScaler\n", "sc_X = StandardScaler()\n", "X_train = sc_X.fit_transform(X_train)\n", "X_test = sc_X.transform(X_test)\n", "print(\"---------------------\")\n", "print(\"Step 6: Feature Scaling\")\n", "print(\"X_train\")\n", "print(X_train)\n", "print(\"X_test\")\n", "print(X_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "完整的项目请前往Github项目100-Days-Of-ML-Code查看。有任何的建议或者意见欢迎在issue中提出~" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3.8.3 64-bit (conda)", "metadata": { "interpreter": { "hash": "1b78ff499ec469310b6a6795c4effbbfc85eb20a6ba0cf828a15721670711b2c" } } }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3-final" } }, "nbformat": 4, "nbformat_minor": 2 }