{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 机器学习与社会科学应用"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 第三章 经典分类算法"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 第四节 决策树算法"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"郭峰 \n",
" 教授、博士生导师 \n",
"上海财经大学公共经济与管理学院 \n",
"上海财经大学数实融合与智能治理实验室 \n",
"邮箱：guofengsfi@163.com "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"本节目录 \n",
"4.1. 分类决策树 \n",
"4.2. 分类决策树案例 \n",
"4.3. 回归决策树"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4.1. 分类决策树"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_iris \n",
"from sklearn.model_selection import train_test_split \n",
"from sklearn.tree import DecisionTreeClassifier \n",
"from sklearn.metrics import accuracy_score, classification_report \n",
"\n",
"# 加载鸢尾花数据集 \n",
"data = load_iris() \n",
" \n",
"# 提取特征和标签 \n",
"X = data.data \n",
"y = data.target \n",
"\n",
"# 划分训练集和测试集 \n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=666) \n",
"\n",
"# 创建决策树分类器 \n",
"dt_classifier = DecisionTreeClassifier() \n",
"\n",
"# 在训练集上训练模型 \n",
"dt_classifier.fit(X_train, y_train) \n",
"\n",
"# 在测试集上进行预测 \n",
"y_pred = dt_classifier.predict(X_test) \n",
"\n",
"# 评估模型性能 \n",
"accuracy = accuracy_score(y_test, y_pred) \n",
"print(\"Accuracy:\", accuracy) \n",
"\n",
"# 输出分类报告 \n",
"print(\"Classification Report:\") \n",
"print(classification_report(y_test, y_pred)) "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_iris \n",
"from sklearn.model_selection import train_test_split, GridSearchCV \n",
"from sklearn.tree import DecisionTreeClassifier \n",
"from sklearn.metrics import accuracy_score, classification_report \n",
"\n",
"# 加载鸢尾花数据集 \n",
"data = load_iris() \n",
"\n",
"# 提取特征和标签 \n",
"X = data.data \n",
"y = data.target \n",
"\n",
"# 划分训练集和测试集 \n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=666) \n",
"\n",
"# 创建决策树分类器 \n",
"dt_classifier = DecisionTreeClassifier() "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 定义调参的超参数组合，以字典的形式指定参数和参数范围 \n",
"param_grid = { \n",
" 'criterion': ['gini', 'entropy'], \n",
" 'max_depth': [None, 5, 10, 15], \n",
" 'min_samples_split': [2, 5, 10], \n",
" 'min_samples_leaf': [1, 2, 5] \n",
"} \n",
"\n",
"# 初始化网格搜索对象 \n",
"grid_search = GridSearchCV(estimator=dt_classifier, param_grid=param_grid, cv=5, n_jobs=-1) \n",
"\n",
"# 在训练集上进行网格搜索，寻找最佳参数组合 \n",
"grid_search.fit(X_train, y_train) \n",
"\n",
"# 输出最佳参数组合和最佳模型 \n",
"print(\"Best Parameters:\", grid_search.best_params_) \n",
"best_model = grid_search.best_estimator_ \n",
"\n",
"# 在测试集上进行预测 \n",
"y_pred = best_model.predict(X_test) \n",
"\n",
"# 评估模型性能 \n",
"accuracy = accuracy_score(y_test, y_pred) \n",
"print(\"Accuracy:\", accuracy) \n",
"\n",
"# 输出分类报告 \n",
"print(\"Classification Report:\") \n",
"print(classification_report(y_test, y_pred)) \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4.2. 分类决策树案例"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"案例：坦特尼克号"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 数据预处理"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 数据预处理\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"path =\"D:/python/机器学习与社会科学应用/演示数据/03经典分类算法/titanic/data/\"\n",
"f1 = open(path+\"train.csv\",encoding='utf8')\n",
"data = pd.read_csv(path+\"train.csv\", encoding='utf8')\n",
"data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 数据预处理\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"path =\"D:/python/机器学习与社会科学应用/演示数据/03经典分类算法/titanic/data/\"\n",
"data = pd.read_csv(path+\"train.csv\", encoding='utf8')\n",
"data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 数据清理\n",
"# 丢弃无用的数据\n",
"data.drop(['Name', 'Ticket', 'Cabin'], axis=1, inplace=True)\n",
"# 处理性别数据\n",
"data['Sex'] = (data['Sex'] == 'male').astype('int')\n",
"# 处理登船港口数据\n",
"labels = data['Embarked'].unique().tolist()\n",
"data['Embarked'] = data['Embarked'].apply(lambda n: labels.index(n))\n",
"# 处理缺失数据\n",
"data = data.fillna(0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 划分数据集"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"\n",
"y = data['Survived'].values\n",
"X = data.drop(['Survived'], axis=1).values\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"\n",
"print('train dataset: {0}; test dataset: {1}'.format(X_train.shape, X_test.shape))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 训练模型"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.tree import DecisionTreeClassifier\n",
"clf = DecisionTreeClassifier()\n",
"clf.fit(X_train, y_train)\n",
"train_score = clf.score(X_train, y_train)\n",
"test_score = clf.score(X_test, y_test)\n",
"print('train score: {0}; test score: {1}'.format(train_score, test_score))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 调参--选择最优模型"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"决策树模型里面有一个参数max_depth控制决策树的最大深度，不同深度将会对模型产生不同的影响。\n",
"下面通过遍历不同的深度参数来训练决策树，并画出模型的socre曲线。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 参数选择max_depth"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 参数选择 max_depth\n",
"def cv_score(d):\n",
" clf = DecisionTreeClassifier(max_depth=d)\n",
" clf.fit(X_train, y_train)\n",
" tr_score = clf.score(X_train, y_train)\n",
" cv_score = clf.score(X_test, y_test)\n",
" return (tr_score, cv_score)\n",
"\n",
"depths = range(2, 15)\n",
"scores = [cv_score(d) for d in depths]\n",
"tr_scores = [s[0] for s in scores]\n",
"cv_scores = [s[1] for s in scores]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 选出最佳深度及score\n",
"best_score_index = np.argmax(cv_scores)\n",
"best_score = cv_scores[best_score_index]\n",
"best_param = depths[best_score_index]\n",
"print('best param: {0}; best score: {1}'.format(best_param, best_score))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 画出不同深度对应的score\n",
"plt.figure(figsize=(10, 6), dpi=144)\n",
"plt.grid()\n",
"plt.xlabel('max depth of decision tree')\n",
"plt.ylabel('score')\n",
"plt.plot(depths, cv_scores, '.g-', label='test score')\n",
"plt.plot(depths, tr_scores, '.r--', label='training score')\n",
"plt.legend()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 不同阈值对决策树的影响"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"如果节点的分裂导致的不纯度的下降程度大于或者等于这个节点的值，那么这个节点将会被分裂。 决策树的参数min_impurity_decrease树早期生长的阈值。如果一个节点的不纯度超过阈值那么这个节点将会分裂，否则它还是一片叶子。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 训练模型，并计算评分\n",
"def cv_score(val):\n",
" clf = DecisionTreeClassifier(criterion='gini',min_impurity_decrease=val)\n",
" clf.fit(X_train, y_train)\n",
" tr_score = clf.score(X_train, y_train)\n",
" cv_score = clf.score(X_test, y_test)\n",
" return (tr_score, cv_score)\n",
"\n",
"# 指定参数范围，分别训练模型，并计算评分\n",
"values = np.linspace(0, 0.01, 100)\n",
"scores = [cv_score(v) for v in values]\n",
"tr_scores = [s[0] for s in scores]\n",
"cv_scores = [s[1] for s in scores]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 找出评分最高的模型参数\n",
"best_score_index = np.argmax(cv_scores)\n",
"best_score = cv_scores[best_score_index]\n",
"best_param = values[best_score_index]\n",
"print('best param: {0}; best score: {1}'.format(best_param, best_score))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 画出模型参数与模型评分的关系\n",
"plt.figure(figsize=(10, 6), dpi=144)\n",
"plt.grid()\n",
"plt.xlabel('threshold of gini')\n",
"plt.ylabel('score')\n",
"plt.plot(values, cv_scores, '.g-', label='test score')\n",
"plt.plot(values, tr_scores, '.r--', label='training score')\n",
"plt.legend()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 使用GridSearchCV类同时寻找最佳的min_impurity_decrease、criterion。\n",
"from sklearn.model_selection import GridSearchCV\n",
"\n",
"entropy_thresholds = np.linspace(0, 1, 50)\n",
"gini_thresholds = np.linspace(0, 0.01, 100)\n",
"\n",
"# Set the parameters by cross-validation\n",
"param_grid = [{'criterion': ['entropy'], \n",
" 'min_impurity_decrease': entropy_thresholds},\n",
" {'criterion': ['gini'], \n",
" 'min_impurity_decrease': gini_thresholds},\n",
" {'max_depth': range(2, 10)},\n",
" {'min_samples_split': range(2, 30, 2)}]\n",
"\n",
"clf = GridSearchCV(DecisionTreeClassifier(), param_grid, cv=5)\n",
"clf.fit(X, y)\n",
"print(\"best param: {0}\\nbest score: {1}\".format(clf.best_params_, \n",
" clf.best_score_))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4.3. 回归决策树"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"在sklearn中我们可以用来提高决策树泛化能力的超参数主要有： \n",
"- max_depth:树的最大深度,也就是说当树的深度到达max_depth的时候无论还有多少可以分支的特征,决策树都会停止运算.\n",
"- min_samples_split: 分裂所需的最小数量的节点数.当叶节点的样本数量小于该参数后,则不再生成分支.该分支的标签分类以该分支下标签最多的类别为准 \n",
"- min_samples_leaf; 一个分支所需要的最少样本数,如果在分支之后,某一个新增叶节点的特征样本数小于该超参数,则退回,不再进行剪枝.退回后的叶节点的标签以该叶节点中最多的标签你为准\n",
"- min_weight_fraction_leaf: 最小的权重系数 \n",
"- max_leaf_nodes:最大叶节点数,None时无限制,取整数时,忽略max_depth\n",
"- 与分类决策树一样的地方在于,最大深度的增加虽然可以增加对训练集拟合能力的增强,但这也就可能意味着其泛化能力的下降"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.tree import DecisionTreeRegressor\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# 创建随机数据集\n",
"rng = np.random.RandomState(1)\n",
"X = np.sort(10 * rng.rand(160, 1), axis=0)\n",
"y = np.sin(X).ravel()\n",
"y[::5] += 2 * (0.5 - rng.rand(32)) # 每五个点增加一次噪音\n",
"\n",
"# 拟合回归决策树\n",
"regr_1 = DecisionTreeRegressor(max_depth=2)\n",
"regr_2 = DecisionTreeRegressor(max_depth=5)\n",
"regr_3 = DecisionTreeRegressor(max_depth=8)\n",
"regr_1.fit(X, y)\n",
"regr_2.fit(X, y)\n",
"regr_3.fit(X, y)\n",
"\n",
"# 预测\n",
"X_test = np.arange(0.0, 10.0, 0.01)[:, np.newaxis]\n",
"y_1 = regr_1.predict(X_test)\n",
"y_2 = regr_2.predict(X_test)\n",
"y_3 = regr_3.predict(X_test)\n",
"\n",
"# 画出结果\n",
"plt.figure()\n",
"plt.scatter(X, y, s=20, edgecolor=\"black\",\n",
" c=\"darkorange\", label=\"data\")\n",
"plt.plot(X_test, y_1, color=\"cornflowerblue\",\n",
" label=\"max_depth=2\", linewidth=2)\n",
"plt.plot(X_test, y_2, color=\"yellowgreen\", label=\"max_depth=5\", linewidth=2)\n",
"plt.plot(X_test, y_3, color=\"r\", label=\"max_depth=8\", linewidth=2)\n",
"plt.xlabel(\"data\")\n",
"plt.ylabel(\"target\")\n",
"plt.title(\"Decision Tree Regression\")\n",
"plt.legend()\n",
"plt.show()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 本节结束"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
}
},
"nbformat": 4,
"nbformat_minor": 4
}