{ "cells": [ { "cell_type": "markdown", "id": "195068e6", "metadata": {}, "source": [ "# 机器学习与社会科学应用" ] }, { "cell_type": "markdown", "id": "3613afae", "metadata": {}, "source": [ "# 第六章 无监督学习算法" ] }, { "cell_type": "markdown", "id": "ce45c17d", "metadata": {}, "source": [ "# 第四节 LDA主题模型" ] }, { "cell_type": "markdown", "id": "831c2908", "metadata": {}, "source": [ "郭峰 \n", " 教授、博士生导师 \n", "上海财经大学公共经济与管理学院 \n", "上海财经大学数实融合与智能治理实验室 \n", " 邮箱：guofengsfi@163.com " ] }, { "cell_type": "markdown", "id": "fa6fb483", "metadata": {}, "source": [ "本节目录 \n", "4.1.数据预处理 \n", "4.2.训练LDA模型 \n", "4.3.确定最优主题数 \n", "4.4.最优主题数的主题图 \n", "4.5.绘制主题词云图 \n", "4.6.两篇文章的主题相似度 " ] }, { "cell_type": "markdown", "id": "fe12d5b2", "metadata": {}, "source": [ "## 4.1. 数据预处理" ] }, { "cell_type": "code", "execution_count": null, "id": "36ad683b", "metadata": {}, "outputs": [], "source": [ "import pandas as pd \n", "import numpy as np\n", "import re \n", "import os \n", "import time\n", "from tqdm import tqdm\n", "# import warnings\n", "# warnings.filterwarnings(\"ignore\")\n", "\n", "os.chdir('D:/python/机器学习与社会科学应用/演示数据/06无监督学习/lda主题模型/')\n", "df = pd.read_csv('cssci_clean_test.csv', encoding='utf8')\n", "print(\"原始样本量：\",df.shape)\n", "\n", "# 保存需要的列，并更改列名\n", "df = df[['title', 'keyword', 'abstract']]\n", "\n", "\n", "# 将标题、摘要、内容合并\n", "df['keyword'] = df['keyword'].fillna(\";\")\n", "df['content'] = df['title'] + ';' + df['keyword'] + df['abstract']\n", "df = df[df['content'].str.len() > 100]\n", "print(\"标题+内容大于100字的数量：\", len(df))\n", "\n", "# 剔除标题和内容为空的行\n", "df = df.dropna()\n", "# 剔除重复行\n", "df = df.drop_duplicates()\n", "# 剔除内容字数少于100的行\n", "df = df[df['content'].str.len() > 100]\n", "print(\"剔除Title为空的行以及重复的行后的数量：\", len(df))\n", "\n", "df.to_csv('cssci_lda.csv', encoding='utf8', index=False)" ] }, { "cell_type": "markdown", "id": "33abcc51", "metadata": {}, "source": [ "## 4.2. 训练LDA模型" ] }, { "cell_type": "code", "execution_count": null, "id": "5e36828d", "metadata": {}, "outputs": [], "source": [ "import pandas as pd \n", "import numpy as np \n", "import jieba\n", "import jieba.analyse\n", "import re\n", "import gensim \n", "import pyLDAvis\n", "import pyLDAvis.gensim_models\n", "import os \n", "import time\n", "from collections import defaultdict\n", "from tqdm import tqdm\n", "from cntext.dictionary import STOPWORDS_zh\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "os.chdir('D:/python/机器学习与社会科学应用/演示数据/06无监督学习/lda主题模型/')\n", "\n", "# 分词\n", "def cut_words(string):\n", " string = re.sub(r\"[0-9a-z\\s+]+\", \"\",string) \n", " # 加载自定义词典\n", " jieba.load_userdict(\"keywords.txt\")\n", " # 删除单个字符的词\n", " cuts = [w for w in jieba.cut(string) if len(w)>1]\n", " # 删除停用词\n", " cuts = [w for w in cuts if w not in STOPWORDS_zh] \n", " return cuts\n", "\n", "# 创建词袋向量（bag of words）\n", "def create_bow(texts_cut):\n", "# frequency = defaultdict(int)\n", "# for text in texts_cut:\n", "# for token in text:\n", "# frequency[token] += 1\n", " # texts_cut = [[token for token in text if frequency[token] > 0] for text in texts_cut] #只保留词频大于1的\n", " dictionary = gensim.corpora.Dictionary(texts_cut)\n", " # print(len(dictionary))\n", " # print(dictionary.token2id) \n", " # 过滤，每个词汇至少在30篇文档中存在，每个词至多存在于99%的文档，保留最高词频30000个词\n", " dictionary.filter_extremes(no_below=30, no_above=0.9, keep_tokens=None)\n", " # print(len(dictionary))\n", " # 删除词频最高的两个词\n", " dictionary.filter_n_most_frequent(2)\n", " corpus = [dictionary.doc2bow(text) for text in text_cut]\n", " return dictionary, corpus\n", "\n", "# 训练并保存lda模型\n", "def training_ldamodel(topic_num):\n", " lda = gensim.models.ldamodel.LdaModel(corpus, id2word=dictionary, num_topics=topic_num, passes=10)\n", " if not os.path.exists(\"lda_model/\" + str(topic_num)):\n", " os.makedirs(\"lda_model/\" + str(topic_num))\n", " lda.save('lda_model/'+ str(topic_num) + '/' + 'lda_model.model') \n", "\n", " \n", "if __name__ == '__main__':\n", " starttime = time.time()\n", " # *****************【此部分需要修改】\n", " # 读取数据\n", " df = pd.read_csv('cssci_lda.csv')\n", " print(\"语料库样本Size:\", df.shape)\n", " df = df.sample(5000) # 演示时为了减少时间，减少数据规模\n", " # 文档所在的列名\n", " content = 'content' \n", " # 设定主题数目区间\n", " topics_range = range(5, 15) # 【根据需要设定主题数目的范围】\n", " \n", " # 分词\n", " tqdm.pandas()\n", " df['text_cut'] = df[content].progress_apply(cut_words)\n", " # 如果数据量大可以保存分词结果\n", " df.to_csv('cssci_lda_cut.csv', encoding='utf8', index=False)\n", " # 创建词袋向量\n", " text_cut = df['text_cut'].tolist() # 将列转化为数组形式\n", " dictionary, corpus = create_bow(text_cut)\n", " \n", " # 训练LDA模型 \n", " for i in tqdm(topics_range):\n", " training_ldamodel(i)\n", " # 保存corpus,由于dictionary会随着LDA model的保存而一并保存，corpus需要单独保存\n", " gensim.corpora.MmCorpus.serialize('lda_model/corpus.mm', corpus)\n", "\n", " endtime = time.time()\n", " print(\"总耗时：\", (endtime - starttime))" ] }, { "cell_type": "markdown", "id": "95bbce43", "metadata": {}, "source": [ "## 4.3. 确定最优主题数" ] }, { "cell_type": "markdown", "id": "7d5fbfef", "metadata": {}, "source": [ "评价模型： \n", "- 通常用于评价聚类算法好坏的方法有两种，其一是使用带分类标签的测试数据集，然后使用一些算法来判断聚类结果与真实结果的差距；其二是使用无分类标签的测试数据集，通过一些指标进行衡量，通常衡量LDA主题的指标有两个：困惑度与一致性，其中一致性指标更好。\n", "- 困惑度指标可以通俗的理解为某个文档属于某个主题的概率大小，如果概率越小，表示困惑越大，即越不可能属于这个主题。因此困惑的越小越好。\n", "- 一致性指标可以通俗的理解为每个主题内的主题词的相似性。例如，一个主题都是关于农业的词汇，包括农村、农民、乡村、化肥等词，显然一致性比较稿；但是如果这个关于农业的主题还包括股票、货币、资本等词汇，显然这个主题的一致性就比较低。\n", "- 因此，对于LDA主题模型评价要求困惑度越低越好，一致性越高越高。在实践中我们通常画出LDA的困惑度和一致性指标的折线图进行评估，有时还需要结合具体的主题的主题关键词进行判断是否最优。" ] }, { "cell_type": "markdown", "id": "8a386780", "metadata": {}, "source": [ "### 困惑度\n", "\n", "- log_perplexity()这个函数没有对主题数目做归一化，因此不同的topic数目不能直接比较.\n", "- gensim包的作者Radim现身回答说perplexity不是一个好的评价topic质量的指标\n", "- 参考：https://blog.csdn.net/qq_23926575/article/details/79472742\n", "- 困惑度的公式\n", "$$\n", "perplexity(D)=exp{\\frac{-\\sum^M_{d=1}log{p(w_d)}}{\\sum^M_{d=1}N_d}}\n", "$$\n", "其中，M是测试语料库的大小，$N_d$是第d篇文本的大小（即单词个数）。\n", "$$\n", "p(w_d)=\\sum_z p(z)p(w|z, gamma)\n", "$$\n", "其中z是主题，w是文档，gamma是训练集学出来的文本-主题分布。\n", "\n", "所以perlexity的上半部分就是生成整个文档的似然估计（表示训练集训练出的参数的生成能力）的负值，由于概率值取值范围为[0, 1]，按照对数函数的定义，分子值是一个大数，而分母是整个测试集的单词数目。也就是说模型生成能力越强，perplexity值越小。" ] }, { "cell_type": "markdown", "id": "ebdd97e8", "metadata": {}, "source": [ "### 一致性 \n", "\n", "- 更好的内在评估方法\n", "- 衡量生成的主题的连贯性\n", "- 一个好的主题模型能够生成更连贯的主题\n", "\n", "单词入侵 \n", "- 想法：为主题注入一个随机词，比如农业主题{farmers, farm, food, rice, agricluture}$\\rightarrow${farmers, farm, food, rice, cat, agriculture}\n", "- 让用户猜测哪个是入侵词\n", "- 猜对$\\rightarrow$ 话题连贯\n", "\n", "PMI$\\approx$一致性\n", "- 一对词的PMI高$\\rightarrow$词是相关的；PMI(farm, rice)较高，PMI(farmers, cat)较低；\n", "- 如果主题中的所有词对都具有高PMI$\\rightarrow$主题是连贯的\n", "- 如果大多数主题具有高PMI$\\rightarrow$良好的主题模型\n", "- 哪里可以获得PMI的词共现统计数据\n", " - 主题模型可以使用相同的语料库\n", " - 更好的方法是使用外部语料库（例如维基百科）\n", " \n", "PMI\n", "- 计算主题中前N个词的称对PMI\n", "$$\n", "PMI(t) = \\sum^N_{j=2}\\sum^{j-1}_{i=1}log{\\frac{P(w_i, w_j)}{P(w_i)P(w_j)}}\n", "$$\n", "- 给定主题：{farmers, farm, food, rice, agriculture}\n", "- 连贯性 = 所有词对的总和PMI：\n", " - PMI(farmers, farm)+PMI(farmers, food)+...+PMI(rice, agriculture)\n", "- 标准化PMI\n", "$$\n", "NPMI(t) = \\sum^N_{j=2}\\sum^{j-1}_{i=1} \\frac{log{\\frac{P(w_i, w_j)}{P(w_i)P(w_j)}}}{-logP(w_i,w_j)}\n", "$$" ] }, { "cell_type": "code", "execution_count": null, "id": "ab0abd55", "metadata": { "scrolled": true, "tags": [] }, "outputs": [], "source": [ "# 困惑度或一致性指标与主题个数折线图\n", "import matplotlib.pyplot as plt\n", "import matplotlib\n", "import pandas as pd \n", "import numpy as np\n", "import time\n", "import gensim\n", "from gensim.models.coherencemodel import CoherenceModel\n", "import ast\n", "from tqdm import tqdm\n", "import os \n", "\n", "os.chdir('D:/python/机器学习与社会科学应用/演示数据/06无监督学习/lda主题模型/')\n", "\n", "\n", "#计算困惑度\n", "def compute_perplexity(topic_num):\n", " ldamodel = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model')\n", " # corpus = gensim.corpora.MmCorpus('lda_model/corpus.mm')\n", " return ldamodel.log_perplexity(corpus)\n", "\n", "\n", "#计算一致性指标\n", "def compute_coherence(topic_num):\n", " ldamodel = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model')\n", " dictionary = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model.id2word')\n", " # corpus = gensim.corpora.MmCorpus('lda_model/corpus.mm')\n", " ldacm = CoherenceModel(model=ldamodel, texts=text_cut, dictionary=dictionary, coherence='c_v')\n", " return ldacm.get_coherence()\n", "\n", "if __name__ == '__main__':\n", " starttime = time.time()\n", " df = pd.read_csv('cssci_lda_cut.csv') # 【需要修改】\n", " df['text_cut'] = df['text_cut'].map(ast.literal_eval)\n", " text_cut = df['text_cut'].tolist() # 将列转化为数组形式\n", " corpus = gensim.corpora.MmCorpus('lda_model/corpus.mm')\n", " \n", " x = range(5, 15) # 【与前文的topics_range一致】\n", " # 困惑度\n", " y = [compute_perplexity(i) for i in tqdm(x)]\n", " # 一致性\n", " z = [compute_coherence(i) for i in tqdm(x)] \n", " print(\"perplexity and coherence are: \", y, z)\n", "\n", " plt.subplot(2, 1, 1)\n", " plt.plot(x, y, 'ko-')\n", " plt.xlabel('topics_num')\n", " plt.ylabel('perplexity size')\n", " plt.rcParams['font.sans-serif']=['SimHei']\n", " matplotlib.rcParams['axes.unicode_minus']=False\n", " plt.title('topics-perplexity')\n", "\n", " plt.subplot(2, 1, 2)\n", " plt.plot(x, z, 'r.-')\n", " plt.xlabel('topics_num')\n", " plt.ylabel('coherence size')\n", " plt.title('topics-coherence')\n", " plt.show()\n", " endtime = time.time()\n", " print(\"总耗时：\", (endtime - starttime))\n", " \n", "# 从下面的困惑度与一致性折线图可以看出，困惑度的折线图从5-11缓慢上升，主题数为12时有所下降，此后开始大幅上升\n", "# 因此根据困惑图，我们将主题数定在12及以下比较合适\n", "# 再看一致性图，在主题数为7-9时，一致性指标达到较高的之，此后大幅下降，并从主题数12时后又大幅上升\n", "# 通过困惑度图和一致性图，我们初步将主题数定为7-9，后面我们将根据具体的主题词分布正式确定主题数。" ] }, { "cell_type": "markdown", "id": "3223e588", "metadata": {}, "source": [ "## 4.4. 最优主题数的主题图" ] }, { "cell_type": "markdown", "id": "a1ecb4b6", "metadata": {}, "source": [ "上文中，我们初步确定主题数为7-9，这里，我们查看每个主题的交叉情况和关键词的分布情况，最终确定主题数。" ] }, { "cell_type": "code", "execution_count": null, "id": "728e9536", "metadata": {}, "outputs": [], "source": [ "import pyLDAvis \n", "from pyLDAvis import gensim_models\n", "import gensim\n", "from gensim import corpora\n", "import os \n", "\n", "os.chdir('D:/python/机器学习与社会科学应用/演示数据/06无监督学习/lda主题模型/')\n", "\n", "# 加载保存的LDA模型\n", "topic_num = 8 # 【依次输入7、8、9，观察每个主题模型的主题分布】\n", "lda_model = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model')\n", "dictionary = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model.id2word')\n", "corpus = corpora.MmCorpus('lda_model/corpus.mm')\n", "\n", "# 输出模型\n", "print(lda_model.print_topics(num_topics=topic_num, num_words=3)) # 输出10个主题，每个主题4个词汇\n", "print(lda_model.print_topic(0, topn=5)) # 输出第0个主题，排名前十的关键词\n", " \n", "# 主题可视化\n", "pyLDAvis.enable_notebook(local=False)\n", "plot = pyLDAvis.gensim_models.prepare(lda_model, corpus, dictionary)\n", "# 保存主题可视化为网页，但是由于网页依赖的第三方插件存在问题，打开前使用TXT打开并将文本中的“cdn”替换为“fastly”，然后关闭再用浏览器打开\n", "# pyLDAvis.save_html(plot, 'lda_model/'+ str(topic_num) + '/'+'LDAvis.html')\n", "# pyLDAvis.display(plot, local=True) # 【有时需要单独运行才有效】" ] }, { "cell_type": "code", "execution_count": null, "id": "8b0d047d", "metadata": {}, "outputs": [], "source": [ "pyLDAvis.display(plot, local=True) # 【有时需要单独运行才有效】\n", "# 图示说明\n", "# 左图：每一个圆圈代表一个主题，圆的大小表示主题再在文档中所占的比例，如第一个主题的圆面积最大，表示该主题在所有文档中所占的比例最大，圆面积随着主题顺序的递增而逐渐减少，表示后面的主题在文档的分布越来越小\n", "# 圆与圆之间的距离表示两者的相似度，距离越近，相似度越高。因此，圆圈越分散，表示主题的分类越清晰，模型越好。\n", "# 右图柱状图，灰色代表总的词频，红色代表主题内的关键词词频\n", "# 相关性lambda可以右上角蓝色条进行调整，含义类似于TF-IDF\n", "# 通过对比，发现主题数为7和8时，分类效果较好，最终的分类还可以根据每个主题的关键词进行判断\n", "# 通过观察，我们发现主题数为8时，主题的关键词的一致性较好，因此最终我们选择主题数为8\n", "# 在实践中，不一定需要确定最优主题，可以根据需要选择与用户需要的主题相似的主题数就可以了。" ] }, { "cell_type": "markdown", "id": "7b4f8f4e", "metadata": {}, "source": [ "## 4.5. 绘制主题词云图" ] }, { "cell_type": "code", "execution_count": null, "id": "a60a0e49", "metadata": {}, "outputs": [], "source": [ "#LDA主题模型的关键词词云\n", "import codecs\n", "import numpy as np\n", "import pandas as pd\n", "import gensim \n", "# from wordcloud import WordCloud, ImageColorGenerator\n", "# from PIL import Image\n", "from pyecharts.charts import WordCloud\n", "from pyecharts import options as opts\n", "import os\n", "\n", "os.chdir('D:/python/机器学习与社会科学应用/演示数据/06无监督学习/lda主题模型/')\n", "\n", "topic_num = 8 #【设定主题数目】\n", "topicn = 2 #【第几个主题】\n", "\n", "lda = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model')\n", "#topic_word = lda.print_topics(num_topics=1, num_words=10)\n", "#print(topic_word)\n", "topic_word = lda.print_topic(topicn,topn=100)\n", "print(topic_word)\n", "topic_word = topic_word.split(' + ')\n", "word_score_list = []\n", "for tw in topic_word:\n", " temp1 = tw.split('*')\n", " #print(temp1)\n", " temp1[1] = eval(temp1[1])\n", " word_score_list.append(temp1)\n", "#print(word_score_list)\n", "print(word_score_list[:][0])\n", "word=[x[1] for x in word_score_list]\n", "count=[x[0] for x in word_score_list]\n", "print(word[0:10])\n", "print(count[0:10])\n", "\n", "data=[]\n", "for i in range(len(word_score_list)):\n", " temp=[(word[i],count[i])]\n", " data=data+temp\n", " \n", "# 创建实例对象\n", "c = WordCloud()\n", "c.add(series_name=\"\",data_pair=data)\n", "# 设置标题\n", "c.set_global_opts(title_opts=opts.TitleOpts(\"主题词云\"))\n", "# 展示图片\n", "c.render_notebook() " ] }, { "cell_type": "markdown", "id": "f0845ea9", "metadata": {}, "source": [ "## 4.6. 计算两篇文档的相似度" ] }, { "cell_type": "code", "execution_count": null, "id": "989141b8", "metadata": {}, "outputs": [], "source": [ "import pandas as pd \n", "import numpy as np \n", "import gensim \n", "import jieba \n", "import scipy.stats\n", "import time\n", "import os \n", "import ast\n", "\n", "os.chdir('D:/python/机器学习与社会科学应用/演示数据/06无监督学习/lda主题模型/')\n", "\n", "def lda_similarity(text1, text2):\n", " # 文档1\n", "# text1 = text1.split()\n", " print(text1)\n", " corpus1 = dictionary.doc2bow(text1)\n", " lda_text1 = lda_model[corpus1]\n", " # 输出新文档的主题分布\n", " print(lda_text1)\n", " list_text1 = [i[1] for i in lda_text1]\n", " print('list_text1: ', list_text1)\n", " \n", " # 文档2\n", "# text2 = text2.split()\n", " print(text2)\n", " corpus2 = dictionary.doc2bow(text2)\n", " lda_text2 = lda_model[corpus2]\n", " print(lda_text2)\n", " list_text2 = [i[1] for i in lda_text2]\n", " print('list_text2: ', list_text2)\n", " \n", " # 相似度计算\n", " try:\n", " # 欧式距离相似度；dot()返回的时两个数组的点积；np.linalg.norm求欧式距离；值越大表示越相似\n", " # sim = np.dot(list_text1, list_text2) / (np.linalg.norm(list_text1) * np.linalg.norm(list_text2))\n", " # KL divergence，也称KL散度，衡量两个概率分布的相似性；散度越小，即距离越小，表示两个概率分布越相似\n", " sim = scipy.stats.entropy(list_text1, list_text2)\n", " except ValueError:\n", " sim = 0\n", " \n", "if __name__ == '__main__':\n", " starttime = time.time()\n", " topic_num = 8 #【设定主题数目】\n", " # 加载模型\n", " lda_model = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model')\n", " dictionary = gensim.models.ldamodel.LdaModel.load('lda_model/'+ str(topic_num) + '/' + 'lda_model.model.id2word')\n", " # 读取数据\n", " df = pd.read_csv('cssci_lda_cut.csv')\n", " df['text_cut'] = df['text_cut'].map(ast.literal_eval)\n", " df = df.sample(200).reset_index()\n", " text1 = df['text_cut'][20] #【第20个文档】\n", " text2 = df['text_cut'][50] #【第50个文档】\n", " sim = lda_similarity(text1, text2)\n", " endtime = time.time()\n", " print(\"总耗时：\",(endtime - starttime))" ] }, { "cell_type": "code", "execution_count": null, "id": "8ecb1478", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 5 }