ML-For-Beginners NLP 系列实战NLTK 深入学习作业与 515K 欧洲酒店评论的数据探索【免费下载链接】ML-For-Beginners12 weeks, 26 lessons, 52 quizzes, classic Machine Learning for all项目地址: https://gitcode.com/GitHub_Trending/ml/ML-For-Beginners本篇文章围绕 ML-For-Beginners 课程 NLP 章节第 4 课酒店评论情感分析·上篇的课后作业展开以自然语言处理工具库NLTK的深入学习为任务主线完整复现课程讲义translations/en/6-NLP/4-Hotel-Reviews-1/README.md中的数据准备与探索性数据分析EDA全流程。读完本文你将掌握 515,000 条酒店评论数据集17 个字段的字段语义、pandas 数据框实操技巧频率统计、分组聚合、apply逐行计算、性能对比并为下一课使用 NLTK 执行停用词移除与 VADER 情感分析打下坚实基础。作业背景一份以 NLTK 为核心的深入学习任务本节课程的课后作业assignment.md将任务目标指向 NLTK 本身NLTK 是什么NLTKNatural Language Toolkit是计算语言学与 NLP 领域广为人知的 Python 库提供语料库访问、词性标注、分词、停用词、情感分析等一整套文本处理能力。作业内容通读《Natural Language Processing with Python》即 NLTK 官方在线书籍《NLTK Book》并动手完成其中的练习借此机会更深入地认识这个库。作业性质这是一份**不计分ungraded**的探索型作业学习价值大于考核意义——它要求你在完成本课数据探索之后进一步理解后续情感分析环节所依赖的工具。该作业与本课讲义直接衔接讲义末尾的 Assignment 一节即链接至本作业。本课的核心目标是在大量真实文本上做探索性分析——这正是 NLTK 及其情感分析能力将要大显身手的舞台。为什么先做数据探索酒店评论情感分析项目概览本课的实战背景是一个真实任务基于情感分析与住客评分构建酒店推荐机器人。数据集为515K Hotel Reviews Data in Europe由 Jiashen Liu 通过 Booking.com 公开信息爬取采用 CC0 公共领域许可解压后约 230 MB覆盖欧洲 6 座城市的 1493 家酒店。按课程要求数据集需下载到 NLP 课程的 data 目录6-NLP/data/README.md 即为该目录占位说明。围绕该数据集课程提出了三个值得用 Python NLTK 回答的问题评论中最常使用的单词与短语是什么酒店官方Tags是否与评分相关例如带幼童的家庭类住客是否比独自旅行者更常给出差评从而暗示酒店更适配后者NLTK 的情感得分是否与住客给出的数值评分一致动手前请确认环境能够运行 Python 3 的.ipynb笔记本、pandas、本地安装的 NLTK以及已下载到 data 目录的数据集。本节课程的探索代码全部收录在配套笔记本 solution/notebook.ipynb 中可随时对照。数据集字段全解析17 列到底在说什么课程要求先用编辑器VS Code / Excel 等打开 CSV 目视检查一遍数据字段表头如下Hotel_Address, Additional_Number_of_Scoring, Review_Date, Average_Score, Hotel_Name, Reviewer_Nationality, Negative_Review, Review_Total_Negative_Word_Counts, Total_Number_of_Reviews, Positive_Review, Review_Total_Positive_Word_Counts, Total_Number_of_Reviews_Reviewer_Has_Given, Reviewer_Score, Tags, days_since_review, lat, lng按业务语义可将其分为四组理解每一组的含义是后续 EDA 的前提。酒店列Hotel_Name、Hotel_Address、lat纬度、lng经度借助lat/lng可用 Python 在地图上标出酒店位置甚至按正面/负面评论做颜色区分。Hotel_Address对建模价值不大更适合替换为国家/城市字段以方便排序与检索。酒店元评论列Average_Score按数据集创建者的说明该列是基于最近一年最新评论计算的酒店平均分。这种口径颇为特殊课程建议暂且按字面接受——同时留下一个思考题能否用数据集中的其他列自行计算另一种平均分后文的 EDA 会给出答案。Total_Number_of_Reviews酒店收到的评论总数。不写代码无法确认它是否等同于数据集内该酒店的评论条数后续探索会证明两者不一致。Additional_Number_of_Scoring表示只打了分、但未撰写正面或负面评论文本的评分条数。评论列Reviewer_Score数值评分最多一位小数取值范围 2.510。为何下限是 2.5 而非 0数据集没有给出解释。Negative_Review若未写差评字段值为No Negative偶尔会有住客在差评栏写下正面内容如 there is nothing bad about this hotel。Review_Total_Negative_Word_Counts差评单词数。一般而言差评词数越多评分越低即便不做情感分析也能观察到此趋势。Positive_Review若未写好评字段值为No Positive同样存在反转情况如 there is nothing good about this hotel at all。Review_Total_Positive_Word_Counts好评单词数通常与评分正相关。Review_Date与days_since_review可用于衡量评论的新鲜度/陈旧度——例如酒店换管理团队、翻新或新增泳池后旧评论的参考价值会下降。Tags住客自行勾选的短标签用于描述出行类型独自/家庭等、房型、入住时长与提交方式。该列问题较多详见后文专门讨论。评论者列Total_Number_of_Reviews_Reviewer_Has_Given评论者累计发表的评论数。理论上可作为推荐模型的特征例如高产评论者可能更挑剔但数据集中评论者没有唯一编码、无法跨评论关联虽有 30 位评论者发表过 100 条评论但难以借此为推荐模型增益。Reviewer_Nationality评论者国籍。课程特别提醒不要基于某国国民更倾向于打高分/低分这类文化成见建模——这是刻板印象每位评论者的评分都经由其既往住宿经历、旅行距离、性格等因素过滤把分数差异简单归因于国籍缺乏依据。一条真实的极端样本Average ScoreTotal Number ReviewsReviewer ScoreNegative ReviewPositive ReviewTags7.819452.5This is currently not a hotel but a construction site...约 115 词描述施工噪音、拒绝换房、多收费等经历Nothing. Terrible place. Stay away.Business trip, Couple, Standard Double Room, Stayed 2 nights这位住客体验极差酒店平均分高达 7.8、累计 1945 条评论但他只给 2.5 分并在差评栏写了 115 词好评栏仅 7 个词告诫他人避开。如果只统计词数而不做情感分析就会误判其真实意图——这正是本课后续引入 NLTK 情感分析的动机。同时最低分 2.5 而非 0再次暴露评分体系口径的疑问。Tags 列看似好用实则难以标准化Tags乍看非常适合做数据分类但它是非标准化的自由文本A 酒店可能用Single room、Twin room、Double roomB 酒店却用Deluxe Single Room、Classic Queen Room、Executive King Room——指代可能相同的房型写法却千差万别。课程给出的两条处理思路尝试统一全部术语Classic single room尚可映射到Single room但Superior Queen Room with Courtyard Garden or City View这类标签几乎无法可靠映射成本极高。改用 NLP 手段直接统计Solo、Business Traveller、Family with young kids等关键短语的出现频率并将其纳入推荐。Tags通常包含 56 个逗号分隔的值涵盖出行类型、住客类型、房型、入住晚数、提交设备但由于部分评论者留空各值并不总是保持固定顺序。例如仅过滤Family with就能命中 80,000 条含 Family with young children 或 Family with older children 的记录——说明该列有潜力但需要付出额外处理成本下一课会给出完整的标签清洗方案。平均分疑云数据集自带字段并不总是自洽讲义对与平均分相关的 5 个字段Hotel_Name、Additional_Number_of_Scoring、Average_Score、Total_Number_of_Reviews、Reviewer_Score做了交叉验证发现了明显矛盾数据集内评论最多的酒店是Britannia International Hotel Canary Wharf数据集内 4789 条评论但Total_Number_of_Reviews却是 9086即便加上Additional_Number_of_Scoring26824789 2682 7471仍比 9086 少 1615。Kaggle 上对Average_Score的说明是基于最近一年最新评论计算的酒店平均分该口径似乎没什么实际帮助。用数据集内评分自行计算后该酒店的平均分为 6.8而字段值是 7.1差异可能来自Additional_Number_of_Scoring的纯评分评论但无法证实。评论数第二多的酒店其计算平均分为 8.12字段值为 8.1——是巧合还是口径差异基于这些不一致课程决定编写小程序亲自探索数据判断这些字段能用还是不能用。 注意事项 处理文本数据时你的代码将基于文本自动计算无需也不应逐条人工阅读。但数据集来自公开网站少数差评可能包含种族主义、性别歧视或年龄歧视等冒犯性内容有的评论琐碎且无关如抱怨天气有的则令人不适。建议让代码去评估情感避免把自己暴露在潜在的不良内容中——尽管这类评论只占少数但确实存在。动手实践第一步加载数据课程强调写代码之前先充分理解数据及其怪癖尤其是文本数据更需要仔细审视。从加载开始# Load the hotel reviews from CSV import pandas as pd import time # importing time so the start and end time can be used to calculate file loading time print(Loading data file now, this could take a while depending on file size) start time.time() # df is DataFrame - make sure you downloaded the file to the data folder df pd.read_csv(../../data/Hotel_Reviews.csv) end time.time() print(Loading took str(round(end - start, 2)) seconds)该 CSV 包含超过 50 万行、仅 17 列。pandas 的快速 CSV 加载器将其读入 DataFrame而 DataFrame 提供了强大的行级操作能力。请把这段加载代码保留在程序顶部后续所有探索都基于它。本例数据本身已干净不包含会干扰仅识别英文算法处理的多语言字符但现实中你很可能需要先做格式预处理——可以思考一下遇到非英文字符时你会如何处理探索性数据分析九个验证任务与完整代码答案本课的核心练习是在不修改数据框的前提下通过写代码核验数据集中的数值是否与 pandas 计算结果一致。课程给出的通用原则是编程任务往往有多种解法选择最简单、最容易理解尤其方便日后回看的方案即可——DataFrame 的完整 API 通常已经提供了高效途径。请将下列问题当作编程任务先尝试不看答案自行求解再对照下文打印数据框的shape行数与列数。计算评论者国籍的频率统计Reviewer_Nationality列有多少个不同取值分别是哪些数据集中最常见的评论者国籍是哪个打印国家与评论数出现频率最高的前 10 个国籍及其频次是什么前 10 大评论者国籍各自评论最多的酒店是哪家数据集中每家酒店的评论数酒店频率统计是多少数据集虽自带Average_Score但也可以自行计算平均分对每家酒店求数据集内全部评论者评分的均值。新增一列Calc_Average_Score存放计算结果。是否存在Average_Score与Calc_Average_Score均保留 1 位小数相同的酒店尝试编写一个接收 Series行作为参数、比较两值并在不等时打印消息的 Python 函数再用.apply()方法逐行调用它。计算并打印Negative_Review值为 No Negative 的行数。计算并打印Positive_Review值为 No Positive 的行数。计算并打印Positive_Review为 No Positive且Negative_Review为 No Negative 的行数。任务 1数据框形状print(The shape of the data (rows, cols) is str(df.shape)) The shape of the data (rows, cols) is (515738, 17)数据共515,738 行 × 17 列。这正是配套笔记本 solution/notebook.ipynb 中 EDA 代码的第一步。任务 2评论者国籍频率# value_counts() creates a Series object that has index and values # in this case, the country and the frequency they occur in reviewer nationality nationality_freq df[Reviewer_Nationality].value_counts() print(There are str(nationality_freq.size) different nationalities) # print first and last rows of the Series. Change to nationality_freq.to_string() to print all of the data print(nationality_freq) There are 227 different nationalities United Kingdom 245246 United States of America 35437 Australia 21686 Ireland 14827 United Arab Emirates 10235 ... Comoros 1 Palau 1 Northern Mariana Islands 1 Cape Verde 1 Guinea 1 Name: Reviewer_Nationality, Length: 227, dtype: int64数据集包含227 个不同国籍其中英国以 245,246 条评论遥遥领先。print(The highest frequency reviewer nationality is str(nationality_freq.index[0]).strip() with str(nationality_freq[0]) reviews.) # Notice there is a leading space on the values, strip() removes that for printing # What is the top 10 most common nationalities and their frequencies? print(The next 10 highest frequency reviewer nationalities are:) print(nationality_freq[1:11].to_string()) The highest frequency reviewer nationality is United Kingdom with 245246 reviews. The next 10 highest frequency reviewer nationalities are: United States of America 35437 Australia 21686 Ireland 14827 United Arab Emirates 10235 Saudi Arabia 8951 Netherlands 8772 Switzerland 8678 Germany 7941 Canada 7894 France 7296细节国籍值带有前导空格打印时需要.strip()处理前 10 名中英国独占近半。任务 3各国籍最常评论的酒店# What was the most frequently reviewed hotel for the top 10 nationalities # Normally with pandas you will avoid an explicit loop, but wanted to show creating a new dataframe using criteria # (dont do this with large amounts of data because it could be very slow) for nat in nationality_freq[:10].index: # First, extract all the rows that match the criteria into a new dataframe nat_df df[df[Reviewer_Nationality] nat] # Now get the hotel freq freq nat_df[Hotel_Name].value_counts() print(The most reviewed hotel for str(nat).strip() was str(freq.index[0]) with str(freq[0]) reviews.) The most reviewed hotel for United Kingdom was Britannia International Hotel Canary Wharf with 3833 reviews. The most reviewed hotel for United States of America was Hotel Esther a with 423 reviews. The most reviewed hotel for Australia was Park Plaza Westminster Bridge London with 167 reviews. The most reviewed hotel for Ireland was Copthorne Tara Hotel London Kensington with 239 reviews. The most reviewed hotel for United Arab Emirates was Millennium Hotel London Knightsbridge with 129 reviews. The most reviewed hotel for Saudi Arabia was The Cumberland A Guoman Hotel with 142 reviews. The most reviewed hotel for Netherlands was Jaz Amsterdam with 97 reviews. The most reviewed hotel for Switzerland was Hotel Da Vinci with 97 reviews. The most reviewed hotel for Germany was Hotel Da Vinci with 86 reviews. The most reviewed hotel for Canada was St James Court A Taj Hotel London with 61 reviews.注意这里用显式for循环只是为了演示按条件构造新数据框的思路处理海量数据时循环通常较慢应优先使用 pandas 的向量化操作。任务 4每家酒店的评论数# First create a new dataframe based on the old one, removing the uneeded columns hotel_freq_df df.drop([Hotel_Address, Additional_Number_of_Scoring, Review_Date, Average_Score, Reviewer_Nationality, Negative_Review, Review_Total_Negative_Word_Counts, Positive_Review, Review_Total_Positive_Word_Counts, Total_Number_of_Reviews_Reviewer_Has_Given, Reviewer_Score, Tags, days_since_review, lat, lng], axis 1) # Group the rows by Hotel_Name, count them and put the result in a new column Total_Reviews_Found hotel_freq_df[Total_Reviews_Found] hotel_freq_df.groupby(Hotel_Name).transform(count) # Get rid of all the duplicated rows hotel_freq_df hotel_freq_df.drop_duplicates(subset [Hotel_Name]) display(hotel_freq_df)Hotel_NameTotal_Number_of_ReviewsTotal_Reviews_FoundBritannia International Hotel Canary Wharf90864789Park Plaza Westminster Bridge London121584169Copthorne Tara Hotel London Kensington71053578.........Mercure Paris Porte d Orleans11010Hotel Wagner13510Hotel Gallitzinberg1738关键发现数据集内实际统计到的评论数Total_Reviews_Found与字段Total_Number_of_Reviews并不一致——后者可能指酒店实际收到的总评论数但未全部被抓取也可能是其他口径。由于无法确认Total_Number_of_Reviews不会被用于后续建模。任务 5自行计算平均分并对比# define a function that takes a row and performs some calculation with it def get_difference_review_avg(row): return row[Average_Score] - row[Calc_Average_Score] # mean is mathematical word for average df[Calc_Average_Score] round(df.groupby(Hotel_Name).Reviewer_Score.transform(mean), 1) # Add a new column with the difference between the two average scores df[Average_Score_Difference] df.apply(get_difference_review_avg, axis 1) # Create a df without all the duplicates of Hotel_Name (so only 1 row per hotel) review_scores_df df.drop_duplicates(subset [Hotel_Name]) # Sort the dataframe to find the lowest and highest average score difference review_scores_df review_scores_df.sort_values(by[Average_Score_Difference]) display(review_scores_df[[Average_Score_Difference, Average_Score, Calc_Average_Score, Hotel_Name]])差异最大的酒店如下负值表示计算平均分更高正值表示字段值更高Average_Score_DifferenceAverage_ScoreCalc_Average_ScoreHotel_Name-0.87.78.5Best Western Hotel Astoria-0.78.89.5Hotel Stendhal Place Vend me Paris MGallery-0.77.58.2Mercure Paris Porte d Orleans-0.77.98.6Renaissance Paris Vendome Hotel-0.57.07.5Hotel Royal Elys es............0.77.56.8Mercure Paris Op ra Faubourg Montmartre0.87.16.3Holiday Inn Paris Montparnasse Pasteur0.96.85.9Villa Eugenie0.98.67.7MARQUIS Faubourg St Honor Relais Ch teaux1.37.25.9Kube Hotel Ice Bar由于无法确知字段值为何与计算值部分一致、部分不同课程建议用数据集内真实评分自行计算平均值更稳妥。而表中显示差异通常很小仅一家酒店差异超过 1 分因此实践中忽略差异、直接采用计算平均分是安全的。任务 69统计空评论文本lambda 版本# with lambdas: start time.time() no_negative_reviews df.apply(lambda x: True if x[Negative_Review] No Negative else False , axis1) print(Number of No Negative reviews: str(len(no_negative_reviews[no_negative_reviews True].index))) no_positive_reviews df.apply(lambda x: True if x[Positive_Review] No Positive else False , axis1) print(Number of No Positive reviews: str(len(no_positive_reviews[no_positive_reviews True].index))) both_no_reviews df.apply(lambda x: True if x[Negative_Review] No Negative and x[Positive_Review] No Positive else False , axis1) print(Number of both No Negative and No Positive reviews: str(len(both_no_reviews[both_no_reviews True].index))) end time.time() print(Lambdas took str(round(end - start, 2)) seconds) Number of No Negative reviews: 127890 Number of No Positive reviews: 35946 Number of both No Negative and No Positive reviews: 127 Lambdas took 9.64 seconds另一种方式用sum代替 lambda# without lambdas (using a mixture of notations to show you can use both) start time.time() no_negative_reviews sum(df.Negative_Review No Negative) print(Number of No Negative reviews: str(no_negative_reviews)) no_positive_reviews sum(df[Positive_Review] No Positive) print(Number of No Positive reviews: str(no_positive_reviews)) both_no_reviews sum((df.Negative_Review No Negative) (df.Positive_Review No Positive)) print(Number of both No Negative and No Positive reviews: str(both_no_reviews)) end time.time() print(Sum took str(round(end - start, 2)) seconds) Number of No Negative reviews: 127890 Number of No Positive reviews: 35946 Number of both No Negative and No Positive reviews: 127 Sum took 0.19 seconds两种写法结果一致但性能差异显著课程文档记录lambda 版约 9.64 秒sum版约 0.19 秒——这正是选择最简单高效方法的实证。结论有 127 行占 515,738 行的 0.02%同时缺少正面与负面评论文本说明评论者只打了分但拒绝写评语比例极小不太可能显著歪曲模型结果但这类评论数据集里没有评论的异常正是探索数据才能发现的陷阱。从数据探索走向 NLTK下一课的过滤与情感分析延伸完成本课 EDA 后下一课6-NLP/5-Hotel-Reviews-2/README.md将真正进入 NLTK 的实战先清洗数据再执行情感分析。这部分可以视为本课 NLTK 作业的落地版练习地址精简用replace_address函数把Hotel_Address映射为 6 个城市, 国家组合Amsterdam, Netherlands / Barcelona, Spain / London, United Kingdom / Milan, Italy / Paris, France / Vienna, Austria并可用groupby(Hotel_Address).agg({Hotel_Name: nunique})按国家聚合酒店数伦敦 400 家、巴黎 458 家等。标签处理先str.strip([])去掉方括号、str.replace( , , ,, regexFalse)去掉引号再把乱序的多值标签拆入临时列合并后value_counts()可得到 2428 个唯一标签最终只保留 8 类有用标签Leisure trip、Couple、Solo traveler、Business trip、Group、Family with young children、Family with older children、With a pet并逐列以 0/1 编码。停用词移除NLTK 的stopwords.words(english)可先行剔除不携带情感的常用词从而加速后续分析课程文档记录最长差评 395 词去停用词后剩 195 词515,000 行两列去停用词约耗时 3.3 秒具体因设备而异。VADER 情感分析nltk.sentiment.vader.SentimentIntensityAnalyzer对每条评论输出compound得分No Negative/No Positive直接返回 0保存为Negative_Sentiment与Positive_Sentiment两列。课程提醒个别情感得分会完全错误——例如极度讽刺的评论我当然爱死了一个没有暖气的房间会被误判为正面的人类一读便知的讽刺是规则型分析器的天然盲区依据 Hutto Gilbert, 2014 的 VADER 论文。至此完整的数据流水线共 4 步原始Hotel_Reviews.csv→本课探索→Hotel_Reviews_Filtered.csv过滤笔记本→Hotel_Reviews_NLP.csv情感分析笔记本→ 供 NLP 挑战赛使用。作业完成后的延伸挑战与自修建议本课作业在NLTK 深入探索之外还提供了一个开放挑战像本课一样找几个文本密集型数据集尝试找出其中可能向模型引入偏差或倾斜情感的区域——文本数据的怪癖如空评论文本、非标准化标签、口径不明的评分字段往往只有通过严谨的探索性分析才能暴露。这与本课反复强调的核心理念一致对数据动手之前先彻底理解它的数据与怪癖而 NLTK 这类工具库正是帮助你在海量文本中让代码替人解读含义的关键基础设施。完成本课与作业后可以沿着两条线索继续自修一是把 NLTK 官方书籍的练习逐章做完尤其是语料库、分词与情感分析章节二是参考课程推荐的 NLP 学习路径尝试构建语音与文本密集型模型时可用的更多工具。准备好之后就带着探索结论进入下一课用 NLTK 真正跑通评论 → 情感得分的完整链路。【免费下载链接】ML-For-Beginners12 weeks, 26 lessons, 52 quizzes, classic Machine Learning for all项目地址: https://gitcode.com/GitHub_Trending/ml/ML-For-Beginners创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考