用 Gensim 打造完整 NLP 流水線：從文本預處理到語義搜索的系統實踐

Halo咯咯

發布于 2025-11-10 08:41

瀏覽

0收藏

在自然語言處理（NLP）領域，很多人會問：如何從零開始，構建一個真正可用的 NLP 流水線？

常見的教程往往只聚焦于某一個環節，比如“訓練一個 Word2Vec 模型”或者“跑一次 LDA 主題建模”。但在真實項目中，往往需要一個系統性的流程：從原始文本 → 預處理 → 特征建模 → 相似度分析 → 語義搜索 → 可視化。

今天分享的就是這樣一個完整的 端到端 NLP 流水線，基于 Gensim 構建，并且包含：

文本預處理與語料構建
Word2Vec 詞向量建模與相似度分析
LDA 主題建模與主題可視化
TF-IDF 文檔相似度建模
語義搜索與文檔分類
模型評估（Coherence Score）

本文將完整保留所有代碼，并配上逐段講解，方便你直接運行或復用到自己的項目中。

1. 環境準備與依賴安裝

在 Google Colab 或本地 Python 環境中運行，先安裝依賴。

!pip install --upgrade scipy==1.11.4
!pip install gensim==4.3.2 nltk wordcloud matplotlib seaborn pandas numpy scikit-learn
!pip install --upgrade setuptools


print("Please restart runtime after installation!")
print("Go to Runtime > Restart runtime, then run the next cell")


import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from wordcloud import WordCloud
import warnings
warnings.filterwarnings('ignore')


from gensim import corpora, models, similarities
from gensim.models import Word2Vec, LdaModel, TfidfModel, CoherenceModel
from gensim.parsing.preprocessing import preprocess_string, strip_tags, strip_punctuation, strip_multiple_whitespaces, strip_numeric, remove_stopwords, strip_short


import nltk
nltk.download('punkt', quiet=True)
nltk.download('stopwords', quiet=True)
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

?? 說明

固定??scipy==1.11.4?? 與??gensim==4.3.2??，避免兼容問題；
??nltk?? 用于分詞和停用詞處理；
??WordCloud??? 和??Seaborn?? 負責可視化；
設置 warnings 忽略掉無關提示，輸出更干凈。

2. 構建統一的 NLP 流水線類

我們將所有步驟封裝進一個 AdvancedGensimPipeline 類，方便調用和復用。

class AdvancedGensimPipeline:
   def __init__(self):
       self.dictionary = None
       self.corpus = None
       self.lda_model = None
       self.word2vec_model = None
       self.tfidf_model = None
       self.similarity_index = None
       self.processed_docs = None
      
   def create_sample_corpus(self):
       """Create a diverse sample corpus for demonstration"""
       documents = [           
           "Data science combines statistics, programming, and domain expertise to extract insights",
           "Big data analytics helps organizations make data-driven decisions at scale",
           "Cloud computing provides scalable infrastructure for modern applications and services",
           "Cybersecurity protects digital systems from threats and unauthorized access attempts",
           "Software engineering practices ensure reliable and maintainable code development",
           "Database management systems store and organize large amounts of structured information",
           "Python programming language is widely used for data analysis and machine learning",
           "Statistical modeling helps identify patterns and relationships in complex datasets",
           "Cross-validation techniques ensure robust model performance evaluation and selection",
           "Recommendation systems suggest relevant items based on user preferences and behavior",
           "Text mining extracts valuable insights from unstructured textual data sources",
           "Image classification assigns predefined categories to visual content automatically",
           "Reinforcement learning trains agents through interaction with dynamic environments"
       ]
       return documents

?? 說明這里我們用一組涵蓋 數據科學、機器學習、推薦系統、云計算、安全 的文檔作為示例語料，保證后續的模型有足夠多樣的主題。

3. 文本預處理

對原始文本做清洗、分詞、去停用詞等處理。

def preprocess_documents(self, documents):
       """Advanced document preprocessing using Gensim filters"""
       print("Preprocessing documents...")
      
       CUSTOM_FILTERS = [
           strip_tags, strip_punctuation, strip_multiple_whitespaces,
           strip_numeric, remove_stopwords, strip_short, lambda x: x.lower()
       ]
      
       processed_docs = []
       for doc in documents:
           processed = preprocess_string(doc, CUSTOM_FILTERS)
          
           stop_words = set(stopwords.words('english'))
           processed = [word for word in processed if word notin stop_words and len(word) > 2]
          
           processed_docs.append(processed)
      
       self.processed_docs = processed_docs
       print(f"Processed {len(processed_docs)} documents")
       return processed_docs

?? 說明

使用 Gensim 內置過濾器去掉標點、數字、HTML 標簽等；
用 NLTK 的停用詞表進一步清洗；
最終得到干凈的 token 列表。

4. 構建字典與語料庫

def create_dictionary_and_corpus(self):
       """Create Gensim dictionary and corpus"""
       print("Creating dictionary and corpus...")
      
       self.dictionary = corpora.Dictionary(self.processed_docs)
       self.dictionary.filter_extremes(no_below=2, no_above=0.8)
       self.corpus = [self.dictionary.doc2bow(doc) for doc in self.processed_docs]
      
       print(f"Dictionary size: {len(self.dictionary)}")
       print(f"Corpus size: {len(self.corpus)}")

?? 說明

??Dictionary?? 是詞典，把單詞映射為唯一 ID；
??doc2bow?? 將文檔轉為稀疏向量（Bag of Words）；
??filter_extremes?? 去掉太稀有或太頻繁的詞。

5. Word2Vec 詞向量建模與相似度分析

def train_word2vec_model(self):
       """Train Word2Vec model for word embeddings"""
       print("Training Word2Vec model...")
      
       self.word2vec_model = Word2Vec(
           sentences=self.processed_docs,
           vector_size=100,
           window=5,
           min_count=2,
           workers=4,
           epochs=50
       )
       print("Word2Vec model trained successfully")
      
   def analyze_word_similarities(self):
       """Analyze word similarities using Word2Vec"""
       print("\n=== Word2Vec Similarity Analysis ===")
      
       test_words = ['machine', 'data', 'learning', 'computer']
      
       for word in test_words:
           if word in self.word2vec_model.wv:
               similar_words = self.word2vec_model.wv.most_similar(word, topn=3)
               print(f"Words similar to '{word}': {similar_words}")
      
       try:
           if all(w in self.word2vec_model.wv for w in ['machine', 'computer', 'data']):
               analogy = self.word2vec_model.wv.most_similar(
                   positive=['computer', 'data'],
                   negative=['machine'],
                   topn=1
               )
               print(f"Analogy result: {analogy}")
       except:
           print("Not enough vocabulary for complex analogies")

?? 說明

??vector_size=100?? 表示每個詞被嵌入 100 維空間；
??most_similar?? 可找出語義上最接近的詞；
類比分析示例：computer + data - machine ≈ ?。

6. LDA 主題建模與可視化

def train_lda_model(self, num_topics=5):
       """Train LDA topic model"""
       print(f"\nTraining LDA model with {num_topics} topics...")
      
       self.lda_model = LdaModel(
           corpus=self.corpus,
           id2word=self.dictionary,
           num_topics=num_topics,
           random_state=42,
           passes=20,
           alpha='auto'
       )
       print("LDA model trained successfully")
      
   def analyze_topics(self, num_words=5):
       """Display discovered topics"""
       print("\n=== LDA Topics ===")
       topics = self.lda_model.print_topics(num_words=num_words)
       for idx, topic in topics:
           print(f"Topic {idx}: {topic}")
       return topics
      
   def visualize_topics(self):
       """Visualize topic distributions with word clouds"""
       print("\nGenerating topic word clouds...")
       fig, axes = plt.subplots(1, 3, figsize=(15, 5))
      
       for i, ax in enumerate(axes.flatten()):
           if i >= self.lda_model.num_topics:
               break
           words = dict(self.lda_model.show_topic(i, topn=15))
           wc = WordCloud(width=400, height=300, background_color='white')
           wc.generate_from_frequencies(words)
           ax.imshow(wc, interpolation='bilinear')
           ax.set_title(f'Topic {i}')
           ax.axis('off')
      
       plt.tight_layout()
       plt.show()

?? 說明

LDA 可將文檔分解為多個主題分布；
??alpha='auto'?? 讓模型自動調節主題稀疏度；
通過WordCloud可直觀展示每個主題的核心詞。

7. TF-IDF 相似度建模與語義搜索

def build_tfidf_similarity_index(self):
       """Build TF-IDF similarity index for documents"""
       print("\nBuilding TF-IDF similarity index...")
       self.tfidf_model = TfidfModel(self.corpus)
       corpus_tfidf = self.tfidf_model[self.corpus]
       self.similarity_index = similarities.MatrixSimilarity(corpus_tfidf)
       print("Similarity index created")
      
   def perform_semantic_search(self, query, topn=3):
       """Perform semantic search using TF-IDF"""
       print(f"\n=== Semantic Search Results for: '{query}' ===")
      
       query_processed = preprocess_string(query)
       query_bow = self.dictionary.doc2bow(query_processed)
       query_tfidf = self.tfidf_model[query_bow]
       similarities_scores = self.similarity_index[query_tfidf]
      
       ranked_results = sorted(enumerate(similarities_scores), key=lambda x: -x[1])[:topn]
       for idx, score in ranked_results:
           print(f"Document {idx}: {score:.3f}")
       return ranked_results

?? 說明

通過 TF-IDF 計算文檔間的余弦相似度；
可以對任意查詢做語義搜索，找出最相關的文檔。

8. 模型評估與文檔分類

def evaluate_topic_coherence(self):
       """Evaluate topic coherence for LDA model"""
       print("\nEvaluating topic coherence...")
       coherence_model = CoherenceModel(
           model=self.lda_model,
           texts=self.processed_docs,
           dictionary=self.dictionary,
           coherence='c_v'
       )
       coherence = coherence_model.get_coherence()
       print(f"Topic coherence score: {coherence:.3f}")
       return coherence
      
   def classify_document(self, doc_index):
       """Classify document into most probable topic"""
       print(f"\nClassifying document {doc_index}...")
       doc_bow = self.corpus[doc_index]
       topics = self.lda_model.get_document_topics(doc_bow)
       topics_sorted = sorted(topics, key=lambda x: -x[1])
       print(f"Document {doc_index} topics: {topics_sorted}")
       return topics_sorted[0] if topics_sorted elseNone

?? 說明

??Coherence Score?? 衡量主題模型效果（越高越好）；
??classify_document?? 可給定某篇文檔，輸出其最可能的主題分類。

9. 主函數運行

def main():
   print("=== Advanced NLP Pipeline Demonstration ===\n")
   pipeline = AdvancedGensimPipeline()

   documents = pipeline.create_sample_corpus()
   print("Sample documents:")
   for i, doc in enumerate(documents[:3]):
       print(f"Doc {i}: {doc}")
   print("...")

   processed_docs = pipeline.preprocess_documents(documents)
   pipeline.create_dictionary_and_corpus()

   pipeline.train_word2vec_model()
   pipeline.analyze_word_similarities()

   pipeline.train_lda_model(num_topics=3)
   pipeline.analyze_topics(num_words=7)
   pipeline.visualize_topics()

   pipeline.build_tfidf_similarity_index()
   pipeline.perform_semantic_search("machine learning algorithms", topn=2)

   pipeline.evaluate_topic_coherence()
   pipeline.classify_document(0)

if __name__ == "__main__":
   main()

運行后你會看到：