• Corpora and Vector Spaces
  
  • From Strings to Vectors
  • Corpus Streaming – One Document at a Time
  • Corpus Formats
  • Compatibility with NumPy and SciPy
    
• Topics and Transformations
  
  • Transformation interface
  • Available transformations
    
• Similarity Queries
  
  • Similarity interface
  • Where next?
    
• Experiments on the English Wikipedia
  
  • Preparing the corpus
  • Latent Semantic Analysis
  • Latent Dirichlet Allocation
    
• Distributed Computing
  
  • Why distributed computing?
  • Prerequisites
  • Core concepts
  • Available distributed algorithms
    
  
  

All the examples can be directly copied to your Python interpreter shell. IPython‘s cpaste command is especially handy for copypasting code fragments, including the leading >>> characters.

Gensim uses Python’s standard logging module to log various stuff at various priority levels; to activate logging (this is optional), run

First, let’s import gensim and create a small corpus of nine documents and twelve features [1]:

In gensim a corpus is simply an object which, when iterated over, returns its documents represented as sparse vectors. In this case we’re using a list of list of tuples. If you’re not familiar with the vector space model, we’ll bridge the gap between raw strings, corpora and sparse vectorsin the next tutorial on Corpora and Vector Spaces.

If you’re familiar with the vector space model, you’ll probably know that the way you parse your documents and convert them to vectors has major impact on the quality of any subsequent applications.

Next, let’s initialize a transformation:

A transformation is used to convert documents from one vector representation into another:

Here, we used Tf-Idf, a simple transformation which takes documents represented as bag-of-words counts and applies a weighting which discounts common terms (or, equivalently, promotes rare terms). It also scales the resulting vector to unit length (in the Euclidean norm).

Transformations are covered in detail in the tutorial on Topics and Transformations.

To transform the whole corpus via TfIdf and index it, in preparation for similarity queries:

and to query the similarity of our query vector vec against every document in the corpus:

How to read this output? Document number zero (the first document) has a similarity score of 0.466=46.6%, the second document has a similarity score of 19.1% etc.

Thus, according to TfIdf document representation and cosine similarity measure, the most similar to our query document vec is document no. 3, with a similarity score of 82.1%. Note that in the TfIdf representation, any documents which do not share any common features with vec at all (documents no. 4–8) get a similarity score of 0.0. See the Similarity Queries tutorial for more detail.