Skip to content

Performs multi document summarization. Includes a method to generate summaries: The method uses a sentence importance score calculator based on various semantic features and a semantic similarity score to select sentences that would be most representative of the document. It uses stack-decoder algorithm as used as a template and builds on it to …

Notifications You must be signed in to change notification settings

inspirationhyl/Summarization

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Aim:
The project takes in documents and output an extraction based summary i.e. selects sentences from the given set that satisfy the length constraint and also maximize the overall score of the summary. (Importance Score of a summary is the sum of scores of individual sentences.)

Main class:
Expects one argument - path to the directory containing the data.
For the project, the sample data files are in the 'data' directory.

Method:
The project contains the implementation of the StackDecoder method for summary generation.
http://cs.stanford.edu/people/ssandeep/reports/PACLIC-09.pdf

The current similarity scorer set is 'CosineSimilarityScorer'. This can be changed to any other scorer by implementing the SimilarityScorer interface.
Other alternative included is JaccardSimilarityScorer.

Final report of the project:
http://cs.stanford.edu/people/ssandeep/reports/CS224N.pdf

Output:
The output summaries are generated in the 'summaries' folder.

External sources: (files not mentioned below were all coded by me)
Files that are not coded by me but provided as starter codes for the 'Natural Language Processing' class at Stanford:
com.util package: 
	Counter, CounterMap, MapFactory, PorterStemmer, PQ
com.score.importance:
	All the files
com.decoding.stackdecoder:
	SpecialPQ
	
Contributions:
I have written the framework code for loading the dataset and segregating it into topics, documents, summaries and sentences, also designed the structure.
I have also implemented the 'Stackdecoder' algorithm using the above framework.
The common platform also made it easy for others to plugin their code and implement other methods.

NOTE:
LSP is an external program that is not currently included. An open source alternative is to use NTLK (http://nltk.org/) tokenizer.

About

Performs multi document summarization. Includes a method to generate summaries: The method uses a sentence importance score calculator based on various semantic features and a semantic similarity score to select sentences that would be most representative of the document. It uses stack-decoder algorithm as used as a template and builds on it to …

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published