Text Summarization using Word Frequency.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"The death of an American tourist who illegally visited the isolated North Sentinel Island had drawn the world's attention to the small island's reclusive inhabitants.They're one of the few mostly uncontacted groups left in the world, and they owe that isolation partly to geography -- North Sentinel is a small island, off the main shipping routes, surrounded by a shallow reef with no natural harbors -- partly to protective laws enforced by the Indian government, and partly to their own fierce defense of their home and their privacy. But they're not entirely uncontacted; over the last 200 years, outsiders have visited the island several times, and it often ended badly for both sides.According to a 2011 census effort, and based on anthropologists' estimates of how many people the island could support, there are probably somewhere between 80 and 150 people on North Sentinel Island, although it could be as many as 500 or as few as 15. The Sentinelese people are related to other indigenous groups in the Andaman Islands, a chain of islands in India's Bay of Bengal, but they've been isolated for long enough that other Andaman groups, like the Onge and the Jarawa, can't understand their language.Based on a single visit to a Sentinelese village in 1967, we know that they live in lean-to huts with slanted roofs; Pandit described a group of huts, built facing one another, with a carefully-tended fire outside each one. We know that they build small, narrow outrigger canoes, which they maneuver with long poles in the relatively shallow, calm waters inside the reef. From those canoes, the Sentinelese fish and harvest crabs. They are hunter-gatherers, and if their lifestyle is anything like that of related Andamanese peoples, they probably live on fruits and tubers that grow wild on the island, eggs from seagulls or turtles, and small game like wild pigs or birds. They carry bows and arrows, as well as spears and knives, and unwelcome visitors have learned to respect their skill with all of the above. Many of those tools and weapons are tipped with iron, which the Sentinelese probably find washed ashore and work to suit their needs.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"the death of an american tourist who illegally visited the isolated north sentinel island had drawn the world's attention to the small island's reclusive inhabitants.they're one of the few mostly uncontacted groups left in the world, and they owe that isolation partly to geography -- north sentinel is a small island, off the main shipping routes, surrounded by a shallow reef with no natural harbors -- partly to protective laws enforced by the indian government, and partly to their own fierce defense of their home and their privacy. but they're not entirely uncontacted; over the last 200 years, outsiders have visited the island several times, and it often ended badly for both sides.according to a 2011 census effort, and based on anthropologists' estimates of how many people the island could support, there are probably somewhere between 80 and 150 people on north sentinel island, although it could be as many as 500 or as few as 15. the sentinelese people are related to other indigenous groups in the andaman islands, a chain of islands in india's bay of bengal, but they've been isolated for long enough that other andaman groups, like the onge and the jarawa, can't understand their language.based on a single visit to a sentinelese village in 1967, we know that they live in lean-to huts with slanted roofs; pandit described a group of huts, built facing one another, with a carefully-tended fire outside each one. we know that they build small, narrow outrigger canoes, which they maneuver with long poles in the relatively shallow, calm waters inside the reef. from those canoes, the sentinelese fish and harvest crabs. they are hunter-gatherers, and if their lifestyle is anything like that of related andamanese peoples, they probably live on fruits and tubers that grow wild on the island, eggs from seagulls or turtles, and small game like wild pigs or birds. they carry bows and arrows, as well as spears and knives, and unwelcome visitors have learned to respect their skill with all of the above. many of those tools and weapons are tipped with iron, which the sentinelese probably find washed ashore and work to suit their needs.\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = text.lower()\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove spaces, punctuations and numbers\n",
    "\n",
    "clean_text = re.sub(\"[^A-Za-z]\", \" \", text)\n",
    "clean_text = re.sub(\"\\s+\", \" \", clean_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the death of an american tourist who illegally visited the isolated north sentinel island had drawn the world s attention to the small island s reclusive inhabitants they re one of the few mostly uncontacted groups left in the world and they owe that isolation partly to geography north sentinel is a small island off the main shipping routes surrounded by a shallow reef with no natural harbors partly to protective laws enforced by the indian government and partly to their own fierce defense of their home and their privacy but they re not entirely uncontacted over the last years outsiders have visited the island several times and it often ended badly for both sides according to a census effort and based on anthropologists estimates of how many people the island could support there are probably somewhere between and people on north sentinel island although it could be as many as or as few as the sentinelese people are related to other indigenous groups in the andaman islands a chain of islands in india s bay of bengal but they ve been isolated for long enough that other andaman groups like the onge and the jarawa can t understand their language based on a single visit to a sentinelese village in we know that they live in lean to huts with slanted roofs pandit described a group of huts built facing one another with a carefully tended fire outside each one we know that they build small narrow outrigger canoes which they maneuver with long poles in the relatively shallow calm waters inside the reef from those canoes the sentinelese fish and harvest crabs they are hunter gatherers and if their lifestyle is anything like that of related andamanese peoples they probably live on fruits and tubers that grow wild on the island eggs from seagulls or turtles and small game like wild pigs or birds they carry bows and arrows as well as spears and knives and unwelcome visitors have learned to respect their skill with all of the above many of those tools and weapons are tipped with iron which the sentinelese probably find washed ashore and work to suit their needs '"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"the death of an american tourist who illegally visited the isolated north sentinel island had drawn the world's attention to the small island's reclusive inhabitants.they're one of the few mostly uncontacted groups left in the world, and they owe that isolation partly to geography -- north sentinel is a small island, off the main shipping routes, surrounded by a shallow reef with no natural harbors -- partly to protective laws enforced by the indian government, and partly to their own fierce defense of their home and their privacy.\",\n",
       " \"but they're not entirely uncontacted; over the last 200 years, outsiders have visited the island several times, and it often ended badly for both sides.according to a 2011 census effort, and based on anthropologists' estimates of how many people the island could support, there are probably somewhere between 80 and 150 people on north sentinel island, although it could be as many as 500 or as few as 15. the sentinelese people are related to other indigenous groups in the andaman islands, a chain of islands in india's bay of bengal, but they've been isolated for long enough that other andaman groups, like the onge and the jarawa, can't understand their language.based on a single visit to a sentinelese village in 1967, we know that they live in lean-to huts with slanted roofs; pandit described a group of huts, built facing one another, with a carefully-tended fire outside each one.\",\n",
       " 'we know that they build small, narrow outrigger canoes, which they maneuver with long poles in the relatively shallow, calm waters inside the reef.',\n",
       " 'from those canoes, the sentinelese fish and harvest crabs.',\n",
       " 'they are hunter-gatherers, and if their lifestyle is anything like that of related andamanese peoples, they probably live on fruits and tubers that grow wild on the island, eggs from seagulls or turtles, and small game like wild pigs or birds.',\n",
       " 'they carry bows and arrows, as well as spears and knives, and unwelcome visitors have learned to respect their skill with all of the above.',\n",
       " 'many of those tools and weapons are tipped with iron, which the sentinelese probably find washed ashore and work to suit their needs.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split into sentence list\n",
    "\n",
    "sentence_list = nltk.sent_tokenize(text)\n",
    "sentence_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords = nltk.corpus.stopwords.words(\"english\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Word Frequencies\n",
    "\n",
    "word_frequencies = {}\n",
    "for word in nltk.word_tokenize(clean_text):\n",
    "    if word not in stopwords:\n",
    "        if word not in word_frequencies:\n",
    "            word_frequencies[word] = 1\n",
    "        else:\n",
    "            word_frequencies[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "maximum_frequency = max(word_frequencies.values())\n",
    "\n",
    "for word in word_frequencies:\n",
    "    word_frequencies[word] = word_frequencies[word] / maximum_frequency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calculate Sentence Scores\n",
    "\n",
    "sentence_scores = {}\n",
    "\n",
    "for sentence in sentence_list:\n",
    "    for word in nltk.word_tokenize(sentence):\n",
    "        if word in word_frequencies and len(sentence.split(\" \")) < 30:\n",
    "            if sentence not in sentence_scores:\n",
    "                sentence_scores[sentence] = word_frequencies[word]\n",
    "            else:\n",
    "                sentence_scores[sentence] += word_frequencies[word]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'death': 0.14285714285714285,\n",
       " 'american': 0.14285714285714285,\n",
       " 'tourist': 0.14285714285714285,\n",
       " 'illegally': 0.14285714285714285,\n",
       " 'visited': 0.2857142857142857,\n",
       " 'isolated': 0.2857142857142857,\n",
       " 'north': 0.42857142857142855,\n",
       " 'sentinel': 0.42857142857142855,\n",
       " 'island': 1.0,\n",
       " 'drawn': 0.14285714285714285,\n",
       " 'world': 0.2857142857142857,\n",
       " 'attention': 0.14285714285714285,\n",
       " 'small': 0.5714285714285714,\n",
       " 'reclusive': 0.14285714285714285,\n",
       " 'inhabitants': 0.14285714285714285,\n",
       " 'one': 0.42857142857142855,\n",
       " 'mostly': 0.14285714285714285,\n",
       " 'uncontacted': 0.2857142857142857,\n",
       " 'groups': 0.42857142857142855,\n",
       " 'left': 0.14285714285714285,\n",
       " 'owe': 0.14285714285714285,\n",
       " 'isolation': 0.14285714285714285,\n",
       " 'partly': 0.42857142857142855,\n",
       " 'geography': 0.14285714285714285,\n",
       " 'main': 0.14285714285714285,\n",
       " 'shipping': 0.14285714285714285,\n",
       " 'routes': 0.14285714285714285,\n",
       " 'surrounded': 0.14285714285714285,\n",
       " 'shallow': 0.2857142857142857,\n",
       " 'reef': 0.2857142857142857,\n",
       " 'natural': 0.14285714285714285,\n",
       " 'harbors': 0.14285714285714285,\n",
       " 'protective': 0.14285714285714285,\n",
       " 'laws': 0.14285714285714285,\n",
       " 'enforced': 0.14285714285714285,\n",
       " 'indian': 0.14285714285714285,\n",
       " 'government': 0.14285714285714285,\n",
       " 'fierce': 0.14285714285714285,\n",
       " 'defense': 0.14285714285714285,\n",
       " 'home': 0.14285714285714285,\n",
       " 'privacy': 0.14285714285714285,\n",
       " 'entirely': 0.14285714285714285,\n",
       " 'last': 0.14285714285714285,\n",
       " 'years': 0.14285714285714285,\n",
       " 'outsiders': 0.14285714285714285,\n",
       " 'several': 0.14285714285714285,\n",
       " 'times': 0.14285714285714285,\n",
       " 'often': 0.14285714285714285,\n",
       " 'ended': 0.14285714285714285,\n",
       " 'badly': 0.14285714285714285,\n",
       " 'sides': 0.14285714285714285,\n",
       " 'according': 0.14285714285714285,\n",
       " 'census': 0.14285714285714285,\n",
       " 'effort': 0.14285714285714285,\n",
       " 'based': 0.2857142857142857,\n",
       " 'anthropologists': 0.14285714285714285,\n",
       " 'estimates': 0.14285714285714285,\n",
       " 'many': 0.42857142857142855,\n",
       " 'people': 0.42857142857142855,\n",
       " 'could': 0.2857142857142857,\n",
       " 'support': 0.14285714285714285,\n",
       " 'probably': 0.42857142857142855,\n",
       " 'somewhere': 0.14285714285714285,\n",
       " 'although': 0.14285714285714285,\n",
       " 'sentinelese': 0.5714285714285714,\n",
       " 'related': 0.2857142857142857,\n",
       " 'indigenous': 0.14285714285714285,\n",
       " 'andaman': 0.2857142857142857,\n",
       " 'islands': 0.2857142857142857,\n",
       " 'chain': 0.14285714285714285,\n",
       " 'india': 0.14285714285714285,\n",
       " 'bay': 0.14285714285714285,\n",
       " 'bengal': 0.14285714285714285,\n",
       " 'long': 0.2857142857142857,\n",
       " 'enough': 0.14285714285714285,\n",
       " 'like': 0.42857142857142855,\n",
       " 'onge': 0.14285714285714285,\n",
       " 'jarawa': 0.14285714285714285,\n",
       " 'understand': 0.14285714285714285,\n",
       " 'language': 0.14285714285714285,\n",
       " 'single': 0.14285714285714285,\n",
       " 'visit': 0.14285714285714285,\n",
       " 'village': 0.14285714285714285,\n",
       " 'know': 0.2857142857142857,\n",
       " 'live': 0.2857142857142857,\n",
       " 'lean': 0.14285714285714285,\n",
       " 'huts': 0.2857142857142857,\n",
       " 'slanted': 0.14285714285714285,\n",
       " 'roofs': 0.14285714285714285,\n",
       " 'pandit': 0.14285714285714285,\n",
       " 'described': 0.14285714285714285,\n",
       " 'group': 0.14285714285714285,\n",
       " 'built': 0.14285714285714285,\n",
       " 'facing': 0.14285714285714285,\n",
       " 'another': 0.14285714285714285,\n",
       " 'carefully': 0.14285714285714285,\n",
       " 'tended': 0.14285714285714285,\n",
       " 'fire': 0.14285714285714285,\n",
       " 'outside': 0.14285714285714285,\n",
       " 'build': 0.14285714285714285,\n",
       " 'narrow': 0.14285714285714285,\n",
       " 'outrigger': 0.14285714285714285,\n",
       " 'canoes': 0.2857142857142857,\n",
       " 'maneuver': 0.14285714285714285,\n",
       " 'poles': 0.14285714285714285,\n",
       " 'relatively': 0.14285714285714285,\n",
       " 'calm': 0.14285714285714285,\n",
       " 'waters': 0.14285714285714285,\n",
       " 'inside': 0.14285714285714285,\n",
       " 'fish': 0.14285714285714285,\n",
       " 'harvest': 0.14285714285714285,\n",
       " 'crabs': 0.14285714285714285,\n",
       " 'hunter': 0.14285714285714285,\n",
       " 'gatherers': 0.14285714285714285,\n",
       " 'lifestyle': 0.14285714285714285,\n",
       " 'anything': 0.14285714285714285,\n",
       " 'andamanese': 0.14285714285714285,\n",
       " 'peoples': 0.14285714285714285,\n",
       " 'fruits': 0.14285714285714285,\n",
       " 'tubers': 0.14285714285714285,\n",
       " 'grow': 0.14285714285714285,\n",
       " 'wild': 0.2857142857142857,\n",
       " 'eggs': 0.14285714285714285,\n",
       " 'seagulls': 0.14285714285714285,\n",
       " 'turtles': 0.14285714285714285,\n",
       " 'game': 0.14285714285714285,\n",
       " 'pigs': 0.14285714285714285,\n",
       " 'birds': 0.14285714285714285,\n",
       " 'carry': 0.14285714285714285,\n",
       " 'bows': 0.14285714285714285,\n",
       " 'arrows': 0.14285714285714285,\n",
       " 'well': 0.14285714285714285,\n",
       " 'spears': 0.14285714285714285,\n",
       " 'knives': 0.14285714285714285,\n",
       " 'unwelcome': 0.14285714285714285,\n",
       " 'visitors': 0.14285714285714285,\n",
       " 'learned': 0.14285714285714285,\n",
       " 'respect': 0.14285714285714285,\n",
       " 'skill': 0.14285714285714285,\n",
       " 'tools': 0.14285714285714285,\n",
       " 'weapons': 0.14285714285714285,\n",
       " 'tipped': 0.14285714285714285,\n",
       " 'iron': 0.14285714285714285,\n",
       " 'find': 0.14285714285714285,\n",
       " 'washed': 0.14285714285714285,\n",
       " 'ashore': 0.14285714285714285,\n",
       " 'work': 0.14285714285714285,\n",
       " 'suit': 0.14285714285714285,\n",
       " 'needs': 0.14285714285714285}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'we know that they build small, narrow outrigger canoes, which they maneuver with long poles in the relatively shallow, calm waters inside the reef.': 3.2857142857142847,\n",
       " 'from those canoes, the sentinelese fish and harvest crabs.': 1.2857142857142856,\n",
       " 'they carry bows and arrows, as well as spears and knives, and unwelcome visitors have learned to respect their skill with all of the above.': 1.571428571428571,\n",
       " 'many of those tools and weapons are tipped with iron, which the sentinelese probably find washed ashore and work to suit their needs.': 2.8571428571428568}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "we know that they build small, narrow outrigger canoes, which they maneuver with long poles in the relatively shallow, calm waters inside the reef. many of those tools and weapons are tipped with iron, which the sentinelese probably find washed ashore and work to suit their needs. they carry bows and arrows, as well as spears and knives, and unwelcome visitors have learned to respect their skill with all of the above. from those canoes, the sentinelese fish and harvest crabs.\n"
     ]
    }
   ],
   "source": [
    "#Text Summarization\n",
    "#Getting top 5 sentences\n",
    "\n",
    "import heapq\n",
    "summary = heapq.nlargest(5, sentence_scores, key = sentence_scores.get)\n",
    "\n",
    "print(\" \".join(summary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}