nso

README.md

@@ -1,3 +1,19 @@
 # CourseProject
+## Coursera Transcripts Analyzer
+
+## Document submission list
+
+### Project Proposal.pdf  -  Project Proposal
+
+### Progress Report.pdf   -  Project Progress Report
+
+### Final Documentation.pdf  -  Project Final Report
+
+### Demo410.mp4  - Presentation and App demonstration video
+
+### Slides.pdf  -  Presentation Slides
+
+### app.py  -  Source code
+
+
 
-Please fork this repository and paste the github link of your fork on Microsoft CMT. Detailed instructions are on Coursera under Week 1: Course Project Overview/Week 9 Activities.

app.py

@@ -0,0 +1,94 @@
+import spacy
+import gensim
+import gensim.corpora as corpora
+from gensim import models
+import streamlit as st
+from spacy.lang.en import English
+import nltk
+from nltk.corpus import wordnet as wn
+from nltk.stem.wordnet import WordNetLemmatizer
+import os, glob
+import pickle
+import pyLDAvis
+import pyLDAvis.gensim_models as gensimvis
+from streamlit import components
+
+# Load "en_core_web_sm"
+nlp = spacy.load("en_core_web_sm")
+parser = English()
+
+# Functions
+#Text Cleaning
+def tokenize(text):
+    lda_tokens = []
+    tokens = parser(text)
+    for token in tokens:
+        if token.orth_.isspace():
+            continue
+        elif token.like_url:
+            lda_tokens.append('URL')
+        elif token.orth_.startswith('@'):
+            lda_tokens.append('SCREEN_NAME')
+        else:
+            lda_tokens.append(token.lower_)
+    return lda_tokens
+
+def get_lemma(word):
+    lemma = wn.morphy(word)
+    if lemma is None:
+        return word
+    else:
+        return lemma
+
+def get_lemma2(word):
+    return WordNetLemmatizer().lemmatize(word)
+
+def prepare_text_for_lda(text):
+    tokens = tokenize(text)
+    tokens = [token for token in tokens if len(token) > 4]
+    tokens = [token for token in tokens if token not in en_stop]
+    tokens = [get_lemma(token) for token in tokens]
+    return tokens
+
+nltk.download('wordnet')
+nltk.download('omw-1.4')
+nltk.download('stopwords')
+en_stop = set(nltk.corpus.stopwords.words('english'))
+
+# HTML Start here
+st.header('Topic Modeling for Course Transcripts', )
+st.subheader(" - By Free Topics Team")
+course_name = st.text_input('Input your course number here and press enter (from 1 to 11):')
+BASE = 'data/'
+
+text_data = []
+
+path = BASE + course_name
+for filename in glob.glob(os.path.join(path, '*.txt')):
+    with open(os.path.join(os.getcwd(), filename), 'r') as f:  # open in readonly mode
+        print("NAMES", filename)
+        txt = ""
+        for line in f:
+            txt += line + " "
+        tokens = prepare_text_for_lda(txt)
+        text_data.append(tokens)
+
+dictionary = corpora.Dictionary(text_data)
+corpus = [dictionary.doc2bow(text) for text in text_data]
+
+# LDA with Gensim
+pickle.dump(corpus, open('corpus.pkl', 'wb'))
+dictionary.save('dictionary.gensim')
+
+NUM_TOPICS = 5
+ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=NUM_TOPICS, id2word=dictionary, passes=15)
+ldamodel.save('model5.gensim')
+topics = ldamodel.print_topics(num_words=4)
+
+for topic in topics:
+    st.write(topic)
+
+# pyLDAvis
+vis = gensimvis.prepare(ldamodel, corpus, dictionary)
+html_string = pyLDAvis.prepared_data_to_html(vis)
+st.components.v1.html(html_string, width=1300, height=800)

data/1/01_welcome.en.txt

@@ -0,0 +1,17 @@
+In 1789 philosopher and professor of Law, Jeremy
+Bentham, famously argued that it was the ability to suffer and not the ability to reason or
+to talk that should be the benchmark for decisions relating to how we treat animals. Welcome
+to the Animal Behaviour and Welfare course My name is Professor Nat Waran and I am the Director
+of the Jeanne Marchig International Centre for Animal Welfare Education at the University
+of Edinburgh's Royal (Dick) School of Veterinary studies, here in bonnie Scotland. Whilst the
+idea that animals such as Peanut and Bella, here, might not have feelings seems nonsensical
+to most people, we have to appreciate that it is only relatively recently become accepted
+that animals are sentient and therefore feel things like pain and fear and stress as well
+as more positive emotions like happiness and pleasure, all of which are important for their
+welfare. Animals play a huge part in the lives of many people and although we rely on them
+for all aspects of our well-being, for food, draft power, medical advances, clothing, sport,
+as well as for pleasure, companionship, protection and comfort, often their quality of life is
+questionable. By the end of this course we would have gone well beyond those early scientists and philosophers who believed that animals like Latchino were like machines and we would
+have explored why animal welfare is so internationally, politically and socially important and how
+enlighting a scientific approach to the study of animal welfare can be.
+We look forward to welcoming you to the course.

data/1/02_an-introduction-to-farm-animal-welfare.en.txt

@@ -0,0 +1,12 @@
+In this week's course, we suggest you
+start with the interactive session on the production cycle of animals, as this will go through how farm
+animals are actually produced and where the likely welfare challenges
+happen during the production cycle. If you're more used to farm animals,
+you might want to skip that section. Otherwise, there are three lectures. One by myself on dairy cows,
+one by my colleague Susan Jarvis on pigs, and another by my colleagues from
+the Aviation Research Center on poultry. Finally, there is another interactive
+session on live animal transport, and we suggest you leave that until
+after you've watched the lectures. At the end of the week, you might want to
+do the quizzes to test your knowledge, start to ask questions on
+the discussion board, and join us in the Google Hangouts to see us answer
+your most frequently asked questions.

data/1/02_behaviour-based-husbandry.en.txt

@@ -0,0 +1,167 @@
+Hello everyone, and welcome to the next
+week of your MOOC in Animal Behavior and Welfare here at
+the University of Edinburgh. This week, we're going to be
+looking at the behavior and welfare of wild animals that
+are managed under human care. For simplicity, we'll refer to them
+as wild animals in captivity and I'll talk a little bit later
+about why we use that term. Today, we'll be looking at animals that
+are managed within the zoo context. And animal welfare is a fundamental
+part of what all good zoos do. The role of the modern zoo involves
+education, conservation, and research. And animal welfare is an essential
+foundation for these activities. We can't produce good conservation
+programs where animals are engaging in natural breeding activities or are fit for reintroduction into the wild if the
+welfare of those animals is not excellent. We also cannot educate the public
+about natural wild animal behaviors if animal
+welfare isn't ensured and those animals are not exhibiting those
+behaviors in the captive setting. Good animal welfare is also essential for
+ensuring that good and valid research data is collected. If animals are physiologically stressed or
+emotionally stressed, then they're not going to give us
+good research data to collect. And so we can see that good animal
+welfare is the cornerstone of all of the activities that good
+zoos may be engaged in. [SOUND] We can provide good welfare by
+taking a behavior-based husbandry approach to how we manage animals. This means that we don't focus
+on what we're providing, we're focusing on what the animal's
+behavior is telling us that it needs by recognizing that all of the animals
+behaviors are meaningful and, therefore, helpful in informing
+us of what an animal may need. Behavior based husbandry incorporates
+all elements of good animal welfare, good health, psychological well being,
+and natural behavioral expression. Behavior-based husbandry considers not
+just elements, such as enclosure design or enrichment, but also human animal
+relationships, the animal's choices with regards to its environment, mating
+opportunities or social interactions, and a level of control over its
+environment and its daily routines. Positive human animal interactions are the
+foundation of providing good welfare for the animals that we manage. If we consider that these animals rely
+on us to provide for all of their needs. Food, shelter, enrichment,
+mating opportunities and companionship, we can understand that if we are
+aggressive, unpredictable, or negative in our interactions to our animals,
+we can create significant stress for them. Perhaps we're having a bad day or we're
+worried about something in our lives. That may be the case, but
+our animals do not understand that. And if we are abrupt, unresponsive, or short-tempered with them,
+our behavior can create anxiety for them. It is important to recognize
+that not all stress is negative. Continuous or high level stress from which
+an animal cannot escape can be negative, and may mean that an animal is no longer
+able to cope with its captive environment. On the other hand, an environment which
+does not offer any kind of challenge may be safe for an animal, but can also be
+both physically and mentally frustrating. Just as we humans engage in interesting
+and exciting activities, such as sports or socializing, to stimulate our minds and bodies, it is important that we provide
+animals with similar challenges. Opportunities for mating, enrichment, and social interaction can all
+provide positive, stimulating and short-term stress,
+which may be beneficial for our animals. Operant conditioning is increasingly
+popular in captive wildlife facilities to facilitate the husbandry procedures and
+support the human and animal bond. In order to train animals, most facilities
+employ a positive reinforcement methodology, recognizing that punishment
+and negative reinforcement often end up creating unnecessary anxiety and fear
+in order to force particular behaviors. In these videos, we can see direct
+punishment of an elephant hit with a stick and negative reinforcement of elephants
+in a circus, where the trainers are using hand held spikes to force
+the elephants into position. Positive reinforcement rewards
+the desired behaviors and so, if delivered properly, should not be
+stressful for the animal being trained. Whilst operant conditioning may
+provide cognitive enrichment for our animals, it's important that in order
+to do so, it is properly delivered. And the animal is adequately rewarded for
+its efforts. Any operant conditioning or training
+program must be both progressive and enjoyable in order to be enriching for
+the animal. Training programs, which are repetitive,
+boring, or which create frustration for the animal may have
+a negative welfare impact. The management of training
+sessions is important. By engaging interactions with our animals,
+we must recognize that we may create disruption to their daily
+routines and normal social interactions. Separating an animal from its conspecifics
+for training may create stress, anxiety, or frustration in
+the individuals not selected to receive the additional attention and food
+rewards that training programs confer. Additionally, if training programs
+are not well-planned or structured or delivered, they may create frustration or
+confusion. And this can lead to a negative experience
+for both the animal and the trainer. It is also important to understand that
+interactions with our animals should be much broader than just
+intimate in training sessions. In this video,
+we see a socially isolated chimpanzee. He has lived alone since its mate
+died a few years before, and since then his social interaction
+has been limited to his keepers. As the keeper focus is on cleaning,
+feeding, and basic husbandry, and they have had limited
+training in animal behavior, they have, so far, been unable to recognize his need for
+social interaction. We can see a visiting applied ethologist,
+who understands his social cues, and is responding to his communication and desire for social interaction,
+engaging in play behavior with him. For social animals,
+such as a chimpanzee, this type of social interaction is vitally important to
+ensuring that the animal does not suffer. Similarly, here we see
+a singly housed gibbon, who's on the floor of her enclosure and
+pressed against the barrier, attempting to solicit contact for
+social grooming from humans. As the barrier is very narrow, we had to use a stick to provide
+her with tactile contact. Throughout the process, she turned and
+presented each side of her body. Tactile grooming like this is
+essential for social primates. But it is important that we
+consider social interactions and how our own behavior may impact
+the emotional states of all of the species that we keep in captivity. Environmental or cognitive enrichment is often seen as
+a panacea for many behavioral problems. But whilst they can be very useful,
+it needs to be applied strategically with clear goals in mind and
+a recognition of possible limitations. For example, in this photograph, the bear has been provided
+with an enriched environment. It has been given a tire, a type of
+occupational or physical enrichment. But I'm sure that we can all recognize
+that regardless of this enrichment, this environment is still totally
+unsuitable for this animal. In this case, the environment has been
+enriched with a log pile and a bowl, but still remains barren and is completely
+unsuitable to meet the psychological and physical needs of this species. Enrichment comprises many
+different categories and where practical a complex variety
+enrichment categories should be applied. These categories include physical
+enrichments, such as bedding material, branch work, burrows, nesting boxes,
+pools, substrates, and appropriate vegetation. Occupational enrichment, these
+are natural or non-natural objects that can be manipulated,
+such as toys or traffic cones. Feeding enrichment, this is the provision
+of food related activities, novel food items and devices,
+or the scattering of food. Sensory enrichment,
+the provision of novel or familiar scents, sounds,
+visual, or tactile stimuli. >> [INAUDIBLE]. >> Cognitive enrichment,
+items that present mental challenges, and engage the animal in
+problem-solving often for a reward. >> [INAUDIBLE] >> And social enrichment,
+provision of intra or interspecies social stimuli, training and
+positive human-animal relationships. >> [INAUDIBLE]
+>> For enrichment to be successful, we need to
+consider what needs we are trying to meet. The Shape of Enrichment, an organization
+dedicated to developing and delivering zoo animal enrichment and Disney's
+Animal Kingdom have both developed strategic approaches to developing
+goal-oriented enrichment programs. And I'd suggest that you
+check out their websites. This is an example of
+goal-orientated enrichment. This browse hanger was developed for
+moloch gibbons to provide both cognitive enrichment, and
+promote natural foraging behavior. Sometimes, an overly simplistic
+approach can be taken towards goal orientated enrichment. A classic example is that of live feeding. Some people believe that providing
+a domestic prey animal to a predator within a confined environment
+supports the predator's natural behaviors. However, when we consider the various
+complex stages of predation, including prey finding through olfactory
+or visual activity, stalking and pouncing or striking, we can see that
+actually the killing of a chicken by tiger does not offer
+much of a cognitive or physical challenge, and
+is unlikely to be particularly enriching. Even in the case of reptiles,
+the provision of live prey simply only replicates the strike and the
+constrict aspects of the hunting sequence. And these behaviors can easily be
+provided for by offering warm, humanely killed carcasses,
+which elicits the same responses in the predator without the negative
+welfare experience for the prey animal. Attempts to provide predators with genuine
+simulated hunting experiences are much more challenging. This lion rover was developed
+by Mark Kingston Jones to mimic natural predatory behaviors
+in a pride of captive lions. And whilst not perfect, it can be
+controlled by the keepers to offer the lions enrichment based on prey
+location, stalking, and chasing. The benefit of a goal-oriented enrichment
+program is that it considers the needs of the individual, rather than taking
+a scattergun approach to enrichment. Enrichment items are presented,
+assessed, and reviewed to ensure they
+are meeting the goal. In this example, we see two young bear cubs rescued from
+the illegal wildlife trade in Vietnam, where they were destined for
+bear farms as part of the bear bile trade. These young cubs have already
+experienced much trauma, maternal separation, transport, and
+significant human interactions. And so it is important the new
+experiences provide positive learning opportunities for them. We can see in this video that Misty,
+the female cub, is very confident and
+curious to explore her new bubble bath. Her brother Rain, however,
+is a little more anxious and quite startled to see his
+sister looking so strange. By monitoring this interaction, we can
+assess their individual behaviors and responses, and modify their enrichment to
+ensure that it does not cause anxiety or stress, but
+instead remains a source of pleasure and supports their environmental exploration. Positive human-animal
+interactions can lead to improved public engagement and education. And improved welfare and enrichment supports a greater
+range of natural animal behaviors. But offering our animals
+a variety of opportunities, they can choose their
+preferred activities. And this choice and control over their daily activities is
+likely to result in improved welfare. At all times, it is important that we are aware of
+how what we do may affect our animals.