EXCLUSIVE – Coupling top-down and bottom-up approaches to Natural Language Processing

“I love deadlines. I
love the whooshing noise they make as they go by.”

Most humans would be able to detect the sarcasm in the quote
above, even if it takes them a moment or two. But imagine making a computer
understand the sentiment expressed in the above sentence.

That is the sort of challenge, Dr. Erik Cambria (Assistant
Professor at the School of Computer Science and Engineering at Nanyang
Technological University) and his team at SenticNet
are trying to tackle. They are
dealing with the fundamental problems of natural language processing (NLP) for
sentiment analysis. Natural language, which is the language we use for
communicating with each other, is rather different from the way we communicate
with computers. Natural language is ambiguous, complex, chaotic. Constructed
languages, such as programming languages, adhere to strict rules and logic.

Wikipedia defines sentiment analysis as the use of NLP, text
analysis, computational linguistics, and biometrics to systematically identify,
extract, quantify, and study affective states and subjective information.
Applications involve analysing the positive, negative and neutral
sentiments in online customer reviews, surveys, feedback, social media postings
and this has great utility in range of fields, from marketing to finance and healthcare. 

The problem
is much more complicated than it seems. For instance, if a statement is
sarcastic, as the one above, something which looks positive is actually
negative (love is hate). Understanding this polarity (whether a sentiment is
positive or negative) is a core aspect of sentiment analysis. It involves the
use of deep learning, psychology, and also linguistics, demonstrating the
multi-disciplinary nature of the field.

Deep
learning helps detect some patterns, such as the usual occurrence of a big
shift in polarity in a sarcastic comment (positive followed by negative), linguistics
provide insights on sentence structure, while psychology is important because
whether a statement is sarcastic or not can be dependent on the personality of
the individual.

To take
another example, saying “This phone is expensive but nice” is not the same as
saying “This phone is nice but expensive.” In fact, the sentiments expressed
are polar opposites, though the words used are the same. Here, the
understanding of sentence structure based on linguistics is key. When the ‘but’
conjunction is used, positive followed by negative yields negative but negative
followed by positive yields positive.

To
understand the approach of SenticNet to dealing with such challenges and
improving sentiment analysis, we need to look at its origins.

Origins from a commonsense knowledge base

SenticNet
started as a project in MIT Media Lab in 2009.

“They had
this big knowledge base of commonsense and I thought, why don’t we use it
for sentiment analysis,” Dr. Cambria said, “back then, sentiment analysis was
not very popular but in the past few years, its popularity has increased
dramatically. Because of the research challenges, and also because of
the business opportunities. For instance, so many companies want to know
what their customers like about their products.”

In AI
research, commonsense knowledge is the collection of facts and information that
an ordinary person is expected to know. Facts so obvious, so trivial, that no
one would think of mentioning them explicitly, like a chair is for sitting
down, or that we drink water to quench our thirst. 

Natural language is only
used to communicate knowledge which we don’t have based on shared experience. The
challenge is to get this general knowledge that most people possess,
represented in a way that it is available to AI programs.

A knowledge
base here refers to a semantic network with millions of nodes, connected by
links that encode the commonsense piece of information. For example, beer and drink
could be two nodes and connecting the two would represent the taken-for-granted
information that “beer is a drink”.

The MIT Media Lab has a portal called the Open Mind Common Sense
(OMCS), which collects pieces of knowledge from volunteers on the Internet by
enabling them to enter commonsense into the system with no special training or
knowledge of computer science.

Volunteers on
the web would answer questions like– “what is a bed used for?”, “what is a
beer for?”, “where do you usually find the knife?”. Only those answers
which occurred more than a few times would be inserted into the semantic graph.
“If many people said that the bed is for sleeping, you take that as a good
piece of commonsense” Dr. Cambria said. 

ConceptNet is a semantic
network based on the information in the OMCS database. SenticNet was built
based on ConceptNet, focusing on concepts that are either positive or negative,
because the eventual objective of SenticNet is to conduct sentiment analysis.

“We started
as just a knowledge base, then from there we went on into the fundamental
problems of natural language processing for sentiment analysis. While
before we were just focusing on knowledge representation, later we got more and
more interested in commonsense reasoning and linguistics. We went from having
just SenticNet to having Sentic patterns and other
reasoning techniques like AffectiveSpace and things that altogether allow us to
do sentiment analysis in a human-like way,” Dr. Cambria said describing
the evolution of SenticNet.

Machine learning is not enough

Dr. Cambria
said, “We try to take inspiration from how the human brain actually understands
things, which is a very different approach from pure machine learning.”  

The big
difference between Sentic computing and other techniques is that Sentic
computing is a hybrid approach that uses machine learning alongside
knowledge representation, reasoning and linguistics. 

With recent developments in machine learning methods like deep
networks, most researchers are pinning their hopes on feeding massive volumes
of data to algorithms. Dr.
Cambria believes that commonsense is key to improving AI. Simply relying only
on statistics, probabilities, co-occurrence frequencies is not enough.

He went on to highlight three big issues with machine
learning. The first is ‘Dependency’, as machine learning requires a lot of
training data and is domain-dependent.

The second issue is ‘Consistency’, as changes or tweaks in
the learning model may lead to different results. The third is ‘Transparency’,
that is, the way machine learning performs decision-making is a black box. We
do not know why the algorithms arrived at the conclusions they did. In fact,
this very same fact makes machine learning a powerful tool. Researchers don’t
need to understand the data. They
can just feed data to a neural network or whatever learning algorithm they are
using, this learns the features automatically, and then it takes decisions.
But we never know why the algorithm takes those decisions. This lack of
transparency can be a major problem if we are using AI to perform activities
that involves ethics like, say, selecting candidates for a job opening.  

In the context of NLP, Dr. Cambria said that these issues
are crucial because, unlike in other fields, they prevent AI from achieving
human-like performance. AI researchers need to bridge the gap between
statistical NLP and many other disciplines that are necessary for understanding
human language, such as linguistics, commonsense reasoning, and affective
computing (affective computing is the study and development of systems and
devices that can recognise, interpret, process, human affects or emotions).

Coupling top-down and
bottom-up AI

Because of the reasons discussed above, Dr. Cambria
advocates a combination
of symbolic and sub-symbolic AI. Symbolic models, such as semantic networks,
represent a top-down approach to encode meaning. Sub-symbolic methods, such as
neural networks, represent a bottom-up approach to infer syntactic patterns
from data (syntax is the set of rules, principles, and processes that govern
word order and sentence structure). The top-down approach helps gain
transparency, while data-driven deep learning enables the automatic detection
of patterns.

In a paper titled “SenticNet 5: Discovering Conceptual
Primitives for Sentiment Analysis by Means of Context Embeddings”, Dr.
Cambria along with his co-authors explores how the two approaches might
complement each other. The paper talks about the use of the bag-of-concepts
model (as opposed to bag-of-words in which a text is represented as a bag or
set of its constituent words) for sentiment analysis. The bag-of-concepts has
the advantage over bag-of-words of being able to deal with multiword
expressions like ‘pretty ugly’ or ‘sad smile’, which would be split up in the
latter model and hence lose their polarity, i.e., their positive or negative
meaning (as in pretty used as an adjective rather than an adverb). And it
avoids the blind use of keywords and word co-occurrence counts.

But now the problem is that the bag-of-concepts model cannot
achieve a comprehensive coverage of meaningful concepts, i.e., a full list of
multiword expressions that actually make sense. Models could be used to extract
concepts from raw data but such approaches are prone to errors due to the richness
and ambiguity of natural language. This is based on the idea that there is a
finite set of mental primitives for affect-bearing concepts and a finite set of
principles of mental combination governing their interaction.

The paper goes on to propose the generalisation of concepts
with related meaning, such as ‘munch toast’ and ‘slurp noodles’, into the
conceptual primitive ‘EAT FOOD’. Sub-symbolic AI could now be used to automatically
discover the conceptual primitives that can better generalise SenticNet’s
commonsense knowledge.

This approach would also help in tackling the symbol
grounding problem. Our understanding of language is grounded in the physical
world, in sensations, in memory. A computer does not learn meaning like that. A
meaning of a word on a page or computer screen is ungrounded. And looking it up
in a dictionary would not help.

This article
explains the problem like this: “If I tried to look up the meaning of a word I
did not understand in a (unilingual) dictionary of a language I did not already
understand, I would just cycle endlessly from one meaningless definition to
another. My search for meaning would be ungrounded. In contrast, the meaning of
the words in my head — the ones I do understand — are
"grounded" (by a means that cognitive neuroscience will eventually
reveal to us). And that grounding of the meanings of the words in my head
mediates between the words on any external page I read (and understand) and the
external objects to which those words refer.”

In the approach presented in the paper, several adjectives
and verbs are defined in function of only one ‘primitive’ item thereby
grounding those meanings in that one primitive. It does not solve the symbol
grounding problem but reduces it.

Current applications

SenticNet’s
research is being applied in several projects spanning from
fundamental knowledge representation problems to applications of commonsense
reasoning in contexts such as big social data analysis and human-computer
interaction. 

For
instance, a project in collaboration with Prof. Roy Welsch from MIT
Sloan School of Management focuses on natural
language based financial forecasting (NLFF). Markets are driven by
sentiments. Understanding those sentiments from data can be used for predicting
market movements.

SenticNet
is also developing tools that allow patients to easily and efficiently measure
their health related quality of life and improving human-computer interaction (HCI) by developing dialogue systems
with commonsense.

Another
project, called PONdER (Public Opinion of Nuclear Energy) aims to
collect, aggregate, and analyse opinions towards nuclear energy in different
languages and across Singapore, Malaysia, Indonesia, Thailand, and Vietnam.
Understanding how the public perceives nuclear energy in the region enables
policymakers to make informed national policies and decisions pertaining to
nuclear energy, as well as shape communication strategies to inform the public
about nuclear energy. 

Dr. Cambria
said that personally he is more interested in the fundamental problems of
AI and sentiment analysis. For example, solving the symbol grounding
problem or building machines that can really understand language (IQ),
emotions (EQ), and culture (CQ).

“Today, we
still don’t have machines that really understand natural
language. Siri does not understand natural language, Watson is an amazing answering
machine but it does not understand language. At SenticNet, we want to go beyond
rule-based and stats-based systems. What we are working on is
not really NLP research anymore; it is natural language understanding.”