We are creating some awesome events for you. Kindly bear with us.

EXCLUSIVE – Coupling top-down and bottom-up approaches to Natural Language Processing

EXCLUSIVE – Coupling top-down and bottom-up approaches to Natural Language Processing

“I love deadlines. I
love the whooshing noise they make as they go by.”

Most humans would be able to detect the sarcasm in the quote
above, even if it takes them a moment or two. But imagine making a computer
understand the sentiment expressed in the above sentence.

That is the sort of challenge, Dr. Erik Cambria (Assistant
Professor at the School of Computer Science and Engineering at Nanyang
Technological University) and his team at SenticNet
are trying to tackle. They are
dealing with the fundamental problems of natural language processing (NLP) for
sentiment analysis. Natural language, which is the language we use for
communicating with each other, is rather different from the way we communicate
with computers. Natural language is ambiguous, complex, chaotic. Constructed
languages, such as programming languages, adhere to strict rules and logic.

Wikipedia defines sentiment analysis as the use of NLP, text
analysis, computational linguistics, and biometrics to systematically identify,
extract, quantify, and study affective states and subjective information.
Applications involve analysing the positive, negative and neutral
sentiments in online customer reviews, surveys, feedback, social media postings
and this has great utility in range of fields, from marketing to finance and healthcare. 

The problem
is much more complicated than it seems. For instance, if a statement is
sarcastic, as the one above, something which looks positive is actually
negative (love is hate). Understanding this polarity (whether a sentiment is
positive or negative) is a core aspect of sentiment analysis. It involves the
use of deep learning, psychology, and also linguistics, demonstrating the
multi-disciplinary nature of the field.

learning helps detect some patterns, such as the usual occurrence of a big
shift in polarity in a sarcastic comment (positive followed by negative), linguistics
provide insights on sentence structure, while psychology is important because
whether a statement is sarcastic or not can be dependent on the personality of
the individual.

To take
another example, saying “This phone is expensive but nice” is not the same as
saying “This phone is nice but expensive.” In fact, the sentiments expressed
are polar opposites, though the words used are the same. Here, the
understanding of sentence structure based on linguistics is key. When the ‘but’
conjunction is used, positive followed by negative yields negative but negative
followed by positive yields positive.

understand the approach of SenticNet to dealing with such challenges and
improving sentiment analysis, we need to look at its origins.

Origins from a commonsense knowledge base

started as a project in MIT Media Lab in 2009.

“They had
this big knowledge base of commonsense and I thought, why don’t we use it
for sentiment analysis,” Dr. Cambria said, “back then, sentiment analysis was
not very popular but in the past few years, its popularity has increased
dramatically. Because of the research challenges, and also because of
the business opportunities. For instance, so many companies want to know
what their customers like about their products.”

research, commonsense knowledge is the collection of facts and information that
an ordinary person is expected to know. Facts so obvious, so trivial, that no
one would think of mentioning them explicitly, like a chair is for sitting
down, or that we drink water to quench our thirst. 

Natural language is only
used to communicate knowledge which we don’t have based on shared experience. The
challenge is to get this general knowledge that most people possess,
represented in a way that it is available to AI programs.

A knowledge
base here refers to a semantic network with millions of nodes, connected by
links that encode the commonsense piece of information. For example, beer and drink
could be two nodes and connecting the two would represent the taken-for-granted
information that “beer is a drink”.

Image credit: SenticNet

The MIT Media Lab has a portal called the Open Mind Common Sense
(OMCS), which collects pieces of knowledge from volunteers on the Internet by
enabling them to enter commonsense into the system with no special training or
knowledge of computer science.

Volunteers on
the web would answer questions like– “what is a bed used for?”, “what is a
beer for?”, “where do you usually find the knife?”. Only those answers
which occurred more than a few times would be inserted into the semantic graph.
“If many people said that the bed is for sleeping, you take that as a good
piece of commonsense” Dr. Cambria said. 

ConceptNet is a semantic
network based on the information in the OMCS database. SenticNet was built
based on ConceptNet, focusing on concepts that are either positive or negative,
because the eventual objective of SenticNet is to conduct sentiment analysis.

“We started
as just a knowledge base, then from there we went on into the fundamental
problems of natural language processing for sentiment analysis. While
before we were just focusing on knowledge representation, later we got more and
more interested in commonsense reasoning and linguistics. We went from having
just SenticNet to having Sentic patterns and other
reasoning techniques like AffectiveSpace and things that altogether allow us to
do sentiment analysis in a human-like way,” Dr. Cambria said describing
the evolution of SenticNet.

Machine learning is not enough

Dr. Cambria
said, “We try to take inspiration from how the human brain actually understands
things, which is a very different approach from pure machine learning.”  

The big
difference between Sentic computing and other techniques is that Sentic
computing is a hybrid approach that uses machine learning alongside
knowledge representation, reasoning and linguistics. 

With recent developments in machine learning methods like deep
networks, most researchers are pinning their hopes on feeding massive volumes
of data to algorithms. Dr.
Cambria believes that commonsense is key to improving AI. Simply relying only
on statistics, probabilities, co-occurrence frequencies is not enough.

He went on to highlight three big issues with machine
learning. The first is ‘Dependency’, as machine learning requires a lot of
training data and is domain-dependent.

The second issue is ‘Consistency’, as changes or tweaks in
the learning model may lead to different results. The third is ‘Transparency’,
that is, the way machine learning performs decision-making is a black box. We
do not know why the algorithms arrived at the conclusions they did. In fact,
this very same fact makes machine learning a powerful tool. Researchers don’t
need to understand the data. They
can just feed data to a neural network or whatever learning algorithm they are
using, this learns the features automatically, and then it takes decisions.
But we never know why the algorithm takes those decisions. This lack of
transparency can be a major problem if we are using AI to perform activities
that involves ethics like, say, selecting candidates for a job opening.  

In the context of NLP, Dr. Cambria said that these issues
are crucial because, unlike in other fields, they prevent AI from achieving
human-like performance. AI researchers need to bridge the gap between
statistical NLP and many other disciplines that are necessary for understanding
human language, such as linguistics, commonsense reasoning, and affective
computing (affective computing is the study and development of systems and
devices that can recognise, interpret, process, human affects or emotions).

Coupling top-down and
bottom-up AI

Because of the reasons discussed above, Dr. Cambria
advocates a combination
of symbolic and sub-symbolic AI
. Symbolic models, such as semantic networks,
represent a top-down approach to encode meaning. Sub-symbolic methods, such as
neural networks, represent a bottom-up approach to infer syntactic patterns
from data (syntax is the set of rules, principles, and processes that govern
word order and sentence structure). The top-down approach helps gain
transparency, while data-driven deep learning enables the automatic detection
of patterns.

In a paper titled “SenticNet 5: Discovering Conceptual
Primitives for Sentiment Analysis by Means of Context Embeddings
”, Dr.
Cambria along with his co-authors explores how the two approaches might
complement each other. The paper talks about the use of the bag-of-concepts
model (as opposed to bag-of-words in which a text is represented as a bag or
set of its constituent words) for sentiment analysis. The bag-of-concepts has
the advantage over bag-of-words of being able to deal with multiword
expressions like ‘pretty ugly’ or ‘sad smile’, which would be split up in the
latter model and hence lose their polarity, i.e., their positive or negative
meaning (as in pretty used as an adjective rather than an adverb). And it
avoids the blind use of keywords and word co-occurrence counts.

But now the problem is that the bag-of-concepts model cannot
achieve a comprehensive coverage of meaningful concepts, i.e., a full list of
multiword expressions that actually make sense. Models could be used to extract
concepts from raw data but such approaches are prone to errors due to the richness
and ambiguity of natural language. This is based on the idea that there is a
finite set of mental primitives for affect-bearing concepts and a finite set of
principles of mental combination governing their interaction.

The paper goes on to propose the generalisation of concepts
with related meaning, such as ‘munch toast’ and ‘slurp noodles’, into the
conceptual primitive ‘EAT FOOD’. Sub-symbolic AI could now be used to automatically
discover the conceptual primitives that can better generalise SenticNet’s
commonsense knowledge.

This approach would also help in tackling the symbol
grounding problem. Our understanding of language is grounded in the physical
world, in sensations, in memory. A computer does not learn meaning like that. A
meaning of a word on a page or computer screen is ungrounded. And looking it up
in a dictionary would not help.

This article
explains the problem like this: “If I tried to look up the meaning of a word I
did not understand in a (unilingual) dictionary of a language I did not already
understand, I would just cycle endlessly from one meaningless definition to
another. My search for meaning would be ungrounded. In contrast, the meaning of
the words in my head — the ones I do understand — are
"grounded" (by a means that cognitive neuroscience will eventually
reveal to us). And that grounding of the meanings of the words in my head
mediates between the words on any external page I read (and understand) and the
external objects to which those words refer.”

In the approach presented in the paper, several adjectives
and verbs are defined in function of only one ‘primitive’ item thereby
grounding those meanings in that one primitive. It does not solve the symbol
grounding problem but reduces it.

Image credit: SenticNet (A sketch of SenticNet 5’s graph showing part of the semantic network for the primitive INTACT taken from SenticNet 5: Discovering Conceptual Primitives for Sentiment Analysis by Means of Context Embeddings, by Erik Cambria, Soujanya Poria, Devamanyu
Hazarika, Kenneth Kwok
/ Copyright 2018, Association for the Advancement of Artificial Intelligence)

Current applications

research is being applied in several projects spanning from
fundamental knowledge representation problems to applications of commonsense
reasoning in contexts such as big social data analysis and human-computer

instance, a project in collaboration with Prof. Roy Welsch from MIT
Sloan School of Management focuses on natural
language based financial forecasting
 (NLFF). Markets are driven by
sentiments. Understanding those sentiments from data can be used for predicting
market movements.

is also developing tools that allow patients to easily and efficiently measure
their health related quality of life and improving human-computer interaction (HCI) by developing dialogue systems
with commonsense

project, called PONdER (Public Opinion of Nuclear Energy) aims to
collect, aggregate, and analyse opinions towards nuclear energy in different
languages and across Singapore, Malaysia, Indonesia, Thailand, and Vietnam.
Understanding how the public perceives nuclear energy in the region enables
policymakers to make informed national policies and decisions pertaining to
nuclear energy, as well as shape communication strategies to inform the public
about nuclear energy. 

Dr. Cambria
said that personally he is more interested in the fundamental problems of
AI and sentiment analysis. For example, solving the symbol grounding
problem or building machines that can really understand language (IQ),
emotions (EQ), and culture (CQ).

“Today, we
still don’t have machines that really understand natural
language. Siri does not understand natural language, Watson is an amazing answering
machine but it does not understand language. At SenticNet, we want to go beyond
rule-based and stats-based systems. What we are working on is
not really NLP research anymore; it is natural language understanding.”


Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging problems. A private company, Qlik offers real-time data integration and analytics solutions, powered by Qlik Cloud, to close the gaps between data, insights and action. By transforming data into Active Intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer relationships. Qlik serves more than 38,000 active customers in over 100 countries.


CTC Global Singapore, a premier end-to-end IT solutions provider, is a fully owned subsidiary of ITOCHU Techno-Solutions Corporation (CTC) and ITOCHU Corporation.

Since 1972, CTC has established itself as one of the country’s top IT solutions providers. With 50 years of experience, headed by an experienced management team and staffed by over 200 qualified IT professionals, we support organizations with integrated IT solutions expertise in Autonomous IT, Cyber Security, Digital Transformation, Enterprise Cloud Infrastructure, Workplace Modernization and Professional Services.

Well-known for our strengths in system integration and consultation, CTC Global proves to be the preferred IT outsourcing destination for organizations all over Singapore today.


Planview has one mission: to build the future of connected work. Our solutions enable organizations to connect the business from ideas to impact, empowering companies to accelerate the achievement of what matters most. Planview’s full spectrum of Portfolio Management and Work Management solutions creates an organizational focus on the strategic outcomes that matter and empowers teams to deliver their best work, no matter how they work. The comprehensive Planview platform and enterprise success model enables customers to deliver innovative, competitive products, services, and customer experiences. Headquartered in Austin, Texas, with locations around the world, Planview has more than 1,300 employees supporting 4,500 customers and 2.6 million users worldwide. For more information, visit www.planview.com.


SIRIM is a premier industrial research and technology organisation in Malaysia, wholly-owned by the Minister​ of Finance Incorporated. With over forty years of experience and expertise, SIRIM is mandated as the machinery for research and technology development, and the national champion of quality. SIRIM has always played a major role in the development of the country’s private sector. By tapping into our expertise and knowledge base, we focus on developing new technologies and improvements in the manufacturing, technology and services sectors. We nurture Small Medium Enterprises (SME) growth with solutions for technology penetration and upgrading, making it an ideal technology partner for SMEs.


HashiCorp provides infrastructure automation software for multi-cloud environments, enabling enterprises to unlock a common cloud operating model to provision, secure, connect, and run any application on any infrastructure. HashiCorp tools allow organizations to deliver applications faster by helping enterprises transition from manual processes and ITIL practices to self-service automation and DevOps practices. 


IBM is a leading global hybrid cloud and AI, and business services provider. We help clients in more than 175 countries capitalize on insights from their data, streamline business processes, reduce costs and gain the competitive edge in their industries. Nearly 3,000 government and corporate entities in critical infrastructure areas such as financial services, telecommunications and healthcare rely on IBM’s hybrid cloud platform and Red Hat OpenShift to affect their digital transformations quickly, efficiently and securely. IBM’s breakthrough innovations in AI, quantum computing, industry-specific cloud solutions and business services deliver open and flexible options to our clients. All of this is backed by IBM’s legendary commitment to trust, transparency, responsibility, inclusivity and service.

Send this to a friend