Close this search box.

We are creating some awesome events for you. Kindly bear with us.

Big data driven insights into tomorrow’s marketing environments

Big data driven insights into tomorrow’s marketing environments

The business of Experian, a global leader
in credit reporting and marketing services with annual revenues exceeding US$4.3
(for 2017), is all about data.

Experian has four main business units:
Credit Information Services, Decision Analytics, Business Information Services,
and Marketing Services. Experian Marketing Services (EMS) helps marketers
connect with customers through relevant communications across a variety of
channels, driven by advanced analytics on an extensive database of geographic,
demographic, and lifestyle data. EMS has built its business on the effective
collection, analysis, and use of data.

of records

The company has always handled large
amounts of data, billions and quadrillions of records, on who consumers are,
how they’re connected, how they interact. With today’s proliferation of digital
channels and information of social media likes, web interactions and email
responses, older systems no longer have the capacity to deal with the data

In the past, there was no requirement to
provide data in real-time. Experian sent customer database updates to clients
once a month for campaign adjustments, allowing Experian to process large
volumes of data through a number of diverse platforms, which were mostly mainframe

That’s changing. Today’s consumers leave a
digital trail of behaviors and preferences for marketers to leverage so they
can enhance the customer experience. Experian’s clients, which includes many of
the top retail companies in the world, are asking for more frequent updates on
consumers’ latest purchasing behaviors, online browsing patterns and social
media activity so they can respond in real time. They are increasingly looking
for a single, integrated view of their customer.

infrastructure for real-time reporting

Meeting the need for immediacy of
information and customisation of data in real time for clients, would require a
technological infrastructure that can accommodate rapid processing, large-scale
storage, and flexible analysis of multi-structured data. Experian’s mainframes
were hitting their limits in terms of performance, flexibility and scalability.

EMS set an internal goal to process more
than 100 million records of data per hour, translating to 28,000 records per

The team decided to look for new
architectures that could handle the new volumes of data. About 30 criteria were
identified for the new platform, ranging from depth and breadth of offering to
support capabilities to price to unique distribution features. Two criteria
were prioritized: Both batch and real-time data processing capabilities; and scalability
to accommodate large and growing data volumes.

The North America Experian Marketing
Services group led the evaluation of NoSQL technologies within Experian. Hadoop
and HBase quickly surfaced as a natural fit for Experian’s needs. EMS engineers
downloaded raw Apache Hadoop.

They saw certain gaps that could be filled
by a commercial distribution. EMS evaluated several distributions and selected
Cloudera to meet EMS’ enterprise-level Hadoop needs, such as meeting client
SLAs (service level agreements) and having 24×7 reliability.

Experian invested in Cloudera Enterprise,
which is comprised of three things: Cloudera’s open source Hadoop stack (CDH),
a management toolkit (Cloudera Manager), and expert technical support.

A production version of Experian’s
Cross-Channel Identity Resolution (CCIR) engine was launched. CCIR is a linkage
engine that is used to keep a persistent repository of client touch points.
CCIR runs on HBase,
a high-performance, distributed data store that integrates with Cloudera's
platform to deliver a secure and easy-to-manage NoSQL database.

EMS’ HBase system spanned five billion rows
of data, as of 2017, and the number is expected to grow tenfold in the near
future. HBase offers a shared architecture that is distributed, fault tolerant,
and optimised for storage. In addition, HBase enables both batch and real-time
data processing.

Experian feeds data into the CDH-powered
CCIR engine using custom extract, transform, load (ETL) scripts from in-house
mainframes and relational databases including IBM DB2, Oracle, SQL Server, and
Sybase IQ.

performance accelerated by 50x

The new platform is delivering operational
efficiency to Experian by accelerating processing performance by 50x, at a
fraction of the cost of the legacy environment. The new system can process 100
million records per hour compared to 50 million matches per day earlier.

Cloudera Enterprise allows Experian to get
maximum operational efficiency out of their Hadoop clusters. Due to a wide
variation in use cases for customers, the team had to do a lot of tweaking on
the platform to get the performance we need. Cloudera Enterprise provides the
ability to store these store different configuration settings and version those

McCullough added, “Not only has Cloudera
Manager simplified our process, but it’s made it possible at all. Without a
Linux background, I would not have been able to deploy Hadoop across a cluster
and configure it and have anything up and running in nearly the timeframe that
we had.”

Furthermore, Cloudera Manager enabled the
deployment and configuration of Hadoop across a cluster in the timeframe
Experian had. Cloudera Manager monitors services running on cluster and reports
when servers are unhealthy, services have stopped, and/or nodes are bad. It automates
distribution across the cluster, monitors CPU usage across various applications
and data storage availability and provides a single portal to see into all
cluster details.

The deployment allowed Experian to process
orders of magnitude more information through its systems. Experian’s platform is
the first data management platform of its kind that accepts data, links
information together across an entire marketing ecosystem, and puts it into a
usable format for an enhanced customer experience. These data processing capabilities
combined with Experian’s expertise in bringing together data assets provided
new insights into tomorrow’s marketing environments.


In January 2017, it was announced
that Experian was integrating Cloudera Enterprise onto
its cloud environment for its Credit Information Services, Decision Analytics
and Business Information Services business lines, with the aim of improved credit data processing speeds for
clients. Thus, Cloudera continues to transform the way Experian provides
consumer and business credit data to its clients.

All content from customer
success story
 and case


Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging problems. A private company, Qlik offers real-time data integration and analytics solutions, powered by Qlik Cloud, to close the gaps between data, insights and action. By transforming data into Active Intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer relationships. Qlik serves more than 38,000 active customers in over 100 countries.


As a Titanium Black Partner of Dell Technologies, CTC Global Singapore boasts unparalleled access to resources.

Established in 1972, we bring 52 years of experience to the table, solidifying our position as a leading IT solutions provider in Singapore. With over 300 qualified IT professionals, we are dedicated to delivering integrated solutions that empower your organization in key areas such as Automation & AI, Cyber Security, App Modernization & Data Analytics, Enterprise Cloud Infrastructure, Workplace Modernization and Professional Services.

Renowned for our consulting expertise and delivering expert IT solutions, CTC Global Singapore has become the preferred IT outsourcing partner for businesses across Singapore.


Planview has one mission: to build the future of connected work. Our solutions enable organizations to connect the business from ideas to impact, empowering companies to accelerate the achievement of what matters most. Planview’s full spectrum of Portfolio Management and Work Management solutions creates an organizational focus on the strategic outcomes that matter and empowers teams to deliver their best work, no matter how they work. The comprehensive Planview platform and enterprise success model enables customers to deliver innovative, competitive products, services, and customer experiences. Headquartered in Austin, Texas, with locations around the world, Planview has more than 1,300 employees supporting 4,500 customers and 2.6 million users worldwide. For more information, visit


SIRIM is a premier industrial research and technology organisation in Malaysia, wholly-owned by the Minister​ of Finance Incorporated. With over forty years of experience and expertise, SIRIM is mandated as the machinery for research and technology development, and the national champion of quality. SIRIM has always played a major role in the development of the country’s private sector. By tapping into our expertise and knowledge base, we focus on developing new technologies and improvements in the manufacturing, technology and services sectors. We nurture Small Medium Enterprises (SME) growth with solutions for technology penetration and upgrading, making it an ideal technology partner for SMEs.


HashiCorp provides infrastructure automation software for multi-cloud environments, enabling enterprises to unlock a common cloud operating model to provision, secure, connect, and run any application on any infrastructure. HashiCorp tools allow organizations to deliver applications faster by helping enterprises transition from manual processes and ITIL practices to self-service automation and DevOps practices. 


IBM is a leading global hybrid cloud and AI, and business services provider. We help clients in more than 175 countries capitalize on insights from their data, streamline business processes, reduce costs and gain the competitive edge in their industries. Nearly 3,000 government and corporate entities in critical infrastructure areas such as financial services, telecommunications and healthcare rely on IBM’s hybrid cloud platform and Red Hat OpenShift to affect their digital transformations quickly, efficiently and securely. IBM’s breakthrough innovations in AI, quantum computing, industry-specific cloud solutions and business services deliver open and flexible options to our clients. All of this is backed by IBM’s legendary commitment to trust, transparency, responsibility, inclusivity and service.