Close this search box.

We are creating some awesome events for you. Kindly bear with us.

AI Helps Solve Research Reproducibility Concerns

Scientists are becoming increasingly concerned that a lack of reproducibility in research could lead to flaws that hinder scientific productivity and erode public trust in science, among other things. A group of researchers now claims that constructing a prediction market, in which Artificially Intelligent (AI) agents make predictions on hypothetical replication studies, might lead to an explainable, scalable method of estimating confidence in published scholarly work.

Experiments and research that are replicated, which is an important step in the scientific process, assist establish confidence in the results and showing whether they can be generalised across contexts. Scientists increasingly lack the resources for effective replication efforts as studies have become more sophisticated, costly, and time-consuming – what they now refer to as the replication problem.

As scientists, we want to do work, and we want to know that our work is good. Our approach to helping address the replication crisis is to use AI to help predict whether a finding would replicate if repeated and why.

− Sarah Rajtmajer, Assistant Professor, Information Sciences and Technology, Penn State

Based on trading patterns and elements of the papers and claims that drove the bots’ behaviour, a bot-based approach scales and offers some explainability of its findings. Bots are trained to recognise key features in academic research papers, such as authors and institutions, statistics, and linguistic cues, downstream mentions, and similar studies in the literature, and then assess the confidence that the study is robust enough to replicate in future studies, according to the team’s approach. The bot then bids on its level of confidence, just like a person betting on the outcomes of a sporting event. The outcomes of the AI-powered bots are compared to human predictions.

While human-based prediction markets are well-known and have been successfully applied in a variety of industries, prediction markets are innovative when it comes to analysing research data. Prediction markets are a skill that humans have mastered. They are, however, using bots for our market, which is unique.

The system offered confidence scores for about 68 of the 192 publications that were subsequently duplicated, or ground truth replication studies, according to the researchers, who presented their findings at a recent meeting of the Association for the Advancement of Artificial Intelligence. The accuracy on the set of papers was roughly 90%.

Because humans are better at predicting research reproducibility than bots, the researchers propose that combining the two approaches — humans and bots working together — could give the best of both worlds: a system with more accuracy while still being scalable.

They train the bots in front of human traders on a regular basis, then deploy them offline when we need a speedy result or when replication efforts at scale are required. They can also construct bot markets that take advantage of such intangible human wisdom.

As reported by OpenGov Asia, according to experts at the University of California, Irvine, producing smarter, more accurate systems necessitates a mixed human-machine approach. They describe a new mathematical model that can increase performance by integrating human and computational predictions and confidence scores in a paper published this month in Proceedings of the National Academy of Sciences.

To put the framework to the test, researchers ran an image classification experiment in which human volunteers and computer algorithms competed to accurately identify altered images of animals and everyday objects like chairs, bottles, bicycles, and trucks. The human participants assessed their level of confidence in each image’s correctness as low, medium, or high, while the machine classifier generated a continuous score. Across photos, the results revealed significant variations in confidence between people and AI computers.


Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging problems. A private company, Qlik offers real-time data integration and analytics solutions, powered by Qlik Cloud, to close the gaps between data, insights and action. By transforming data into Active Intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer relationships. Qlik serves more than 38,000 active customers in over 100 countries.


CTC Global Singapore, a premier end-to-end IT solutions provider, is a fully owned subsidiary of ITOCHU Techno-Solutions Corporation (CTC) and ITOCHU Corporation.

Since 1972, CTC has established itself as one of the country’s top IT solutions providers. With 50 years of experience, headed by an experienced management team and staffed by over 200 qualified IT professionals, we support organizations with integrated IT solutions expertise in Autonomous IT, Cyber Security, Digital Transformation, Enterprise Cloud Infrastructure, Workplace Modernization and Professional Services.

Well-known for our strengths in system integration and consultation, CTC Global proves to be the preferred IT outsourcing destination for organizations all over Singapore today.


Planview has one mission: to build the future of connected work. Our solutions enable organizations to connect the business from ideas to impact, empowering companies to accelerate the achievement of what matters most. Planview’s full spectrum of Portfolio Management and Work Management solutions creates an organizational focus on the strategic outcomes that matter and empowers teams to deliver their best work, no matter how they work. The comprehensive Planview platform and enterprise success model enables customers to deliver innovative, competitive products, services, and customer experiences. Headquartered in Austin, Texas, with locations around the world, Planview has more than 1,300 employees supporting 4,500 customers and 2.6 million users worldwide. For more information, visit


SIRIM is a premier industrial research and technology organisation in Malaysia, wholly-owned by the Minister​ of Finance Incorporated. With over forty years of experience and expertise, SIRIM is mandated as the machinery for research and technology development, and the national champion of quality. SIRIM has always played a major role in the development of the country’s private sector. By tapping into our expertise and knowledge base, we focus on developing new technologies and improvements in the manufacturing, technology and services sectors. We nurture Small Medium Enterprises (SME) growth with solutions for technology penetration and upgrading, making it an ideal technology partner for SMEs.


HashiCorp provides infrastructure automation software for multi-cloud environments, enabling enterprises to unlock a common cloud operating model to provision, secure, connect, and run any application on any infrastructure. HashiCorp tools allow organizations to deliver applications faster by helping enterprises transition from manual processes and ITIL practices to self-service automation and DevOps practices. 


IBM is a leading global hybrid cloud and AI, and business services provider. We help clients in more than 175 countries capitalize on insights from their data, streamline business processes, reduce costs and gain the competitive edge in their industries. Nearly 3,000 government and corporate entities in critical infrastructure areas such as financial services, telecommunications and healthcare rely on IBM’s hybrid cloud platform and Red Hat OpenShift to affect their digital transformations quickly, efficiently and securely. IBM’s breakthrough innovations in AI, quantum computing, industry-specific cloud solutions and business services deliver open and flexible options to our clients. All of this is backed by IBM’s legendary commitment to trust, transparency, responsibility, inclusivity and service.