Now in its second year, the Big Data Maturity Survey 2016 (formerly the “Hadoop Maturity Survey”) delivers a wealth of insight into Hadoop as it continues to be the fastest growing force in Big Data.
But how do you separate hype from reality? How do you find out what works and what doesn’t? AtScale has partnered with Cloudera, Hortonworks, MapR, Tableau, Trifacta and Cognizant to identify companies that are working with Big Data or about to. We asked them how they got value from it, what tools they are using and the tactics they used to succeed.
With answers from over 2,550 participants worldwide, this survey is the first global and industry’s largest study on Big Data Maturity. The study focuses on companies that have deployed Big Data, their best practices and their intentions.
The Indian Space Research Organisation’s (ISRO) Polar Satellite Launch Vehicle (PSLV) has launched nine satellites, including eight nanosatellites, into space from the first launch pad at the Satish Dhawan Space Centre in Andhra Pradesh.
The 44-metre-long rocket’s primary payload is the Earth Observation Satellite-6 (EOS-6) or Oceansat-3, a third-generation satellite to monitor oceans. It is a follow up to OceanSat-1 or IRS-P4 and OceanSat-2 launched in 1999 and 2009, respectively. Oceansat-3 will provide data about ocean colour, sea surface temperature, and wind vector data for oceanography, climatology, and meteorological applications.
The Oceansat-3 was placed in the polar orbit at a height of about 740 kilometres above sea level. While it weighs approximately 1,100 kilogrammes, which is only slightly heavier than Oceansat-1, for the first time in this series, it houses three ocean observing sensors. These include an Ocean Colour Monitor (OCM-3), Sea Surface Temperature Monitor (SSTM), and Ku-Band scatterometer (SCAT-3). There is also an ARGOS payload, a press release mentioned.
The OCM-3, with a high signal-to-noise ratio, is expected to improve accuracy in the daily monitoring of phytoplankton. This has a wide range of operational and research applications including fishery resource management, ocean carbon uptake, harmful algal bloom alerts, and climate studies. The SSTM will provide ocean surface temperature, which is a critical ocean parameter to provide various forecasts ranging from fish aggregation to cyclone genesis and movement. Temperature is a key parameter required to monitor the health of the coral reefs, and if needed, to provide coral bleaching alerts. The Ku-Band Pencil beam scatterometre will provide a high-resolution wind vector (speed and direction) at the ocean surface, which will be useful for seafarers, including fishermen and shipping companies. Data regarding temperature and wind is also particularly important for ocean and weather models to improve their forecast accuracies.
ARGOS is a communication payload jointly developed with France and it is used for low-power (energy-efficient) communications including marine robotic floats (Argo floats), fish-tags, drifters, and distress alert devices valuable in search and rescue operations.
The Minister of State (Independent Charge) for Science and Technology, Jitendra Singh, stated that ISRO will continue to maintain the orbit of the satellite and its standard procedures for data reception and archiving. Major operational users of this satellite include Ministry of Earth Sciences (MoEs) institutions such as the Indian National Centre for Ocean Information Services (INCOIS) and the National Centre for Medium Range Weather Forecasting (NCMRWF).
INCOIS has also established a state-of-the-art satellite data reception ground station within its campus with technical support from the National Remote Sensing Centre (ISRO-NRSC). Singh asserted that ocean observations such as this will serve as a solid foundation for India’s blue economy and polar region policies. A representative from MoES noted that the launch of Oceansat-3 is significant as it is the first major ocean satellite launch from India since the initiation of the UN Decade of Ocean Science for Sustainable Development (UNDOSSD, 2021-2030).
The Indian Space Research Organisation is the national space agency of India, headquartered in Bengaluru. It operates under the Department of Space, which is overseen by the country’s Prime Minister.
Astronomers from the California Institute of Technology (Caltech) have completely automated the classification of 1,000 supernovae using a machine-learning (ML) algorithm. The Zwicky Transient Facility, or ZTF, a sky survey instrument located at Caltech’s Palomar Observatory, collected data that the algorithm was then used to analyse.
“We needed a helping hand, and we knew that once we trained our computers to do the job, they would take a big load off our backs,” says Christoffer Fremling, a staff astronomer at Caltech and the mastermind behind the new algorithm tagged as SNIascore.
A year and a half after SNIascore classified its first supernova in April 2021, they are approaching the pleasant milestone of 1,000 supernovae. Every night, ZTF scans the night sky for alterations known as transient events. This covers everything, from asteroids in motion to recently devoured stars by black holes to exploding stars known as supernovae.
ZTF notifies astronomers worldwide of these transient events by sending out hundreds of thousands of alerts each night. Other telescopes are then used by astronomers to monitor and learn more about the nature of the shifting objects. Thousands of supernovae have so far been found thanks to ZTF data.
Members of the ZTF team cannot organise all the data on their own due to the constant flow of data that comes in every night. According to Matthew Graham, project scientist for ZTF and research professor of astronomy at Caltech, “the traditional notion of an astronomer sitting at the observatory and sieving through telescope images carries a lot of romanticism but is drifting away from reality.”
Instead, to help with the searches, the team has created ML algorithms. SNIascore was created to categorise potential supernovae. There are two main categories of supernovae: Type I and Type II. In contrast to Type II supernovae, Type I supernovae are devoid of hydrogen.
When material from a companion star flows onto a white dwarf star, causing a thermonuclear explosion, a Type I supernova is produced. When a massive star collapses due to its own gravity, a Type II supernova happens. Type Ia supernovae, or the “standard candles” in the sky, can be classified by SNIascore. These are dying stars that explode with a steady-state thermonuclear blast.
Astronomers can gauge the universe’s expansion rate thanks to Type Ia supernovae. Fremling and colleagues are currently expanding the algorithm’s capabilities to classify additional types of supernovae soon.
Every night, after ZTF has recorded sky flashes that may be supernovae, it sends the data to the SEDM spectrograph at Palomar, which is in a dome a short distance away (Spectral Energy Distribution Machine).
To determine which supernovae are likely Type Ias, SNIascore collaborates with SEDM. As a result, the ZTF team is working quickly to compile a more trustworthy data set of supernovae that will allow astronomers to conduct additional research and, ultimately, learn more about the physics of the potent stellar explosions.
“SNIascore is incredibly precise. We have observed the performance of the algorithm in the real world after 1,000 supernovae” says Fremling. Since the initial launch in April 2021, they have found no clearly misclassified events, and they are now planning to implement the same algorithm with other observing facilities.
According to Ashish Mahabal, who oversees ZTF’s machine learning initiatives and is the centre’s lead computational and data scientist at Caltech, their work demonstrates how ML applications are maturing in near real-time astronomy.
The SNIascore was created as part of the ZTF’s Bright Transient Survey (BTS), which is currently the most comprehensive supernova survey available to the astronomical community. The entire BTS dataset contains nearly 7000 supernovae, 90 per cent of which were discovered and classified by ZTF while the remaining 10 per cent were contributed by other groups and facilities.
The Victoria University of Wellington’s division of Science, Health, Engineering, Architecture, and Design Innovation (SHEADI) will inaugurate a Centre of Data Science and Artificial Intelligence in the first half of 2023.
According to a statement from the University, the centre will offer areas of expertise in modelling and statistical learning; evolutionary and multi-objective learning; deep learning and transfer learning; image, text, signal, and language processing; scheduling and combinational optimisation; and interpretable AI/ML learning.
These technological themes will be applied across a wide range of areas including primary industry, climate change and environment; health, biology, medical outcomes; security, energy, high-value manufacturing; and social, public policy, and ethics applications. On top of traditional research, the centre will also establish a pipeline of scholarships/internships for Maori students, train early career researchers, and focus on industry, intellectual property, and commercialisation.
The centre will build on the current success and international leadership in this space at the University, the Pro Vice-Chancellor of the division, Ehsan Mesbahi, stated. The institute is continuing to grow its national and international partnerships to create local and global value. The centre will provide a distinctive identity for the growing excellence and innovation in data science and AI research at the University, capabilities which domestic and global partners are increasingly demanding across a vast array of application domains.
In May, the University announced it would offer the first undergraduate major in Artificial Intelligence in the country. It provides students with knowledge of AI concepts, techniques, and tools. They learn how to apply that knowledge to solve problems, combined with programming skills that will enable them to build software tools incorporating AI technology that will help shape the future.
Students studying AI at the University are taught by academics from its internationally renowned AI/ML research group, which is one of the largest in the southern hemisphere. The major is designed to open doors for graduates to opportunities nationally and around the world. There has been an increase in the adoption of AI technologies globally, and a growing demand for people who can apply AI techniques to address a wide range of problems, which the University aims to address.
After completing their degree, graduates will have a wide variety of career options, such as AI scientist, business consultant, AI architect, data analyst, machine learning engineer, and robotic scientist among others. They will also have the option to further their study through the University’s Master of Artificial Intelligence.
OpenGov Asia reported earlier that New Zealand’s Education Technology (EdTech) is set to become one of the country’s key industries. Worth NZ$ 173.6 million in 2020, EdTech software is poised to grow to NZ$ 319.6 million by 2025. At the heart of the digital transformation of education technology has been the pandemic. COVID-19 is seen as the driving force behind the digital transformation of learning, permanently changing the way education is consumed and delivered — right from preschool through post-tertiary education and lifelong learning. The global EdTech market size was valued at US$ 254.8 billion in 2021. Experts believe the market will reach US$ 605.4 billion by 2027.
With the introduction of its Kooha Version 2.0 during the recently held 2022 National Science and Technology Week celebration, the Department of Science and Technology-Advanced Science and Technology Institute (DOST-ASTI) showered photo enthusiasts with helpful tips on interactive smartphone photography.
Kooha is a photo-sharing app derived from the Filipino word “kuha,” which means “to take.” It capitalises on the Philippines’ status as “the selfie capital of the world,” with thousands of photographs shared on various social media platforms every day.
With the help of the camera app Kooha, users may take pictures that go beyond simple snapshots. Multiple sensors are embedded into mobile devices; Kooha uses these sensor data while users snap pictures and embeds them in the image.
Users will be able to quickly learn the location where the photo was shot, the background noise when they shoot a selfie, the network provider’s signal strength, the device battery level, camera settings, environment sensor data, motion sensor, and more. All the photographs captured by the app are shared on Kooha Community. Users’ photos become more than just images when they post them to the community; they become contributions.
When the sensor data from the images is combined with the large pool of sensor data from other users, the data becomes societally important. The data can assist data scientists in generating insights and fresh knowledge that can be used by decision-makers across the country. Kooha is a free app that can be downloaded from Google Play.
According to the DOST-ASTI, Kooha uses the built-in sensors of a mobile device to gather real-time data like sound level, temperature, and humidity and embeds it into a snapshot, making it particularly valuable in research operations across industries thanks to the fresh knowledge it produces.
It added that even more useful Kooha features include the ability to contribute images to the community section, rate shared photos based on “awards” from other users, map the locations of pinned photos, and unlock “badges” by completing specific “achievements.”
As a useful tool application, Kooha reflects the reality that science and the arts may collaborate effectively to produce meaningful results. In addition, the DOST- ASTI’s Quality Management System (QMS) was recertified in accordance with the ISO 9001:2015 standard.
Director of DOST-ASTI Franz A. de Leon stated that the ISO recertification demonstrates the DOST-ASTI’s dedication to continuously enhance its operations and assure successful service delivery – bringing science and technology closer to the people.
He added that their partners and stakeholders can be confident that the institute will constantly offer high-quality products and services because they adhere to the quality policy of developing relevant, timely, and impactful ICT- and electronics-based innovations.
The ISO certificate was the result of the DOST-ASTI management and staff’s collaborative efforts to expand its technologies and ensure the smooth execution of its mandate and functions. Reviewing and improving processes is critical to achieving the agency’s purpose of contributing to the achievement of national development priorities and the growth of Philippine firms through the provision of creative solutions centred on ICT and electronics technology.
This is DOST-ASTI’s second recertification since transitioning to the ISO 9001:2015 standard in 2018. Subject to regular surveillance assessments, the certificate is valid until November 2025.
The Second Minister for Trade and Industry, Tan See Leng, and the Republic of Korea (RoK) Minister for Trade, Dukgeun Ahn, have signed the Korea-Singapore Digital Partnership Agreement (KSDPA).
Under the agreement, the two sides will work to establish digital trade rules and norms to promote interoperability between digital systems. This will enable more seamless cross-border data flows and build a trusted and secure digital environment for businesses and consumers. A government press release wrote that KSDPA will also deepen bilateral cooperation in new emerging areas such as personal data protection, e-payments, artificial intelligence, and source code protection.
The Ministers also signed a memorandum of understanding (MoU) on Implementing the Korea-Singapore Digital Economy Dialogue, which will act as a platform to promote digital economy collaboration between industry players and academic experts from both sides. The MoU is part of bilateral efforts to develop cooperative projects to implement the KSDPA. Key features of the KSDPA include:
Facilitating end-to-end digital trade
Electronic Payments (e-payments): The two sides will adopt transparent and facilitative rules (e.g. encouraging open Application Programming Interfaces (APIs)) to promote secure cross-border e-payments.
Paperless Trading: Singapore and RoK will accept electronic versions of trade administration documents to support the digitalisation and seamless exchange of key commercial documents.
Open Government Data: Both countries will ensure that government data will be publicly available in a machine-readable and open format, with easy-to-use and freely available APIs.
Enabling trusted data flows
Cross-border Data Flows (including for financial services): Businesses in Singapore and RoK will be allowed to transfer information, including those which are generated or held by financial institutions, across borders if the requisite regulations are met and with adequate personal data protection safeguards in place.
Prohibiting Data Localisation: The two nations will establish rules against data localisation requirements so that businesses can choose where their data is stored and processed, and their cloud technology of choice.
Facilitate trust in digital systems and participation in the Digital Economy
Artificial Intelligence (AI): The countries will promote the adoption of AI governance and ethical frameworks that support the trusted, safe, and responsible use of AI-based technologies.
Cryptography: Neither country will require the transfer of or access to private keys and related technologies, as a condition of market access.
Source Code Protection: To ensure software developers can trust the market within which they operate and ensure that source code is protected, neither country will require the transfer of, or access to, source code as a condition of market access. This includes the algorithm expressed in the source code.
Online Consumer Protection: The two sides will adopt laws that guard against fraudulent or deceptive conduct that causes harm to consumers engaged in online commercial activities.
Small and Medium Enterprises Cooperation: Singapore and RoK will promote jobs and growth for SMEs. They will also encourage their participation in platforms that help link them with international suppliers, buyers, and other potential business partners.
Digital Identities: The countries will promote interoperability of digital identity regimes, which can lead to reliable identity verification and the faster processing of applications. This will enable businesses and consumers to navigate the digital economy with ease and security.
The Ministry of Electronics and Information Technology (MietY) is deliberating on various aspects of digital personal data and its protection and has formulated a draft bill titled ‘The Digital Personal Data Protection Bill 2022’. The Ministry has invited feedback from the public on the draft Bill. The submissions will not be disclosed and held in a fiduciary capacity, to enable people submitting feedback to provide the same freely. The government has said no public disclosure of the submissions will be made.
According to a press release, the purpose of the draft Bill is to provide for the processing of digital personal data in a manner that recognises both the right of individuals to protect their personal data and the need to process personal data for lawful purposes and matters connected therewith or incidental thereto. The draft Bill employs plain and simple language to facilitate ease of understanding and is available on the Ministry’s website along with an explanatory note that provides a brief overview of its provisions.
There are presently over 760 million active Internet users and over the next coming years, this is expected to touch 1.2 billion. There is an increasing need to regulate content and data collection on the Internet.
The Digital Personal Data Protection Bill frames out the rights and duties of the citizen (Digital Nagrik) on one hand and the obligations to use collected data lawfully of the Data Fiduciary on the other. The bill is based on seven principles around the Data Economy.
The first principle is that usage of personal data by organisations must be done in a manner that is lawful, fair, and transparent. The second principle of purpose limitation is that the personal data is used for the purposes for which it was collected.
The third principle of data minimisation is that only those items of personal data required for attaining a specific purpose must be collected. The fourth principle of the accuracy of personal data is that a reasonable effort must be made to ensure that the personal data of the individual is accurate and kept up to date. The fifth principle of storage limitation is that personal data is not stored perpetually by default. The storage should be limited to such duration as is necessary for the stated purpose for which personal data was collected.
The sixth principle is that reasonable safeguards are taken to ensure that there is no unauthorised collection or processing of personal data. This is intended to prevent a personal data breach. The seventh principle is that the person who decides the purpose and means of the processing of personal data should be accountable for such processing.
The Bill will establish a comprehensive legal framework governing digital personal data protection in the country. The Bill provides for the processing of digital personal data in a manner that recognises the right of individuals to protect their personal data, societal rights, and the need to process personal data for lawful purposes.
To enhance digital-based governance, the government is getting ready to construct four National Data Centers (PDN). Hence, the implementation of data-driven policies is encouraged using digital government ideas and initiatives.
According to Semuel Abrijani Pangerapan, Director General of Informatics Applications at the Ministry of Communication and Informatics, PDN is a strategic move by the government to advance effectiveness, efficiency, the sovereignty of state data, and the consolidation of national data within the One Data Indonesia framework.
He said during the “Groundbreaking Ceremony for the Development of the National Data Centre (Strengthening of E-Government), in Cikarang, West Java, “The PDN is one of the instructions of the President of the Republic of Indonesia in order to expedite digital transformation within government agencies.
The National Data Centre is expected to result in smart and contemporary governance because the installed technology in the PDN ecosystem comprises cloud computing, big data analytics and artificial intelligence, blockchain, and the metaverse.
Director General Semuel noted that the groundbreaking represented the introduction of the Bekasi Regency PDN development project to the central government, local government, the private sector, and the community.
The establishment of PDN is also one of the primary factors boosting Indonesia’s digital innovation. Especially in the context of effectiveness, efficiency, consolidation of national data, security, and sovereignty of state information, as well as encouraging the implementation of One Data Indonesia.
The Ministry has designed four PDN development locations, including the Deltamas Industrial Estate (Jabodetabek) region, the Nongsa Digital Park (Batam) area, the new National Capital City (IKN) in East Kalimantan, and Labuan Bajo, East Nusa Tenggara.
The Ministry indicated that the initial PDN was constructed in Cikarang, West Java, namely in the Deltamas Industrial Estate region, around forty kilometres from Jakarta. The second PDN will be constructed in the Nongsa neighbourhood of Batam City, Province of the Riau Archipelago. A fibre optic network capable of connecting the area and its environs to western Indonesia already exists at this site.
The decision to locate a data centre in Batam is based on the comprehensiveness of the supporting infrastructure, which includes fibre optic infrastructure, electricity supply, water, and direct paths to the global internet backbone. IKN and Labuan Bajo, East Nusa Tenggara are slated to house the second PDN development location.
Meanwhile, Usman Kansong, Director General of Information and Public Communication at the Ministry of Communication and Informatics declared that the government intends to use metaverse technology to promote virtual tourism at the Borobudur Temple.
To safeguard the tourist attraction, Director General Usman claims that the discussion on the use of this metaverse technology began concurrently with the implementation of a ban or restriction on general visitors’ access to the Borobudur Temple edifice. According to the Ministry, using this technology allows tourists who visit the Borobudur Temple can still climb this ancient structure without being there with the help of the metaverse.
Led by the Minister of Communication and Informatics Johnny G. Plate, the Ministry is optimistic that the implementation of this cutting-edge technology will be realised. The government would also offer help and training for waste management as well as for distributing local handicrafts in the vicinity of the temple and growing tourist settlements. This tourist system has the potential to offset the pandemic’s significant economic impact on the travel and tourism industry.