Using Talend Big Data Platform to build a cloud data lake on the Microsoft Azure Cloud Platform, learn how a company was able to integrate and cleanse data from multiple sources and deliver real-time insights.
The process of extracting value from your data, especially in industries such as financial services, can be a challenge. There’s the sheer volume of data, which is on pace to double every two years.
Data comes from multiple sources, can be spread across your infrastructure or stuck in silos, and can be hard to integrate when it’s distributed across cloud and on-premises platforms. Increasing and changing regulations can hinder the process, too. As well, your customers demand self-service access in real time, and your teams want more insights, and faster.
In this eBook, learn why a cloud data lake you trust can help you scale the process of integrating data, assessing its quality, tracking lineage, and extracting value from data—with confidence.
Indonesia’s Ministry of Communication and Informatics conducted a familiarisation workshop for health workers and non-medical staff about Electronic Medical Records (RME). According to Health Ministry regulations, health facilities in Indonesia are required to use RME. Indonesia Social Security Administrator (Health BPJS) now offers an integrated RME computing system. Some digitally based hospitals have used the system for RME.
“Through this workshop and seminar, the Ministry of Communication and Informatics introduces Electronic Medical Records (RME) so that participants can understand what RME is and how the supporting technology works,” said I Nyoman Adhiarna, Director of Digital Economy at the Ministry of Communication and Information.
Essentially, health facilities must use the electronic record for patient registration activities, clinical information filling, storage and transfer of medical records, ownership and contents of patient medical records, security, and data protection.
Meanwhile, Setiaji, the Chief of the Ministry of Health’s Digital Transformation Officer (CTO), emphasises the importance of information technology as the backbone of the ongoing transformation of the national health system. “One of the major agenda items is the implementation of electronic medical records, which has begun with the launch of the SATU SEHAT platform.” “This platform connects sixty thousand health services across Indonesia,” he said.
The seminar also covered several topics, such as the role of medical recorders and health information (PMIK) in the success of RME adoption, RME integration with cyber security, change management in RME adoption, and RME adoption and its implications for hospital services.
Director Nyoman acknowledged challenges in implementing RME in hospitals, such as internet network connectivity and cyber security. However, adequate digital infrastructure would make digital transformation in health care more accessible. As a result, the Ministry of Communication and Informatics is currently focusing on developing digital infrastructure in remote areas, known as the 3T acronyms for frontier, outermost, and lagging.
A multifunctional satellite, Satellite Indonesia Raya (SATRIA) 1, will provide support for remote internet access. The broadband satellite will launch in 2023 to reach 150,000 public service points.
As the health sector becomes increasingly digitalised, cyber and data security has become a significant concern. According to Director Nyoman, all PSEs, both public and private, that manage personal data are urged to pay close attention to the feasibility and dependability of personal processing data, particularly those related to technology, governance, and human resources.
He is also concerned about the role of the Data Protection Officer (DPO) following the implementation of the government’s data protection law. Each electronic system operator must legally have a person in charge of data protection.
“Later,” he said, “a Personal Data Protection Agency will be formed, which will most likely be under the Ministry of Communication and Informatics.”
Director Nyoman also emphasised the Ministry of Communication and Informatics’ role in assisting the Ministry of Health in becoming the leading sector in the health sector to accelerate digital transformation in the health sector.
The Ministry of Communication and Information welcomed the One Healthy Indonesia Health Service (IHS) platform, launched in July 2021 by the Ministry of Health as a digital transformation programme based on an integrated and standardised health data system.
Australia’s national science agency, CSIRO, recently revealed details of an AU$15 million project to develop a national soil information system, aimed at improving the sustainable management of one of the nation’s most precious assets.
Supporting the National Soil Strategy, and funded by the Australian Government’s Department of Agriculture, Fisheries and Forestry, the Australian National Soil Information System (ANSIS) project is a collaboration between the government, research organisations, industry, the private sector and the community.
Using innovative processes and technologies, ANSIS will allow improved sharing of nationally consistent soil data and information through online access for users. This will help Australians to better understand their nation’s diverse range of soils and make better decisions about managing our important soil resources. Currently, soil data is collected using different methods, by different organisations, and at a range of depths in the soil. This makes it hard to access, compare and use data from diverse sources.
The Project Lead at CSIRO stated that improving access to the best soil data and information can help promote digital agriculture innovation and is key to sustainably managing Australia’s soils. By using ANSIS, farmers and agricultural advisors will have access to more soil data and be better placed to more sustainably manage the soil on which they rely.
Soil is vital to agricultural production and natural environments, as well as health and well-being. This information system will help everyone care for this important natural resource. Productive, healthy, and resilient soil means more economic, environmental, and social benefits to Australia. Monitoring soil also helps scientific understanding of how the natural world is changing.
This work will provide insight into biodiversity, water resources, landscapes and coastlines, fauna, climate, and geology. By harmonising Australia’s soil data, we can make it accessible across many fields of science and exploration. The project is being delivered under the Federal Government’s National Soil Strategy, which is about prioritising soil health, empowering soil innovation and stewards, and strengthening soil knowledge and capability. The new ANSIS system will be available for use in 2023.
ANSIS will provide improved access to nationally consistent soil data and information needed to help sustainably manage Australian soil. ANSIS will provide:
- More soil data
- More data sets are available that in other soil systems
- Enables more certainty in products developed
- Opportunity to develop new products
- Improved access
- Multiple data sets are now discoverable and accessible
- National coverage
- Most up-to-date data available
- Efficient provision
- Organised and standardised data for immediate use
- Can feed into many users’ requirements
- Consistent delivery
- Substantial reduction in time to prepare information products
- Trusted location
- Certainty that data is from an authoritative source, verified and satisfies standards.
Indonesia has great ambitions for its digital economy and has deployed strategies to achieve its ambitions with a goal to reach USD315 billion by 2030. The 2021-2024 Indonesia Digital Roadmap is set on 4 pillars, namely digital infrastructure, digital government, digital economy and digital society.
As part of its strategy, the government is promoting four important digital skills to accelerate its digital economy. The government believes that the future demand for digital skills will be focused on four areas Artificial Intelligence, Bitcoin, Cloud Computing, and Data Analytics (ABCD). The ABCD skills are projected to help the national economy hit its US$315 billion by 2030 target.
Therefore, the Indonesian government is encouraging young people to start businesses through a variety of free programs such as Beta School, 1,000 Startup Movement, Startup Studio, HUB.ID and IGDX.
“Aside from university disciplines, the ABCD is becoming increasingly important for everyone. I believe that all young people require ABCD,” stated Dedy Permadi, Expert Staff of the Minister of Communication and Informatics, in a discussion forum.
Mastering ABCD technical hard skills apart, Indonesian digital talents are also expected to be proficient in non-technical or soft skills known as the 4C’s, which are Complex Problem Solving, Critical Thinking, Creativity and Communication.
The Director of SDPPI Kominfo, Ismail, expressed his hope that the young generation in Indonesia would capture the golden opportunity for digitalisation. Digitalisation will transform Indonesia from a consumer country to a prominent player in the new normal.
The government recognises the importance of good infrastructure support in boosting the digital economy. As a result, the government is working to ensure an equitable distribution of internet connection networks across Indonesia, particularly in frontier, remote, and underdeveloped (3T) areas.
According to Ismail, the development of ICT infrastructure must meet three criteria: broad coverage, the deployment of a fibre-optic cable network on the backbone, and affordability, which means that the price is reasonable for the community.
Private operators focus on developing infrastructure in high-demand urban areas and, as a result, the digital divide between cities and towns has grown wider. Consequently, the government is beginning to develop 3T telecommunications in rural, underserved areas.
“We cannot rely solely on private-sector investment. To speed up and accelerate digital transformation, the government must invest in infrastructure,” Ismail said emphatically.
The Ministry of Communication and Information Agency and Telecommunications and Information Accessibility (BAKTI) have also worked to improve and expand internet access for public services throughout Indonesia. BAKTI is working with telecommunications companies to build Base Transceiver Stations (BTS) in remote areas of Indonesia.
“We hope to finish building BTS in all remote areas by 2023 and connect them to the 4G network,” Deddy stated.
Indonesia is a vast archipelagic country. So, relying solely on fibre optic cable networks will make it difficult to provide connectivity. As a result, the government is combining the fibre optic cable network constructed with the 150 Gbps SATRIA 1 satellite.
This multifunctional satellite can provide internet access to 150,000 public service locations in Indonesia, including educational institutions, local governments, defence and security administration, and health facilities. This satellite is scheduled to launch in 2023.
The government has begun construction of the first National Data Centre in the Delta Mas Region, GIIC, Cikarang District, Bekasi Regency, West Java Province, in connection with its digital strategy. It will then gradually expand data centres in Nongsa Digital Park in Batam, Riau Archipelago, the new National Capital City (IKN) in Balikpapan, East Kalimantan, and Labuan Bajo, East Nusa Tenggara.
The creation of this government data centre is intended to promote efficiency, effectiveness, state data sovereignty, and national data consolidation as part of the One Data Indonesia initiative. “This (data centre) is critical because government data management is critical to developing society’s transformation into a digital society,” Deddy said.
The Indonesian government disclosed four potential uses of Big Data and AI to improve its e-government programmes. These two technologies, they feel, have the potential to support disaster identification and preventive action, prevention of illegal activities and cyber-attacks and increase workforce effectiveness.
The Director General of Informatics Applications, Semuel A. Pangerapan, explained several scenarios for Big Data. According to him, the government can use Big Data to improve critical event management and the quality of the response by identifying problem points through Big Data Analytics. For example, the agencies can be better prepared to prevent and mitigate natural disasters such as drought, epidemics or massive accidents occur.
In addition, Big Data can also enhance the government’s ability to prevent money laundering and fraud through better surveillance to detect such illegal activities.
Furthermore, Big Data significantly reduces the possibility of cyber-attacks. Cyber-attacks can come from external parties, data leaks or internally for a variety of reasons. An analysis of patterns and unusual activities can help in preventing or managing such cyber issues.
Big Data and analytics can contribute to workforce effectiveness by increasing monitoring. In addition, it can be used for policy design, decision-making and gaining insights.
Semuel stressed the importance of data analysis after collecting all data in the right fashion. Data is only valuable if it is collected correctly and then analysed – data will only provide benefits if processed in the right way. “In its implementation, AI helps analyse existing Big Data, providing data understanding or insight to help make decisions,” he explained.
Another advantage of AI is the ability to speed up new implementation services and corrections in real-time. At the evaluation stage, AI can also provide suggestions for adjustments and improvements to subsequent policies.
Currently, the encourages the improvement of the quality of Big Data and AI innovation through the development of e-government. The Indonesian government is also open to third parties to accelerate Big Data and AI use.
E-government has made progress in recent years and received appreciation from the United Nations in 2020. The UN said that Indonesia’s e-government development index rose to rank 88 from previously ranked 107 in 2018. Indonesia’s e-participation index has also increased from rank 92 in 2018 to 57 in 2022.
“The two rankings show an increase in the quality of Indonesia’s e-government and the level of community activity in using e-government services,” said Semuel.
However, the government faced challenges in implementing these two technologies. Overlapping and data replication is one of the main problems. “Regulatory obstacles in the procurement of government Big Data infrastructure also need to be overcome. Then compliance with international standards for the national Big Data ecosystem is also still the government’s homework.”
To optimise AI use, Semuel emphasised the need for a skilled workforce, regulations governing the ethics of using AI, infrastructure, and industrial and public sector adoption of AI innovations.
The government is implementing several solutions to overcome challenges. First, they have provided suitable facilities in the form of National Data Centres (NDCs) in four separate locations. The NDCs will accommodate Government Cloud and contain national data across sectors.
Optimisation of data centre utilisation needs to be supported by staff with qualified expertise. For this reason, the government is holding digital skills training on AI and Big Data through the Digital Talent Scholarship (DTS) and Digital Leadership Academy (DLA) programs.
Apart from facilities and upskilling, Indonesia is looking to develop a business ecosystem that utilises AI and Big Data. Support for this comes from the National Movement of 1000 Digital Startups, Startup Studio Indonesia (SSI) and HUB.ID.
The Cyberspace Administration of China (CAC) announced a new certification for personal information protection and implementation. The office has decided to implement such certification to enhance its information protection capabilities and to promote the rational processing of personal information.
The certification implementation follows the Personal Information Protection Certification Implementation Rules. The implementation rules clarify that personal information processors must comply with the requirements of GB/T 35273 Information Security Technology Personal Information Security Specifications. The rules outline requirements for on-site audits, the evaluation and approval of certification results, post-certification supervision and certification time limits.
Organisations engaged in personal information protection certification work need approvals to carry out activities. The regulation applies to every personal information processor that carries out private information collection, storage, use, processing, transmission, provision, disclosure, deletion and cross-border processing activities.
The State Administration for Market Regulation and the State Internet Information Office decided to implement personal Information protection certification. The step is relevant to provisions of the Personal Information Protection Law of the People’s Republic of China (‘PIPL’). The body requires the Specifications for Security Certification of Cross-Border Processing of Personal Information for cross-border personal information processing.
The latest versions of the standards include technical verification, on-site audit, and post-certification supervision. In addition, the certification body shall clarify the requirements for certification entrustment materials, including but not limited to the basic materials of the certification client, the certification power of attorney, and relevant certification documents.
To get certified, an organisation must submit certification entrustment materials according to the certification body’s requirements and the certification body shall give timely feedback on whether it is accepted after reviewing the materials.
The materials are then used for determining the certification plan, including the type and quantity of personal information, the scope of personal information processing activities, information on technical verification institutions, etc., before notifying the organisation seeking certification.
The CAC stated certification is valid for three years. An organisation must submit a certification commission within six months before the expiration of the validity period. The certification body shall adopt the method of post-certification supervision and reissue new certificates to those that meet the certification requirements.
Violations, cheating, and other behaviours that seriously affect the implementation of the certification on the certification client or personal information processor will cancel the certificate. Therefore, certification bodies shall adopt appropriate methods to implement post-certification supervision to ensure that certified personal information processors continue to meet certification requirements. The certification body comprehensively evaluates the post-certification surveillance conclusions and other relevant information. If the evaluation is passed, the certification certificate can continue to be maintained.
The organisation shall actively cooperate with the certification activities. During the validity period of the certification certificate. If the name and registered address of the certified personal information processor, or the certification requirements, certification scope, etc., change, the certification principal shall submit a change entrustment to the certification body.
When changes happen, the certification body must evaluate the change in entrustment materials. The result will determine whether the body can approve the change. If technical verification or on-site audit is required, the body shall conduct technical and on-site audits before the change is approved.
When a certified personal information processor no longer meets the certification requirements, the certification body will promptly suspend or revoke the certification certificate. The certification principal can apply for the suspension and cancellation of the certification certificate within the validity period of the certification certificate.
Caltech engineers collaborated with the University of Southampton in England to design an ultrahigh-speed data transfer chip. The chip integrates both an electronics chip and a photonics chip which uses light to transfer data. It took four years to complete, from the initial idea to the final test in the lab.
“As the world becomes increasingly connected, and every device generates more data, it is exciting to show that we can achieve such high data rates while burning a fraction of power compared to the traditional techniques. We had to optimise the entire system all at the same time, which enabled achieving a superior power efficiency,” said Azita Emami, the Andrew and Peggy Cherng Professor of Electrical Engineering and Medical Engineering, Executive Officer for Electrical Engineering and senior author of the paper.
The research paper is titled “A 100Gb/s PAM4 Optical Transmitter in A 3D-Integrated SiPh-CMOS Platform Using Segmented MOSCAP Modulators.” Rockley Photonics and the U.K. Engineering and Physical Sciences Research Council funded this research.
The need for high processing power and transmission creates the inevitable excess heat. Heat is the enemy of the speed and the amount of data a computer device can manage. It happens not just for personal computers or laptops but also for data centres.
While a laptop may heat up while when in use, servers in data centres also heat up as they work – but at a much grander scale. Therefore, managing heat in the data centre is essential. The less heat, the more computing power is generated and the greater the volume of information it can handle.
Hence, engineers tried to find a way to increase the processing speed while keeping the heat low. The solution was to design and co-optimise an electronics chip and a photonics chip. The chip is innovative because it integrates an electronic circuit essential for data processing, combined with a photonics chip which is the most efficient piece for data transmission.
The Caltech/Southampton integrated chip can transmit 100 gigabits of data per second! Moreover, the integrated chip generates minimal heat, producing just 2.4 pico-Joules per transmitted bit. The result increases the electro-optical power efficiency by 3.6 times compared to the current technology.
Handling Next-level Computing
In the future, data centres will manage very high volumes of data compared to today. The new design integrated chip will answer a continuous demand for increasing data communication speed in data centres and high-performance computers.
“As the computing power of the chips scale, the communication speed can become the bottleneck, especially under stringent energy constraints,” Emami explained.
The high-demand data transmission and processing from a data-demanding task, such as a video call, streaming a movie, or playing an online video game, need high processing power in the data centre.
“There are more than 2,700 data centres in the U.S. and more than 8,000 worldwide, with towers of servers stacked on top of each other to manage the load of thousands of terabytes of data going in and out every second,” says a Caltech graduate student Arian Hashemi Talkhooncheh (MS ’16), lead author of a paper describing the two-chip innovation that was published in the IEEE Journal of Solid-State Circuits.
The Indian Space Research Organisation’s (ISRO) Polar Satellite Launch Vehicle (PSLV) has launched nine satellites, including eight nanosatellites, into space from the first launch pad at the Satish Dhawan Space Centre in Andhra Pradesh.
The 44-metre-long rocket’s primary payload is the Earth Observation Satellite-6 (EOS-6) or Oceansat-3, a third-generation satellite to monitor oceans. It is a follow up to OceanSat-1 or IRS-P4 and OceanSat-2 launched in 1999 and 2009, respectively. Oceansat-3 will provide data about ocean colour, sea surface temperature, and wind vector data for oceanography, climatology, and meteorological applications.
The Oceansat-3 was placed in the polar orbit at a height of about 740 kilometres above sea level. While it weighs approximately 1,100 kilogrammes, which is only slightly heavier than Oceansat-1, for the first time in this series, it houses three ocean observing sensors. These include an Ocean Colour Monitor (OCM-3), Sea Surface Temperature Monitor (SSTM), and Ku-Band scatterometer (SCAT-3). There is also an ARGOS payload, a press release mentioned.
The OCM-3, with a high signal-to-noise ratio, is expected to improve accuracy in the daily monitoring of phytoplankton. This has a wide range of operational and research applications including fishery resource management, ocean carbon uptake, harmful algal bloom alerts, and climate studies. The SSTM will provide ocean surface temperature, which is a critical ocean parameter to provide various forecasts ranging from fish aggregation to cyclone genesis and movement. Temperature is a key parameter required to monitor the health of the coral reefs, and if needed, to provide coral bleaching alerts. The Ku-Band Pencil beam scatterometre will provide a high-resolution wind vector (speed and direction) at the ocean surface, which will be useful for seafarers, including fishermen and shipping companies. Data regarding temperature and wind is also particularly important for ocean and weather models to improve their forecast accuracies.
ARGOS is a communication payload jointly developed with France and it is used for low-power (energy-efficient) communications including marine robotic floats (Argo floats), fish-tags, drifters, and distress alert devices valuable in search and rescue operations.
The Minister of State (Independent Charge) for Science and Technology, Jitendra Singh, stated that ISRO will continue to maintain the orbit of the satellite and its standard procedures for data reception and archiving. Major operational users of this satellite include Ministry of Earth Sciences (MoEs) institutions such as the Indian National Centre for Ocean Information Services (INCOIS) and the National Centre for Medium Range Weather Forecasting (NCMRWF).
INCOIS has also established a state-of-the-art satellite data reception ground station within its campus with technical support from the National Remote Sensing Centre (ISRO-NRSC). Singh asserted that ocean observations such as this will serve as a solid foundation for India’s blue economy and polar region policies. A representative from MoES noted that the launch of Oceansat-3 is significant as it is the first major ocean satellite launch from India since the initiation of the UN Decade of Ocean Science for Sustainable Development (UNDOSSD, 2021-2030).
The Indian Space Research Organisation is the national space agency of India, headquartered in Bengaluru. It operates under the Department of Space, which is overseen by the country’s Prime Minister.