Hong Kong’s Smart Government Innovation Lab has been putting out a slew of innovations develops by its various incubatees.
Recently, one of its supported firms developed a solution which is now ready to be acquired by other firms, government agencies and academic institutions.
Solution description
To support hyper-scale storage management across multiple geographical regions, the firm posits that Hyperconverged Infrastructure (HCI) should be geared towards geo-distributed deployment.
A geo-distributed HCI can follow a hierarchical topology: it is composed of multiple DCs that reside in different geographical regions, in which each DC is composed of multiple nodes (or servers) for storage.
Thus, a geo-distributed HCI unifies the storage space across different geographical regions to form a global-scale storage pool.
One critical deployment requirement of a geo-distributed HCI is to maintain availability and durability guarantees in the presence of failures, ranging from transient failures where data is temporarily unavailable (e.g., power loss, network disconnection, system upgrades, reboots), to permanent failures where data is permanently lost (e.g., disk crashes, sector errors). Cloud outages are common in practice.
Storing data with redundancy is a typical fault-tolerant approach, as any lost or unavailable data can be recovered through the available redundant data. Traditional distributed storage systems use replication to provide fault tolerance.
The idea is to create multiple identical copies (called replicas) for each data chunk and distribute the replicas across different nodes.
However, the storage overhead of replication is significant and poses a scalability concern with the unprecedented growth of data volume.
The incubatee presents its core technology, network coding, proposed by the research team at the Chinese University of Hong Kong in 2000.
Network coding provides a promising low-cost redundancy mechanism to achieve fault tolerance, while significantly reducing the storage overhead compared to traditional replication and achieving optimal repair performance with theoretical guarantees.
It belongs to a special class of erasure coding; first formalising the terminologies of erasure coding, and then elaborating how network coding improves the repair performance.
Application Areas
The solution can be deployed in the areas of Infrastructure as well as GovCloud.
Technologies Used
The solution employs Cloud Computing and Network Coding.
Use case
The benefits of the solution are as follows:
- High storage efficiency: nCloud leverages network coding to minimize the amount of storage redundancy compared to traditional replication, thereby saving the long-term operational costs.
- High performance: nCloud mitigates the cross-DC traffic to maintain high repair performance. It also mitigates the overhead on other applications that share the cross-DC bandwidth.
- High fault tolerance: nCloud achieves high fault tolerance via network coding in two aspects: (i) availability, which means that any unavailable data remains accessible through data redundancy, and (ii) durability, which means there is no data loss in the face of node or DC failures.
- Security via diversity: nCloud exploits secret sharing (see our previous work CDStore [22]) to generate secure coded chunks (called shares), such that an adversary cannot recover the stored data if the number of shares is insufficient.
- Software-defined storage management: nCloud provides a configurable, software-defined storage management framework to readily address the heterogeneity of storage resources and application requirements.
About the Smart Government Innovation Lab
In 2018, the Government established the Smart Government Innovation Lab to explore hi-tech products such as AI and relevant technologies, including machine learning, big data analytics, cognitive systems and intelligent agent, as well as blockchain and robotics from firms, especially local start-ups.
The Lab is always on the lookout for innovation and technology (I&T) solutions that are conducive to enhancing public services or their operational effectiveness.
I&T suppliers are encouraged to regularly visit the Lab’s website to check on the current business and operational needs in public service delivery and propose innovative solutions or product suggestions to address them.
