Oak Ridge Transitions to Kronos, Marking a New Data Storage Era

Summary

Sources

docs.olcf.ornl.govdocs.olcf.ornl.gov

After decades of service, supporting hundreds of users and archiving over one hundred and sixty petabytes of data, the High Performance Storage System, known as HPSS, is reaching the end of its operational life and will be decommissioned in early twenty twenty-five. The transition from HPSS to the new nearline storage resource, Kronos, marks a significant change in data storage strategy at the Oak Ridge Leadership Computing Facility. Kronos, set to be available for production use in late July twenty twenty-four, represents the next generation of nearline storage. It features a multi-tiered architecture that combines both disk and tape storage solutions. The system leverages IBM Storage Scale, also known as General Parallel File System or GPFS, for the disk sub-system, which will then automatically store data on the systems tape sub-system. Initially, Kronos will offer an impressive capacity of one hundred and thirty-four petabytes, with the ability to expand as the need arises. Its bandwidth capabilities are expected to reach up to two hundred gigabytes per second, facilitating efficient data transfers from the centers Data Transfer Nodes. Regarding accessibility, Kronos will be integrated with moderate Data Transfer Nodes, Frontier login nodes, and Andes login nodes, but will not be mounted on the compute nodes of Frontier or Andes. Its integration allows for the use of standard UNIX commands and tools for interaction, simplifying the user experience. The storage system will also be accessible through the Oak Ridge Leadership Computing Facility Data Transfer Node endpoint in Globus, ensuring that all projects with allocations to Summit, Frontier, or Andes will receive an allocation on Kronos. The directory structure of Kronos is designed to be similar to other center-wide storage resources, facilitating a smoother transition for users familiar with the systems environment. The transition to Kronos from HPSS is a critical step. HPSS will become read-only from August thirty, twenty twenty-four, as part of the preparation for its decommissioning in January twenty twenty-five. This change necessitates users to begin migrating their needed data from HPSS to Kronos as early as possible to avoid potential time constraints. The Oak Ridge Leadership Computing Facility emphasizes the importance of coordinating large HPSS data migrations to mitigate access impacts expected due to the high volumes of data being transferred. It is recommended to use the HPSS hsi utility for the most efficient data migration experience, especially for data stored only on tape, to ensure a smooth transition to Kronos. This shift not only highlights the technological advancements in data storage systems but also underscores the continuous effort to provide more efficient, reliable, and scalable storage solutions to meet the growing demands of high-performance computing environments. The decommissioning of the High Performance Storage System, HPSS, marks the end of an era in the realm of data storage at the Oak Ridge Leadership Computing Facility. Over its operational lifespan, HPSS played a pivotal role in data storage for an extensive user base, successfully archiving over one hundred and sixty petabytes of data. This significant archive symbolizes the vast amount of research, development, and innovation facilitated by HPSS, underscoring its importance in advancing computational science and engineering. As the facility transitions from HPSS to Kronos, there are several key dates of which users and stakeholders must be aware. The process begins on August thirty, twenty twenty-four, when HPSS will transition to a read-only format. This critical step is a clear signal for users to initiate the migration of their essential data from HPSS to Kronos. The read-only status means that while existing data can still be accessed, no new data can be added to HPSS, emphasizing the importance of beginning the migration process in a timely manner. The final stage in the decommissioning process is set for January thirty-first, twenty twenty-five, when all remaining data on HPSS will be permanently deleted. This irreversible action underscores the urgency for users to migrate their data to avoid loss. It is a stark reminder of the finite nature of technological systems and the need for constant adaptation and evolution in the face of progressing technological capabilities. The migration from HPSS to Kronos is not without its challenges. Coordinating large data migrations is a complex task that requires meticulous planning and execution. The sheer volume of data, over one hundred and sixty petabytes, presents a logistical challenge, necessitating efficient and effective strategies for data transfer. Users are encouraged to plan their migrations carefully, considering factors such as data prioritization, transfer speeds, and potential bottlenecks. The Oak Ridge Leadership Computing Facility is providing support to facilitate this transition. Recognizing the potential for access impacts due to the expected volumes of data migrating off of HPSS, efforts are being made to spread the migration load. Users are encouraged to coordinate their large HPSS data migrations by reaching out for assistance. This support is crucial in mitigating the challenges associated with such a large-scale data migration, ensuring a smoother transition for all involved. In preparation for the transition, users are advised against deleting data from HPSS due to the added overhead of data deletion. Instead, the focus should be on transferring necessary data to Kronos using the most efficient methods available. The HPSS hsi utility is recommended for its performance and organization capabilities, especially for data stored on tape, which may have longer retrieval times. As the facility moves forward with the decommissioning of HPSS, the emphasis is on ensuring a seamless transition for users to Kronos. Despite the challenges, the transition represents a step forward in enhancing data storage capabilities, offering improved efficiency, reliability, and scalability for the future needs of high-performance computing. Kronos emerges as the new frontier in data storage at the Oak Ridge Leadership Computing Facility, heralding a new era of efficiency and capacity in nearline storage solutions. This innovative system is designed with a multi-tiered architecture that ingeniously combines both disk and tape storage, optimizing data storage and retrieval processes to meet the burgeoning demands of modern high-performance computing environments. At the heart of Kronoss architecture is an initial staggering capacity of one hundred and thirty-four petabytes, a testament to its ability to accommodate vast amounts of data. This capacity is not merely about numbers; it represents the future of computational research, offering ample space for the expansive data sets that todays scientific research generates. Furthermore, Kronos boasts an expected bandwidth of up to two hundred gigabytes per second, a significant upgrade that ensures the rapid transfer of data, facilitating efficient workflows and reducing the time researchers spend waiting for data access. Accessibility is a cornerstone of the Kronos design. The system is seamlessly integrated with various nodes, including moderate Data Transfer Nodes, Frontier login nodes, and Andes login nodes, although it will not be mounted on the compute nodes of Frontier or Andes. This strategic choice of accessibility points underscores a commitment to user convenience and operational flexibility. Moreover, the use of a standard UNIX command interface for user interaction with Kronos highlights a thoughtful approach to ensuring that the transition for users is as smooth as possible, leveraging familiar command structures to lower the learning curve and streamline data management tasks. Comparing Kronos with its predecessor, HPSS, reveals significant advancements and benefits. Both systems offer accessibility through Data Transfer Nodes and login nodes, but Kronos introduces enhancements that streamline data interaction. For instance, the transition from specialized transfer tools like hsi and htar in HPSS to the integration of globus and standard UNIX transfer utilities in Kronos marks a leap forward in simplifying data transfers. This evolution in transfer tools reflects a broader shift towards more user-friendly and accessible data management practices. In terms of file and directory management, Kronos departs from the HPSSs reliance on the hsi command interface, embracing standard UNIX utilities that many users are already familiar with. This change not only eases the transition for existing users but also makes the system more accessible to new users, promoting broader adoption and utilization of the storage resource. Perhaps one of the most significant improvements is in data retrieval speeds. While data retrieval from HPSS could fluctuate based on the datas location, potentially seeing delays if the data was only stored on tape, Kronos offers a more consistent access experience. All data stored on disk in Kronos ensures rapid and reliable retrieval, minimizing delays and enhancing productivity for users. In summation, Kronos represents a significant leap forward in data storage technology at the Oak Ridge Leadership Computing Facility. Its multi-tiered architecture, combining the strengths of disk and tape storage, its impressive capacity and bandwidth, and its enhanced accessibility and user interaction mechanisms, position Kronos as a pivotal advancement in supporting the next generation of scientific research. As the transition from HPSS to Kronos progresses, the benefits of this new system promise to redefine the landscape of data storage and management in high-performance computing environments.