Skip to main content

Data Migration

Using the scp command, users can copy data from other servers or storage to the HPC Phase III platform (network connectivity needs to be confirmed in advance).

Network Connectivity Confirmation​

Use the ping command on the HPC Phase III platform to confirm whether the source data can be accessed.

# Confirm the network connectivity of the HPC Phase II platform

ping hpc2login.hkust-gz.edu.cn

Data Migration of the HPC Phase II Platform​

rsync​

Suitable for large-scale file data migration operations.

rsync -vrlptDSH username@hpc2login.hpc.hkust-gz.edu.cn:<source_path> <target_path>

-v: Represents the verbose mode. It will display detailed information during the synchronization process, such as the names of the files being transferred, the transfer progress, etc., making it convenient for users to understand the progress of the synchronization.

-r: Represents recursive. It will synchronize all subdirectories and files under the source directory to the target directory, ensuring that the entire directory structure is completely copied.

-l: Used to preserve symbolic links. That is, during the synchronization process, if there are symbolic link files in the source directory, the symbolic links themselves will be copied to the target directory instead of the original files pointed to by the symbolic links.

-p: Represents preserving file permissions, ensuring that the files in the target directory have the same permission settings (such as read, write, and execute permissions) as the source files. This is very important for some programs or files with strict permission requirements.

-t: Used to preserve file timestamps, including file modification time, access time, etc. In this way, the files in the target directory can be consistent with the source files in terms of time, which is helpful for tracing the change history of files based on time information when needed.

-D: Equivalent to -a -X. -a represents the archive mode, which will recursively copy files and preserve all file attributes, such as permissions, owners, timestamps, etc.; -X represents copying extended attributes, including file ACL (Access Control List) and other information. However, in this command, -D may be used to handle device files and special files, and the interpretation of this option may vary slightly in different versions of rsync.

-S: Represents compressing data during transmission. The files will be compressed during the transmission process, which can reduce the amount of data transferred and improve the transmission speed, especially effective when the network bandwidth is limited. However, this will increase the CPU load because compression and decompression operations are required.

-H: Represents preserving hard links. If there are multiple files in the source directory linked to the same inode (i.e., hard links), it will ensure that these files also maintain the hard link relationship in the target directory instead of being copied into multiple independent files.

scp​

Use the scp command to copy data from the storage of the HPC Phase II platform to the storage of the HPC Phase III platform.

# File copy

scp -p username@hpc2login.hpc.hkust-gz.edu.cn:<source_file> <target_path>

# Directory copy

scp -rp username@hpc2login.hpc.hkust-gz.edu.cn:<source_path> <target_path>