Data transfer onto the HPC

At some point, you will inevitably need to get data from the HPC back onto your local machine, or vice versa. There are a number of different tools that can handle this, but for most use cases either rsync or scp should suffice.

Rsync is the most recommended:

• Efficient for large or incremental transfers.
• Automatically skips files that haven't changed.
• Can resume interrupted transfers.

To copy files to the HPC:

eg. A text file called error.txt

(base) azeezoe@BBMP814:~$ rsync -avhP error.txt [email protected]:/hpc/faculty/azeezoe/error.txt
[email protected]'s password:
sending incremental file list
error.txt
            786 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 900 bytes  received 35 bytes  143.85 bytes/sec
total size is 786  speedup is 0.84

-a = archive (preserves permissions, etc.)

-v = verbose

-h = human-readable

-P = progress + resume

eg. To copy a folder (called test_transfer) from the HPC to your local computer

[azeezoe@hpc1 ~]$ ls test_transfer/
test_data.txt

Transfer it across from your local computer

(base) azeezoe@BBMP814:~$ rsync -avhP [email protected]:/hpc/faculty/azeezoe/test_transfer .
[email protected]'s password:
receiving incremental file list
test_transfer/
test_transfer/test_data.txt
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/2)

sent 47 bytes  received 156 bytes  45.11 bytes/sec
total size is 0  speedup is 0.00

We can see that both the folder and data are transferred across

(base) azeezoe@BBMP814:~$ ls test_transfer/
test_data.txt

Scp can still be appropriate, for quick, secure, one-time transfers. It's use is similar to rsync.

Transfer to HPC: scp -r /path/to/local_data [email protected]:/path/to/remote_folder/

Transfer from HPC: scp -r [email protected]:/path/to/remote_data /path/to/local_folder/

-r = recursive (needed for copying directories)