Skip to main content

Data transfer onto the HPC

At some point, you will inevitably need to get data from the HPC back onto your local machine, or vice versa. There are a number of different tools that can handle this, but for most use cases either rsync or scp should suffice.

Transfer using GUI application

If you would like to use a graphical user interface application, CyberDuck is one such option. After you have downloaded and opened the application, you will be met with a screen like this:

To connect to your home directory on the HPC, click on 'Open Connection', then select SFTP from the dropdown. Enter in the address of the HPC in the Server section, your username and password, as well as the SSH private key used to connect to the HPC if you have one (not shown in this example).

After hitting connect (click 'Allow' on the Unkown fingerprint popup), you will see the directory structure of the folders on the HPC in your home directory.

To download a file or directory simply right click it and hit download

And to upload a file you click the 'Upload' button in the top right and browse your local files to upload

Transfer using command line

Rsync is the most recommended:

  • Efficient for large or incremental transfers.
  • Automatically skips files that haven't changed.
  • Can resume interrupted transfers.

To copy files to the HPC:

eg. A text file called error.txt

(base) azeezoe@BBMP814:~$ rsync -avhP error.txt [email protected]:/hpc/faculty/azeezoe/error.txt
[email protected]'s password:
sending incremental file list
error.txt
            786 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 900 bytes  received 35 bytes  143.85 bytes/sec
total size is 786  speedup is 0.84

-a = archive (preserves permissions, etc.)

-v = verbose

-h = human-readable

-P = progress + resume

eg. To copy a folder (called test_transfer) from the HPC to your local computer

[azeezoe@hpc1 ~]$ ls test_transfer/
test_data.txt

Transfer it across from your local computer

(base) azeezoe@BBMP814:~$ rsync -avhP [email protected]:/hpc/faculty/azeezoe/test_transfer .
[email protected]'s password:
receiving incremental file list
test_transfer/
test_transfer/test_data.txt
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/2)

sent 47 bytes  received 156 bytes  45.11 bytes/sec
total size is 0  speedup is 0.00

We can see that both the folder and data are transferred across

(base) azeezoe@BBMP814:~$ ls test_transfer/
test_data.txt

Scp can still be appropriate, for quick, secure, one-time transfers. It's use is similar to rsync.

Transfer to HPC: scp -r /path/to/local_data [email protected]:/path/to/remote_folder/

Transfer from HPC: scp -r [email protected]:/path/to/remote_data /path/to/local_folder/

-r = recursive (needed for copying directories)

No comments to display

Back to top