Data Transfers via the command line

There are the usual ways, how you can transfer your data to the cluster:

  • scp: The regular way of transferring data in Linux
  • rsync: Also a standard tool in Linux like scp, but has the advantage that it can resume broken transfers as it normally keeps the metadata of the files
  • rclone: A mighty tool for data transfers, normally to cloud storage. May also be utilized for data transfers to Palma and might be faster than the former solutions when transferring many small files
  • WinSCP: The standard tool when using Windows

Data Transfers via the web interface

This part is still in a testing phase! Please be careful when using it!

If you have access to Palma, you can use a Nextcloud for data transfers. To make use of it, do the following

  • Create a directory called "transfer" at /scratch/tmp/$USER: "mkdir /scratch/tmp/$USER/transfer"
  • Log in at https://palma-web.uni-muenster.de
  • Click on the "scratch" directory. (You might have to click multiple times in the first run). Data you put in here will be stored in the scratch directory. Also, data that you put in /scratch/tmp/$USER/transfer will be visible here
  • The recommended way to use this is to upload your data via the webinterface and move it to another location on the cluster afterwards. The speed of the nextcloud will probably decrease, if the directory becomes too crowded.
  • In future iterations, this is intended to be used for realizing data transfers between different HPC clusters.

Transferring data to the Nextcloud via rclone

If you want to copy data to and from Palma from the outside and have rclone available, it can be used for a performant data transfer:

Setting up rclone:

  • rclone config 
    • n) New remote
    • name> palma-web
    • Storage> 47 (this is WebDAV)
    • url> https://palma-web.uni-muenster.de/remote.php/dav/files/$username (replace $username with your user id)
    • vendor> 2
    • user> $username (replace $username with your user id)
    • y/g/n> y (y) Yes, type in my own password)
    • password: Create an app password in the web interface at Settings → Security (Einstellungen → Sicherheit) and put it here.
    • bearer_token> (Leave empty)
    • n) No (default)
    • y) Yes this is OK (default)
    • q) Quit config

Data transfer with rclone:

  • From remote to palma: `rclone folder palma-web:scratch/ --progress --verbose`
  • From palma to remote: `rclone  palma-web:scratch/_folder_ . --progress --verbose`

See also: https://linuxpip.org/rclone-examples

  • Keine Stichwörter