HPC

Difference between revisions of "Training/Transferring Data"

From HPC
Jump to: navigation , search
m
 
(57 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
__TOC__
 
__TOC__
−
{|
+
 
−
|style="width:5%; border-width: 0" | [[File:icon_warning.png]]
+
 
−
|style="width:95%; border-width: 0" | Please Note: Data stored on Viper is not backed up, if required please backup data to an external source. See the  [[Training/Data_Backup|Data backup]] page here.
+
==Methods==
 +
 
 +
===Small to medium amounts of data===
 +
 
 +
This can be carried out using a number of SFTP-styled packages such as sftp, scp, rsync, and Filezilla to name a few. Filezilla can be downloaded as a  Windows application and may be more suitable for people who prefer a Windows environment, rather than the command line.
 +
 
 +
'''Note''' FTP (insecure) is disabled on viper.hull.ac.uk and SFTP (secure FTP) is the recommended method.
 +
 
 +
===Large to a huge amount of data (100GB +)===
 +
To avoid excessive network congestion and the long period of time to transfer such data please raise a [https://support.hull.ac.uk/ support ticket] for further details. This may typically involve connecting a USB data drive directly to VIPER to copy the data quickly into your user area.
 +
 
 +
===OneDrive to Viper===
 +
 
 +
With the University using MS OneDrive, it is possible to copy files to Viper. For multiple files, zip your files up on OneDrive first (considerably easier), as shown here:
 +
 
 +
{| class="wikitable mw-collapsible"
 +
| style="width:100%; background: #aaaaaa;" | <Strong>From OneDrive to Viper</Strong>
 
|-
 
|-
−
|}
+
|
 +
* Starting with the zipped file MyData.zip on OneDrive, copied previously and compressed from local storage.
 +
* Click on the '''share icon''' or options for the file here (three horizontal dots) and select '''share'''.
 +
 
 +
[[File:onedrive-01.png]]
 +
 
 +
* Then
 +
 
 +
[[File:onedrive-02.png]]
 +
 
 +
* Click on '''Copy'''
 +
 
 +
[[File:onedrive-03.png]]
 +
 
 +
* On your Viper console application, type '''wget''' and paste in the copied link and follow with '''-O yourfilename''' (otherwise Linux will name the file to the OneDrive link - ''argh!'')
 +
 
 +
[[File:onedrive-04.png]]
 +
 
 +
* And here is the received file
  
−
==Small to medium amounts of data==
+
[[File:onedrive-05.png]]
−
This can be carried out using a number of SFTP-styled packages such as sftp, scp, rsync, and Filezilla to name a few. Filezilla can be downloaded as a  Windows application and may be more suitable for people who prefer a Windows environment, rather than the command line.
 
  
−
'''Note''' that plain FTP is disabled on viper.hull.ac.uk and SFTP (secure FTP) is the recommended method.
+
'''Remember''' to un-share the file on OneDrive and use unzip (if required) on Linux to restore the file structure.
  
−
==Large to a huge amount of data (100GB +)==
+
|}
−
To avoid excessive network congestion and the long period of time to transfer such data please contact [https://support.hull.ac.uk/ Topdesk] for further details. This may typically involve connecting a USB data drive directly to VIPER to copy the data quickly into your user area.
 
  
 
==Universal==
 
==Universal==
Line 19: Line 51:
 
The example user is user1; you will need to enter your own username here:
 
The example user is user1; you will need to enter your own username here:
  
−
{| class="wikitable"
+
{| class="wikitable mw-collapsible"
−
| style="width:100%" | <Strong>From your local computer to Viper</Strong>
+
| style="width:100%; background: #aaaaaa;" | <Strong>* To Viper from local computer</Strong>
 
|-
 
|-
 
|scp myfile user1@viper.hull.ac.uk:myfile
 
|scp myfile user1@viper.hull.ac.uk:myfile
Line 26: Line 58:
 
|rsync -av my-directory viper.hull.ac.uk:/data/users/YourID
 
|rsync -av my-directory viper.hull.ac.uk:/data/users/YourID
 
|}
 
|}
−
{| class="wikitable"
+
 
−
| style="width:100%" | <Strong>From Viper to your local Computer</Strong>
+
{| class="wikitable mw-collapsible"
 +
| style="width:100%; background: #aaaaaa;" | <Strong>* From Viper to your local computer</Strong>
 
|-
 
|-
−
|scp user1@viper.hull.ac.uk:bio/YourUCInetID/hpc-file
+
| scp user1@viper.hull.ac.uk:bio/YourUCInetID/hpc-file
 
|-
 
|-
 
|rsync -av viper.hull.ac.uk:/data/users/YourID/mydatadirectory
 
|rsync -av viper.hull.ac.uk:/data/users/YourID/mydatadirectory
 
|}
 
|}
 +
 
==Windows==
 
==Windows==
 
The following example uses MobaXterm, for transferring large files it is recommended to use FileZilla. For instructions on how to use FileZilla view the Linux section. follow the instruction in the Getting Connected section for installing MobaXterm.
 
The following example uses MobaXterm, for transferring large files it is recommended to use FileZilla. For instructions on how to use FileZilla view the Linux section. follow the instruction in the Getting Connected section for installing MobaXterm.
−
===Uploading Files===
 
−
1. To upload files to Viper, navigate to the directory you wish to upload files to use the MobaXterm file browser and click the 'Upload to current folder' button.
 
−
[[File:upload_button.jpg]]
 
−
2. Choose the file/files to upload and click open or alternatively drag and drop the file/files you wish to upload over the Viper file browser.
 
  
−
3. The upload may take some time depending on upload speed and file size.
+
{| class="wikitable mw-collapsible"
 +
| style="width:100%; background: #aaaaaa;" | <Strong>* Uploading Files</Strong>
 +
|-
 +
|* To upload files to Viper, navigate to the directory you wish to upload files to use the MobaXterm file browser.
 +
* Choose the file/files to upload and click open or alternatively drag and drop the file/files you wish to upload over the Viper file browser.
 +
* The upload may take some time depending on upload speed and file size.
 +
|}
 +
 
  
−
===Downloading Files===
+
{| class="wikitable mw-collapsible"
−
1. To download files from Viper, navigate to the directory using the MobaXterm file browser.
+
| style="width:100%; background: #aaaaaa;" | <Strong>* Downloading Files</Strong>
 +
|-
 +
|* To download files from Viper, navigate to the directory using the MobaXterm file browser.
 
[[File:mobaxterm_file_browser.jpg]]
 
[[File:mobaxterm_file_browser.jpg]]
−
2. Select the file/files you wish to download and click the 'download selected files button'.
+
* Select the file/files you wish to download and click the 'download selected files button'.
 
[[File:mobaxterm_download_icon.jpg]]
 
[[File:mobaxterm_download_icon.jpg]]
−
3. Choose the directory where you want the files to download and click ok.
+
* Choose the directory where you want the files to download and click ok.
−
 
+
* The download will start, depending on download speed and file size this may take some time.
−
4. The download will start, depending on download speed and file size this may take some time.
+
|}
  
 
==MacOS==
 
==MacOS==
−
The following example uses Cyberduck, you can download Cyberduck from https://cyberduck.io macOS also supports FileZilla for an example on how to use FileZilla please view the Linux section
+
The following example uses Cyberduck, you can download Cyberduck from https://cyberduck.io macOS also supports FileZilla for an example of how to use FileZilla please view the Linux section.
−
===Connecting===
 
−
1. Click Open Connection.
 
  
−
2. From the drop-down menu select '''SFTP''' (SSH File Transfer Protocol)
+
{| class="wikitable mw-collapsible"
−
 
+
| style="width:100%; background: #aaaaaa;" | <Strong>Connecting</Strong>
−
3. Enter server viper.hull.ac.uk, Port: 22, Username: University UID, Password: university password
+
|-
−
 
+
|
−
4. Click Connect
+
* Click Open Connection.
 +
* From the drop-down menu select '''SFTP''' (SSH File Transfer Protocol)
 +
* Enter server viper.hull.ac.uk, Port: 22, Username: University UID, Password: university password
 +
* Click Connect.
  
 
[[File:cyberduck_connection.jpg]]
 
[[File:cyberduck_connection.jpg]]
 +
|}
  
−
===Uploading Files===
+
{| class="wikitable mw-collapsible"
−
1. Click Upload
+
| style="width:100%; background: #aaaaaa;" | <Strong>Uploading Files</Strong>
 +
|-
 +
|
 +
* Click Upload
 +
* Choose the file/folder you want to upload and click Choose.
 +
* The transfer window will appear.
 +
* Please note due to file size the upload may take some time.
 +
|}
  
−
2. Choose the file/folder you want to upload and click choose
+
{| class="wikitable mw-collapsible"
 +
| style="width:100%; background: #aaaaaa;" | <Strong>Downloading Files</Strong>
 +
|-
 +
|
 +
* From the file browser right-click on the file/folder to download and select download from the menu.
 +
* The transfer window will appear.
 +
* Please note due to file size the download may take some time.
 +
|}
  
−
3. The transfer window will appear
+
==Linux==
−
 
 
−
4. Please note due to file size the upload may take some time
 
  
−
===Downloading Files===
+
The following example uses FileZilla, you can download FileZilla from https://filezilla-project.org/ (''Windows/Linux and MacOS'').
−
1. From the file browser right-click on the file/folder to download and select download from the menu
+
* '''Note''': the FileZilla installer sometimes comes with additional trial software packages. We would advise against installing these
  
−
2. The transfer window will appear
+
{| class="wikitable mw-collapsible"
−
 
+
| style="width:100%; background: #aaaaaa;" | <Strong>Using FileZilla</Strong>
−
3. Please note due to file size the download may take some time
 
−
==Linux==
 
−
{|
 
−
|style="width:5%; border-width: 0" | [[File:icon_warning.png]]
 
−
|style="width:95%; border-width: 0" | Filezilla can also be used on Windows and macOS.
 
 
|-
 
|-
−
|}
+
|
−
The following example uses FileZilla, you can download FileZilla from https://filezilla-project.org/
+
*Enter the Host: viper.hull.ac.uk, username, password, and port: 22, using an SFTP connection.
−
 
 
−
1. Enter the Host: viper.hull.ac.uk, username, password, and port: 22, using an SFTP connection.
 
 
[[File:filezilla_linux_login.jpg]]
 
[[File:filezilla_linux_login.jpg]]
−
2. A pop-up will appear informing that the server host key is unknown. Click always trust this host and click ok. In the remote site box, your Viper directory will appear.
+
* A pop-up will appear informing that the server host key is unknown. Click Always trust this host and click ok. In the remote site box, your Viper directory will appear.
−
 
+
* To transfer files click and drag the file/folder from either your pc to Viper (left to right) or from Viper to your pc (right to left). The transfer will start depending on transfer speed and file size this may take some time.
−
3. To transfer files click and drag the file/folder from either your pc to Viper (left to right) or from Viper to your pc (right to left). The transfer will start depending on transfer speed and file size this may take some time.
 
 
[[File:filezilla_transferring_files.jpg]]
 
[[File:filezilla_transferring_files.jpg]]
−
 
+
|}
  
 
==Navigation==
 
==Navigation==

Latest revision as of 12:45, 8 June 2023


Methods

Small to medium amounts of data

This can be carried out using a number of SFTP-styled packages such as sftp, scp, rsync, and Filezilla to name a few. Filezilla can be downloaded as a Windows application and may be more suitable for people who prefer a Windows environment, rather than the command line.

Note FTP (insecure) is disabled on viper.hull.ac.uk and SFTP (secure FTP) is the recommended method.

Large to a huge amount of data (100GB +)

To avoid excessive network congestion and the long period of time to transfer such data please raise a support ticket for further details. This may typically involve connecting a USB data drive directly to VIPER to copy the data quickly into your user area.

OneDrive to Viper

With the University using MS OneDrive, it is possible to copy files to Viper. For multiple files, zip your files up on OneDrive first (considerably easier), as shown here:

From OneDrive to Viper
  • Starting with the zipped file MyData.zip on OneDrive, copied previously and compressed from local storage.
  • Click on the share icon or options for the file here (three horizontal dots) and select share.

Onedrive-01.png

  • Then

Onedrive-02.png

  • Click on Copy

Onedrive-03.png

  • On your Viper console application, type wget and paste in the copied link and follow with -O yourfilename (otherwise Linux will name the file to the OneDrive link - argh!)

Onedrive-04.png

  • And here is the received file

Onedrive-05.png

Remember to un-share the file on OneDrive and use unzip (if required) on Linux to restore the file structure.

Universal

The following can only be performed on Windows after installing MobaXterm or PuTTY.

The example user is user1; you will need to enter your own username here:

* To Viper from local computer
scp myfile user1@viper.hull.ac.uk:myfile
rsync -av my-directory viper.hull.ac.uk:/data/users/YourID
* From Viper to your local computer
scp user1@viper.hull.ac.uk:bio/YourUCInetID/hpc-file
rsync -av viper.hull.ac.uk:/data/users/YourID/mydatadirectory

Windows

The following example uses MobaXterm, for transferring large files it is recommended to use FileZilla. For instructions on how to use FileZilla view the Linux section. follow the instruction in the Getting Connected section for installing MobaXterm.

* Uploading Files
* To upload files to Viper, navigate to the directory you wish to upload files to use the MobaXterm file browser.
  • Choose the file/files to upload and click open or alternatively drag and drop the file/files you wish to upload over the Viper file browser.
  • The upload may take some time depending on upload speed and file size.


* Downloading Files
* To download files from Viper, navigate to the directory using the MobaXterm file browser.

Mobaxterm file browser.jpg

  • Select the file/files you wish to download and click the 'download selected files button'.

Mobaxterm download icon.jpg

  • Choose the directory where you want the files to download and click ok.
  • The download will start, depending on download speed and file size this may take some time.

MacOS

The following example uses Cyberduck, you can download Cyberduck from https://cyberduck.io macOS also supports FileZilla for an example of how to use FileZilla please view the Linux section.

Connecting
  • Click Open Connection.
  • From the drop-down menu select SFTP (SSH File Transfer Protocol)
  • Enter server viper.hull.ac.uk, Port: 22, Username: University UID, Password: university password
  • Click Connect.

Cyberduck connection.jpg

Uploading Files
  • Click Upload
  • Choose the file/folder you want to upload and click Choose.
  • The transfer window will appear.
  • Please note due to file size the upload may take some time.
Downloading Files
  • From the file browser right-click on the file/folder to download and select download from the menu.
  • The transfer window will appear.
  • Please note due to file size the download may take some time.

Linux

The following example uses FileZilla, you can download FileZilla from https://filezilla-project.org/ (Windows/Linux and MacOS).

  • Note: the FileZilla installer sometimes comes with additional trial software packages. We would advise against installing these
Using FileZilla
  • Enter the Host: viper.hull.ac.uk, username, password, and port: 22, using an SFTP connection.

Filezilla linux login.jpg

  • A pop-up will appear informing that the server host key is unknown. Click Always trust this host and click ok. In the remote site box, your Viper directory will appear.
  • To transfer files click and drag the file/folder from either your pc to Viper (left to right) or from Viper to your pc (right to left). The transfer will start depending on transfer speed and file size this may take some time.

Filezilla transferring files.jpg

Navigation

Retrieved from ‘http://hpc.mediawiki.hull.ac.uk/index.php?title=Training/Transferring_Data&oldid=41603’