Difference between revisions of "Quickstart/Data Management"
m (→Larger amounts of data (100GB +)) |
m (→Linux) |
||
| (8 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
{| | {| | ||
|style="width:5%; border-width: 0" | [[File:icon_warning.png]] | |style="width:5%; border-width: 0" | [[File:icon_warning.png]] | ||
| − | |style="width:95%; border-width: 0" | <strong>''Viper storage is considered 'scratch', please remember to back up your data! '' | + | |style="width:95%; border-width: 0" | <strong>''Viper storage is considered 'scratch', please remember to back up your data! ''<br>''Users are not permitted to store personal or sensitive information on Viper.''</strong> |
|- | |- | ||
|} | |} | ||
| Line 11: | Line 11: | ||
For less than 100GB: you can skip to using [[#Universal| universal method]] or skip to your OS: [[#Windows| Windows]] / [[#MacOS| Mac]] / [[#Linux| Linux]]. | For less than 100GB: you can skip to using [[#Universal| universal method]] or skip to your OS: [[#Windows| Windows]] / [[#MacOS| Mac]] / [[#Linux| Linux]]. | ||
===Small to medium amounts of data=== | ===Small to medium amounts of data=== | ||
| − | This can be carried out using | + | This can be carried out using several SFTP-styled packages such as sftp, scp, rsync, and Filezilla to name a few. Filezilla can be downloaded as a Windows application and may be more suitable for people who prefer a Windows environment, rather than the command line. |
'''Note''' that plain FTP is disabled on viper.hull.ac.uk and SFTP (secure FTP) is the recommended method. | '''Note''' that plain FTP is disabled on viper.hull.ac.uk and SFTP (secure FTP) is the recommended method. | ||
| Line 18: | Line 18: | ||
The following can only be performed on Windows after installing MobaXterm or PuTTY. | The following can only be performed on Windows after installing MobaXterm or PuTTY. | ||
| − | The example user is user1; you will need to enter your | + | The example user is user1; you will need to enter your username here: |
{| class="wikitable" | {| class="wikitable" | ||
| style="width:100%" | <Strong>From your local computer to Viper</Strong> | | style="width:100%" | <Strong>From your local computer to Viper</Strong> | ||
|- | |- | ||
| − | |scp myfile user1@viper.hull.ac.uk:myfile | + | |''scp myfile user1@viper.hull.ac.uk:myfile |
| − | |- | + | |-'' |
| − | |rsync -av my-directory | + | |''rsync -av my-directory viperlogin.hpc.hull.ac.uk:/data/users/YourID'' |
|} | |} | ||
{| class="wikitable" | {| class="wikitable" | ||
| style="width:100%" | <Strong>From Viper to your local Computer</Strong> | | style="width:100%" | <Strong>From Viper to your local Computer</Strong> | ||
|- | |- | ||
| − | |scp user1@ | + | |''scp user1@viperlogin.hpc.hull.ac.uk:bio/YourUCInetID/hpc-file'' |
|- | |- | ||
| − | |rsync -av viper.hull.ac.uk:/data/users/YourID/mydatadirectory | + | |''rsync -av viper.hull.ac.uk:/data/users/YourID/mydatadirectory'' |
|} | |} | ||
| + | |||
====Windows==== | ====Windows==== | ||
| − | The following example uses MobaXterm, for transferring large files it is recommended to use FileZilla. For instructions on how to use FileZilla view the Linux section. follow the | + | The following example uses MobaXterm, for transferring large files it is recommended to use FileZilla. For instructions on how to use FileZilla view the Linux section. follow the instructions in the Getting Connected section for installing MobaXterm. |
=====Uploading Files===== | =====Uploading Files===== | ||
1. To upload files to Viper, navigate to the directory you wish to upload files to use the MobaXterm file browser and click the 'Upload to current folder' button. | 1. To upload files to Viper, navigate to the directory you wish to upload files to use the MobaXterm file browser and click the 'Upload to current folder' button. | ||
[[File:upload_button.jpg]] | [[File:upload_button.jpg]] | ||
| − | 2. Choose the file/files to upload and click open or | + | 2. Choose the file/files to upload and click open or drag and drop the file/files you wish to upload over the Viper file browser. |
3. The upload may take some time depending on upload speed and file size. | 3. The upload may take some time depending on upload speed and file size. | ||
| Line 53: | Line 54: | ||
====MacOS==== | ====MacOS==== | ||
| − | The following example uses Cyberduck, you can download Cyberduck from https://cyberduck.io macOS also supports FileZilla for an example | + | The following example uses Cyberduck, you can download Cyberduck from https://cyberduck.io macOS also supports FileZilla for an example of how to use FileZilla please view the Linux section |
=====Connecting===== | =====Connecting===== | ||
1. Click Open Connection. | 1. Click Open Connection. | ||
| Line 59: | Line 60: | ||
2. From the drop-down menu select '''SFTP''' (SSH File Transfer Protocol) | 2. From the drop-down menu select '''SFTP''' (SSH File Transfer Protocol) | ||
| − | 3. Enter server | + | 3. Enter server viperlogin.hpc.hull.ac.uk, Port: 22, Username: University UID, Password: university password |
4. Click Connect | 4. Click Connect | ||
| − | |||
| − | |||
=====Uploading Files===== | =====Uploading Files===== | ||
| Line 81: | Line 80: | ||
3. Please note due to file size the download may take some time | 3. Please note due to file size the download may take some time | ||
====Linux==== | ====Linux==== | ||
| − | + | ||
| − | |||
| − | |||
| − | |||
| − | |||
The following example uses FileZilla, you can download FileZilla from https://filezilla-project.org/ | The following example uses FileZilla, you can download FileZilla from https://filezilla-project.org/ | ||
| − | 1. Enter the Host: | + | 1. Enter the Host: viperlogin.hpc.hull.ac.uk, username, password, and port: 22, using an SFTP connection. |
| − | + | ||
| − | 2. A pop-up will appear informing that the server host key is unknown. Click | + | 2. A pop-up will appear informing that the server host key is unknown. Click Always trust this host and click ok. In the remote site box, your Viper directory will appear. |
3. To transfer files click and drag the file/folder from either your pc to Viper (left to right) or from Viper to your pc (right to left). The transfer will start depending on transfer speed and file size this may take some time. | 3. To transfer files click and drag the file/folder from either your pc to Viper (left to right) or from Viper to your pc (right to left). The transfer will start depending on transfer speed and file size this may take some time. | ||
[[File:filezilla_transferring_files.jpg]] | [[File:filezilla_transferring_files.jpg]] | ||
| − | |||
===Larger amounts of data (100GB +)=== | ===Larger amounts of data (100GB +)=== | ||
| − | To avoid excessive network congestion and the long period | + | To avoid excessive network congestion and the long period to transfer such data please contact us via the [https://hull.service-now.com/ Support Portal] for further details. This may typically involve connecting a USB data drive directly to Viper to copy the data quickly into your user area. |
[[Main Page #Quickstart| Back]] / [[Quickstart/Virtual Environments| Next (Virtual Environments)]] | [[Main Page #Quickstart| Back]] / [[Quickstart/Virtual Environments| Next (Virtual Environments)]] | ||
Latest revision as of 15:53, 22 August 2024
|
Viper storage is considered 'scratch', please remember to back up your data! Users are not permitted to store personal or sensitive information on Viper. |
Contents
Transferring Data on Viper
For less than 100GB: you can skip to using universal method or skip to your OS: Windows / Mac / Linux.
Small to medium amounts of data
This can be carried out using several SFTP-styled packages such as sftp, scp, rsync, and Filezilla to name a few. Filezilla can be downloaded as a Windows application and may be more suitable for people who prefer a Windows environment, rather than the command line.
Note that plain FTP is disabled on viper.hull.ac.uk and SFTP (secure FTP) is the recommended method.
Universal
The following can only be performed on Windows after installing MobaXterm or PuTTY.
The example user is user1; you will need to enter your username here:
| From your local computer to Viper |
| scp myfile user1@viper.hull.ac.uk:myfile |
| rsync -av my-directory viperlogin.hpc.hull.ac.uk:/data/users/YourID |
| From Viper to your local Computer |
| scp user1@viperlogin.hpc.hull.ac.uk:bio/YourUCInetID/hpc-file |
| rsync -av viper.hull.ac.uk:/data/users/YourID/mydatadirectory |
Windows
The following example uses MobaXterm, for transferring large files it is recommended to use FileZilla. For instructions on how to use FileZilla view the Linux section. follow the instructions in the Getting Connected section for installing MobaXterm.
Uploading Files
1. To upload files to Viper, navigate to the directory you wish to upload files to use the MobaXterm file browser and click the 'Upload to current folder' button.
2. Choose the file/files to upload and click open or drag and drop the file/files you wish to upload over the Viper file browser.
3. The upload may take some time depending on upload speed and file size.
Downloading Files
1. To download files from Viper, navigate to the directory using the MobaXterm file browser.
2. Select the file/files you wish to download and click the 'download selected files button'.
3. Choose the directory where you want the files to download and click ok.
4. The download will start, depending on download speed and file size this may take some time.
MacOS
The following example uses Cyberduck, you can download Cyberduck from https://cyberduck.io macOS also supports FileZilla for an example of how to use FileZilla please view the Linux section
Connecting
1. Click Open Connection.
2. From the drop-down menu select SFTP (SSH File Transfer Protocol)
3. Enter server viperlogin.hpc.hull.ac.uk, Port: 22, Username: University UID, Password: university password
4. Click Connect
Uploading Files
1. Click Upload
2. Choose the file/folder you want to upload and click Choose
3. The transfer window will appear
4. Please note due to file size the upload may take some time
Downloading Files
1. From the file browser right-click on the file/folder to download and select download from the menu
2. The transfer window will appear
3. Please note due to file size the download may take some time
Linux
The following example uses FileZilla, you can download FileZilla from https://filezilla-project.org/
1. Enter the Host: viperlogin.hpc.hull.ac.uk, username, password, and port: 22, using an SFTP connection.
2. A pop-up will appear informing that the server host key is unknown. Click Always trust this host and click ok. In the remote site box, your Viper directory will appear.
3. To transfer files click and drag the file/folder from either your pc to Viper (left to right) or from Viper to your pc (right to left). The transfer will start depending on transfer speed and file size this may take some time.
Larger amounts of data (100GB +)
To avoid excessive network congestion and the long period to transfer such data please contact us via the Support Portal for further details. This may typically involve connecting a USB data drive directly to Viper to copy the data quickly into your user area.