General/Data Backup

From HPC
(Redirected from Training/Data Backup)
Jump to: navigation , search

Introduction

Although at present there are not any official methods for backing up your home directory, there is a mitigation in the form of an in-house backup script which uses your box.com account.

The programs are called:

Program Description
viper2box This compresses the current directory and then sending the compressed file up to your personal box account provided by the University.
box2viper This shows the current backups on box.com and allows you to select one to download to Viper directly


Prerequisites

SSO login credentials

viper2box (and box2viper) does not support SSO login credentials and so you will need to create a Box specific password to supplement your SSO login.

  • Log into box.com with your university credentials

Sso-viper1.jpg

  • Select Account Settings from the drop-down menu on the top right hand side
  • Move down to the Create External Password as below and enter your password to allow this. (For ease this can be the same standard password ).

Sso-viper2.jpg

Usage

viper2box

  • By moving to the directory you require to be backed up, you can then type viper2box to start the process as below:



[pysdlb@login01 DEVELOPMENT]$ viper2box

>>> viper2box starting...
>>> Please wait performing analysis on this directory
>>> The requested directory size is 85 M and this could take 5 minutes to complete

>>> Your box account Darren Bird is set to [mailto:d.bird@hull.ac.uk d.bird@hull.ac.uk]
<<< Type your password or press just return to abort :
>>> 1. Compressing to backup-home-pysdlb-DEVELOPMENT-2019-6-19.tar.gz...
>>> 2. Compression finished
>>> 3. Checking compressed file
>>> 4. Sending up to box.com
>>> 5. Removing temporary file
>>> 6. Upload took 10 seconds
>>> 7. Emailing you the results
>>> 8. All done...

[pysdlb@login01 DEVELOPMENT]$


  • The indicated file backup-home-327198-DEVELOPMENT-2017-3-14.tar.gz will be placed into ViperBackups on box.com.


box2viper

  • By moving to the directory you require the backup to be placed, you can then type box2viper to start the process as below:

[user@login01 ~]$ box2viper

>>> box2viper starting...
>>> Your box account is set to [mailto:d.duck@hull.ac.uk d.duck@hull.ac.uk]
Type your password or press just return to abort : >>> Querying box.com

1)      backup-etc-cron.daily-2017-3-13.tar.gz
2)      backup-home-321321-CODE_SAMPLES-2017-2-13.tar.gz
3)      backup-home-321321-CUDA1-2017-2-13.tar.gz
4)      backup-home-321321-CUDA2-2017-2-13.tar.gz
5)      backup-home-321321-CUDA3-2017-2-13.tar.gz
6)      backup-home-321321-DEVELOPMENT-2017-3-13.tar.gz
7)      backup-home-321321-DEVELOPMENT-2017-3-14.tar.gz
8)      backup-home-321321-PSY-Stuff-2017-2-13.tar.gz
9)      backup-home-321321-TENSORFLOW-2017-3-13.tar.gz
10)     backup-var-www-2017-3-13.tar.gz

Type in which file to download or blank to abort : 10
>>> Downloading -> backup-var-www-2017-3-13.tar.gz
>>> Download complete
>>> (Use 'tar -xvjf backup-var-www-2017-3-13.tar.gz' to expand this file)
>>> Connection closed
[user@login01 ~]$

In the above example you can uncompress the file backup-var-www-2017-3-13.tar.gz by the following command:


[user@login01 ~]$ tar -xvjf backup-var-www-2017-3-13.tar.gz

drwxr-xr-x pysdlb/py         0 2017-01-23 14:21 ./Tensorflow-learn/
-rw-r--r-- pysdlb/py      8995 2016-12-01 16:01 ./Tensorflow-learn/word2vec.py

[user@login01 ~]$

Icon warning.png
  • The tar file can be moved to any other directory and be uncompressed there.
  • Within Linux use the command tar -xvjf <file_name.tar.gz>
  • Within Windows use the application 7-zip which can handle this type of compressed file.
  • Beware that uncompressing the file may overwrite existing files in that directory. Uncompress in a different folder

Limitations

Icon exclam3.png
  • The main issue here is the speed of box.com in reading and writing files, so backing up large directories (and their sub-directories) may take some time. The utility viper2box does mitigate this by compressing the directory quickly by using some of the multi core processing power of the login node and is faster than just using utilities like Filezilla (SFTP).
  • box.com operate a 2 TeraByte/month limit for Business/Enterprise users like the University of Hull. This does not include downloads from within an account by the content owner and by collaborators while logged in to their account.
  • box.com has a 15G byte file limit.

Further Information

Navigation