Thursday, March 28, 2013

Cloud Backup on QNAP NAS

After finding Symform a bit too slow for my needs I decided to move to Amazon S3 to give them a go and also to try out some more simple scripting.

QNAP does have a web interface for backing up to S3, but it is a bit buggy and did not work for me. Looking in the forums people complain of issues with files that are too big or trying to copy too many files. Anyhow this is where s3cmd comes in. 

Written in python s3cmd is a command line tool that can be used to upload / sync files to S3.

Installing this tool on a QNAP is covered by

http://bit.ly/Zuuhou summarised by natebeaty, http://bit.ly/Z2Lzch

1. Install Python from the QNAP admin QPKG Center
2. Download latest s3cmd and transfer to a QNAP share directory
3. SSH in to your QNAP box
4. python s3cmd --configure
5. write shell scripts, add to crontab, etc

The original guide mentions that the config file is deleted on reboot, so need to check that out

scrip file; with your backup its important to know the system is working and your files are being backed up. So a log file is essential, and why not have this notified by email as well. all this can be achieved with scripts.


2 comments:

  1. Hello Andrew, Thanks for your blog article it's really nice and give me a good idea on how to start off. I have the same situation as you do, and I'm trying to get the s3cmd to work on my TS-219P.

    I got this one weird problem which I was wondering if you have experienced. If you could help me I would really be appreciated!

    Right now, I got everything to work (on command line), even running the script file (s3-backup.sh) on command line:

    #!/bin/bash

    /share/maintenance/s3cmd-1.5.0-rc1/s3cmd --rr sync -rv /share/all-shared-folders/emilie/ s3://kingjim-backup/kingjim-nas/emilie/ >> /share/maintenance/log/s3cmd/backup_`date "+%Y%m%d-%H-%M"`.log

    ==========================================

    The problem though, is the cronjob. I can confirm it was run, because my log file (the one above) was generated, but it was empty.

    This is my cronjob task:
    14 3 * * * /share/maintenance/s3-backup.sh 2>&1 | logger

    ==========================================

    I've done a number of different variations on the above, but just couldn't find out what was missing.

    I fee like some dependency is missing when the crontab is running, as compared to when I run it on command prompt.

    It's just so funny, so I was seeing if you could shed me some light.

    Thank you very much in advance!!

    Cheers,
    Thomas

    ReplyDelete
    Replies
    1. I gave up on the scripted approach with S3 once google drive came out with their cheaper cloud storage rates. the script would often fail for some reason or another.

      There is a qnap app for replicating data to google drive which is fairly robust and you can setup a schedule.

      As for the non-functional script in your crontab I'm not too sure. Are you sure the cron run job has the permissions and passwords to access your bucket, I seam to remember I had to copy the config each time.

      I'm not familiar with the last bit of the command "2>&1 | logger" what does this do exactly? I put all the loggin into the scrip file itself.

      Here is a copy of my scrip that I used to use, maybe this can help you?

      ! /bin/sh
      PSUM=photo_summary_$(date +%F).log
      PATH=$PATH:/share/homes/andy/s3cmd
      cp /share/homes/andy/s3cmd/.s3cfg ~

      echo "[Photos] Sync Started" > ${PSUM}
      date >> ${PSUM}
      echo >> ${PSUM}

      s3cmd sync --skip-existing /share/Multimedia/Photos/Archive s3://qnap-photos/ >> ${PSUM}
      s3cmd sync --skip-existing /share/Multimedia/Photos/Camera\ Roll s3://qnap-photos/ >> ${PSUM}
      s3cmd sync --skip-existing /share/Multimedia/Photos/Photostream s3://qnap-photos/ >> ${PSUM}


      echo >> ${PSUM}

      echo "[Photos] Sync Ended" >> ${PSUM}
      date >> ${PSUM}
      echo >> ${PSUM}

      echo "[Videos] Sync Started" >> ${PSUM}
      date >> ${PSUM}
      echo >> ${PSUM}

      s3cmd sync --skip-existing /share/Multimedia/Videos s3://qnap-videos/ >> ${PSUM}
      echo >> ${PSUM}

      echo "[Videos] Sync Ended" >> ${PSUM}
      date >> ${PSUM}


      echo "Subject: Amazon S3 Sync Log" > mail.txt
      echo "From: QNAP " >> mail.txt
      echo "To: Andrew Wingate " >> mail.txt
      echo >> mail.txt

      cat ${PSUM} >> mail.txt

      sendmail myemail@gmail.com < mail.txt

      Delete