up

README.md

@@ -1,3 +1,17 @@
 # rsyncbackup
 
 To backup and synchronize directories with redondancy on multiple support type (nas,computers,usb,cloud) by the most automated way. Push or pull with rsync.
+
+## Requirements
+
+- /datas should contain P1 P2 Pxx with the same user
+- connection with public/private keys (without password)
+
+## banana_push_pull.sh
+
+To synchronize datas between computers. Hosts should be available by ssh resolver (dns or .ssh/config) and rsync should be available installed on two sides.
+
+## rsync_usb*.sh_
+
+To sync mounted keys.
+Copy script to key directory and execute it.

backup_strategy.md

@@ -0,0 +1,97 @@
+# stratégie de classement et de sauvegarde des documents
+
+# type de documents
+
+* critères de tri
+- taille (Ko/Mo/Go)
+- durée de rétention (vie/1y/5y/10y) 
+- lisible en console
+  - oui:
+    - txt
+    - audio
+    - ods
+    - keepass
+  - non:
+    - video
+  - possible:
+    - pdf
+- format tolérant les pertes
+  - oui (gravable):
+    - audio
+    - video
+  - non:
+    - archive (zip/tar/xz/7z/...)
+    - pdf
+- convertible avec perte sous un format archivé
+  - oui:
+    - pdf
+  - non:
+    - 
+
+|-------------------|-----------|-------|-------|--------|--------|-----|-----|-----|----------|----------|-----|--------------|
+| type              | extension | duree | perso | repro  | taille | cdr | usb | cli | disaster | rotation | maj | compressible |
+|-------------------|-----------|-------|-------|--------|--------|-----|-----|-----|----------|--------- |-----|--------------|
+| keepass           | kbdx      |  vie  |  oui  | oui    | small  | non | oui | oui | oui      | oui      | oui | non          |
+| paie              | pdf       |  vie  |  oui  | non    | small  | oui | oui | oui | oui      | non      | oui | non          |
+| scans officiel    | png/pdf   |  vie  |  oui  | oui    | small  | oui | oui | non | oui      | non      | oui | non          |
+| releves banques   | pdf       |  10y  |  oui  | oui    | middle | oui | oui | oui | non      | oui      | oui | non          |
+| video youtube     | mp4       |  -    |  non  | oui    | big    | non | non | non | non      | non      | non | non          |
+| video famille     | mp4       |  vie  |  oui  | non    | big    | oui | non | non | oui      | non      | oui | non          |
+| video entourage   | mp4       |  vie  |  oui  | non    | big    | oui | non | non | non      | non      | oui | non          |
+| video idiotes     | mp4       |  -    |  oui  | oui    | big    | oui | non | non | non      | non      | oui | non          |
+| video osm         | mp4       |  -    |  oui  | oui    | big    | non | non | non | non      | non      | maj | non          |
+| video artois      | mp4       |  -    |  oui  | non    | big    | oui | non | non | non      | non      | non | non          |
+| video copie       | mp4       |  -    |  non  | oui    | big    | non | non | non | non      | non      | non | non          |
+| video anime       | mp4       |  vie  |  non  | non    | big    | oui | non | non | non      | non      | non | non          |
+| video spectacle   | mp4       |  10y  |  non  | non    | big    | oui | non | non | non      | non      | non | non          |
+| books pdf study   | pdf       |  10y  |  non  | non    | middle | oui | non | oui | non      | non      | non | non          |
+| books epub study  | epub      |  10y  |  non  | non    | middle | oui | non | oui | non      | non      | non | non          |
+| audiobooks        | ogg       |  10y  |  non  | non    | big    | oui | non | oui | non      | non      | non | non          |
+| music originaux   | mp3/wma   |  10y  |  non  | non    | middle | oui | non | oui | non      | non      | non | non          |
+| music dump cd     | flac/...  |  -    |  non  | oui    | middle | oui | non | oui | non      | non      | non | non          |
+| iso winx cds      | iso.xz    |  10y  |  oui  | non    | big    | non | non | non | non      | non      | non | oui          |
+|                   |           |       |       |        |        |     |     |     |          |          |     |              |
+|                   |           |       |       |        |        |     |     |     |          |          |     |              |
+|                   |           |       |       |        |        |     |     |     |          |          |     |              |
+|                   |           |       |       |        |        |     |     |     |          |          |     |              |
+
+# matrice des prioritées
+
+|-----|-------|-------|--------|----------|-----|
+| num | duree | perso | taille | archives | cdr |
+|-----|-------|-------|--------|----------|-----|
+| P1  |  vie  |  oui  | faible |    non   | non |
+| P2  |  vie  |  oui  | moyen  |    oui   | non |
+| P3  |  vie  |  oui  | eleve  |    oui   | non |
+| P4  |  vie  |  oui  | eleve  |    non   | oui |
+| P5  |  vie  |  oui  | moyen  |    non   | non |
+| P6  |  10y  |  non  |        |          | oui |
+| P7  |  10y  |  non  |        |          | oui |
+| P8  |  -    |       |        |          | non |
+| P9  |  -    |       |        |          | oui |
+|-----|-------|-------|--------|----------|-----|
+
+
+# tailles disques cibles
+
+|----------------|----------|----|----|----|----|----|-----|-----|-----|
+| volume         | taille   | P1 | P2 | P3 | P4 | P5 | P6  | P7  | P8  |
+|----------------|----------|----|----|----|----|----|-----|-----|-----|
+|                | cible    | 4  | 4  | 8  | 16 | 32 | 64  | 128 | 256 |
+|----------------|----------|----|----|----|----|----|-----|-----|-----|
+|                | agregats |    | 8  | 16 | 32 | 64 | 128 | 256 | 512 |
+|----------------|----------|----|----|----|----|----|-----|-----|-----|
+|                | replicas | 9  | 8  | 7  | 6  | 5  | 4   | 3   | 2   |
+|----------------|----------|----|----|----|----|----|-----|-----|-----|
+| cle usb 4Go    | 4G       | x  |    |    |    |    |     |     |     |
+| cle usb 8Go    | 8G       | x  | x  |    |    |    |     |     |     |
+| cle usb 16Go   | 16G      | x  | x  | x  |    |    |     |     |     |
+| cle usb 32Go   | 32G      | x  | x  | x  | x  |    |     |     |     |
+| cle usb 64Go   | 64G      | x  | x  | x  | x  | x  |     |     |     |
+| cle usb 128G   | 128G     | x  | x  | x  | x  | x  | x   |     |     |
+| noisette home  | 256G     | x  | x  | x  | x  | x  | x   | x   |     |
+| externe 500Go  | 512G     | x  | x  | x  | x  | x  | x   | x   | x   |
+| vg seagate 1To | 950G     | x  | x  | x  | x  | x  | x   | x   | x   |
+|----------------|----------|----|----|----|----|----|-----|-----|-----|
+
+

banana_push_pull.sh

@@ -0,0 +1,93 @@
+#!/bin/bash
+
+action=$(basename $0)
+declare -a dirlist
+logfile=/tmp/push_pull.log
+rsync_options="--archive --perms --compress --delete-during --exclude=*.git* --exclude=git.code-workspace"
+
+case "$(hostname)" in
+  pistache|noix|noisette)
+    dirlist=( /datas/workspace /datas/git /datas/P1 /datas/P2 /datas/P3 /datas/P4 /datas/P5 /datas/P6 /datas/P7 /datas/rebuild )
+    ;;
+  queensland)
+    dirlist=( /datas/workspace /datas/git /datas/P1 /datas/P2 /datas/P3 /datas/P4 /datas/P5 /datas/P6 )
+    ;;
+  pecan)
+    dirlist=( /datas/workspace /datas/git /datas/P1 /datas/P2 /datas/P3 /datas/P4 /datas/P5 )
+    ;;
+  eee)
+    dirlist=( /datas/workspace /datas/git /datas/P1 /datas/P2 /datas/P3 )
+    ;;
+  mllse)
+    dirlist=( /datas/P1/KEEPASS /datas/P1/PARENTS /datas/P2/*_THUOT /datas/P6/*_THUOT )
+    ;;
+  *)
+    dirlist=( /datas/worspace )
+    ;;
+esac	
+
+# dirlist=( /datas/P1 /datas/P2 /datas/P3 /datas/P4 /datas/P5 /datas/P6 /datas/P7 ) 
+
+ask() {
+    USER_ANSWER="?";
+    while [ "x$USER_ANSWER" != "xno" -a "x$USER_ANSWER" != "xnon" -a "x$USER_ANSWER" != "xyes" -a "$USER_ANSWER" != "xoui" ] ; do
+        echo -n "$1 $2 $3 ? [yes/no] "
+        read -n 3 USER_ANSWER
+        echo "";
+    done
+    if [ "x$USER_ANSWER" == "xyes" -o "x$USER_ANSWER" == "xoui" ] ; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+echo "Snapshot git repositories (without history)"
+
+case "$(hostname)" in
+    noix)
+  	[ -e ${logfile} ] && rm ${logfile}
+        rsync --dry-run ${rsync_options} --exclude=*.terraform* --out-format="%B %l %M %f" /datas/git/* /datas/P1/git/ |tee ${logfile}
+        if [ -s ${logfile} ]
+	      then
+	        if ask snapshot git repo /datas/git /datas/P1/git
+	        then
+            rsync --verbose ${rsync_options} --exclude=*.terraform* /datas/git/* /datas/P1/git/
+	        fi
+	      fi
+    ;;
+esac
+
+echo "List of files differences from master"
+
+for directory in ${dirlist[@]}
+do
+  echo "check ${directory}"
+  [ -w ${directory} ] || ( echo "${directory} is not writable" && exit 3 )
+  [ -e ${logfile} ] && rm ${logfile}
+  case "${action}" in
+    pull) 
+      rsync --dry-run ${rsync_options} --out-format="%B %l %M %f" banana:${directory}/ ${directory} |tee ${logfile}
+      [ -s ${logfile} ] || continue
+      if ask rsync banana:${directory}/ ${directory} 
+      then
+        echo "rsync --verbose ${rsync_options} banana:${directory}/ ${directory}"
+        rsync --verbose ${rsync_options} banana:${directory}/ ${directory}
+      fi
+      ;;
+    push) 
+      rsync --dry-run ${rsync_options} --out-format="%B %l %M %f" ${directory}/ banana:${directory} |tee ${logfile}
+      [ -s ${logfile} ] || continue
+      if ask rsync ${directory}/ banana:${directory} 
+      then
+        echo "rsync --verbose ${rsync_options} ${directory}/ banana:${directory}"
+        rsync --verbose ${rsync_options} ${directory}/ banana:${directory}
+      fi
+      ;;
+    *)
+      echo "command ${command} is unknown ... push or pull ??"
+      exit 2
+      ;;
+  esac
+done
+

export_bookmarks.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+backupdirectory=/datas/P1/BOOKMARKS
+bookmarksexportdir=/datas/git/bookmarksexport
+
+export_bookmark()
+{
+  echo "export ${browser}"
+  browser=$1
+  lastfile=$(find ${backupdirectory} -name "2[0-9]*_${browser}_bookmarks.html" -print |sort -n | tail -n 1)
+  lastfilechecksum=$(sha512sum "${lastfile}" |cut -d' ' -f 1)
+  python3 ${bookmarksexportdir}/bookmark_exporter.py --browser "${browser}" --formats html --output /tmp --prefix bookmark
+  lastexport=$(find /tmp -name "bookmark_2[0-9_]*" -print 2>/dev/null |sort -n |tail -n 1)
+  lastexportchecksum=$(sha512sum "${lastexport}" |cut -d' ' -f 1)
+  echo "compare ${lastexportchecksum} == ${lastfilechecksum}"
+  if [ "${lastexportchecksum}" == "${lastfilechecksum}" ]
+  then
+    echo "${browser} bookmarks not modified"
+  else
+    echo mv -i "${lastexport}" ${backupdirectory}/"$(date +%Y%m%d%H%M%S)_${browser}_bookmarks.html"
+  fi
+}
+
+for browser in Firefox Chromium Chrome
+do
+  export_bookmark ${browser}
+  sleep 3
+done

git_prepare_and_push.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+source=/datas/git
+destination=/datas/git.ccyr.eu
+
+echo "check diff from ${source} to ${destination}"
+[ -e /tmp/rsync_push.log ] && rm /tmp/rsync_push.log
+rsync -anpz --delete-after --exclude=*.git* --out-format="%B %l %M %f" ${source}/ ${destination}  |tee /tmp/rsync_push.log
+[ -s /tmp/rsync_push.log ] && echo "to push to distant, copy following command : \
+rsync -apvz --delete-after --exclude=*.git* --out-format=\"%B %l %M %f\" ${source}/ ${destination}"
+
+commit_repository()
+{
+  echo "check repository $1"
+  cd ${destination}/$1
+  git status
+}
+
+
+for repository in ansible computers financial genealogie
+do
+  commit_repository ${repository}
+done

mtp_put_directory.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# execute script from gvfs directory
+
+directory=$1
+
+# directory target name
+destination=$(basename "${directory}")
+
+
+echo "liste des fichiers du dossier ${directory}"
+for item in "${directory}"/*
+do
+    filename=$(basename "${item}")
+    echo "examen du fichier ${filename}"
+    if [ -e "${destination}"/"${filename}" ]
+    then
+        echo "${item} already on"
+    else
+        echo "copie du fichier ${filename}"
+        echo "cp --verbose ${item} ${destination}/${filename}"
+        cp --verbose "${item}" "${destination}"/"${filename}"
+    fi
+done
+
+# vim: tabstop=4 expandtab shiftwidth=4 softtabstop=4 foldmethod=indent

rsync_usb128go.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+for directory in "P1" "P2" "P3" "P4" "P5" "P6"
+do
+rsync -avzp --delete-during /datas/${directory}/ ./${directory}/
+done

rsync_usb64go.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+for directory in "P1" "P2" "P3" "P4" "P5"
+do
+rsync -avzp --delete-during /datas/${directory}/ ./${directory}/
+done