User Tools

Site Tools


linux:parallel_rsync

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
parallel_rsync [2012/12/06 17:52] dodgerlinux:parallel_rsync [2022/02/11 11:36] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +====== [SCRIPT] psync (parallel rsync) ======
 +
 ====== Description ====== ====== Description ======
 This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers. This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers.
  
 ====== Instructions ====== ====== Instructions ======
 +I suggest you tu launch psync with the following line:
 +<code bash>
 +./psync.sh /path/to/folder
 +</code>
 +Don't launch it with the FINAL SLASH:
 +  * NOP: <del>./psync.sh /path/to/folder/</del>
 +  * YES: ./psync.sh /path/to/folder
 +
 +
 ===== Pre-Reqs ===== ===== Pre-Reqs =====
   * gnu screen   * gnu screen
Line 8: Line 19:
   * ssh   * ssh
  
-===== psync =====+===== psync.sh ===== 
 +==== Description ==== 
 This script will: This script will:
   * Check if the directory to transfer exists   * Check if the directory to transfer exists
Line 16: Line 29:
   * Think that the //${MAXPARALEL}// is flexible because of the "//sleep 1//" in the "//check_max_processes()//" function.   * Think that the //${MAXPARALEL}// is flexible because of the "//sleep 1//" in the "//check_max_processes()//" function.
  
 +==== Code ====
  
-<code>+<file bash psync.sh>
 #!/bin/bash #!/bin/bash
 [ ! $1 ] && echo "Usage: $0 /path/to/run" && exit 1 [ ! $1 ] && echo "Usage: $0 /path/to/run" && exit 1
Line 44: Line 58:
     local let MAXPARALEL=20     local let MAXPARALEL=20
  
-    LAUCHRSYNC="/root/autosync/launch_rsync.sh"+    LAUCHRSYNC="$(dirname $0)/launch_rsync.sh"
     local let y=0     local let y=0
     for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do     for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do
Line 72: Line 86:
  
 sync_this ${TARGET} sync_this ${TARGET}
-</code>+</file>
  
 ==== Script Variables ==== ==== Script Variables ====
Line 81: Line 95:
 |<code>let MAXPARALEL=20</code> | Maximum number of rsync's launched at a time | |<code>let MAXPARALEL=20</code> | Maximum number of rsync's launched at a time |
 |<code>LAUCHRSYNC="/root/autosync/launch_rsync.sh"</code> | The rsync script itself | |<code>LAUCHRSYNC="/root/autosync/launch_rsync.sh"</code> | The rsync script itself |
 +
 +===== launch_rsync.sh =====
 +==== Description ====
 +This script will:
 +  * Launch rsync non-parallel or parallel
 +  * Log the exit code of rsync to know if everything gones fine or not
 +
 +
 +==== Code ====
 +<file bash launch_rsync.sh>
 +#!/bin/bash
 +# launch_rsync.sh
 +RECURSIVE=$(echo $1 | tr '[[:upper:]]' '[[:lower:]]')
 +TARGET=$2
 +SCREENNAME=$3
 +LOGDIR=$4
 +DSTSERVER="1.1.1.1"
 +DESTINATION="${TARGET}"
 +
 +
 +if [[ "${RECURSIVE}" =~ ^\-{1,2}(nr|non-recursive)$ ]] ; then
 + rsync -cdlptgoDv --partial ${TARGET}/* ${DSTSERVER}:${DESTINATION}/ 2>&1 > ${LOGDIR}/transferlogs/${SCREENNAME}_NOTRECURSIVE.log
 + RES=$?
 +elif [[ "${RECURSIVE}" =~ ^\-{1,2}(r|recursive)$ ]] ; then
 + rsync -cazv --partial ${TARGET}/* ${DSTSERVER}:${DESTINATION}/ 2>&1 > ${LOGDIR}/transferlogs/${SCREENNAME}.log
 + RES=$?
 +else
 + echo "$0 -nr|-r|--non-recursive|--recursive"
 + exit 1
 +fi
 +
 +if [ $RES -eq 0 ] ; then
 + echo "$RES : ${TARGET}" >> ${LOGDIR}/${RECURSIVE//-/}_TRANSFERS.OK
 +else
 + echo "$RES : ${TARGET}" >> ${LOGDIR}/${RECURSIVE//-/}_TRANSFERS.FAIL
 +fi
 +</file>
 +==== Variables ====
 +^ Variable ^ Description ^
 +|<code>RECURSIVE=$(echo $1 | tr '[[:upper:]]' '[[:lower:]]')</code> | Parallel or not, **DON'T MODIFY** |
 +|<code>TARGET=$2</code> | The directory that will be transferred, **DON'T MODIFY** |
 +|<code>SCREENNAME=$3</code> | Name of the screen in which that script is running, **DON'T MODIFY** |
 +|<code>LOGDIR=$4</code> | Where the ressults will be logged, **DON'T MODIFY** |
 +|<code>DSTSERVER="1.1.1.1"</code> | Destination server |
 +|<code>DESTINATION="${TARGET}"</code> | Destination folder, actually is the same of //${TARGET}//, but you will wish to modify it :-) |
  
  
  
linux/parallel_rsync.1354816367.txt.gz · Last modified: 2012/12/06 17:52 by dodger