linux:parallel_rsync
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
parallel_rsync [2012/12/06 17:52] – dodger | linux:parallel_rsync [2022/02/11 11:36] (current) – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== [SCRIPT] psync (parallel rsync) ====== | ||
+ | |||
====== Description ====== | ====== Description ====== | ||
This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers. | This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers. | ||
====== Instructions ====== | ====== Instructions ====== | ||
+ | I suggest you tu launch psync with the following line: | ||
+ | <code bash> | ||
+ | ./psync.sh / | ||
+ | </ | ||
+ | Don't launch it with the FINAL SLASH: | ||
+ | * NOP: < | ||
+ | * YES: ./psync.sh / | ||
+ | |||
+ | |||
===== Pre-Reqs ===== | ===== Pre-Reqs ===== | ||
* gnu screen | * gnu screen | ||
Line 8: | Line 19: | ||
* ssh | * ssh | ||
- | ===== psync ===== | + | ===== psync.sh ===== |
+ | ==== Description ==== | ||
This script will: | This script will: | ||
* Check if the directory to transfer exists | * Check if the directory to transfer exists | ||
Line 16: | Line 29: | ||
* Think that the // | * Think that the // | ||
+ | ==== Code ==== | ||
- | <code> | + | <file bash psync.sh> |
#!/bin/bash | #!/bin/bash | ||
[ ! $1 ] && echo " | [ ! $1 ] && echo " | ||
Line 44: | Line 58: | ||
local let MAXPARALEL=20 | local let MAXPARALEL=20 | ||
- | LAUCHRSYNC=" | + | LAUCHRSYNC=" |
local let y=0 | local let y=0 | ||
for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do | for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do | ||
Line 72: | Line 86: | ||
sync_this ${TARGET} | sync_this ${TARGET} | ||
- | </code> | + | </file> |
==== Script Variables ==== | ==== Script Variables ==== | ||
Line 81: | Line 95: | ||
|< | |< | ||
|< | |< | ||
+ | |||
+ | ===== launch_rsync.sh ===== | ||
+ | ==== Description ==== | ||
+ | This script will: | ||
+ | * Launch rsync non-parallel or parallel | ||
+ | * Log the exit code of rsync to know if everything gones fine or not | ||
+ | |||
+ | |||
+ | ==== Code ==== | ||
+ | <file bash launch_rsync.sh> | ||
+ | #!/bin/bash | ||
+ | # launch_rsync.sh | ||
+ | RECURSIVE=$(echo $1 | tr ' | ||
+ | TARGET=$2 | ||
+ | SCREENNAME=$3 | ||
+ | LOGDIR=$4 | ||
+ | DSTSERVER=" | ||
+ | DESTINATION=" | ||
+ | |||
+ | |||
+ | if [[ " | ||
+ | rsync -cdlptgoDv --partial ${TARGET}/* ${DSTSERVER}: | ||
+ | RES=$? | ||
+ | elif [[ " | ||
+ | rsync -cazv --partial ${TARGET}/* ${DSTSERVER}: | ||
+ | RES=$? | ||
+ | else | ||
+ | echo "$0 -nr|-r|--non-recursive|--recursive" | ||
+ | exit 1 | ||
+ | fi | ||
+ | |||
+ | if [ $RES -eq 0 ] ; then | ||
+ | echo "$RES : ${TARGET}" | ||
+ | else | ||
+ | echo "$RES : ${TARGET}" | ||
+ | fi | ||
+ | </ | ||
+ | ==== Variables ==== | ||
+ | ^ Variable ^ Description ^ | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
linux/parallel_rsync.1354816367.txt.gz · Last modified: 2012/12/06 17:52 by dodger