linux:parallel_rsync
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
parallel_rsync [2012/12/06 17:48] – dodger | linux:parallel_rsync [2022/02/11 11:36] (current) – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== [SCRIPT] psync (parallel rsync) ====== | ||
+ | |||
====== Description ====== | ====== Description ====== | ||
This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers. | This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers. | ||
====== Instructions ====== | ====== Instructions ====== | ||
+ | I suggest you tu launch psync with the following line: | ||
+ | <code bash> | ||
+ | ./psync.sh / | ||
+ | </ | ||
+ | Don't launch it with the FINAL SLASH: | ||
+ | * NOP: < | ||
+ | * YES: ./psync.sh / | ||
+ | |||
+ | |||
===== Pre-Reqs ===== | ===== Pre-Reqs ===== | ||
* gnu screen | * gnu screen | ||
Line 8: | Line 19: | ||
* ssh | * ssh | ||
- | ===== psync ===== | + | ===== psync.sh ===== |
+ | ==== Description ==== | ||
This script will: | This script will: | ||
* Check if the directory to transfer exists | * Check if the directory to transfer exists | ||
Line 14: | Line 27: | ||
* Parallel Transfer of the upper directories from deep 1 to deep // | * Parallel Transfer of the upper directories from deep 1 to deep // | ||
* Parallel transfer of the directories at deep // | * Parallel transfer of the directories at deep // | ||
- | * Think that the // | + | * Think that the // |
+ | ==== Code ==== | ||
- | <code> | + | <file bash psync.sh> |
#!/bin/bash | #!/bin/bash | ||
[ ! $1 ] && echo " | [ ! $1 ] && echo " | ||
Line 44: | Line 58: | ||
local let MAXPARALEL=20 | local let MAXPARALEL=20 | ||
- | LAUCHRSYNC=" | + | LAUCHRSYNC=" |
local let y=0 | local let y=0 | ||
for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do | for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do | ||
Line 72: | Line 86: | ||
sync_this ${TARGET} | sync_this ${TARGET} | ||
- | </ | + | </ |
+ | |||
+ | ==== Script Variables ==== | ||
+ | ^ Variable ^ Description ^ | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |||
+ | ===== launch_rsync.sh ===== | ||
+ | ==== Description ==== | ||
+ | This script will: | ||
+ | * Launch rsync non-parallel or parallel | ||
+ | * Log the exit code of rsync to know if everything gones fine or not | ||
+ | |||
+ | |||
+ | ==== Code ==== | ||
+ | <file bash launch_rsync.sh> | ||
+ | # | ||
+ | # launch_rsync.sh | ||
+ | RECURSIVE=$(echo $1 | tr ' | ||
+ | TARGET=$2 | ||
+ | SCREENNAME=$3 | ||
+ | LOGDIR=$4 | ||
+ | DSTSERVER=" | ||
+ | DESTINATION=" | ||
+ | |||
+ | |||
+ | if [[ " | ||
+ | rsync -cdlptgoDv --partial ${TARGET}/* ${DSTSERVER}: | ||
+ | RES=$? | ||
+ | elif [[ " | ||
+ | rsync -cazv --partial ${TARGET}/* ${DSTSERVER}: | ||
+ | RES=$? | ||
+ | else | ||
+ | echo "$0 -nr|-r|--non-recursive|--recursive" | ||
+ | exit 1 | ||
+ | fi | ||
+ | |||
+ | if [ $RES -eq 0 ] ; then | ||
+ | echo "$RES : ${TARGET}" | ||
+ | else | ||
+ | echo "$RES : ${TARGET}" | ||
+ | fi | ||
+ | </ | ||
+ | ==== Variables ==== | ||
+ | ^ Variable ^ Description ^ | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |< | ||
+ | |||
+ |
linux/parallel_rsync.1354816094.txt.gz · Last modified: 2012/12/06 17:48 by dodger