User Tools

Site Tools


linux:parallel_rsync

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
parallel_rsync [2012/12/06 17:55] dodgerlinux:parallel_rsync [2022/02/11 11:36] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +====== [SCRIPT] psync (parallel rsync) ======
 +
 ====== Description ====== ====== Description ======
 This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers. This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers.
  
 ====== Instructions ====== ====== Instructions ======
 +I suggest you tu launch psync with the following line:
 +<code bash>
 +./psync.sh /path/to/folder
 +</code>
 +Don't launch it with the FINAL SLASH:
 +  * NOP: <del>./psync.sh /path/to/folder/</del>
 +  * YES: ./psync.sh /path/to/folder
 +
 +
 ===== Pre-Reqs ===== ===== Pre-Reqs =====
   * gnu screen   * gnu screen
Line 20: Line 31:
 ==== Code ==== ==== Code ====
  
-<code>+<file bash psync.sh>
 #!/bin/bash #!/bin/bash
 [ ! $1 ] && echo "Usage: $0 /path/to/run" && exit 1 [ ! $1 ] && echo "Usage: $0 /path/to/run" && exit 1
Line 47: Line 58:
     local let MAXPARALEL=20     local let MAXPARALEL=20
  
-    LAUCHRSYNC="/root/autosync/launch_rsync.sh"+    LAUCHRSYNC="$(dirname $0)/launch_rsync.sh"
     local let y=0     local let y=0
     for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do     for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do
Line 75: Line 86:
  
 sync_this ${TARGET} sync_this ${TARGET}
-</code>+</file>
  
 ==== Script Variables ==== ==== Script Variables ====
Line 86: Line 97:
  
 ===== launch_rsync.sh ===== ===== launch_rsync.sh =====
 +==== Description ====
 +This script will:
 +  * Launch rsync non-parallel or parallel
 +  * Log the exit code of rsync to know if everything gones fine or not
 +
 +
 ==== Code ==== ==== Code ====
-<code>+<file bash launch_rsync.sh>
 #!/bin/bash #!/bin/bash
 # launch_rsync.sh # launch_rsync.sh
Line 114: Line 131:
  echo "$RES : ${TARGET}" >> ${LOGDIR}/${RECURSIVE//-/}_TRANSFERS.FAIL  echo "$RES : ${TARGET}" >> ${LOGDIR}/${RECURSIVE//-/}_TRANSFERS.FAIL
 fi fi
-</code> +</file> 
 +==== Variables ==== 
 +^ Variable ^ Description ^ 
 +|<code>RECURSIVE=$(echo $1 | tr '[[:upper:]]' '[[:lower:]]')</code> | Parallel or not, **DON'T MODIFY** | 
 +|<code>TARGET=$2</code> | The directory that will be transferred, **DON'T MODIFY** | 
 +|<code>SCREENNAME=$3</code> | Name of the screen in which that script is running, **DON'T MODIFY** | 
 +|<code>LOGDIR=$4</code> | Where the ressults will be logged, **DON'T MODIFY** | 
 +|<code>DSTSERVER="1.1.1.1"</code> | Destination server | 
 +|<code>DESTINATION="${TARGET}"</code> | Destination folder, actually is the same of //${TARGET}//, but you will wish to modify it :-) |
  
  
  
linux/parallel_rsync.1354816509.txt.gz · Last modified: 2012/12/06 17:55 by dodger