Ciberterminal Wiki

User Tools

Site Tools

Trace:
linux:parallel_rsync

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
parallel_rsync [2012/12/06 17:54] – dodgerlinux:parallel_rsync [2019/07/29 12:11] – [Code] dodger
Line 1: Line 1:
 +====== [SCRIPT] psync (parallel rsync) ======
 +
 ====== Description ====== ====== Description ======
 This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers. This set of scripts will parallelize the transfer of a huge directory tree keeping in mind a maximum number of simultaneous transfers.
  
 ====== Instructions ====== ====== Instructions ======
 +I suggest you tu launch psync with the following line:
 +<code bash>
 +./psync.sh /path/to/folder
 +</code>
 +Don't launch it with the FINAL SLASH:
 +  * NOP: <del>./psync.sh /path/to/folder/</del>
 +  * YES: ./psync.sh /path/to/folder
 +
 +
 ===== Pre-Reqs ===== ===== Pre-Reqs =====
   * gnu screen   * gnu screen
Line 9: Line 20:
  
 ===== psync.sh ===== ===== psync.sh =====
 +==== Description ====
 +
 This script will: This script will:
   * Check if the directory to transfer exists   * Check if the directory to transfer exists
Line 16: Line 29:
   * Think that the //${MAXPARALEL}// is flexible because of the "//sleep 1//" in the "//check_max_processes()//" function.   * Think that the //${MAXPARALEL}// is flexible because of the "//sleep 1//" in the "//check_max_processes()//" function.
  
 +==== Code ====
  
-<code>+<file bash psync.sh>
 #!/bin/bash #!/bin/bash
 [ ! $1 ] && echo "Usage: $0 /path/to/run" && exit 1 [ ! $1 ] && echo "Usage: $0 /path/to/run" && exit 1
Line 44: Line 58:
     local let MAXPARALEL=20     local let MAXPARALEL=20
  
-    LAUCHRSYNC="/root/autosync/launch_rsync.sh"+    LAUCHRSYNC="$(dirname $0)/launch_rsync.sh"
     local let y=0     local let y=0
     for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do     for FOLDER in $(find ${TARGET} -mindepth ${MAXDEPTH} -maxdepth ${MAXDEPTH} -type d) ; do
Line 72: Line 86:
  
 sync_this ${TARGET} sync_this ${TARGET}
-</code>+</file>
  
 ==== Script Variables ==== ==== Script Variables ====
Line 83: Line 97:
  
 ===== launch_rsync.sh ===== ===== launch_rsync.sh =====
 +==== Description ====
 +This script will:
 +  * Launch rsync non-parallel or parallel
 +  * Log the exit code of rsync to know if everything gones fine or not
 +
 +
 ==== Code ==== ==== Code ====
-<code>+<file bash launch_rsync.sh>
 #!/bin/bash #!/bin/bash
 # launch_rsync.sh # launch_rsync.sh
Line 111: Line 131:
  echo "$RES : ${TARGET}" >> ${LOGDIR}/${RECURSIVE//-/}_TRANSFERS.FAIL  echo "$RES : ${TARGET}" >> ${LOGDIR}/${RECURSIVE//-/}_TRANSFERS.FAIL
 fi fi
-</code> +</file> 
 +==== Variables ==== 
 +^ Variable ^ Description ^ 
 +|<code>RECURSIVE=$(echo $1 | tr '[[:upper:]]' '[[:lower:]]')</code> | Parallel or not, **DON'T MODIFY** | 
 +|<code>TARGET=$2</code> | The directory that will be transferred, **DON'T MODIFY** | 
 +|<code>SCREENNAME=$3</code> | Name of the screen in which that script is running, **DON'T MODIFY** | 
 +|<code>LOGDIR=$4</code> | Where the ressults will be logged, **DON'T MODIFY** | 
 +|<code>DSTSERVER="1.1.1.1"</code> | Destination server | 
 +|<code>DESTINATION="${TARGET}"</code> | Destination folder, actually is the same of //${TARGET}//, but you will wish to modify it :-) |
  
  
  
linux/parallel_rsync.txt · Last modified: 2022/02/11 11:36 by 127.0.0.1