Demo entry 6356649

bash function, while loop, sed regexps

   

Submitted by Alex on Apr 19, 2017 at 18:36
Language: Bash. Code size: 759 Bytes.

FILTER=".zip"
LIST="s3_list.txt"
BUCKET="s3bucket"

function get_files {
  offset=$1
  for l in {A..Z}; do
    first_line_number=`cat $LIST |grep $FILTER | grep -n "/dr/$l" | head -n1| grep -Eo '^[^:]+'`
    start_line_number=$(($first_line_number+$offset))
    echo $start_line_number
    size=`cat $LIST |grep $FILTER |head -n$start_line_number |tail -n1| awk '{print $3}'`
    total_size=$(($total_size+$size))
    url=`cat $LIST |grep $FILTER |head -n$start_line_number |tail -n1|sed -n 's/.*\/\(.*\)/\1/p'`
    s3cmd get "s3://$BUCKET/$url"
  done
}

total_size=0
limit=$((14*1024*1204*1024))
i=1
while [ $total_size -lt $limit ]; do
  get_files $i
  echo "$i $limit  $total_size"
  let i=i+1
done
echo $(($total_size/1042/1024))

This snippet took 0.00 seconds to highlight.

Back to the Entry List or Home.

Delete this entry (admin only).