Script: Migrating data between drives.

Chris - KJ7PPK

My main server at home has a simple 48TB mergerfs pool where my non-critical data is stored. In the past, I’ve just manually run rsync to move data over. This time, I opted to put together a script to make the process a little easier.

Disclaimer: I can’t stand AI, but in the past month or two, I’ve found ChatGPT4 (which I’ve now decided to call SKYNET moving forward) pretty handy for creating quick scripts like this so most of this was generated and sculpted by wasting tremendous environmental resources instead of investing the time to Google and come up with what I wanted by hand. That said, getting a script to perform and appear exactly as you envision still involves a lot of back and forth to get dialed in.

I downed all of my Docker containers that reference the mount point of my mergerfs pool, then used the following script to migrate data from the old 10TB drives to the new 12TB drives. The script offers the user the ability to save a log file, then presents all attached storage devices to the user with assigned numbers. The user chooses the drive(s) they’d like to migrate from, then the drive(s) to migrate to. We’re moving the contents of partitions here, not cloning the disks themselves.

Since my workflow sometimes involves connecting a drive via USB enclosure, the script will also tell you how the drive is connected (SATA, NVME, USB) and run a quick check with smartmontools to give an indicator of the drive’s perceived health status. If the partitions selected are not yet mounted, it will create a temporary mount path for the transfer.

Afterward, I’ll remove the now-empty drives and repopulate the mergerfs pool with the appropriate disk UUIDs, then run a balance & deduplicate and bring my Docker containers back online. One note: This script will only unmount file systems that it mounts during the execution. If you’re migrating to or from a mounted file system, you’ll have to unmount it yourself.

I ended up calling this “DiskHopper” and putting it on GitHub if you prefer to grab it that way. Original script below, just don’t forget to chmod +x!

#!/bin/bash

RSYNC_OPTS="-aAXH --whole-file --inplace --no-compress --info=stats2"
TEMP_MOUNT_BASE="/mnt/migratetemp"
TEMP_MOUNTS=()

LOG_ENABLED=false
LOG_FILE=""

RED='\033[1;31m'
GREEN='\033[1;32m'
YELLOW='\033[1;33m'
BLUE='\033[1;34m'
NC='\033[0m'

cleanup() {
    echo -e "\n${BLUE}Cleaning up temp mounts...${NC}"
    echo -e "${YELLOW}TEMP_MOUNTS to cleanup:${NC} ${TEMP_MOUNTS[*]}"
    
    for mnt in "${TEMP_MOUNTS[@]}"; do
        echo -e "${YELLOW}Unmounting $mnt...${NC}"
        sudo umount "$mnt" 2>/dev/null
        sudo rmdir "$mnt" 2>/dev/null
    done

    if [ -d "$TEMP_MOUNT_BASE" ]; then
        if [ -z "$(ls -A "$TEMP_MOUNT_BASE")" ]; then
            echo -e "${GREEN}✅ Temp mount base $TEMP_MOUNT_BASE is empty. Removing...${NC}"
            sudo rmdir "$TEMP_MOUNT_BASE"
        else
            echo -e "${RED}⚠️ Warning:${NC} $TEMP_MOUNT_BASE is not empty after cleanup."
            echo -e "${RED}⚠️ You may need to investigate or manually clean it up.${NC}"
            echo -e "${YELLOW}Suggested command:${NC} sudo rm -rf $TEMP_MOUNT_BASE/*"
        fi
    fi
}

trap cleanup SIGINT

log() {
    if $LOG_ENABLED; then
        echo -e "$@" >> "$LOG_FILE"
    fi
}

get_os_disks() {
    ROOT_DEV=$(findmnt -n -o SOURCE /)
    EFI_DEV=$(findmnt -n -o SOURCE /boot/efi 2>/dev/null)
    ROOT_DISK=$(lsblk -no PKNAME "$ROOT_DEV")
    EFI_DISK=$(lsblk -no PKNAME "$EFI_DEV" 2>/dev/null)

    echo "$ROOT_DISK $EFI_DISK" | tr ' ' '\n' | sort -u | grep -v '^$'
}

get_smart_status() {
    local disk="$1"
    if smartctl -H "$disk" &>/dev/null; then
        health=$(smartctl -H "$disk" | grep "SMART overall-health" | awk '{print $NF}')
        case "$health" in
            PASSED) echo "✅" ;;
            FAILED) echo "❌" ;;
            *) echo "⚠️" ;;
        esac
    else
        echo "⚠️"
    fi
}

truncate_text() {
    local text="$1"
    local length="$2"
    if [ ${#text} -gt $length ]; then
        echo "${text:0:$(($length - 3))}..."
    else
        printf "%-${length}s" "$text"
    fi
}

list_partitions() {
    OS_DISKS=($(get_os_disks))
    PARTITION_LIST=()

    echo -e "${BLUE}SMART Health Legend:${NC} ✅ Healthy   ⚠️ Unknown   ❌ Failed"
    echo -e "${BLUE}========== Available Partitions (Excluding OS Disk) ==========${NC}"
    printf "${BLUE}%-4s| %-15s| %-6s| %-17s| %-22s| %-15s| %-22s${NC}\n" \
        "#" "Partition" "Size" "Model" "Mount Point" "Usage" "Host Disk (Type)"
    printf "${BLUE}%s\n${NC}" "----|-----------------|--------|-------------------|------------------------|----------------|------------------------"

    local count=1
    while read -r line; do
        eval "$line"
        PARTITION="/dev/$NAME"
        PARENT="/dev/$PKNAME"

        if printf '%s\n' "${OS_DISKS[@]}" | grep -q -w "$PKNAME"; then continue; fi
        if [ -z "$PKNAME" ]; then continue; fi

        TRANSPORT=$(lsblk -dn -o TRAN "$PARENT")
        [ -z "$TRANSPORT" ] && TRANSPORT="unknown"
        SMART_ICON=$(get_smart_status "$PARENT")

        MOUNT_POINT=$(lsblk -nr -o MOUNTPOINT "$PARTITION" | grep '/' || echo "UNMOUNTED")
        SHORT_MOUNT="$MOUNT_POINT"
        if [[ "$MOUNT_POINT" =~ ^/srv/dev-disk-by-uuid- ]]; then
            UUID=$(echo "$MOUNT_POINT" | sed 's|/srv/dev-disk-by-uuid-||')
            SHORT_MOUNT="/srv/uuid-${UUID:0:8}...${UUID: -8}"
        elif [[ ${#MOUNT_POINT} -gt 22 ]]; then
            SHORT_MOUNT="${MOUNT_POINT:0:19}..."
        fi

        PARTITION_SIZE=$(lsblk -n -o SIZE "$PARTITION")

        if [[ "$MOUNT_POINT" != "UNMOUNTED" ]]; then
            USAGE_INFO=$(df -h --output=used,size,pcent "$MOUNT_POINT" | tail -1 | awk '{print $1" / "$2" ("$3")"}')
            MOUNT_COLOR=$GREEN
        else
            USAGE_INFO="N/A"
            MOUNT_COLOR=$YELLOW
        fi

        MODEL=$(udevadm info --query=all --name="$PARENT" | grep "ID_MODEL=" | cut -d= -f2)
        [[ -z "$MODEL" ]] && MODEL="Unknown"

        printf "%-4s| %-15s| %-6s| %-17s| ${MOUNT_COLOR}%-22s${NC}| %-15s| %-22s\n" \
            "$count" "$(truncate_text "$PARTITION" 15)" "$(truncate_text "$PARTITION_SIZE" 6)" \
            "$(truncate_text "$MODEL" 17)" "$(truncate_text "$SHORT_MOUNT" 22)" \
            "$(truncate_text "$USAGE_INFO" 15)" "$(truncate_text "$PARENT (${TRANSPORT^^}) $SMART_ICON" 22)"

        PARTITION_LIST+=("$PARTITION")
        ((count++))
    done < <(lsblk -P -o NAME,SIZE,MODEL,PKNAME | grep -v "loop")

    export PARTITION_LIST
}

select_partitions() {
    local prompt=$1
    local result_var=$2
    read -p "Select $prompt partition numbers (space-separated): " -a selection
    eval "$result_var=(${selection[@]})"
}

ensure_mounted() {
    local partition=$1
    local index=$2
    local temp_mount="$TEMP_MOUNT_BASE/partition$index"

    sudo mkdir -p "$temp_mount"
    sudo mount "$partition" "$temp_mount" || {
        echo -e "${RED}❌ Failed to mount $partition. Exiting.${NC}"
        exit 1
    }

    echo "$temp_mount"
}

migrate_data() {
    local total_bytes_moved=0

    for src in "${SOURCE_MOUNTS[@]}"; do
        echo -e "\n📂 ${BLUE}Processing source:${NC} $src"
        log "Processing source: $src"

        mapfile -t items < <(find "$src" -mindepth 1 -maxdepth 1)

        if [ ${#items[@]} -eq 0 ]; then
            echo -e "${YELLOW}⚠️  No files to transfer in $src${NC}"
            log "No files to transfer in $src"
            continue
        fi

        for item in "${items[@]}"; do
            dst="${DEST_MOUNTS[$((RANDOM % ${#DEST_MOUNTS[@]}))]}"
            echo -e "➡️  ${BLUE}Copying${NC} $(basename "$item") ${BLUE}to${NC} $dst"
            log "Copying $(basename "$item") to $dst"

            RSYNC_OUTPUT=$(rsync $RSYNC_OPTS "$item" "$dst"/)
            result=$?

            echo "$RSYNC_OUTPUT"

            transferred_bytes=$(echo "$RSYNC_OUTPUT" | grep "Total transferred file size:" | awk '{print $5}')

            if [[ "$transferred_bytes" =~ ^[0-9]+$ ]]; then
                total_bytes_moved=$((total_bytes_moved + transferred_bytes))
            fi

            if [ $result -eq 0 ]; then
                echo -e "${GREEN}✅ rsync OK, removing source:${NC} $(basename "$item")"
                log "✅ rsync OK, removing source: $(basename "$item")"
                rm -rf "$item"
            else
                echo -e "${RED}❌ rsync failed:${NC} $item"
                log "❌ rsync failed: $item"
                echo -e "${YELLOW}Cleaning up failed destination...${NC}"
                rm -rf "$dst/$(basename "$item")"
                log "Cleaned up destination: $dst/$(basename "$item")"
            fi
        done
    done

    total_gb_moved=$(awk "BEGIN {printf \"%.2f\", $total_bytes_moved / (1024*1024*1024)}")
    echo -e "\n${BLUE}Total Data Transferred:${NC} ${GREEN}${total_gb_moved} GB${NC}"
    log "Total Data Transferred: ${total_gb_moved} GB"
}

# ===== MAIN =====
clear
echo -e "${BLUE}=== Partition-Based Data Migration Tool ===${NC}"

LOG_DIR="$(pwd)"
LOG_FILE="$LOG_DIR/migration-log-$(date +%F_%H-%M-%S).txt"

echo
echo -e "${BLUE}Enable logging to file? (y/n):${NC}"
echo -e "Logs will be saved to: ${YELLOW}${LOG_FILE}${NC}"
read -p "> " LOG_CHOICE

if [[ "$LOG_CHOICE" == "y" ]]; then
    LOG_ENABLED=true
    echo "Logging to $LOG_FILE"
    echo "Migration Log - $(date)" > "$LOG_FILE"
fi

list_partitions
select_partitions "SOURCE" SOURCE_SELECTION
select_partitions "DESTINATION" DEST_SELECTION

SOURCE_MOUNTS=()
for num in "${SOURCE_SELECTION[@]}"; do
    index=$((num-1))
    partition="${PARTITION_LIST[$index]}"
    mount_point=$(ensure_mounted "$partition" "$index")

    echo -e "${GREEN}Mounted: $partition -> $mount_point${NC}"

    SOURCE_MOUNTS+=("$mount_point")
    TEMP_MOUNTS+=("$mount_point")
    echo -e "${YELLOW}TEMP_MOUNTS now:${NC} ${TEMP_MOUNTS[*]}"
done

DEST_MOUNTS=()
for num in "${DEST_SELECTION[@]}"; do
    index=$((num-1))
    partition="${PARTITION_LIST[$index]}"
    mount_point=$(ensure_mounted "$partition" "$index")

    echo -e "${GREEN}Mounted: $partition -> $mount_point${NC}"

    DEST_MOUNTS+=("$mount_point")
    TEMP_MOUNTS+=("$mount_point")
    echo -e "${YELLOW}TEMP_MOUNTS now:${NC} ${TEMP_MOUNTS[*]}"
done

echo -e "${BLUE}✅ Sources:${NC} ${SOURCE_MOUNTS[*]}"
echo -e "${BLUE}✅ Destinations:${NC} ${DEST_MOUNTS[*]}"
echo -e "${BLUE}✅ Temp mounts to cleanup:${NC} ${TEMP_MOUNTS[*]}"

read -p "Ready to start migration? (y/n): " CONFIRM
if [[ "$CONFIRM" != "y" ]]; then
    echo -e "${RED}Exiting.${NC}"
    cleanup
    exit 0
fi

migrate_data

echo -e "\n${GREEN}✅ Migration complete!${NC}"
log "✅ Migration complete!"

cleanup

Related Posts

One-Liner: Pull details of all attached storage.

I took one Linux Desktop class back before I realized what a waste of time college was. Name something in Windows, I know the commands and shortcuts because years of practical experience > anything in a classroom. In *nix, I find myself googling and when I do have something that works, I save it into […]