UnRAID: Data, Cache, and the Mover

So I’ll say up front, it’s possible that I haven’t set my storage up in the optimum way, and that choosing ‘just’ a 500GB cache drive has caused me some small issues, but I think that in daily operation, things should be fine.

My biggest challenge with the transition to the new machine was always going to be moving the data from old to new, whilst keeping the old one running and serving media.
As it transpired, we’ve had some internet issues over the last week which has meant the Plex server has been inaccessible to the outside world most of the time anyway, but I had already hatched a plan and that was what I stuck to.

I had 4x6TB (not 4x4TB as I said in my first post) in my Windows machine configured in Windows Storage Spaces. Due to the way it was configured, I could only remove one of the disks, despite having just over 1 disk-worth of data.
Therefore I’d need to move everything before I could destroy the array on my Windows machine and move the other disks.

The cache drive was a savior here, both in terms of storage and speed.

The way that the cache drive works in UnRAID is simple and brilliant. The idea is that your array of spinning rust does not need to be constantly spun up as you transfer all of this data at whatever speeds your combination of hardware limits you to.
The data copies across the network to the Cache drive, in my case a fast NVMe drive, and can then be moved to the array at a later date.

This moving is taken care of by a process called the Mover. It does exactly what you’d think – it takes the data from the cache drive and puts it on the array, but the data is always exposed to other devices and dockers from the same share path.

I had the Mover setup to run once a night, but to move all of this data I would need to invoke it more often. Essentially I could fill up to 500GB of space on the cache drive before the transfer would fall over, and I had around 4TB of data to move.

Back to Community Applications and the install of a plugin called User Scripts which, you guessed it, allows you to add user-created scripts to UnRAID and execute them on a schedule.

I found a script on Reddit that did what I needed – checked firstly to see if Mover was already running, then checked the available space on Cache. If it was over 80% (a configurable value), it would run the Mover. The Mover moved data from Cache to Array slightly faster than I could copy to Cache, so it worked out pretty well, though I’ll admit I was on my laptop a lot of the time and could keep an eye on things, and did invoke the Mover by hand a few times.

It took a couple of days, but my data is now fully migrated! As I said, I had just over 1 disk’s worth of data, which means that a chunk of it is sitting on the Cache drive, but that’s not a problem as it’s non-volatile storage. This means that I can power down my Windows box, swap the disks over, then reboot Blackjack and get to work adding them.

Speaking of which, a note that I discovered from SpaceInvaderOne’s excellent Parity video (as linked in a previous post) is to always preclear your new disks. What this does is write 0s to every sector of the disk. That means that when the new disks are added to the array, parity does not have to be recalculated!
This means your parity does not need to spend hours recalculating itself, and your data remains secure throughout.

Now all that’s left to do is to try and migrate my Plex metadata from Windows to docker … let’s see how that turns out …

The Mover Script

#!/bin/bash

# This unRAID bash script will run the mover if the cache drive is over a certain percentage full

# Modify the value to set what percentage is the threshold to start the mover
SETPOINT=80

# Only run script if cache disk enabled and in use (script chunk taken from stock mover script at /usr/local/sbin/mover)
if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If the mover is already running then exit (script chunk taken from stock mover script at /usr/local/sbin/mover - just added the pipe to logger)
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running" | /usr/bin/logger -s -tpercent_mover.sh[$$]
      exit 0
  fi
fi

# Check how full the cache drive is
AMOUNTFULL=$(df | grep /mnt/cache | awk '{ printf "%d", $5 }')

# Simple logic - if setpoint has been met or exceeded, run the mover. If not, don't run.
if [ "$AMOUNTFULL" -ge "$SETPOINT" ]
    then
        echo "Cache is $AMOUNTFULL% full, which is greater than or equal to the set point of $SETPOINT%. Running mover." | /usr/bin/logger -s -tpercent_mover.sh[$$]
        /usr/local/sbin/mover
    else
        echo "Cache is $AMOUNTFULL% full, which is less than the set point of $SETPOINT%. Not running mover." | /usr/bin/logger -s -tpercent_mover.sh[$$]
fi

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s