diff --git a/know-how/hardware/_posts/2017-05-28-speed-up-synology-reshape.md b/know-how/hardware/_posts/2017-05-28-speed-up-synology-reshape.md new file mode 100644 index 0000000..006259d --- /dev/null +++ b/know-how/hardware/_posts/2017-05-28-speed-up-synology-reshape.md @@ -0,0 +1,149 @@ +--- +title: Speed up Synology volume reshaping +layout: default +language: en +created: 2017-05-28 23:45:40 +0200 +updated: 2017-05-28 23:45:40 +0200 +toc: false +tags: + - know-how + - hardware + - synology + - diskstation + - dsm + - raid + - lvm + - reshape + - storage +--- +I wanted to increase the available storage on my Synology DS415+ by replacing 2x 2TB disks with +4TB ones, changing my 4-4-2-2 configuration to 4-4-4-4 and the overall storage space from 8TB to +12TB. + +After replacing the first disk, there was the data reconstruction which finished after 2 hours +or so. But then came the reshaping which I left running over night (~10 hours) and it still only +got to 20%. + +That made me wonder how long the overall process might take and if that is normal. + +Google brought me to [this](https://forum.synology.com/enu/viewtopic.php?t=79595) and +[this](https://forum.synology.com/enu/viewtopic.php?t=124489) forum post which suggested this +command for checking the status and estimated runtime: + +``` +root@DiskStation:~# cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] +md3 : active raid5 sdd6[2] sda6[0] sdb6[1] + 1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU] + [=====>...............] reshape = 25.1% (491263360/1953494784) finish=5198.4min speed=4687K/sec + +md2 : active raid5 sdd5[4] sda5[0] sdc5[5] sdb5[1] + 5846049792 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU] + +md1 : active raid1 sdd2[3] sda2[0] sdb2[1] sdc2[2] + 2097088 blocks [4/4] [UUUU] + +md0 : active raid1 sdd1[3] sda1[0] sdb1[1] sdc1[2] + 2490176 blocks [4/4] [UUUU] + +unused devices: +``` + +As you can see, for me it showed about 5200 minutes left. That would be almost 87 hours, or a +little bit over 3½ days. Given that some people reported runtimes of 16 days, I assumed I was +lucky already. + +However, the `speed` indicator of the above command showed values around 4-6 MB/sec which is +rather slow for hard drives capable of 100 MB/sec and more. + +People mentioned running services on the DiskStation to be a problem, especially indexing services +for e.g. the PhotoStation. So I stopped every service I didn't need (from the *Package Manager*) +and paused those services which appeared at the top for the `top` command, i.e. which were using +the CPU: + +``` +root@DiskStation:~# synoservice --pause synomkthumbd +root@DiskStation:~# synoservice --pause pkgctl-FileStation +root@DiskStation:~# synoservice --pause scemd +root@DiskStation:~# synoservice --pause synorelayd +root@DiskStation:~# synoservice --pause synoindexd +``` + +(Just remember to `--resume` them after everything is done.) + +Now to the tweaking of the lvm parameters. Since I didn't want to just change settings to values +from the forums without knowing why and what they do, I searched for a site explaining the +different parameters and I found [this nice page](https://baptiste-wicht.com/posts/2015/03/how-to-speed-up-raid-5-6-growing-with-mdadm.html) +which explains the commands and their effects in terms of speed and memory consumption. + +The `sysctl dev.raid.speed_limit_max` already showed a reasonable value of `200000` (200 MB/sec) +but I raised it to 500 MB/sec - just to be sure. + + sysctl -w dev.raid.speed_limit_max=500000 + +As expected, this didn't do anything. On to the next parameter from that page - the +`stripe_cache_size`. First, check the current (default) value: + + root@DiskStation:~# cat /sys/block/md3/md/stripe_cache_size + 256 + +This means the stripe cache is 256 pages of 4096 Bytes each = 1 MiB per disk. Not very much. +So I increased it as per the recommendation to 4096 pages = 16 MiB per disk: + + echo 4096 > /sys/block/md2/md/stripe_cache_size + echo 4096 > /sys/block/md3/md/stripe_cache_size + +(`md2` and `md3` are my `raid5` devices. See `mdstat` output above.) + +This brought the biggest effect. The estimated runtime left immediately fell to around 1300 +minutes, fluctuating between 1100 and 1800 minutes: + +``` +root@DiskStation:~# cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] +md3 : active raid5 sdd6[2] sda6[0] sdb6[1] + 1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU] + [=====>...............] reshape = 26.0% (509001728/1953494784) finish=1338.0min speed=17992K/sec +``` + +That was already impressive, cutting down the overall time from 3.5 days to 22 hours. +And we had another parameter to tweak: the read-ahead. Again, we first want to figure out the +default value: + + root@DiskStation:~# blockdev --getra /dev/md3 + 384 + +Since the value is in "sectors" of 512 Bytes, that comes in at 192 KiB. As we have plenty of RAM +(upgraded my DS to 8GB), I've set this to the recommended 32 MiB: + + root@DiskStation:~# blockdev --setra 65536 /dev/md3 + root@DiskStation:~# blockdev --setra 65536 /dev/md2 + +And that gave another small boost: + +``` +root@DiskStation:~# cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] +md3 : active raid5 sdd6[2] sda6[0] sdb6[1] + 1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU] + [=====>...............] reshape = 26.1% (511295616/1953494784) finish=1001.1min speed=24008K/sec +``` + +Another 300 minutes (5 hours) shaved off the remaining time. With a remaining time of less +than 17 hours for something which initially needed 3.5 days, I was very pleased. + +And the process seems to have sped up even more as it is now - 9.5 hours later - nearing completion: + +``` +root@DiskStation:~# cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] +md3 : active raid5 sdd6[2] sda6[0] sdb6[1] + 1953494784 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU] + [===================>.] reshape = 95.9% (1874314432/1953494784) finish=20.1min speed=65521K/sec +``` + + + + +*[MiB]: Mebibyte (1024 KiB), similar to Megabyte (1000 kB) +*[KiB]: Kibibyte (1024 Bytes), similar to Kilobyte (1000 Bytes)