====== ZFS Commands ======
<code>zfs list</code> shows USED and AVAILABLE after parity/redundancy are removed
<code>zpool list</code> shows ALLOCATED and FREE including space occupied by parity/redundancy
----
====== Create ZFS dataset ======

===== PostgreSQL =====
~~REFNOTES~~
==== notes ====
  * use ZFS's cache is postgresql dataset exceeds RAM, use postgres's cache if dataset fits in RAM
    * disable postgres cache, enable ZFS cache
    * enable postgres cache, disable ZFS cache

==== zfs ====
  * **atime=off** -- disable access time recording for files when they are read
  * **compression=lz4** -- turn on compression
  * **recordsize=8k** -- set record size to postgres's default write block size 8k
  * **logbias=throughput**  -- 
  * **primarycache=metadata** -- ARC only cache metadata (dnode entries), not page data itself. DO NOT disable if PostgreSQL's cache is disabled!
  * quota=500G

==== postgres ====
  * disable full_page_writes
  * disable PostgreSQL TOAST compression
  * dedicate 15%-25% physical ram to ARC, and 50% to shared_buffers
=== raw benchmark data ===
{{tablelayout?tableSort=1}}
^ pgbench -c 10 -j 2 -t 10000 ^ inc. connections est. ^ exc. connections est. ^ latency ^
| BASE |  6775 |  6776 |  1.476 ms |
| primarycache=metadata |  3850 |  3850 |  2.597 ms |
| primarycache=metadata compression=lz4 |  4815 |  4816 |  2.077 ms |
| primarycache=metadata compression=lz4 recordsize=16k |  4160 |  4160 |  2.404 ms |
| primarycache=metadata, compression=lz4, recordsize=16k, atime=off |  3415 |  3415 |  2.928 ms |
| primarycache=metadata, compression=lz4, recordsize=8k, atime=off |  4362 |  4363 |  2.292 ms |
| primarycache=metadata, compression=lz4, recordsize=8k, atime=off, \\ shared_buffers=2048MB |  5862 |  5863 |  1.706 ms |
| primarycache=metadata, compression=lz4, recordsize=16k, atime=off, \\ shared_buffers=2048MB |  5819 |  5820 |  1.718 ms |
| compression=lz4, recordsize=16k, atime=off, \\ shared_buffers=2048MB |  7628 |  7628 |  1.311 ms |
| compression=lz4, recordsize=16k, atime=off, \\ shared_buffers=4096MB |  7332 |  7333 |  1.364 ms |
| compression=lz4, recordsize=16k, atime=off, logbias=throughput, \\ shared_buffers=4096MB, synchronous_commit=off |  27052 |   27067 |  0.370 ms |
| compression=lz4, recordsize=16k, atime=off, logbias=throughput, \\ shared_buffers=4096MB, synchronous_commit=off, full_page_writes=off |  30348 |  30366 |  0.330 ms |


----
====== /boot/loader.conf ======
<code>
# ZFS tuning. arc min at 2GB, max at 8GB. ashift to 12 for 4k sectors
# settings are in 1024*1024, not sure if need be
vfs.zfs.min_auto_ashift="12"
vfs.zfs.arc_min="2147483648"
# real memory  = 6442450944 (6144 MB)
# avail memory = 5916946432 (5642 MB)
# Setting limit to 2/3 of available memory
vfs.zfs.arc_max="3944630954"
</code>


Create ZFS raid6 zpool named ''storage'' with ''da'' devices
<code>zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6</code>

Create ZFS dataset ''media''
<code>zfs create storage/media</code>

Create ZFS dataset ''data'' with lz4 compression, acl mode passthrough, acl inherit, and visible snapdir
<code>zfs create -o aclmode=passthrough -o aclinherit=passthrough -o snapdir=visible -o compression=lz4 storage/data</code>

Create folders in storage/media and storage/data, set proper owner:group
<code>
mkdir /storage/media/4k/
mkdir /storage/media/anime/
mkdir /storage/media/anime_movies/
mkdir /storage/media/documentary/
mkdir /storage/media/international/
mkdir /storage/media/japanese/
mkdir /storage/media/movies/
mkdir /storage/media/mp3/
mkdir /storage/media/russian_movies/
mkdir /storage/media/russian_shows/
mkdir /storage/media/shows/

mkdir /storage/data/smb/

chown -R nobody:nobody /storage/media
chown -R nobody:nobody /storage/data
</code>

----
===== Benchmarks =====
{{tablelayout?tableSort=1}}
^ Type ^ zpool command ^
| 4 x stripe | zpool create storage da0 da1 da2 da3 |
| 7 x stripe | zpool create storage da0 da1 da2 da3 da4 da5 da6 |
| 4 x raid5  | zpool create storage raidz da0 da1 da2 da3 |
| 7 x raid5  | zpool create storage raidz da0 da1 da2 da3 da4 da5 da6 |
| 7 x raid6  | zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6 |

{{tablelayout?tableSort=1}}
^ name ^ command ^ 
| dd 30gb zero write | dd if=/dev/zero of=/storage/30gb.zero.test bs=1m count=30000 |
| dd 30gb zero read  | dd if=/storage/30gb.zero.test of=/dev/null bs=1m |
| fio                | fio fio-seq-read.job |
| fio                | fio fio-seq-write.job |
| fio                | fio fio-seq-rw.job |
| fio                | fio fio-rand-read.job |
| fio                | fio fio-rand-write.job |
| fio                | fio fio-rand-rw.job |
| iozone             | iozone -a -b results.xls -f /storage/iozone.tmp |


<datatables>
^ zfs type  ^ dd 30gb read      ^ dd 30gb write     ^ 1 ^ 2 ^ 3 ^ 4 ^
| 4x stripe | 747370298  | 761272125  |   |   |   |   |
| 7x stripe | 1174415124 | 1254250847 |   |   |   |   |
| 4x raid5  | 500529216  | 479107026  |   |   |   |   |
| 7x raid5  | 954371168  | 665795046  |   |   |   |   |
| 7x raid6  | 736157517  | 571027892  | | | | |
</datatables>

<gchart hbar>
4x stripe = 764801997
7x stripe = 1254250847
4x raid5  = 479107026
7x raid5  = 665795046
</gchart>

===== Links =====
ZFS Tuning Guide[(https://wiki.freebsd.org/ZFSTuningGuide)]
FreeBSD Enterprise 1 PB Storage[(https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/)]