This is an old revision of the document!

ZFS Commands

zfs list

shows USED and AVAILABLE after parity/redundancy are removed

zpool list

shows ALLOCATED and FREE including space occupied by parity/redundancy

Create ZFS dataset

PostgreSQL

~~REFNOTES~~

notes

use ZFS's cache is postgresql dataset exceeds RAM, use postgres's cache if dataset fits in RAM
- disable postgres cache, enable ZFS cache
- enable postgres cache, disable ZFS cache

zfs

atime=off – disable access time recording for files when they are read
compression=lz4 – turn on compression
recordsize=8k – set record size to postgres's default write block size 8k
logbias=throughput –
primarycache=metadata – ARC only cache metadata (dnode entries), not page data itself. DO NOT disable if PostgreSQL's cache is disabled!
quota=500G

postgres

disable full_page_writes
disable PostgreSQL TOAST compression
dedicate 15%-25% physical ram to ARC, and 50% to shared_buffers

raw benchmark data

pgbench -c 10 -j 2 -t 10000

tweaks	inc. connections est.	exc. connections est.	latency
BASE	6775	6776	1.476 ms
primarycache=metadata	3850	3850	2.597 ms
primarycache=metadata compression=lz4	4815	4816	2.077 ms
primarycache=metadata compression=lz4 recordsize=16k	4160	4160	2.404 ms
primarycache=metadata, compression=lz4, recordsize=16k, atime=off	3415	3415	2.928 ms
primarycache=metadata, compression=lz4, recordsize=8k, atime=off	4362	4363	2.292 ms
primarycache=metadata, compression=lz4, recordsize=8k, atime=off, shared_buffers=2048MB	5862	5863	1.706 ms
primarycache=metadata, compression=lz4, recordsize=16k, atime=off, shared_buffers=2048MB	5819	5820	1.718 ms
compression=lz4, recordsize=16k, atime=off, shared_buffers=2048MB	7628	7628	1.311 ms
compression=lz4, recordsize=16k, atime=off, shared_buffers=4096MB	7332	7333	1.364 ms
compression=lz4, recordsize=16k, atime=off, logbias=throughput, shared_buffers=4096MB, synchronous_commit=off	27052	27067	0.370 ms
compression=lz4, recordsize=16k, atime=off, logbias=throughput, shared_buffers=4096MB, synchronous_commit=off, full_page_writes=off	30348	30366	0.330 ms

/boot/loader.conf

# ZFS tuning. arc min at 2GB, max at 8GB. ashift to 12 for 4k sectors
# settings are in 1024*1024, not sure if need be
vfs.zfs.min_auto_ashift="12"
vfs.zfs.arc_min="2147483648"
# real memory  = 6442450944 (6144 MB)
# avail memory = 5916946432 (5642 MB)
# Setting limit to 2/3 of available memory
vfs.zfs.arc_max="3944630954"

Create ZFS raid6 zpool named storage with da devices

zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6

Create ZFS dataset media

zfs create storage/media

Create ZFS dataset data with lz4 compression, acl mode passthrough, acl inherit, and visible snapdir

zfs create -o aclmode=passthrough -o aclinherit=passthrough -o snapdir=visible -o compression=lz4 storage/data

Create folders in storage/media and storage/data, set proper owner:group

mkdir /storage/media/4k/
mkdir /storage/media/anime/
mkdir /storage/media/anime_movies/
mkdir /storage/media/documentary/
mkdir /storage/media/international/
mkdir /storage/media/japanese/
mkdir /storage/media/movies/
mkdir /storage/media/mp3/
mkdir /storage/media/russian_movies/
mkdir /storage/media/russian_shows/
mkdir /storage/media/shows/

mkdir /storage/data/smb/

chown -R nobody:nobody /storage/media
chown -R nobody:nobody /storage/data

Benchmarks

Type	zpool command
4 x stripe	zpool create storage da0 da1 da2 da3
7 x stripe	zpool create storage da0 da1 da2 da3 da4 da5 da6
4 x raid5	zpool create storage raidz da0 da1 da2 da3
7 x raid5	zpool create storage raidz da0 da1 da2 da3 da4 da5 da6
7 x raid6	zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6

name	command
dd 30gb zero write	dd if=/dev/zero of=/storage/30gb.zero.test bs=1m count=30000
dd 30gb zero read	dd if=/storage/30gb.zero.test of=/dev/null bs=1m
fio	fio fio-seq-read.job
fio	fio fio-seq-write.job
fio	fio fio-seq-rw.job
fio	fio fio-rand-read.job
fio	fio fio-rand-write.job
fio	fio fio-rand-rw.job
iozone	iozone -a -b results.xls -f /storage/iozone.tmp

zfs type	dd 30gb read	dd 30gb write
4x stripe	747370298	761272125
7x stripe	1174415124	1254250847
4x raid5	500529216	479107026
7x raid5	954371168	665795046
7x raid6	736157517	571027892

</datatables>

Links

ZFS Tuning Guide[(https://wiki.freebsd.org/ZFSTuningGuide)] FreeBSD Enterprise 1 PB Storage[(https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/)]

sleepy.wtf

Table of Contents