Table of Contents
ZFS Commands
zfs list
shows USED and AVAILABLE after parity/redundancy are removed
zpool list
shows ALLOCATED and FREE including space occupied by parity/redundancy
Create ZFS dataset
PostgreSQL
~~REFNOTES~~
notes
- use ZFS's cache is postgresql dataset exceeds RAM, use postgres's cache if dataset fits in RAM
- disable postgres cache, enable ZFS cache
- enable postgres cache, disable ZFS cache
zfs
- atime=off – disable access time recording for files when they are read
- compression=lz4 – turn on compression
- recordsize=8k – set record size to postgres's default write block size 8k
- logbias=throughput –
- primarycache=metadata – ARC only cache metadata (dnode entries), not page data itself. DO NOT disable if PostgreSQL's cache is disabled!
- quota=500G
postgres
- disable full_page_writes
- disable PostgreSQL TOAST compression
- dedicate 15%-25% physical ram to ARC, and 50% to shared_buffers
raw benchmark data
| pgbench -c 10 -j 2 -t 10000 | inc. connections est. | exc. connections est. | latency |
|---|---|---|---|
| BASE | 6775 | 6776 | 1.476 ms |
| primarycache=metadata | 3850 | 3850 | 2.597 ms |
| primarycache=metadata compression=lz4 | 4815 | 4816 | 2.077 ms |
| primarycache=metadata compression=lz4 recordsize=16k | 4160 | 4160 | 2.404 ms |
| primarycache=metadata, compression=lz4, recordsize=16k, atime=off | 3415 | 3415 | 2.928 ms |
| primarycache=metadata, compression=lz4, recordsize=8k, atime=off | 4362 | 4363 | 2.292 ms |
| primarycache=metadata, compression=lz4, recordsize=8k, atime=off, shared_buffers=2048MB | 5862 | 5863 | 1.706 ms |
| primarycache=metadata, compression=lz4, recordsize=16k, atime=off, shared_buffers=2048MB | 5819 | 5820 | 1.718 ms |
| compression=lz4, recordsize=16k, atime=off, shared_buffers=2048MB | 7628 | 7628 | 1.311 ms |
| compression=lz4, recordsize=16k, atime=off, shared_buffers=4096MB | 7332 | 7333 | 1.364 ms |
| compression=lz4, recordsize=16k, atime=off, logbias=throughput, shared_buffers=4096MB, synchronous_commit=off | 27052 | 27067 | 0.370 ms |
| compression=lz4, recordsize=16k, atime=off, logbias=throughput, shared_buffers=4096MB, synchronous_commit=off, full_page_writes=off | 30348 | 30366 | 0.330 ms |
/boot/loader.conf
# ZFS tuning. arc min at 2GB, max at 8GB. ashift to 12 for 4k sectors # settings are in 1024*1024, not sure if need be vfs.zfs.min_auto_ashift="12" vfs.zfs.arc_min="2147483648" # real memory = 6442450944 (6144 MB) # avail memory = 5916946432 (5642 MB) # Setting limit to 2/3 of available memory vfs.zfs.arc_max="3944630954"
Create ZFS raid6 zpool named storage with da devices
zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6
Create ZFS dataset media
zfs create storage/media
Create ZFS dataset data with lz4 compression, acl mode passthrough, acl inherit, and visible snapdir
zfs create -o aclmode=passthrough -o aclinherit=passthrough -o snapdir=visible -o compression=lz4 storage/data
Create folders in storage/media and storage/data, set proper owner:group
mkdir /storage/media/4k/ mkdir /storage/media/anime/ mkdir /storage/media/anime_movies/ mkdir /storage/media/documentary/ mkdir /storage/media/international/ mkdir /storage/media/japanese/ mkdir /storage/media/movies/ mkdir /storage/media/mp3/ mkdir /storage/media/russian_movies/ mkdir /storage/media/russian_shows/ mkdir /storage/media/shows/ mkdir /storage/data/smb/ chown -R nobody:nobody /storage/media chown -R nobody:nobody /storage/data
Benchmarks
| Type | zpool command |
|---|---|
| 4 x stripe | zpool create storage da0 da1 da2 da3 |
| 7 x stripe | zpool create storage da0 da1 da2 da3 da4 da5 da6 |
| 4 x raid5 | zpool create storage raidz da0 da1 da2 da3 |
| 7 x raid5 | zpool create storage raidz da0 da1 da2 da3 da4 da5 da6 |
| 7 x raid6 | zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6 |
| name | command |
|---|---|
| dd 30gb zero write | dd if=/dev/zero of=/storage/30gb.zero.test bs=1m count=30000 |
| dd 30gb zero read | dd if=/storage/30gb.zero.test of=/dev/null bs=1m |
| fio | fio fio-seq-read.job |
| fio | fio fio-seq-write.job |
| fio | fio fio-seq-rw.job |
| fio | fio fio-rand-read.job |
| fio | fio fio-rand-write.job |
| fio | fio fio-rand-rw.job |
| iozone | iozone -a -b results.xls -f /storage/iozone.tmp |
<datatables>
| zfs type | dd 30gb read | dd 30gb write | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|---|
| 4x stripe | 747370298 | 761272125 | ||||
| 7x stripe | 1174415124 | 1254250847 | ||||
| 4x raid5 | 500529216 | 479107026 | ||||
| 7x raid5 | 954371168 | 665795046 | ||||
| 7x raid6 | 736157517 | 571027892 |
</datatables>
Links
ZFS Tuning Guide[(https://wiki.freebsd.org/ZFSTuningGuide)] FreeBSD Enterprise 1 PB Storage[(https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/)]