Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NAS-132425 / None / sync truenas/zfs-2.3-release with upstream 2.3.0-rc3 tag #260

Merged
merged 48 commits into from
Nov 14, 2024

Commits on Oct 11, 2024

  1. zpool/zfs: allow --json wherever -j is allowed

    Mostly so that with the JSON formatting options are also used, they all
    look the same. To my eye, `-j --json-flat-vdevs` suggests that they are
    different or unrelated, while `--json --json-flat-vdevs` invites no
    further questions.
    
    Sponsored-by: Klara, Inc.
    Sponsored-by: Wasabi Technology, Inc.
    Reviewed-by: Umer Saleem <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16632
    robn authored and behlendorf committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    6669036 View commit details
    Browse the repository at this point in the history
  2. zdb: fix printf format in dump_zap()

    When compiling zdb.c on 32-bit platforms, a format conversion error 
    is reported for a printf() in dump_zap().  Change %l to macro 
    %" PRIu64 " to match the platform size of a 64-bit unsigned integer.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Martin Matuska <[email protected]>
    Closes openzfs#16635
    mmatuska authored and behlendorf committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    5816296 View commit details
    Browse the repository at this point in the history
  3. CI: Stick with ubuntu-22.04 for CodeQL analysis

    The ubuntu-latest alias now refers to ubuntu-24.04 instead of
    ubuntu-22.04 which causes CodeQL's autobuild to fail with:
    
        cpp/autobuilder: deptrace not supported in ubuntu 24.04
    
    Until deptrace is supported by ubuntu-24.04 hosted runners request
    ubuntu-22.04 which is supported.
    
    Signed-off-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tino Reichardt <[email protected]>
    Reviewed-by: George Melikov <[email protected]>
    Closes openzfs#16639
    behlendorf committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7f830d7 View commit details
    Browse the repository at this point in the history
  4. ZTS: Slightly increase dedup_quota limit

    As described in the comment above this check the space used by
    logged entries is not accounted for and some margin needs to be
    added in.  While uncommon we have slightly exceeded the 600,000
    threshold on some CI run so we increase the limit a bit more.
    
    Reviewed-by: Alexander Motin <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Reviewed-by: George Melikov <[email protected]>
    Signed-off-by: Brian Behlendorf <[email protected]>
    Closes openzfs#16637
    behlendorf committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    5bc27ac View commit details
    Browse the repository at this point in the history
  5. ZTS: Increase zpool_import_parallel_pos import margin

    Increase the pool import time allowed by assuming a minimum reduction
    to 1/2 instead of 1/3 when comparing sequential to parallel import
    times.  This is sufficient to verify parallel imports are working as
    intended and should address the occasional false positive failure
    when the time is slightly exceeded.
    
    Reviewed-by: Tino Reichardt <[email protected]>
    Reviewed-by: George Melikov <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Signed-off-by: Brian Behlendorf <[email protected]>
    Closes openzfs#16638
    behlendorf committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    c645b07 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Fallback to strerror() when strerror_l() isn't available

    Some C libraries, such as uClibc, do not provide strerror_l() in
    which case we fallback to strerror().
    
    Reviewed-by: Tino Reichardt <[email protected]>
    Signed-off-by: Brian Behlendorf <[email protected]>
    Closes openzfs#16636 
    Closes openzfs#16640
    behlendorf committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    56871e4 View commit details
    Browse the repository at this point in the history
  2. ZTS: Optimize Kernel Same-page Merging (KSM)

    Kernel same-page Merging (KSM) allows KVM guests to share identical
    memory pages. These shared pages are usually common libraries or other
    identical, high-use data.
    
    The current configuration was a bit to lazy - so KSM didn't work very
    well. With the new configuration I could run 3 Linux VMs in parralel.
    
    FreeBSD can't benefit from it. But FreeBSD is not so memory hungry in
    general, so there is no need for it ;)
    
    Reviewed-by: George Melikov <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Tino Reichardt <[email protected]>
    Closes openzfs#16641
    mcmilk authored and behlendorf committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    77df762 View commit details
    Browse the repository at this point in the history
  3. ZTS: Make use of optimal CPU pinning

    With CPU pinning, we should get some speedup because of better
    cpu cache re-use.
    
    Reviewed-by: George Melikov <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Tino Reichardt <[email protected]>
    Closes openzfs#16641
    mcmilk authored and behlendorf committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    b5a3825 View commit details
    Browse the repository at this point in the history
  4. Tag 2.3.0-rc2

    Signed-off-by: Brian Behlendorf <[email protected]>
    behlendorf committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    0409c47 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. freebsd: Use compiler.h from FreeBSD's base's linuxkpi

    The FreeBSD linux/compiler.h in OpenZFS was copied from a very old
    version of FreeBSD's linuxkpi's linux/compiler.h. There's no need for
    this duplication. Use FreeBSD's linuxkpi version instead, and provide
    zfs_fallthrough to augment it (it's all that's needed). Use #pragma once
    to avoid naming issues for guard variables. Since this is a complete
    rewrite, use my copyright here (the original code in FreeBSD still
    credits everybody). This works back at least to FreeBSD 12.4, which
    is not out of support, and all newer releases.
    
    Remove extra copies of macros that were defined elsewhere, but are now
    properly defined in LinuxKPI so are redundant.
    
    Sponsored-by: Netflix
    Reviewed-by: Tony Hutter <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Warner Losh <[email protected]>
    Closes openzfs#16650
    bsdimp authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3d9129a View commit details
    Browse the repository at this point in the history
  2. Fix inconsistent mount options for ZFS root

    While mounting ZFS root during boot on Linux distributions from initrd,
    mount from busybox is effectively used which executes mount system call
    directly. This skips the ZFS helper mount.zfs, which checks and enables
    the mount options as specified in dataset properties. As a result,
    datasets mounted during boot from initrd do not have correct mount
    options as specified in ZFS dataset properties.
    
    There has been an attempt to use mount.zfs in zfs initrd script,
    responsible for mounting the ZFS root filesystem (PR#13305). This was
    later reverted (PR#14908) after discovering that using mount.zfs breaks
    mounting of snapshots on root (/) and other child datasets of root have
    the same issue (Issue#9461).
    
    This happens because switching from busybox mount to mount.zfs correctly
    parses the mount options but also adds 'mntpoint=/root' to the mount
    options, which is then prepended to the snapshot mountpoint in
    '.zfs/snapshot'. '/root' is the directory on Debian with initramfs-tools
    where root filesystem is mounted before pivot_root. When Linux runtime
    is reached, trying to access the snapshots on root results in
    automounting the snapshot on '/root/.zfs/*', which fails.
    
    This commit attempts to fix the automounting of snapshots on root, while
    using mount.zfs in initrd script. Since the mountpoint of dataset is
    stored in vfs_mntpoint field, we can check if current mountpoint of
    dataset and vfs_mntpoint are same or not. If they are not same, reset
    the vfs_mntpoint field with current mountpoint. This fixes the
    mountpoints of root dataset and children in respective vfs_mntpoint
    fields when we try to access the snapshots of root dataset or its
    children. With correct mountpoint for root dataset and children stored
    in vfs_mntpoint, all snapshots of root dataset are mounted correctly
    and become accessible.
    
    This fix will come into play only if current process, that is trying to
    access the snapshots is not in chroot context. The Linux kernel API
    that is used to convert struct path into char format (d_path), returns
    the complete path for given struct path. It works in chroot environment
    as well and returns the correct path from original filesystem root.
    
    However d_path fails to return the complete path if any directory from
    original root filesystem is mounted using --bind flag or --rbind flag
    in chroot environment. In this case, if we try to access the snapshot
    from outside the chroot environment, d_path returns the path correctly,
    i.e. it returns the correct path to the directory that is mounted with
    --bind flag. However inside the chroot environment, it only returns the
    path inside chroot.
    
    For now, there is not a better way in my understanding that gives the
    complete path in char format and handles the case where directories from
    root filesystem are mounted with --bind or --rbind on another path which
    user will later chroot into. So this fix gets enabled if current
    process trying to access the snapshot is not in chroot context.
    
    With the snapshots issue fixed for root filesystem, using mount.zfs in
    ZFS initrd script, mounts the datasets with correct mount options.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Reviewed-by: Ameer Hamza <[email protected]>
    Signed-off-by: Umer Saleem <[email protected]>
    Closes openzfs#16646
    usaleem-ix authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    36a67b5 View commit details
    Browse the repository at this point in the history
  3. libspl/backtrace: dump registers in libunwind backtraces

    More useful stuff, especially when trying to follow a disassembly.
    
    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tino Reichardt <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16653
    robn authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    bcd61d9 View commit details
    Browse the repository at this point in the history
  4. libspl/backtrace: helper macros for output

    My eyes are going blurry looking at all those write calls. This is much
    nicer.
    
    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tino Reichardt <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Close openzfs#16653
    robn authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d5db840 View commit details
    Browse the repository at this point in the history
  5. libspl/backtrace: rename and document hex conversion function

    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tino Reichardt <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16653
    robn authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    f52d7aa View commit details
    Browse the repository at this point in the history
  6. libspl/backtrace: comment and harden libunwind backtracer

    This is the sort of code that we get right once and never look at again.
    Anyone reading this code is already likely in the middle of a debugging
    nightmare, and then they have a wall of manual string construction and
    an unfamiliar and idiosyncratic library to deal with. So, comment the
    whole thing to try to make it clear what's going on.
    
    In pursuit of the above, I've added return checks to some of the
    libunwind calls, fixed the frame loop to not skip the "top" frame
    (however unseful it may be), and fix a couple of calls to
    spl_bt_u64_to_hex_str() which requested 18 digits instead of 16.
    
    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tino Reichardt <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16653
    robn authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    b4cd10c View commit details
    Browse the repository at this point in the history
  7. zfs_debug: Restore log size limit for userspace

    For some reason it was dropped when split from kernel, that makes
    raidz_test to accumulate in RAM up to 100GB of logs we don't need.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Igor Kozhukhov <[email protected]>
    Reviewed-by:  Rob Norris <[email protected]>
    Reviewed-by: Tino Reichardt <[email protected]>
    Signed-off-by:	Alexander Motin <[email protected]>
    Sponsored by:	iXsystems, Inc.
    Closes openzfs#16492
    Closes openzfs#16566
    Closes openzfs#16664
    amotin authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    ace2e17 View commit details
    Browse the repository at this point in the history
  8. Fix compile-time warnings caused by duplicate struct typedefs

    Some compiler/versions warn these typedefs according to openzfs#16660.
    
    The platform specific header sys/abd_os.h shouldn't define or use abd_t,
    as it's defined in its non-platform specific consumer sys/abd.h.
    Do the same as what FreeBSD header does.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Tomohiro Kusumi <[email protected]>
    Closes openzfs#16660 
    Closes openzfs#16665
    kusumi authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    2b359c7 View commit details
    Browse the repository at this point in the history
  9. zdb: show bp in uberblock dump

    Just another useful nugget of info in times of strife.
    
    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: George Melikov <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tino Reichardt <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16667
    robn authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    78d39d9 View commit details
    Browse the repository at this point in the history
  10. config: fix dequeue_signal check for kernels <4.20

    Before 4.20, kernel_siginfo_t was just called siginfo_t. This was
    causing the kthread_dequeue_signal_3arg_task check, which uses
    kernel_siginfo_t, to fail on older kernels.
    
    In d6b8c17, we started checking for the "new" three-arg
    dequeue_signal() by testing for the "old" version. Because that test is
    explicitly using kernel_siginfo_t, it would fail, leading to the build
    trying to use the new three-arg version, which would then not compile.
    
    This commit fixes that by avoiding checking for the old 3-arg
    dequeue_signal entirely. Instead, we check for the new one, as well as
    the 4-arg form, and we use the old form as a fallback. This way, we
    never have to test for it explicitly, and once we're building
    HAVE_SIGINFO will make sure we get the right kernel_siginfo_t for it, so
    everything works out nice.
    
    Original-patch-by: Finix <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16666
    robn authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    e30c693 View commit details
    Browse the repository at this point in the history
  11. spl/thread: explicitly define thread_func_t as noreturn

    All of our thread entry functions have this signature:
    
        void (*)(void*) __attribute__((noreturn))
    
    The low-level `__thread_create()` function accepts a `thread_func_t` as
    the entry point, which is defined more simply as:
    
        void (*)(void *)
    
    And then the `thread_create()` and `thread_create_named()` macros cast
    the passed-in function point down to `thread_func_t`, that is, casting
    away the `noreturn` attribute.
    
    Clang considers casting between these two types to be invalid because
    both the caller and the callee may have elided parts of the stack frame
    save and restore, knowing that they won't be needed.
    
    Recent Linux appears to be setting `-Wcast-function-type-strict`, which
    causes this invalid cast to emit a warning, which with `-Werror` is
    converted to an error, breaking the build.
    
    This commit fixes this in the simplest possible way: adding `noreturn`
    to the `thread_func_t` attribute. Since all our thread entry functions
    already have this attribute, it's arguably a just a consistency fix
    anyway.
    
    I considered removing the casts in the macros, which silences the
    warnings, but it turns out that Clang has a bug that won't emit this
    error for implicit conversions, only explicit casts. So leaving them
    there seems like a reasonable belt-and-suspenders approach. Also,
    frankly, this whole mechanism seems a little undercooked inside LLVM, so
    I'm content go with my intuition about the smallest, least invaisve
    change.
    
    **NOTE**: `__thread_create` is exported by `spl.ko` and has a
    `thread_func_t` arg, so this is an ABI break. Whether that matters in
    practice, I have no idea.
    
    Further reading:
    - llvm/llvm-project@1aad641
    - llvm/llvm-project#7325
    - llvm/llvm-project#41465
    
    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16672 
    Closes openzfs#16673
    robn authored and behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    ede715d View commit details
    Browse the repository at this point in the history
  12. ZTS: Add additional exceptions

    The following tests have been observed to occasionally fail when
    running under the CI.  Updated our exceptions list to track them.
    
    Signed-off-by: Brian Behlendorf <[email protected]>
    Closes openzfs#16670
    behlendorf committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    5237760 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Fix dependency install on Debian 11 (openzfs#16683)

    Adding cryptsetup breaks some dialog things on Debian 11.
    Apply some workaround for it.
    
    Signed-off-by: Tino Reichardt <[email protected]>
    Reviewed-by: George Melikov <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    mcmilk authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    77d8197 View commit details
    Browse the repository at this point in the history
  2. Pack dmu_buf_impl_t by 16 bytes

    On 64bit FreeBSD this reduces one from 296 to 280 bytes.  On small
    block workloads dbufs may consume gigabytes of ARC, and this saves
    5% of it.
    
    Reviewed-by: Tino Reichardt <[email protected]>
    Reviewed-by: Brian Atkinson <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by:	Alexander Motin <[email protected]>
    Sponsored by:	iXsystems, Inc.
    Closes openzfs#16684
    amotin authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    7e3ce4e View commit details
    Browse the repository at this point in the history
  3. ZTS: Add LUKS sanity test

    Add a LUKS sanity test to trigger: openzfs#16631
    
    Reviewed-by: Tino Reichardt <[email protected]>
    Reviewed-by: Rob Norris <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Tony Hutter <[email protected]>
    Closes openzfs#16681
    tonyhutter authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    f7b4bca View commit details
    Browse the repository at this point in the history
  4. Fix gcc unused value warning in FreeBSD simd.h

    The macros `simd_stat_init()` and `simd_stat_fini()` in FreeBSD's
    `simd.h` are defined as zero, but they are actually only used as
    statements. Replace the definitions with `do {} while (0)` instead, to
    avoid gcc `-Wunused-value` warnings.
    
    Reviewed-by: Tino Reichardt <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Reviewed-by: Toomas Soome <[email protected]>
    Signed-off-by: Dimitry Andric <[email protected]>
    Closes openzfs#16693
    DimitryAndric authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    fd2cae9 View commit details
    Browse the repository at this point in the history
  5. Fix gcc uninitialized warning in FreeBSD zio_crypt.c

    In FreeBSD's `zio_do_crypt_data()`, ensure that two `struct uio`
    variables are cleared before copying data out of them. This avoids
    accessing garbage data, and fixes gcc `-Wuninitialized` warnings.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Toomas Soome <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Dimitry Andric <[email protected]>
    Closes openzfs#16688
    DimitryAndric authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    f3823a9 View commit details
    Browse the repository at this point in the history
  6. On the first vdev open ignore impossible ashift hints

    If on the first open device's logical ashift is bigger than set
    by pool's ashift property, ignore the last as unusable instead of
    creating vdev that will fail most of I/Os due to misalignment.
    
    Reviewed-by: Rob Norris <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Ameer Hamza <[email protected]>
    Signed-off-by:  Alexander Motin <[email protected]>
    Sponsored by:   iXsystems, Inc.
    Closes openzfs#16690
    amotin authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    bbc0d34 View commit details
    Browse the repository at this point in the history
  7. Add warning for external consumers of dmu_tx_callback_register

    While reading some code @grwilson came across the above function that
    seemingly had no consumers besides a ztest callback that ensures that
    the tx_callback infrastructure works correctly. It turns out that Lustre
    is the main (and potentially the only) consumer of this. Refer to
    `osd_trans_commit_cb` of `lustre/osd-zfs/osd_handler.c` in the Lustre
    repo for more info. Let's add a comment highlighting this before someone
    removes it by mistake.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Serapheim Dimitropoulos <[email protected]>
    Closes openzfs#16698
    sdimitro authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    8ac70aa View commit details
    Browse the repository at this point in the history
  8. vdev_disk: try harder to ensure IO alignment rules

    It seems out our notion of "properly" aligned IO was incomplete. In
    particular, dm-crypt does its own splitting, and assumes that a logical
    block will never cross an order-0 page boundary (ie, the physical page
    size, not compound size). This effectively means that it needs to be
    possible to split a BIO at any page or block size boundary and have it
    work correctly.
    
    This updates the alignment check function to enforce these rules (to the
    extent possible).
    
    Our response to misaligned data is to make some new allocation that is
    properly aligned, and copy the data into it. It turns out that
    linearising (via abd_borrow_buf()) is not enough, because we allocate eg
    4K blocks from a general purpose slab, and so may receive (or already
    have) a 4K block that crosses pages.
    
    So instead, we allocate a new ABD, which is guaranteed to be aligned
    properly to block sizes, and then copy everything into it, and back out
    on the way back.
    
    Sponsored-by: Klara, Inc.
    Sponsored-by: Wasabi Technology, Inc.
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16687 openzfs#16631 openzfs#15646 openzfs#15533 openzfs#14533
    robn authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    19a8dd4 View commit details
    Browse the repository at this point in the history
  9. vdev_disk: move abd return and free off the interrupt handler

    Freeing an ABD can take sleeping locks to update various stats. We
    aren't allowed to sleep on an interrupt handler. So, move the free off
    to the io_done callback.
    
    We should never have been freeing things in the interrupt handler, but
    we got away with it because we were usually freeing a linear ABD, which
    at most is returning two objects to a cache and never sleeping. Scatter
    ABDs can be used now, and those have more complex locking.
    
    Sponsored-by: Klara, Inc.
    Sponsored-by: Wasabi Technology, Inc.
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16687
    robn authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    86b5853 View commit details
    Browse the repository at this point in the history
  10. Added output to zpool online and offline

    I was surprised to discover today that `zpool online` and
    `zpool offline` don't print any information about why they failed in
    many cases, they just return 1 with no information about why.
    
    Let's improve that where we can without changing the library function.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Reviewed-by: Allan Jude <[email protected]>
    Signed-off-by: Rich Ercolani <[email protected]>
    Closes openzfs#16244
    rincebrain authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    903d3f9 View commit details
    Browse the repository at this point in the history
  11. zdb: add extra -T flag to show histograms of BRT refcounts

    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16692
    robn authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    7546fbd View commit details
    Browse the repository at this point in the history
  12. ZTS: Add Fedora 41, remove Fedora 39

    Fedora 41 was released 10/29/24, and Fedora 39 will be EOL on 11/12/24.
    Update Fedora runners in the test suite.  Some minor tweaks also needed
    to support ksh 1.0.10.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: George Melikov <[email protected]>
    Signed-off-by: Tony Hutter <[email protected]>
    Closes openzfs#16700
    tonyhutter authored and behlendorf committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    d367ef2 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. zfs(4): remove "experimental" from zfs_bclone_enabled

    I think we've done enough experiments.
    
    Sponsored-by: https://despairlabs.com/sponsor/
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Reviewed-by: George Melikov <[email protected]>
    Signed-off-by: Rob Norris <[email protected]>
    Closes openzfs#16189 
    Closes openzfs#16712
    robn authored and behlendorf committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    880b739 View commit details
    Browse the repository at this point in the history
  2. Reduce dirty records memory usage

    Small block workloads may use a very large number of dirty records.
    During simple block cloning test due to BRT still using 4KB blocks
    I can easily see up to 2.5M of those used.  Before this change
    dbuf_dirty_record_t structures representing them were allocated via
    kmem_zalloc(), that rounded their size up to 512 bytes.
    
    Introduction of specialized kmem cache allows to reduce the size
    from 512 to 408 bytes.  Additionally, since override and raw params
    in dirty records are mutually exclusive, puting them into a union
    allows to reduce structure size down to 368 bytes, increasing the
    saving to 28%, that can be a 0.5GB or more of RAM.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Brian Atkinson <[email protected]>
    Signed-off-by:	Alexander Motin <[email protected]>
    Sponsored by:	iXsystems, Inc.
    Closes openzfs#16694
    amotin authored and behlendorf committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    55cbd1f View commit details
    Browse the repository at this point in the history
  3. Verify parent_dev before calling udev_device_get_sysattr_value

    Not all udev devices have parent devices.
    Calling udev_device_get_ functions yield an assertion error
    if called with a NULL pointer.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Sietse <[email protected]>
    Co-authored-by: Sietse <[email protected]>
    Closes openzfs#16705 
    Closes openzfs#16717
    Uglymotha authored and behlendorf committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    b96845b View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Revert "Avoid BUG in migrate_folio_extra"

    This reverts commit b052035.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Reviewed-by: Brian Atkinson <[email protected]>
    Signed-off-by: tstabrawa <[email protected]>
    Closes openzfs#16568
    Closes openzfs#16723
    tstabrawa authored and behlendorf committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    ae48c2f View commit details
    Browse the repository at this point in the history
  2. Use simple folio migration function

    Avoids using fallback_migrate_folio, which starts unnecessary writeback
    (leading to BUG in migrate_folio_extra).
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Tony Hutter <[email protected]>
    Reviewed-by: Brian Atkinson <[email protected]>
    Signed-off-by: tstabrawa <[email protected]>
    Closes openzfs#16568
    Closes openzfs#16723
    tstabrawa authored and behlendorf committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    661bb43 View commit details
    Browse the repository at this point in the history
  3. ZFS send should use spill block prefetched from send_reader_thread

    Currently, even though send_reader_thread prefetches spill block,
    do_dump() will not use it and issues its own blocking arc_read. This
    causes significant performance degradation when sending datasets with
    lots of spill blocks.
    
    For unmodified spill blocks, we also create send_range struct for them
    in send_reader_thread and issue prefetches for them. We piggyback them
    on the dnode send_range instead of enqueueing them so we don't break
    send_range_after check.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Chunwei Chen <[email protected]>
    Co-authored-by: david.chen <[email protected]>
    Closes openzfs#16701
    tuxoko authored and behlendorf committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    c82eb27 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Update ABD stats for linear page Linux

    a10e552 updated abd_free_linear_page() to no longer call
    abd_update_scatter_stat(). This meant that linear pages that were not
    attached to Direct I/O requests were not doing waste accounting for the
    ARC. This led to performance issues due to incorrect ARC accounting that
    resulted in 100% of CPU time being spent in arc_evict() during prolonged
    I/O workloads with the ARC.
    
    The call to abd_update_scatter_stats() is now conditionally called in
    abd_free_linear_page() when the ABD is not from a Direct I/O request.
    
    Reviewed-by: Mark Maybee <[email protected]>
    Reviewed-by: Tony Nguyen <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Brian Atkinson <[email protected]>
    Closes openzfs#16729
    bwatkinson authored and behlendorf committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    8131793 View commit details
    Browse the repository at this point in the history
  2. Use <fcntl.h> instead of <sys/fcntl.h>

    When building on musl, we get:
    
    ```
    In file included from tests/zfs-tests/cmd/getversion.c:22:
    /usr/include/sys/fcntl.h:1:2: error: #warning redirecting incorrect
     #include <sys/fcntl.h> to <fcntl.h> [-Werror=cpp]
     1 | #warning redirecting incorrect #include <sys/fcntl.h> to <fcntl.h>
    
    In file included from module/os/linux/zfs/vdev_file.c:36:
    /usr/include/sys/fcntl.h:1:2: error: #warning redirecting incorrect
     #include <sys/fcntl.h> to <fcntl.h> [-Werror=cpp]
     1 | #warning redirecting incorrect #include <sys/fcntl.h> to <fcntl.h>
    ```
    
    Bug: https://bugs.gentoo.org/925235
    Reviewed-by: Brian Behlendorf <[email protected]>
    Signed-off-by: Sam James <[email protected]>
    Closes openzfs#15925
    thesamesam authored and behlendorf committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    e12d761 View commit details
    Browse the repository at this point in the history
  3. JSON: fix user properties output for zfs list

    This commit fixes JSON output for zfs list when user properties are
    requested with -o flag. This case needed to be handled specifically
    since zfs_prop_to_name does not return property name for user
    properties, instead it is stored in pl->pl_user_prop.
    
    Reviewed-by: Ameer Hamza <[email protected]>
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Umer Saleem <[email protected]>
    Closes openzfs#16732
    usaleem-ix authored and behlendorf committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    9061a4d View commit details
    Browse the repository at this point in the history
  4. Tag 2.3.0-rc3

    Signed-off-by: Brian Behlendorf <[email protected]>
    behlendorf committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    1a54b13 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2024

  1. Merge tag 'zfs-2.3.0-rc3' into truenas/zfs-2.3-release

    OpenZFS 2.3.0-rc3
    
    Signed-off-by: Ameer Hamza <[email protected]>
    ixhamza committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    b1835da View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. JSON: fix user properties output for zpool list

    This commit fixes JSON output for zpool list when user properties are
    requested with -o flag. This case needed to be handled specifically
    since zpool_prop_to_name does not return property name for user
    properties, instead it is stored in pl->pl_user_prop.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Umer Saleem <[email protected]>
    Closes openzfs#16734
    usaleem-ix authored and ixhamza committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    41f89d0 View commit details
    Browse the repository at this point in the history
  2. Fix user properties output for zpool list

    In zpool_get_user_prop, when called from zpool_expand_proplist and
    collect_pool, we often have zpool_props present in zpool_handle_t equal
    to NULL. This mostly happens when only one user property is requested
    using zpool list -o <user_property>. Checking for this case and
    correctly initializing the zpool_props field in zpool_handle_t fixes
    this issue.
    
    Interestingly, this issue does not occur if we query any other property
    like name or guid along with a user property with -o flag because while
    accessing properties like guid, zpool_prop_get_int is called which
    checks for this case specifically and calls zpool_get_all_props.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Alexander Motin <[email protected]>
    Signed-off-by: Umer Saleem <[email protected]>
    Closes openzfs#16734
    usaleem-ix authored and ixhamza committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    092cb94 View commit details
    Browse the repository at this point in the history
  3. L2ARC: Move different stats updates earlier

    ..., before we make the header or the log block visible to others.
    It should fix assertion on allocated space going negative if the
    header is freed once the lock is dropped, while the write is still
    going.
    
    Reviewed-by: Brian Behlendorf <[email protected]>
    Reviewed-by: Rob Norris <[email protected]>
    Signed-off-by: Alexander Motin <[email protected]>
    Sponsored by:	iXsystems, Inc.
    Closes openzfs#16040
    Closes openzfs#16743
    amotin authored and ixhamza committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    5a0ec4a View commit details
    Browse the repository at this point in the history
  4. zvol_os.c: Increase optimal IO size

    Since zvol read and write can process up to (DMU_MAX_ACCESS / 2) bytes
    in a single operation, the current optimal I/O size is too low. SCST
    directly reports this value as the optimal transfer length for the
    target SCSI device. Increasing it from the previous volblocksize results
    in performance improvement for large block parallel I/O workloads.
    
    Signed-off-by: Ameer Hamza <[email protected]>
    ixhamza committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b79bddb View commit details
    Browse the repository at this point in the history