AuriStor File System News

The AuriStorFS v2021.05-30 release includes important bug fixes for clients, servers and administrative tooling.

• New platform

  Linux 6.5 kernels.

• Fix for rxkad_krb5 superuser tokens

  OpenAFS security update OPENAFS-SA-2013-003 "Brute force DES attack permits compromise of AFS cell" introduced a bug that results in the expiration of rxkad_krb5 protected superuser connections that are intended to never expire. The security advisory was addressed by a feature change which extended the rxkad security class to support the use of Kerberos v5 service tickets with stronger encryption keys such as aes256-cts-hmac-sha1-96. The Kerberos v5 service ticket is exchanged and decrypted during the CHALLENGE | RESPONSE phase of establishing an Rx connection protected by rxkad. Once validated, any non-DES session key is used to derive a 56-bit key that can be used by rxkad for wire privacy. This Kerberos v5 extended version of rxkad is referred to as "rxkad_krb5" to distinguish it from the earlier versions that only supported DES encrypted service tickets.

  rxkad was designed based Kerberos v4 which used unsigned 32-bit integers to represent timestamps. The valid range being Unix epoch (0) through 07 Feb 2106 06:28:15 GMT (0xffffffff). Kerberos v5 timestamps are represented as signed 32-bit integers with a valid range of 13 Dec 1901 20:45:52 GMT (-2147483648) through 19 Jan 2038 03:14:07 GMT (0x7fffffff). This mismatch in the valid time ranges resulted in a subtle bug when generating rxkad superuser tokens such as created for server-to-server authentication and the -localauth switch for tools such as bos, pts, and vos.

  An rxkad superuser token is defined as having a start time of Unix Epoch (0) and an end time of 0xffffffff. A superuser token can never expire even though the maximum lifetime of an rxkad token is 30 days (108000 seconds). Since the maximum endtime that can be represented by a Kerberos v5 service ticket is 0x7fffffff that is what is set when the requested end time exceeds the maximum. When the Rx acceptor validates the rxkad_krb5 RESPONSE packet, it observes the lifetime range as not being equivalent to a never expiring ticket and therefore the maximum token lifetime applies.

  This is not normally a problem because most Rx connections do not exist for longer than 30 days. However, the fileserver connections to the protection service do and the fileserver does not expect these connections to expire. Therefore, there was no logic in place to handle rxkad expiration errors.

  There are two changes that have been introduced in v2021.05-30 related to rxkad_krb5 superuser token lifetimes. The first is that when validating an rxkad_krb5 RESPONSE packet, if the service ticket is Kerberos v5 and the start time is 0 and the end time is 0x7fffffff, then treat that as a Kerberos v4 ticket with a valid range of start 0 and end 0xffffffff. In other words, accept the Kerberos v5 ticket with the range 0..0x7fffffff as a never expiring token.

  The second change is to the fileserver to handle rxkad expiration errors by generating a new token and establishing a new rxkad_krb5 connection to the protection service.

  Sites that have deployed yfs-rxgk keys to all AuriStorFS servers within their cells are unaffected by this bug. This is true even if the yfs-rxgk/_afs@ service principal has yet to be created or enabled.

• Reliability of IP ACLs.

  Transarc introduced IP ACLs as part of the AFS 3.1 release. The implementation in use to this day assumed that the PR_HostCPS RPC issued by the fileserver to the protection service could only fail for two reasons. First, the protection service did not support the PR_HostCPS opcode. Second, the requested entry did not exist. All failures were therefore treated as if the Host's Current Protection Set (set of group memberships) is equivalent to the empty set which was then cached for the fileserver configured "hostcpsrefresh" time which defaults to two hours.

  A PR_HostCPS call can fail for reasons that were not handled. For example, a protection server can return an out of memory error, all of the protection servers might be unreachable, or as has been discovered an rxkad expiration error might be returned. Processing the incoming call from a cache manager without the correct HostCPS set can result a change in the rights granted to the client which can result in an application failure.

  This release of the fileserver reacts to an inability to obtain the HostCPS in the same way that it behaves if the end user's CPS cannot be obtained. The incoming RPC from the client is rejected and a VBUSY error is returned. The VBUSY signal to the client to retry the request either to this fileserver or another one (if the data is replicated.) The fileserver will no longer cache the empty HostCPS set in case of failure.

  The implemented changes will improve the consistency of fileserver's handling of IP ACLs. However, it is worth reminding everyone that the network world when IP ACLs were created is very different from today's networks. IP ACLs can result in unpredictable behavior when clients are multi-homed (IPv4 + IPv6 or multiple IPv4 addresses), move between networks, or connect via VPN connections, The use of IP ACLs is therefore discouraged. As an alternative to IP ACLs, AuriStorFS permits rights to be granted or revoked from groups of authenticated keyed cache managers.

• Reconnection of yfs-rxgk authenticated connections after rekeying

  An under-specification of rxgk security class protocol and a flaw in the yfs-rxgk implementation has been identified. It was assumed that an Rx acceptor would not receive an initial Rx connection that had already been rekeyed. As a result of that assumption yfs-rxgk acceptors reject an rxgk authenticator encrypted with a key generation number other than zero. This assumption is incorrect. Even though it is true that all yfs-rxgk connections are created with an initial key generation number of zero there are at least two common circumstances during which an Rx acceptor will receive a call on a previously unknown Rx connection. The first case is when an Rx connection has been idle for more than ten minutes and it has been garbage collected by the acceptor but is still in use by the initiator. The second case is when an initiator changes endpoints because it has changed networks, joined a VPN, or is communicating behind a NAT/PAT device with a short lived port mapping. In these circumstances the acceptor might receive an rxgk RESPONSE packet for an Rx connection that has already been rekeyed at least once. yfs-rxgk connections are rekeyed after each 1.7TB of data is transferred.

  Failure to accept the RESPONSE can result in a cache manager marking a fileserver down or in the user's rxgk tokens being discarded. A complete fix requires that both the initiator and the acceptor be updated to v2021.05-30 or beyond.

• Performance of "vos convertROtoRW"

  Many sites use read-only volumes as an online backup mechanism. If a vice partition hosting read-write volumes fails, then a read-only volume on another fileserver is converted to become a replacement read-write volume. This release includes a number of performance and behavioral enhancements to reduce the time necessary to perform the conversion and simply the process. First, it is now assumed that if a read-write instance is unusable that any read-only or backup instance co-located are also unusable. Second, if the fileserver hosting the read-only that will be converted is v2021.05-26 or later, then the readonly volume will not be converted to a read-write volume but instead will be cloned to a read-write volume. These two changes reduce the number of required location service write transactions by 50% and the number of volserver vnode operations by 50%.

• Performance of "vos source", "vos interactive", "vos eachvol", "pts source", "pts interactive" and "pts eachuser/eachgroup"

  When executing "vos convertROtoRW" on large numbers of read-only volumes, use of "vos source" can be significantly faster than executing individual "vos convertROtoRW" commands. In addition to saving the overhead of starting a new process, if the "vos source" command can reuse a single Rx connection pool for all of the commands then it is possible to eliminate the cost of probing for the best server, identifying the ubik coordinator, and performing any required CHALLENGE/RESPONSE exchange.

  Prior to this release inclusion of a -cell , -verbose, -localauth, or -encrypt switch on the command being executed by a "source", "interactive" or "eachxxx" command would force the allocation of a new Rx connection pool for each executed command. Beginning with this release new Rx connection pools are allocated only if the state required by the command differs from the state of the top level command. This change increases the likelihood that the Rx connection pools can be reused; thereby saving time. If 10ms is saved for each of 100,000 executed commands, that is a savings of greater than 16 minutes.

• Reliability of volume operations initiated from behind a NAT

  Another change of note affects the use of "vos move[site]", "vos release", "vos copy[site]" when the "vos" process and one or both of the volservers are separated by a NAT/PAT/firewall, These vos operations involve a volume forward operation between two volservers. While the data is flowing between the volservers the active call between the vos process and the source volserver is has no DATA packets flowing in either direction. A PING ACK is sent by vos once every 100s. Unfortunately, many NATs have port mapping timeouts as short as 30s. If the volume forward takes longer to complete than the NAT mapping timeout, the volume forward completes successfully but the response is not received by vos. Instead, vos observes a network timeout and then attempts to clean up from what appears to be a failed volume forward. The v2021.05-29 release was intended to address this situation but the implemented changes were incomplete. v2021.05-30 ensures that the maximum time period between PING ACKs when an outgoing call is waiting for a response is 20s which should be sufficient to maintain the port mapping on NAT/PAT devices that aggressively prune the port mapping tables.

• AuriStorFS Clients and Servers version v2021.05 Patch 29 released.
• New platform: Linux 6.4 kernels.
• Execution of fs commands such as examine, whereis, listquota, listacl, cleanacl, setacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the yfs.ko kernel module.
• On Linux, prevents a kernel panic if the configured cache directory is located on a filesystem such as overlayfs which does not support the functionality required to be a cache.
• Improved operation of vos operations that involve a volume forward operation between two volservers when the vos process is separated from one or both of the volservers by a NAT/PAT/firewall which times out port mappings within 100 seconds.

• AuriStorFS Clients and Servers version v2021.05 Patch 28 released.
• Fixes for volserver bugs.
• New GPG signing key deployed for package compatibility with Fedora 38.

• AuriStorFS Clients and Servers version v2021.05 Patch 27 released.
• Fixes for bug introduced in v2021.05 Patch 26 within volserver and vos.

• AuriStorFS Clients and Servers version v2021.05 Patch 26 released.
• With this release AuriStorFS adds support for several new platforms:
  • Linux 6.3 kernel
  • SUSE SLES 15 (SUSE SolidDriver certified)
  • CrayOS 2.3 / 2.4
  • Red Hat EL8 Real Time kernels
  • Red Hat EL9 Real Time kernels
• AuriStor has begun to publish repositories for Linux aarch64. The first three platforms for which aarch64 have been published are:
  • Red Hat EL8, Alma8, Rocky8
  • Red Hat EL9, Alma9, Rocky9
  • Fedora 38
  Additional distributions will gain aarch64 repositories over the Summer.
• AuriStor has been a Red Hat Partner for many years and that partnership continues to get stronger and closer. AuriStor has now established partnership relationships with SUSE and HPE Cray in order to support their respective products. With SUSE AuriStor is now a member of the SolidDriver program which ensures that SUSE will not reject a support request from a SLES customer that has the AuriStorFS kernel modules installed. Instead, SUSE and AuriStor have an established procedure for mutually investigating support requests submitted to SUSE for SLES. HPE Cray's CrayOS is forked from SLES 15 and customized for each Cray installation. AuriStor's partnership with HPE Cray provides AuriStor with the ability to provide HPE Cray customers with custom builds of AuriStorFS for their Cray installations.
• AuriStor has also expanded our membership with TSANet to obtain mutual support for our customers with many industry leading hardware and software vendors: https://tsanet.org/members/.
• In addition to the new platform support, the v2021.05-26 has major improvements in the following areas:
  • Continued improvements in volume lifecycle management in the fileserver with substantial reductions in fileserver shutdown and restart times. Not all sites have more than a million volumes hosted on each fileserver, but if your does, the changes included in v2022.05-26 reduce the fileserver shutdown time from tens of minutes to a small number of seconds. Fileserver shutdown and restart time is critical to reducing the impact of fileserver updates on client systems.
  • The cache manager includes a number of bug fixes and one significant design change. The v2021.05-26 release alters the logic used to track the availability of volume content on each site. This state information is particular important when volumes become temporarily unavailable due to volume management operations such as backup, move, release, salvage, etc; or due to volumes being unavailable due to a fileserver being in the process of shutting down or not yet up again. The changes in this release force the client to immediately failover to an alternate site for any replicated volume when a fileserver indicates that it is restarting or that a volume is missing. Prior cache managers would stick with a fileserver when told that it was restarting either until the fileserver came back up or until the fileserver was shutdown for a sufficient period of time. If the fileserver took tens of minutes to shutdown the client would fail to failover until it did.
  • This release includes a significant change to the RX RPC stack's lifecycle management of incoming calls. A call once received transitions through the following states PRECALL -> QUEUED -> ACTIVE -> HOLD -> DALLY before it is garbage collected. The PRECALL state indicates that a call is not yet eligible to be attached or scheduled to a worker. The most common reasons are that authentication or reachability testing is pending or that the first DATA packet in a multi-packet call has yet to arrive. A QUEUED call is waiting for a worker to be assigned. An ACTIVE call is assigned to a worker. HOLD indicates that the call has completed but is waiting for the initiator to acknowledge the response DATA packets. DALLY indicates that the call is detached and ready for garbage collection. In prior releases if a call is in the PRECALL state when an ABORT from the initiator arrives, the call would be terminated immediately. However, a call in the QUEUED state required that it be assigned to a worker before it could be terminated. On extremely busy servers it has been observed that ABORTed calls can sit in the queue for up to 52 seconds before a worker is available to be assigned. Many initiated calls have connection dead times well under 52 seconds and these calls would timeout if the prior call issued in the same call channel keeps the call channel busy for longer than the dead time period. This release makes it possible for calls in the QUEUE state to be terminated immediately. This change not only prevents timeouts of incoming calls but also reduces the work load on the worker pool. These changes address timeout problems observed with vos processes performing write transactions with extremely busy location service coordinators.
  • The vos process has received some significant improvements.
    • verbose output is more detailed and complete. Site information (server and partition) is now included for each transaction and operation. Failures are more reliably logged.
    • A new keepalive mechanism has been added to ensure that idle volserver transactions do not timeout until vos is ready to terminate them. As volume sizes and the number of vnodes within a volume have increased many sites have reported that vos operations are failing because of transaction timeouts. These timeouts occur when a vos operation involves multiple servers and a volume clone must be created or deleted on one server prior to performing the next operation on another server's transaction. If a volserver transaction is idle for five minutes or longer, it is subject to automatic garbage collection. In this release, the vos process will schedule a keep alive operation on each idle volserver transaction once a minute to ensure that the transaction is not viewed as idle.
    • Many improvements have been made to "vos release" logic to address failures in various edge cases.
    • When both volservers are v2021.05-26 or later, moving a volume to a site containing a pre-existing ROVOL is now more efficient as the data in the ROVOL can be used as the starting point for an incremental move instead of a full move. This is particularly beneficial for volumes large numbers of vnodes that have not changed since the ROVOL was created.
  • There are improvements in AuriStor's DKMS processing.
  • Man page updates.

• AuriStorFS Clients and Servers version v2021.05 Patch 25 released.
• v2021.05-25 supports Linux kernels based upon the 6.1 or 6.2 mainline releases.
• The v2021.05-25 release is important for AuriStorFS database servers deployed in environments without reliable clock synchronization. The AuriStor UBIK database replication protocol requires synchronization of clocks within two seconds. The changes to the v2021.05-25 release will strictly enforce that votes cast for the coordinator be within the window ranging from two seconds before the start of the election to two seconds after the vote was received. Receipt of a vote with a timestamp outside of that window will result in the vote being discarded and the peer being marked down. Prior to this change it was possible for a vote to be cast with a timestamp up to 23 (ubik bigtime) seconds prior to the end of the election. As a coordinator is elected for a term of 21 seconds from the start of the election and an election is conducted every 10 seconds, it was possible for a coordinator to be elected to a term that either was about to expire or had already expired. This resulted in sporadic failures of write transactions.
• The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
• For Debian/Ubuntu systems which use DKMS to install the AuriStorFS kernel module, the v2021.05 release includes new functionality to retry DKMS if the matching AuriStorFS kernel module is missing from the system. This change will permit recovery when the first reboot into a new kernel of a Debian/Ubuntu system occurs without network access or prior to the release of a matching AuriStorFS kernel module.
• With this release the Linux /proc/fs/yfs directory tree has been moved to /proc/fs/auristorfs. A symlink from /proc/fs/yfs to /proc/fs/auristorfs is provided to ensure backward compatibility.
• A new /proc/fs/auristorfs/rxstats file can be used to read the RX statistics counters. This set of statistics uses 64-bit counters unlike the output from "rxdebug -rxstat" which is limited to 32-bit counters.

• AuriStorFS Clients and Servers released for Apple MacOS 13 (Ventura) on both Apple Silicon and Intel architectures.

• New Supported Platforms:
  • Fedora 37
  • Linux 6.0 kernels
• UNIX Cache Manager
  • Locally configured cell aliases can now be used when evaluating magic mount paths /afs/.@mount/<cell-name-or-alias>/<volume-name>/.
  • RXRPC Changes
    • RX calls are now created with a fixed initial congestion window instead of using the final congestion window from the most recently completed call. The use of a stashed value is inconsistent with RFC5681 and can reduce the transfer rate of subsequent calls.
    • The Sent ABORT packet statistics counter is now maintained and reported. It was missing since the initial AuriStorFS release.
  • Location Service (aka vlserver):
    • Write transactions that were begun but failed prior to completion are no longer committed. This avoids unnecessary propagation of a database relabeling without any data change.
    • This release introduces new 64-bit YFSVL CreateEntry and ReplaceEntry RPC to replace the broken but never called versions introduced in 2018 (v0.169).
    • ReplaceEntry RPC audit log entries now include the volume type and release type parameters. With this change it is possible to determine from the audit event if an implement ReleaseLock was performed.
    • GetEndpoints and GetAddrsU RPC audit log entries now include the requested fileserver UUID and Uniqifier values.
  • vos command:
    • Location server instances are categorized by the RPCs that known to be supported based upon which issued RPCs succeed or fail. The categorization algorithm was broken resulting in AuriStorFS vos commands never issuing the latest 64-bit RPCs and potentially falling back to the set of RPCs supported by IBM AFS 3.3. This release revises the algorithhm such that the AuriStorFS locations servers will be issued the latest 64-bit RPCs and fallback will proceed incrementally through older AuriStorFS server versions, OpenAFS, IBM AFS 3.6 and finally IBM AFS 3.3.
    • vos dump -file <file> will now unlink the <file> if the stored dump stream is incomplete.

• Location Service
  • Introduce new RPCs YFSVL_GetAuthoritativeEntryByName, YFSVL_GetAuthoritativeEntryByID, and YFSVL_AuthoritativeListAttributesU2. When available on all servers and the vos command these RPCs will be used to ensure that vos commands that modify the database will only fetch data from the elected Location Service coordinator.
• UBIK clients (vos, pts, afsbackup)
  • Avoid unnecessary delays when issuing RPCs that are not supported on all of the servers in the UBIK quorum.
• Volume Server
  • Since the introduction of v2021.05 vos listmaxacl, vos listrootacl and vos listseclevels failed to end the ITReadOnly volume transaction created to perform the query. This was due to a permission failure in the Volume Server.

• AuriStorFS Clients and Servers version v2021.05 Patch 21 released.
  • Volume Group Object Store
    • Volume clone removal time reduced by eliminating unnecessary backing store I/O operations.
    • A data-version-vnode object creation race that might result in spurious creation failures was fixed.
    • A data-version-vnode object reference count race that might result in improper deletion was fixed.
  • vos command line tool
    • vos backup and vos backupsys can now update a pre-existing backup volume while a vos release is in progress.
  • RX RPC
    • No longer send ACK packets in response to DATA packets if an ABORT packet has been sent.
    • The Sent RX BUSY packets counter is once again reported to rxdebug server port -rxstats.
    • Improve resiliency when the RX peer advertises unreasonable values for ACK packet trailer fields. This change permits continued communication with broken RX RPC implementations.
  • Linux kernel module
    • Linux 6.0 mainline kernels are now supported
    • Fix a build error with Linux mainline 5.19 or later kernels when the architecture is aarch64.
    • The kernel module now includes description, author and version information that can be displayed via modinfo.

• AuriStorFS Clients and Servers version v2021.05 Patch 20 released. This release:
  • Volume Server changes
    • The volserver validation checks introduced in v2021.05-19 break the restoration of incremental volume dumps. This release fixes the regression and adds tests to validate the behavior.

• AuriStorFS Clients and Servers version v2021.05 Patch 19 released. This release:
  • FileServer Updates
    • Reorganize how Volumes are initialized for use with the VLRU. This change avoids the possibility of an assertion failure if a volume is placed into an error state without it being detached.
    • Prevent a race during startup which could result in a core dump due during startup of the VLRU Scanner Thread. This race was introduced in 2021.05-18.
    • When creating a cross-directory hard link, delay the assignment of a per-file ACL until after the copy-on-write operation succeeds.
  • Protect Service changes:
    • Prevent crash if authenticated Kerberos v5 identity contains a dot in the first component and "allow-dotted-principals" is disabled.
  • Volume Server changes
    • When a Volume Dump RPC fails due to an RX Peer Unreachable error log the ICMP error details (if available).
    • Introduce additional consistency checks when receiving a dump stream.
  • Salvage Service changes
    • Do not assign a parent directory vnode to orphaned files with attached per-file ACLs.
  • Backup Service changes
    • Prior releases faailed to re-open log files after receiving a signal from logrotate.
  • Volume Package changes
    • The volume group link table limit that prevented a vnode from being linked to more than seven volumes within a volume group has been removed. Exceeding this limit when creating a new volume clone (readonly, backup or other) could result in the entire volume group becoming unusable. The limit of seven was derived from the design of the IBM/OpenAFS volume group object store format. There is no design limit in the AuriStorFS object store format.
  • UBIK Service changes
    • Prevent threads attempting to perform a write transaction from jumping ahead of a queue of threads waiting for exclusive access.
  • UBIK Client changes
    • When it is known that a coordinator is required to complete an RPC, increase the RX connection dead time from 12s to 60s in case the coordinator is under heavy load.
    • When it is known that a coordinator is required to complete an RPC, disable the RX hard dead timeout since it is not safe for a write transaction RPC to be retried.
  • RX RPC
    • Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
    • Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
    • Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers on macOS (KERNEL) and Linux (userspace). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.
  • vos subcommand changes
    • Do not default the -clone switch to yes if the volume type is readonly or backup. There is no benefit to using a clone for these volume types as the volumes are not taken away from the fileserver during the volume operation. dump, shadow, and copy are affected.
    • Error move volume sooner if the volume type is not read-write.
    • Error movesite volume sooner if the volume type is not readonly.
    • Error copysite volume sooner if the volume type is not readonly.
    • When dumping a volume using a clone the clone will be flagged as temporary to ensure that if the transaction is interrupted that the clone will be automatically garbage collected.

• AuriStorFS Clients and Servers version v2021.05 Patch 18 released. This release:
  • New Supported Platforms:
    • Linux Kernel 5.19
  • FileServer Updates
    • Improved tracking of idle but in-use volumes to avoid unnecessary volume salvaging after an emergency fileserver restart or transition from active to passive server instance.
    • Streamlined the fileserver shutdown process to reduce volume contention. These changes ensure that a fileserver with millions of attached volumes can perform a clean shutdown in just a few seconds.
    • Prevent idle dead timeouts during StoreData calls exceeding 8MB of data over slow lossy networks.
    • Protect against signed extension parameter overflow when processing RXAFS_FetchData calls.
  • Volume Server changes
    • Include temporary volumes (those flagged as 'destroyMe') in the output of AFSVolListVolumes and AFSVolListOneVolume calls.
  • Cache Manager
    • Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
    • The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
  • RX RPC
    • Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
    • The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
  • Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

• AuriStorFS Clients and Servers version v2021.05 Patch 17 released. This release:
  • improves the reliability of the RX RPC protocol ACK packet processing and congestion avoidance algorithms.
  • hints the creation of outgoing fileserver callback connections to improve the selection of a compatible network interface.
  • improves the reliablity of UBIK client processes such as vos that mix database reads with database writes. Such clients can write stale data to the database if the source data is read from a non-coordinator replica of the database that is temporarily out of sync with the coordinator.
  • New Supported Platforms:
    • Red Hat Enterprise Linux 9.0
    • Red Hat Enterprise Linux 8.6
    • Fedora 36
    • Ubuntu 22.04
    • Linux Kernel 5.18
    • Debian "arm hard float"
  • Cell Service Database Updates
    • Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
    • Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
    • Add ee.cooper.edu
    • Restore ams.cern.ch, md.kth.se, italia

• AuriStorFS Client installers beginning with the v2021.05-9 release are supported on macOS 12 Monterey on both Apple Silicon and Intel Macs. Note: please upgrade Big Sur, Catalina, and Mojave macOS systems to AuriStorFS v2021.05-9 before upgrading macOS to Monterey. macOS Monterey will deadlock during shutdown when previous versions of AuriStorFS built for Big Sur, Catalina and Mojave are installed.

• Leveraging AFS Storage Systems to Ease Global Software Deployment
  Tracy J. Di Marco White, Goldman Sachs
  Using AFS as both a file store and an object store, we provide software to hundreds of thousands of client systems within both public and private cloud. As we see a continual increase in the frequency of software deployments, in the number of different software packages, and in the number of versions of each software package, we have also adapted our software deployment systems. Both of our software deployment systems use AFS, but one is unaware of AFS, and one makes specific use of various AFS features. I'll cover how the infrastructure has grown from several private data centers, and how our use of AFS has eased migration to both private and public cloud. I'll discuss the changes we are making to both the AFS-unaware and AFS-aware deployment systems, as well as discuss bugs, bottlenecks, and patterns of software development and usage that we've discovered through the change process.
• Hands-Off Testing for Networked Filesystems
  Daria Phoebe Brashear, AuriStor, Inc.
  Cross-platform network filesystems require testing, but in-kernel interface testing is problematic under the best of circumstances. This talk will discuss the techniques used at AuriStor for automating hands-off testing using buildbot, TAP, docker, and kvm.

• This release of AuriStorFS is primarily bug fixes for the UNIX Cache Managers and UBIK services (Location Service, Protection Service, and Backup Database Service).
• The UBIK service changes include several important corrections for mistakes introduced as part of the v0.208 and v2021.04 performance and reliability improvements. AuriStor recommends that all UBIK servers be updated to v2021.05.
• The UNIX Cache Manager changes provide additional data integrity checks during directory lookup operations. Some sites have observed the sporadic creation of negative dentries on LINUX in directories that have not been modified in over a year. Negative dentries are created when a lookup request fails with an ENOENT error. A year ago there were nearly locations where an ENOENT error could be returned. There are now only six and each location is one where an ENOENT is the correct result. The additional directory integrity checks are intended to ensure that an ENOENT cannot be generated as a result of the lookup being performed on invalid data. If a data integrity error is detected, an EIO error will be returned and a warning will be logged including the directory FileID and the disk cache chunk index value.
• Other bug fixes include Direct I/O and StoreMini RPCs would overwrite fileserver returned error codes such as VBUSY, VOFFLINE, ENOSPC, EACCES, EPERM, etc with a locally generated RXGEN_CC_UNMARSHAL (-451) error. These two code paths were missed when equivalent changes were applied elsewhere in the v0.205 release.
• The UNIX Cache Manager also received replacements for the subsystems responsible for managing the volume locations servers for each cell; per cell token management for each user/pag, and rx connection vectors. The new infrastructure does not rely upon the legacy global lock for thread safety and therefore permits increased parallism. The volume location server management supports multi-homed servers. Previously, each endpoint of a multi-homed server was treated as a unique server which potentially increased the delays when failover due to server restart or network path connectivity issues occurred. With the new model server probes select the best available endpoint. Failover due to Ubik errors no longer issues an RPC to an alternative endpoint belonging to a previously queried server.
• This release permits the selection of the "auth" (integrity protection only) rx security level in addition to "clear" and "crypt" (privacy and integrity protection). Use "fs setcrypt auth". For the first time users can request yfs-rxgk tokens that prefer "clear" or "auth" levels instead of "crypt". Use the new "aklog -levels <levels-list>" where <levels-list> is a comma separated ordered list of preferred minimum levels. When generating a yfs-rxgk token for an AuriStorFS cell, the RXGK service will pick the first level that is acceptable to the cell's policy. By default, the order is "crypt,auth,clear". Use of "clear" and "auth" is discouraged but might be useful to obtain additional speed by eliminating cryptographic algorithm overhead. v2021.05 is the new recommended release.
• A variety of improvements for the fileserver and volserver are included in this release but nothing urgent.

• AuriStorFS v2021.04 contains another round of significant improvements to the UBIK replication infrastructure used by the location, protection and backup services. The changes remove a serialization point by providing further data isolation between read-transactions and write-transactions. This change further reduces the risk of a thundering herd of reader threads subsequent to write-transaction completion in a heavily loaded cell (six ubik peers, peaks of 7000 reads/second/peer, and average of 22 writes/second). This release also adds additional data consistency protections to the UBIK application services (vlserver, ptserver, and buserver). Sites that take advantage of the protection service's groups of groups (aka "supergroups") capability will appreciate a noticeable reduction in CPU usage when responding to fileserver queries for a user's current protection set (CPS). The RXGK service co-located with the location service includes a change to prepare for issuance of tokens that do not mandate wire privacy and integrity protection. AuriStor recommends that all UBIK database service instances be updated to v2021.04.
• AuriStor recommends that fileservers be updated when convenient to do so. There is a risk of partition lock deadlock during fileserver startup introduced in v0.209 and a potential vnode metadata consistency issue when clients perform cross-directory renames of vnodes with a link count greater than one.
• The UNIX / Linux cache manager changes are primarily bug fixes for issues that have been present for years. A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs. A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes. On Linux, the potential of an overwritten ERESTARTSYS error during fetch or store data rpcs could result in transient failures. Upgrading to v2021.04 is recommended but not urgent.

• v0.209 introduces a new cache manager architecture on all Linux platforms. The new architecture includes a redesign of:
  • kernel extension load
  • kernel extension unload
  • /afs mount
  • /afs unmount
• New platform support includes:
  • Linux mainline 5.11 and 5.12 kernels
  • gcc11 compilation
  • Updates for Linux ppc64 and ppc64le architectures
  • Hardware accelerated cryptographic routines for Linux __aarch64__.
  • Ubuntu 18.04 and 20.04 -oem kernel modules.
• Higher throughput read and write transactions for UBIK based location, protection and backup database services. The combination of write transaction isolation and lookup caching substantially increases the rate of read and write transactions while reducing the risk that read transactions can block for extended periods.
  Compared to v0.200, these changes have reduced the minimum write transaction time by 75%, the mean by 50% and the maximum by 13%. The rate of write transaction completion increased by 35%.
• unix cache manager negative volume name lookup caching
• bos getfile - similar to bos getlog but can be used to transfer files with arbitrary binary contents.
• asetkey delete by key sub-type.
• CIDR based assignment of server endpoint priorities.
• Over 600 additional improvements.

• v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:
  • kernel extension load
  • kernel extension unload (not available on Big Sur)
  • /afs mount
  • /afs unmount
  • userspace networking
• The conversion to userspace networking will have two user visible impacts for end users:
  • The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.
  • Observed network throughput is likely to vary compared to previous releases.
• On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.
• AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
• AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

The v0.200 release is an important release targeted at AuriStorFS server deployments and Unix cache managers.

fileserver: remote denial of service
Impacted versions: All releases from v0.116 through v0.198 are affected.
CVE: None
CVSS Base Score:             4.9
Impact Subscore:             3.6
Exploitability Subscore:     1.2
CVSS Temporal Score:         4.8
CVSS Environmental Score:    3.1
Modified Impact Subscore:    1.8
Overall CVSS Score:          3.1
CVSS v3.1:
AV:N/AC:L/PR:H/UI:N/S:U/C:N/I:N/A:H/E:F/RL:U/RC:C/CR:X/IR:X/AR:L/MAV:N/MAC:L/MPR:H/MUI:N/MS:U/MC:N/MI:N/MA:H
Details on this denial of service vulnerability will be disclosed thirty
(30) days after release to give customer sites an opportunity to update
their fileservers to v0.200.

Linux cache managers prior to v0.199 are susceptible to a general protection fault if a server unreachable network error occurs during a direct i/o operation.

The macOS backgrounder in prior releases can repeatedly segmentation fault and restart when there is no network connectivity.

This release includes major improvements in the handling of RPCs that are interrupted by NAT/PAT devices timing out their udp endpoint mappings.

The v0.198 release is a security release targeted at AuriStorFS server deployments.

vlserver: remote denial of service
Impacted versions: All releases from v0.193 through v0.197 are affected.
CVE: CVE-2020-26119
CVSS Base Score:             8.6
Impact Subscore:             4.0
Exploitability Subscore:     3.9
CVSS Temporal Score:         8.0
CVSS Environmental Score:    9.3
Modified Impact Subscore:    5.9
Overall CVSS Score:          9.3
CVSS v3.1:
AV:N/AC:L/PR:N/UI:N/S:C/C:N/I:N/A:H/E:F/RL:O/RC:C/CR:X/IR:X/AR:H/MAV:N/MAC:L/MPR:N/MUI:N/MS:C/MC:N/MI:N/MA:H
Details on this denial of service vulnerability will be disclosed only
after all customer sites have updated their location servers to v0.198.

The v0.197 release includes significant performance improvements for UNIX cache managers. Especially for macOS and Red Hat Enterprise Linux (including derivatives). macOS users accessing AuriStorFS cells with yfs-rxgk will experience AuriStor's proprietary hardware accelerated implementations of aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha384-192, and optimistic caching of vnode status information for the first time. Linux users will benefit from more aggressive optimistic caching of status information as well as support for SELinux labels and the world's first path-ioctl implementation for FUSE. CentOS users will appreciate the dedicated repository for CentOS kernel modules. This release also introduces support for the Linux mainline 5.8 and 5.9 kernels.

Cell administrators will appreciate an improved vos eachvol and new pts eachuser and pts eachgroup commands. The new pts whoami -rxkad switch and improved logging of yfs-rxgk authentication failures ease debugging of authentication and authorization configuration errors.

All servers have been updated to improve reliability and performance. New audit events have been added to the fileserver when a rename operation unlinks a existing target. A rename that replaces a directory will no longer orphan the unlinked directory. Volserver transactions can no longer be stolen by another cell administrator. UBIK service coordinators protect themselves against peers that accept a transaction rpc but never complete it.

Finally but not least, AuriStor has introduced fs ignorelist to replace fs blacklist. Likewise ignorelist-dns, ignorelist-volroot, ignorelist-afsmountdir, and ignorelist-volrootprefix replace the format [afsd] "blacklist" variants. In all cases "blacklist" is accepted as a hidden alias.

The v0.195 release is a CRITICAL update for all macOS and Linux cache managers. The changes in v0.195 correct bugs that can result in data corruption.

v0.195 also introduces support for Linux 5.7 kernels.

The v0.194 release is a CRITICAL update for all UBIK servers (vlserver, ptserver, buserver) and the macOS and Linux cache managers. The changes in v0.194 correct bugs that can result in data corruption in all of the above.

v0.194 also introduces support for Linux 5.6 kernels and Red Hat Enterprise Linux 7.8.

The 0.192 release is primarily a bug fix release focused on the Linux cache manager, the fileserver/salvageserver and "vos".

This fileserver fixes are critical for any server with a vice partition supporting Linux reflinks (xfs+reflinks, btrfs, ocfs2). AuriStor is unaware of any customers operating fileservers with these configurations. There is also a fix to permit salvaging volumes containing more than 76 million vnodes.

The Unix cache manager changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the auristorfs cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

Changes to the "vos move" and "vos movesite" commands preserve the volume's last update timestamp ensuring proper calculation of the change set for incremental transfers. The bug can result in reversion to the contents of the BACK volume if a RW volume with an existing BACK volume is moved twice without first removing or updating the BACK volume.

The AuriStor File System v0.191 release addresses bugs that were identified after the release of v0.190, compatibility with RHEL7 kernels suffering a regression, and improvements in Linux cache manager startup/shutdown. Notable changes include:

• Fileserver bug fixes when Linux reflinks are supported by vice partition backing stores. xfs+reflinks, btrfs or ocfs2.
• Re-enabling SIMD processor extensions for non-RHEL userland Linux processes when the kernel module does not export __kernel_fpu_begin or __kernel_fpu_end.
• Improvements in Linux kernel module startup and shutdown that reduce the risk that a reboot will be required.
• Work-around for a RHEL 7.6 and 7.7 regression that impacts Linux systems configured to export /afs via nfsd.

The AuriStor File System v0.190 release addresses bugs that were identified shortly after the release of v0.189.

The v0.189 release includes a broad range of performance improvements, new features, and of course, bug fixes. The highlights include:

• UNIX Cache Manager performance improvements
• Faster "git status" operation on repositories stored in /afs.
• Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See UNIX Cache Manager large file performance improvements).
• New Fileserver support for Data Loss Prevention scanning and Backup solutions that walk the directory tree. (See New Feature: Fileserver Implicit ACLs).
• A major rewrite of the Ubik recovery engine to eliminate contention with the coordinator election procedures. These changes ensure that regardless of the database size or network latency, Ubik recovery can no longer alter the timing of election ballots. In prior releases a lengthy recovery could prevent an election from being conducted which could result in the expiration of the coordinator's term. Reduction in thread contention will also enhance the performance of the vlserver, ptserver and buserver.
• Reduced Ubik transaction times resulting from parallelization of quorum updates.
• Many improvements intended to reduce the amount of data included in an incremental volume dump and/or avoid the need for generating a full volume dump during a "vos release".
• New support for acceptor-only keys that permit key rotation without single points of failure or flag days. (See "New Feature: Key Rotation Using Acceptor Only Keys).
• Auditing improvements.
• Updated platform support including Fedora Core 31 and Oracle Linux
• and much more ...

AuriStorFS v0.188 installer for macOS Catalina (10.15) released in conjunction with Apple's release of macOS Catalina (10.15).

v0.188 addresses three issues experienced by customers.

The first is UBIK coordinator term expiration of the location service after periodic load spikes that increased the size of the vlserver thread pool from 100 threads to more than 13,000 threads. The load spike would last for under a minute, the thread pool would scale back to 100 threads after 20 minutes. Forty minutes later another load spike would occur repeating the pattern. The existing rx packet allocator behaved very poorly under this workload pattern resulting in allocation of additional rx packets with each load spike. After a week more than five million rx packets had been allocated on some vlservers.

As the allocated packet counts increased and the number of threads decreased, the packets per thread ratio increased as well. When the thread pool resized to 100 threads the number of packets assigned to each thread grew to the point where packet transfers began to interfere with rx data transfer and event processing. The UBIK coordinator election algorithm is time sensitive and a failure to deliver votes or timeout RPCs in timely manner can result in election failure.

v0.188 addresses the root causes by replacing:

• the rx multi call implementation used to conduct UBIK elections with a new variation that manages its own timeouts and does not rely upon timeouts set upon each individual rx rpc.
• the condvar timed wait implementation with a version that has finer grained clock resolution: 1ns instead of 1s.
• the rx packet allocator with a new implementation that is better suited for use with dynamic thread pools and larger window sizes. The new allocator also significantly reduces lock contention when obtaining and releasing packets.

The second problem is loss of volume access after the fileserver writes to the FileLog:
CopyOnWrite corruption prevention: detected zero nlink for volume N inode vnode:V unique:U tag:T (dest), forcing volume offline

No data corruption occurs but after each occurence the volume is salvaged. After the 16th automatic salvage the volume is taken offline until there is manual intervention by an administrator.

This bug, a file descriptor leak, was introduced in v0.184 as a side effect of one of the fixes for the libyfs_vol reference counting errors.

The final issue is the on-going problems that some customers have experienced with Linux clients either with the shell reporting:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

or "mount --bind" failing with

mount: mount(2) failed: No such file or directory

or

$ cd /afs/example.com/
$ ls -al /proc/self/cwd

generates

/proc/self/cwd -> /afs/example.com (deleted)

The symptom occurs when a Linux dentry (directory entry) object ends up in an unhashed state although it is referenced by an inode.

Since v0.180 AuriStor has revised code paths to improve error code reporting and avoid race conditions that can generate this behavior. Apparently, there are still additional conditions that have yet to be identified. v0.188 includes a band-aid whereby an unhashed denty will be rehashed when needed. However, AuriStor is still trying to find and address the root cause. Therefore the AuriStorFS kernel module will log a warning when a dentry is rehashed

Changes since v0.184 are primarily focused on the UNIX/Linux cache manager and fixing operational issues reported in the volserver and fileserver. The v0.184 release implemented major changes to the UNIX/Linux cache manager. This release fixes bugs introduced in v0.184 and missed edge cases. It also continues the refactoring of internal interfaces to propagate error codes and signals to userland applications. VolserLog messages related to the volserver transaction lifecycle have been thoroughly revamped. Additional reliability improvements within RX are included.

• New Platforms:
  • Red Hat Enterprise 8
  • xfs reflinks (requires new filesystem) changes AuriStorFS StoreData RPC copy-on-write performance from O(filelength) to O(write length).
  • significant improvements in udp performance compared to rhel7.
  • extended berkeley packet filters provides for fairer distribution of rx call processing across multiple rx listener threads.
  • Fedora 30
• Unix CM:
• v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.
• v0.184 introduced a potential deadlock during directory processing. This is fixed.

• Many sites have noticed that clients with v0.184 installed might log Lost contact with xxxx server ... referencing a strange negative error code and that fileservers might log FetchData Write failure ... errors from any Linux client version.
  These errors might correlate to corruption of pages in the Linux page cache. The corruption is that one or more contiguous pages might be inappropriately zero filled.
  This release implements many code changes intended prevent Linux page cache are AFS disk cache corruption.

  • Better data version checks
  • More invalidation of cache chunk data version when zapping
  • Only zero fill pages past the server end of file
  • Always advance RPC stream pointer when skipping over missing pages or when populating pages from the disk cache chunk.
  • Never match a data version number equal to -1.
  • Avoid truncation races between find_get_page() and page locking.

• Some sites have experienced failures of Linux mount --bind of /afs paths or getcwd returning ENOENT. This release fixes a dentry race that can produce an unhashed directory entry.
  Some uses of the directory will continue to work, as the first lookup following the race will associate a new dentry with the inode, as an additional alias. Directories are not supposed to have aliases on Linux, so the vfs code assumes that d_alias is at most a list of 1 element, and accesses the entry in a slightly different way in a few places. Some sites get the new hashed dentry, others get the original unhashed one.

• Propagate EINTR and ERESTARTSYS during location server queries to userland.
• Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT
• Log all server down and server up events. Transition events from server probes failed to log messages.
• Avoid leaking local errors to the fileserver if a failure occurs during Direct IO processing.
• RX RPC networking:
• If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
  Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.
• Avoid event cancellation race with rx call termination during process shutdown. This race when lost can prevent a process such as vos from terminating after successfully completing its work.
• Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.
• Volserver:
• Frequent issuance of "vos listvol" commands can no longer interfere with volume transaction idle timeout processing.

• Since IBM AFS 3.5 the volserver has logged transaction status every 30 seconds to the VolserLog. In v0.184 the volserver logs the following lifecycle messages at level 0:

  1. trans id on volume id is older than s seconds
  2. trans id on volume id has timed out
  3. trans id on volume id has been idle for more than s seconds

  
  On a busy volserver these messages can flood the VolserLog.
  This change raises the level of messages 1 and 3 to 125 and introduces a new "Created trans id on volume id" message logged at level 5.
  With this change level 0 logs unexpected termination of each transaction. Level 125 will include the 30 second updates for sites that require them.
  The partition, volume parentid and transaction iflags fields have been added to each log message.

• RPCs issued by vos listvol will no longer block in the volserver if the requested volume requires salvaging. The volume attachment retries can block the salvageserver from acquiring an exclusive volume lock resulting in a salvage failure and a soft-deadlock. From now on the vos listvol command will fail immediately.
• Fileserver:
• If the vice partition's backing store is unmounted or otherwise becomes unavailable the fileserver could terminate unexpectedly due to a segmentation fault. Beginning with this release the fileserver will survive but all requests for objects stored on the missing vice partition will fail.

• Introduce the ability to configure random error injection during FetchStatus, FetchData, and StoreData RPC processing.

• Add File IDs to "FetchData Write Failure" FileLog messages.
• Ubik services:
• This release introduces the ability to configure a separate debug log level for ubik than for the application service. By default, when the "ubik_debug" level is unspecified or set to zero, the application's log level determines which "ubik: " log entries are written to the log.

• Security improvements include volserver validation of destination volserver security policies prior to transmitting marshaled volume data. Prior to v0.184 the volservers were trusted to reject volumes whose security policy could not be enforced. Linux cache managers can no longer be keyed with rxkad tokens. Introduction of a pam module capable of managing tokens for both AuriStorFS and/or Linux Kernel kAFS.

• The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error". These changes are expected to reduce the likelihood of "mount --bind" and getcwd failures on Linux with "No such file or directory" errors.

• A potentially serious race condition and reference counting error in the vol package shared by the Fileserver and Volserver could prevent volumes from being detached which in turn could prevent the Fileserver and Volserver from shutting down. After 30 minutes the BOSServer would terminate both processes. The reference counting errors could also prevent a volserver from marshaling volume data for backups, releases, or migrations.

• This release is moves the location of the cache manager's cmstate.dat from /etc/yfs/ to /var/yfs/ or /var/lib/yfs depending upon the operating system. The cmstate.dat file stores the cache manager's persistent UUID which must be unique. The cmstate.dat file must not be replicated. If virtual machines are cloned the cmstate.dat must be removed. The cmstate.dat file must not be managed by a configuration management system.

• The release includes two new vos command options:
  
  * "vos addsite -force"
  * "vos listvol -id "
  

• Finally, this release includes a Linux PAM module as well as support for the Amazon Linux 2 distribution and many more quality and performance improvements

AuriStor, Inc. is pleased to sponsor and invite AFS and Linux kernel developers to the second Linux kernel AFS (kAFS) Hackathon and Birds of a Feather meeting. The hackathon and BoF will be co-located with the USENIX Vault '19 - Linux Storage and Filesystems Conference. Read more...

• This release improves RX call reliability across network paths with a high degree of packet loss and/or round trip times larger than 60ms. The corrected bugs have been present in all IBM derived RX implementations dating back to the mid 90s. The impact of these bugs is an increased risk of timeouts and performance degradation for long lived calls over high latency network paths that periodically experience packet loss. Volume operations such as moves, releases, backups and restores over WAN connections are particularly susceptible due to the amount of data transmitted in each RPC.

• One feature change is experimental support for RX windows larger than 255 packets (360KB). This release extends the RX flow control state machine to support windows larger than the Selective Acknowledgment table. The new maximum of 65535 packets (90MB) could theoretically fill a 100 gbit/second pipe provided that the packet allocator and packet queue management strategies could keep up.

• A change to volume use statistics tracking when volumes are moved, copied, and restored. The AFS volume dump stream format which is used for volume archives and volume transfers can store the daily and weekly vnode access counts but none of the other extended volume statistics maintained by the fileserver. When a volume is moved it makes sense for the use counts to be migrated with the volume to the new location. When a volume is copied it makes sense that the new location should start its counters from zero instead of the values collected at the location that was used as the source. Finally, when restoring a volume or releasing a new snapshot of a volume to readonly or backup sites, the use counts should remain unaltered. Beginning with this release, when AuriStorFS v0.178 or later "vos" is used in combination with an AuriStorFS v0.178 or later destination "volserver" the desired use count management will take place. At the moment the weekly access counts are only accessible when using the "vos examine -format" switch.

• Many more quality and performance improvements

• AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

  When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

• Workarounds for an IBM AFS and OpenAFS RX header userStatus field information leakage bug. This bug inadvertently interferes with the RX service upgrade mechanism that permits AuriStorFS clients (including Linux kafs) and services to detect each other without undesireable timeouts or extra round trips.

  When an affected IBM or OpenAFS cache manager (or fileserver) establishes a connection to an AuriStorFS server the bug can result in an unintentional RX service upgrade. For example, if a pre-v0.175 fileserver incorrectly upgraded an incoming RX connection from RXAFS to RXYFS, it would mistakenly believe the client offered the RXYFSCB callback service; which it doesn't. The failure to establish a successful connection to the RXYFSCB service would cause the fileserver to reject the client's RXAFS requests with a VBUSY error.

• A fileserver is supposed to be able to serve data from a .readonly or .backup volume while the volserver is dumping or forwarding the volume contents. This functionality introduced in IBM AFS 3.3 was fatally broken in AuriStorFS v0.157 when the volume disk interface was overhauled to avoid data corruption. Then starting with v0.168 "vos release" failed to terminate the volume transaction used to clone the RW volume to the RO site on the same server. Attempts to read from volumes that were exclusively in-use by the volserver would return VOFFLINE (106) errors.

  As of AuriStorFS v0.175 release .readonly and .backup volumes can once again be attached to fileservers while a "vos release" or "vos dump" command is in process. Since some of the fixed defects were in "vos" and others in the fileserver both "vos" and the fileserver must be updated to v0.175 to ensure correct behavior.

• A major security model change to the Backup Tape Controller (butc), backup coordinator command, and the backup service to address OPENAFS-SA-2018-001.txt

  Starting with v0.175 butc supports:

  • yfs-rxgk and rxkad authentication
  • AES256-CTS-HMAC-SHA1-96 or 56-bit fcrypt wire encryption
  • super user authorization
  • auditing of all remote procedure call requests

  The new security model is incompatible with the existing "backup" and "butc" processes. The new "butc" always executes using "localauth" credentials just as any other cell service does; it can no longer be executed using tokens obtained via aklog.

  The butc service will by default require all incoming RPCs to be authenticated as a super user either via use of -localauth credentials or end user identities found in the UserListExt or ReaderList bosserver configuration.

  As a side effect of these changes, both backup and butc gain IPv6 support.

  As the new security model is incompatible with the existing deployed butc and backup processes, the 0.175 version includes configuration knobs to force the use of the old security model for backward compatibility. Use of these knobs restores the privilege escalation vulnerability. Please contact AuriStorFS support if your site requires use of this configuration.

• New data input validation improvements within the vlserver and volserver. These changes ensure that the vlserver cannot store volume location records referencing invalid fileservers or volume site parameters; and that the volserver cannot forward volume data to volservers that are not registered with the cell's location service.

  The same validation has been added to vos to ensure that it cannot be instructed to violate cell constraints.

• Correct Linux disk cache management to support AppArmor sand boxes.
• Many more quality and performance improvements.