fileserver - Initializes the File Server component of the dafs process
fileserver [-afscompat [yes | no]] [-auditlog [<path to log file> | yes | on]] [-audit-interface (file | syslog | sysvmq)] [-debug <debug level>] [-spare <number of spare blocks>] [-pctspare <percentage spare>] [-b <buffers>] [-l <directory vnodes>] [-s <file vnodes>] [-vc <volume cachesize>] [-banner] [-novbc] [-implicit <admin mode bits: rlidwka>] [-readonly] [-hr <number of hours between refreshing the host cps>] [-busyat <redirect clients when queue > n>] [-rxpck <number of Rx extra packets>] [-rxdbg] [-rxmaxmtu <bytes>] [-rxwindow <packets] [-rxbind] [-allow-dotted-principals] [-L] [-S] [-realm <Kerberos realm name>] [-udpsize <size of socket buffer in bytes>] [-sendsize <size of send buffer in bytes>] [-abortthreshold <abort threshold>] [-enable_peer_stats] [-enable_process_stats] [-syslog [< loglevel >]] [-mrafslogs] [-cve-2018-7168 <re-enables opcode RXYFS_StoreOpaqueACL and RXYFS_StoreACL >] [-help] [-vhandle-setaside <fds reserved for non-cache io>] [-vhandle-max-cachesize <max open files>] [-vhandle-initial-cachesize <fds reserved for non-cache io>] [-vattachpar <number of volume attach threads>] [-m <min percentage spare in partition>] [-ignore-state] [-vhashsize <log(2) of number of volume hash buckets>] [-vlrudisable] [-vlruthresh <minutes before eligibility for soft detach>] [-vlruinterval <seconds between VLRU scans>] [-vlrumax <max volumes to soft detach in one VLRU scan>] [-unsafe-nosalvage] [-offline-timeout <timeout in seconds>] [-offline-shutdown-timeout <timeout in seconds>] [-sync <sync behaviour>] [-logfile <log file] [-config <configuration file] [-maxthreads <maximum number of worker threads>] [-threadidle <maximum idle lifetime for a worker thread in seconds>]
The fileserver command initializes the File Server component of the dafs bnode.
In the conventional configuration,
its binary file is located in the /usr/local/libexec/yfs directory on a file server machine.
The fileserver command is not normally issued at the command shell prompt but instead executed by the bosserver(8) as part of the dafs bnode created using bos create(8).
When -syslog mode is disabled, the File Server creates the /var/log/yfs/FileLog log file during startup, if the file does not already exist. It does not write a detailed trace by default, but the -debug option may be used to increase the amount of detail. The bos getlog command can be used to remotely fetch the contents of the log file.
The File Server's configuration permits the administrator to adjust many aspects of the File Server's performance, as detailed in "OPTIONS". By default the File Server sets values for many arguments that are suitable for common deployments. The following list describes the parameters and corresponding arguments for which the File Server sets default values, and the table below summarizes these settings.
The maximum number of threads the File Server may attempt to create to handle requests for data; corresponds to the -maxthreads argument.
The minimum acceptable value is 6.
The default can be between 512 and 16384 depending upon the operating system and the configured RLIMIT_NOFILE value for the process.
A minimum of 4 file descriptors is required by each worker thread.
The actual number of threads that can be created is determined by the operating system.
The number of worker threads dynamically grows and shrinks to satisfy the number of simultaneously received requests. New threads are created on demand and threads retire after being idle for -threadidle seconds.
The maximum number of directory buffers the File Server caches in memory; corresponds to the -b argument.
The default number of directory blocks is the larger of 120 and twice the maximum number of worker threads that can be created.
Each cached directory buffer consumes 2,092 bytes of memory.
The initial volume header cache size,
which determines how many volume headers the File Server can cache in memory before having to retrieve data from disk; corresponds to the -vc argument.
The default number of volume headers is the larger of 600 and the maximum number of worker threads that can be created.
Each volume header consumes 576 bytes of memory.
The maximum number of directory (aka large) vnodes the File Server caches in memory for manipulating directory objects; corresponds to the -l argument.
The default is the larger of 600 and three times the size of the volume header cache.
Each directory vnode consumes 320 bytes of memory.
The maximum number of file (aka small) vnodes the File Server caches in memory for tracking file objects; corresponds to the -s argument.
The default is the larger of 600 and ten times the size of the volume header cache.
Each file vnode consumes 320 bytes of memory.
The initial number of Rx packets the File Server allocates at startup; corresponds to the -rxpck argument.
The default is 200 packets.
Each packet consumes 1544 bytes of memory.
The default values are:
Parameter (Argument) Default Value ---------------------------------------------------------------------- Minimum number of workers (-minthreads) 6 Maximum number of workers (-maxthreads) min(16384, OS limit) Number of cached dir blocks (-b) max(120, 2 x (-maxthreads)) Volume header cache size (-vc) max(600, (-maxthreads)) Number of cached directory vnodes (-l) max(600, 3 x (-vc)) Number of cached file vnodes (-s) max(600, 10 x (-vc)) Number of initial Rx packets (-rxpck) 200
To override any of the values, provide the indicated argument.
The amount of memory required for the File Server varies based on configuration. If additional memory is available, increasing the value of the -b, -l, -s, and -vc arguments beyond their configured values can improve File Server performance.
By default, the File Server allows a volume to exceed its quota by 1 MB when an application is writing data to an existing file in a volume that is full. The File Server still does not allow users to create new files in a full volume. To change the default, use one of the following arguments:
Set the -spare argument to the number of extra kilobytes that the File Server allows as overage. A value of 0 allows no overage.
Set the -pctspare argument to the percentage of the volume's quota the File Server allows as overage.
By default, the File Server implicitly grants the a (administer) and l (lookup) permissions to system:administrators on the access control list (ACL) of every directory in the volumes stored on its file server machine. In other words, the members of the system:admininstrators group can exercise those two permissions regardless of which rights are granted to the caller by an ACL. The implicit rights are granted even when a matching negative access control entry revokes the a (administer) and l (lookup) permissions. To change the implicit permissions, use the -implicit argument. The a (administer) permission is always granted to members of the system:administrators group as an implicit permission and cannot be revoked.
The File Server maintains a host current protection set (host CPS) for each client machine from which it has received a data access request. Like the CPS for a user, a host CPS lists all of the Protection Database groups to which the machine belongs, and the File Server compares the host CPS to a directory's ACL to determine in what manner users on the machine are authorized to access the directory's contents. When the pts adduser or pts removeuser command is used to change the groups to which a machine belongs, the File Server must recompute the machine's host CPS in order to notice the change. By default, the File Server contacts the Protection Server every two hours to recompute host CPSs, implying that it can take that long for changed group memberships to become effective. To change this frequency, use the -hr argument.
The File Server stores volumes in partitions. A partition is a filesystem or directory on the server machine that is named /vicepX or /vicepXX where XX is "a" through "z" or "aa" though "zz". Up to 255 partitions can be attached to a fileserver. The File Server expects that the /vicepXX directories are each on a dedicated filesystem. The File Server will only use a /vicepXX if it's a mountpoint for another filesystem, unless the file /vicepXX/AlwaysAttach exists. A partition will not be mounted if the file /vicepXX/NeverAttach exists. If both /vicepXX/AlwaysAttach and /vicepXX/NeverAttach are present, then /vicepXX/AlwaysAttach wins. The data in the partition is a special format object store that can only be accessed using AuriStorFS commands or an AuriStorFS or AFS client.
The File Server generates the following message when a partition is nearly full:
No space left on device
Do not specify both the -spare and -pctspare arguments. Doing so causes the File Server to exit, leaving an error message in the /var/log/yfs/FileLog file.
The maximum size of a volume quota is 2 zettabytes (2^74 bytes) and the maximum size of a /vicepX partition on a File Server is 2 zettabytes (2^74 bytes).
The maximum number of directory entries is 64,000 if all of the entries have names that are 15 octets or less in length. A name whose length is up to 15 octets long requires the use of only one block in the directory. Additional sequential blocks are required to store entries with names that are longer than 15 octets. Each additional block provides an additional length of 32 octets for the name of the entry. Note that if file names use an encoding like UTF-8, a single character may be encoded into multiple octets.
In real world use, the maximum number of objects in an AFS directory is usually between 16,000 and 25,000, depending on the average name length.
Sets the abort threshold, which is triggered when a Cache Manager sends a number of RPC requests in a row and all of them fail due to unexpected error. When the abort threshold is reached, the File Server begins to delay the responses to the problem client in order to reduce the load on the File Server. In most cases errors such as EPERM, EACCES, ENOENT, and EEXIST are expected and will not be counted towards the abort threshold count.
Setting the threshold to 0 disables the throttling behavior.
Ensures that no data or metadata will be accepted that prevents interoperability with AFS3-protocol clients such as IBM AFS 3.6 and OpenAFS. It does not disable the AuriStorFS protocol extensions or use of the yfs-rxgk security class.
For the File Server, -afscompat prohibits explicit references to per-file ACLs and cross-directory hard links, use of dates outside the range from 1 January 1970 to 19 January 2038, growth of a volume beyond 2TB, and restrictions on the vnode id numeric space.
By default, the RXKAD security layer will disallow access by Kerberos v5 principals with a dot in the first component of their name. This is to avoid the confusion where principals user/admin and user.admin are both mapped to the Kerberos v4 user.admin PTS entry. Sites whose Kerberos realms don't have these collisions between principal names may disable this check by starting the server with this option.
The default is no.
Turns on audit logging, and optionally sets the path for the audit log. The audit log records information about RPC calls, including the name of the RPC call, the host that submitted the call, the authenticated entity (user) that issued the call, the parameters for the call, and if the call succeeded or failed. If yes is specified, /var/log/yfs/FileAuditLog is used in conjunction with -audit-interface file.
Specifies the audit interface to use. The file interface writes audit messages to the file specified by -auditlog. The sysvmq interface writes audit messages to a SYSV message queue (see msgget(2) and msgrcv(2)). The message queue the sysvmq interface writes to has the key ftok(path, 1), where path is the path specified in the -auditlog option. The syslog interface writes audit messages to the syslog(1) facility. The path provided is used as the syslog ident.
Defaults to file.
Sets the number of directory buffers. Provide a positive integer.
Prints the following banner to /dev/console about every 10 minutes.
File Server is running at I<time>.
The default is no.
Defines the number of incoming RPCs that can be waiting for a response from the File Server before the File Server returns the error code VBUSY to the Cache Manager that sent the latest RPC. In response, the Cache Manager retransmits the RPC after a delay. This argument prevents the accumulation of so many waiting RPCs that the File Server can never process them all. Provide a positive integer. The default is 50% greater than the -rxpck value.
Sets an alternate location for the configuration file that will be parsed for configuration options for this service. If this option is not specified, /etc/yfs/server/yfs-server.conf will be used.
To address CVE-2018-7168 fileserver opcodes RXYFS_StoreOpaqueACL and RXYFS_StoreACL are disabled by default. These opcodes are required to permit AuriStorFS clients older than v0.168 to store ACLs.
This option re-enables support for RXYFS_StoreOpaqueACL and RXYFS_StoreACL. It should be used with caution as it permits clients to store improperly constructed ACLs. AuriStorFS clients v0.116 through v0.167 are known to ignore the fs setacl -negative flag. When -negative is intended, the specified rights are granted instead of denied.
The default is no.
Sets the detail level for the debugging trace written to the /var/log/yfs/FileLog file. Provide one of the following values, each of which produces an increasingly detailed trace: 0, 1, 5, 25, and 125. The default value of 0 produces messages during startup and shutdown, and error messages when processing Rx calls.
Activates the collection of Rx statistics and allocates memory for their storage. For each connection with a specific UDP port on another machine, a separate record is kept for each type of RPC (FetchFile, GetStatus, and so on) sent or received. To display or otherwise access the records, use the Rx Monitoring API.
The default is no.
Activates the collection of Rx statistics and allocates memory for their storage. A separate record is kept for each type of RPC (FetchFile, GetStatus, and so on) sent or received, aggregated over all connections to other machines. To display or otherwise access the records, use the Rx Monitoring API.
The default is no.
Prints the online help for this command. All other valid options are ignored.
Specifies how often the File Server refreshes its knowledge of the machines that belong to protection groups (refreshes the host CPSs for machines). The File Server must update this information to enable users from machines recently added to protection groups to access data for which those machines now have the necessary ACL permissions.
When starting up the File Server, don't reload any state that was previously saved to disk. The default is to restore callback state when the File Server restarts.
Defines the set of permissions granted by default to the system:administrators group on the ACL of every directory in a volume stored on the file server machine. Provide one or more of the standard permission letters (rlidwka) and auxiliary permission letters (ABCDEFGH), or one of the shorthand notations for groups of permissions (all, none, read, and write). To review the meaning of the permissions, see the fs setacl reference page. The a (administer) permission is always granted to system:administrators regardless of this setting.
Sets the number of directory (aka large) vnodes available in memory for caching directory elements. Provide a positive integer.
Sets the file to use for server logging. If logfile is not specified and no other logging options are supplied, this will be /var/log/yfs/FileLog. Note that this option is intended for debugging and testing purposes. Changing the location of the log file will prevent bos getlog from being able to read its contents.
Specifies the percentage of each AFS server partition that the File Server creates as a reserve. Specify an integer value between 0 and 30. The default is 8 on AIX and 0 on all other platforms. A value of 0 means that the partition can become completely full, which can have serious negative consequences.
Specifies the maximum number of worker threads that can be started to service incoming remote procedure calls. The minimum acceptable value is 6. The default can be between 512 and 16384 depending upon the operating system and the configured RLIMIT_NOFILE value for the process. A minimum of 4 file descriptors is required by each worker thread. The actual number of threads that can be created is determined by the operating system.
The number of worker threads dynamically grows and shrinks to satisfy the number of simultaneously received requests. New threads are created on demand and threads retire after being idle for -threadidle seconds.
Prevents the File Server from breaking the callbacks that Cache Managers hold on a volume that the File Server is reattaching after the volume was offline (as a result of the vos restore command, for example). Use of this flag is strongly discouraged as Cache Managers will be unaware of metadata and data changes within the volume.
The default is no.
This option behaves similarly to -offline-timeout but applies to volumes that are going offline as part of the File Server shutdown process. If the value specified is N, the File Server will interrupt any outstanding Rx calls after N seconds have passed. The Rx call will terminate with an RX_RESTARTING error.
Setting this option to 0 means to interrupt all Rx calls immediately during the shutdown process. Setting this option to -1 means to wait forever for Rx calls to finish.
If -offline-timeout is specified, the default value of -offline-shutdown-timeout is the value specified for -offline-timeout. Otherwise, the default value is -1.
Setting this option to N means that if any clients are reading from a volume when the File Server wants to offline that volume (for example, as part of releasing a volume), the File Server will wait N seconds for any outstanding Rx calls to complete. If the Rx calls do not finish after N seconds have passed, the File Server will interrupt the Rx calls and send an error to the callers, allowing the volume to go offline.
If an Rx call is interrupted, from the Cache Managers's point of view, it will appear as if it had accessed the volume after it had gone offline. For RO volumes, this means the Cache Manager should fail-over to another valid RO site for that volume. This option may speed up volume releases if volumes are being accessed by Cache Managers that have slow or unreliable network connections.
Setting this option to 0 means to interrupt clients immediately if a volume is waiting to go offline. Setting this option to -1 means to wait forever for Rx calls to finish. The default value is -1.
Specifies the amount by which the File Server allows a volume to exceed its quota, as a percentage of the quota. Provide an integer between 0 and 99. A value of 0 prevents the volume from ever exceeding its quota. Do not combine this argument with the -spare argument.
Do not permit data or metadata changes to this fileserver. All readwrite volumes are treated as readonly.
The default is no.
Specifies a list of one or more Kerberos realm names used by the File Server for local authentication. When provided, this argument overrides any realm names listed in the [kerberos] realm setting.
Bind the Rx socket if only a single interface remains after applying the netinfo and netrestrict configuration from yfs-server.conf(5)
The default is no.
Writes a trace of the File Server's operations on Rx packets to the file /var/log/yfs/FileRxLog.
The default is no.
Specify the maximum transmission unit (MTU) value. The value must be between the minimum (520) and maximum (16384) Rx packet sizes.
It artificially limits the maximum Rx data packet size that will be transmitted. It can be used when the maximum size that can be successfully transmitted is smaller than the reported network interface MTU.
The initial number of Rx packets the File Server allocates at startup. Provide a positive integer.
Specify the maximum sliding window size that RX may use on the wire. Larger windows improve performance on networks with a high latency, at the expense of higher memory usage. The value specified must be between 16 and 8192. The default is 8192.
Sets the number of file (aka small) vnodes available in memory for caching file objects. Provide a positive integer.
Specifies the acceptable security class and level combinations as a list of colon-separated pairs.
The following classes are recognized: rxnull, rxkad, yfs-rxgk
The following levels are recognized: clear, auth, crypt
For example, to enforce that all connections use the yfs-rxgk class with a minimum level of auth: -security yfs-rxgk:auth yfs-rxgk:crypt
Sets the size of the buffer used to transfer data (in either direction) between Rx calls and the vice partition backing store. The buffer cannot be smaller than 16384 bytes. The default is 16384 bytes.
Specifies the number of additional kilobytes an application can store in a volume after the quota is exceeded. Provide a positive integer; a value of 0 prevents the volume from ever exceeding its quota. Do not combine this argument with the -pctspare argument.
This option changes how hard the fileserver tries to ensure that data written to volumes actually reaches the physical disk or network attached device.
Normally, when the File Server writes to disk, the underlying filesystem or Operating System may delay writes from actually going to disk, and reorder which writes hit the disk first. So, during an unclean shutdown of the machine (if the power goes out, or the machine crashes, etc), file data may become lost that the server previously told Cache Managers was successfully written.
To try to mitigate this, the File Server will try to "sync" file data to the physical disk at numerous points during I/O processing. However, this can result in significantly reduced performance. Depending on the usage patterns, this may or may not be acceptable. This option dictates specifically what the File Server does when it wants to perform a "sync".
There are several options; pass one of these as the argument to -sync. The default is onclose.
This causes a sync operation to always sync immediately and synchronously. This is the slowest option that provides the greatest protection against data loss in the event of a crash.
This causes a sync to do nothing immediately, but causes the relevant file to be flagged as potentially needing a sync. When a volume is detached, volume metadata files flaged for synced are synced, as well as data files that have been accessed recently. Events that cause a volume to detach include: performing volume operations (dump, restore, clone, etc), a clean shutdown of the File Server, or during "soft detachment".
Effectively this option is the same as never while a volume is attached and actively being used, but if a volume is detached, there is an additional guarantee for the data's consistency.
This causes all syncs to never do anything. This is the fastest option, with the weakest guarantees for data consistency.
Depending on the underlying filesystem and Operating System, there may be guarantees that any data written to disk will hit the physical media after a certain amount of time. For example, Linux's pdflush process usually makes this guarantee, and ext3 can make certain various consistency guarantees according to the options given. ZFS on Solaris can also provide similar guarantees, as can various other platforms and filesystems. Consult the documentation for your platform if you are unsure.
Which option you choose is not an easy decision to make. Various developers and experts sometimes disagree on which option is the most reasonable, and it may depend on the specific scenario and workload involved. Some argue that the always option does not provide significantly greater guarantees over any other option, whereas others argue that choosing anything besides the always option allows for an unacceptable risk of data loss. This may depend on your usage patterns, your platform and filesystem, and who you talk to about this topic.
Regardless of the -sync setting selected, a volume that has been idle for 10 minutes will have all modificatons flushed to disk and the on-disk Volume header will be updated to indicate that the volume is idle and does not require a salvage if the File Server was to shutdown unexpectedly.
Use syslog instead of the normal logging location for the fileserver process. If provided, log messages are at <loglevel> instead of the default LOG_USER.
This option causes the File Server to bypass the normal safety check when attaching volumes that checks the inUse field in the volume header. With this option, volumes that were in use at the time of an unclean shutdown will not be salvaged immediately the next time they are accessed, and thus risk (possibly silent and/or irrevocable) corruption. Volumes will still be salvaged when an internal inconsistency is detected or other cases where a salvage would normally occur.
Due to the increased risk of data corruption, the use of this option is strongly discouraged and should only be used after consulting with AuriStor support.
Specifies the maximum number of seconds that a worker thread can remain idle before it will be terminated. The default is 20 minutes (1200 seconds). This value should be increased if the fileserver workload is known to experience a spike in the number of serviced remote procedure calls on a periodic cycle longer than the default.
Sets the size of the UDP buffer, which is 64 KB by default. Provide a positive integer, preferably larger than the default.
The number of threads assigned to attach and detach volumes. The default is 1. Setting this value to a multiple of the number of vice partitions can reduce File Server startup and shutdown times.
Sets an initial lower-bound for the number of volume headers the File Server can cache in memory. This value cannot be smaller than the maximum number of threads, -maxthreads, the File Server is permitted to create. Provide a positive integer.
Number of file handles set aside for I/O in the cache. Defaults to 128.
Maximum number of available file handles.
Number of file handles set aside for I/O not in the cache. Defaults to 128.
The log(2) number of initial volume hash buckets. The default ranges from 8 to 24 depending upon the effective size of the volume header cache. The minimum that can be specified is 6 (64 hash buckets). The maximum that can be specified is 24 (16,777,216 buckets). Regardless of the initial value, the volume hash table will grow dynamically as the number of volumes pre-attached by the File Server increases.
This option completely disables the VLRU mechanism, which means volumes will only "soft detach" as a result of the volume header cache -vc filling. By default the VLRU mechanism is disabled. Set to no to enable the VLRU mechanism.
The default is yes.
The number of minutes of inactivity before a volume is eligible for soft detachment. Default is 120 minutes.
The number of seconds between VLRU candidate queue scan. The default is 120 seconds.
The maximum number of volumes which can be soft detached in a single pass of the scanner. Default is 8 volumes.
The following bos create command creates a dafs process on the File Server machine fs2.your-cell-name.com. Type the command on a single line:
% bos create -server fs2.your-cell-name.com -instance dafs -type dafs \
-cmd "/usr/local/libexec/yfs/fileserver \
/usr/local/libexec/yfs/volserver \
/usr/local/libexec/yfs/salvageserver \
/usr/local/libexec/yfs/salvager
Sending process signals to the File Server Process can change its behavior in the following ways:
Process Signal OS Result
---------------------------------------------------------------------
File Server XCPU Unix Prints a list of client IP
Addresses.
File Server USR2 Windows Prints a list of client IP
Addresses.
File Server POLL HPUX Prints a list of client IP
Addresses.
Any server TSTP Any Increases Debug level by a power
of 5 -- 1,5,25,125, etc.
This has the same effect as the
-debug XXX command-line option.
Any Server WINCH Any Resets Debug level to 0
Any Server HUP Any Force log file to be reopened.
File Server TERM Any Run minor instrumentation over
the list of descriptors.
Other Servers TERM Any Causes the process to quit.
File Server QUIT Any Causes the File Server to Quit.
Bos Server knows this.
The basic metric of whether an AFS file server is doing well is the number of connections waiting for a thread, which can be found by running the following command:
% rxdebug <server> | grep "calls waiting for"
Each line returned by rxdebug that contains the text "waiting_for" represents a connection that's waiting for a file server thread.
If the blocked connection count is ever above 0, the server is having problems replying to clients in a timely fashion. If it gets above 10, roughly, there will be noticeable slowness by the user. The total number of connections is a mostly irrelevant number that goes essentially monotonically for as long as the server has been running and then goes back down to zero when it's restarted.
The most common cause of blocked connections rising on a server is some process somewhere performing an abnormal number of accesses to that server and its volumes. If multiple servers have a blocked connection count, the most likely explanation is that there is a volume replicated between those servers that is absorbing an abnormally high access rate.
To get an access count on all the volumes on a server, run:
% vos listvol <server> -long
and save the output in a file. The results will look like a bunch of vos examine output for each volume on the server. Look for lines like:
40065 accesses in the past day (i.e., vnode references)
and look for volumes with an abnormally high number of accesses. Anything over 10,000 is fairly high, but some volumes like root.cell and other volumes close to the root of the cell will have that many hits routinely. Anything over 100,000 is generally abnormally high. The count resets about once a day.
Another approach that can be used to narrow the possibilities for a replicated volume, when multiple servers are having trouble, is to find all replicated volumes for that server. Run:
% vos listvldb -server <server>
where <server> is one of the servers having problems to refresh the VLDB cache, and then run:
% vos listvldb -server <server> -part <partition>
to get a list of all volumes on that server and partition, including every other server with replicas.
Once the volume causing the problem has been identified, the best way to deal with the problem is to move that volume to another server with a low load or to stop any runaway programs that are accessing that volume unnecessarily. Often the volume will be enough information to tell what's going on.
If you still need additional information about who's hitting that server, sometimes you can guess at that information from the failed callbacks in the FileLog log in /var/log/afs on the server, or from the output of:
% /Library/Auristor/Tools/sbin/rxdebug <server> -rxstats
but the best way is to turn on debugging output from the file server. (Warning: This generates a lot of output into FileLog on the AFS server.) To do this, log on to the AFS server, find the PID of the fileserver process, and do:
kill -TSTP <pid>
where <pid> is the PID of the file server process. This will raise the debugging level so that you'll start seeing what people are actually doing on the server. You can do this up to three more times to get even more output if needed. To reset the debugging level back to normal, use (The following command will NOT terminate the file server):
kill -WINCH <pid>
The debugging setting on the File Server should be reset back to normal when debugging is no longer needed. Otherwise, the AFS server may well fill its disks with debugging output.
The FileLog can be restarted by renaming the existing file and then executing the command:
kill -HUP <pid>
This will force the fileserver process to close and reopen the FileLog.
The lines of the debugging output that are most useful for debugging load problems are:
SAFS_FetchStatus, Fid = 2003828163.77154.82248, Host 171.64.15.76
SRXAFS_FetchData, Fid = 2003828163.77154.82248
(The example above is partly truncated to highlight the interesting information). The Fid identifies the volume and inode within the volume; the volume is the first long number. So, for example, this was:
% vos examine 2003828163
pubsw.matlab61 2003828163 RW 1040060 K On-line
afssvr5.Stanford.EDU /vicepa
RWrite 2003828163 ROnly 2003828164 Backup 2003828165
MaxQuota 3000000 K
Creation Mon Aug 6 16:40:55 2001
Last Update Tue Jul 30 19:00:25 2002
86181 accesses in the past day (i.e., vnode references)
RWrite: 2003828163 ROnly: 2003828164 Backup: 2003828165
number of sites -> 3
server afssvr5.Stanford.EDU partition /vicepa RW Site
server afssvr11.Stanford.EDU partition /vicepd RO Site
server afssvr5.Stanford.EDU partition /vicepa RO Site
and from the Host information one can tell what system is accessing that volume.
Note that the output of vos_examine(1) also includes the access count, so once the problem has been identified, vos examine can be used to see if the access count is still increasing. Also remember that you can run vos examine on the read-only replica (e.g., pubsw.matlab61.readonly) to see the access counts on the read-only replica on all of the servers that it's located on.
The issuer must be logged in as the superuser root on a file server machine to issue the command at a command shell prompt. It is conventional instead to create and start the process by issuing the bos create command.
BosConfig(5), FileLog(5), bos_create(8), bos_getlog(8), fs_setacl(1), msgget(2), msgrcv(2), salvager(8), volserver(8), vos_examine(1) fileserver(8), yfs-server.conf(5)
IBM Corporation 2000. http://www.ibm.com/ All Rights Reserved.
This documentation is covered by the IBM Public License Version 1.0. It was converted from HTML to POD by software written by Chas Williams and Russ Allbery, based on work by Alf Wachsmann and Elizabeth Cassell.
"AFS" is a registered mark of International Business Machines Corporation, used under license. (USPTO Registration 1598389)
"OpenAFS" is a registered mark of International Business Machines Corporation. (USPTO Registration 4577045)
The "AuriStor" name, log 'S' brand mark, and icon are registered marks of AuriStor, Inc. (USPTO Registrations 4849419, 4849421, and 4928460) (EUIPO Registration 015539653).
"Your File System" is a registered mark of AuriStor, Inc. (USPTO Registrations 4801402 and 4849418).
"YFS" and "AuriStor File System" are trademarks of AuriStor, Inc.