WASD Hypertext Services - Technical Overview

18 - WATCH Facility

18.1 - Server Instances
18.2 - Event Categories
18.3 - Request Filtering
18.4 - Report Format
18.5 - Usage Suggestions
18.6 - Command-Line Use
[next] [previous] [contents] [full-page]

The WATCH facility is a powerful adjunct in server administration. From the Server Administration facility (17 - Server Administration) it provides an online, real-time, in-browser-window view of request processing in the running server. The ability to observe live request processing on an ad hoc basis, without changing server configuration or shutting-down/restarting the server process, makes this facility a great configuration and problem resolution tool. It allows (amongst other uses)

assessment of mapping rules
assessment of authorization rules
investigation of request processing problems
observation of script interaction
general observation of server behaviour

A single client per server process can access the WATCH facility at any one time. It can be used in one of two modes.

Options immediately below the duration selector allows the WATCH output to concurrently be included in the server process log. This allows a permanent record (at least as permanent as server logs) to be simply produced.


18.1 - Server Instances

With a single instance (6.1 - Server Instances) access to WATCH is always through the one server process. If multiple instances are configured WATCH requests, in common with all others, will be serviced by any one of the associated processes depending on the indeterminate current state of the round-robin distribution.

This is often an issue for request WATCHing. The simplest scenario involves two instances. When the WATCH report is activated it will be serviced by the first process, when the request wishing to be WATCHed is accessed it (in the absence of any other server activity) will be serviced by the other process and will not be reported by WATCH on the first.

There is no simple solution for this issue as the round-robin distribution of requests is determined by the network device driver. When debugging server or script behaviour often the simplest solution is to temporarily reset to one the number of instances running on the system. See 17.3 - Server Instances, 17.6 - HTTPd Server Action and 5.5.2.6 - Instances.


18.2 - Event Categories

An event is considered any significant point for which the server code has a reporting call provided. These have been selected to provide maximum information with minimum clutter and impact on server performance. Obvious examples are connection acceptance and closure, request path resolution, error report generation, network reads and writes, etc. Events are collected together into groupings to allow clearly defined areas of interest to be selected for reporting.

[graphic]  WATCH Selection Graphic

The report menu provides for the inclusion of any combination of the following categories.


Request


Response


General


Network


Other


Proxy


Code Modules

If the server has been compiled using the WATCH_MOD=1 macro a set of module WATCHing statements is included. These provide far more detailed processing information than available with the generic WATCH, are intended primarily for debugging the server during development and testing. This is considered a specialized tool, with the quantity and level of detail produced most likely proving counter-productive in addressing general site configuration issues. The module WATCH facility is not obvious but is available by selecting the "H" character of the "WATCH" item on the Server Administration page. If this functionality has not been built into the server image the message "Module WATCHing is not a compiled option!" is displayed.


18.3 - Request Filtering

By default all requests to all services are WATCHed. Fine control may be exercised over exactly which requests are reported, allowing only a selected portion of all requests being processed to be concentrated on, even on a live and busy server. This is done by filtering requests according the following criteria.

These filters are controlled using fully-specified or wildcarded strings. Requests that do not match the filter are not reported. In the case of originating client and destination service, requests are eliminated before ever appearing in the report. Path and track filtering is slightly different, requiring some request processing before the target can be determined. Depending on the categories selected this may result in some events begin displayed. It will be eliminated, with an accompanying explanatory message, as soon the path or track identifier has been determined.

The following examples are grouped in the same order as the categories listed above; client, service and path.

  alpha.wasd.dsto.defence.gov.au
  *.wasd.dsto.gov.au  
  131.185.250.202
  131.185.250.*
 
  beta.wasd.dsto.defence.gov.au:8000
  beta.wasd.dsto.defence.gov.au:*
  http://*
  https:*
  *:80
 
  /ht_root/src/*
  /cgi-bin/*
  /web/*/cyrillic/*
  $ORoKJAOef8sAAAkuACc
  http://proxied.host.name/*


18.4 - Report Format

The following example illustrates the format of the WATCH report. It begins with multi-line heading. The first two record the date, time and official server name, with underline. The third provides the WASD server version. The fourth provides some TCP/IP agent information. Lines following can show OpenSSL version (if deployed), system infomation, server startup command-line, and then current server process quotas. The last three lines of the header provide a list of the categories being recorded, the filters in use, and the last, column headings described as follows:

time the event was recorded
the module name of the originating source code
the line in the code module
a unique item number for each thread being WATCHed
event category name
free-form, but generally interpretable event data

[graphic]  WATCH Report Graphic

Note that some items also include a block of data. The request header category does this, providing the blank-line terminated text comprising the HTTP header. Rule mapping also provides a block of information representing each rule as it is interpreted. Generally WATCH-generated information can be distinguished from other data by it's uniform format and delimiting vertical bars. Initiative and imagination is sometimes required to interpret the free-form data but a basic understanding of HTTP serving and a little consideration is generally all that is required to deduce the essentials of any report.

  31-MAY-2000 18:20:25  WATCH REPORT  delta.wasd.dsto.defence.gov.au:80
  ---------------------------------------------------------------------
  HTTPd-WASD/7.0.0 OpenVMS/AXP SSL (30-MAY-2000 22:56:54.37 VMS 7.2-1 DECC 60290003)
  Compaq-TCP/IP TCPIP$IPC_SHR V5.0A-1 (20-MAY-1999 22:31:52.08)
  OpenSSL 0.9.5 28 Feb 2000 (5-MAR-2000 14:58:56.83)
  AlphaServer DS20 500 MHz VMS V7.2-1 (ODS-5 enabled, VMS NAML, VMS FIB)
  $ HTTPD /PRIORITY=4 /SYSUAF=(ID)
  AST:1995/2000 BIO:509/512 BYT:479200/499296 DIO:512/512 ENQ:504/512 FIL:295/300 PGFL:192992/200000 PRC:0/100 TQ:98/100
  BYTLM-available:495008 BYTLM-per-subproc:14592 (approx 26 subprocesses) BYTLM-net-accept:1024 BYTLM-net-listen:1024
  Watching: connect, request, response (19)
  Client: "*" Service: "*" Path: "*"
  |Time_______|Module__|Line|Item|Category|Event...|
  |18:20:46.91 NET      1626 0001 CONNECT  ACCEPT 131.185.250.108,47111 on https://131.185.250.203:443|
  |18:20:47.00 REQUEST  1211 0001 REQUEST  HEADER 285 bytes|
  GET /ht_root/ HTTP/1.0
  If-Modified-Since: Thursday, 07-Oct-99 03:51:23 GMT; length=4644
  Connection: Keep-Alive
  User-Agent: Mozilla/3.0Gold (X11; I; OSF1 V4.0 alpha)
  Pragma: no-cache
  Host: delta.wasd.dsto.defence.gov.au
  Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
 
  |18:20:47.00 NET      1148 0001 CONNECT  VIRTUAL delta.wasd.dsto.defence.gov.au:443|
  |18:20:47.00 REQUEST  2270 0001 REQUEST  GET /ht_root/|
  |18:20:47.00 MAPURL   0312 0001 MAPPING  PATH /ht_root/|
  /ht_root/  ..  REDIRECT  /*.*.htmlx  /*.htmlx?httpd=ssi&__part=*
  /ht_root/  ..  PASS      /httpd/-/change/*  /httpd/-/change/*
  /ht_root/  ..  MAP       /htroot/*  /ht_root/*
  /ht_root/  ..  SET       /web/*   stmLF
  /ht_root/  ..  SET       /ht_root/*   stmLF
  /ht_root/  ..  MAP       /httpd-internal-icons/*  /httpd/-/*
  /ht_root/  ..  PASS      /ht_root/runtime/*  /ht_root/runtime/*
  /ht_root/  ..  PASS      /*/-/*  /ht_root/runtime/*/*
  /ht_root/  Y-  PASS      /ht_root/*  
  |18:20:47.00 MAPURL   0335 0001 MAPPING  RESULT /ht_root/ /ht_root/ HT_ROOT:[000000] - -|
  |18:20:47.00 CACHE    0604 0001 RESPONSE CACHE DSA811:[HT_ROOT.][000000]HOME.HTML|
  |18:20:47.00 NET      2427 0001 RESPONSE HEADER 231 bytes|
  HTTP/1.0 200 Success
  Server: HTTPd-WASD/7.0.0 OpenVMS/AXP Digital-TCPIP SSL
  Date: Wed, 31 May 2000 18:20:47 GMT
  Last-Modified: Tue, 30 May 2000 03:51:23 GMT
  Content-Type: text/html; charset=ISO-8859-1
  Content-Length: 4644
 
  |18:20:47.00 REQUEST  0407 0001 REQUEST  STATUS 200 rx:457 tx:5084 bytes 0.0000 seconds|
  |18:20:47.00 REQUEST  0252 0001 CONNECT  KEEP-ALIVE 131.185.250.108,47111|
  |18:20:52.06 NET      1502 0001 CONNECT  CLOSE 131.185.250.108,47111|


18.5 - Usage Suggestions

The following provides a brief explanation on the way WATCH operates and any usage implications.

A single client may be connected to the WATCH facility at any given time. When connecting the client is sent an HTTP response header and the WATCH report heading lines. The request then remains connected until the WATCH duration expires or the client overtly aborts the connection. During this period the browser behaves as if receiving a sometimes very slow, sometimes stalled, plain-text document. As the server processes WATCHable events the text generated is sent to the WATCH-connected client.

If the connection is aborted by the user some browsers will consider document retrieval to be incomplete and attempt to reconnect to the service if an attempt is made to print or save the resulting document. As the printing of WATCH information is often quite valuable during problem resolution this behaviour can result in loss of information and generally be quite annoying. Appropriate use of the duration selector when requesting a report can work around this, as at expiry the server disconnects, browsers generally interpreting this as legitimate end-of-document (when no content-length has been specified).

During report processing some browsers may not immediately update the on-screen information to reflect received data without some application activity. If scroll-bars are present on the document window manipulating either the horizonal or vertical slider will often accomplish this. Failing that minimizing then restoring the application will usually result in the most recent information being visible.

Browser reload/refresh may be used to restart the report. A browser will quite commonly attempt to remain at the current position in the document, which with a WATCH report's sustained but largely indeterminate data stream may take some time to reach. It is suggested the user ensure that any vertical scroll-bar is at the beginning of the current report, then refresh the report.

Selecting a large number of categories, those that generate copious output for a single event (e.g. response body) or collecting for extended periods can all result in the receipt of massive reports. Some browsers do not cope well with documents megabytes in size.

NOTE

WATCH reports are written using blocking I/O. This means when large bursts of data are being generated (e.g. when WATCHing network data, response bodies, etc.) significant granularity may be introduced to server processing. Also if the WATCH client fails or blocks completely server processing could halt completely! (This has been seen when WATCHing through a firewall.)

When supplying WATCH output as part of a problem report please ZIP the file and include it an an e-mail attachment. Mailers often mangle the report format making it difficult to interpret.


18.6 - Command-Line Use

Although intended primarily as a tool for online use WATCH can be deployed at server startup with a command-line qualifier and provide report output to the server process log. This is slightly more cumbersome than the Web interface but may still be useful in some circumstances. Full control over event categories and filters is possible.

The following examples illustrate the command-line WATCH specification.

  /NOWATCH
  /WATCH=NOSTARTUP,ITEMS=(REQUEST,RESPONSE,MAPPING)
  /WATCH="ITEMS=(REQUEST,RESPONSE,ERROR),*,*,/cgi-bin/*"
  /WATCH=LIST


[next] [previous] [contents] [full-page]