/*****************************************************************************/
/*
                                 GZIP.c

Provides ZLIB-enabled GZIP compression (gzip, deflate) for WASD.

Dynamically maps required functions from a ZLIB shareable image.  If the image
is found and all required functions present the ZLIB compression/decompression
of network streams in enabled for suitable requests/responses.  If not it is
just left disabled.

Based on ZLIB v1.2.1 port by Jean-François Piéronne (jf.pieronne@laposte.net).
Requires this package to be installed and started on the runtime system for
dynamic activation.  The sharable image must be INSTALLed (without any
particular privileges) before it can be activated by the privileged WASD HTTPd
image.  This module will compile on a system that does not have the LIBZ (ZLIB)
header files.  The WASD GZIP.H header contains all of the ZLIB structure and
macro defintions required.  Of course with WASD dynamically activating the LIBZ
sharable image no static linking is required.


RESPONSE CONTENT-ENCODING
-------------------------
The WASD_CONFIG_GLOBAL directive [GzipResponse] controls whether this feature
is enabled for the gzip content-encoding of suitable response bodies.  This
directive requires at least one parameter, the compression level in the range
1..9.  Smaller values provide faster but poorer compression ratios while larger
values better compression at the cost of more CPU cycles and latency.  This
corresponds to the GZIP utility's -1..-9 CLI switches.  Two optional parameters
could allow ZLIB's 'memLevel' and 'windowBits' to be adjusted by ZLIB
afficiendos (level[,memory,window]).  A small amount of experimentation by this
author indicates minor changes in memory usage and compression ratio by
fiddling with these.

Be aware that GZIP encoding is MEMORY INTENSIVE.  From 132kB to 265kB has been
observed per compressing request (WATCH provides this in a summary line). 
These values apply across a wide range of transfer sizes (from kilobytes to
tens of megabytes).  It also is very CPU INTENSIVE and adds response latency
(on the author's 333MHz system anyway), though that might be well be offset by
significant reductions in transfer time on the Internet or other slower,
non-intranet infrastructures.  Text content compression has been observed from
30% to 10% of the original file size (even down to 1% in the case of the
extremely redundant content of [EXAMPLE]64K.TXT).  VMS executables (for want of
another binary test case) at around 40%.  In other words, GZIP encoding may not
be suitable or efficient for every site or every request!

Once enabled WASD will GZIP the responses for all suitable contents provided
the client accepts the encoding and the response is not one of the following
(see GzipShouldDeflate()):

  o  less than 1400 bytes (no point in the overhead)
  o  already content-encoded script or proxy response
  o  script requested that the response not be GZIPed
  o  a compressed image (e.g. GIF, JPEG, etc)
  o  a video stream (presumably already compressed, e.g. MPEG)
  o  a compressed audio stream
  o  an obviously compressed application stream (e.g. GZIP, ZIP, JAR)
  o  a PDF file, which seem to experience plug-in problems when GZIPed
     (this can be overridden using a 'set **.pdf response=gzip=all')
  o  a Shockwave Flash file, which are chocka with compressed objects
  o  SSI source file ('text/x-shtml' content type)

Additional control may be exercised with the following path SETings:

  o  response=GZIP=all        matching paths will always have GZIP encoding
                              performed (the above constraints still apply)
  o  response=GZIP=none       matching paths will never have GZIP encoding
  o  response=GZIP=<integer>  responses with content-lengths greater than
                              the specified number of kilobytes will be GZIP
                              content-encoded (if the content-length cannot
                              be determined it will NOT not encoded and the
                              above constraints still apply)


REQUEST CONTENT-ENCODING
------------------------
Decoding of GZIP content-encoded request bodies is enabled using the
WASD_CONFIG_GLOBAL directive [GzipAccept].  Enabling this using a value 15 (or 1)
results in the server advertising it's acceptance of GZIPed requests using the
"Accept-Encoding: gzip, deflate" response header.  Requests containing bodies
GZIP compressed will have these decoded as they are read from the client and 
before further processing, such as the upload of files into server accessable
file-system space.  THIS DECODING IS OPTIONAL and not the default with DCL and
DECnet script processing.  That is, a request body will be passed to the script
still encoded unless specific mapping directs otherwise.  Decoding by the
server into the original data prior to transfering to the script can be enabled
for all or selected scripts using the following path settings.

  o  script=body=decode       script gets the decoded stream
  o  script=body=NOdecode     script gets the raw, encoded stream (default)

Note that scripts need to be specially aware of both GZIP encoded bodies and
those already decoded by the server.  In the first case the stream must be read
to the specified content-length and then decoded.  In the second case, a
content-length cannot be provided by the server (without unencoding the entire
stream ahead of time it cannot predict the final size).  Where the server is to
decode the request body before transfering it to the script it changes the
CGI variable CONTENT_LENGTH to a single question-mark ("?").  Scripts may use
this to detect the server's intention and then must ignore any
transfer-encoding and/or content-encoding header information and read the
request body until end-of-file is received.

GZIP decoding (decompression) is understandably much less memory and CPU
intensive.  Experimentation indicates it does not contribute significantly to
latency either.


TESTING GZIP
------------
Most modern browsers accept a GZIP deflated response so that's
straight-forward.  Either the document displays correctly or not.  Server
processing of GZIP deflated request bodies is a little more involved.  See the
decription in the BODY.C module.


VERSION HISTORY
---------------
01-SEP-2024  MGD  add [SRC.LIBZ] and WASD_LIBZ_SHR32
22-APR-2020  MGD  add WATCH reporting points for NETWORK
24-AUG-2013  MGD  GzipInit() ZLIB shareable image via logical names
                   WASD_LIBZ_SHR32, then GNV$LIBZSHR32, finally LIBZ_SHR32
18-AUG-2009  MGD  bugfix; compress PDF if path set all
25-APR-2007  MGD  GzipShouldDeflate() do not compress Shockwave Flash
                    increase minimum size before compression to 1400 bytes
28-FEB-2007  MGD  bugfix; GzipDeflateCache() allow for cached CGI header
26-OCT-2006  JPP  bugfix; GzipShouldDefault() uninitialized 'cptr' when no
                    content-type would cause WatchThis() "!AZ" to barf if
                    'cptr' was non-NULL but pointed into an invalid page
04-JUL-2006  MGD  use PercentOf32() for more accurate percentages
04-JUL-2005  MGD  bugfix; adjust CacheMemoryInUse/CachePermMemoryInUse
                    bugfix; GzipDeflateCache() ambit buffer size calculation
                    too small for small content lengths (just allow heaps!)
28-MAY-2005  MGD  bugfix; GzipDeflateCache() ambit buffer size calculation
                    (captr->ContentLength >> 9) now (.. >> 7) (jpp@esme.fr)
26-MAR-2005  MGD  GzipDeflateCache() to deflate cache entry content,
                  GzipShouldDeflate() modified so 'rqptr' is optional
04-FEB-2005  MGD  GzipShouldDeflate() disable PDF deflation by default
22-JAN-2005  MGD  bugfix; GzipDeflate() delay avail input until output non-zero
08-JAN-2005  MGD  timer-based flush mechanism
15-NOV-2004  MGD  bugfix; GzipDeflate() (with NET.C)
16-OCT-2004  MGD  initial
*/
/*****************************************************************************/

#ifdef WASD_VMS_V7
#undef _VMS__V6__SOURCE
#define _VMS__V6__SOURCE
#undef __VMS_VER
#define __VMS_VER 70000000
#undef __CRTL_VER
#define __CRTL_VER 70000000
#endif

#include <stdio.h>
#include <ctype.h>

#include "wasd.h"
#include "gzip.h"

#define WASD_MODULE "GZIP"

/******************/
/* global storage */
/******************/

BOOL  GzipAccept,
      GzipAvailable,
      GzipResponse;

int  GzipDeflateCompLevel,
     GzipDeflateMemLevel,
     GzipDeflateWindowBits,
     GzipFindImageStatus,
     GzipFlushSeconds,
     GzipFlushInitialSeconds,
     GzipInflateWindowBits;

char  *GzipZlibNamePtr,
      *GzipZlibVersionPtr;

char  *GzipZlibError [] = { "Unknown ZLIB error!",
                            "ZLIB errno error.",
                            "ZLIB stream error.",
                            "ZLIB data error.",
                            "ZLIB memory error.",
                            "ZLIB buffer error."
                            "ZLIB version error." };

int (*ZlibDeflateFn)(z_stream*, int);
int (*ZlibDeflateInit2Fn)(z_stream*, int, int, int, int, int, char*, int);
int (*ZlibDeflateEndFn)(z_stream*);
int (*ZlibInflateFn)(z_stream*, int);
int (*ZlibInflateInit2Fn)(z_stream*, int, char*, int);
int (*ZlibInflateEndFn)(z_stream*);
char* (*ZlibVersionFn)(void);

/********************/
/* external storage */
/********************/

extern int  CacheChunkInBytes,
            HttpdTickSecond;

extern unsigned long  CacheGzipDeflateCount,
                      CacheMemoryInUse,
                      CachePermMemoryInUse;

extern unsigned long  SysPrvMask[];

extern uint64  CacheGzipDeflateBytesIn64,
               CacheGzipDeflateBytesOut64;

extern char  ErrorSanityCheck[];

extern ACCOUNTING_STRUCT  *AccountingPtr;
extern CONFIG_STRUCT  Config;
extern WATCH_STRUCT  Watch;

/*****************************************************************************/
/*
Attempt to get the required functions from the ZLIB shareable image.  Any
errors, including the obvious one of image not found, just disables any gzip
encoding.
*/

GzipInit ()

{
#define LIB$M_FIS_MIXEDCASE 0x10

   static $DESCRIPTOR (LibzImageDsc, "LIBZ_SHR32");
   static $DESCRIPTOR (GnvLibzImageDsc, "GNV$LIBZSHR32");
   static $DESCRIPTOR (WasdLibzImageDsc, "WASD_LIBZ_SHR32");
   static $DESCRIPTOR (DeflateDsc, "deflate");
   static $DESCRIPTOR (DeflateInit2Dsc, "deflateInit2_");
   static $DESCRIPTOR (DeflateEndDsc, "deflateEnd");
   static $DESCRIPTOR (InflateDsc, "inflate");
   static $DESCRIPTOR (InflateInit2Dsc, "inflateInit2_");
   static $DESCRIPTOR (InflateEndDsc, "inflateEnd");
   static $DESCRIPTOR (ZlibVersionDsc, "zlibVersion");

   int  status;
   struct dsc$descriptor_s  *dscptr;

   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_NET))
      WatchThis (WATCHALL, WATCH_MOD_NET, "GzipInit()");

   /* if it's not configured then don't initialize */
   if (!(Config.cfMisc.GzipAcceptWindowBits ||
         Config.cfMisc.GzipResponseCompLevel)) return;

   lib$establish (GzipInitHandler);

   /* need SYSPRV to access the (default) protected WASD_EXE: location */
   sys$setprv (1, &SysPrvMask, 0, 0);

   status = lib$find_image_symbol (dscptr = &WasdLibzImageDsc, &ZlibVersionDsc,
                                   &ZlibVersionFn, 0, 0);

   sys$setprv (0, &SysPrvMask, 0, 0);

   if (status == RMS$_FNF)
      status = lib$find_image_symbol (dscptr = &GnvLibzImageDsc, &ZlibVersionDsc,
                                      &ZlibVersionFn, 0, 0);
   if (status == RMS$_FNF)
      status = lib$find_image_symbol (dscptr = &LibzImageDsc, &ZlibVersionDsc,
                                      &ZlibVersionFn, 0, 0);
   if (status == RMS$_FNF)
   {
      GzipFindImageStatus = status;
      GzipZlibNamePtr = dscptr->dsc$a_pointer;
      FaoToStdout ("%HTTPD-W-GZIP, shareable image not found\n");
      return;
   }

   if (VMSok (status))
      status = lib$find_image_symbol (dscptr, &DeflateDsc,
                                      &ZlibDeflateFn, 0, 0);
   if (VMSok (status))
      status = lib$find_image_symbol (dscptr, &DeflateInit2Dsc,
                                      &ZlibDeflateInit2Fn, 0, 0);
   if (VMSok (status))
      status = lib$find_image_symbol (dscptr, &DeflateEndDsc,
                                      &ZlibDeflateEndFn, 0, 0);
   if (VMSok (status))
      status = lib$find_image_symbol (dscptr, &InflateDsc,
                                      &ZlibInflateFn, 0, 0);
   if (VMSok (status))
      status = lib$find_image_symbol (dscptr, &InflateInit2Dsc,
                                      &ZlibInflateInit2Fn, 0, 0);
   if (VMSok (status))
      status = lib$find_image_symbol (dscptr, &InflateEndDsc,
                                      &ZlibInflateEndFn, 0, 0);
   lib$revert ();

   GzipFindImageStatus = status;
   GzipZlibNamePtr = dscptr->dsc$a_pointer;

   if (VMSok (status))
   {
      GzipAvailable = true;
      if (Config.cfMisc.GzipAcceptWindowBits) GzipAccept = true;
      if (Config.cfMisc.GzipResponseCompLevel) GzipResponse = true;

      GzipInflateWindowBits = Config.cfMisc.GzipResponseWindowBits;
      if (GzipInflateWindowBits < 8)
         GzipInflateWindowBits = 15;  /* assume the default */
      else
      if (GzipInflateWindowBits > 15)
         GzipInflateWindowBits = 15;

      GzipDeflateCompLevel = Config.cfMisc.GzipResponseCompLevel;
      if (GzipDeflateCompLevel > 9) GzipDeflateCompLevel = 9;

      GzipDeflateMemLevel = Config.cfMisc.GzipResponseMemLevel;
      /* if not specified default to ZLIB default */
      if (!GzipDeflateMemLevel) GzipDeflateMemLevel = 8;
      if (GzipDeflateMemLevel > 9) GzipDeflateMemLevel = 9;

      GzipDeflateWindowBits = Config.cfMisc.GzipResponseWindowBits;
      /* if not specified default to ZLIB default */
      if (!GzipDeflateWindowBits) GzipDeflateWindowBits = 15;
      if (GzipDeflateWindowBits < 8)
         GzipDeflateWindowBits = 15;  /* assume the default */
      else
      if (GzipDeflateWindowBits > 15)
         GzipDeflateWindowBits = 15;

      GzipFlushSeconds = Config.cfMisc.GzipFlushSeconds;
      if (GzipFlushSeconds < 0) GzipFlushSeconds = 0;

      GzipFlushInitialSeconds = Config.cfMisc.GzipFlushInitialSeconds;
      if (GzipFlushInitialSeconds < 0) GzipFlushSeconds = 0;

      GzipZlibVersionPtr = ZlibVersionFn();
      FaoToStdout ("%HTTPD-I-GZIP, using !AZ V!AZ\n",
                   GzipZlibNamePtr, GzipZlibVersionPtr);
   }
   else
      FaoToStdout ("%HTTPD-W-GZIP, !AZ %X!8XL\r\n-!-!&M\n",
                   GzipZlibNamePtr, status);
}

/*****************************************************************************/
/*
Just continue, to report an error if the image couldn't be activated or the
required symbol not found.
*/

int GzipInitHandler ()

{
   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_NET))
      WatchThis (WATCHALL, WATCH_MOD_NET, "GzipInitHandler()");

   return (SS$_CONTINUE);
}

/*****************************************************************************/
/*
Return a pointer to the appropriate ZLIB message string.
*/

char* GzipZlibMsg
(
z_stream *zsptr,
int zstatus
)
{
   char  *cptr;

   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_NET))
      WatchThis (WATCHALL, WATCH_MOD_NET, "GzipZlibMsg() !UL", zstatus);

   if (!(cptr = zsptr->msg))
   {
      zstatus = -zstatus;
      if (zstatus < 0 || zstatus > 5) zstatus = 0;
      cptr = GzipZlibError[zstatus];
   }
   return (cptr);
}

/*****************************************************************************/
/*
A function to deflate a cache entry's content (see CACHE.C) to provide an
already GZIPed equivalent for serving, obviously ameliorating the expense of
doing it dynamically for each request.  Performs the full initialization,
deflate and end ZLIB compression life-cycle in the one call.
*/

BOOL GzipDeflateCache
(
REQUEST_STRUCT *rqptr,
FILE_CENTRY *captr
)
{
   int  zstatus,
        BufferCount,
        BufferSize,
        SizeInBytes;
   char  *BufferPtr;
   z_stream  zStream;
   z_stream  *zsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipDeflateCache()");

   zsptr = &zStream;

   zsptr->opaque = rqptr;
   zsptr->zalloc = &GzipAlloc;
   zsptr->zfree = &GzipFree;

   /* plus 16 is the magic number to get a gzip header and trailer!! */
   zstatus = ZlibDeflateInit2Fn (zsptr,
                                 GzipDeflateCompLevel,
                                 Z_DEFLATED,
                                 GzipDeflateWindowBits + 16,
                                 GzipDeflateMemLevel,
                                 Z_DEFAULT_STRATEGY,
                                 ZLIB_VERSION,
                                 sizeof(z_stream));

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "ZlibDeflateInit2Fn(!UL,!UL+16,!UL) !SL !&Z",
                 GzipDeflateCompLevel, GzipDeflateWindowBits,
                 GzipDeflateMemLevel, zstatus, zsptr->msg);

   if (zstatus != Z_OK)
   {
      ErrorNoticed (rqptr, SS$_BUGCHECK, GzipZlibMsg(zsptr,zstatus), FI_LI);
      return (false);
   }

   /* ambit buffer size will be length times two - heaps */
   BufferSize = captr->ContentLength + captr->ContentLength + 12;

   if (captr->CgiHeaderLength)
   {
      zsptr->next_in = captr->ContentPtr + captr->CgiHeaderLength + 1;
      zsptr->avail_in = captr->ContentLength - captr->CgiHeaderLength - 1;
   }
   else
   {
      zsptr->next_in = captr->ContentPtr;
      zsptr->avail_in = captr->ContentLength;
   }

   zsptr->next_out = BufferPtr = VmGetCache (BufferSize);
   zsptr->avail_out = BufferSize;

   zstatus = ZlibDeflateFn (zsptr, Z_FINISH);

   if (zstatus != Z_STREAM_END)
   {
      ZlibDeflateEndFn (zsptr);
      VmFreeCache (BufferPtr, FI_LI);
      ErrorNoticed (rqptr, SS$_BUGCHECK, GzipZlibMsg(zsptr,zstatus), FI_LI);
      return (false);
   }

   /* whatever's in the buffer */
   BufferCount = BufferSize - zsptr->avail_out;

   SizeInBytes = BufferCount;
   if (SizeInBytes % CacheChunkInBytes) SizeInBytes += CacheChunkInBytes;
   SizeInBytes = SizeInBytes / CacheChunkInBytes;
   SizeInBytes *= CacheChunkInBytes;
   if (!SizeInBytes) SizeInBytes = CacheChunkInBytes;

   if (rqptr->rqPathSet.CachePermanent)
   {
      captr->GzipContentPtr = VmGetPermCache (SizeInBytes);
      CachePermMemoryInUse += SizeInBytes;
   }
   else
   {
      captr->GzipContentPtr = VmGetCache (SizeInBytes);
      CacheMemoryInUse += SizeInBytes;
   }
   captr->GzipEntrySize = SizeInBytes;
   captr->GzipContentLength = BufferCount;
   memcpy (captr->GzipContentPtr, BufferPtr, BufferCount);

   CacheGzipDeflateCount++;
   CacheGzipDeflateBytesIn64 += captr->ContentLength;
   CacheGzipDeflateBytesOut64 += captr->GzipContentLength;

   zstatus = ZlibDeflateEndFn (zsptr);
   if (zstatus != Z_OK)
      ErrorNoticed (rqptr, SS$_BUGCHECK, GzipZlibMsg(zsptr,zstatus), FI_LI);

   VmFreeCache (BufferPtr, FI_LI);

   if (WATCHING (rqptr, WATCH_RESPONSE))
      WatchThis (WATCHITM(rqptr), WATCH_RESPONSE,
                 "CACHE deflate !UL->!UL bytes, !UL%",
                 captr->ContentLength, captr->GzipContentLength,
                 PercentOf32(captr->GzipContentLength,captr->ContentLength));

   return (true);
} 

/*****************************************************************************/
/*
Initialize the ZLIB structure.  Allocate memory from the ZLIB output buffer.
Returns true and set the pointer to the ZLIB structure if successful.  Returns
false and leaves the pointer NULL if not.  Reports any error via the usual WASD
mechanism.
*/

BOOL GzipDeflateBegin
(
REQUEST_STRUCT *rqptr,
GZIP_COMPRESS *gzptr,
int BufferSize
)
{
   int  zstatus;
   z_stream  *zsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "GzipDeflateBegin() !UL", BufferSize);

   gzptr->DeflateStartStream = true;
   gzptr->DeflateEndOfStream = false;
   gzptr->DeflateMemoryAllocated = rqptr->GzipMemoryAllocated = 0;

   zsptr = (z_stream*)VmGetHeap (rqptr, sizeof(z_stream));

   zsptr->opaque = rqptr;
   zsptr->zalloc = &GzipAlloc;
   zsptr->zfree = &GzipFree;

   /* plus 16 is the magic number to get a gzip header and trailer!! */
   zstatus = ZlibDeflateInit2Fn (zsptr,
                                 GzipDeflateCompLevel,
                                 Z_DEFLATED,
                                 GzipDeflateWindowBits + 16,
                                 GzipDeflateMemLevel,
                                 Z_DEFAULT_STRATEGY,
                                 ZLIB_VERSION,
                                 sizeof(z_stream));

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "ZlibDeflateInit2Fn(!UL,!UL+16,!UL) !SL !UL !&Z",
                 GzipDeflateCompLevel, GzipDeflateWindowBits,
                 GzipDeflateMemLevel, zstatus, rqptr->GzipMemoryAllocated,
                 zsptr->msg);

   if (zstatus == Z_OK)
   {
      gzptr->DeflateZstreamPtr = zsptr;
      gzptr->DeflateBufferSize = BufferSize;
      gzptr->DeflateBufferPtr = VmGetHeap (rqptr, BufferSize);
      if (GzipFlushSeconds)
      {
         gzptr->FlushTickSeconds = HttpdTickSecond;
         if (GzipFlushSeconds > GzipFlushInitialSeconds)
            gzptr->FlushTickSeconds += GzipFlushInitialSeconds;
         else
            gzptr->FlushTickSeconds += GzipFlushSeconds;
      }
      return (true);
   }

   rqptr->rqResponse.HttpStatus = 500;
   ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI);
   return (false);
} 

/*****************************************************************************/
/*
Deallocate ZLIB structures.
Returns true if there was no last-minute problem reported by ZLIB, false and
reports the error if there was.
*/

BOOL GzipDeflateEnd
(
REQUEST_STRUCT *rqptr,
GZIP_COMPRESS *gzptr
)
{
   int  zstatus;
   z_stream  *zsptr;

   /*********/
   /* begin */
   /*********/

   if (gzptr->DeflateZstreamPtr)
   {
      if (WATCHING (rqptr, WATCH_RESPONSE))
         if (!WATCHING (rqptr, WATCH_NETWORK))
            WatchThis (WATCHITM(rqptr), WATCH_RESPONSE,
                       "DEFLATE !UL->!UL bytes, !UL% (!ULkB)",
                       gzptr->DeflateBytesIn, gzptr->DeflateBytesOut,
                       PercentOf32(gzptr->DeflateBytesOut,
                                   gzptr->DeflateBytesIn),
                       rqptr->GzipMemoryAllocated >> 10);
   }

   gzptr->DeflateEndOfStream = true;

   InstanceMutexLock (INSTANCE_MUTEX_HTTPD);
   AccountingPtr->GzipDeflateCount++;
   AccountingPtr->GzipDeflateBytesIn64 += gzptr->DeflateBytesIn;
   AccountingPtr->GzipDeflateBytesOut64 += gzptr->DeflateBytesOut;
   InstanceMutexUnLock (INSTANCE_MUTEX_HTTPD);

   if (zsptr = gzptr->DeflateZstreamPtr)
   {
      gzptr->DeflateMemoryAllocated = rqptr->GzipMemoryAllocated;
      zstatus = ZlibDeflateEndFn (zsptr);
      /* this as NULL indicates that deflate is no longer in use */
      gzptr->DeflateZstreamPtr = NULL;
   }
   else
      zstatus = Z_OK;

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "ZlibDeflateEnd() !SL", zstatus);

   if (zstatus == Z_OK) return (true);

   rqptr->rqResponse.HttpStatus = 500;
   ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI);
   return (false);
} 

/*****************************************************************************/
/*
Data to be written to the network is intercepted and passed to this function. 
It uses the ZLIB deflate function to compress the data into the WASD buffer
allocated during initialization.

The '*InDataPtrPtr' and '*InDataLengthPtr' point to the input buffer.  If
'*InDataPtrPtr' is NULL then it's end-of-stream.  When input space is available
and the input buffer consumed '*InDataLengthPtr' is modified to be zero.  This
detected by the calling routine to indicate more input data (actually server
output data, but input to GZIP) should be obtained.

The '*OutDataPtrPtr' and '*DataLengthPtr' are then modified to reflect any
compressed data available for writing (zero indicates none and more should be
fed to the input).

Return true to indicate success, false if the write (and request) should be
terminated due to error (which is also reported by the usual WASD mechanism).
*/

BOOL GzipDeflate
(
REQUEST_STRUCT *rqptr,
GZIP_COMPRESS *gzptr,
char **InDataPtrPtr,
int *InDataLengthPtr,
char **OutDataPtrPtr,
int *OutDataLengthPtr
)
{
   int  zflush, zstatus,
        InDataLength,
        OriginalAvailIn;
   char  *InDataPtr;
   z_stream  *zsptr;

   /*********/
   /* begin */
   /*********/

   zsptr = gzptr->DeflateZstreamPtr;

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "GzipDeflate() !&B !UL !UL !UL !&B",
                 !*InDataPtrPtr, *InDataLengthPtr, 
                 zsptr->avail_in, zsptr->avail_out,
                 gzptr->FlushTickSeconds < HttpdTickSecond);

   /* a NULL pointer indicates end-of-stream */
   if (*InDataPtrPtr)
   {
      if (GzipFlushSeconds)
      {
         if (gzptr->DeflateBytesOut > GZIP_SIZE_OF_HEADER &&
             gzptr->FlushBytesOut != gzptr->DeflateBytesOut)
         {
            /* bytes have been output by ZLIB, reset the 'flush' timer */
            gzptr->FlushTickSeconds = HttpdTickSecond + GzipFlushSeconds;
            gzptr->FlushBytesOut = gzptr->DeflateBytesOut;
         }
         else
         if (gzptr->FlushTickSeconds < HttpdTickSecond) 
         {
            /* no bytes output from ZLIB for the 'flush' number of seconds */
            gzptr->DeflateFlushStream = true;
            gzptr->FlushTickSeconds = HttpdTickSecond + GzipFlushSeconds;
            gzptr->FlushBytesOut = gzptr->DeflateBytesOut;
         }
      }
      if (gzptr->DeflateFlushStream)
         zflush = Z_SYNC_FLUSH;
      else
         zflush = Z_NO_FLUSH;
   }
   else
      zflush = Z_FINISH;

   *OutDataLengthPtr = 0;
   *OutDataPtrPtr = NULL;

   /* if have data to process or have previously processed some */
   if (*InDataLengthPtr || zsptr->total_in)
   {
      if (!zsptr->avail_in)
      {
         /* requires more input */
         zsptr->next_in = InDataPtr = *InDataPtrPtr;
         zsptr->avail_in = InDataLength = *InDataLengthPtr;
         *InDataLengthPtr = 0;
      }
      if (!zsptr->avail_out)
      {
         /* output buffer needs (re)setting */
         zsptr->next_out = gzptr->DeflateBufferPtr;
         zsptr->avail_out = gzptr->DeflateBufferSize;
      }

      zstatus = ZlibDeflateFn (zsptr, zflush);

      gzptr->DeflateBytesIn = zsptr->total_in;
      gzptr->DeflateBytesOut = zsptr->total_out;

      if (zstatus == Z_OK)
      {
         if (gzptr->DeflateFlushStream)
         {
            /* this many bytes are available for flushing */
            gzptr->DeflateBufferCount = gzptr->DeflateBufferSize -
                                        zsptr->avail_out;
            if (zsptr->avail_out)
            {
               /* not a buffer-full though, go back to not flushing */
               gzptr->DeflateFlushStream = false;
               /* reset the output buffer */
               zsptr->next_out = gzptr->DeflateBufferPtr;
               zsptr->avail_out = gzptr->DeflateBufferSize;
            }
         }
         else
         if (!zsptr->avail_out)
         {
            /* the output buffer has filled, flush the contents */
            gzptr->DeflateBufferCount = gzptr->DeflateBufferSize;
         }
         else
            gzptr->DeflateBufferCount = 0;
      }
      else
      if (zstatus == Z_STREAM_END)
      {
         /* whatever's in the buffer */
         gzptr->DeflateBufferCount = gzptr->DeflateBufferSize -
                                     zsptr->avail_out;
      }
      else
      {
         /* yes JPP, you're right, kludgy as it looks - it seems necessary! */
         if (zstatus == Z_BUF_ERROR) zstatus = Z_OK;
         gzptr->DeflateBufferCount = 0;
      }

      *OutDataPtrPtr = gzptr->DeflateBufferPtr;
      *OutDataLengthPtr = gzptr->DeflateBufferCount;

      if (WATCHMOD (rqptr, WATCH_MOD_NET))
         WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                    "!SL !SL !UL/!UL !&X:!UL !&X:!UL !UL->!UL !&Z",
                    zflush, zstatus, zsptr->avail_in, zsptr->avail_out, 
                    InDataPtr, InDataLength,
                    *OutDataPtrPtr, *OutDataLengthPtr,
                    gzptr->DeflateBytesIn, gzptr->DeflateBytesOut, zsptr->msg);

      if (gzptr->DeflateBufferCount)
         if (WATCHING (rqptr, WATCH_NETWORK))
             WatchThis (WATCHITM(rqptr), WATCH_NETWORK,
                        "DEFLATE !UL->!UL bytes, !UL% (!ULkB)",
                        gzptr->DeflateBytesIn, gzptr->DeflateBytesOut,
                        PercentOf32(gzptr->DeflateBytesOut,
                                    gzptr->DeflateBytesIn),
                        rqptr->GzipMemoryAllocated >> 10);
   }
   else
   {
      /* no data has ever been processed, just clean up the ZLIB stuff */
      zstatus = Z_STREAM_END;
   }

   rqptr->BytesTxGzipPercent = PercentOf32 (gzptr->DeflateBytesOut,
                                            gzptr->DeflateBytesIn);

   if (zstatus == Z_OK) return (true);

   if (zstatus == Z_STREAM_END)
      if (GzipDeflateEnd (rqptr, gzptr))
         return (true);

   rqptr->rqResponse.HttpStatus = 500;
   ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI);

   GzipDeflateEnd (rqptr, gzptr);

   return (false);
} 

/*****************************************************************************/
/*
Can and should the content body be deflated?
Return true to indicate deflate, false to leave as-is.
*/

BOOL GzipShouldDeflate
(
REQUEST_STRUCT *rqptr,
char *ContentTypePtr,
int ContentLength
)
{
   BOOL  EncodeGzip;
   char  *cptr;
   char  ResponseKbytes [32];

   /*********/
   /* begin */
   /*********/

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "GzipShouldDeflate() !&B !&Z !SL",
                 GzipAvailable, ContentTypePtr, ContentLength);

   if (!GzipAvailable) return (false);

   if (ContentTypePtr)
      EncodeGzip = true;
   else
   {
      cptr = "no content-type";
      ContentTypePtr = "";
      EncodeGzip = false;
   }

   if (rqptr)
   {
      /* request related considerations */
      if (!rqptr->rqHeader.AcceptEncodingGzip)
      {
         EncodeGzip = false;
         cptr = "client does not accept";
      }
      else
      if (rqptr->rqResponse.ContentIsEncodedGzip ||
          rqptr->rqResponse.ContentIsEncodedUnknown)
      {
         EncodeGzip = false;
         cptr = "response already encoded";
      }
      else
      if (MATCH0 (ContentTypePtr, "application/pdf", 15) &&
          rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_ALL &&
          rqptr->rqCgi.ContentEncodingGzip != CGI_OUTPUT_GZIP)
      {
         EncodeGzip = false;
         cptr = "PDF files are not compressed unless specified by SET rule";
      }    
      else
      if (rqptr->rqCgi.ContentEncodingGzip == CGI_OUTPUT_NO_GZIP)
      {
         EncodeGzip = false;
         cptr = "script requested response NOT to be GZIPed";
      }
      else
      if (rqptr->rqPathSet.ResponseGzip == MAPURL_RESPONSE_GZIP_NONE &&
          rqptr->rqCgi.ContentEncodingGzip != CGI_OUTPUT_GZIP)
      {
         /* note that the script requesting GZIP overrides the mapping rule */
         EncodeGzip = false;
         cptr = "response=GZIP=none";
      }
      else
      if (!rqptr->rqPathSet.ResponseGzip &&
          ContentLength >= 0 &&
          ContentLength < GZIP_MIN_CONTENT_LENGTH)
      {
         /* when fairly small avoid the compression overhead */
         EncodeGzip = false;
         cptr = "minimal content-length";
      }
      else
      if (rqptr->rqPathSet.ResponseGzip &&
          rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_ALL &&
          rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_NONE &&
          (ContentLength < 0 ||
           rqptr->rqPathSet.ResponseGzip >= ContentLength >> 10))
      {
         EncodeGzip = false;
         if (WATCHING (rqptr, WATCH_RESPONSE))
         {
            if (ContentLength >= 0)
               FaoToBuffer (cptr = ResponseKbytes, sizeof(ResponseKbytes), NULL,
                            "response=GZIP=!UL (!ULkB)",
                            rqptr->rqPathSet.ResponseGzip, ContentLength >> 10);
            else
               FaoToBuffer (cptr = ResponseKbytes, sizeof(ResponseKbytes), NULL,
                            "response=GZIP=!UL (?kB)",
                            rqptr->rqPathSet.ResponseGzip);
         }
      }
      else
      if (rqptr->rqResponse.NoGzip)
      {
         EncodeGzip = false;
         cptr = "response NOT to be GZIPed";
      }
   }

   if (EncodeGzip)
   {
      /* content-type considerations */
      if (MATCH6 (ContentTypePtr, "image/") &&
          !MATCH3 (ContentTypePtr+6, "bmp") &&
          !MATCH4 (ContentTypePtr+6, "tiff") &&
          !MATCH8 (ContentTypePtr+6, "x-bitmap") &&
          !MATCH8 (ContentTypePtr+6, "x-pixmap"))
      {
         /* most image content is already compressed (except the above) */
         EncodeGzip = false;
         cptr = "compressed image";
      }
      else
      if (MATCH6 (ContentTypePtr, "audio/") &&
          !MATCH3 (ContentTypePtr+6, "wav") &&
          !MATCH5 (ContentTypePtr+6, "x-wav"))
      {
         /* generally audio is already compressed (except WAVs AFAIK) */
         EncodeGzip = false;
         cptr = "compressed audio";
      }
      else
      if (MATCH6 (ContentTypePtr, "video/"))
      {
         /* video is (hopefully!) already compressed */
         EncodeGzip = false;
         cptr = "compressed video";
      }
      else
      if (MATCH0 (ContentTypePtr, "application/", 12) &&
          (strstr (ContentTypePtr+12, "zip") ||
           strstr (ContentTypePtr+12, "gtar") ||
           strstr (ContentTypePtr+12, "archive") ||
           strstr (ContentTypePtr+12, "stuffit") ||
           strstr (ContentTypePtr+12, "x-compressed")))
      {
         /* compressed application stream */
         EncodeGzip = false;
         cptr = "compressed archive";
      }
      else
      if (MATCH0 (ContentTypePtr, "multipart/", 10) &&
          strstr (ContentTypePtr+10, "zip"))
      {
         /* same, same */
         EncodeGzip = false;
         cptr = "compressed archive";
      }
      else
      if (MATCH0 (ContentTypePtr, "application/pdf", 15) &&
          rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_ALL &&
          rqptr->rqCgi.ContentEncodingGzip != CGI_OUTPUT_GZIP)
      {
         EncodeGzip = false;
         cptr = "PDF files are not compressed";
      }
      else
      if (MATCH0 (ContentTypePtr, "application/x-shockwave-flash", 29))
      {
         EncodeGzip = false;
         cptr = "Shockwave Flash files are not compressed";
      }
      else
      if (MATCH0 (ContentTypePtr, "text/x-shtml", 12))
      {
         EncodeGzip = false;
         cptr = "SSI source are not compressed";
      }
      else
      if (ContentLength >= 0 &&
          ContentLength < GZIP_MIN_CONTENT_LENGTH)
      {
         /* when fairly small avoid the compression overhead */
         EncodeGzip = false;
         cptr = "minimal content-length";
      }
   }

   if (WATCHING (rqptr, WATCH_RESPONSE))
      if (!EncodeGzip)
         WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "DEFLATE no, !AZ", cptr);

   return (EncodeGzip);
} 

/*****************************************************************************/
/*
Initialize the ZLIB structure.  Allocate memory from the ZLIB output buffer.
Returns true and set the pointer to the ZLIB structure if successful.  Returns
false and leaves the pointer NULL if not.  Reports any error via the usual WASD
mechanism.
*/

BOOL GzipInflateBegin
(
REQUEST_STRUCT *rqptr,
GZIP_COMPRESS *gzptr,
int BufferSize
)
{
   int  zstatus;
   z_stream  *zsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipInflateBegin()");

   gzptr->InflateStartStream = true;
   gzptr->InflateEndOfStream = false;
   gzptr->InflateMemoryAllocated = rqptr->GzipMemoryAllocated = 0;

   zsptr = (z_stream*)VmGetHeap (rqptr, sizeof(z_stream));

   zsptr->opaque = rqptr;
   zsptr->zalloc = &GzipAlloc;
   zsptr->zfree = &GzipFree;

   /* plus 16 is the magic number to remove a gzip header and trailer!! */
   zstatus = ZlibInflateInit2Fn (zsptr,
                                 GzipInflateWindowBits + 16,
                                 ZLIB_VERSION,
                                 sizeof(z_stream));

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "ZlibInflateInit2Fn(!UL+16) !SL !UL !&Z",
                 GzipDeflateWindowBits, zstatus,
                 rqptr->GzipMemoryAllocated, zsptr->msg);

   if (zstatus == Z_OK)
   {
      gzptr->InflateZstreamPtr = zsptr;
      gzptr->InflateBufferSize = BufferSize;
      gzptr->InflateBufferPtr = VmGetHeap (rqptr, BufferSize);
      return (true);
   }

   rqptr->rqResponse.HttpStatus = 500;
   ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI);
   return (false);
} 

/*****************************************************************************/
/*
Deallocate ZLIB structures.
Returns true if there was no last-minute problem reported by ZLIB, false and
reports the error if there was.
*/

BOOL GzipInflateEnd
(
REQUEST_STRUCT *rqptr,
GZIP_COMPRESS *gzptr
)
{
   int  zstatus;
   z_stream  *zsptr;

   /*********/
   /* begin */
   /*********/

   if (gzptr->InflateZstreamPtr)
   {
      if (WATCHING (rqptr, WATCH_RESPONSE))
         if (!WATCHING (rqptr, WATCH_NETWORK))
            WatchThis (WATCHITM(rqptr), WATCH_RESPONSE,
                       "INFLATE !UL->!UL bytes, !UL% (!ULkB)",
                       gzptr->InflateBytesIn, gzptr->InflateBytesOut,
                       PercentOf32(gzptr->InflateBytesOut,gzptr->InflateBytesIn),
                       rqptr->GzipMemoryAllocated >> 10);
   }

   gzptr->InflateEndOfStream = true;

   InstanceMutexLock (INSTANCE_MUTEX_HTTPD);
   AccountingPtr->GzipInflateCount++;
   AccountingPtr->GzipInflateBytesIn64 += gzptr->InflateBytesIn;
   AccountingPtr->GzipInflateBytesOut64 += gzptr->InflateBytesOut;
   InstanceMutexUnLock (INSTANCE_MUTEX_HTTPD);

   if (zsptr = gzptr->InflateZstreamPtr)
   {
      gzptr->InflateMemoryAllocated = rqptr->GzipMemoryAllocated;
      zstatus = ZlibInflateEndFn (zsptr);
      /* this as NULL indicates that inflate is no longer in use */
      gzptr->InflateZstreamPtr = NULL;
   }
   else
      zstatus = Z_OK;

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                 "ZlibInflateEnd() !SL", zstatus);

   if (zstatus == Z_OK) return (true);

   if (zstatus == Z_BUF_ERROR) return (true);

   rqptr->rqResponse.HttpStatus = 500;
   ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI);
   return (false);
} 

/*****************************************************************************/
/*
Data read from the network is intercepted and passed to this function. 
It uses the ZLIB inflate function to decompress the data into the WASD buffer
allocated during initialization.  The 'DataPtr' and 'DataLength' are then
modified to reflect the decompressed data and it can then be further processed.
Return true to indicate success, false if the write (and request) should be
terminated due to error (which is also reported by the usual WASD mechanism).
*/

BOOL GzipInflate
(
REQUEST_STRUCT *rqptr,
GZIP_COMPRESS *gzptr,
char **DataPtrPtr,
int *DataLengthPtr
)
{
   int  zstatus;
   z_stream  *zsptr;

   /*********/
   /* begin */
   /*********/

   if (WATCHMOD (rqptr, WATCH_MOD_NET))
      WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipInflate() !&X !UL",
                 *DataPtrPtr, *DataLengthPtr);

   zsptr = gzptr->InflateZstreamPtr;

   if (!zsptr->avail_in)
   {
      zsptr->next_in = *DataPtrPtr;
      zsptr->avail_in = *DataLengthPtr;
   }

   if (zsptr->avail_in || zsptr->total_in)
   {
      /* if we have data to process or have previously processed some */
      if (!zsptr->avail_out)
      {
         /* output buffer needs (re)setting */
         zsptr->next_out = gzptr->InflateBufferPtr;
         zsptr->avail_out = gzptr->InflateBufferSize;
      }

      if (WATCHMOD (rqptr, WATCH_MOD_NET))
         WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "!UL(!UL)/!UL",
                    zsptr->avail_in,  gzptr->InflateAvailIn, zsptr->avail_out);

      zstatus = ZlibInflateFn (zsptr, Z_NO_FLUSH);

      gzptr->InflateBytesIn = zsptr->total_in;
      gzptr->InflateBytesOut = zsptr->total_out;

      if (zstatus == Z_OK && !zsptr->avail_out)
         gzptr->InflateBufferCount = gzptr->InflateBufferSize;
      else
      if (zstatus == Z_STREAM_END)
         gzptr->InflateBufferCount = gzptr->InflateBufferSize -
                                     zsptr->avail_out;
      else
         gzptr->InflateBufferCount = 0;

      if (WATCHMOD (rqptr, WATCH_MOD_NET))
         WatchThis (WATCHITM(rqptr), WATCH_MOD_NET,
                    "!SL !UL/!UL !UL !UL->!UL !&Z",
                    zstatus, zsptr->avail_in, zsptr->avail_out, 
                    gzptr->InflateBufferCount, gzptr->InflateBytesIn,
                    gzptr->InflateBytesOut, zsptr->msg);

      if (gzptr->InflateBufferCount)
         if (WATCHING (rqptr, WATCH_NETWORK))
            WatchThis (WATCHITM(rqptr), WATCH_NETWORK,
                       "INFLATE !UL->!UL bytes, !UL% (!ULkB)",
                       gzptr->InflateBytesIn, gzptr->InflateBytesOut,
                       PercentOf32(gzptr->InflateBytesOut,gzptr->InflateBytesIn),
                       rqptr->GzipMemoryAllocated >> 10);
   }
   else
   {
      /* no data has ever been processed, just clean up the ZLIB stuff */
      zstatus = Z_STREAM_END;
      gzptr->InflateBufferCount = 0;
   }

   if (zstatus == Z_OK ||
       zstatus == Z_STREAM_END)
   {
      if (zstatus == Z_STREAM_END)
         if (!GzipInflateEnd (rqptr, gzptr))
            return (false);

      gzptr->InflateAvailIn = zsptr->avail_in;
      *DataPtrPtr = gzptr->InflateBufferPtr;
      *DataLengthPtr = gzptr->InflateBufferCount;

      return (true);
   }

   rqptr->rqResponse.HttpStatus = 500;
   ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI);

   GzipInflateEnd (rqptr, gzptr);

   return (false);
} 

/*****************************************************************************/
/*
Allocate request heap memory on behalf of ZLIB demands.
Shouldn't leak any memory this way (not that ZLIB is likely to).
*/

char* GzipAlloc
(
REQUEST_STRUCT *rqptr,
int items,
int size
)
{
   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_NET))
      WatchThis (WATCHALL, WATCH_MOD_NET,
                 "GzipAlloc() !UL !UL", items, size);

   rqptr->GzipMemoryAllocated += size*items; 
   return (VmGetHeap (rqptr, size*items));
} 

/*****************************************************************************/
/*
Free GzipAlloc()ed request heap memory.
*/

GzipFree
(
REQUEST_STRUCT *rqptr,
char *mptr
)
{
   /*********/
   /* begin */
   /*********/

   if (WATCH_MODULE(WATCH_MOD_NET))
      WatchThis (WATCHALL, WATCH_MOD_NET, "GzipFree()");

   VmFreeFromHeap (rqptr, mptr, FI_LI);
} 

/*****************************************************************************/

