ViCompress Manual

1. Forward to the host given in the URL

   In this setup, ViCompress acts as a regular forward proxy. Given a URL, such as:

   https://<hostname.com>/path/index.html

   ViCompress performs a DNS lookup of the hostname specified in the request URL, and forwards the request to that origin webserver. If no webserver entries are specified, then ViCompress defaults to this mode.

    

2. Forward to a set of backend webservers

   In this setup, ViCompress acts as a reverse proxy for a website with the same set of backend webservers. Instead of performing a DNS lookup of the hostname in the URL, ViCompress chooses one of the backend webservers specified in the configuration file:

   webserver 192.168.0.10 80
webserver 192.168.0.10 81
webserver 192.168.0.20 80
webserver 192.168.0.21 80


   Since ViCompress load-balances among the webservers, they should be identical. They should contain the same content, and contain the same websites (virtual hosts). If you are running different websites (virtual hosts) on different IP addresses/ports, then you should use the third setup below:

3. Forward a virtual host to a specific webserver

   In this setup, ViCompress acts as a reverse proxy for multiple websites, where different websites (virtual hosts) are run on different machines/ports. The webserver option is used to specify which machine/ports are running which websites:

   webserver 192.168.0.10 80 company.com www.company.com
webserver 192.168.0.20 80 company.com www.company.com
webserver 192.168.0.10 81 company2.com www.company2.com
webserver 192.168.0.20 81 company2.com www.company2.com
webserver 192.168.0.30 80
webserver 192.168.0.40 80


   In the example above:

   Requests for host company.com or www.company.com are forwarded to 192.168.0.10 port 80 and 192.168.0.20 port 80.

   Requests for host company2.com or www.company2.com are forwarded to 192.168.0.10 port 81 and 192.168.0.20 port 81

   Requests for any other host are forwarded to 192.168.0.30 port 80 and 192.168.0.40 port 80

Network related options:
listen
webserver
outgoingip


• HTML
• Javascript
• CSS stylesheets
• PDF documents
• Microsoft Word documents

• Images (GIF, PNG, JPEG)
• Video
• Music
• Executables

ViCompress sends a special HTTP header when it compresses any pages, so that the browser is aware the page is compressed:

Content-Encoding: gzip

The web browser automatically decompresses the gzipped page, and displays it to the user. No additional or special client software is needed. All modern web browsers recognize gzip content encoding, including:

• Internet Explorer 4 and above
• Firefox (all versions)
• Opera 4 and above
• Chrome (all versions)
• Netscape 6 and above.

In addition, ViCompress will not send gzipped content to browsers that do not support it. Browsers that recognize gzip compression send a special HTTP header in the request:

Accept-Encoding: gzip

ViCompress first checks for this header before gzipping any pages.

Compression related options:
enable_compression

ViCompress will not cache web pages that are generated dynamically, such as through ASP, PHP, or CGI scripts. ViCompress looks for two HTTP headers

Last-Modified
Expires


to determine whether a response is dynamically generated or not. Only responses which contain an Expires or Last-Modified header will be cached. Most web servers, such as Apache and Microsoft IIS, include a Last-Modified header when serving static files, such as images, HTML, or javascript files. The Last-Modified header is set to the date the file was last modified.

If the Expires header is present, the object is cached, and the expiration time is set to the time in that header. Or, if the Last-Modified header is present, the object is cached, and the expiration time is set to half of the item's age. For example, a web page that was last modified 8 days ago will remain in the ViCompress cache for 4 days.

When the in-memory cache becomes full, items that have not been recently accessed are removed to make room for new items in the cache.

Users can view the list of URLs in the memory cache by logging into the ViCompress machine and sending the following special URL to ViCompress:

ViCompress will return a plain text list of the URLs in the cache, one per line. You can retrieve this list only from an http client on the same machine as ViCompress. Outside clients cannot access the cached URL list.

Caching related options:
enable_caching
cache_memory
max_cacheditem_size
cache_expires


If ViCompress fails to connect to a backend web server, that web server is marked as down, and will be skipped for future requests. Clients that had previous sessions with that web server will be forwarded to a new backend web server. ViCompress will try to re-connect to a down web server every 3 minutes. If the connection succeeds, the web server is marked as up again. If all backend web servers are down when a request arrives, ViCompress will simply choose among the down web servers.

Many web applications keep session information for each client, such as shopping cart items. Session information may be stored on a central database, or may be stored locally on individual web servers. If your website stores session information on individual web servers, then a client's requests cannot be distributed across multiple web servers. To force a client to use the same backend web server throughout a session, ViCompress provides the enable_sessions option. When enabled, ViCompress will send the client a cookie to indicate which backend web server to use:

Set-Cookie: vicompressid=1

For the duration of the session, the client browser will send the vicompressid Cookie for every request:

Cookie: vicompressid=1

ViCompress will forward the requests to the backend webserver specified by the cookie. When the client browser is closed, the browser discards the vicompressid Cookie, and the session is ended. Note that if sessions are enabled, client connections may not be evenly distributed across the multiple backend web servers.

Load Balancing related options:
webserver
enable_sessions


ViCompress produces two log files: an access log and error log.

The accesslog stores information about each client request on a single line. It is used for gathering website statistics. The logformat option controls the format of the access log file. ViCompress supports two log formats:

• The Apache Combined Log Format, described at https://httpd.apache.org/docs/logs.html
• The Squid Access Log Format, described at https://wiki.squid-cache.org/SquidFaq/SquidLogs

Log Rotation

The accesslog file can grow quickly under heavy load. Therefore, ViCompress will automatically rotate log files once they reach a certain size. This size is given by the configuration option:

rotatesize <size in megabytes>

ViCompress will save up to two previous copies of the accesslog. When rotation occurs, ViCompress will execute the following:

and will then create a new, empty accesslog file.

Error Log

The error log stores error and debugging messages from ViCompress. The path of the errorlog is determined by the configuration option errorlog:

errorlog "C:\Program Files\ViCompress\log\errorlog.txt"
errorlog /usr/local/vicompress/log/errorlog

By default, ViCompress logs just basic startup and shutdown messages, as well as the up/down status of backend web servers. If the enable_debug option is set to yes, ViCompress will log additional messages to the errorlog, including

• When a new client is accepted
• The http request header received from the client
• The destination IP address ViCompress connects to
• The http reply header received from the origin server
• Whether ViCompress is compressing/caching the reply

Log related options:
accesslog
errorlog
rotatesize
enable_debug


C:\Program Files\ViCompress\logstats\YYYYMMstats.html (Windows)
/usr/local/vicompress/logstats/YYYYMMstats.html (Unix)

where YYYY is the year, and MM is the month. For example:

C:\Program Files\ViCompress\logstats\200501stats.html (Jan 2005)
C:\Program Files\ViCompress\logstats\200502stats.html (Feb 2005)
C:\Program Files\ViCompress\logstats\200503stats.html (Mar 2005)

/usr/local/vicompress/logstats/200501stats.html (Jan 2005)
/usr/local/vicompress/logstats/200502stats.html (Feb 2005)
/usr/local/vicompress/logstats/200503stats.html (Mar 2005)


This report will show the bandwidth saved with caching and compression for

• The entire month
• Each day of the month
• Each day of the week
• Each hour of the day

A sample report for day of the week is shown below:

Weekly bandwidth

Sun Mon Tue Wed Thu Fri Sat

Bandwidth Requests Day Without gzip With gzip Without cache With cache Total Gzipped Cached Sun 294.80 MB 160.30 MB (54%) 294.80 MB 167.08 MB (56%) 27220 4440 (16%) 16892 (62%) Mon 605.43 MB 320.20 MB (52%) 605.43 MB 350.32 MB (57%) 50370 8437 (16%) 32154 (63%) Tue 634.86 MB 337.99 MB (53%) 634.86 MB 362.19 MB (57%) 51054 8766 (17%) 32518 (63%) Wed 558.14 MB 300.22 MB (53%) 558.14 MB 330.64 MB (59%) 51262 8438 (16%) 32144 (62%) Thu 636.47 MB 357.06 MB (56%) 636.47 MB 360.93 MB (56%) 50480 8524 (16%) 31859 (63%) Fri 523.90 MB 296.97 MB (56%) 523.90 MB 284.05 MB (54%) 39884 6770 (16%) 25395 (63%) Sat 291.60 MB 164.25 MB (56%) 291.60 MB 163.88 MB (56%) 21725 3666 (16%) 13518 (62%)

View a complete sample report.

In addition, an HTML date index file will be created containing hyperlinks to all the monthly statistics.
C:\Program Files\ViCompress\logstats\statsindex.html (Windows)
/usr/local/vicompress/logstats/statsindex.html (Unix)

The statistics can also be generated independent of the vicompress server, by running update_log_stats on the command line with the following arguments:

On Windows:

On Unix:

Log Statistics related options
accesslog
logstats

• It supports persistent connections with both clients and origin servers
• It supports chunked transfer encoding
• It supports the Cache-Control header
• It supports cache hits on Range HTTP requests
• It supports PUT method uploads and HTTP/1.1 100 Continue replies
• It supports X-Forwarded-For header

• Handles thousands of simultaneous connections
  ViCompress uses non-blocking network I/O with modern polling mechanisms (epoll, kqueue, /dev/poll) rather than spawning a thread/process per request. This allows ViCompress to handle thousands of connections simultaneously.
• Scales to multiple processors
  ViCompress uses multiple threads for the CPU intensive gzip compression. ViCompress will automatically determine how many processors your system has, and will spawn one compression thread per processor.
• Persistent connections
  A persistent connection allows a client to request and download multiple objects (like images) using a single TCP connection, rather than multiple TCP connections. A persistent connection reduces the latency of each request, since it bypasses the TCP handshake needed for a new TCP connection. ViCompress supports persistent connections to both the client and server. Even when compressing data on-the-fly, ViCompress still maintains a persistent connection by using HTTP 1.1 chunked transfer-encoding.
• DNS lookup caching
  When configured as a forward proxy, DNS lookup time can often be slow. ViCompress will cache DNS lookups, thereby improving response time.

ViCompress supports 3 modes of forwarding

1. (Forward proxy) Forward to the host given in the URL
2. (Reverse proxy) Forward to a set of backend webservers
3. (Reverse proxy) Forward each website (virtual host) to a specific webserver

This option is used if ViCompress is being used as a reverse proxy, as opposed to a forward proxy.

Specify the IP address and port of the backend webserver to forward requests to. Multiple webserver entries may be specified, each on a separate line. Both IPv4 and IPv6 addresses are supported.

If you are running multiple websites (virtual hosts) on different IP addresses/ports, then you can also specify the virtual hosts in addition to the IP address and port. Multiple virtual hosts can be specified for a single webserver entry. When ViCompress receives a request, it will parse out the hostname in the URL and find a webserver entry with a matching virtual host. For example:

webserver 192.168.0.10 80 company.com www.company.com
webserver 192.168.0.20 80 company.com www.company.com
webserver 192.168.0.10 81 company2.com www.company2.com
webserver 192.168.0.20 81 company2.com www.company2.com
webserver 192.168.0.30 80
webserver 192.168.0.40 80


In the example above:

• Requests for host company.com or www.company.com are
  forwarded to 192.168.0.10 port 80 and 192.168.0.20 port 80.
• Requests for host company2.com or www.company2.com are
  forwarded to 192.168.0.10 port 81 and 192.168.0.20 port 81
• Requests for any other host are forwarded to 192.168.0.30 port 80 and 192.168.0.40 port 80

Specify the IP address and port for ViCompress to listen on. Multiple listen entries may be specified, each on a separate line. Both IPv4 and IPv6 addresses are supported. The default value is all IPv4 interfaces, 0.0.0.0, on port 80, the standard HTTP port. Only servers started by root can bind to ports less than 1024. The older (1.0.7 and earlier) configuration options listenip <IP address> and listenport <port> have been deprecated.

Specify the IP address for ViCompress to bind to when making outgoing connections. Both IPv4 and IPv6 addresses are supported. The default value is any interface, 0.0.0.0.

Enable or disable sticky sessions. The default value is yes. This option is only used when two or more backend webservers are specified in the ViCompress configuration. When enabled, ViCompress will use HTTP Cookies to ensure that a client is sent to the same backend web server for the duration of its session. An HTTP session lasts until the client browser is closed. See the Load Balancing and Failover section for more details.

Enable or disable gzip compression. The default value is yes. When enabled, ViCompress will gzip HTML and text pages before sending the response to the client. See the Compression section for details about how gzip compression is performed.

Enable or disable caching of pages. The default value is yes. When enabled, ViCompress will cache static pages and images in memory. These cached pages are only in memory, they are not written to disk.

See the Caching section for details about what objects are cached, and for how long.

Specify the size of the in-memory cache, in megabytes. The default value is 100. Note that under high load, ViCompress will also use around 12 MB for compression and 10 MB for basic operation. The total memory (cache_memory + 22 MB) should not exceed the amount of RAM memory available. If cache_memory is set to 0, caching is disabled.

Web pages larger than this size will not be cached. In order to have a high hit rate, ViCompress should cache many small pages, rather than a few large pages. To prevent large pages from being cached, use this option. The default value is 512 kilobytes.

When a web page is cached, it remains in the cache based on its age. The expiration time is set to half of the item's age. For example, a page that is 4 days old will be cached for 2 days.

This option is used to place an upper limit on the expiration time. The default value is 240 hours (10 days). After 10 days, the page is removed from the cache.

Enable or disable caching of dns lookups. The default value is yes. When enabled, ViCompress will store the hostname-to-IP address mappings in memory.

Specify the amount of time a cached DNS mapping is valid. The default is 2 hours.

The user to run the server as. ViCompress will switch to this user after binding to the listening port (usually port 80). ViCompress is generally started as root, since only root can bind to ports less than 1024. However, it is unsafe to run a server program as root. Therefore, ViCompress will switch to a non-root user after binding to port 80. That user is specified by the "user" option given above.

The default value is the user who started the server.

This option only applies when accelerating a website. It does not apply to forward proxies. Specify the hostname to use in the HTTP Host header, when sending the HTTP request to the backend webservers. By default, ViCompress will just send the same HTTP Host header it receives from the client browser.

Specify the file path where the access log should be stored. The file must be writable by the username given in the "user" option. If the filename contains spaces, the filename must be enclosed in quotes. If the accesslog is not specified, no access log file will be created or written to.

Specify the file path where the error log should be stored. The file must be writable by the username given in the "user" option. If the filename contains spaces, the filename must be enclosed in quotes. If the errorlog is not specified, no error log file will be created or written to.

Every hour, the vicompress server will run the update_log_stats program to generate the log statistics from the accesslog. The logstats option specifies the directory to store the log statistics in. The directory should already exist, and should contain the template.html page. If the logstats directory is not specified, no statistics will be created or written to.

Specify the file path where the HTML error page is located. This HTML page will be sent back to users when ViCompress is unable to lookup or connect to the origin web server in the HTTP request. If no file is specified, then no HTML content is sent back on HTTP error replies.

Rotate the log files when they reach the specified size, in megabytes. The default value is 100. The maximum value is 2047, or about 2 gigabytes. When rotation occurs, the following commands are executed:

and a blank log file is created at <accesslog>. See Log Files for further details about log file rotation.

Specify the format of the accesslog file. The supported formats are the Apache Combined Log Format, and the Squid Access Log Format. The default value is the Squid format.

A brief summary of the two formats is given below. Note that the Apache format does not include the hostname of the request URL, only the URL path.

Apache Log Format

clientip       : The IP Address of the client.
hit/compression: "hit" or "miss", followed by the length before compression.
username       : The username sent for authentication, or "-" if not given.
date           : The date of the response [day/month/year:hour:min:sec +/-timezone]
firstline      : The first line of the HTTP request (method url version).
replycode      : The server HTTP reply status code.
contentlength  : The reply length after compression, in bytes.
referer        : The URL which referred the user to this website.
useragent      : The platform and version of the client browser.

A sample Apache log entry is shown below:

15.13.130.10 miss15923 - [21/Aug/2003:17:26:45 -0700] "GET /index.html HTTP/1.0"
200 1852 "https://www.google.com/" "Mozilla 4.0 (IE 6.0 compatible)"

Squid Log Format

date          : The date of the response, the number of seconds since 1970.
duration      : The duration of the response, in milliseconds.
client        : The IP address of the client.
hit status    : TCP_HIT if the request is a cache hit, else TCP_MISS.
replycode     : The server HTTP reply status code.
contentlength : The reply length after compression, in bytes.
method        : The HTTP request method (GET, POST, etc).
URL           : The requested URL.
compression   : The reply length before compression, in bytes.
peerstatus    : NONE if the request is a cache hit, else DIRECT.
peerhost      : The IP address of the backend web server, or "-" if a cache hit.
contenttype   : The content type of the HTTP reply, or "-" if not given.

A sample Squid log entry is shown below:

1112387949.000 529 15.13.130.249 TCP_MISS/200 1031 GET https://www.myhost.com/somefile.jpg
8523 DIRECT/15.0.110.12 text/html

See the Log Files section for further details.

Enable or disable debugging. The default value is no. When enabled, ViCompress will write debug messages to the errorlog. By default, ViCompress logs just basic startup and shutdown messages, as well as the status of backend web servers. The messages are shown below.

Each debug message in the errorlog contains the date, the IP address:port of the client connection, and the debug message. Here is a sample entry:

Below is the complete list of debugging messages:

For the <HTTP request> and <HTTP reply>, ViCompress will print out the full http request and reply headers. For <status message>, ViCompress will print out one of the messages below:

Below is a sample debug log session for a non-cached request, that is being compressed