Documentation: Chapter 9
Custom Log Formats

---

FlashStats is very adept at automatically understanding all of the major log file formats. It also supports self-identifying log file formats (ie, those which write an extra header explaining their format). However, it is possible to configure some web servers so that they generate a non-standard format that confuses FlashStats. To support these systems, FlashStats allows you to specify a custom format.

Unsupported Log Formats

---

FlashStats does not support the following log file formats:

• Database (SQL, Access, etc).
• Any other binary (non-text) format.
• Quad-log systems such as old Apache web servers which generate four separate logs (access_log, referer_log, agent_log, error_log). More accurately, FlashStats does support analyzing the access_log (which should be in Common Log Format), but FlashStats will ignore the referer_log, agent_log, and error_log. Therefore you will only get the standard reports (eg, Top URLs Requested) but not the cool extended reports (eg, Search Phrases, Most Common Browsers). If your web server generates these types of logs, you should upgrade it to a later version which supports integrated logs. More details are available here.

Natively Supported Log Formats

---

Here is a list of the standard log formats. Check one of your web server's log files to see if they match one of these formats.

Common Log Format
This log format separates parameters with a space. The fields are:
1. Client host name or IP address
2. Authenticated realm (often just a placeholder -)
3. Authenticated username (often just a placeholder -)
4. Date and time (enclosed in [brackets])
5. The request string (enclosed in "quotes") (see note below)
6. The result code (eg, 200, 404, etc)
7. The number of bytes returned to the client (excluding header bytes)

The custom log file format string for this format is:

LogFormat=Shmw1qrc
This is a standard format built into FlashStats; you don't need to specify it with a LogFormat= parameter.

Note: Field #5 normally looks like "GET /dir/file.ext HTTP/1.0". However, some web servers generate log files that include the host name at the start of the document so that the fields look like "GET http://www.domain.com:80/dir/file.ext HTTP/1.0". If this is how your log files look, then when you are requesting a report you will have to specify the host name along with the path that you're interested in in the Page To Analyze field. For example, instead of just specifying / you would have to specify http://www.domain.com:80/.

---

Extended (or Combined) Log Format
This log format is a simple extension of the Common Log Format. The first seven fields are the same as the Common Log Format, and they are followed by two additional fields (referer and user agent):

1. Client host name or IP address
2. Authenticated realm (often just a placeholder -)
3. Authenticated username (often just a placeholder -)
4. Date and time (enclosed in [brackets])
5. The request string (enclosed in "quotes")
6. The result code (eg, 200, 404, etc)
7. The number of bytes returned to the client (excluding header bytes)
8. The HTTP REFERER field (enclosed in "quotes")
9. The HTTP USER_AGENT field (enclosed in "quotes")

The custom log file format string for this format is:

LogFormat=Shmw1qrcfa
This is a standard format built into FlashStats; you don't need to specify it with a LogFormat= parameter.

---

Windows Log Format (O'Reilly WebSite)
This log format is supported only by the WebSite web server from O'Reilly Software. If you are running WebSite then we recommend that you use their Combined Log Format as described above. However, if you are using the Windows Log Format, FlashStats will still support it. The fields are tab-delimited. Here is a list of the fields:

1. Date and time
2. Client host name or IP address
3. Server host name or IP address (for multi-homed sites)
4. Authenticated realm
5. Authenticated username
6. HTTP method (eg, GET, POST, HEAD, etc)
7. The path portion of the requested URL (does not contain the query string)
8. The HTTP REFERER field
9. The email address of the client (not used)
10. The HTTP USER_AGENT field
11. The result code (eg, 200, 404, etc)
12. The number of bytes returned to the client (excluding header bytes)
13. The time required to process the request (in milliseconds)

The custom log file format string for this format is:

LogFormat=T4h?mw?uf?arc?
This is a standard format built into FlashStats; you don't need to specify it with a LogFormat= parameter.

---

WebSTAR Log Format (Macintosh)
The WebSTAR log format is self-identifying. Therefore, any combination of fields can be read by FlashStats. See Chapter 2 for details on which fields are required by FlashStats.

Please note that some third-party log rotation utilities do not properly initialize the new log file with the WebSTAR header fields. Because of this, when FlashStats begins to read the file it can't determine what the fields are. Therefore, we recommend that you do not use a third-party log rotation utility. You should instead use the rotater that comes with WebSTAR. (The rotater that came with version 1 of WebSTAR was not very good, but the current version is much better and should be able to handle all of your needs.)

Quid Pro Quo uses a compatible log file format.

The custom log file format string for this format depends upon the fields selected (and their order).

---

MacHTTP Log Format (Macintosh)
The MacHTTP web server generates the following log file format, with fields separated by tabs:

1. Date
2. Time (24-hour format)
3. The result code as a string (usually OK or ERR)
4. Client host name or IP address
5. The path portion of the requested URL (does not contain the query string)
6. The number of bytes returned to the client (excluding header bytes)

The custom log file format string for this format is:

LogFormat=T29khxc
This is a standard format built into FlashStats; you don't need to specify it with a LogFormat= parameter.

---

W3C Extended Log File Format (IIS 4 and above)
With IIS 4 and IIS 5, the default log file format used is the W3C Extended Log File Format. This is a flexible, self-identifying format and is the format preferred by FlashStats.

The IIS administrative interface lets you decide which fields to write to your log files. At a minimum you should include the following fields in your log files:

1. Date
2. Time
3. Client IP Address
4. Method
5. URI Stem
6. URI Query
7. Http Status
8. Bytes Sent
9. User Agent
10. Referrer

See Chapter 2 for full details on configuring IIS logging.

Support for this standard format is built into FlashStats; you don't need to specify it with a LogFormat= parameter.

---

Microsoft IIS 3 Log File Format (Windows)
Microsoft IIS 3 uses a special log file format. This format is natively supported by FlashStats. The fields are terminated by a comma and a space (even the last field). The fields are:

1. Client host name or IP address
2. Authenticated username
3. Date (in local format)
4. Time (24-hour format)
5. The string "W3SVC" indicating a web server entry (as opposed to FTP or Gopher)
6. The NetBIOS name of the web server
7. The IP address of the web server connected to (useful for multi-homed sites; FlashStats treats this as the Authenticated Realm field)
8. The time required to process the request (in milliseconds)
9. The number of bytes received from the client
10. The number of bytes returned to the client (excluding header bytes)
11. The result code (eg, 200, 404, etc)
12. Windows [internal] status code
13. HTTP method (eg, GET, POST, HEAD, etc)
14. The path portion of the requested URL (does not contain the query string)
15. The query string portion of the requested URL

The custom log file format string for this format is:

LogFormat=Chw29v?m??cr??xe
This is a standard format built into FlashStats; you don't need to specify it with a LogFormat= parameter.

This old log format is also available on IIS 4 and IIS 5 but should not be used as it is an inferior format.

---

IIS / PWS FlashLog Format (Maximized Software)
The Microsoft IIS 3 "in*.log" log file format described above does not record the referer or user agent information for a request. Therefore, Maximized Software has released an ISAPI filter plug-in called FlashLog which adds those fields to the log file format. FlashLog is available as a free download from our web site, and is also included as a standard part of FlashStats. FlashLog would only be used in conjunction with the IIS 3 Log File Format, not with the preferred W3C Extended Log File Format.

FlashLog adds the two fields to the log file in a compatible style: they are added to the end of the line, so that they do not interfere with any other parsing software. FlashStats will automatically detect and use the two extended fields if they are present.

Therefore, when running with the FlashLog plug-in, the log file fields are:

1. Client host name or IP address
2. Authenticated username
3. Date (in local format)
4. Time (24-hour format)
5. The string "W3SVC" indicating a web server entry (as opposed to FTP or Gopher)
6. The NetBIOS name of the web server
7. The IP address of the web server connected to (useful for multi-homed sites; FlashStats treats this as the Authenticated Realm field)
8. The time required to process the request (in milliseconds)
9. The number of bytes received from the client
10. The number of bytes returned to the client (excluding header bytes)
11. The result code (eg, 200, 404, etc)
12. Windows [internal] status code
13. HTTP method (eg, GET, POST, HEAD, etc)
14. The path portion of the requested URL (does not contain the query string)
15. The query string portion of the requested URL
16. The HTTP REFERER field
17. The HTTP USER_AGENT field

The custom log file format string for this format is:

LogFormat=Chw29v?m??cr??xefa
This is a standard format built into FlashStats; you don't need to specify it with a LogFormat= parameter.

Specifying A Custom Log Format

---

You can use the LogFormat= parameter to tell FlashStats about a custom log format. You do this by creating a short string containing characters representing the log file fields, in proper order. The first character in the string indicates how the field are separated in the log file; the remaining characters represent the different fields. All characters in the custom format string are case-sensitive!

As an example, the format for the Extended Log Format is: Shmw1qrcfa.

Note that if you include more characters than there are fields in the log, any extra characters will be ignored. For example, the Extended Log Format could also be used for Common Log Format files, and the last two characters (f representing the referer field, and a representing the user agent field) would simply be ignored because the Common Log Format file would not contain those fields.

Custom Log Format Characters

Character Meaning C Fields are separated by a comma and a space (used by IIS 3) S Fields are separated by a space; fields containing a space are enclosed with "quotes" or [brackets] or are URL-encoded T Fields are separated by a tab 1 (one) Date and time in format: 31/Dec/1999:23:59:59 -0800 2 Date, format: 12/31/99 3 Date, format: 1999-12-31 4 Date and time, format: 12/31/99 23:59:59 5 (Usually WebSTAR) Date, format: 12/31/99 or 1999-12-31 or Fri, 31 Dec 1999 9 Time, format: 23:59:59 a The HTTP USER_AGENT field, format: Mozilla/3.01 (Win95; I) c Number of bytes returned to client e The request's query string, format: key=value&key=value f The HTTP REFERER field, format: http://www.domain.com/path/source.ext?query h Client host name or IP address k Response code as a string (OK or ERR) m Authenticated realm (often just - if none) n (WebSTAR) Client host name if DNS lookup enabled; otherwise just - p (WebSTAR) Client host IP address if DNS lookup disabled; otherwise just - q Full HTTP request line in format: GET /url HTTP/1.0 r Result code as a numeric string (eg, 200) u URL requested (with query string), format: /path/path/file.ext?query v The string "W3SVC" indicating an IIS 3 log file format w Authenticated user (often just - if none) x Base portion of URL (no query string), format: /path/file.ext ? Ignore this field (used for any other field type)

Using The LogFormat Parameter

---

If you need to specify a custom log format, you will usually do so in the FlashStats configuration file. When specified on a line all by itself, this parameter applies to all log files for all user accounts. If necessary, an individual user account can be overridden with a different value by including a LogFormat= parameter in the user account definition line. This usually isn't necessary.

The LogFormat= parameter can also be passed in from a form, but there usually is no reason to do it this way.

Version 1.5 -- Last update: 2/8/01

---