Raw Logs: set up sending of logs associated with your CDN resources (Paid option)

It is a paid option

You can receive CDN resource logs from CDN servers and shielding (if it is activated).

Contact us via chat or email to support@gcorelabs.com to enable the option.

Once the feature is enabled you will be able to set it up in your personal account->CDN section->Raw logs->Receive raw logs.

____________________1_eng.png

We use two protocols to deliver logs: S3 and FTP.

You should set up a storage on your side to start receiving the logs.

 

Configure logs receiving on S3

Configure logs receiving on FTP

Log path example

Log format

Example log

Log Fields

Configure logs receiving on S3

Storage is Amazon 

If you want to receive logs to your Amazon storage, specify AWS Access Key ID, AWS Secret Access Key and Region (optional).

Check the box Add logs from origin shielding to receive logs from CDN servers and shielding.

______________eng.png

Select where you want to send logs:

  • into one folder

___________.png

  •  into different folders

_____________.png

Note! The system doesn't create folders (bucket) in your storage. Specify an existing folder (bucket).

Click Save Changes.

 

Storage is not Amazon  

If you choose Other, fill in Hostname, Access Key ID and Secret Access Key.

If you use Yandex.Cloud or our S3 storage, fill in Bucket hostname.

Check the box Add logs from origin shielding to receive logs from CDN servers and shielding.

______________s3________eng.png

Select where you want to send logs:

  • into one folder

___________.png

  •  into different folders

_____________.png

Note! The system doesn't create folders (bucket) in your storage. Specify an existing folder (bucket).

Click Save Changes.

Configure logs receiving on FTP

Specify Hostname, Login, Password and Prepend folder (optional).

Note! A port number can be added to your Hostname if needed: <hostname>:<port>.

Check the box Add logs from origin shielding to receive logs from CDN servers and shielding.

______________ftp_eng.png

Select where you want to send logs:

  • into one folder

___________.png

  •  into different folders

_____________.png

Note! The system doesn't create folders (bucket) in your storage. Specify an existing folder (bucket).

Click Save Changes.

 

Log path example 

 s3://log-bucket-name/2019/08/20/15/nodename_primarycname.gcdn.co_access.log.gz 

Log Format

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status" 
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"
"$sent_http_content_size" "$scheme" "$host" "$request_time"
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"
"$geoip_city_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"
"$request_id" "$http_x_forwarded_proto" "$http_x_forwarded_request_id" "$ssl_cipher"
"$ssl_session_id" "$ssl_session_reused";

The log format can be changed at any time. New fields are always added to the end of the line. When analyzing logs, only the known fields from the beginning of the line should be taken into account.

Example Log

"0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1" "200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36" "1514848" "-" "https" "origin.cdn.com" "1.500" "0.445" "157" "-" "[kal]" "MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000" "0.200" "0.0.0.0" "asnumber" "106980391" "1" "c1c0f12ab35b7cccccd5dc0a454879c5" "-" "-" "ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"

Log Fields

Field Example log value

Description

$remote_addr 0.0.0.0

IP address of the client

$remote_user -

User name supplied with the Basic authentication

[$time_local] [26/Apr/2019:09:47:40 +0000]

Local time in the Common Log Format

$request

GET /ContentCommon/images/image.png HTTP/1.1

HTTP request type, requested path to a file, HTTP version
$status

200

Response status
$body_bytes_sent

1514283

Number of bytes sent to a client, disregarding the response header
$http_referer

https://example.com/videos/10

Referrer: specify the URL of the user

$http_user_agent

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36

User Agent which a user applied to request content
$bytes_sent

1514848

Number of bytes sent to a client
$sent_http_content_size -

Value of the HTTP Content-Size header

$scheme https

Request's protocol (HTTP ot HTTPS)

$host cdn.example.com

CDN-resource's Hostname received from a request

$request_time 1.500

Request processing time counted in seconds with a milliseconds resolution; time elapsed from the first bytes sent by the client till the log record, made after a client sent last bytes

$upstream_response_time 0.445

Time spent on receiving the response from the upstream server; time counted in seconds with millisecond resolution. Time of several responses is shown in commas and colon

$request_length 157

Request length (including request line, header and request body)

$http_range -

File fragment value received by Range request

[$responding_node] kal

The name of the responded CDN-node

$upstream_cache_status MISS

CDN cache status of a requested file:

HIT –  file's given from the cache

STALE – file outdated as the origin server didn't respond or responded incorrectly due to cache's being updated

UPDATING - file outdated as it's being updated after the earlier request was sent

REVALIDATED – the proxy_cache_revalidate directive was enabled and NGINX verified that the currently cached content is still valid

EXPIRED - cache lifetime ended, but the file is relevant to the origin's file. A request is sent to the origin server for the recache

MISS – file isn't given from the cache, it's proxied from the origin server

$upstream_response_length  10485760

Length of the response obtained from the origin server; stored in bytes. Lengths of several responses are separated by commas and colons

$upstream_addr 0.0.0.0:80

IP address and port of the prigin server

$gcdn_api_client_id 123

Your ID in our system

$gcdn_api_resource_id 01

Your CDN-resource ID in our system

$uid_got -

Cookie name and received user ID

$uid_set -

Cookie name and provided user ID

$geoip_country_code KZ

User’s country code

$geoip_city -

User’s city code

$shield_type shield_no

Specifies if a shield is enabled or not

shield_old - enabled

shield_no - disabled

 

$server_addr 0.0.0.0

IP address of anycast zone or edge-server

$server_port 80

Port of the server which accepted a request

$upstream_status 206

Origin server's response code

$upstream_connect_time 0.000

Time spent to access the origin server; time counted in seconds with millisecond resolution.

$upstream_header_time 0.200

Time spent to receive a header from the origin server;  time counted in seconds with millisecond resolution.

$shard_addr 0.0.0.0

IP address of an edge-server that received the first client request if the Cache Sharding feature is enabled.

$geoip2_data_asnumber asnumber

The autonomous system number where the user request came from.

$connection_requests 1

Current number of requests made through a connection.

$request_id c1c0f12ab35b7cccccd5dc0a454879c5

Unique request identifier generated from 16 random bytes, in hexadecimal.

$http_x_forwarded_proto -

Initial protocol of the incoming request (http or https).

$http_x_forwarded_request_id -

Initial ID of the incoming request.

$ssl_cipher ECDHE-RSA-AES256-GCM-SHA384

Returns the name of the cipher used for an established SSL connection.

$ssl_session_id 28a4184139cb43cdc79006cf2d1a4ac93bdc****

Returns the session identifier of an established SSL connection.

$ssl_session_reused  r

Returns “r” if an SSL session was reused, or “.” otherwise.

 

 

Was this article helpful?
Recently viewed articles