Raw Logs (paid feature). Export logs to your storage

What is Raw Logs?
Export to an S3 storage 
    • Amazon storage
    • Non-Amazon storage
Export to an FTP/SFTP storage
Export time intervals
Log path example
Log format
Log example
Log fields

What is Raw Logs?

Raw Logs is an option that enables an automatic export of CDN resource logs to your storage. Logs contain information about user requests sent to cache servers and pre-cache servers (if origin shielding is enabled).

The feature is paid. To activate, contact us via support@gcorelabs.com. After activation, enable Raw Logs in your control panel and configure export to S3, FTP or SFTP storage.

image_1375.png

Export logs to an S3 storage

Amazon storage

1. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

2. For storage type, select "Amazon".

3. Specify your access key ID. In your Amazon personal account, it is called "AWS access Key ID". You can find it using the instruction. An access key ID and secret access key are required to configure log export to your storage.

4. Specify your secret access key. In your Amazon personal account, it is called "AWS secret access Key". You can find it using the instruction.

5. Specify your AWS region — location of a server where your storage is hosted. This is optional: for most storages, the region is determined automatically. You can leave the field empty. But we recommend filling it out so to ensure that your logs are exported successfully.

6. Choose how to organize storage: to put logs of all CDN resources into one bucket or to use separate buckets for each CDN resource.

7. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. If you want to export logs to a specific folder within a bucket, specify a folder name.

8. Click "Save changes".

image_1376.png

Non-Amazon storage

1. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

2. For storage type, select "Other".

3. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use G-Core Labs S3 storage, you can find its access key ID in your personal account in the "Hostname" field.

4. Specify your access key ID. Along with a secret access key, it is required to configure log export to your storage. If you use G-Core Labs S3 storage, you can find its access key ID in your personal account in the "Access key" field.

5. Specify your secret access key. If you use G-Core Labs S3 storage, you can find its secret access key in your personal account in the "Secret key" field.

6. Specify a bucket hostname — a bucket ID that is used by your S3 storage system in the {bucket_name}.{hostname} format. It is required to ensure that logs are exported to a correct bucket within a storage. This field is optional: for some storages, a bucket hostname is determined automatically. If you use G-Core Labs or Yandex.Cloud storage, a bucket hostname is required. A bucket hostname of the G-Core Labs storage looks as follows: {bucket name}.{hostname from step 3}. For example: examplename.s-ed1.cloud.gcore.lu. A bucket hostname of Yandex.Cloud storage looks as follows: {bucket name}.{Yandex.Cloud hostname}}. For example: examplename.storage.yandexcloud.net.

5. Specify a region — location ID of a server where your storage is hosted. This is optional: for some storages, the region is determined automatically. You can leave the field empty. If you use G-Core Labs S3 storage, a location ID is required. You can find it in the "Details" of the storage. Your location ID is a part of your hostname to the first dot.

image_1377.png

7. Choose how to organize storage: put logs of all CDN resources into one bucket or to use separate buckets for each CDN resource.

8. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. If you want to export logs to a specific folder within a bucket, specify a folder name.

9. Click "Save changes".

image_1378.png

Export logs to an FTP/SFTP storage

1. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

2. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use G-Core Labs SFTP storage, you can find its hostname in the "Details" of the storage in the "Hostname" field. It looks as follows: drf.origin.edgecore.ru . Additionally, you can specify an FTP or SFTP storage port by adding a colon after the hostname. For example: mia.origin.gcore.lu:2200.

3. Specify a storage username. If you use G-Core Labs SFTP storage, you can find its username in the "Details" of your storage in the "Storage/User name" field.

4. Enter your storage password.

5. Specify a folder for export. If you use G-Core Labs SFTP storage, specify the root (home) folder where other folders originate from. You can find its name in the "Details" of your SFTP storage at the end of the "Upload path" field.

image_1379.png

If you use an SFTP storage from another provider, clarify whether a root folder that includes other folders is created by default. If not, leave the field empty. If yes, specify a folder name.

6. Choose how to organize storage: put logs of all CDN resources into one folder or to use separate folders for each CDN resource. Then specify a folder name. If you specify a non-existent folder, logs will be exported to a root folder.

7. Click "Save changes".

image_1380.png

Export time intervals

Logs are exported at the end of each hour. If you activate Raw Logs at 00:30, the first logs will be exported between 00:45 and 01:00, the next ones — between 01:45 and 02:00.

If CDN servers are not requested, an empty log file (± 20 bytes) will be sent to your storage.

You can see the status of the Raw Logs option in your control panel:

  • "Pending" is a status for the time interval between the connection to a storage and the very first log export
  • "OK" is a status showing that logs are exported from at least one CDN server
  • "Failed" is a status indicating that an error occurred while connecting to a storage or that the service failed to export logs within 24 hours
  • "Pause" is a status showing that the option is paused

image_1381.png

Log path example

s3://log-bucket-name/2019/08/20/15/nodename_primarycname.domain.ru_access.log.gz

Log format

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"
"$edgename" "$scheme" "$host" "$request_time"
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"
"$geoip_city_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"
"$request_id" "$http_x_forwarded_proto" "$http_x_forwarded_request_id" "$ssl_cipher"
"$ssl_session_id" "$ssl_session_reused"
"$sent_http_content_type" "$tcpinfo_rtt"

Please don’t be surprised if you see a field that is not listed above. We occasionally add new fields. If some fields are added in logs you will receive an email about it. New fields are added to the end of the line.

Log example

"0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1"
"200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36"
"1514848" "[dh-up-gc18]" "https" "origin.cdn.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]"
"MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000"
"0.200" "0.0.0.0" "asnumber" "106980391" "1" "c1c0f12ab35b7cccccd5dc0a454879c5" "-" "-"
"ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"
"application/json" "21"

Log fields

Not all fields are important. Some of them relate to our internal CDN system and are not meaningful for you. In the table below, we have highlighted such system fields in italics. Other fields can be helpful for traffic analysis or statistics. 

Field Log value example

Description

$remote_addr

0.0.0.0

User's IP address

$remote_user

(internal system variable)

-

Username used in Basic authentication 

[$time_local]

[26/Apr/2019:09:47:40 +0000]

Local time in Common Log Format

$request

GET /ContentCommon/images/image.png HTTP/1.1

HTTP method, requested file path, HTTP version

$status

200

Response status code from a CDN server

$body_bytes_sent

1514283

Number of bytes sent to a user, excluding the response header size

$http_referer

https://example.com/videos/10

Referrer - a URL requested by a user

$http_user_agent

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36

User agent that was used to send a request (browser or other application)

$bytes_sent

1514848

Number of bytes sent to a user

$edgename

[dh-up-gc18]

CDN server that forwarded a requested file

$scheme

https

Protocol (HTTP or HTTPS) of a request

$host

cdn.example.com

Requested hostname of a CDN resource

$request_time

1.500

Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request were processed and logging after the last bytes were sent to a user

$upstream_response_time

0.445

Number of seconds (accurate to milliseconds) it took to receive a response from an origin. In case of multiple responses, commas and colons are used 

$request_length

157

Request length (including request line, header and request body)

$http_range

bytes=0-1901653

File fragment size in a Range request

[$responding_node]

dh

Responding data center

$upstream_cache_status

MISS

Status of a requested file in CDN cache:

HIT is a status of a response served from CDN cache.

STALE is a status of an outdated response that failed to update because an origin was not responding or responding incorrectly.

UPDATING is a status of an outdated response that is still updating since a previous request.

REVALIDATED is a status of a response that is identical to the one on an origin based on the proxy_cache_revalidate directive.

EXPIRED is a status of a response that has expired in cache, but still matches the one on an origin. A request has been sent to an origin for re-caching.

MISS is a status of a response that has been served directly from an origin, rather than from cache

$upstream_response_length

 10485760

Response length from an origin in bytes. In case of multiple responses, commas and colons are used

$upstream_addr

0.0.0.0:80

Origin's IP address and port

$gcdn_api_client_id

(internal system variable)

123

Your ID in our system

$gcdn_api_resource_id

(internal system variable)

01

Your CDN-resource ID in our system

$uid_got

(internal system variable)

-

Cookie name and received user ID

$uid_set

(internal system variable)

-

Cookie name and provided user ID

$geoip_country_code

KZ

User’s country code

$geoip_city

-

User’s city code

$shield_type

(internal system variable)

shield_no

This field shows whether the shielding option is enabled:

shield_old - enabled

shield_no - disabled

$server_addr

(internal system variable)

0.0.0.0

IP address of an Anycast zone or CDN server

$server_port

(internal system variable)

80

Requested port

$upstream_status

206

Origin response code

$upstream_connect_time

0.000

Number of seconds (accurate to milliseconds) it took to access an origin server

$upstream_header_time

0.200

Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server

$shard_addr

(internal system variable)

0.0.0.0

IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled

$geoip2_data_asnumber

asnumber

Number of an autonomous system that sent a request

$connection

(internal system variable)

2897494295

Connection serial number

$connection_requests

(internal system variable)

1

Current number of requests made through a connection

$request_id

(internal system variable)

c1c0f12ab35b7cccccd5dc0a454879c5

Unique request identifier generated from 16 random bytes, in hexadecimal form

$http_x_forwarded_proto

-

Initial protocol of an incoming request (HTTP or HTTPS)

$http_x_forwarded_request_id

(internal system variable)

-

Initial ID of an incoming request

$ssl_cipher

(internal system variable)

ECDHE-RSA-AES256-GCM-SHA384

Cipher name used for an established SSL connection

$ssl_session_id

(internal system variable)

28a4184139cb43cdc79006cf2d1a4ac93bdc****

Session ID of an established SSL connection

$ssl_session_reused

(internal system variable)

 r

The filed shows whether a session was reused (“r” ) or not (“.”)

$sent_http_content_type

application/json

Value of the Content-Type HTTP header, indicating the MIME type of a transmitted file

$tcpinfo_rtt

21

Time required to transmit a complete TCP segment: from the first bit to the last one

 

Was this article helpful?
Recently viewed articles