Lifecycle policy. Delete objects from the bucket automatically

Our S3 storage supports lifecycle policy.

Lifecycle configuration is an XML file with a set of rules which define files expiration time. Once the files expire, they are removed from the storage.

There are different ways to set lifecycle configuration on a bucket. Read more on them here.

Lifecycle configuration elements

Lifecycle configuration is an XML file:

<LifecycleConfiguration> 
<Rule>
...
</Rule>
</LifecycleConfiguration>

Inside the <Rule>,</Rules> tags the following elements should be specified:

  • Metadata: ID and status,
  • Prefix (leave empty since the policy is applied to the content of the bucket),
  • Action (file expiration time).

Rule ID

Each rule must have a unique identifier. ID length is limited to 255 characters. Numbers, Latin letter and underscores are allowed.

<ID>rule_identificator</ID>

Rule status

The status value can be either Enabled or Disabled. If a rule is Disabled, no actions specified in the rule will be performed.

<Status>Enabled</Status>

Prefix

The policy is applied to the content of the entire bucket so you can leave the Prefix tag empty.

<Prefix/>

Actions

Specify the number of days the file should be kept in your bucket.

<Expiration>
<Days>1</Days>
</Expiration>

Example: Adding lifecycle configuration with aws cli

Create an XML file:

  • ID — one_day,
  • the rule is applied to the entire bucket (<Prefix/>),
  • the rule is active (status Enabled),
  • the file expiration time is 1 day.
<LifecycleConfiguration>
<Rule>
<ID>one_day</ID>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>1</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>

If you are using aws cli to add configuration to your bucket, you won't be able to upload an XML file so you should convert it to JSON first (save it as lifecycle.json):

{
"Rules": [
{
"ID": "one_day",
"Prefix": "",
"Status": "Enabled",
"Expiration": {
"Days": 1
}
}
]
}

Important! The following header's value ("Prefix": "") applies a lifecycle configuration to a whole bucket. If you need to apply the policy to a specific folder in a bucket, specify the folder. For example, if files are in deleteme / [files ...] folder, the value will be: "Prefix": "deleteme /"

Start the aws cli from a directory with the lifecycle.json file and run the command below. Don't forget to replace my_bucket name in the example with your bucket name. To choose the correct value for the --endpoint-url parameter use this article.

$aws s3api put-bucket-lifecycle --bucket my_bucket --lifecycle-configuration file://lifecycle.json --endpoint-url=https://s-ed1.cloud.gcore.lu

To check if the policy is uploaded correctly use the following command:

$aws s3api get-bucket-lifecycle-configuration --bucket my_bucket --endpoint-url=https://s-ed1.cloud.gcore.lu

In the response you should get the uploaded JSON file:

{
"Rules": [
{
"Expiration": {
"Days": 1
},
"ID": "one_day",
"Prefix": "",
"Status": "Enabled"
}
]
}

To remove the policy from the bucket use this command:

aws s3api delete-bucket-lifecycle --bucket my-bucket --endpoint-url=https://s-ed1.cloud.gcore.lu

Lifecycle policy logic

The file removal process starts around midnight (UTC timezone).

Lifecycle policy is set after the files are uploaded

If you upload a file on January, 1st (any time), then set lifecycle policy with a file expiration time of 1 day, the file will be deleted on January, 3rd at around midnight.

Lifecycle policy is set before the files are uploaded

If the lifecycle policy is already set for 1 day and you upload your file on January, 2nd, the file will be removed on January, 4th at around midnight.

Lifecycle policy is removed

If you delete the lifecycle policy from the bucket, the files it was applied to won't be removed.

Was this article helpful?
Recently viewed articles