23.2 TB of Inactive S3 Data: A Storage Ownership Case Study

Summary

• Client: A media and entertainment company with a large Amazon S3 footprint (anonymized).
• Problem: S3 charges had reached $12,350 a month, and the content teams were sure they needed everything they stored.
• Finding: 23.2 TB in S3 Standard had not been accessed in more than two years. Most of it belonged to identifiable teams, but 3.2 TB had no clear owner.
• Outcome: After every owning team signed off, the data moved to S3 Glacier Deep Archive, and nothing was deleted. Storage cost for the 23.2 TB fell from about $546 to about $24 a month, saving about $523 a month. Lifecycle policies and ownership requirements now cover all new buckets.

The situation

The client’s Director of Cloud Operations asked us to look at S3 costs. As a media company, they expected heavy storage use, but $12,350 a month seemed high even for them. The content teams insisted they needed all of it. The request was to find out what could change without disrupting production workflows.

The S3 estate:

• 156 buckets across two regions
• 97.4 TB of stored data
• $12,350 a month in S3 charges
• Three storage classes in use: Standard, Standard-IA and One Zone-IA

There was no archive storage at all, despite the company’s long-term retention needs.

Initial findings

Analysis of the buckets showed four patterns worth investigating:

Storage analysis findings:
- 23.2 TB in S3 Standard with no access for more than 2 years
- 45 buckets with "temp", "old" or "archive" in their names
- No lifecycle policies on 143 of 156 buckets
- 17.6 TB of uncompressed video in S3 Standard

Access patterns and age

Last access for the 23.2 TB:

- Last GET request:  more than 730 days ago
- Last PUT request:  more than 730 days ago
- Last LIST request: more than 90 days ago

Age of the content:

- 8.4 TB:  2 to 3 years old
- 11.7 TB: 3 to 5 years old
- 3.1 TB:  more than 5 years old

Cost (us-east-1 list prices):

Current: 23.2 TB in S3 Standard
- 23,757 GB x $0.023 per GB-month   = about $546/month

Projected: 23.2 TB in S3 Glacier Deep Archive
- 23,757 GB x $0.00099 per GB-month = about $24/month

Savings: about $523/month ($6,275/year)

The inactive 23.2 TB accounted for about $546 of the $12,350 monthly S3 bill, so moving it was never going to transform the bill on its own. What made it worth doing was that nobody knew what much of the data was, or who was responsible for it.

What the data was

A review of object metadata and bucket history sorted the 23.2 TB into three groups:

Completed projects (14.7 TB)

• Marketing campaigns from 2018 to 2020
• Raw footage from completed TV productions
• Former client projects that the company was contractually required to keep

Legacy systems (5.3 TB)

• Database backups from decommissioned systems
• Media libraries from an old content management system
• Retired website assets

Unknown origin (3.2 TB)

• Unlabeled data dumps
• Test data with no clear owner
• Personal backups made by employees, against company policy

Finding the owners

Each group needed a decision from the team responsible for it, so we interviewed the teams that owned the first two groups.

• Marketing confirmed the campaigns were finished. They needed to keep them for reference but accessed them at most once a year, and hadn’t realized they were in the most expensive storage class.
• Production confirmed the footage came from shows delivered years earlier. They needed it only if a client requested a reshoot or remaster, which was rare, and retrieval within 12 hours would be acceptable.
• IT confirmed the system backups were kept for compliance under a seven-year retention policy. They had never needed to restore from them, and there was no reason for them to be in S3 Standard.

Risk assessment

The change was low risk:

1. Importance: the data was needed for compliance and contracts, but rarely accessed.
2. Retrieval time: 12 to 24 hours was acceptable for everything identified.
3. Workflow impact: no active projects used the data.
4. Compliance: S3 Glacier Deep Archive met the seven-year retention requirement.

The plan

Phase 1: Transition (days 0 to 30)

• Create lifecycle policies for all identified data
• Transition in stages: Standard, then Standard-IA, then Glacier Deep Archive
• Record what was transitioned, with its metadata

Phase 2: Policy (days 31 to 60)

• Introduce company-wide storage lifecycle policies
• Require tags for project, owner, expiry date and access requirements
• Build a dashboard showing storage costs by team

Phase 3: Ongoing management (day 61 onwards)

• Quarterly reviews of storage use
• Automated alerts for storage that may no longer be needed
• Showback of storage costs to the owning teams

Execution

• Week 1: Built a complete inventory of the 23.2 TB.
• Week 2: Collected sign-off from every owning team.
• Week 3: Applied lifecycle policies and started the transitions.
• Week 4: Documented the new storage policies and trained the teams.

Results

• Storage cost for the 23.2 TB: from about $546 to about $24 a month
• Savings: about $523 a month, or $6,275 a year
• Data deleted: none
• Time to resolution: 10 hours of investigation, plus three weeks of implementation
• Governance: lifecycle policies and ownership tags required for all new buckets

Contributing factors

Four factors had let the data accumulate:

1. “Storage is cheap” thinking. True compared with on-premises storage, but cloud storage costs still add up.
2. No ownership. Nobody was accountable for what stored data cost.
3. No visibility. Teams didn’t know what their storage cost.
4. No automation. No lifecycle policies moved data to cheaper classes as it aged.

Prevention measures

Implemented with the client:

• Guidance on storage classes, so teams choose the right class from the start
• Required tags for project, owner, expiry date and access requirements
• Team-level cost dashboards for storage
• Default lifecycle policies on every new bucket
• Quarterly cleanup sessions, with dedicated time for teams to review and archive their data

Takeaways

The dollar savings in this case were modest. The more important outcome was ownership: every part of the 23.2 TB was signed off by a responsible team before it moved, and new buckets now come with an owner tag and a default lifecycle policy. Before this work, 143 of the 156 buckets had no lifecycle policy, so data stayed in its original storage class however old it got, and nobody was accountable for what it cost.

No files were deleted. Matching storage classes to how the data was actually used let the company keep everything it needed while paying much less to store the inactive part of it.

Two points are worth checking before a move like this. S3 Glacier Deep Archive has a 180-day minimum storage duration, and lifecycle transitions are charged per object, so buckets with very large numbers of small objects can cost more to transition than they save.

Find unowned data in your own accounts

These 23.2 TB sat in S3 Standard with no access for more than two years, and nobody was accountable for what they cost: 3.2 TB was of unknown origin, and the teams that owned the rest didn’t realize what it was costing in the most expensive storage class. Cloud Sleuth is being built as the ownership and context layer for cloud infrastructure, around three questions: who owns this resource, what is it actually for, and is it allowed to exist in its current state? The Cloud Sleuth CLI will support AWS and Azure from launch: it will run on your machine or in your VPC, with nothing leaving your environment, and show unowned resources, common orphans and basic allocation gaps in your own accounts. Both the local tool and Cloud Sleuth Cloud are pre-launch. Get early access.