Module 9 closes with the same shape that Module 5 and Module 8 closed with: one company, one decade of published engineering posts, a set of patterns that transfer to teams orders of magnitude smaller. This time the company is Pinterest, and the topic is cost. Pinterest has run a multi-year cost-reduction programme on their data platform, written about it openly on their engineering blog, and produced one of the cleanest public examples of how a large data org actually cuts a cloud bill in half without tearing the platform down.
The lessons map onto the levers Module 9 has covered: storage tiering (lesson 65 in this module’s plan), compute right-sizing and instance choice (lesson 66), query optimisation (lesson 67), file format and partitioning (lesson 68), FinOps and chargeback (lesson 69), and the negotiation-and-rebuild meta-question (lesson 71). Pinterest pulled most of them at once, in roughly that order, and the published numbers add up to savings in the high tens of millions of dollars annually. The architectural lesson is that none of the individual levers is novel; it is the discipline of applying them as an ongoing programme rather than a one-time cleanup that produces the result.
The scale and the starting point
Pinterest’s published numbers shift over time. The rough order of magnitude as of the mid-2020s: around five hundred million monthly active users, a multi-petabyte data warehouse on Amazon S3, tens of thousands of batch jobs per day, and hundreds of data engineers, analysts, and data scientists interacting with the platform. Annual data infrastructure spend ran into the tens to low hundreds of millions of dollars, with AWS as the primary cloud provider.
The pressure that started the programme is the pressure every fast-growing company faces. The data platform was growing faster than the rest of the business. New use cases (recommendations, ads, content moderation, experimentation) drove new pipelines, which drove more storage, more compute, and more queries. The bill grew super-linearly with revenue at exactly the moment the company needed margin discipline. Cost optimisation went from “something the platform team should think about” to “a strategic priority with executive attention”.
The Pinterest engineering blog (https://medium.com/pinterest-engineering, retrieved 2026-05-01) carries the public account of the programme. The posts cluster around storage optimisation on S3 and Iceberg, Spark cost-efficiency work, query and pipeline optimisation, and Graviton-based migration. The exact savings figures are not always disclosed to the dollar, but the directional claims are explicit: roughly half the data-infrastructure cost was taken out over the course of the programme, with individual projects contributing tens of percent each.
Storage tiering and compaction
The first and largest tranche of savings came from storage. Multi-petabyte data lakes accumulate two pathologies that compound over time. Cold data sits on the same expensive storage tier as hot data, even though it is read once a quarter, if ever. And data lands on S3 as many small files, because that is how streaming and Spark’s default output behave, and small files are expensive to operate on at scale.
Pinterest’s storage work, described in the engineering blog posts on Iceberg adoption and S3 cost optimisation, attacked both. The headline moves were:
Tiering by access pattern. Tables and partitions with old, rarely-read data were moved from S3 Standard to S3 Infrequent Access and Glacier tiers, where the per-gigabyte storage cost is a fraction of Standard. The catch is that Glacier introduces retrieval latency and per-request fees, so the tiering policy has to be informed by actual access logs, not guesswork. Pinterest built the access tracking, then tuned the policy so the tiering moved data only after the access pattern justified it. This is conceptually simple and operationally fiddly, which is why most teams skip it until the bill forces the issue.
Small-file compaction. A pipeline that emits ten thousand small Parquet files per partition produces a folder that is expensive to list, expensive to query, and expensive to read. Pinterest’s compaction layer (running on top of Apache Iceberg, which they adopted heavily, partly drawing on Netflix’s open-sourced project covered in lesson 40) periodically rewrites small files into fewer, larger ones. The savings came from two places: fewer S3 GET and LIST requests at query time (request fees are a real line on multi-petabyte workloads), and faster query execution (fewer files means fewer parallel tasks and less coordination overhead).
Iceberg table format adoption. Iceberg’s metadata layer enables partition pruning, snapshot-based reads, and schema evolution that older Hive-style table layouts cannot. The cost effect shows up as fewer files scanned per query (because the planner can prune at metadata level) and better query plans (because statistics are accurate). The format change is infrastructure work that pays back in lower compute cost on every query that runs against the migrated tables.
Lifecycle policies. Old snapshots, orphan files, and historical Iceberg metadata accumulate. Without a deletion policy, they sit on S3 forever. Pinterest’s lifecycle automation cleans them up, and the savings are pure overhead reduction.
The combined storage programme produced double-digit percentage savings on the storage line, and a meaningful additional saving on the compute side because queries got faster. The architectural pattern: treat storage as a system to be operated, not as a passive substrate that just accumulates whatever the pipelines write.
Spark efficiency and Graviton
The compute side of the programme has been described in posts on Pinterest’s Spark efficiency work and the migration to ARM-based AWS Graviton instances.
Right-sizing. Many Spark jobs were over-provisioned. Cluster sizes had been picked by guess, run once, and never revisited. A systematic pass through the largest jobs found that a meaningful fraction were running with two-to-five-times the executor count they needed. Reducing the over-provisioning is conceptually trivial and operationally rare, because nobody has the time or the dashboards to know which jobs are over-provisioned without an explicit programme. Pinterest built the dashboards and ran the programme, and the savings on compute were substantial.
Spot instances. Batch workloads that tolerate interruption (most ETL and most ML feature engineering) can run on AWS Spot at sixty-to-eighty percent off on-demand pricing. The catch is that Spot instances can be reclaimed with two minutes’ notice, so the workload has to be checkpointable or restartable. Pinterest moved a large share of their batch Spark workload to Spot, with the orchestrator (covered in lesson 57 of Module 8) handling retries when capacity vanished. The savings are direct: same workload, much smaller bill.
Graviton (ARM) migration. AWS Graviton instances run on ARM processors and offer roughly twenty percent better price-performance than the x86 equivalents for many workloads. The migration is not free: software has to be rebuilt or repackaged for ARM, performance has to be validated, and edge cases (native libraries, JNI bindings, vendor agents) have to be handled. Pinterest published their experience moving Spark workloads to Graviton, including the gotchas around JVM tuning and native dependencies. The result was a structural cost reduction on every workload that migrated, which is a different kind of saving than tuning a single job: it compounds across the entire fleet for as long as the cluster runs.
Instance type matching. Beyond ARM, the programme included matching instance types to workload shape. Memory-heavy aggregations on memory-optimised instances. Compute-heavy machine-learning feature builds on compute-optimised instances. Storage-heavy scans on instances with local NVMe. The default of “use whatever m5.xlarge gives us” leaves money on the table; matching deliberately reclaims it.
The compute programme delivered the largest single line of savings in the overall effort. The lesson is that compute waste, like storage waste, is rarely visible without a programme designed to expose it.
Query optimisation: the top-N approach
The third tranche came from query optimisation, but with a discipline that many teams miss. In a warehouse with tens of thousands of recurring queries, the cost distribution is sharply skewed. A handful of queries (the top fifty, sometimes the top ten) account for a disproportionate share of the total compute. Optimising the long tail produces small, scattered savings; optimising the head produces visible, cumulative ones.
Pinterest’s approach, described in their engineering posts on warehouse efficiency, was explicitly top-N. Instrument every query with cost attribution. Rank by total cost (frequency multiplied by per-run cost). Take the top fifty. Hand them to a query-optimisation team. Rewrite, repartition, add the right indexes or sort keys, change the join order, push predicates down, prune unused columns, materialise the expensive subqueries. Move on to the next batch.
The mechanics are workmanlike and the results are out of proportion to the effort. A single rewrite of a daily query that scans a petabyte saves the cost of that scan every day, every year, until somebody changes the underlying schema. The compounding is severe in the right direction. The same engineering hours spread across the long tail would have produced a tenth of the savings.
The pattern transfers. Every analytics warehouse has a top-N query list. Every data team that has not run the exercise has waste in the head of that list. The first pass through the top fifty is usually the highest-leverage week of cost work a data engineer can do.
Right-sizing clusters and warehouses
Beyond Spark, Pinterest’s programme touched the always-on systems. Warehouses (Trino, Snowflake-like services), notebook clusters, BI backends, and streaming pipelines all carry baseline cost that runs whether anyone is using them or not. The right-sizing pass on these targeted three things.
Idle scaling. Clusters that ran twenty-four-seven but only saw load for ten hours a day got scaled down or stopped during the idle window. The savings are direct, and the operational risk is low if the scale-up is automated. The systems where this is hardest are the ones with stateful caches that take time to warm; for those, the trade-off is cache warmth against cost, and the answer is workload-dependent.
Concurrency-based sizing. Warehouses sized for peak concurrency that occurs once a day were costing peak prices for twenty-three hours of off-peak. Auto-scaling, where supported, addressed this. Where not supported, scheduled resizing did.
Workload isolation by SLA. Critical interactive queries were separated from batch and ad-hoc workloads, so the interactive cluster could be sized for low concurrency and high responsiveness, while the batch cluster could be sized for high throughput and tolerate queuing. Mixing them forced both to be sized for the worst-case combined load, which is more expensive than the sum of two appropriately sized clusters.
The pattern: pay for what you use, when you use it. Defaults rarely match this; deliberate configuration does.
The kill list
The least technical and most under-appreciated part of the programme is the audit-and-delete pass. Every mature data org has dashboards nobody opens, pipelines that compute outputs nobody consumes, and tables that haven’t been queried in a year. They cost money for nothing, and they accumulate because creating them is easy and deprecating them is somebody else’s job.
Pinterest’s programme included an explicit kill list. Dashboards with no views in ninety days got flagged. Pipelines whose outputs had no downstream readers got flagged. Tables with zero queries over an audit window got flagged. The flags went to owners, who either justified the asset or agreed to its deletion. The deletion phase reclaimed real money, both directly (the compute and storage of the dead asset) and indirectly (the operational cost of monitoring and maintaining it).
The pattern is universal. Any team that has been running for more than two years has assets that nobody uses and nobody has gotten around to deleting. The kill list is the cheapest cost-saving exercise on the platform, and it is the one most teams skip.
Cost visibility and chargeback
The cultural underpinning of the whole programme is visibility. None of the individual savings are possible without dashboards that show, by team and by product, what the data platform is costing. Without those dashboards, every cost discussion is anecdotal, every prioritisation is political, and the team that volunteers to optimise carries the burden alone.
Pinterest, like most large data orgs, built dashboards that attribute compute, storage, and request cost to the teams that generated it. The attribution model can be chargeback (the team’s budget actually pays for what they used) or showback (the team sees what they used but the platform team pays). Pinterest’s specific model is internal; the public posts emphasise the visibility itself, not the financial mechanics.
The effect is what economists call internalising the externality. When a team can see that their dashboard is costing the company a hundred thousand dollars a month, they tend to ask whether the dashboard is worth that. They tend to optimise the queries, partition the tables better, archive the old data, and delete the unused assets. The platform team’s role shifts from being the only optimiser to being the enabler of optimisation by every team.
The cumulative result
The published claim is that the multi-year programme cut data infrastructure cost roughly in half, against a counterfactual of unconstrained growth. In absolute terms, this is a savings figure in the high tens of millions of dollars annually, depending on the period and the baseline. The savings are persistent: the storage tiering, the Graviton migration, the kill list, and the right-sizing all keep paying as long as the systems run.
The cumulative effect matters more than any individual line. None of the levers is novel. All of them are documented in cloud-vendor whitepapers, in open-source project documentation, and in the public posts of half a dozen other companies. What Pinterest did was apply them systematically, as an ongoing programme rather than a one-time clean-up, and the compounding produced the headline result.
The lessons
Five takeaways, in the same shape Module 5’s Netflix case (lesson 40), Module 6’s Uber case (lesson 48), and Module 8’s Airbnb case (lesson 64) used.
Cost optimisation is its own engineering discipline. Treating it as a project misses half the savings. Treating it as an ongoing programme, with dedicated owners, dashboards, and a quarterly review cadence, produces compounding results. Pinterest’s published account is, more than anything else, an account of the discipline, not the individual techniques.
The 80/20 rule applies hard. Most savings come from a small number of high-leverage changes. Top-N query rewrites, the largest cluster’s right-sizing, the biggest table’s tiering. The rest is incremental. A team starting from scratch should focus on the few largest opportunities and accept that the long tail will take longer to harvest.
Make cost visible by team and by product. Dashboards are the precondition for everything else. Without them, optimisation is anecdote and politics. With them, it is engineering. The dashboards are not glamorous to build, and they are the highest-leverage piece of FinOps infrastructure most teams can ship.
The kill list of jobs nobody runs is real money. Every data org has dashboards, pipelines, and tables that nobody uses. Audit them, flag them, delete the ones that cannot justify themselves. This is the cheapest tranche of savings, and it is the one most teams have not run.
Spot and Graviton are real money when workloads tolerate them. Both technologies are mature in 2026. Spot is a sixty-to-eighty percent discount on batch compute, gated by checkpointability. Graviton is a structural twenty-percent improvement, gated by software portability. The migration costs are real and one-time; the savings are recurring and compound across every workload that runs on the migrated infrastructure.
Cross-references back into Module 9
The Pinterest programme exercises every lever Module 9 introduced. Storage tiering and compaction are the operational form of the storage layout patterns from earlier in the module. Spark efficiency, Spot, and Graviton are the compute right-sizing levers. Top-N query rewrites are query optimisation in its most leveraged form. Cost dashboards and chargeback are the FinOps culture that makes the rest sustainable. The kill list is the asset-lifecycle hygiene that closes the loop. Lesson 71’s build-versus-buy meta-question is the framing under which a programme like this gets approved at all: when the bill is large enough that optimising it is worth a dedicated team’s time.
The shapes transfer to teams running a hundredth the workload. A small data team with a six-figure monthly bill has the same levers, in the same proportions, just with smaller absolute numbers. The discipline is the same; the savings curve is the same; the cumulative effect is the same.
Module 9 closes here
Module 9 has been about making the data platform sustainable on the cost axis. Storage layout, compute right-sizing, query optimisation, format choices, FinOps culture, vendor relationships, and the cumulative programme that ties them together. A platform that is reliable but expensive will eventually be cut. A platform that is cheap but unreliable costs the business more than it saves. The two axes are linked, and the techniques in this module are how a mature team holds both.
Module 10 opens by stepping out of the data platform and into the broader question of how a system divides responsibility across services. The first lesson covers microservices: the term, the actual benefits, the costs that the early advocates underweighted, and the pendulum that has swung back toward modular monoliths in many of the cases where microservices were once the default answer. The architectural decisions look different from the data-platform decisions of the last nine modules, but the underlying logic is the same: every choice is a trade-off, and the job is to know which axis matters for the system in front of you.
Citations and further reading
- Pinterest Engineering blog,
https://medium.com/pinterest-engineering(retrieved 2026-05-01). The collection of public posts on data platform cost reduction, Iceberg adoption, Spark efficiency, and Graviton migration that this lesson summarises. - Pinterest Engineering, posts on data platform efficiency and cost optimisation (retrieved 2026-05-01). The directional savings claims and the breakdown of the programme into storage, compute, query, and lifecycle phases come from this body of work.
- Apache Iceberg project,
https://iceberg.apache.org/(retrieved 2026-05-01). Documentation on partition pruning, snapshot expiration, and small-file compaction, the underlying primitives Pinterest’s storage programme relies on. - Apache Spark documentation,
https://spark.apache.org/docs/latest/(retrieved 2026-05-01). Reference for the configuration and tuning levers (executor sizing, shuffle behaviour, dynamic allocation) that the Spark efficiency work touches. - AWS Graviton documentation,
https://aws.amazon.com/ec2/graviton/(retrieved 2026-05-01). Vendor reference for the price-performance claim and the migration considerations relevant to the Graviton tranche. - AWS S3 storage classes documentation,
https://aws.amazon.com/s3/storage-classes/(retrieved 2026-05-01). The pricing tiers and retrieval semantics that the storage tiering programme operates on. - AWS Spot Instances documentation,
https://aws.amazon.com/ec2/spot/(retrieved 2026-05-01). The pricing and interruption semantics that govern the Spot migration.