Coffee Data, Decoded: What Your Café Can Learn from Spotify's RAP

The Data Problem at Your Local Café

Think about a coffee shop that keeps tabs on every bean, every brew, every customer's go-to order. You want to know Sarah's usual—a cortado with oat milk—or that Ethiopian Yirgacheffe flies off the shelves on weekends, or that the espresso machine's had 2,000 shots and needs a checkup. But pulling up Sarah's order history shouldn't mean wading through months of sales data. That's the kind of headache Spotify just tackled with its Random Access Parquet (RAP) architecture—and it's a trick for any coffee business drowning in numbers.

Why Your Data Lake Feels Like a Swamp

Data lakes are great for the big picture. You can run a huge query to spot trends—like which beans are hot in your city—or feed a machine learning model to guess tomorrow's pastry sales. But when you need one specific record, like a single order or a customer profile, the same system crawls. Query engines like Trino and BigQuery are built for scanning, not for plucking a needle from a haystack. So you end up copying data into a separate operational database just to handle that one quick lookup.

Spotify hit this exact wall. They had petabytes of data in Bigtable for online services and exabytes in a data lake on Google Cloud Storage. Copying all that to another system was getting absurdly expensive. RAP is their answer: an external index that maps a key—like a user ID—straight to a specific Parquet file and row position. The query engine checks the index, grabs the data, reads a few kilobytes. No full scan, no copy.

Bringing It Home: Your Coffee Shop

You might not have exabytes of coffee data, but the principle holds. Say you keep all your sales, inventory, and loyalty info in a data lake on S3 or Google Cloud Storage. You want to run broad analytics—like which drinks sell best in the afternoon—and also answer quick questions: "What did that regular customer order last week?" or "How many bags of that new single-origin do we have left?"

Usually, you'd have to maintain a separate database for those fast lookups, duplicating data and risking inconsistency. With something like RAP, you could add an index on top of your existing files, serving both analytics and quick queries from the same source. No syncing, no extra storage costs.

The Index: Your Coffee Roast Log, But Automated

Think of the index as a well-organized roast log. Instead of digging through every bag in the warehouse to find a roast date, you have a card catalog that says exactly which shelf and which row it's on. RAP stores the mapping between keys and file positions. When new data arrives—say, a batch of orders—the index builder appends without touching the immutable Parquet files. Like adding new cards to your catalog without rewriting the old ones.

This keeps your existing data intact. You can still run your usual analytics, train your ML models, and let your AI assistants query the same data. But now you can also serve low-latency requests to your mobile app or POS system, all from the same lake.

Speed Tricks: How to Make Point Queries Zip

Spotify didn't stop at the index. They also fiddled with physical data layout to make point queries even faster. One trick is sorting data by the query key—say, customer ID—so a single customer's records are stored together. That way, a point query only reads a few files instead of hundred.

Another trick is interleaving value columns. Instead of storing all "drink type" values together and all "milk type" values together, you store them side by side for each record. So when you fetch Sarah's order, you get her drink and milk preference in one continuous read. It's like keeping beans and grinder settings in the same drawer—no rummaging through the whole kitchen.

Covering indexes are another option. If a query only needs a few fields, the index itself can provide them without touching the Parquet file at all. Like a quick-reference card that lists roast date and origin without opening the bag. These tricks trade a bit of extra storage for much faster reads—sometimes just a single range request covering a few kilobytes.

Multiple Paths: Secondary Indexes

What if you sometimes search by customer ID, other times by product ID? Spotify's RAP supports secondary indexes, so you can add new access paths without rewriting data files. You can have a hash index for exact matches—like "find customer 12345"—and a sorted index for range queries—like "all orders between these dates."

That's huge for a coffee shop. You might look up a customer by phone number, or find all orders for a specific roast. With secondary indexes, you can add these lookup paths on the fly without changing your data pipeline. The indexes live at the service layer, so Parquet data stays untouched for analytics.

Fancy Stuff: Z-Ordering and Hilbert Curves

For multidimensional queries—like "all orders for this customer in this date range"—you can use spatial layout tricks like Z-ordering or Hilbert curves. These methods cluster data that's close in multiple dimensions, so a single query touches fewer files. It's like organizing beans by both roast date and origin, so you can quickly find a batch roasted last week from Colombia.

These techniques aren't just for tech giants. Open-source tools and cloud services are increasingly supporting them. You can apply similar ideas to your own data lake, even without building a custom index from scratch. The key is to think about how your data will be queried, not just how it'll be analyzed.

What's Next: One Data Lake to Rule Them All

Spotify's RAP is part of a bigger push: making data lakes serve not just analytics, but also online apps and AI agents. Google Cloud has been pushing a similar vision with Apache Iceberg-based lakehouses. The goal is one copy of data, used for everything, without the cost and complexity of multiple systems.

For coffee businesses, that means a single source of truth for beans, sales, customers, equipment. Analytics team runs big queries, mobile app fetches a customer's favorite drink in milliseconds, AI learns from the same data. No more syncing between data warehouse and transactional database.

Tech is still evolving, but the direction is clear. Whether you're a small café with a spreadsheet or a national chain with a data team, understanding the idea of an index layer on top of your data files is worth it. Could save money, reduce complexity, make your data work harder.

Start Brewing: Practical Steps

So what can you do today? Look at your own data setup. Are you duplicating data just to serve fast queries? Could you add an index to your data lake instead? Even if you're not ready to build a custom RAP, experiment with tools like Apache Iceberg and Parquet—they're open-source and widely supported.

Think about your query patterns. Do you need to look up individual customers, products, orders? If so, consider sorting data by those keys and adding indexes. You might be surprised how much faster your apps feel and how simpler your infrastructure becomes.

Spotify's RAP is a technical fix, but its lesson is simple: with a little clever indexing, you can have your data and drink it too. And in the coffee world, that's worth a toast.