How to setup duration based profiling in Sentry

I recently came across a gnarly issue: Every morning we’d see a latency spike in our API for for about 7–8 minutes. More importantly, it happened for a small fraction of requests (0.1%), making it pretty hard to reproduce.

I figured leveraging Sentry’s profiling feature would be a good way to catch this. But this has its own problems.

The first problem is that it’s very expensive to profile all requests: a billion samples where most of them are useless is a waste of money.

Sentry allows sampling requests, which somewhat takes care of the cost problem.

The second problem is that Sentry by default samples based on the count of events. So if you have a sampling rate of 1%, you’ll get approximately 1 in 100 requests. But, since most transactions are pretty fast, these are still useless. And 10 million requests instead of a billion is still a lot of wasted money.

I required a way to sample requests by duration: instead of sampling by the request ID, I want to sample by duration. Give me 50% of all requests that took more than 2 seconds, and 0.0001% of all requests that took less than 2 seconds.

The magic of this kind of sampling is that it gives me the most important spans, while being much cheaper than the default sampling.

The docs don’t mention this is possible at all, but combing through the source code, I found a way to do this. Here’s how it works: