How much does Cursor adoption increase productivity at a security company with almost 100 engineers
We’re building Bilanc, helping companies measure and improve developer productivity using AI. We recently ran a joint study on the impact of AI on developer productivity with one of our customers. Our customer is a well-known series B security vendor that has raised well over $100M. Due to the nature of their work, we both feel it's our best interests to redact their name from this article, but since we’re both Y Combinator portco's we still thought it would be cool to share the learnings. We hope you'll still enjoy!
We’ll cover:
How we designed this experiment
The results
How we designed this experiment?
Our customer had already started rolling out Cursor across their organisation before becoming a Bilanc customer. Their Cursor rollout and the timing of our onboarding led itself very nicely into a post-hoc quasi-experiment.
They had 1 month worth of cursor usage data, so we began our observation period from the month prior. We conducted a pre-post analysis comparing productivity scores before and after adoption, using non-AI adopters as a control group to isolate the impact of AI on productivity.
How to define an AI adopter?
We do this in two ways, for 2 different calculations
For overall AI adoption %’s we look at whether engineers that submit PR’s have used Cursor AI features at all in the given time period.
For calculating the productivity boost at an engineer level, we calculate the differential in productivity score for PR’s which are shipped on days when the engineer is using AI features in Cursor vs. when they aren’t using AI features (mostly before the rollout).
Note: In our dataset, once engineers start using Cursor AI features, they almost always use it every single day.
Results
Huge thanks to our customer for working with us to publish the results of this study.
