Calculating chemical similarity at scale is essential to drug discovery but remains a compute-intensive challenge. Yeedu’s Turbo Engine processes 2.4 million compound comparisons in under 2 hours at just $18, compared to $2,400 using traditional Spark. This breakthrough enables researchers to accelerate molecular analysis, reduce infrastructure costs, and increase R&D productivity.
Chemical similarity plays a critical role in drug discovery, helping researchers:
In simple terms, chemical similarity measures how alike two chemical compounds are.
These comparisons rely on molecular fingerprints—binary strings that encode specific structural features of molecules. The Tanimoto coefficient, a widely used metric, quantifies this similarity.
The computation of chemical similarity, particularly at scale, presents a significant challenge. Take the ChEMBL database, for instance, a critical resource in drug discovery containing information on roughly 2.4 million compounds. Calculating the Tanimoto coefficient between each pair of compounds requires a staggering 5.76 trillion comparisons (2.4 million × 2.4 million).
This level of computation is a massive hurdle, as it demands enormous processing power, making it difficult for many organizations to perform this efficiently.
Running similarity calculations for millions of compounds is far from trivial. It is a process that can become computationally prohibitive, even for organizations with large-scale computing infrastructure. Yet, scientists and researchers depend on these computations to drive breakthroughs in drug discovery. Without rapid, efficient ways to process these massive datasets, research progress can slow to a crawl.
We put this challenge to the test using Yeedu's Turbo Engine (currently in Beta). First, we generated molecular fingerprints (2048-bit Morgan fingerprints, radius 2) for all 2.4 million compounds in the ChEMBL database.
We processed the Tanimoto coefficients for all pairs of compounds on a 192-core AWS EC2 instance (C7i.x48large)
Results:
Baseline Comparison:
The number of computations scales quadratically with the number of compounds, since the coefficient of each compound needs to be calculated with respect to all the other compounds. Therefore, time taken for computations also scales quadratically, with the number of compounds. In other words, if the number of compounds doubles, the number of calculation and hence the time taken is expected to increase by four times.
To illustrate it mathematically, if T(n) is the time required to compute Tanimoto coefficient for n compounds, then time required to compute Tanimoto coefficient for 2 million compounds is T(2M) = 1.75 hours (i.e. 1 hour, 45 minutes).
As per quadratic scaling, the expected times for the compound similarity will be as under:
However, with Yeedu, while the number of calculations increases quadratically, the time required to complete the calculations, and hence the cost scales sub-quadratically.
This is due to the efficiency of Yeedu’s Turbo engine – a rearchitected spark execution engine that speeds up processing with:
.png)
While the benchmark focuses on scale, day-to-day researchers often need fast comparisons for smaller batches (1–100 compounds).
With Yeedu:
This enables real-time insights without the need for heavy backend infrastructure.
Chemical similarity accelerates:
With Yeedu, this can now happen:
Researchers can spend less time waiting for jobs to finish and more time pushing scientific boundaries.
Traditional Spark infrastructure simply wasn’t built with this level of efficiency in mind. Yeedu changes that.
By cutting costs from $2,400 to $18, and enabling sub-second latency for small queries, Turbo Engine proves that chemical similarity at scale doesn’t have to be prohibitively expensive.
