Drug Combination Benchmark Group

Drug combination screening offers significant potential for expanding the use of existing drugs and in improving their efficacy. For instance, the simultaneous modulation of multiple targets can address the common mechanisms of drug resistance seen in the treatment of cancers. However, experimentally exploring the entire space of possible drug combinations is not a feasible task. Thus, computational models that can predict synergistic combinations can prove very valuable.

We measure drug combination response using the complementary metrics of sensitivity and synergy. The drug combination sensitivity (CSS) score is derived using relative IC50 values of compounds and the area under their dose-response curves. For synergy, we use metrics derived from four different models: Bliss model, Highest Single Agent (HSA), Loewe additivity model and Zero Interaction Potency (ZIP). Synergy is a measure of deviation of an observed drug combination response from the expected effect of non-interaction.

In this benchmark group, we use the DrugComb dataset. To access, type:

from tdc import BenchmarkGroup
group = BenchmarkGroup(name = 'DrugCombo_Group', path = 'data/', file_format='pkl')

predictions = {}

for benchmark in group:
    name = benchmark['name']
    train_val, test = benchmark['train_val'], benchmark['test']

    ## --- train your model --- ##
    predictions[name] = y_pred_test

out = group.evaluate(predictions)

print(out)
'''
Note that the output includes the evaluations across tissues:

{'drugcomb_css': {'mae': 22.963}, 'drugcomb_css_kidney': {'mae': 21.811}, 'drugcomb_css_lung': {'mae': 21.367},
'drugcomb_css_breast': {'mae': 18.25}, 'drugcomb_css_hematopoietic_lymphoid': {'mae': 40.379}, 'drugcomb_css_colon': {'mae': 25.001},
'drugcomb_css_prostate': {'mae': 21.874}, 'drugcomb_css_ovary': {'mae': 19.468}, 'drugcomb_css_skin': {'mae': 18.683}, 
'drugcomb_css_brain': {'mae': 21.89}, 'drugcomb_hsa': {'mae': 4.562}, 'drugcomb_loewe': {'mae': 11.109}, 'drugcomb_bliss': {'mae': 4.686}, 
'drugcomb_zip': {'mae': 4.545}}
'''

Follow the instruction on how to use the BenchmarkGroup class and for other useful functions to facilitate your model building.

For every dataset, we use drug combination split and hold out a 20% test set. The evaluation metrics is MAE.

We encourage submissions that reports results for all five synergy scores. Note that the tissue types are automatically calculated based on the test set prediction on CSS data type.

Summary

Label	Number	Task	Metric	Split
CSS	297,098	Regression	MAE	Combination
HSA	297,098	Regression	MAE	Combination
Loewe	297,098	Regression	MAE	Combination
Bliss	297,098	Regression	MAE	Combination
Zip	297,098	Regression	MAE	Combination

Tissue	Label	# of Cell Lines	Test Set Size	Metric
Kidney	CSS	8	8,096	MAE
Lung	CSS	9	9,108	MAE
Breast	CSS	5	20,551	MAE
Hema	CSS	6	6,072	MAE
Colon	CSS	7	7,084	MAE
Prostate	CSS	2	2,024	MAE
Ovary	CSS	7	7,084	MAE
Skin	CSS	9	9,108	MAE
Brain	CSS	6	6,072	MAE

Leaderboard on Combination Response Prediction

Rank	Model	Contact	Link	#Params	CSS	HSA	Loewe	Bliss	Zip
1	MLP	Yusuf Roohani	GitHub, Paper	7,141,297	16.858 ± 0.005	4.453 ± 0.002	9.184 ± 0.001	4.560 ± 0.000	4.027 ± 0.003

: Click to sort models for specific benchmark column.

Leaderboard on Combination Sensitivity Scores Across Tissues

Rank	Model	Contact	Link	#Params	Kidney	Lung	Breast	Hema	Colon	Prost.	Ovary	Skin	Brain
1	MLP	Yusuf Roohani	GitHub, Paper	7,141,297	14.570 ± 0.003	15.653 ± 0.017	13.432 ± 0.049	28.764 ± 0.201	17.729 ± 0.042	15.692 ± 0.005	15.263 ± 0.041	15.663 ± 0.065	15.694 ± 0.006

: Click to sort models for specific benchmark column.