ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI

TL;DR: This paper proposes a common evaluation framework for automatic stroke lesion segmentation from MRIP, describes the publicly available datasets, and presents the results of the two sub‐challenges: Sub‐Acute Stroke Lesion Segmentation (SISS) and Stroke Perfusion Estimation (SPES).

About: This article is published in Medical Image Analysis.The article was published on 2017-01-01 and is currently open access. It has received 417 citations till now.

Summary

1. Introduction

• Still, segmenting stroke lesions from MRI images poses a challenging problem.
• First, the stroke lesions' appearance varies significantly over time, not only between but even within the clinical phases of stroke development.
• In the acute phase, the DWI denotes the infarcted region as hyperintensity.
• They may or may not be aligned with the vascular supply territories and multiple lesions can appear at the same time (e.g. caused by an embolic shower).
• Finally, a good segmentation approach must comply with the clinical workflow.

1.1. Current methods

• The quantification of stroke lesions has gained increasing interest during the past years ( Fig. 1 ).
• A recent review of non-chronic stroke lesion segmenta-Table 1 Listing of publications describing non-chronic stroke lesion segmentation in MRI with evaluation on human image data since Rekik et al. (2012) .
• Column Metrics denotes the metrics used in the evaluation.

Method

• In the present benchmark study, the authors approach the urgent problem of comparability.
• To this end, the authors planned, organized, and pursued the I schemic S troke LE sion S egmentation challenge: A direct, fair, and independently controlled comparison of automatic methods on a carefully selected public dataset.
• ISLES combined two subchallenges dealing with different phases of the stroke lesion evolution: First, the S troke P erfusion ES timation (SPES) challenge dealing with the image interpretation of the acute phase of stroke; second, the S ub-acute I schemic S troke lesion S egmentation (SISS) challenge dealing with the later stroke image patterns.

2. Setup of ISLES

• Interested research teams could register for one or both subchallenges.
• All submitted algorithms were required to be fully automatic; no other restrictions were imposed.
• Until the day of the challenge, the SMIR platform listed over 120 registered users for the ISLES 2015 challenge and a similar count of training dataset downloads.
• Of these, 14 teams provided testing dataset results for SISS and 7 algorithms participated in SPES.

3.2. SPES image data and ground truth

• The training dataset is additionally equipped with a manually created DWI segmentation ground truth set, which roughly denotes the stroke's core area.
• These are not considered in the challenge.

3.3. Evaluation metrics

• Please note that the ASSD and HD values were computed excluding the failed cases (they do, however, incur the lowest vacant rank for these cases).
• The last row shows the inter-observer results for comparison.
• The saturation of the node colors indicates the strength of a method, where better methods are highlighted with more saturated colors.
• Note that all teams with the same number of incoming and outgoing edges perform, statistically spoken, equally well.

Rank

• Similar, the HD is defined as the maximum of all surface distances with EQUATION.
• The distance measure d ( ) employed in both cases is the Euclidean distance, computed taking the voxel size into account.

3.4. Ranking

• The authors chose a third approach based on the ideas of Murphy et al. (2011) that builds on the concept that a ranking reveals only the direction of a relationship between two items (i.e. higher, lower, equal) but not its magnitude.
• Basically, each participant's results are ranked per case according to each of the three metrics and then the obtained ranks are averaged.

3.5. Label fusion

• The specific design of each automatic segmentation algorithm will result in certain strengths and weaknesses for particular challenges in the present image data.
• These algorithms enable a suitable selection and/or fusion to best combine complementary segmentation approaches.
• First, majority vote ( Xu et al., 1992 ) , which simply counts the number of foreground votes over all classification results for each voxel separately and assigns a foreground label if this number is greater than half the number of algorithms.
• Second, the STAPLE algorithm ( Warfield et al., 2004 ) , which performs a simultaneous truth and performance level estimation, that calculates a global weight for each rater and attempts to remove the negative influence of poor algorithms during majority voting.
• Third, the SIMPLE algorithm ( Langerak et al., 2010 ) , which employs a selective and iterative method for performance level estimation by successively removing the algorithms with poorest accuracy as judged by their respective Dice score against a weighted majority vote, where the weights are determined by the previously estimated performances.

4.1. Inter-observer variance

• Comparing the two ground truths of SISS against each other provides (1) the baseline above which an automatic method can be considered to produce results superior to a human rater and (2) a measure of the task's difficulty ( Table 7 , last row).
• The two expert segmentations overlap at least partially for all cases.

4.2. Leaderboard

• The evaluation measures and ranking system employed are described in the method part of this article ( Section 3.4 ).
• No participating method succeeded in segmenting all 36 testing cases successfully (DC > 0) and the best scores are still substantially below the human rater performance.
• Note that for all following experiments, the authors will focus on DC averages only as the ASSD and HD values cannot be readily computed for the failed cases and are thus not suitable for a direct comparison of methods with differing numbers of failure cases.

4.3. Statistical analysis

• The two highest ranking methods, UK-Imp2 and CN-Neu, show no statistically significant differences with a confidence of 95% (i.e. p < 0.025).
• No other algorithm performs better than them, and they both are better than the 12 remaining ones.
• Next comes a group of four methods (FI-Hus, BE-Kul2, US-Odu, De-UzL) to which only the two winners prove superior.
• But among these, FI-Hus takes the highest position as it is statistically better than eight other methods, while the other three only prove superior to at most four competitors.
• The established leaderboard ranking is largely confirmed by the statistical analysis.

4.4. Impact of multi-center data

• Since the training dataset contained only cases from the first center, the difference in performance should reveal the methods' generalization abilities.
• The authors observed that not a single algorithm reached second center scores comparable to its first center scores.

4.5. Combining the participants' results by label fusion

• Applying the three label fusion algorithms presented in Section 3.5 lead to the results presented in Table 7 at the bottom.
• The authors found that the SIMPLE algorithm performed best and could reduce outliers as evident by a lower Hausdorff distance.
• When using majority voting or STAPLE, the negative influence of multiple failed segmentations that are correlated yielded a lower accuracy than at least the two top ranked algorithms.

4.6. Dependency on observer variations

• The average DC scores of each method differed only slightly over the ground truth sets.
• Only in a single case, UK-Imp2, the difference was significant (paired Student's t -test with p < 0.05), but the higher results were obtained for the, formerly unseen, GT02 set.
• The authors can hence conclude that all algorithms generalized well with respect to expert segmentations of different raters.
• An additional data analysis showed that the ranking of the methods does not change if only one or the other of the ground truth sets is employed for evaluation.

4.7. Outlier cases

• The three cases that were successfully processed by nearly all algorithms show large, clearly outlined lesions with a strongly hyperintense FLAIR signal.
• In two of these cases, the DWI signal is relatively weak, in some areas nearly isointense.
• Still, for these cases the algorithms displayed the highest confidence.
• Another case (10) , equally showing a small lesion, has a stronger FLAIR support, but also displays large periventricular WMHs that seem to confuse most algorithms despite missing DWI hyperintensities.
• But many algorithms additionally delineated parts of the periventricular WMHs, which again only show up in the FLAIR sequence.

4.8. Correlation with lesion characteristics

• Significant moderate correlation was found between the lesion volume and the average DC values.
• A statistically significant difference of means was found when comparing cases with haemorrhage present and cases without, as well as between left hemispheric and right hemispheric lesions.
• Since the characteristics cannot be assumed to be independent, the authors furthermore tested the last two groupings for significant differences in lesion volumes between the groups.
• This was found in both cases (see secondary test for each of these two characteristics).
• The authors could not reliably establish a significant influence on the results for any single parameter.

5.1. Leaderboard

• The authors opted not to calculate the HD for SPES as it does not reflect the clinical interest of providing volumetric information of the penumbra region.
• In addition, since some lesions in SPES contained holes, the HD was not a useful metric for gauging segmentation quality.
• This ranking is the outcome of the challenge event and was used to determine the competition winners.
• Visualization of significant differences between the 7 participating methods' case ranks.
• Note that all teams with the same number of incoming and outgoing edges perform, statistically spoken, equally well.

5.2. Statistical analysis

• The authors do not observe significant differences between the two first ranked methods nor between the third and fourth place.
• Hence, SPES has two first ranked, two second ranked, and one third ranked method according to the statistical analysis.

5.4. Outlier cases

• For case 05, the authors can be observed two previous embolisms that cause a compensatory perfusion change, depicted as two hyperintensity regions within the lesion area in the diffusion image and as hypoperfused areas in the Tmax map.
• In summary, the main difficulties faced by the algorithms are related to physiological aspects, such as collateral flow, previous infarcts, etc.

6.1. The most suitable algorithm and the remaining challenges

• The conclusions drawn here are meant to be general and valid for most of the participating methods.
• Any interested reader is invited to download the participants' training dataset results and perform her/his own analysis to test whether these findings hold true for a particular algorithm.

6.2. Recommendations and limitations

• Instead, the findings of this investigation can help them to identify suitable solutions that can serve as support tools:.
• In particular clearly outlined, large lesions are already segmented with good results, which are usually tedious to outline by hand.
• For smaller and less pronounced lesions the manual approach is still recommended.
• Furthermore, they should be aware that individual adaptations to each data source are most likely required -either by tuning the hyperparameters or through machine learning.

7. Discussion: SPES

• The discrepancy between the relatively good results reported by Olivot et al. (2009a ) , Christensen et al. (2010) and Straka et al. (2010) and the poor performance observed in this study can be partially explained by the different end-points (expert segmentation on PWI-MRI vs. follow-up FLAIR/T2), the different evaluation measures (DC/ASSD vs. volume similarity), and the different data.
• This only serves to highlight the need for a public evaluation dataset.
• From an image processing point of view, the volume correlation is not a suitable measure to evaluate segmentations as it can lead to good results despite completely missed lesions.

7.1. The most suitable algorithm and the remaining challenges

• An automated method has to fulfill the strict requirements of clinical routine.
• With 6 min (CH-Insel) and 20 sec (DE-UzL), including all pre-and post-processing steps, the two winning methods fit the requirements, DE-UzL even leaving room for overhead.

7.2. Recommendations and limitations

• While MCA strokes are most common and well suited for mechanical reperfusion therapies ( Kemmling et al., 2015 ) , the restriction to low-noise MCA cases limits the result transfer to clinical routine.
• The generality of the results is additionally reduced by providing only single-center, single-ground truth data.
• Finally, voxel-sized errors in the ground truth prevented the evaluation of the HD, which would have provided additional information.

8. Conclusion

• For the next version of ISLES, the authors would like to focus on the acute segmentation problem from a therapeutical point of view.
• By modeling a benchmark reflecting the time-critical decision making processes for cerebrovascular therapies, the authors hope to promote the transfer from methods to clinical routine and further the exchange between the disciplines.
• A multi-center dataset with hundreds of cases will allow the participants to develop complex solutions.

Figures

Table 2 List of all participants in the ISLES challenge. All teams are color coded for easier reference in all further listings. The ML column denotes whether the submitted algorithm is based on machine learning. Refer to the SISS and SPES columns for the subchallenges each team participated in. Additionally, a very short summary of each method is provided. For a detailed description of each algorithm and used abbreviations see Appendix A .

Fig. 8. Visualization of significant differences between the 7 participating methods’ case ranks. Each node represents a team, each edge a significant difference of the tail side team over the head side team according to a two-sided Wilcoxon signedrank test ( p < 0.025). Therefore, the less outgoing and the more incoming edges a team has (denoted by numbers in brackets (# out/ # in ) for easier interpretation), the weaker its method compared to the others. The saturation of the node colors roughly denotes the strength of a method, where better methods are depicted with stronger colors. Note that all teams with the same number of incoming and outgoing edges perform, statistically spoken, equally well.

Table 8 Correlation between the SISS case characteristics and the average DC values over all teams. A ρ denotes a Spearman correlation, a t a Student’s t -test. All pvalues are two tailed ( p 2 ). Significant results according to a 95% confidence interval are denoted by a ∗ . Secondary tests appearing in the table were performed against the lesion volume rather than the average DC values.

Fig. 9. DC score result of all 7 SPES teams for each of the testing dataset cases. Most methods show a similar pattern. Please refer to the online version for color.

Table 3 Stroke lesion characteristics of the 64 SISS cases. The strong diversity is representative for stroke lesions and emphasizes the difficulty of the task. μ denotes the mean value, [ min, max ] the interval and n the total count. Abbreviations are: anterior cerebral artery (ACA), middle cerebral artery (MCA), posterior cerebral artery (PCA) and basilar artery (BA).

Fig. 7. Visual results for selected difficult (10, 17, 23), easy (2, 5, 13), and second center (29, 32) cases from the SISS testing dataset. The first row shows the distribution of all 14 submitted results on a slice of the FLAIR volume. The second row shows the same image with the ground truth (GT01) outlined in red. And the third row shows the corresponding DWI sequence. Please refer to the online version for colors.

Fig. 10. Sequences of some cases with a low (05 and 11) and high (15) average DC score over all 7 teams participating in SPES. The ground truth is painted red into the DWI sequence slices in the first column. The last column shows the distribution of the resulting segmentations on the gray-scale version of the TTP. All perfusion maps are windowed equally for direct comparison. Please refer to the online version for colors.

Fig. 4. Adaptation to the data from the second medical center. The graph shows each method’s average DC scores on the 28 cases from the first and the eight cases from the second medical center. The methods are color coded. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 5. Differences in performance on the two ground truth sets. The graph shows each methods average DC scores on the 36 testing dataset cases broken down by ground truth set. A star ( ∗) before a team’s name denotes statistical significant difference according to a paired Student’s t -test with p < 0.05. The methods are color coded. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 3. Significant differences between the 14 participating methods’ case ranks according to a two-sided Wilcoxon signed-rank test ( p < 0.025). Each node represents a team, each edge a significant difference of the tail side team over the head side team. Therefore, the less outgoing and the more incoming edges a team has (denoted by numbers in brackets (# out/ # in ) for easier interpretation), the weaker its method compared to the others. The saturation of the node colors indicates the strength of a method, where better methods are highlighted with more saturated colors. Note that all teams with the same number of incoming and outgoing edges perform, statistically spoken, equally well. A higher importance of incoming over outgoing edges or vice-versa cannot be readily established. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Table 1 Listing of publications describing non-chronic stroke lesion segmentation in MRI with evaluation on human image data since Rekik et al. (2012) . Column A denotes the lesion phase, i.e., (A)cute, (S)ub-acute or (C)hronic. Column T denotes the method type, i.e., (A)utomatic or (S)emi-automatic. Column N denotes the number of testing cases (mostly leave-one-out evaluation scheme is employed). Column Sequences denotes the used MRI sequences. Column DC denotes the reported Dice’s coefficient score if available. Column Metrics denotes the metrics used in the evaluation. Abbreviations are: V = visual evaluation, VE = volume error, PPV = positive prediction value, + = other metrics, m = median reported. Note that the lesion phases were adapted to our definition if sufficient information was available.

Fig. B.11. Ranking schema as employed in the ISLES challenge.

Table 9 SPES challenge leaderboard after evaluating the 7 participating methods on the testing dataset. The rank is the final measure for ordering the algorithms’ performances relative to each other. The cases column denotes the number of successfully (i.e., all DC > 0) segmented cases. All evaluation measures are given in mean ± STD. Since no method failed completely on a single case, the reported ASSD values are suitable for a direct comparison between methods. The three next-to-last rows display the results obtained with different fusion approaches. The last two rows denote thresholding methods employed in clinical studies.

Frequently asked questions

Q1. What are the contributions mentioned in the paper "Isles 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri" ?

Al ( MRI ) volumes are intensely res datasets and evaluation scheme Stroke Lesion Segmentation ( ISL In this paper the authors propose a com present the results of the two Perfusion Estimation ( SPES ). However, algorit Overall, no algorithmic characte the characteristics of stroke les studied in detail.