Prospective Comparison of Free-Breathing Accelerated Cine Deep Learning Reconstruction Versus Standard Breath-Hold Cardiac MRI Sequences in Patients With Ischemic Heart Disease
Abstract
BACKGROUND. Cine cardiac MRI sequences require repeated breath-holds, which can be difficult for patients with ischemic heart disease (IHD).
OBJECTIVE. The purpose of the study was to compare a free-breathing accelerated cine sequence using deep learning (DL) reconstruction and a standard breath-hold cine sequence in terms of image quality and left ventricular (LV) measurements in patients with IHD undergoing cardiac MRI.
METHODS. This prospective study included patients undergoing 1.5- or 3-T cardiac MRI for evaluation of IHD between March 15, 2023, and June 21, 2023. Examinations included an investigational free-breathing cine short-axis sequence with DL reconstruction (hereafter, cine-DL sequence). Two radiologists (reader 1 [R1] and reader 2 [R2]), in blinded fashion, independently assessed left ventricular ejection fraction (LVEF), left ventricular end-diastolic volume (LVEDV), left ventricular end-systolic volume (LVESV), and subjective image quality for the cine-DL sequence and a standard breath-hold balanced SSFP sequence; R1 assessed artifacts.
RESULTS. The analysis included 26 patients (mean age, 64.3 ± 11.7 [SD] years; 14 men, 12 women). Acquisition was shorter for the cine-DL sequence than the standard sequence (mean ± SD, 0.6 ± 0.1 vs 2.4 ± 0.6 minutes; p < .001). The cine-DL sequence, in comparison with the standard sequence, showed no significant difference for LVEF for R1 (mean ± SD, 51.7% ± 14.3% vs 51.3% ± 14.7%; p = .56) or R2 (53.4% ± 14.9% vs 52.8% ± 14.6%; p = .53); significantly greater LVEDV for R2 (mean ± SD, 171.9 ± 51.9 vs 160.6 ± 49.4 mL; p = .01) but not R1 (171.8 ± 53.7 vs 165.5 ± 52.4 mL; p = .16); and no significant difference in LVESV for R1 (mean ± SD, 88.1 ± 49.3 vs 86.0 ± 50.5 mL; p = .45) or R2 (85.2 ± 48.1 vs 81.3 ± 48.2 mL; p = .10). The mean bias between the cine-DL and standard sequences by LV measurement was as follows: LVEF, 0.4% for R1 and 0.7% for R2; LVEDV, 6.3 mL for R1 and 11.3 mL for R2; and LVESV, 2.1 mL for R1 and 3.9 mL for R2. Subjective image quality was better for cine-DL sequence than the standard sequence for R1 (mean ± SD, 2.3 ± 0.5 vs 1.9 ± 0.8; p = .02) and R2 (2.2 ± 0.4 vs 1.9 ± 0.7; p = .02). R1 reported no significant difference between the cine-DL and standard sequences for off-resonance artifacts (3.8% vs 23.1% examinations; p = .10) and parallel imaging artifacts (3.8% vs 19.2%; p = .19); blurring artifacts were more frequent for the cine-DL sequence than the standard sequence (42.3% vs 7.7% examinations; p = .008).
CONCLUSION. A free-breathing cine-DL sequence, in comparison with a standard breath-hold cine sequence, showed very small bias for LVEF measurements and better subjective quality. The cine-DL sequence yielded greater LV volumes than the standard sequence.
CLINICAL IMPACT. A free-breathing cine-DL sequence may yield reliable LVEF measurements in patients with IHD unable to repeatedly breath-hold.
TRIAL REGISTRATION. ClinicalTrials.gov NCT05105984
Highlights
Key Finding
▪ In patients with IHD, a free-breathing cine sequence with DL reconstruction showed very low bias for LVEF (R1: 0.4%; R2: 0.7%), better subjective quality (R1: 2.3 ± 0.5 vs 1.9 ± 0.8; R2: 2.2 ± 0.4 vs 1.9 ± 0.7), and faster acquisitions (0.6 ± 0.1 vs 2.4 ± 0.6 minutes) versus a standard breath-hold cine sequence.
Importance
▪ The highly accelerated free-breathing cine-DL sequence allows reliable LVEF quantification in patients with IHD who may have difficulty performing serial breath-holds.
In patients with ischemic heart disease (IHD), coronary artery disease can lead to various manifestations of impaired left ventricular (LV) systolic function. IHD typically involves an irreversible loss of myocardial mass after an acute myocardial infarction, although IHD can also involve contractile dysfunction in a viable myocardial territory that experiences a chronic reduction in coronary flow [1]. IHD is the most common cause of heart failure worldwide [2]. The left ventricular ejection fraction (LVEF), calculated from the left ventricular end-diastolic volume (LVEDV) and left ventricular end-systolic volume (LVESV), is the key parameter for assessing LV systolic function and has important therapeutic and prognostic implications in patients with IHD [3]. Reliable LVEF measurements are critical for guiding management in such patients. For example, evidence of persistent LV dysfunction despite optimal drug therapy is an indication for intracardiac defibrillator implantation [3, 4]. In addition, LVESV is a major predictor of mortality after acute myocardial infarction [5].
Cardiac MRI plays an important role in the diagnostic, prognostic, and pretherapeutic evaluation of patients with IHD, allowing characterization of myocardial morphology, kinetics, and tissue composition in a single examination [6, 7]. MRI is currently the reference standard for the assessment of LV volumes and function [8, 9]. The cine balanced SSFP (bSSFP) sequence is an essential component of cardiac MRI examinations performed to calculate LV volumes by endocardial contouring. This ultrafast gradient-echo sequence has the advantages of excellent spatial resolution, good SNR, and clear differentiation between the myocardium and blood pool [10]. When this sequence is performed, at least 20 phases should be acquired per cardiac cycle to achieve optimal LVEF calculation [11]. Analysis of successive phases of the cardiac cycle on short-axis cine images enables measurement of LV diameters and wall thickness as well as assessment of global and segmental contractility. The analysis of LV volumes and LVEF can be performed using dedicated postprocessing software [9]. However, the cardiac MRI examination is long and requires the patient to perform repeated breath-holds to avoid respiratory artifacts. These breath-holds may be difficult for patients with IHD and can lead to suboptimal image quality, resulting in possible measurement errors.
Given these issues, there is a need to reduce the duration of cardiac MRI sequences while maintaining satisfactory image quality and reliable LV measurements. Recent efforts to reduce sequence times have primarily involved various undersampling techniques to accelerate acquisitions [12–14]. Cardiac cine sequences may also be accelerated through the use of novel deep learning (DL) methods for image reconstruction. These DL image reconstruction techniques could also potentially improve image quality by improving SNR and reducing artifacts. Moreover, combining DL image reconstruction algorithms with other acceleration methods could allow very rapid acquisitions in which the entire short-axis cine sequence of the LV is performed in sufficiently short time periods to permit a free-breathing, rather than breath-holding, technique. Such an ability would be particularly useful in dyspneic patients unable to perform repeated breath-holds.
The aim of this study was to compare a free-breathing accelerated cine sequence using DL image reconstruction and a standard breath-hold cine sequence in terms of image quality and LV measurements in patients with IHD undergoing cardiac MRI.
Methods
Population
This single-center prospective study was approved by an independent ethics committee. The study was registered at ClinicalTrials.gov (NCT05105984). All participants provided written informed consent. One author (J.P.) is an employee of GE HealthCare and provided technical support for the implementation of the cine-DL sequence protocol and the use of the offline reconstruction pipeline described in this study. The remaining authors, who are not employees, had full control of the data and information submitted for publication.
All adult patients scheduled to undergo nonemergent cardiac MRI as part of an initial workup or follow-up evaluation for IHD from March 15, 2023, to June 21, 2023, received a letter before their appointment inviting them to participate in the current study. Patients interested in participating communicated this interest when arriving for their appointment, at which time informed consent was obtained. No additional screening criteria were applied. In patients who consented to participate, the MRI examination included, for investigational purposes, a prototype free-breathing highly accelerated cine sequence with DL image reconstruction (hereafter, cine-DL sequence) in addition to the institution's standard cardiac MRI protocol.
The sex, age, height, BMI, and heart rate of study participants were recorded.
MRI Examinations
MRI examinations were performed on either a 1.5-T scanner (SIGNA Artist, GE HealthCare) or a 3-T scanner (SIGNA Premier, GE HealthCare) using a 20-channel surface coil (MultiPurpose AIR Coil, medium size, GE HealthCare).
The standard MRI protocol for patients with IHD included a standard cine bSSFP sequence (in LV short-axis, two-chamber, and four-chamber planes), a short-axis myocardial perfusion sequence (in LV short-axis plane), and late enhancement sequences (in LV short-axis, two-chamber, and four-chamber planes). The standard cine sequence used a breath-hold 2D segmented ECG-triggered bSSFP (FIESTA, GE HealthCare) acquisition.
The investigational cine-DL sequence was inserted in the protocol before the myocardial perfusion sequence and obtained 2D images in the LV short-axis plane. The sequence was performed with a variable-density k-t acceleration factor of 12 [15], allowing acquisition of each slice over a single heartbeat during free breathing while maintaining a rapid temporal resolution. Respiratory triggering was performed using a pneumatic belt with the acquisition window set to 40%.
Images were reconstructed using a DL algorithm designed for cine cardiac imaging [16]. The pipeline for the cine-DL reconstruction was previously developed in 22 healthy volunteers, unrelated to the present investigation [16]. The present cine-DL sequence incorporated further customization of the previously developed sequence including an additional soft-gating feature [17–19] to compensate for respiratory motion. The Supplemental Methods provides further details about the cine-DL sequence and cine-DL reconstruction technique.
Table 1 summarizes acquisition parameters for the cine-DL and standard cine sequences at 1.5 and 3 T. The temporal resolution for the cine-DL sequence was 58 ms at 1.5 T and 55 ms at 3 T; for the standard sequence, the temporal resolution was 41–57 ms at 1.5 T and 41 ms at 3 T. The in-plane acquired voxel size for the cine-DL sequence was 2.2 × 2.2 mm at 1.5 T and 2.0 × 2.0 mm at 3 T; for the standard sequence, it was 1.8 × 1.6 mm at 1.5 T and 1.7 × 1.5 mm at 3 T. The number and position of the slices were identical for the cine-DL and standard cine sequences.
| Parameter | Cine-DL Sequence | Standard Sequence | ||
|---|---|---|---|---|
| 1.5 T | 3 T | 1.5 T | 3 T | |
| Respiratory technique | Free breathing | Free breathing | Breath-hold | Breath-hold |
| Temporal resolution (ms) | 58 | 55 | 41–57 | 41 |
| Acquired voxel size (mm) | 2.2 × 2.2 | 2.0 × 2.0 | 1.8 × 1.6 | 1.7 × 1.5 |
| FOV (mm) | 400 × 400 | 400 × 400 | 400 × 400 | 380 × 380 |
| Acceleration method | VDkt sampling | VDkt sampling | ASSET | ASSET |
| Acceleration factor | 12 | 12 | 1.5 | 2 |
| Interslice spacing (mm) | 2 | 2 | 2 | 0 |
| Flip angle (°) | 51 | 55 | 49 | 49 |
| Slice thickness (mm) | 8 | 7 | 8 | 7 |
| Phases per cardiac cycle | 30 | 30 | 30 | 30 |
Note—DL = deep learning, VDkt = variable-density k-t, ASSET = array spatial sensitivity-encoding technique.
For each examination, the duration of each of the two cine sequences was recorded. For the standard sequence, this time reflected only time during breath-holds; the recovery times between breath-holds were not included. In addition, the clinical reports from the MRI examinations were reviewed for the presence of transmural or nontransmural infarction.
MRI Analysis
Two radiologists (C.R. and C.D., with 20 and 3 years, respectively, of posttraining experience in cardiovascular imaging; hereafter, reader 1 [R1] and reader 2 [R2], respectively) independently reviewed the examinations while blinded to clinical information and the sequence being reviewed. Examinations were reviewed during two sessions separated by 1 week. Each session included for each patient either the standard sequence or the cine-DL sequence (selected at random) such that each session included a combination of the two sequences across patients. All images were deidentified before analysis.
The readers performed analysis using commercially available software designed for cardiac MRI interpretation (CVI42MR, version 5.11, Circle Cardiovascular Imaging). The readers performed endocardial and epicardial contouring of the LV, including trabeculations and papillary muscles within the LV volume. Selection of basal and apical slices as well as of telediastolic and telesystolic phases was at the reader's discretion. On the basis of the contours, LVEF, LVEDV, LVESV, and left ventricular end-diastolic mass (LVEDM) were calculated.
The readers also placed ROIs in normal-appearing interventricular septum and in the LV cavity on a medioventricular short-axis image, avoiding trabeculations and papillary muscles. On the basis of the mean values of these ROIs, the myocardium-to–blood pool ratio was calculated for each reader (hereafter, the signal ratio). R1 also recorded the SD of each ROI, which was used to calculate the homogeneity index as the ratio between the ROI's mean and SD, whereby a higher homogeneity index indicates a more uniform signal intensity distribution.
In addition, the readers assessed each sequence's subjective visual image quality using a 4-point scale: 0, not interpretable due to severe artifacts; 1, artifacts hampering reliable interpretation; 2, good image quality with artifacts, outside of the heart region, that do not interfere with interpretation; and 3, excellent image quality with very few minor artifacts.
R1 assessed images for the presence or absence of off-resonance artifacts, parallel imaging artifacts, blurring artifacts, and flow artifacts.
Statistical Analysis
Data were summarized using descriptive measures, including counts and frequencies as well as mean ± SD. The LV measures, signal ratio, homogeneity index, subjective image quality, and examination durations were compared between the two sequences using Wilcoxon signed rank tests. For each reader, scatterplots were constructed of LVEF between the two sequences; corresponding regression lines were fit to the plots, and coefficient of determinations (R2) were calculated. Bland-Altman analysis was used to assess agreement (in terms of bias and 95% limits of agreement) between the two sequences for the LV measurements [20]. The frequency of artifacts was compared between the sequences using the Fisher exact test. The intraclass correlation coefficient (ICC) was used to assess agreement between readers or sequences for the quantitative measures; ICC was assessed using the classification by Koo and Li [21]: less than 0.50, poor; 0.50–0.75, moderate; 0.76–0.90, good; 0.91–1, excellent. Cohen kappa coefficient was used to assess agreement between readers for subjective image quality scores according to the scale proposed by Landis and Koch [22]: less than 0, poor; 0.01–0.20, slight; 0.21–0.40, fair; 0.41–0.60, moderate; 0.61–0.80, substantial or strong; and 0.81–1.00, almost perfect. Additional subanalyses were performed in examinations stratified by field strength (1.5 or 3 T).
p values less than .05 were considered statistically significant. All analyses were performed using RStudio (version 1.3.1093) and R (version 3.6.3).
Results
Study Sample
A total of 31 adult patients were scheduled to undergo nonemergent cardiac MRI as part of initial workup or follow-up evaluation for IHD during the study period. All of these patients received the recruitment letter for study participation, were interested in the study, and subsequently consented to participation; these patients underwent cardiac MRI that included the investigational free-breathing cine-DL sequence. Five patients who consented and underwent MRI were excluded from the analysis. These five patients included three patients with arrhythmia in whom the standard breath-hold sequence was of insufficient quality to obtain the quantitative measurements as well as two patients in whom reconstruction of the cine-DL had a technical failure related to data loss or corruption during transfer of the raw data from the MRI console to the external reconstruction system. Thus, the final analyzed study sample included 26 patients (mean age, 64.3 ± 11.7 (SD) years; 14 men, 12 women). Figure 1 shows the flow of patient selection. The characteristics of the study sample are summarized in Table 2. Sixteen and 10 MRI examinations were performed at 1.5 and 3 T, respectively. Figures 2 and 3 provide representative images for the two sequences at 1.5 and 3 T, respectively. MRI showed transmural infarction in seven patients, nontransmural infarction in six patients, and no infarction in 13 patients.
| Characteristic | Value |
|---|---|
| Sex, no. (%) | |
| Male | 14 (54) |
| Female | 12 (46) |
| Age (y) | 64.3 ± 11.7 |
| Height (cm) | 171.0 ± 7.9 |
| BMI | 27.7 ± 5.6 |
| Heart rate (beats/min) | 70.7 ± 13.1 |
Note—Except where otherwise indicated, data are reported as mean ± SD.
Fig. 2A —44-year-old patient with ischemic heart disease who underwent cardiac MRI at 1.5 T.
A, Short-axis images from standard cine balanced SSFP breath-hold sequence (A) and free-breathing accelerated cine sequence with deep learning reconstruction (B) show normal appearance of left ventricular myocardium.
Fig. 2B —44-year-old patient with ischemic heart disease who underwent cardiac MRI at 1.5 T.
B, Short-axis images from standard cine balanced SSFP breath-hold sequence (A) and free-breathing accelerated cine sequence with deep learning reconstruction (B) show normal appearance of left ventricular myocardium.
Fig. 3A —85-year-old patient with ischemic heart disease who underwent cardiac MRI at 3 T.
A, Short-axis images from standard cine balanced SSFP breath-hold sequence (A) and free-breathing accelerated cine sequence with deep learning reconstruction (B) show normal appearance of left ventricular myocardium.
Fig. 3B —85-year-old patient with ischemic heart disease who underwent cardiac MRI at 3 T.
B, Short-axis images from standard cine balanced SSFP breath-hold sequence (A) and free-breathing accelerated cine sequence with deep learning reconstruction (B) show normal appearance of left ventricular myocardium.
Acquisition Times
The mean acquisition time (± SD) was significantly shorter for the cine-DL sequence than for the standard sequence (0.6 ± 0.1 vs 2.4 ± 0.6 minutes; p < .001).
LVEF and Volumes
Table 3 reports the LV measurements (mean ± SD) for each sequence for each reader. The cine-DL sequence, in comparison with the standard sequence, showed no significant difference for LVEF for R1 (51.7% ± 14.3% vs 51.3% ± 14.7%; p = .56) or R2 (53.4% ± 14.9% vs 52.8% ± 14.6%; p = .53); no significant difference for LVEDV for R1 (171.8 ± 53.7 vs 165.5 ± 52.4 mL; p = .16) but significantly greater LVEDV for R2 (171.9 ± 51.9 vs 160.6 ± 49.4 mL; p = .01); no significant difference for LVESV for R1 (88.1 ± 49.3 vs 86.0 ± 50.5 mL; p = .45) or R2 (85.2 ± 48.1 vs 81.3 ± 48.2 mL; p = .10); and significantly greater LVEDM for R1 (139.7 ± 54.4 vs 130.3 ± 49.1 g; p = .006) but no significant difference for LVEDM for R2 (128.2 ± 50.8 vs 125.1 ± 50.0 g; p = .26).
| Parameter | Reader 1 | Reader 2 | ||||
|---|---|---|---|---|---|---|
| Cine-DL | Standard | p | Cine-DL | Standard | p | |
| LVEF (%) | 51.7 ± 14.3 | 51.3 ± 14.7 | .56 | 53.4 ± 14.9 | 52.8 ± 14.6 | .53 |
| LVEDV (mL) | 171.8 ± 53.7 | 165.5 ± 52.4 | .16 | 171.9 ± 51.9 | 160.6 ± 49.4 | .01 |
| LVESV (mL) | 88.1 ± 49.3 | 86.0 ± 50.5 | .45 | 85.2 ± 48.1 | 81.3 ± 48.2 | .10 |
| LVEDM (g) | 139.7 ± 54.4 | 130.3 ± 49.1 | .006 | 128.2 ± 50.8 | 125.1 ± 50.0 | .26 |
| Signal ratioa | 4.1 ± 0.6 | 3.2 ± 1.0 | < .001 | 4.0 ± 0.6 | 3.1 ± 1.0 | < .001 |
| Homogeneity indexb | ||||||
| LV cavity | 36.8 ± 15.0 | 20.8 ± 6.5 | < .001 | — | — | — |
| Myocardium | 22.9 ± 13.5 | 15.4 ± 5.7 | .01 | — | — | — |
| Subjective image qualityc | 2.3 ± 0.5 | 1.9 ± 0.8 | .02 | 2.2 ± 0.4 | 1.9 ± 0.7 | .02 |
Note—Except where otherwise indicated, data are presented as mean ± SD. Dash (—) indicates not evaluated by reader 2. DL = deep learning, LVEF = left ventricular ejection fraction, LVEDV = left ventricular end-diastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass, LV = left ventricular.
a
The myocardium-to–blood pool ratio.
b
A higher homogeneity index indicates a more uniform signal intensity distribution.
c
Subjective visual image quality was assessed using a 4-point scale: 0, not interpretable due to severe artifacts; 1, artifacts hampering reliable interpretation; 2, good image quality with artifacts, outside of the heart region, that do not interfere with interpretation; and 3, excellent image quality with very few minor artifacts.
Figure 4 shows scatterplots of LVEF between the cine-DL and standard sequences for the two readers; the corresponding regression lines yielded R2 of 0.95 for R1 and 0.95 for R2.
Fig. 4A —Scatterplots show LVEF values from free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for R1 and R2. LVEF = left ventricular ejection fraction, DL = deep learning, R1 = reader 1, R2 = reader R2.
A, Scatterplots of LVEF values (circles) from cine-DL and standard sequences for R1 (A) and R2 (B). Straight lines represents corresponding linear regression lines; coefficient of determination (R2) is 0.95 for R1 and 0.95 for R2.
Fig. 4B —Scatterplots show LVEF values from free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for R1 and R2. LVEF = left ventricular ejection fraction, DL = deep learning, R1 = reader 1, R2 = reader R2.
B, Scatterplots of LVEF values (circles) from cine-DL and standard sequences for R1 (A) and R2 (B). Straight lines represents corresponding linear regression lines; coefficient of determination (R2) is 0.95 for R1 and 0.95 for R2.
Interreader agreement for LVEF, LVEDV, LVESV, and LVEDM was excellent for both the cine-DL sequence (ICC, 0.96–0.99) and the standard sequence (ICC, 0.97–0.98) (Table 4).
| Variable | Agreement (95% CI) |
|---|---|
| Cine-DL sequence | |
| LVEF | 0.98 (0.92–0.99) |
| LVEDV | 0.99 (0.97–0.99) |
| LVESV | 0.99 (0.98–1.00) |
| LVEDM | 0.96 (0.64–0.99) |
| Signal ratioa | 0.86 (0.71–0.93) |
| Subjective image quality | 0.69 (0.37–1.00) |
| Standard sequence | |
| LVEF | 0.97 (0.93–0.99) |
| LVEDV | 0.98 (0.95–0.99) |
| LVESV | 0.98 (0.95–0.99) |
| LVEDM | 0.97 (0.93–0.99) |
| Signal ratioa | 0.94 (0.88–0.97) |
| Subjective image quality | 0.87 (0.72–1.00) |
Note—Interreader agreement is expressed using intraclass correlation coefficient for all measures except subjective image quality, for which the kappa coefficient is used. DL = deep learning, LVEF = left ventricular ejection fraction, LVEDV = left ventricular end-diastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
a
The myocardium-to–blood pool ratio.
The two sequences showed excellent agreement for all LV measurements for both readers (ICC, 0.93–0.98), aside from good agreement for LVEDV for R2 (ICC, 0.89). Table 5 summarizes results from the Bland-Altman analysis for agreement of the LV measurements between the two sequences. The mean bias between the cine-DL and standard sequences (i.e., positive and negative values indicating larger and smaller measurements, respectively, for the cine-DL sequence) for LVEF was 0.4% for R1 and 0.7% for R2, for LVEDV was 6.3 mL for R1 and 11.3 mL for R2, for LVESV was 2.1 mL for R1 and 3.9 mL for R2, and for LVEDM was 9.3 g for R1 and 3.1 g for R2. Figure 5 shows the corresponding Bland-Altman plots for both readers.
| Reader, Test | LVEF | LVEDV | LVESV | LVEDM |
|---|---|---|---|---|
| Reader 1 | ||||
| ICC | 0.98 (0.95–0.99) | 0.93 (0.84–0.97) | 0.98 (0.95–0.99) | 0.95 (0.83–0.98) |
| Biasa | 0.4% (−0.8% to 1.6%) | 6.3 mL (−1.7 to 14.3 mL) | 2.1 mL (−2.5 to 6.6 mL) | 9.3 g (3.7−15.0 g) |
| 95% Limits of agreementa | −5.5%, 6.3% | −32.4 mL, 45.1 mL | −20.0 mL, 24.1 mL | −18.2 g, 36.8 g |
| Reader 2 | ||||
| ICC | 0.98 (0.95–0.99) | 0.89 (0.73–0.96) | 0.97 (0.93–0.99) | 0.97 (0.92–0.98) |
| Biasa | 0.7% (−0.6% to 1.9%) | 11.3 mL (2.8–19.9 mL) | 3.9 mL (−0.8 to 8.6 mL) | 3.1 g (−2.2 to 8.4 g) |
| 95% Limits of agreementa | −5.6%, 6.9% | −30.0 mL, 52.7 mL | −18.7 mL, 26.5 mL | −22.8 g, 29.0 g |
Note—Values in parentheses indicate 95% CIs. ICC = intraclass correlation coefficient, DL = deep learning, LVEF = left ventricular ejection fraction, LVEDV = left ventricular end-diastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
a
In Bland-Altman analysis results, positive and negative values indicate larger and smaller measurements, respectively, for cine-DL in comparison with standard sequence.
Fig. 5A —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
A, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Fig. 5B —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
B, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Fig. 5C —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
C, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Fig. 5D —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
D, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Fig. 5E —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
E, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Fig. 5F —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
F, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Fig. 5G —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
G, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Fig. 5H —Bland-Altman plots show agreement between free-breathing accelerated cine sequence with DL reconstruction and standard cine balanced SSFP breath-hold sequence for left ventricular measurements for R1 and R2. DL = deep learning, R1 = reader 1, R2 = reader 2, LVEF = left ventricular ejection fraction, LVEDV = left ventricular enddiastolic volume, LVESV = left ventricular end-systolic volume, LVEDM = left ventricular end-diastolic mass.
H, Bland-Altman plots show agreement between cine-DL and standard sequences for LVEF, LVEDV, LVESV, and LVEDM for R1 (A–D, respectively) and R2 (E–H). Solid horizontal lines represent bias. Dashed horizontal lines represent 95% limits of agreement. Associated data are shown in Table 5.
Signal Ratio and Homogeneity Index
Table 3 reports the signal ratio and homogeneity index for each sequence. Interobserver agreement for the signal ratio was good for the cine-DL sequence (ICC, 0.86) and excellent for the standard sequence (ICC, 0.94). The signal ratio was significantly higher for the cine-DL sequence than for the standard sequence for R1 (mean, 4.1 vs 3.2, respectively; p < .001) and R2 (4.0 vs 3.1; p < .001). On the basis of measurements by one reader, the cine-DL sequence, in comparison with the standard sequence, showed significantly higher homogeneity index in both the LV cavity (mean ± SD, 36.8 ± 15.0 vs 20.8 ± 6.5; p < .001) and the myocardium (22.9 ± 13.5 vs 15.4 ± 5.7; p = .01).
Subjective Image Quality
Interreader agreement for subjective image quality scores was strong for the cine-DL sequence (κ = 0.69) and almost perfect for the standard sequence (κ = 0.87). For the cine-DL sequence, R1 assessed 19 examinations as a score of 2 and seven examinations as a score of 3, whereas R2 assessed 20 examinations as a score of 2 and six examinations as a score of 3. For the standard sequence, R1 assessed nine examinations as a score of 1, 11 examinations as a score of 2, and six examinations as a score of 3, whereas R2 assessed eight examinations as a score 1, 13 examinations as a score of 2, and five examinations as a score of 3. No examination received a score of 0 for either sequence for either reader. Subjective image quality was significantly better for the cine-DL sequence than the standard sequence for R1 (mean ± SD, 2.3 ± 0.5 vs 1.9 ± 0.8; p = .02) and R2 (2.2 ± 0.4 vs 1.9 ± 0.7; p = .02).
R1 reported no significant difference between cine-DL and standard sequences for off-resonance artifacts (3.8% [1/26] vs 23.1% [6/26] examinations; p = .10), parallel imaging artifacts (3.8% [1/26] vs 19.2% [5/26]; p = .19), and flow artifacts (0.0% [0/26] vs 7.7% [2/26]; p = .49). However, blurring artifacts were significantly more frequent in the cine-DL sequence than the standard sequence (42.3% [11/26] vs 7.7% [2/26] examinations; p = .008).
Subanalysis by Magnetic Field Strength
Table S1 summarizes the LV measurements, signal ratios, homogeneity indexes, and subjective image quality scores for each sequence and reader, stratified by field strength. Table S2 presents results of the Bland-Altman analysis for comparison of LV measurements between the two sequences for each reader, stratified by field strength. At 1.5 T, mean LVEF was 47.9% ± 15.5% for the cine-DL sequence versus 47.9% ± 16.1% for the standard sequence for R1 and 49.3% ± 15.9% for the cine-DL sequence versus 49.6% ± 16.3% for the standard sequence for R2. At 3 T, mean LVEF was 57.7% ± 10.1% for the cine-DL sequence versus 56.7% ± 10.6% for the standard sequence for R1 and 59.9% ± 11.1% for the cine-DL sequence versus 57.7% ± 10.4% for the standard sequence for R2. LVEF exhibited mean bias for the cine-DL sequence versus the standard sequence at 1.5 T of 0.0% for R1 and −0.3% for R2; at 3 T, it was 1.0% for R1 and 2.2% for R2.
Table S3 summarizes the distribution of subjective image quality scores for each sequence and reader, stratified by field strength. At 1.5 T, mean subjective image quality score was 2.4 ± 0.5 for the cine-DL sequence versus 2.1 ± 0.8 for the standard sequence for R1 and 2.4 ± 0.5 for the cine-DL sequence versus 2.1 ± 0.8 for the standard sequence for R2. At 3 T, mean subjective image quality score was 2.1 ± 0.3 for the cine-DL sequence versus 1.5 ± 0.5 for the standard sequence for R1 and 2.0 ± 0.0 for the cine-DL sequence versus 1.6 ± 0.5 for the standard sequence for R2. Off-resonance artifacts were observed for the cine-DL sequence in zero of 16 1.5-T examinations and one of 10 3-T examinations and were observed for the standard sequence in zero of 16 1.5-T and six of 10 3-T examinations. Parallel imaging artifacts were observed for the cine-DL sequence in one of 16 1.5-T and zero of 10 3-T examinations and were observed for the standard sequence in one of 16 1.5-T and four of 10 3-T examinations. Flow artifacts were observed for the cine-DL sequence in zero of 16 1.5-T and zero of 10 3-T examinations and were observed for the standard sequence in 0 of 16 1.5-T and two of 10 3-T examinations. Blurring artifacts were observed for the cine-DL sequence in eight of 16 1.5-T and three of 10 3-T examinations and were observed for the standard sequence in two of 16 1.5-T and zero of 10 3-T examinations.
Discussion
This study evaluated a highly accelerated free-breathing cine sequence with DL reconstruction in patients with IHD undergoing cardiac MRI in comparison with a standard breath-hold cine sequence. For both readers, LVEF showed excellent agreement and very small bias between the cine-DL sequence and the standard sequence. The cine-DL sequence, in comparison with the standard sequence, showed significantly higher signal index between the myocardium and blood pool and no significant difference in subjective image quality for either reader. The free-breathing cine-DL sequence may be particularly useful for performing LVEF measurements in patients with IHD who are experiencing dyspnea, which may lead to difficulty in performing serial breath-holds over the course of an MRI examination.
Reliable LV measurements are essential to individualize therapeutic strategies in patients with IHD. LVEF is the primary parameter impacting such decisions, although LV volumes may also play a role. Based on the Bland-Altman analysis from the current study, LV volume measurements were greater for the cine-DL sequence than for the standard sequence. These greater volumes for the cine-DL sequence may reflect variation in heart position resulting from respiration during the sequence's acquisition; this variation may in turn alter the selection of basal and apical slices for endocardial segmentation. Differences in spatial and temporal resolutions between the two sequences may have also contributed to the differences in volume measurements. In addition, blurring artifacts (which were more common for the cine-DL sequence, likely attributed to the sequence's undersampling used for acceleration) may have impacted the delineation of endocardial and epicardial contours, particularly in basal regions, thereby influencing LV measurements.
The improved subjective image quality of the cine-DL sequence likely in part relates to the sequence's respiratory synchronization, in comparison with the standard sequence's reliance on adequate breath-holding, which may be challenging in dyspneic patients. The cine-DL sequence also uses a smoother reconstruction algorithm. Off-resonance, parallel imaging, and flow artifacts were not significantly different between the two sequences. Nonetheless, the standard sequence appeared more prone to off-resonance and parallel imaging artifacts at 3 T than at 1.5 T. Although not shown by the present data, these artifacts could potentially influence the LV measurements.
These results align with the findings of a study by Zucker et al. [23], which to our knowledge is the only prior clinical study to have compared a free-breathing cine sequence using DL image reconstruction versus a standard bSSFP cine sequence. That study used the same underlying cine-DL reconstruction pipeline, which had been initially developed in 22 healthy volunteers [16], as was used in the present investigation; however, the study by Zucker et al. and the current study evaluated distinct customizations of that pipeline. Zucker et al. evaluated the sequence in a sample of 50 children and young adults with cardiomyopathy or congenital heart disease; 45 of the examinations were performed at 1.5 T and five examinations at 3 T. The cine-DL sequence and standard sequence showed excellent concordance for LVEDV (ICC = 0.97) and LVESV (ICC = 0.95) and good concordance for LVEF (ICC = 0.76) with very small bias for LVEF (0.7%). In comparison, the current study found excellent concordance for LVEF (ICC = 0.98 for both readers). In contrast with the current study, Zucker et al. found singly lower mean subjective image quality scores for the cine-DL sequence than for the reference sequence. Finally, Zucker et al. did not compare the two sequences in terms of the presence of artifacts, as was evaluated in the current study.
Other studies have compared LV measures using fast free-breathing sequences, albeit without the use of DL reconstruction. For example, Kido et al. [24] evaluated a free-breathing cine sequence that used compressed sensing without respiratory synchronization in 63 patients with various heart diseases who underwent 3-T MRI. The sequence incorporated prospective cardiac synchronization over two heartbeats to cover the entire cardiac cycle. The free-breathing sequence had a mean acquisition time of 24 ± 4 (SD) seconds but acquired only eight slices. Excellent correlations with the reference sequence were observed for LVEF, LVEDV, and LVESV (r = 0.94, 0.92, and 0.97, respectively) with negligible bias (−1.1%, 2.1 mL, and 2.2 mL). The accelerated sequence showed lower image quality but remained satisfactory in all patients.
In an additional study, Bellenger et al. [25] assessed the performance of a free-breathing cine sequence with echo-navigation synchronization in 15 patients with dyspneic heart failure and 10 healthy volunteers. LVEF, LVEDV, and LVESV were similar to LV measures obtained using the standard sequence. Nevertheless, the applied synchronization technique requires a regular breathing rate and thus may lead to extended acquisition times in dyspneic patients [26]. The acquisition time was slightly lower for the accelerated than the reference sequence (8.7 vs 11.9 minutes) [26]. Also, Kaji et al. [27] evaluated an ultrafast real-time cine sequence with spiral and circular k-space filling in a single radiofrequency pulse. That sequence allowed acquisition of the whole heart during free breathing without cardiac synchronization using interactive positioning of each slice plane. Despite its low spatial resolution, the sequence yielded excellent correlation with the reference sequence for LVEF, LVEDV, and LVESV (all r > 0.98). The acquisition time for the accelerated sequence was 2.2 minutes [27].
This study had limitations. First, the study was performed at a single center in a small sample. Second, there was no ground-truth reference standard for the LV measures. Third, the cine-DL sequence differed from the standard sequence in terms of both the use of undersampling and the use of DL image reconstruction; comparisons were not performed with sequences using only one of these two techniques. Fourth, the two sequences were not compared in terms of diagnostic performance for various pathologic conditions. Fifth, the cine-DL sequence was performed only in the LV short-axis plane. Sixth, despite the readers not being informed of the sequences being evaluated, the readers may have been able to infer the sequence on the basis of the appearance of the images. Seventh, the cine-DL sequence was performed after the standard sequence in all patients. We believe that this aspect of the study design is unlikely to have created a substantial bias given the use of free breathing for the cine-DL sequence (vs if both sequences used a breath-hold technique). Eighth, we did not analyze measures of objective image quality, such as the proposed European cardiovascular MR registry score [28], or edge sharpness measurements [29]. Finally, the cine-DL sequence was trained in volunteers; future work could train the algorithm in patients with specific cardiac conditions to reflect intended patient populations for the sequence's application.
In conclusion, in patients with IHD, a highly accelerated free-breathing cardiac cine sequence with DL reconstruction yielded excellent agreement and very small bias for LVEF measurements in comparison with a standard 2D bSSFP breath-hold cine sequence. Based on Bland-Altman analysis, LV volumes were higher for the cine-DL sequence than for the standard sequence. Subjective image quality was not significantly different between the two sequences, although blurring artifact was more frequent for the cine-DL sequence. This free-breathing cine-DL sequence could be particularly useful in the evaluation of patients with dysp nea who are unable to perform repeated breath-holds.
Footnotes
Provenance and review: Not solicited; externally peer reviewed.
Peer reviewers: Giuseppe Tremamunno, University of Rome La Sapienza; Matthew S. Lazarus, Albert Einstein College of Medicine; Anle Yu, The First Affiliated Hospital of Hainan Medical University; additional individual(s) who chose not to disclose their identity.
Supplemental Content
File (23_30272_suppl.pdf)
- Download
- 1.08 MB
References
1.
Del Buono MG, Moroni F, Montone RA, Azzalini L, Sanna T, Abbate A. Ischemic cardiomyopathy and heart failure after acute myocardial infarction. Curr Cardiol Rep 2022; 24:1505–1515
2.
Felker GM, Shaw LK, O'Connor CM. A standardized definition of ischemic cardiomyopathy for use in clinical research. J Am Coll Cardiol 2002; 39:210–218
3.
McDonagh TA, Metra M, Adamo M, et al.; ESC Scientific Document Group. 2021 ESC guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur Heart J 2021; 42:3599–3726
4.
Bardy GH, Lee KL, Mark DB, et al.; Sudden Cardiac Death in Heart Failure Trial (SCD-HeFT) Investigators. Amiodarone or an implantable cardioverter-defibrillator for congestive heart failure. N Engl J Med 2005; 352:225–237
5.
White HD, Norris RM, Brown MA, Brandt PW, Whitlock RM, Wild CJ. Left ventricular end-systolic volume as the major determinant of survival after recovery from myocardial infarction. Circulation 1987; 76:44–51
6.
Pennell DJ, Sechtem UP, Higgins CB, et al.; Society for Cardiovascular Magnetic Resonance; Working Group on Cardiovascular Magnetic Resonance of the European Society of Cardiology. Clinical indications for cardiovascular magnetic resonance (CMR): consensus panel report. Eur Heart J 2004; 25:1940–1965
7.
Schwitter J, Arai AE. Assessment of cardiac ischaemia and viability: role of cardiovascular magnetic resonance. Eur Heart J 2011; 32:799–809
8.
Petersen SE, Aung N, Sanghvi MM, et al. Reference ranges for cardiac structure and function using cardiovascular magnetic resonance (CMR) in Caucasians from the UK Biobank population cohort. J Cardiovasc Magn Reson 2017; 19:18
9.
Jerosch-Herold M, Kwong RY. Magnetic resonance imaging in the assessment of ventricular remodeling and viability. Curr Heart Fail Rep 2008; 5:5–10
10.
Kramer CM, Barkhausen J, Flamm SD, Kim RJ, Nagel E; Society for Cardiovascular Magnetic Resonance Board of Trustees Task Force on Standardized Protocols. Standardized cardiovascular magnetic resonance imaging (CMR) protocols, Society for Cardiovascular Magnetic Resonance: Board of Trustees Task Force on Standardized Protocols. J Cardiovasc Magn Reson 2008; 10:35
11.
Roussakis A, Baras P, Seimenis I, Andreou J, Danias PG. Relationship of number of phases per cardiac cycle and accuracy of measurement of left ventricular volumes, ejection fraction, and mass. J Cardiovasc Magn Reson 2004; 6:837–844
12.
Curione D, Ciliberti P, Monti CB, et al. Compressed sensing cardiac cine imaging compared with standard balanced steady-state free precession cine imaging in a pediatric population. Radiol Cardiothorac Imaging 2022; 4:e210109
13.
Altmann S, Halfmann MC, Abidoye I, et al. Compressed sensing acceleration of cardiac cine imaging allows reliable and reproducible assessment of volumetric and functional parameters of the left and right atrium. Eur Radiol 2021; 31:7219–7230
14.
Vial J, Bouzerar R, Pichois R, et al. MRI assessment of right ventricular volumes and function in patients with repaired tetralogy of Fallot using kat-ARC accelerated sequences. AJR 2020; 215:807–817
15.
Lai P, Brau A. Improving cardiac cine MRI on 3T using 2D k-t accelerated auto-calibrating parallel imaging. (abstract) J Cardiovasc Magn Reson 2014; 16(suppl 1):W3
16.
Sandino CM, Lai P, Vasanawala SS, Cheng JY. Accelerating cardiac cine MRI using a deep learning-based ESPIRiT reconstruction. Magn Reson Med 2021; 85:152–167
17.
Cheng HLM. Investigation and optimization of parameter accuracy in dynamic contrast-enhanced MRI. J Magn Reson Imaging 2008; 28:736–743
18.
Usman M, Atkinson D, Odille F, et al. Motion corrected compressed sensing for free-breathing dynamic cardiac MRI. Magn Reson Med 2013; 70:504–516
19.
Forman C, Piccini D, Grimm R, Hutter J, Hornegger J, Zenge MO. Reduction of respiratory motion artifacts for free-breathing whole-heart coronary MRA by weighted iterative reconstruction. Magn Reson Med 2015; 73:1885–1895
20.
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1:307–310
21.
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15:155–163
22.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159–174
23.
Zucker EJ, Sandino CM, Kino A, Lai P, Vasanawala SS. Free-breathing accelerated cardiac MRI using deep learning: validation in children and young adults. Radiology 2021; 300:539–548
24.
Kido T, Kido T, Nakamura M, et al. Assessment of left ventricular function and mass on free-breathing compressed sensing real-time cine imaging. Circ J 2017; 81:1463–1468
25.
Bellenger NG, Gatehouse PD, Rajappan K, Keegan J, Firmin DN, Pennell DJ. Left ventricular quantification in heart failure by cardiovascular MR using prospective respiratory navigator gating: comparison with breath-hold acquisition. J Magn Reson Imaging 2000; 11:411–417
26.
Taylor AM, Jhooti P, Wiesmann F, Keegan J, Firmin DN, Pennell DJ. MR navigator-echo monitoring of temporal changes in diaphragm position: implications for MR coronary angiography. J Magn Reson Imaging 1997; 7:629–636
27.
Kaji S, Yang PC, Kerr AB, et al. Rapid evaluation of left ventricular volume and mass without breath-holding using real-time interactive cardiac magnetic resonance imaging system. J Am Coll Cardiol 2001; 38:527–533
28.
Klinke V, Muzzarelli S, Lauriers N, et al. Quality assessment of cardiovascular magnetic resonance in the setting of the European CMR registry: description and validation of standardized criteria. J Cardiovasc Magn Reson 2013; 15:55
29.
Larson AC, Kellman P, Arai A, et al. Preliminary investigation of respiratory self-gating for free-breathing segmented cine MRI. Magn Reson Med 2005; 53:159–168
Information & Authors
Information
Published In
Copyright
© American Roentgen Ray Society.
History
Submitted: September 18, 2023
Revision requested: October 6, 2023
Revision received: December 20, 2023
Accepted: January 29, 2024
Version of record online: February 7, 2024
Keywords
Authors
Metrics & Citations
Metrics
Citations
Export Citations
To download the citation to this article, select your reference manager software.
There are no citations for this item