In Vitro Evaluation of Cerebrospinal Fluid Velocity Measurement Agreement, Reliability, and Reproducibility in Type I Chiari Malformation Using 2D Phase Contrast Magnetic Resonance Imaging and 4D Flow Imaging In Vitro Evaluation of Cerebrospinal Fluid Velocity Measurement

Background: Phase contrast magnetic resonance imaging, PC MRI, is a valuable tool allowing for non-invasive quantification of CSF dynamics, but has lacked adoption in clinical practice for Chiari malformation diagnostics. To improve these diagnostic practices, a better understanding of PC MRI based measurement agreement, repeatability, and reproducibility of CSF dynamics is needed. Methods: An anatomically realistic in vitro subject specific model of a Chiari malformation patient was scanned three times at five different scanning centers using 2D PC MRI and 4D Flow techniques to quantify intra-scanner repeatability, inter-scanner reproducibility, and agreement between imaging modalities. Peak systolic CSF velocities were measured at nine axial planes using 2D PC MRI, which were then compared to 4D Flow peak systolic velocity measurements extracted at those exact axial positions along the model. Results: Comparison of measurement results showed good overall agreement of CSF velocity detection between 2D PC MRI and 4D Flow (p = 0.86), fair intra-scanner repeatability (confidence intervals ±2 cm/s), and poor inter-scanner reproducibility. On average, 4D Flow measurements had a larger variability than 2D PC MRI measurements (standard deviations 1.83 and 1.04 cm/s, respectively). Conclusion: Agreement, repeatability, and reproducibility of 2D PC MRI and 4D Flow detection of peak CSF velocities was quantified using a patient-specific in vitro model of Chiari malformation. In combination, the greatest factor leading to measurement inconsistency was determined to be a lack of reproducibility between different MRI centers. Overall, these findings may help lead to better understanding for application of 2D PC MRI and 4D Flow techniques as diagnostic tools for CSF dynamics quantification in Chiari malformation and related diseases.


INTRODUCTION
The dynamic movement of cerebrospinal fluid (CSF) has long been the subject of scientific investigation, and its important functional role to support central nervous system health is increasingly realized. For this reason, non-invasive phase contrast magnetic resonance imaging (PC MRI) quantification of CSF dynamics has been pursued for diagnosis, prognosis, and treatment of neurological diseases such as hydrocephalus [1,2], Chiari malformation [3], and syringomyelia [4,5]. Variabilities in CSF dynamics, such as increased CSF velocities and/or flow rate, are thought to be indicative of Chiari malformation and related neurological disorders [6,7]. Single-plane two-dimensional, through-plane encoded PC MRI (2D PC MRI) and time-resolved three-dimensional velocity encoded PC MRI (4D Flow) are promising modalities that allow for CSF characterization. 2D PC MRI is one of the best known non-invasive methods and currently the only method for both qualitative and quantitative CSF characterization [8]. Clinical application of 2D PC MRI is widely varied with use in visualizing morphological and functional alterations in normal pressure hydrocephalous patients as well as CSF flow assessment in Chiari malformation populations with and without syringomyelia [9]. 4D Flow has shown potential to advance in vivo assessment of complex hemodynamic and CSF flow patterns [10][11][12].
Originally developed for cardiovascular applications [13], 4D Flow has been applied to analyze CSF velocity differences between healthy controls and Chiari malformation patients, with and without syrinx formation [14]. Contrast-enhanced MRI techniques have also been applied to quantify relatively slow timescale transport phenomena, such as CSF solute transport in humans [15][16][17]. Additionally, MRI has been applied to quantify short timescale phenomena such as dynamic motion of CSF due to respiration and other maneuvers using real-time PC MRI [18][19][20][21] and time-slip MRI [22,23]. These methods show promise to help reveal new insights about CSF system physiology in health and disease.
At present, the diagnostic relevance of PC MRI-based measurement of CSF dynamics remains under debate by the medical community. For example, the recently published National Institutes of Health common data elements (CDEs) for Chiari malformation clinical research does not include any recommended measurements related to CSF dynamics [24]. The lack of adoption of CSF dynamics as a standard measure for Chiari malformation is likely due to the conflicting findings reported in previous studies comparing CSF velocities in Chiari malformation patients and healthy controls [25][26][27][28]. For example, some investigators report elevated CSF velocities in Chiari malformation patients' pre-surgical treatment, and others reported decreased pre-surgical CSF velocities in Chiari malformation patients compared to post-surgery. Also, there are conflicting reports of both elevated and decreased CSF velocities in healthy subjects compared to Chiari malformation patients. These conflicting findings were discussed in a review by Shaffer et al. [6].
The large variance in CSF velocities reported in Chiari malformation patients versus controls in literature is likely due to the wide range in PC MRI acquisition methods and post-processing techniques.
To address the need for improved CSF dynamics quantification, the present study aims to quantify the agreement, reproducibility, and repeatability of 4D Flow and 2D PC MRI measurement of CSF velocities at the craniovertebral junction. Our focus was the craniovertebral junction CSF velocities because these velocities are thought to potentially be a diagnostic indicator of Chiari malformation. To mitigate normal physiological variation in CSF velocities, our approach utilized a subject-specific highresolution 3D printed model of a Chiari malformation patient with computer controlled pulsatile CSF pump [36]. We hypothesized that 2D PC MRI and 4D Flow would have strong measurement agreement, repeatability, and reproducibility.

Study Design
Experiments were performed using an in vitro subject-specific CSF flow model of a Chiari malformation patient that was tested at five different MRI scanners at four different scanning centers (Center 1 through 5 in Figure 2). The centers were physically located at Emory University in Atlanta (, University Hospital in Cologne Germany (3T Achieva, Philips Healthcare, Best, Netherlands), University Hospital in Basel Switzerland (3T, MAGNETOM Prisma, Siemens Healthcare, Erlangen, Germany), and University Hospital in Lausanne Switzerland (3T Tim Trio and 3T PrismaFit, Siemens Healthcare, Erlangen, Germany). To quantify repeatability, the flow model was scanned three times at each center using both 2D PC MRI immediately followed by 4D Flow MRI. To quantify reproducibility, results were compared across the five centers. Agreement between 2D PC MRI and 4D Flow CSF velocity measurements were also quantified. Results were statistically analyzed within and across MRI centers and between measurement techniques using a mixed effects linear model.

Subject Specific In Vitro CSF Flow Model and Experimental Set-up
To control a consistent CSF flow waveform and anatomic shape across MRI measurement centers, we utilized a computer-controlled in vitro model CSF flow system previously developed by our research group [37] (Figure 3a). The model was designed based on T2-weighted anatomical MRI data collected for a five-year-old Chiari malformation patient with 6.8 mm cerebellar tonsillar descent below the foramen magnum (FM), as described in Bunck et al. [14]. The spinal subarachnoid space was manually segmented from the medulla to the upper thoracic spine based on the T2-weighted images.
Dorsal and ventral spinal cord nerve rootlets (NR) were added to the model segmentation based on exvivo anatomic measurements of nerve root location, radicular line, and descending angle. The model was printed by stereolithography with a spatial resolution of 75 µm (see Figure 3b for model dimensions).
4D Flow images were acquired to quantify the subject-specific CSF flow waveform in the same Chiari malformation patient. CSF flow rate as a function of time was quantified based on a region of interest located at the C2-C3 vertebral level. This waveform was input to an in-house designed computer-controlled oscillatory syringe pump with pulse-trigger output (for MRI cardiac gating). To allow MRI scanning, the syringe output was connected to the in vitro models via polyethylene tubing.
The pump was positioned outside of the scanner operating room with tubing connected to the in vitro model through the waveguide. Tubing was taped to the floor and scanner bed during operation to minimize tubing movement / vibrations during operation. Completed details on the in vitro system dimensions and characterization are provided by Thyagaraj et al. [37]. Scanning was repeated three times at each location. After affixing the static fluid bodies in place, each trial consisted of a 2D PC MRI scan immediately followed by a 4D Flow scan. In between subsequent trials, the model was manually repositioned within the scanner bed.

In Vitro Imaging Protocol
4D Flow and 2D PC MRI images were collected at each center using the following settings ( Table 2), adapted from a previous protocol [37]. We sought to have identical imaging parameters applied across all MRI machines and across the 4D Flow and 2D PC MRI protocol. In brief, 4D flow datasets were collected in the sagittal orientation with velocity encoding of 15 cm/s, prospective gating, 16 phases per cardiac cycle leading to a temporal resolution of 30 ms, repetition time (TR) of 7.5 ms, echo time (TE) of 4.6 ms, flip angle (FA) = 5, with 1 mm in-plane resolution and slice thickness 5 mm.
2D PC MRI data was collected at nine axial slice positions along the model located as shown in

MRI Post-Processing
Both 4D Flow and 2D PC MRI data were post-processed using GTFlow software (version 2.2.4, Gyrotools Inc, Zurich, Switzerland) by a single person at a center core lab. An eddy current offset correction was applied based on a static fluid body placed next to the in vitro model, to offset errors arising from non-uniformity of the magnetic field [38]. The flow field was also inspected and corrected for any aliasing artefacts when present. 2D PC MRI velocity data at each of the nine axial positions was exported as Matlab (version R2014b, Mathworks Inc, Natick, MA) readable files for quantitative comparison of CSF velocities. At each 2D PC MRI slice position, a 4D flow slice was selected and also exported to Matlab. The peak systolic CSF velocity for any pixel within the spinal subarachnoid space was obtained for each slice position within the CSF for all MRI centers and trials.

Statistics
Because trial, scanning center, and scan type could have significant effects, we developed the following linear mixed-effects model for each replicate: where y is the velocity measurement along the spine, s are binary covariates, s are the fixed effects, and zs the random effects. Specifically, 1 indicates whether the treatment group is 4D Flow MRI or not, each of the s with = 2, … ,5 indicates whether the measurement was taken at the th scanning center, and each of the s with = 6, … ,13 indicates whether the measurement was taken at one of the eight axial positions (C1, C2M, C2B, C3, C4, C5, C6, and C7). In this model, 0 represents the baseline, which is the mean velocity measurement from 2D PC MRI at scanning center #1 at the FM position along the spline. In other words, this model estimates the difference between another scanning center and Center 1, as well as between another axial position and FM. Additionally, z represents random effects of the scanning centers and axial slice position (note that the treatment of 4D versus 2D is assumed to be a fixed effect and not included in the random effects), which follow a multivariate normal distribution with mean of zero and a symmetric variance-covariance matrix: 13 )′ is the column vector of all the random effects, 0 is a vector of zeros, and is the variance-covariance matrix. We used the Matlab (Ver. 2019a Mathworks Corp., Natick, MA) function "fitlme" to estimate the parameters in this linear mixed-effects model and test whether each of the fixed effect sizes is significantly different from zero. If so, this would indicate a statistically significant impact on the parameter from treatment groups, scanning centers, or axial position of velocity measurements.
Using this linear mixed-effects model, we obtained p-values for the following 14 fixed effect sizes: the baseline (Center 1), scan type (4D Flow MRI or not), scanning centers (Centers 2-5), axial position of measurement (C1, C2M, C2B, C3, C4, C5, C6, and C7). We accounted for multiple comparisons by applying the Bonferroni correction where the threshold for significant p-values was adjusted to be /14, where is the experimentwise type I error rate.

MR images were collected over three trials at five scanning centers using 2D PC MRI and 4D
Flow. Trial 3 at Center 4 and trial 2 at Center 5 were excluded from analysis due to a bubble detected in the entrance tubing during scanning; all other scanning centers (Centers 1, 4, and 5) had three successful trials that were included in analysis.

Agreement of CSF Velocity Detection by 4D Flow versus 2D PC MRI
Our statistical analysis concluded that 4D Flow and 2D PC MRI are comparable methods for CSF velocity measurements at any scanning center and for any vertebral position. No evidence was found indicating disagreement (p = 0.86, Table 3) and there was moderate agreement based on regression analysis (R 2 = 0.395, Figure 5). The confidence ellipse in Figure 5 defines the region that contains 95% of all samples that can be drawn from the underlying Gaussian distribution. In all, there was an average difference of 0.02 cm/s between measurements of each scan type with a 95% confidence interval (CI) of -0.28 to 0.24 cm/s ( Table 3) and a maximum difference of 2.9 cm/s ( Figure 5). Most measurements across scan types, axial position, trials, and scanning center were within a 95% confidence interval ( Figure 5), though no individual center had perfect agreement between 4D Flow and 2D PC MRI values.
While the measurements showed no discernable trend relating to axial position of measurement, distinct clusters formed for each scanning center showing that scanning center likely effects velocity measurement.

Repeatability
Repeatability within centers was relatively consistent with confidence intervals less than ±2 cm/s (15% of the average measured value of 14 cm/s), (  Table 5) and Center 5 (2D PC MRI STD = 1.08 cm/s and 4D Flow STD = 0.92 cm/s, Figure 6, Table 5) each had a moderate degree of repeatability, and Center 2 (2D PC MRI STD = 0.93 cm/s and 4D Flow STD = 0.74 cm/s, Figure 6, Table 5) and Center 4 (2D PC MRI STD = 0.69 cm/s and 4D Flow STD = 1.38 cm/s, Figure 6, Table 5) showed the narrowest range of values and therefore the best repeatability.  Table 3). This lack of reproducibility can be seen in Figure 7, wherein 4D Flow peak systolic velocity measurements displayed worse reproducibility than 2D PC MRI peak velocity measurements. On average, peak systolic velocities at Center 1 were greater than Center 2 through 5, sequentially ( Figure 7). Each center appeared to have a relative offset value of measurements, indicating a calibration factor may be useful in future comparative studies of PC MRI measurement values.

DISCUSSION
This study quantifies agreement, repeatability, and reproducibility of 2D PC MRI and 4D Flow characterization techniques for the measurement of CSF flow velocities at the craniovertebral junction in Chiari malformation. We found that agreement between 2D PC MRI and 4D flow was good, repeatability within any one scanner was fair, and reproducibility across scanners was poor. An anatomically realistic in vitro CSF flow model was used to conduct experiments performed at five MRI scanning centers. Peak systolic velocities were found to range from 8.3 to 17.3 cm/s, which falls within the range of values reported in Chiari malformation patients ( Table 1).

Agreement
Peak systolic velocity values for 2D PC MRI and 4D Flow had overall good agreement for all centers analyzed with an average difference of 0.02 cm/s with 95% CI of -0.28 To 0.24 cm/s ( Table 3 and Figure 5). This finding supports that either technique can be used within a scanning center and the results would be comparable within exact slices. In clinical practice, a specific slice location is required for 2D PC MRI, while the slice location to be analyzed with 4D Flow is selected after image acquisition by re-slicing of the data. This provides added flexibility for analysis of CSF peak velocities that is not possible using 2D PC MRI. Our approach aimed to acquire 2D PC MRI and 4D Flow with similar spatial and temporal resolution ( Table 2). However, it was not possible to identically match all scanner parameters which may have led to some differences in results across protocols.
Notably, variance across measurements was greater in 4D Flow results than 2D PC MRI. This variance could be due to a greater accuracy of 4D Flow measurements with a loss of precision compared to 2D PC MRI. Without a "Gold Standard" known peak CSF velocity in the in vitro model, the underlying factor leading to this variance requires further research. This technique-based measurement variance can be seen in Figure 5, where most measurements lie within the 95% CI represented by the green ellipse. Here, the overall good agreement between the techniques is apparent, but the clustering of values based on scanning center reveals an important insight into the reproducibility and repeatability of techniques. These center-based clusters could be due to scanner-specific effects at each center, wherein each scanner has a quantifiable effect on the measurements it makes. With a more focused research study, these scanner-effects can be understood and potentially mitigated by use of a standardized scanner calibration technique.

Repeatability
Repeatability of measurement values within any scanner was dependent on each individual scanning center. While Centers 2 and 4 had the best repeatability (STD = 0.87 cm/s and 1.18 cm/s, respectively, Table 5) Centers 3 and 5 had moderate repeatability (STD = 1.06 cm/s and 1.25 cm/s, respectively, Table 5) and Center 1 had poor repeatability (STD = 1.50 cm/s, Table 5). This variance could be due to axial slice location relative to the model anatomy as peak velocity can vary significantly across the caudal brain and cervical spine. Therefore, some variance is expected in the model and will likely be even greater in vivo. Figure 6 provides a visual depiction of the repeatability of either technique in each center where each measurement value was subtracted from the average peak systolic CSF velocity across axial positions for each center. Specifically, the horizontal bars across each box represent the median of each dataset; the closer this median bar is to zero, the better the repeatability within that center for that scanning technique. Interestingly, the 4D Flow datasets seem to be closer to zero on average than the 2D PC MRI datasets yet the 2D PC MRI data has a narrower range of values than the 4D Flow data (STD = 1.04 cm/s and 1.83 cm/s, respectively, Table 5).

Reproducibility
Overall, the most important factor leading to measurement inconsistency in our study was lack of reproducibility across MRI scanning centers. Figure 7 shows that across axial positions, each center tended to have a relative offset based on the specific scanner used. In general, Center 1 reported the highest values for peak systolic CSF velocity followed by each other center sequentially, with Center 5 generally having the lowest reported peak systolic velocity values. This scanner-specific relative offset could be indicative of systemic difference across scanners. As mentioned above, this relative offset at each center is an important source of variance between scanning centers and could be corrected by a standardized calibration procedure. This variance between scanning centers could potentially be due to scanner specific field inhomogeneity, eddy current generated during scanning, and/or inconsistency of the in vitro experimental set up. To mitigate any potential experimental inconsistency, experiments were conducted with identical conditions across all centers including use of identical tubing, fittings, and computer controlled oscillatory pump and identical control waveform (see Methods). Additional details on the in vitro system are also provided by Thyagaraj et al. [37].

Literature Review and Comparison of Results to Previous Studies
We conducted a meta-analysis of all CSF flow studies applied in Chiari malformation ( Table 1).
These studies show a range of peak CSF velocities in healthy controls and Chiari patients depending on the measurement position along the spine, voxel size, slice thickness, velocity encoding value (VENC), and number of phases. . This meta-analysis shows peak CSF velocities are elevated in Chiari malformation compared to healthy subjects and the axial position of greatest CSF velocity elevation is most commonly reported at the C1 vertebral level (Figure 1). However, the standard deviation of peak CSF velocities is considerable compared to group differences and this variance makes specification of a diagnostic threshold for patients versus controls difficult.
Reproducibility and repeatability of CSF velocity measurements, measured in cm/s for individual voxels collected for a region of interest at the craniocervical junction, have not been specifically investigated. A number of studies have been conducted on the reliability of arterial hemodynamics using 4D Flow [39] and 2D PC MRI [40,41] measurements. However, arterial flow velocities are typically one order of magnitude greater than CSF velocities. Thus, the reproducibility / repeatability results from these arterial hemodynamics studies are difficult to apply for CSF velocities. Repeatability of 2D PC MRI CSF and cerebral blood flow (mm 3 /s) measurements have been investigated and shown to have moderate in vivo test-retest repeatability [42]. In that study, the authors did not quantify reliability of CSF velocity measurement (cm/s) that has been a focus of interest for CSF-based Chiari malformation diagnostic tests. Repeatability of in vivo 2D PC MRI measurements of CSF flow at the aqueduct of Sylvius has been examined and found to have moderate repeatability [42]. However, aqueductal CSF velocities are typically greater than at the craniocervical junction. Also, the CSF space geometry at the craniocervical junction is more complex than the tube-shaped aqueductal geometry. The craniocervical junction anatomy is an annulus shape that contains spinal cord nerve roots, neuroaxis curvature, and tonsillar descent in Chiari malformation patients. Poor 4D Flow accuracy has been found during timeframes corresponding to low CSF flow rate [30], but further research is necessary before clinical application is feasible. While many studies have previously quantified repeatability and operator effects for PC MRI hemodynamic characterization [43][44][45][46][47], few studies have quantified these parameters for PC MRI CSF dynamics characterization (Table 5). Overall, these previous CSF dynamics studies are stratified into focuses on the cerebral aqueduct, the spinal subarachnoid space (SAS), and the C2-C3 area and are summarized in Table 5. These studies consistently reported strong intra/inter operator agreement and peak velocity measurements are independent of the operator, therefore intra/inter-operator effects are null in this context and were not investigated. A study by Tawfik et al. [48] detailed 2D PC MRI measurement repeatability at the cerebral aqueduct and reported a peak velocity standard deviation of 1.9 cm/s, which is comparable to the 1.83 cm/s peak velocity standard deviation we found in the cervical spine ( Table 5). In vivo studies by Sakhare et al. [42] and Luetmer et al. [49] reported standard deviations of 2D PC MRI CSF flow between 0.04 and 0.98 mL/s but did not look at peak velocity values.
Pahlavian et al. [30] performed an accuracy study on 4D Flow quantification of CSF dynamics using a 3D printed in vitro model similar to the one used here and found fairly high accuracy (95% CI ±1.8 cm/s, Table 5) but did not quantify repeatability nor reproducibility of measurements. These accuracy results from Pahlavian et al. were of similar range as the reproducibility results of this study. To our knowledge, this is the first study to specifically detail the agreement between 2D PC MRI and 4D Flow quantification of peak CSF velocities and characterize reproducibility of measurements across different scanners.

Relevance of Findings to Clinical Diagnostics for Chiari Malformation
A meta-analysis of similar studies in literature and previous investigations of healthy and Chiari CSF dynamics reveal import insights for the clinical application of novel PC MRI peak velocity quantifications in the cervical spine. Figure 1 shows that these previous investigations of CSF dynamics reported consistently elevated peak CSF velocities in Chiari patients compared to healthy controls at every vertebral level, with a maximum difference of 6.9 cm/s at the C1 position. This difference points towards an underlying physiology of Chiari malformation at the C1 vertebral position that could be leveraged for improved diagnostics pending reliable detection, which requires disagreement between groups to be less than the effect size. Good agreement between 2D PC MRI and 4D Flow measurements indicates both methods would be acceptable in clinical use to characterize CSF dynamics. We also found intra-scanner repeatability of either measurement type to be good, but inter-scanner reproducibility was poor. This lack of reproducibility may help us understand previous studies with conflicting results regarding Chiari CSF dynamics. Mitigation of the lack of reproducibility across scanners could be achieved with a standardized calibration procedure such as generating scanner specific reference values for healthy volunteers.

Limitations
Several limitations have been identified within this study, the use of an in vitro model being the primary limitation. We utilized an in vitro subject specific model of a pediatric Chiari patient, but in vivo studies are needed to understand the full range of physiologically-rooted variability that can occur such as the impact of respiration, movement artefacts, etc. The use of a pediatric Chiari patient for a subject specific model results in data that is not representative of all conditions and individual anatomies, limiting the application of these results to adult populations and other disease populations. This model used one representative flow waveform to control the oscillatory pump, which introduces further specificity of these results and limits a broader application. This study was focused on measurement agreement, repeatability, and reproducibility and did not quantify a "Gold Standard" measurement for quantification of accuracy. Accuracy could be quantified by independent measurement of CSF velocities by flow measurement techniques such as laser Doppler anemometry, but was outside the scope of this study. The use of MRI scanners from different companies introduces further variance and limitations to this study. While four of the scanners used for imaging were manufactured by Siemens, one scanner was manufactured by Philips. The use of different scanners was unavoidable here, and the effects of this decision should be delineated further with more research on imaging reproducibility across scanning locations and scanner type to properly characterize these factors.

reproducibility of 2D PC MRI and 4D Flow quantification of peak CSF velocities in a representative
Chiari malformation patient craniocervical junction geometry. The single greatest factor leading to measurement inconsistency of peak CSF velocities was lack of inter-scanner reproducibility. Taken in combination, the results help identify sources of error that can be improved to allow better application of CSF velocity detection for medical diagnostic purposes. Overall, both 2D PC MRI and 4D Flow techniques show promise as diagnostic tools to quantify CSF dynamics in Chiari malformation.

Ethics approval and consent to participate
Not applicable.

Consent for publication
Not applicable.

Availability of data and material
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interest
BAM is a full-time employee at Alcyone Therapeutics and has received grant support from Genentech,    Table 1 for individual values.

Figure 2.
Overall study design. In vitro models were sent to five imaging centers where they were scanned three times with two techniques, 2D PC MRI and 4D Flow MRI, before the velocity measurements were analyzed using a mixed effects linear model.
Computer controlled pulsatile flow pump.
Printed 3D model with nerve rootlets.

Scan Subjects
Anatomical T2 MRI

Print 3D Model
Add Nerve Rootlets.

Render 3D Model
Segment Geometry.

CSF Flow Waveform Control
Scan Models (a)    Table 1. Literature review of in vivo PCMRI measurements of peak CSF velocities in healthy (H) and Chiari malformation patient (P) cases (Note: the peak velocities denoted by an asterisk were measured at points/probes and not throughout the axial plane).