Function-Structure Associations of the Brain: Evidence from Multimodal Connectivity and Covariance Studies (2024)

The publisher's final edited version of this article is available at Neuroimage

Prevalent brain imaging modalities

High-resolution T1-weighted imaging (which we will refer to as sMRI from now on), is the most common method for depicting structural properties of the brain, which enables the assessment of differences in the local concentration or volume of gray matter (GM) and white matter (WM) at each voxel, by using approaches such as voxel-based morphometry (VBM) (Ashburner and Friston, 2000), voxel-based cortical thickness (VBCT) (Haier et al., 2009), or higher order morphometric and shape changes through programs such as FreeSurfer (Fischl, 2012). DTI, on the other hand, for a given voxel, measures the directional diffusion of water molecules. Common parameters derived from DTI are fractional anisotropy (FA) and mean diffusivity (MD), which refer to the overall strength of water diffusion and its directedness regardless of its specific orientation, respectively. Note that tractography based on DTI cannot directly image multiple fiber orientations within a single voxel. Because if a diffusion tensor is calculated, only one direction for the fiber is obtained in the voxel given by the principal eigenvector, and the orientation distribution function (ODF) is a delta function. To address this limitation, a number of methods have been proposed to measure ODF's based on the angular resolution requirement to resolve closely aligned fiber bundles. Among others, these methods include diffusion spectrum imaging (DSI) (Wedeen et al., 2005), Q-ball imaging (Tuch, 2004), and a probabilistic method based on Monte-Carlo simulations (Behrens et al., 2003). Particularly, DSI and related methods were developed to image complex distributions of intra-voxel fiber orientation (Johansen-Berg and Rushworth, 2009). DSI relies on more accurate assumptions regarding the typical structure of white matter, thus enables looking at crossing or kissing fibers (Tefera et al., 2013), and has shown structural basis of functional cerebellar circuits in the human cerebellum in vivo (Granziera et al., 2009). Structural imaging can also be performed with diffusion-weighted imaging (DWI) and fiber tractography, which uses the passive diffusion of water molecules to infer properties of the surrounding tissue (Roberts et al., 2013). Recently, DWI has been increasingly used for its ability to assess WM microstructure and pathways of the whole brain in vivo (Jones, 2008). In this paper, we use diffusion MRI (dMRI) to denote all abovementioned diffusion imaging methods.

In the functional domain, fMRI measures dynamic changes of the hemodynamic response related to neural activity in the brain. Using blood oxygenation-level dependent (BOLD) imaging, changes in regional blood flow predominantly resulting from increased synaptic activity of neurons ultimately leads to an increase in the fMRI signal. Another prevalent method is EEG, which assesses brain electrical activity only at electrodes placed on the scalp. Yet, its higher temporal resolution is beneficial for the estimation of changes in functional network connectivity. Note that the vast majority of EEG studies focus on electrophysiological activity corresponding to frequencies well below 100 Hz, which similar to fMRI also has been shown to reflect synaptic rather than spiking activity of neurons (Hughes, 1996; Satherley et al., 1996).

In many multimodal studies, data from each modality is first reduced to a feature space, which is a lower-dimensional representation of selected brain activities, and subsequently to discover the correlation along the feature datasets by subjects' variations (Calhoun and Adali, 2009). Incorporating fusion analysis at the group level (across multiple subjects) is quite promising because weak (but important) cross modality relationships along with inter-subject variations can be revealed. Such feature-based approaches provide a natural way to find multimodality associations.

fMRI – dMRI

DTI has become a popular tool for the investigation of white matter architecture in the normal brain, while fMRI, as a well-established neuroimaging technology, may act as a reference framework for validating conclusions derived from the relatively newer DTI method (O'Donnell et al., 2011; Reinges et al., 2004). Reports have validated that there are networks of brain regions where maturation of white matter and changes in functional activity show similar developmental trends during human childhood (Olesen et al., 2003), thus it is plausible to assume that functional and anatomical brain properties covary across subjects, an assumption that may well generalize to brain pathology. Hence combining DTI and fMRI may also provide unprecedented opportunities to deepen our understanding of brain disorders (Kim et al., 2012b; Mascalchi et al., 2012; Soldner et al., 2011; Tang et al., 2013).

Functional-structural connectivity

Many of the conducted fMRI-DTI studies addressed disruptions of brain connectivity seen with mental illnesses such as depression, schizophrenia (SZ), Alzheimer's disease and bipolar disorder. In particular, (Matthews et al., 2011) examined structural and functional neural correlates of soldiers with a history of blast-related concussion and who concurrently suffered from major depressive disorder (MDD) and compared them to a matched non-MDD group. They utilized DTI and an emotional face matching fMRI task for their investigation. MDD relative to non-MDD individuals showed greater activity in the amygdala and other emotion processing structures, but lower activity in emotional control structures such as the dorsolateral prefrontal cortex (DLPFC) during fear matching trials, as well as lower FA in WM tracts such as the superior longitudinal fasciculus (SLF). These FA values correlated negatively with depressive symptom severity. Schlosser et al. observed a direct correlation in schizophrenia between frontal FA reduction and fMRI activation in regions of the prefrontal and occipital cortices (Schlosser et al., 2007). This finding highlights a potential relationship between anatomical changes in a frontal-temporal anatomical circuit and functional alterations in the prefrontal cortex. A recent study of Koch et al. (Koch et al., 2011) showed that WM fiber integrity in terms of increased radial diffusivity of the left superior temporal gyrus (STG) is associated with reduced neuronal activation during decision making in lateral frontal and cingulate cortices as well as the dorsal striatum, hypothalamus, left and right cerebellum and the right insula. These findings clearly suggest that intact white matter connectivity plays an important role for the pattern and intensity of functional activations with neuronal networks engaged in decision making and error-related processes.

(Skudlarski et al., 2010) evaluated interactions between measurements of anatomical and functional connectivity collected in the same subjects to study global schizophrenia-related alterations in brain connectivity. Although patients showed deficits in white matter anatomy, functional connectivity alterations were more complex and a decoupling between structural and functional connectivity was found with networks originating in posterior cingulate cortex, the task-positive network, and one of the default mode network (DMN) components (shown in Figure 1).

(Zhou et al., 2008) studied both functional and anatomical connectivities between hippocampus and other regions by utilizing resting fMRI and FA. Bilateral hippocampi showed reduced functional connectivity in schizophrenia compared to regions involved in episodic memory, such as posterior cingulate cortex (PCC), extrastriate cortex, medial prefrontal cortex, and parahippocampus gyrus. Similarly, mean FA of the fornix body was significantly reduced in patients, indicating concurrence of reductions in functional and anatomical connectivity. Similarly, (Yan et al., 2012) investigated the functional and anatomical connectivity of the anterior cingulate cortex (ACC) by using resting fMRI and DTI. Patients with schizophrenia showed abnormal hemispheric asymmetry regarding their functional connectivity profiles of the ACC-cd with multiple brain areas, as for example increased negative connectivity with the left PCC. Mirroring the functional observations, FA of the right anterior cingulum also was significantly decreased in the patient group.

(Soldner et al., 2011) determined associations of structural integrity in the PCC as assessed by DTI and its functional connectivity with both the hippocampus and para-hippocampus during resting state in Alzheimer's disease (AD), mild cognitive impairment (MCI) and healthy controls (HCs). Results suggest that under healthy conditions, effective connectivity in the default mode network (DMN) between PCC and hippocampus is primarily maintained by an indirect pathway via the parahippocampal gyrus. Patients with AD and MCI showed changes in this connectivity with a partial allocation to the direct pathway, most likely reflecting early parahippocampal lesions. In another functional connectivity study, Wang et al. investigated potential disruptions in perigenual anterior cingulate cortex (pACC)-amygdala functional connectivity and corresponding abnormal structural connections in bipolar disorder (BP) during processing of face stimuli (Wang et al., 2009). As expected, functional connectivity was decreased between the pACC and amygdala in bipolar participants compared to controls during processing of both fearful and happy faces. Moreover, a significant positive association between pACC-amygdala functional coupling and FA in ventro-frontal white matter, including the region of the uncinate fasciculus was found; suggesting that disruptions in functional and white matter connectivity may contribute to disturbances in the coordinated responses of the pACC and amygdala during emotional processing in BP. (Staempfli et al., 2008) also illustrated that DTI- and fMRI-derived topologies are similar, and that the fMRI-DTI combination can provide additional information in order to choose reasonable seed regions for identifying functionally relevant networks and to validate reconstructed WM fibers.

Finally, (Bosnell et al., 2011) tested whether practice-related changes in brain activity differ after stroke by fMRI and DWI, in order to explore spatial relationships between activity changes and patterns of structural degeneration. Results indicate that performance in stoke patients gains with motor practice, which can be associated with increased activity in regions that have been either directly or indirectly impaired by loss of connectivity, suggesting that neurorehabilitation interventions may be associated with enhanced activity in regions with impaired structural connectivity.

Inter-modality covariance

In contrast to the above correlational analyses, multivariate data-mining can also be used to access the joint information of multiple measurement modalities by assessing inter-modality covariance across subjects. Specifically, joint independent component analysis (jICA) was used in (Franco et al., 2008) to combine amplitude of low-frequency fluctuations (ALFF) and FA. Their results indicated that activity in the DMN highly depends on the integrity of WM connections between the two hemispheres (connected via the corpus callosum) and within the cingulate bundles. In addition, Sui et al. applied an “mCCA+jICA” approach (Sui et al., 2011), an optimized model for identifying correspondence across modalities, to compare patients (SZ and BP) and HCs based on task-related fMRI contrast maps and FA data. The authors identified distinct regions that could be used to discriminate between controls and patients, specifically, the DLPFC and motor regions in fMRI and the WM fiber tracts including anterior thalamic radiation (ATR), SLF and inferior frontal-occipital fasciculus (IFO). Further, separation of the SZ and BP groups was established by differences in hippocampal, prefrontal and visual cortex impairments in fMRI combined with disrupted WM integrity in SLF, forceps minor and forceps major. A high-level functional-structural network diagram was also derived to identify which known tracts are both intersected by the regions of FA changes and touch the regional fMRI changes. (Camchong et al., 2011) used a hybrid ICA method to extract the group DMN and accompanying time-courses from resting fMRI but also computed a voxel-wise statistical analysis of FA data to look for between-group differences by means of voxel-wise regressions/correlations. Results revealed convergent fMRI and DTI findings consistent with the disconnection hypothesis in schizophrenia, particularly in medial frontal and anterior cingulate regions, while adding some insight of the relationship between differences in brain connectivity and behavior.

The above findings were facilitated by advanced modeling techniques and could not have been revealed through separate uni-modal analyses as typically performed in the majority of neuroimaging experiments. Importantly, fusion of fMRI and DTI data may improve brain disease classification. For example, MCI, often an early stage of AD, is difficult to diagnose due to its rather mild and nearly insignificant symptoms of cognitive impairment. (Wee et al., 2011) integrated information from DTI and resting fMRI by employing multiple-kernel support vector machines (SVMs), yielding statistically significant improvement (>7.4%) in classification accuracy of predicting MCI from HC by using multimodal data (96.3%) compared to using each modality independently. There are additional studies that demonstrate the potential of the fusion of structural and functional data combined with multi-modal classification techniques to provide more accurate and early detection of brain abnormalities (Fan et al., 2008).

fMRI - sMRI

Historically, the combination of structural and functional brain imaging has been used for the purpose of analyzing functional activity in a priori defined brain regions. Whereby the anatomical T1-weighted image is typically used as a reference for extracting or locating ROIs or for the purpose of placing seed-points for cross-correlational functional analyses (Tian et al., 2011). However, much is to be gained if the T1 weighted image is not only used as a mere reference template (Schultz et al., 2012). For example, when trying to identify MCI patients in a classification study, (Kim and Lee, 2012) showed that the integration of sMRI and fMRI can provide complementary information to improve the diagnosis of MCI relative to either one alone (error rate: 6% using both versus 15% using fMRI only and 35% using sMRI only).

Functional-structural connectivity

A central assumption of systems neuroscience is that the structure of the brain can predict and/or is related to functional connectivity. The findings of Segall et al., support this hypothesis. In that particular study, spatial ICA was applied separately to both gray matter density maps derived from T1-weighted images and resting state (rs-) fMRI data from a large dataset of healthy adults. Then, the decomposed structural and functional components were compared by spatial correlation. The results generally show that each single structural component usually corresponds to several resting-state functional components (see Figure 2. for an example) thereby, elucidating the relationship between brain function at rest and GM density maps (Segall et al., 2012). In (Khullar et al., 2011), functional information is able to improve the correspondence of functional boundaries across subjects beyond the standard structural normalization.

One area of research that has gained much from the union of anatomical and functional images is human brain development, where longitudinal studies have demonstrated the synchronicity between structural and functional changes during childhood and adolescence (Casey et al., 2005). Studies of psychopathological phenomena discovered spatial overlaps of structural and functional alterations in schizophrenia or at risk mental state using cognitive tasks and GM volume. (Salgado-Pineda et al., 2004) found three regions including the thalamus, the anterior cingulate and the inferior parietal that showed both structural and functional impairments associated with attentional processing in schizophrenia. A follow up study of the same group (Salgado-Pineda et al., 2011) also found both functional alterations (facial emotion task) and GM volume reductions in the DMN in schizophrenia. (Smieskova et al., 2012) discovered that reduced insular and prefrontal activation (during an N-back task) was associated with a reduction of GM volume in a at risk mental state group.

Additional studies suggest that alterations in neuronal functioning are directly related to alterations in gray matter structure (Schultz et al., 2012). For example, using a version of the tower of London task, (Rasser et al., 2005) found a positive correlation of neuronal activation and cortical gray matter thickness in prefrontal/frontal and parietal areas in first-episode schizophrenia patients, whereas this correlation in most parts was inverted in healthy controls. When examining the relationship between neuronal activation during an N-back working memory task and GM volume in 15 ultra-high-risk subjects and 15 matched HCs, (Fusar-Poli et al., 2011a) found that reduced prefrontal activation during the task was associated with a reduction in GM volume in the same area in high risk subjects only. Using a whole-brain correlational approach, Michael et al, found that the linkage between GM volumes and functional activation (derived from an auditory sensorimotor task) is stronger in HC than in SZ patients (Michael et al., 2010). In a subsequent study of the authors (Michael et al., 2011) investigated the association of neuronal activity during the Sternberg working memory task and GM. The results demonstrated a differential pattern of structure-function associations across subject groups in anterior cingulate, temporal regions and the cerebellum; with negative correlations in SZ and positive correlations in HCs. Patients exhibited a concordance of increased neuronal activation and diminished gray matter concentration.

Inter-modality covariance

Multivariate, data-driven approaches have also been applied in the context of fMRI and sMRI fusion. For example, (Calhoun et al., 2006) analyzed data of schizophrenia patients and healthy controls using jICA with GM volume maps and fMRI task data. Calhoun et al., found that GM group differences in bilateral parietal and frontal as well as posterior temporal regions distinguished groups. Patients had reduced gray matter concentrations (GMC) and less hemodynamic activity during target detection in bilateral anterior temporal lobe regions. An unexpected corollary to their finding was that, in those regions showing largest group differences, GMC were higher in patients than controls, suggesting that higher GMC may be related to lower functional connectivity during performance of an auditory oddball task. (Correa et al., 2008) also showed an interesting joint relationship between fMRI and GM volume by multimodal canonical correlation (mCCA), with schizophrenic patients showing more functional activity in motor areas and less activity in temporal areas. Additionally, both functional observations coincided with reduced GM volume when compared to healthy controls.

EEG – dMRI

There are few studies which attempted to directly associate parameters inferred from DTI with electrophysiological activity at rest. Most other studies only permitted indirect conclusions, for example, by studying age-related changes in functional connectivity, or by comparing patient groups with neurological impairments to healthy controls. Not surprisingly, all relevant studies tried to identify the physiological prerequisites that determine the individual alpha frequency (IAF), that is, the frequency at which an individual exhibits highest activity in the alpha range (8 - 12 Hz).

The majority of studies addressing structure-functions associations utilizing EEG in task-contexts either focused on the relevance of WM characteristics for interhemispheric transfer of information or cognitive control processes, often using DTI to assess structural connectivity. Interhemispheric transfer time (IHTT) and inter-hemispheric signal propagation (ISP) are two measures used to quantify the quality of the transmission of signals across the corpus callosum. IHTT with EEG is computed as the latency-difference between sensory evoked potentials (e.g., the visual P100) between contralateral and ipsilateral electrodes, when stimuli are presented to only one hemisphere in a given trial. ISP, on the other hand, quantifies the similarity (as the ratio) of signals measured above the different hemispheres after unilateral stimulation, with values around one indicating a good signal transfer. (Westerhausen et al., 2006), for example, found a negative correlation between IHTT and MD in the posterior third of the corpus callosum, with sensory event- related potentials (ERPs) measured at parieto-occipital electrodes. Interestingly, such correlations were not found with IHTT estimates derived from reaction times. Similarly, (Whitford et al., 2011) observed callosal fibers connecting visual areas in both hemispheres being isolated by means of tractography. In Whitford et al., the ERP-derived IHTT was significantly explained by both FA and total diffusion within a linear regression model, even after correction for putative influences such as age, fiber length and curvature. (Voineskos et al., 2010) extended these results by computing ISP for transcranial magnetic stimulation evoked potentials generated in the DLPFC and the motor cortex. Again, the tracts connecting hom*ologous regions were extracted and mean FA values for these tracts were computed. Not only were associations indicating better signal transfer with higher FA, but the pattern was also regionally specific; i.e., correlations of ISP with fibers connecting the motor-areas were found when the motor-cortex was stimulated but not when the DLPFC was targeted and vice versa. Moreover, (Chatzikonstantinou et al., 2011) analyzed features of acute DWI abnormalities related to status epilepticus by combining DWI and EEG analysis, which provided clues to seizure localization and propagation, as well as to identify brain structures affected by continuous or frequent ictal activity.

Teipel and colleagues conducted one of the first studies combining resting state-EEG and DTI measurements, through the use of a multivariate network based on principal component analysis (PCA) to determine the effects of coherence on the regional pattern of diffusivity (Teipel et al., 2009). For both MCI and HC, higher temporo-parietal IAF coherence was associated with increased FA but decreased MD in posterior parts of the corpus callosum and in parietal and occipital WM tracts. In the following year, (Valdes-Hernandez et al., 2010) published data which indicated side- and region-specific effects when correlating IAF and FA values. Again, correlations of the occipital IAF were found significant with rather posterior WM tracts, e.g. the superior corona radiata as well as the inferior occipital and longitudinal fascicule. However, correlations in the corpus callosum showed sign variability with positive associations in the isthmus but negative correlations in the splenium. Although the exact nature of this effect is elusive, Valdes-Hernandes et al suggested that either differences in the function of callosal subregions may play a role (excitatory vs. inhibitory connectivity) or that a given IAF might show differential associations depending on a region's time delay when transmitting signals [see also (David and Friston, 2003)]. Hence, resting state EEG data indicate that higher IAF and its coherence is associated with microstructural measures indicating higher conduction velocity, putatively suggesting an increased efficiency in trans-regional communication. This is in accordance with the notion that higher IAF is associated with increased performance in difficult cognitive tasks [e.g., (Klimesch et al., 2003); (Zoefel et al., 2011)]. Yet, the observation of inverted relationships within hom*ologous regions of the two hemispheres awaits replication and further investigation.

Additional DTI-EEG studies with a focus on higher order cognition tried to elucidate the microstructural determinants of brain processes associated with errors. When participants in a task produce an erroneous response, scalp EEG shows a fronto-medially pronounced effect, manifested as negative going event-related potential, the so-called error-related negativity (ERN), corresponding to an increased activity in the theta band e.g. (Cavanagh et al., 2012; Gruendler et al., 2011). The mid-cingulate cortex is believed to be a generator of this response-locked phenomenon. (Westlye et al., 2009) found an association of the ERN amplitude with FA values in the left cingulate bundle, whereby a higher ERN was associated with increased FA, an effect that seemed to be driven by lowered diffusion perpendicular to the main orientation of the bundle. Studying fronto-medial theta in response to errors, (Cohen, 2011) made similar findings; higher theta power was associated with stronger tract connectivity, between regions that are estimated to generate the electrophysiological response and other regions altogether forming a system underlying cognitive control (e.g., the striatum and the ventrolateral frontal cortex). Even more, participants whose fronto-medial theta activity was functionally coupled to many other electrodes on the scalp (indexed by increased phase synchronization) also showed stronger tract connectivity from theta generators through the corpus callosum to superior frontal regions.

EEG - sMRI

Instead of addressing associations of EEG features with micro-structural properties of the white matter, one can assess the relevance of inter-individual differences in regional GM or WM volumes. Again, similarly to joint DTI- EEG studies, inter-hemispheric processing seems to be of special interest. (Zaehle and Herrmann, 2011), for example, used a visual detection task to assess, whether the individual frequency of the stimulus evoked gamma band response (≈ 40 Hz) over occipital areas is associated with regional volume of the corpus callosum. Indeed, positive correlations were found for the truncus and splenium of the corpus callosum, indicating a higher frequency of the evoked gamma band response with regionally increased white matter volume. Similarly, (Fusar-Poli et al., 2011b) found that the auditory oddball P300, measured in a sample of healthy control subjects and participants at risk for developing a mental disorder, were positively correlated with the volume of the genu. Both of the aforementioned studies used voxel-based morphometry for their assessments. (Huster et al., 2011), though, reported negative correlations of the stop-related P300 observed in a lateralized somatosensory stop-signal task and the cross-sectional area of the truncus of the corpus callosum. A study that looked at associations of P300s with both, white and gray matter variability (Cardenas et al., 2005), found that latencies of the P3b increased with decreased white matter volume in tracts connected to the thalamus and central regions, as well as parts of the corpus callosum. Relevant correlations with GM volume were not reported.

Yet, already a decade earlier (Ford et al., 1994) reported that global GM volumes of the frontal and parietal lobes showed differential associations with amplitudes of P300s elicited in a three stimulus oddball task: whereas P300s under conditions of automatic attention capture correlated with frontal cortical volume, P300s under effortful attention were associated with parietal lobe volume. Correlations with GM volume have also been reported for phase synchronization in the gamma band or event-related potentials observed with memory tasks. (Williams et al., 2005), for example, found differential gamma band phase correlations for female and male participants in an auditory oddball task. Whereas women exhibited positive correlations with local gray matter volume in frontal and parietal cortices, men rather showed negative correlations. This was also the case in a task probing the electrophysiology of the recollection of faces (Schiltz et al., 2006), where larger hippocampal volume and diffusion was positively correlation with slow-wave amplitudes between 275 and 325 ms after stimulus presentation. In addition, hippocampus morphology was also positively correlated with behavioral indices in a nonverbal learning task.

Other studies did not so much directly focus on volumetric effects, but rather studied the relevance of gross-morphometric characteristics of the cortex, such as the degree and asymmetry of cortical folding patterns. The mid-cingulate cortex exhibits a variable degree of asymmetry in cortical folding across subjects, with most subjects showing a leftward asymmetry due to the occurrence of a second (superior- or paracingulate gyrus (Huster et al., 2007; Vogt et al., 1995). It has been shown in a series of experiments (Huster et al., 2012; Huster et al., 2011; Huster et al., 2009) that this leftward asymmetry is accompanied with higher amplitudes of fronto-medial negativities as observed in Stroop, go/no-go, and stop signal tasks. This effect is believed to originate from larger extent and/or volume of cytoarchitectural area 32 (Vogt et al., 1995) and furthermore coincides with increased behavioral performance measures of interference control.

Three-way Fusion

General Aspects

Various neurophysiological techniques used to assess functional/structural connectivity such as fMRI, sMRI, diffusion MRI and EEG/MEG, provide data at different spatial and temporal scales. Hence combining multimodal imaging data provides unprecedented opportunities to further deepen our understanding of the patho-physiological core features of brain disorders(Sui et al., 2013) and provide useful information on how these separate elements relate to each other, such as gray and whiter matter alterations, aberrant electrophysiological activity and altered resting-state/task-related functioning. Although the number of multimodal imaging studies is still limited, initial findings are promising (Rykhlevskaia et al., 2008; Sui et al., 2012d).

All of the multimodal studies that are reviewed in this paper are summarized and separated into different modality categories in Table 1. In general, most studies we reviewed demonstrate congruent effects across measurement modalities and combining modalities does provide more differentiating power among multiple diseases. Within the context of clinical research, structure–function relationships may be reformulated as links between structural brain damage and its functional or behavioral consequences (Kolb and Whishaw, 1998; Schlosser et al., 2007; Wang et al., 2009). These structure-function associations are not regionally unspecific but follow a topological pattern; furthermore, they reveal spatial overlaps across different brain pathologies (Bullmore and Sp*rns, 2009; Keightley et al., 2012; Toosy et al., 2004). For example, the medial frontal region stands out as region of central importance for schizophrenia and bipolar disorder (Camchong et al., 2011; Sui et al., 2011). In addition, most studies stress the relevance of fiber tracts in direct vicinity to or those connecting functionally active regions of the cortex (Leergaard et al., 2012). A good example is the quasi-topological ordering of functional associations and callosal fibers. Many of these effects have even been validated while controlling for global variables such as sex, age, and most importantly, intracranial volume or brain size.

Despite the rather limited number of studies that have investigated putative structure-function associations, a clear pattern has emerged. Overall, data indicate that features suggesting an increased connectivity between regions tend to be associated with larger EEG or fMRI signal amplitudes, higher frequencies of intrinsic oscillations, shorter transmission latencies, or stronger functional connectivity. Such features usually refer to microstructural aspects of white matter tracts of which FA is the one most often applied. Most of these DTI measures suggest higher axonal density or an augmented degree of myelination to be an underlying factor. Additionally, pure volumetric effects, whether assessed for white or for gray matter, have been reported as well and they largely point in the same direction. Less common studied features, such as the regional folding of the cortex, reveal similar effects; possibly, because they might as well be traced back to mere volumetric phenomena or because folding patterns themselves result from white matter architecture (Toro and Burnod, 2005).

In this review, many of the structural imaging findings are based on extracted features, i.e. brain morphology obtained by VBM from T1-weighted imaging, and FA values calculated from DTI. These approaches have some limitation, for example VBM does not directly measure volume or cortical thickness and FA does not provide directional information. On the other hand, several more advanced techniques have been increasingly employed in neuroimaging studies, which may have a better chance of characterizing the brain tissue structure. For example, quantitative T1 mapping is one approach which can provide improved segmentation of GM and WM; T1 weighted data can be quantitatively compared across sites and longitudinally with brain development, and for most cortical brain areas (Sereno et al., 2012). In addition, as a measure of local variations in grey matter, VBCT and VBM yield very consistent results but there is evidence that VBCT provides a more sensitive measure than VBM in grey matter (Hutton et al., 2009) which is likely to improve substantially over the next few years with use of quantitative T1 mapping and higher spatial resolution images, for which VBM will be seen as an gradually inappropriate tool (Sereno et al., 2012).

Similarly, DTI as a rather approximate technique (Jones et al., 2013), is unable to resolve complex configurations of fiber tracts, limiting its utility for constructing detailed, anatomically-informed models of brain structure. Therefore, the multimodal fusion results published thus far that involved results derived from DTI should be treated and interpreted carefully, because the basic assumption of this analysis technique is not accurate enough in most white matter voxels. In contrast, DWI with fiber-reconstruction algorithms could trace large WM pathways and be used to explore patterns of WM projections among different brain regions. Recent findings suggest that diffusion-weighted imaging might even be used to measure functional differences in water diffusion during task performance (Jeurissen et al., 2012; Roberts et al., 2013). Furthermore, DSI and fiber tractography is able to reproduce known neuroanatomy with precision and accuracy as indicated in (Phillips et al., 2012). This advantage is partly due to data acquisition procedures: while many DTI protocols measure diffusion in a small number of directions (e.g., 6 or 12), DSI can assess diffusion in 257 directions and at a range of magnetic gradient strengths. In summary, diffusion weighted MRI(Behrens and Johansen-Berg, 2009) carries invaluable in vivo information about tissue microstructure, but in order to extract this information in the most efficient and unbiased way, it is important to make the right choices for the acquisition and analysis of these data, and, even more importantly, for the interpretation of the results, as advised in (Jones et al., 2013).

Methodical Considerations

It is abundantly clear that there is a diverse and growing collection of scientific tools available for non-invasively studying human brain functioning and relating it to cognitive and behavioral measures. Using these technologies, substantial progress has been made in characterizing structural/functional brain abnormalities and their interactions. In addition, a major goal in integrating/fusing approaches is to capitalize on the relative strengths of each modality, providing results synergistically.

Correlational analyses represent an important element in multimodal neuroimaging. Some are based on pure correlations, such as (Eichele et al., 2005; Michael et al., 2010), some are based on predictions, e.g., multivariate regression, relevance vector machine (RVM), as reported in (De Martino et al., 2010; Valente et al., 2011), which often incorporate testing and training phases and thus provide additional information in terms of stability and reproducibility of the results. Also, structural equation modeling (SEM) or dynamic causal modeling (DCM) can be used to examine the associative structure between functional and structural variables (Goncalves and Hall, 2003). However, such linear ROI-based approaches may miss important nonlinear connectivity links and do not provide information about inter-voxel relationships (Oakes et al., 2007; Schlosser et al., 2003).

The availability of multivariate data-driven approaches provides novel ways to investigate the inter-subject covariance among multiple modalities. They include, but are not limited to principal component analysis (PCA), ICA, CCA, and partial least squares (PLS). Data driven approaches are quite useful for identifying interesting relationships among modalities. Such an approach is complementary to model-based approaches which enable targeted questions and incorporation of potentially useful assumptions based on known properties of the data(Sui et al., 2012a). Informed data-driven approaches are in-between these two extremes and will likely play an increasing role in the future. In addition, the correct choice of component number is often ambiguous, with interpretations depending on the shape and scale of the distribution of points in a data set, thus the easiness of interpretation of the results that often is left to the experimenter (selection of components etc), though there are several categories of methods can help for making this decision, such as improved minimum description length (MDL) (Li et al., 2007). In addition, high model order studies have started regularly evaluating the cross-correlation among components which enables one to visualize the relationship among components even in the case where there may be no ideal model order (Allen et al., 2011).

Future Directions

Although recent multimodal imaging results are promising, much work remains to be done. As the field of multimodal imaging is relatively new, most of the studies represent novel findings; however, replication is needed to draw general conclusions about structure-function relationships. Secondly, multimodal fusion proves to be fruitful for a more informative understanding of brain activity and disorders, but fusing as many modalities/features as possible in the training sample does not guarantee best discrimination or classification between groups, as reported in (Calhoun and Adali, 2009; Zhang et al., 2012); thus it would be helpful to compare a combination of uni-modal and multimodal results, as done in (Kim et al., 2010). This work can be pursued in future by using larger data sets and various modalities. Finally, introducing multimodal analyses in longitudinal brain studies has not been done frequently yet, which could be another new direction, as there are many possibilities for modeling the baseline and change over multiple time points.

Furthermore, recently graph theory-based analyses have become popular for investigating brain networks using imaging data (Bullmore and Sp*rns, 2009; Rubinov and Sp*rns, 2010; Yu et al., 2012; Yu et al., 2013). However, only few studies examined topological properties of brain networks from concurrently obtained multiple modalities. Future multimodal studies incorporating graph theoretical analyses may be helpful to further understand brain functioning.

Besides the four frequently used modalities we discussed above, the incorporation of other modalities, e.g. PET, MR spectroscopy, Infrared (FTIR & IR) spectroscopy and/or genetics may even further and in greater detail assist in our understanding of the complexities of the human brain. For instance, using four modalities (fMRI, sMRI, DTI and MRS), (Hao et al., 2011) were able to more confidently make statements about the neural determinants of intellectual abilities of healthy adults. Moreover, the combination of MRS with MRI could provide markers which could clarify the relationship between prodromal phases of psychosis, neuroimaging characteristics, and underlying neurobiological pathways (Fusar-Poli et al., 2011c). More importantly, generating endophenotypes from brain imaging data for genetic association studies can help to identify potential biomarkers for several mental illnesses (Liu et al.; Meda et al.), which potentially could be used to support clinical diagnosis.

Over the last ten years the number of multimodal studies published that employed a combination of the 4 previously mentioned modalities has steadily increased as shown in Figure 4, as searched from PubMed. It's clear that exploratory brain studies that describe the variability across modalities have become more and more popular over the last 5 years. Most neuroimaging data fusion schemes are restricted to two modalities, because although combining 3 or more modalities improves inferences(Correa et al., 2010a), it also increases analytic challenges. A main challenge in multimodal data fusion comes from dissimilarity of the data types being fused and result interpretation. However, emerging from 2009, N-way multimodal fusion may become one of leading directions in future neuroimaging research.

Open in a separate window

Figure 4

Frequency of published multimodal fusion studies using brain imaging data

In summary, we are just beginning to unlock the potential of multimodal imaging. Joint analysis of brain functional and structural data appears to be an effective approach for analyzing brain diseases. It helps to identify the unique and shared variance associated with each imaging modality that underlies cognitive functioning in healthy controls and impairment in mental illness. The most promising avenues for the future may rest on developing better models based on multi-disciplinary knowledge, thereby enabling the broader neurosciences to access neuroimaging so that key questions can be addressed in a theoretically grounded fashion (Friston, 2009).

​

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

• Allen EA, Erhardt EB, Damaraju E, Gruner W, Segall JM, Silva RF, Havlicek M, Rachakonda S, Fries J, Kalyanam R, Michael AM, Caprihan A, Turner JA, Eichele T, Adelsheim S, Bryan AD, Bustillo J, Clark VP, Feldstein Ewing SW, Filbey F, Ford CC, Hutchison K, Jung RE, Kiehl KA, Kodituwakku P, Komesu YM, Mayer AR, Pearlson GD, Phillips JP, Sadek JR, Stevens M, Teuscher U, Thoma RJ, Calhoun VD. A baseline for the multivariate comparison of resting-state networks. Front Syst Neurosci. 2011;5:2. [PMC free article] [PubMed] [Google Scholar]
• Ardnt C. SPIE Proc. 1996. Information gained by data fusion. [Google Scholar]
• Ashburner J, Friston KJ. Voxel-based morphometry--the methods. Neuroimage. 2000;11:805–821. [PubMed] [Google Scholar]
• Behrens T, Johansen-Berg H. Diffusion MRI:from quantitative measurement to in-vivo neuroanatomy. Elsevier; 2009. [Google Scholar]
• Bosnell RA, Kincses T, Stagg CJ, Tomassini V, Kischka U, Jbabdi S, Woolrich MW, Andersson J, Matthews PM, Johansen-Berg H. Motor practice promotes increased activity in brain regions structurally disconnected after subcortical stroke. Neurorehabil Neural Repair. 2011;25:607–616. [PMC free article] [PubMed] [Google Scholar]
• Bullmore E, Sp*rns O. Complex brain networks: graph theoretical analysis of structural and functional systems. Nat Rev Neurosci. 2009;10:186–198. [PubMed] [Google Scholar]
• Calhoun VD, Adali T. Feature-based fusion of medical imaging data. IEEE Trans Inf Technol Biomed. 2009;13:711–720. [PMC free article] [PubMed] [Google Scholar]
• Calhoun VD, Adali T, Giuliani NR, Pekar JJ, Kiehl KA, Pearlson GD. Method for multimodal analysis of independent source differences in schizophrenia: combining gray matter structural and auditory oddball functional data. Hum Brain Mapp. 2006;27:47–62. [PMC free article] [PubMed] [Google Scholar]
• Camchong J, MacDonald AW, 3rd, Bell C, Mueller BA, Lim KO. Altered functional and anatomical connectivity in schizophrenia. Schizophr Bull. 2011;37:640–650. [PMC free article] [PubMed] [Google Scholar]
• Cardenas VA, Chao LL, Blumenfeld R, Song E, Meyerhoff DJ, Weiner MW, Studholme C. Using automated morphometry to detect associations between ERP latency and structural brain MRI in normal adults. Hum Brain Mapp. 2005;25:317–327. [PMC free article] [PubMed] [Google Scholar]
• Casey BJ, Tottenham N, Liston C, Durston S. Imaging the developing brain: what have we learned about cognitive development? Trends Cogn Sci. 2005;9:104–110. [PubMed] [Google Scholar]
• Cavanagh JF, Zambrano-Vazquez L, Allen JJ. Theta lingua franca: a common mid-frontal substrate for action monitoring processes. Psychophysiology. 2012;49:220–238. [PMC free article] [PubMed] [Google Scholar]
• Chatzikonstantinou A, Gass A, Forster A, Hennerici MG, Szabo K. Features of acute DWI abnormalities related to status epilepticus. Epilepsy Res. 2011;97:45–51. [PubMed] [Google Scholar]
• Chen K, Reiman EM, Huan Z, Caselli RJ, Bandy D, Ayutyanont N, Alexander GE. Linking functional and structural brain images with multivariate network analyses: a novel application of the partial least square method. Neuroimage. 2009;47:602–610. [PMC free article] [PubMed] [Google Scholar]
• Chen Z, Cui L, Li M, Jiang L, Deng W, Ma X, Wang Q, Huang C, Wang Y, Collier DA, Gong Q, Li T. Voxel based morphometric and diffusion tensor imaging analysis in male bipolar patients with first-episode mania. Prog Neuropsychopharmacol Biol Psychiatry. 2011;36:231–238. [PubMed] [Google Scholar]
• Choi K, Yang Z, Hu X, Mayberg H. A Combined Functional-Structural Connectivity Analysis of Major Depression Using Joint Independent Components Analysis. ISMRM; Toronto, Cananda: 2008. p. 3555. [Google Scholar]
• Cohen MX. Error-related medial frontal theta activity predicts cingulate-related structural connectivity. Neuroimage. 2011;55:1373–1383. [PubMed] [Google Scholar]
• Correa NM, Adali T, Li YO, Calhoun VD. Canonical Correlation Analysis for Data Fusion and Group Inferences: Examining applications of medical imaging data. IEEE Signal Process Mag. 2010a;27:39–50. [PMC free article] [PubMed] [Google Scholar]
• Correa NM, Eichele T, Adali T, Li YO, Calhoun VD. Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. Neuroimage 2010b [PMC free article] [PubMed] [Google Scholar]
• Correa NM, Li YO, Adali T, Calhoun VD. Canonical Correlation Analysis for Feature-Based Fusion of Biomedical Imaging Modalities and Its Application to Detection of Associative Networks in Schizophrenia. IEEE J Sel Top Signal Process. 2008;2:998–1007. [PMC free article] [PubMed] [Google Scholar]
• David O, Friston KJ. A neural mass model for MEG/EEG: coupling and neuronal dynamics. Neuroimage. 2003;20:1743–1755. [PubMed] [Google Scholar]
• De Martino F, Valente G, de Borst AW, Esposito F, Roebroeck A, Goebel R, Formisano E. Multimodal imaging: an evaluation of univariate and multivariate methods for simultaneous EEG/fMRI. Magn Reson Imaging. 2010;28:1104–1112. [PubMed] [Google Scholar]
• Eichele T, Calhoun VD, Moosmann M, Specht K, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K. Unmixing concurrent EEG-fMRI with parallel independent component analysis. Int J Psychophysiol. 2008;67:222–234. [PMC free article] [PubMed] [Google Scholar]
• Eichele T, Specht K, Moosmann M, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K. Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci U S A. 2005;102:17798–17803. [PMC free article] [PubMed] [Google Scholar]
• Fan Y, Resnick SM, Wu X, Davatzikos C. Structural and functional biomarkers of prodromal Alzheimer's disease: a high-dimensional pattern classification study. Neuroimage. 2008;41:277–285. [PMC free article] [PubMed] [Google Scholar]
• Fischl B. FreeSurfer. Neuroimage. 2012;62:774–781. [PMC free article] [PubMed] [Google Scholar]
• Ford JM, Sullivan EV, Marsh L, White PM, Lim KO, Pfefferbaum A. The relationship between P300 amplitude and regional gray matter volumes depends upon the attentional system engaged. Electroencephalogr Clin Neurophysiol. 1994;90:214–228. [PubMed] [Google Scholar]
• Franco AR, Ling J, Caprihan A, Calhoun VD, Jung RE, Heileman GL, Mayer AR. Multimodal and Multi-tissue Measures of Connectivity Revealed by Joint Independent Component Analysis. IEEE J Sel Top Signal Process. 2008;2:986–997. [PMC free article] [PubMed] [Google Scholar]
• Friston KJ. Modalities, modes, and models in functional neuroimaging. Science. 2009;326:399–403. [PubMed] [Google Scholar]
• Fusar-Poli P, Broome MR, Woolley JB, Johns LC, Tabraham P, Bramon E, Valmaggia L, Williams SC, McGuire P. Altered brain function directly related to structural abnormalities in people at ultra high risk of psychosis: longitudinal VBM-fMRI study. Journal of psychiatric research. 2011a;45:190–198. [PubMed] [Google Scholar]
• Fusar-Poli P, Crossley N, Woolley J, Carletti F, Perez-Iglesias R, Broome M, Johns L, Tabraham P, Bramon E, McGuire P. White matter alterations related to P300 abnormalities in individuals at high risk for psychosis: an MRI-EEG study. Journal of psychiatry & neuroscience: JPN. 2011b;36:239–248. [PMC free article] [PubMed] [Google Scholar]
• Fusar-Poli P, McGuire P, Borgwardt S. Mapping prodromal psychosis: a critical review of neuroimaging studies. Eur Psychiatry. 2011c;27:181–191. [PubMed] [Google Scholar]
• Goncalves MS, Hall DA. Connectivity analysis with structural equation modelling: an example of the effects of voxel selection. Neuroimage. 2003;20:1455–1467. [PubMed] [Google Scholar]
• Granziera C, Schmahmann JD, Hadjikhani N, Meyer H, Meuli R, Wedeen V, Krueger G. Diffusion spectrum imaging shows the structural basis of functional cerebellar circuits in the human cerebellum in vivo. PLoS One. 2009;4:e5101. [PMC free article] [PubMed] [Google Scholar]
• Groves AR, Beckmann CF, Smith SM, Woolrich MW. Linked independent component analysis for multimodal data fusion. Neuroimage. 2011;54:2198–2217. [PubMed] [Google Scholar]
• Groves AR, Smith SM, Fjell AM, Tamnes CK, Walhovd KB, Douaud G, Woolrich MW, Westlye LT. Benefits of multi-modal fusion analysis on a large-scale dataset: life-span patterns of inter-subject variability in cortical morphometry and white matter microstructure. Neuroimage. 2012;63:365–380. [PubMed] [Google Scholar]
• Gruendler TO, Ullsperger M, Huster RJ. Event-related potential correlates of performance-monitoring in a lateralized time-estimation task. PLoS One. 2011;6:e25591. [PMC free article] [PubMed] [Google Scholar]
• Haier RJ, Karama S, Leyba L, Jung RE. MRI assessment of cortical thickness and functional activity changes in adolescent girls following three months of practice on a visual-spatial task. BMC Res Notes. 2009;2:174. [PMC free article] [PubMed] [Google Scholar]
• Haller S, Xekardaki A, Delaloye C, Canuto A, Lovblad KO, Gold G, Giannakopoulos P. Combined analysis of grey matter voxel-based morphometry and white matter tract-based spatial statistics in late-life bipolar disorder. J Psychiatry Neurosci. 2011;36:391–401. [PMC free article] [PubMed] [Google Scholar]
• Hao X, Xu D, Bansal R, Dong Z, Liu J, Wang Z, Kangarlu A, Liu F, Duan Y, Shova S, Gerber AJ, Peterson BS. Multimodal magnetic resonance imaging: The coordinated use of multiple, mutually informative probes to understand brain structure and function. Hum Brain Mapp. 2011;34:253–271. [PMC free article] [PubMed] [Google Scholar]
• Harms MP, Wang L, Csernansky JG, Barch DM. Structure-function relationship of working memory activity with hippocampal and prefrontal cortex volumes. Brain Struct Funct. 2012;218:173–186. [PMC free article] [PubMed] [Google Scholar]
• Hughes JR. A review of the usefulness of the standard EEG in psychiatry. Clin Electroencephalogr. 1996;27:35–39. [PubMed] [Google Scholar]
• Huster RJ, Enriquez-Geppert S, Pantev C, Bruchmann M. Variations in midcingulate morphology are related to ERP indices of cognitive control. Brain Struct Funct 2012 [PubMed] [Google Scholar]
• Huster RJ, Westerhausen R, Herrmann CS. Sex differences in cognitive control are associated with midcingulate and callosal morphology. Brain Struct Funct. 2011;215:225–235. [PubMed] [Google Scholar]
• Huster RJ, Westerhausen R, Kreuder F, Schweiger E, Wittling W. Morphologic asymmetry of the human anterior cingulate cortex. Neuroimage. 2007;34:888–895. [PubMed] [Google Scholar]
• Huster RJ, Wolters C, Wollbrink A, Schweiger E, Wittling W, Pantev C, Junghofer M. Effects of anterior cingulate fissurization on cognitive control during stroop interference. Hum Brain Mapp. 2009;30:1279–1289. [PMC free article] [PubMed] [Google Scholar]
• Hutton C, Draganski B, Ashburner J, Weiskopf N. A comparison between voxel-based cortical thickness and voxel-based morphometry in normal aging. Neuroimage. 2009;48:371–380. [PMC free article] [PubMed] [Google Scholar]
• Jacobson S, Kelleher I, Harley M, Murtagh A, Clarke M, Blanchard M, Connolly C, O'Hanlon E, Garavan H, Cannon M. Structural and functional brain correlates of subclinical psychotic symptoms in 11-13 year old schoolchildren. Neuroimage. 2009;49:1875–1885. [PubMed] [Google Scholar]
• Jagannathan K, Calhoun VD, Gelernter J, Stevens MC, Liu J, Bolognani F, Windemuth A, Ruano G, Assaf M, Pearlson GD. Genetic associations of brain structural networks in schizophrenia: a preliminary study. Biol Psychiatry. 2010;68:657–666. [PMC free article] [PubMed] [Google Scholar]
• Jamadar S, Powers NR, Meda SA, Gelernter J, Gruen JR, Pearlson GD. Genetic influences of cortical gray matter in language-related regions in healthy controls and schizophrenia. Schizophr Res 2010 [PMC free article] [PubMed] [Google Scholar]
• Jeurissen B, Leemans A, Tournier JD, Jones DK, Sijbers J. Investigating the prevalence of complex fiber configurations in white matter tissue with diffusion magnetic resonance imaging. Hum Brain Mapp 2012 [PMC free article] [PubMed] [Google Scholar]
• Johansen-Berg H, Rushworth MF. Using diffusion imaging to study human connectional anatomy. Annu Rev Neurosci. 2009;32:75–94. [PubMed] [Google Scholar]
• Jones DK. Studying connections in the living human brain with diffusion MRI. Cortex. 2008;44:936–952. [PubMed] [Google Scholar]
• Jones DK, Knosche TR, Turner R. White matter integrity, fiber count, and other fallacies: the do's and don'ts of diffusion MRI. Neuroimage. 2013;73:239–254. [PubMed] [Google Scholar]
• Keightley ML, Chen JK, Ptito A. Examining the neural impact of pediatric concussion: a scoping review of multimodal and integrative approaches using functional and structural MRI techniques. Curr Opin Pediatr. 2012;24:709–716. [PubMed] [Google Scholar]
• Khullar S, Michael AM, Cahill ND, Kiehl KA, Pearlson G, Baum SA, Calhoun VD. ICA-fNORM: Spatial Normalization of fMRI Data Using Intrinsic Group-ICA Networks. Front Syst Neurosci. 2011;5:93. [PMC free article] [PubMed] [Google Scholar]
• Kim DI, Sui J, Rachakonda S, White T, Manoach DS, Clark VP, Ho BC, Schulz SC, Calhoun VD. Identification of imaging biomarkers in schizophrenia: a coefficient-constrained independent component analysis of the mind multi-site schizophrenia study. Neuroinformatics. 2010;8:213–229. [PMC free article] [PubMed] [Google Scholar]
• Kim DJ, Park B, Park HJ. Functional connectivity-based identification of subdivisions of the basal ganglia and thalamus using multilevel independent component analysis of resting state fMRI. Hum Brain Mapp 2012a [PMC free article] [PubMed] [Google Scholar]
• Kim J, Lee JH. Integration of structural and functional magnetic resonance imaging improves mild cognitive impairment detection. Magn Reson Imaging 2012 [PubMed] [Google Scholar]
• Kim YB, Kalthoff D, Po C, Wiedermann D, Hoehn M. Connectivity of thalamo-cortical pathway in rat brain: combined diffusion spectrum imaging and functional MRI at 11.7 T. NMR Biomed. 2012b;25:943–952. [PubMed] [Google Scholar]
• Klimesch W, Sauseng P, Gerloff C. Enhancing cognitive performance with repetitive transcranial magnetic stimulation at human individual alpha frequency. The European journal of neuroscience. 2003;17:1129–1133. [PubMed] [Google Scholar]
• Koch K, Wagner G, Schachtzabel C, Schultz CC, Gullmar D, Reichenbach JR, Sauer H, Schlosser RG. Neural activation and radial diffusivity in schizophrenia: combined fMRI and diffusion tensor imaging study. Br J Psychiatry. 2011;198:223–229. [PubMed] [Google Scholar]
• Kolb B, Whishaw IQ. Brain plasticity and behavior. Annu Rev Psychol. 1998;49:43–64. [PubMed] [Google Scholar]
• Leergaard TB, Hilgetag CC, Sp*rns O. Mapping the connectome: multi-level analysis of brain connectivity. Frontiers in neuroinformatics. 2012;6:14. [PMC free article] [PubMed] [Google Scholar]
• Li YO, Adali T, Calhoun VD. Estimating the number of independent components for functional magnetic resonance imaging data. Hum Brain Mapp. 2007;28:1251–1266. [PMC free article] [PubMed] [Google Scholar]
• Liu J, Calhoun VD. Parallel independent component analysis for multimodal analysis: application to fMRI and EEG data; Proc. of IEEE ISBI conference; Arlington, USA. 2007. pp. 1028–1231. [Google Scholar]
• Liu J, Ghassemi MM, Michael AM, Boutte D, Wells W, Perrone-Bizzozero N, Macciardi F, Mathalon DH, Ford JM, Potkin SG, Turner JA, Calhoun VD. An ICA with reference approach in identification of genetic variation and associated brain networks. Front Hum Neurosci. 2012;6:21. [PMC free article] [PubMed] [Google Scholar]
• Liu J, Pearlson G, Windemuth A, Ruano G, Perrone-Bizzozero NI, Calhoun V. Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum Brain Mapp. 2009;30:241–255. [PMC free article] [PubMed] [Google Scholar]
• Martinez-Montes E, Valdes-Sosa PA, Miwakeichi F, Goldman RI, Cohen MS. Concurrent EEG/fMRI analysis by multiway Partial Least Squares. Neuroimage. 2004;22:1023–1034. [PubMed] [Google Scholar]
• Mascalchi M, Ginestroni A, Toschi N, Poggesi A, Cecchi P, Salvadori E, Tessa C, Cosottini M, Stefano ND, Pracucci G, Pantoni L, Inzitari D, Diciotti S. The burden of microstructural damage modulates cortical activation in elderly subjects with MCI and leuko-araiosis. A DTI and fMRI study. Hum Brain Mapp 2012 [PMC free article] [PubMed] [Google Scholar]
• Matthews SC, Strigo IA, Simmons AN, O'Connell RM, Reinhardt LE, Moseley SA. A multimodal imaging study in U.S. veterans of Operations Iraqi and Enduring Freedom with and without major depression after blast-related concussion. NeuroImage. 2011;54(1):S69–75. [PubMed] [Google Scholar]
• Meda SA, Gill A, Stevens MC, Lorenzoni RP, Glahn DC, Calhoun VD, Sweeney JA, Tamminga CA, Keshavan MS, Thaker G, Pearlson GD. Differences in resting-state functional magnetic resonance imaging functional network connectivity between schizophrenia and psychotic bipolar probands and their unaffected first-degree relatives. Biol Psychiatry. 2012a;71:881–889. [PMC free article] [PubMed] [Google Scholar]
• Meda SA, Jagannathan K, Gelernter J, Calhoun VD, Liu J, Stevens MC, Pearlson GD. A pilot multivariate parallel ICA study to investigate differential linkage between neural networks and genetic profiles in schizophrenia. Neuroimage. 2010;53:1007–1015. [PMC free article] [PubMed] [Google Scholar]
• Meda SA, Narayanan B, Liu J, Perrone-Bizzozero NI, Stevens MC, Calhoun VD, Glahn DC, Shen L, Risacher SL, Saykin AJ, Pearlson GD. A large scale multivariate parallel ICA method reveals novel imaging-genetic relationships for Alzheimer's disease in the ADNI cohort. Neuroimage. 2012b;60:1608–1621. [PMC free article] [PubMed] [Google Scholar]
• Michael AM, Baum SA, White T, Demirci O, Andreasen NC, Segall JM, Jung RE, Pearlson G, Clark VP, Gollub RL, Schulz SC, Roffman JL, Lim KO, Ho BC, Bockholt HJ, Calhoun VD. Does function follow form?: methods to fuse structural and functional brain images show decreased linkage in schizophrenia. Neuroimage. 2010;49:2626–2637. [PMC free article] [PubMed] [Google Scholar]
• Michael AM, King MD, Ehrlich S, Pearlson G, White T, Holt DJ, Andreasen NC, Sakoglu U, Ho BC, Schulz SC, Calhoun VD. A Data-Driven Investigation of Gray Matter-Function Correlations in Schizophrenia during a Working Memory Task. Frontiers in human neuroscience. 2011;5:71. [PMC free article] [PubMed] [Google Scholar]
• O'Donnell LJ, Rigolo L, Norton I, Wells WM, 3rd, Westin CF, Golby AJ. fMRI-DTI modeling via landmark distance atlases for prediction and detection of fiber tracts. Neuroimage. 2011;60:456–470. [PMC free article] [PubMed] [Google Scholar]
• Oakes TR, Fox AS, Johnstone T, Chung MK, Kalin N, Davidson RJ. Integrating VBM into the General Linear Model with voxelwise anatomical covariates. Neuroimage. 2007;34:500–508. [PMC free article] [PubMed] [Google Scholar]
• Olesen PJ, Nagy Z, Westerberg H, Klingberg T. Combined analysis of DTI and fMRI data reveals a joint maturation of white and grey matter in a fronto-parietal network. Brain Res Cogn Brain Res. 2003;18:48–57. [PubMed] [Google Scholar]
• Palacios EM, Sala-Llonch R, Junque C, Roig T, Tormos JM, Bargallo N, Vendrell P. White matter integrity related to functional working memory networks in traumatic brain injury. Neurology. 2012;78:852–860. [PubMed] [Google Scholar]
• Phillips JS, Greenberg AS, Pyles JA, Pathak SK, Behrmann M, Schneider W, Tarr MJ. Co-analysis of brain structure and function using fMRI and diffusion-weighted imaging. J Vis Exp 2012 [PMC free article] [PubMed] [Google Scholar]
• Pomarol-Clotet E, Canales-Rodriguez EJ, Salvador R, Sarro S, Gomar JJ, Vila F, Ortiz-Gil J, Iturria-Medina Y, Capdevila A, McKenna PJ. Medial prefrontal cortex pathology in schizophrenia as revealed by convergent findings from multimodal imaging. Mol Psychiatry. 2010;15:823–830. [PMC free article] [PubMed] [Google Scholar]
• Qiu MG, Ye Z, Li QY, Liu GJ, Xie B, Wang J. Changes of brain structure and function in ADHD children. Brain Topogr. 2011;24:243–252. [PubMed] [Google Scholar]
• Rasser PE, Johnston P, Lagopoulos J, Ward PB, Schall U, Thienel R, Bender S, Toga AW, Thompson PM. Functional MRI BOLD response to Tower of London performance of first-episode schizophrenia patients using cortical pattern matching. Neuroimage. 2005;26:941–951. [PubMed] [Google Scholar]
• Reinges MH, Schoth F, Coenen VA, Krings T. Imaging of postthalamic visual fiber tracts by anisotropic diffusion weighted MRI and diffusion tensor imaging: principles and applications. Eur J Radiol. 2004;49:91–104. [PubMed] [Google Scholar]
• Rektorova I, Mikl M, Barrett J, Marecek R, Rektor I, Paus T. Functional neuroanatomy of vocalization in patients with Parkinson's disease. J Neurol Sci. 2012;313:7–12. [PubMed] [Google Scholar]
• Roberts RE, Anderson EJ, Husain M. White matter microstructure and cognitive function. Neuroscientist. 2013;19:8–15. [PMC free article] [PubMed] [Google Scholar]
• Rubinov M, Sp*rns O. Complex network measures of brain connectivity: uses and interpretations. Neuroimage. 2010;52:1059–1069. [PubMed] [Google Scholar]
• Rykhlevskaia E, Gratton G, Fabiani M. Combining structural and functional neuroimaging data for studying brain connectivity: a review. Psychophysiology. 2008;45:173–187. [PubMed] [Google Scholar]
• Salgado-Pineda P, Fakra E, Delaveau P, McKenna PJ, Pomarol-Clotet E, Blin O. Correlated structural and functional brain abnormalities in the default mode network in schizophrenia patients. Schizophr Res. 2011;125:101–109. [PubMed] [Google Scholar]
• Salgado-Pineda P, Junque C, Vendrell P, Baeza I, Bargallo N, Falcon C, Bernardo M. Decreased cerebral activation during CPT performance: structural and functional deficits in schizophrenic patients. Neuroimage. 2004;21:840–847. [PubMed] [Google Scholar]
• Satherley BL, Jones RD, Bones PJ. EEG spectral topography in neurology: I. A review of techniques. Australas Phys Eng Sci Med. 1996;19:172–182. [PubMed] [Google Scholar]
• Savopol F, Armenakis C. Mergine of Heterogeneous Data for Emergency Mapping: Data integration or Data Fusion? Proc ISPRS 2002 [Google Scholar]
• Schiltz K, Szentkuti A, Guderian S, Kaufmann J, Munte TF, Heinze HJ, Duzel E. Relationship between hippocampal structure and memory function in elderly humans. Journal of cognitive neuroscience. 2006;18:990–1003. [PubMed] [Google Scholar]
• Schlosser R, Gesierich T, Kaufmann B, Vucurevic G, Hunsche S, Gawehn J, Stoeter P. Altered effective connectivity during working memory performance in schizophrenia: a study with fMRI and structural equation modeling. Neuroimage. 2003;19:751–763. [PubMed] [Google Scholar]
• Schlosser RG, Nenadic I, Wagner G, Gullmar D, von Consbruch K, Kohler S, Schultz CC, Koch K, Fitzek C, Matthews PM, Reichenbach JR, Sauer H. White matter abnormalities and brain activation in schizophrenia: a combined DTI and fMRI study. Schizophr Res. 2007;89:1–11. [PubMed] [Google Scholar]
• Schonberg T, Pianka P, Hendler T, Pasternak O, Assaf Y. Characterization of displaced white matter by brain tumors using combined DTI and fMRI. Neuroimage. 2006;30:1100–1111. [PubMed] [Google Scholar]
• Schultz CC, Fusar-Poli P, Wagner G, Koch K, Schachtzabel C, Gruber O, Sauer H, Schlosser RG. Multimodal functional and structural imaging investigations in psychosis research. European archives of psychiatry and clinical neuroscience. 2012;262(2):S97–106. [PubMed] [Google Scholar]
• Segall JM, Allen EA, Jung RE, Erhardt EB, Arja SK, Kiehl K, Calhoun VD. Correspondence between structure and function in the human brain at rest. Frontiers in neuroinformatics. 2012;6:10. [PMC free article] [PubMed] [Google Scholar]
• Sereno MI, Lutti A, Weiskopf N, Dick F. Mapping the Human Cortical Surface by Combining Quantitative T1 with Retinotopy. Cereb Cortex 2012 [PMC free article] [PubMed] [Google Scholar]
• Sexton CE, Allan CL, Le Masurier M, McDermott LM, Kalu UG, Herrmann LL, Maurer M, Bradley KM, Mackay CE, Ebmeier KP. Magnetic resonance imaging in late-life depression: multimodal examination of network disruption. Arch Gen Psychiatry. 2012;69:680–689. [PubMed] [Google Scholar]
• Skudlarski P, Jagannathan K, Anderson K, Stevens MC, Calhoun VD, Skudlarska BA, Pearlson G. Brain connectivity is not only lower but different in schizophrenia: a combined anatomical and functional approach. Biol Psychiatry. 2010;68:61–69. [PMC free article] [PubMed] [Google Scholar]
• Skudlarski P, Jagannathan K, Calhoun VD, Hampson M, Skudlarska BA, Pearlson G. Measuring brain connectivity: diffusion tensor imaging validates resting state temporal correlations. Neuroimage. 2008;43:554–561. [PMC free article] [PubMed] [Google Scholar]
• Smieskova R, Allen P, Simon A, Aston J, Bendfeldt K, Drewe J, Gruber K, Gschwandtner U, Klarhoefer M, Lenz C, Scheffler K, Stieglitz RD, Radue EW, McGuire P, Riecher-Rossler A, Borgwardt SJ. Different duration of at-risk mental state associated with neurofunctional abnormalities. A multimodal imaging study. Hum Brain Mapp. 2011;33:2281–2294. [PMC free article] [PubMed] [Google Scholar]
• Smieskova R, Allen P, Simon A, Aston J, Bendfeldt K, Drewe J, Gruber K, Gschwandtner U, Klarhoefer M, Lenz C, Scheffler K, Stieglitz RD, Radue EW, McGuire P, Riecher-Rossler A, Borgwardt SJ. Different duration of at-risk mental state associated with neurofunctional abnormalities. A multimodal imaging study. Hum Brain Mapp. 2012;33:2281–2294. [PMC free article] [PubMed] [Google Scholar]
• Soldner J, Meindl T, Koch W, Bokde AL, Reiser MF, Moller HJ, Burger K, Hampel H, Teipel SJ. Structural and functional neuronal connectivity in Alzheimer's disease: a combined DTI and fMRI study. Nervenarzt. 2011;83:878–887. [PubMed] [Google Scholar]
• Staempfli P, Reischauer C, Jaermann T, Valavanis A, Kollias S, Boesiger P. Combining fMRI and DTI: a framework for exploring the limits of fMRI-guided DTI fiber tracking and for verifying DTI-based fiber tractography results. Neuroimage. 2008;39:119–126. [PubMed] [Google Scholar]
• Sui J, Adali T, Yu Q, Chen J, Calhoun VD. A review of multivariate methods for multimodal fusion of brain imaging data. J Neurosci Methods. 2012a;204:68–81. [PMC free article] [PubMed] [Google Scholar]
• Sui J, He H, Liu J, Yu Q, Adali T, Pearlson GD, Calhoun VD. Three-way FMRI-DTI-methylation data fusion based on mCCA+jICA and its application to schizophrenia; Conf Proc IEEE Eng Med Biol Soc; San Diego, CA, USA. 2012b. pp. 2692–2695. [PubMed] [Google Scholar]
• Sui J, He H, Pearlson GD, Adali T, Kiehl KA, Yu Q, Clark VP, Castro E, White T, Mueller BA, Ho BC, Andreasen NC, Calhoun VD. Three-way (N-way) fusion of brain imaging data based on mCCA+jICA and its application to discriminating schizophrenia. Neuroimage. 2012c;2:119–132. [PMC free article] [PubMed] [Google Scholar]
• Sui J, He H, Yu Q, Chen J, Rogers J, Pearlson GD, Mayer A, Bustillo J, Canive J, Calhoun VD. Combination of Resting State fMRI, DTI, and sMRI Data to Discriminate Schizophrenia by N-way MCCA + jICA. Front Hum Neurosci. 2013;7:235. [PMC free article] [PubMed] [Google Scholar]
• Sui J, Pearlson G, Caprihan A, Adali T, Kiehl KA, Liu J, Yamamoto J, Calhoun VD. Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+joint ICA model. Neuroimage. 2011;57:839–855. [PMC free article] [PubMed] [Google Scholar]
• Sui J, Yu Q, He H, Pearlson GD, Calhoun VD. A selective review of multimodal fusion methods in schizophrenia. Front Hum Neurosci. 2012d;6:27. [PMC free article] [PubMed] [Google Scholar]
• Supekar K, Uddin LQ, Prater K, Amin H, Greicius MD, Menon V. Development of functional and structural connectivity within the default mode network in young children. Neuroimage. 2010;52:290–301. [PMC free article] [PubMed] [Google Scholar]
• Tang CY, Eaves E, Dams-O'Connor K, Ho L, Leung E, Wong E, Carpenter D, Ng J, Gordon W, Pasinetti G. Diffuse Disconnectivity in tBi: a resting state fMri anD Dti stuDy. Transl Neurosci. 2013;3:9–14. [PMC free article] [PubMed] [Google Scholar]
• Tefera GB, Zhou Y, Juneja V, Narayana PA. Evaluation of fiber tracking from subsampled q-space data in diffusion spectrum imaging. Magn Reson Imaging 2013 [PMC free article] [PubMed] [Google Scholar]
• Teipel SJ, Bokde AL, Meindl T, Amaro E, Jr, Soldner J, Reiser MF, Herpertz SC, Moller HJ, Hampel H. White matter microstructure underlying default mode network connectivity in the human brain. Neuroimage. 2010;49:2021–2032. [PubMed] [Google Scholar]
• Teipel SJ, Pogarell O, Meindl T, Dietrich O, Sydykova D, Hunklinger U, Georgii B, Mulert C, Reiser MF, Moller HJ, Hampel H. Regional networks underlying interhemispheric connectivity: an EEG and DTI study in healthy ageing and amnestic mild cognitive impairment. Hum Brain Mapp. 2009;30:2098–2119. [PMC free article] [PubMed] [Google Scholar]
• Tian L, Meng C, Yan H, Zhao Q, Liu Q, Yan J, Han Y, Yuan H, Wang L, Yue W, Zhang Y, Li X, Zhu C, He Y, Zhang D. Convergent evidence from multimodal imaging reveals amygdala abnormalities in schizophrenic patients and their first-degree relatives. PLoS One. 2011;6:e28794. [PMC free article] [PubMed] [Google Scholar]
• Toosy AT, Ciccarelli O, Parker GJ, Wheeler-Kingshott CA, Miller DH, Thompson AJ. Characterizing function-structure relationships in the human visual system with functional MRI and diffusion tensor imaging. Neuroimage. 2004;21:1452–1463. [PubMed] [Google Scholar]
• Valdes-Hernandez PA, Ojeda-Gonzalez A, Martinez-Montes E, Lage-Castellanos A, Virues-Alba T, Valdes-Urrutia L, Valdes-Sosa PA. White matter architecture rather than cortical surface area correlates with the EEG alpha rhythm. Neuroimage. 2010;49:2328–2339. [PubMed] [Google Scholar]
• Valente G, De Martino F, Esposito F, Goebel R, Formisano E. Predicting subject-driven actions and sensory experience in a virtual world with relevance vector machine regression of fMRI data. Neuroimage. 2011;56:651–661. [PubMed] [Google Scholar]
• Vogt BA, Nimchinsky EA, Vogt LJ, Hof PR. Human cingulate cortex: surface features, flat maps, and cytoarchitecture. J Comp Neurol. 1995;359:490–506. [PubMed] [Google Scholar]
• Voineskos AN, Farzan F, Barr MS, Lobaugh NJ, Mulsant BH, Chen R, Fitzgerald PB, Daskalakis ZJ. The role of the corpus callosum in transcranial magnetic stimulation induced interhemispheric signal propagation. Biol Psychiatry. 2010;68:825–831. [PubMed] [Google Scholar]
• Voss HU, Schiff ND. MRI of neuronal network structure, function, and plasticity. Prog Brain Res. 2009;175:483–496. [PubMed] [Google Scholar]
• Wang F, Kalmar JH, He Y, Jackowski M, Chepenik LG, Edmiston EE, Tie K, Gong G, Shah MP, Jones M, Uderman J, Constable RT, Blumberg HP. Functional and structural connectivity between the perigenual anterior cingulate and amygdala in bipolar disorder. Biol Psychiatry. 2009;66:516–521. [PMC free article] [PubMed] [Google Scholar]
• Wee CY, Yap PT, Zhang D, Denny K, Browndyke JN, Potter GG, Welsh-Bohmer KA, Wang L, Shen D. Identification of MCI individuals using structural and functional connectivity networks. Neuroimage. 2011;59:2045–2056. [PMC free article] [PubMed] [Google Scholar]
• Westerhausen R, Kreuder F, Woerner W, Huster RJ, Smit CM, Schweiger E, Wittling W. Interhemispheric transfer time and structural properties of the corpus callosum. Neurosci Lett. 2006;409:140–145. [PubMed] [Google Scholar]
• Westlye LT, Walhovd KB, Bjornerud A, Due-Tonnessen P, Fjell AM. Error-related negativity is mediated by fractional anisotropy in the posterior cingulate gyrus--a study combining diffusion tensor imaging and electrophysiology in healthy adults. Cereb Cortex. 2009;19:293–304. [PubMed] [Google Scholar]
• Whitford TJ, Kubicki M, Ghorashi S, Schneiderman JS, Hawley KJ, McCarley RW, Shenton ME, Spencer KM. Predicting inter-hemispheric transfer time from the diffusion properties of the corpus callosum in healthy individuals and schizophrenia patients: a combined ERP and DTI study. Neuroimage. 2011;54:2318–2329. [PMC free article] [PubMed] [Google Scholar]
• Williams LM, Grieve SM, Whitford TJ, Clark CR, Gur RC, Goldberg E, Flor-Henry P, Peduto AS, Gordon E. Neural synchrony and gray matter variation in human males and females: integration of 40 Hz gamma synchrony and MRI measures. J Integr Neurosci. 2005;4:77–93. [PubMed] [Google Scholar]
• Xu L, Groth KM, Pearlson G, Schretlen DJ, Calhoun VD. Source-based morphometry: the use of independent component analysis to identify gray matter differences with application to schizophrenia. Hum Brain Mapp. 2009;30:711–724. [PMC free article] [PubMed] [Google Scholar]
• Yan H, Tian L, Yan J, Sun W, Liu Q, Zhang YB, Li XM, Zang YF, Zhang D. Functional and anatomical connectivity abnormalities in cognitive division of anterior cingulated cortex in schizophrenia. PLoS One. 2012;7:e45659. [PMC free article] [PubMed] [Google Scholar]
• Yu Q, Allen EA, Sui J, Arbabshirani MR, Pearlson G, Calhoun VD. Brain Connectivity Networks In Schizophrenia Underlying Resting State Functional Magnetic Resonance Imaging. Current topics in medicinal chemistry. 2012;12:2415–2425. [PMC free article] [PubMed] [Google Scholar]
• Yu Q, Sui J, Liu J, Plis SM, Kiehl KA, Pearlson G, Calhoun VD. Disrupted correlation between low frequency power and connectivity strength of resting state brain networks in schizophrenia. Schizophrenia research. 2013;143:165–171. [PMC free article] [PubMed] [Google Scholar]
• Zaehle T, Herrmann CS. Neural synchrony and white matter variations in the human brain-- relation between evoked gamma frequency and corpus callosum morphology. Int J Psychophysiol. 2011;79:49–54. [PubMed] [Google Scholar]
• Zhang D, Wang Y, Zhou L, Yuan H, Shen D. Multimodal classification of Alzheimer's disease and mild cognitive impairment. Neuroimage. 2011;55:856–867. [PMC free article] [PubMed] [Google Scholar]
• Zhang H, Liu L, Wu H, Fan Y. Feature selection and SVM classification of multiple modality images for predicting MCI. OHBM; Beijing, China: 2012. [Google Scholar]
• Zhou Y, Shu N, Liu Y, Song M, Hao Y, Liu H, Yu C, Liu Z, Jiang T. Altered resting-state functional connectivity and anatomical connectivity of hippocampus in schizophrenia. Schizophr Res. 2008;100:120–132. [PubMed] [Google Scholar]
• Zoefel B, Huster RJ, Herrmann CS. Neurofeedback training of the upper alpha frequency band in EEG improves cognitive performance. Neuroimage. 2011;54:1427–1431. [PubMed] [Google Scholar]