Cross-Informant Assessment of Children's Sympathy: Disentangling Trait and State Agreement

Introduction

A recommended practice in developmental and clinical research is the use of different informants (e.g., caregivers, teachers, peers, clinicians, etc.) to assess children's social-emotional development, behavioral functioning, and mental health [1]. From a practical perspective, using data from several sources is important to obtain a comprehensive profile of children's strengths and needs, which can help plan appropriate intervention. Researchers tend to interpret results that are stable across informants as more trustworthy because they do not depend on a specific informant, and the degree of convergence between informants is thus thought to indicate the child's general score for the construct under investigation. However, a large amount of empirical data indicates that only a small-to-moderate amount of agreement exists between different informants of children's social-emotional development and (mal)adaptive behavior [2, 3]. Although several factors may account for this inconsistency (e.g., different contexts of observation and reference points; [3]), the level of analysis at which it occurs remains unclear.

Here, we addressed this gap using the conceptual and methodological frameworks of latent state–trait (LST) theory [4]. We applied LST to assess the extent to which caregivers and teachers converged in their evaluations of children's sympathy (i.e., affective concern for others' welfare; 5) which is a core dimension of social-emotional development [6, 7]. We investigated this question at two different levels: (1) the dispositional or trait level, reflecting children's sympathetic tendencies across time, and (2) the state level, reflecting fluctuations in children's sympathetic responses at a given point in time. We focused on children's sympathy because it is regarded as a core social-emotional skill and has been associated with various positive and negative developmental outcomes (for reviews, see [5, 7]). Its reliable assessment is also highly relevant to clinical contexts ranging in severity (e.g., for the assessment of callous-unemotional traits among high-risk youth; [8] and social-emotional competencies in schools; [6]). We expected caregivers and teachers to agree more at the dispositional vs. situational level of children's sympathy because the latter is by definition more ephemeral and sensitive to contextual features, which likely differ significantly for caregivers and teachers at home and school, respectively.

Cross-Informant Convergence in the Assessment of Children's Sympathy

Sympathy is a specific emotional response that includes feelings of concern or sorrow for another's emotional state or welfare [5]. In comparison to empathy, which generally involves sharing the emotions of another, but not necessarily feeling concern for them, sympathy is more likely to be implicated in prosocial and aggressive behaviors [9, 10].

Different methods (e.g., questionnaires and observations) and informants (e.g., caregivers and teachers) have been used to assess sympathy across childhood and adolescence [11, 12]. However, the majority of these studies relied on—or at least reported findings from—a single informant using questionnaire items, thus offering only a partial perspective of the development of sympathy across different contexts (e.g., home and school). As a notable exception, Kienbaum [11] used a multi-method (observations and questionnaires) and multi-informant (caregiver-, teacher-, and self-reports) approach to investigate the development of children's sympathy from 5 to 7 years of age. Correlations between child observations and self-reported sympathy were statistically significant at each of the three time points, whereas the evaluations of teachers and parents were neither associated with each other nor the other methods (correlations ranged from −0.03 to 0.27). Similarly, Murphy et al. [13] did not find statistically significant relations between teachers' and parents' evaluations of primary school children's sympathy (the correlation coefficient was 0.14).

Several factors might be responsible for this low inter-rater agreement. For instance, caregivers and teachers may perceive children's sympathetic capacities differently based on their shared context with the children, specifically the way in which their respective contexts may differentially set the stage for sympathetic opportunities and ratings. For example, teachers observe children at school amongst a variety of peers (i.e., additional reference points from which to gauge a given child's sympathy), as well as in an environment that generally commands respect for numerous rules. In contrast, caregivers tend to observe their children at home with less reference points (even after considering siblings) and potentially under different sets of rules and expectations. Caregivers may also see their children from a different perspective, given that they are more emotionally involved with the child than the teacher [14]. Disagreement between informants may also stem from the nature of the construct under investigation and how it is perceived. Sympathy is an internal state that is not easily assessed in children because they may feel concern for another without directly showing it [15]. Notably, another important (and less investigated) factor responsible for this disagreement could be the different degree to which the dispositional characteristics of the child and state-like factors affect the evaluation of each informant. For instance, although caregivers and teachers tend to rate children's behavior and psychological functioning in terms of dispositional (trait) tendencies (e.g., how the child usually behaves or feels; [3]), their evaluations can also reflect situational (state) factors. For instance, a teacher may recall a recent event in which a child showed a sympathetic response (e.g., comforting a peer who was teased at school), which may result in an inflated rating of that child's sympathy (compared to his/her general level of sympathy). Therefore, considering that several context- and occasion-specific cues may differently elicit children's sympathy at home (e.g., siblings crying) vs. school (e.g., bullying episodes), the disagreement between caregivers and teachers may be further aggravated when the focus of the evaluation (dispositional sympathy vs. state sympathy) is not clearly distinguished.

In sum, a number of factors may contribute to caregivers and teachers capturing specific aspects of children's sympathy, resulting in difficulties for the interpretation of existing findings, as well as for the integration of information from multiple informants in practical settings [1]. Hereafter, we show how LST theory can shed light on the low cross-informant agreement of children's sympathy by disentangling the level of convergence at both the trait level (dispositional sympathy) and state level (momentary manifestations of children's sympathy).

Disentangling Trait and State Agreement Using LST

Although a full presentation of LST theory (see [16]) is beyond the scope of this paper, we will reference its main assumptions that directly relate to the assessment of trait and state convergence across informants¹. Developed as an extension of classical test theory, LST theory [4] postulates that an observed, manifest variable (e.g., children's sympathy) can be decomposed into three main components: (1) a trait component ξ that represents the general, stable level of the attribute for that individual, (2) an occasion-specific component ζ that represents state-like deviations from the trait component due to situational and/or interactional (i.e., person x situation) effects, and (3) measurement error. Since, by definition, trait components are stable across time and state components are measured at a specific point in time, only longitudinal data allows for their proper estimation and decomposition [16].

For instance, using a structural equation modeling framework, the singletrait-multistate (STMS) model for three observed indicators (e.g., items of a questionnaire) measured at three time points requires the estimation of four latent variables to separate trait and state effects (see Figure 1). First, a common latent trait variable ξ (measured by all nine indicators) is modeled to reflect the general, time-unspecific mean level of the construct under investigation. Importantly, both the factor loading (λ) and intercept (α) of the same item i should be invariant across time to ensure strong (i.e., scalar) longitudinal measurement invariance at the trait level (i.e., the lack of measurement-related alterations due to different use of the rating scale or interpretations of the items over time; [17, 18]). Second, three time-specific, latent state residual factors (ζ₁, ζ₂, and ζ₃; each measured by the three indicators used at each time point) are estimated to capture participants' deviations from the general latent trait. Since latent state residual factors are defined as momentary deviations from the general latent trait, only weak (i.e., metric) longitudinal invariance of factor loadings γ is required (latent state residual factors have a mean of zero by definition).

FIGURE 1

Figure 1. Singletrait-multistate (STMS) Model for Three Waves. Latent variables indicate both trait (ξ) and state (ζ) components. For the sake of simplicity, the mean-structure (i.e., intercepts) of the model is not depicted.

Geiser et al. [16] extended the STMS model to capture the (in)consistency of trait scores across different fixed situations (e.g., trait anxiety in a neutral vs. threatening situation; see Figure 3 on p. 173 of their paper). This revised STMS involves the simultaneous estimation of the same STMS model within each situation (e.g., A and B), thereby allowing the correlation between the resulting latent trait factors ξ_A and ξ_B to be interpreted as an index of the consistency or convergence of the trait scores across the two situations of interest. For our purposes, the revised STMS can also be used to capture (dis)agreement between informants at the trait and state levels. For instance, for caregivers' and teachers' ratings of children's sympathy with a set of items invariant in their content both over time and across informants, the revised STMS allows for the computation of two relative (rank-order) consistency indexes: (1) the time-unspecific cross-informant correlation coefficient at the trait level (ξ _caregiver with ξ _teacher) with a squared value indicating the degree of cross-informant consistency at the dispositional level of children's sympathy (i.e., both informants rated child A as, in general, more sympathetic than child B); (2) the time-specific cross-informant correlation at the state level (ζ _caregiver with ζ _teacher at time t; see Figure 2) with a squared value indicating the degree of cross-informant consistency at the momentary, fluctuating level of children's sympathy (i.e., both informants rated child A as more sympathetic than child B at a specific time point). Importantly, since latent means are estimated for trait factors, absolute mean-level differences across informants in the construct of interest (e.g., trait sympathy) can also be investigated via latent difference score (LDS) models (see [19]). The absolute mean-level differences represent a further index of (dis)agreement as they indicate to what extent both observers perceive children as having exactly the same mean level of dispositional sympathy (this index is similar to the concept of absolute stability in personality psychology; [20]).

FIGURE 2

Figure 2. Combined Singletrait-multistate (STMS) Model for Three Waves and Two Informants. Latent variables indicate both trait (ξ) and state (ζ) components for each informant. Cross-informant trait consistency coefficient (T-CC) and cross-informant occasion-specific consistency coefficients (OS-CC) are reported. For the sake of simplicity, the mean-structure (i.e., intercepts) of the model is not depicted.

All these coefficients (dispositional, state, and absolute) reflect distinct indexes of cross-informant (dis)agreement. Failing to distinguish and understand them may lead to misleading interpretations/diagnoses in multi-informant assessment practices (e.g., the ASEBA system; [21]) which, in turn, may affect the selection of appropriate intervention strategies for children.

Finally, three other advantages of the LST approach are worthy of mention. First, the STMS model disentangles true trait and state components using latent variables (ξ and ζ) that are free of measurement error, which is often considered a serious concern in this area of research [2]. Second, it allows us to ascertain the presence of possible differences between caregivers and teachers in their use of the instruments/ratings of items by testing a series of increasingly restrictive measurement invariance models (i.e., configural, metric, and scalar). Establishing strong (scalar) measurement invariance across informants allows us to interpret cross-informant differences as true disagreements rather than as biases due to differential use of the rating scales ([19]; also see [22]). Third, the LDS model allows the inclusion of predictors (e.g., children's gender) to explain mean-level inconsistencies across informants ([16]; for a more technical introduction to LDS models, see [23]).

The Present Study

In sum, existing evidence suggests small and not statistically significant cross-informant agreement in the assessment of children's sympathy, especially when informants (i.e., caregivers and teachers) reported children's sympathy from different contexts (i.e., home vs. school; [11]). However, these studies have failed to separate convergence in evaluations of children's dispositional sympathetic tendencies from convergence in evaluations of the fluctuating components of children's sympathy. Moreover, previous works did not clearly focus on distinguishing between agreement in terms of rank-order consistency (e.g., child A is consistently rated as more sympathetic than child B by both informants) and absolute mean-level agreement (e.g., child A has exactly the same mean level of dispositional sympathy according to both informants).

In the present study, we aimed to fill this gap using analyses grounded in LST theory [4] and its conceptual extension for fixed situations [16]. Specifically, we investigated the convergence of caregivers' and teachers' evaluations of children's sympathy at the trait and state level from age 6 to 12. We expected a higher degree of rank-order convergence between the evaluations of caregivers and teachers at the stable, trait level of children's sympathy (i.e., in terms of how much the child is sympathetic in general) compared to the ephemeral, state level of their sympathy at each time point. We also modeled absolute mean-level (dis)agreement across caregivers and teachers via LDS analysis. Finally, since previous studies reported girls as more sympathetic than boys [5], we explored possible differences in mean-level discrepancies of sympathy between genders.

Methods

Participants

For illustrative purposes of the STMS model, we analyzed data published in Zuffianò et al. [10]. Data were from a cohort of 6-year-olds (reassessed at ages 9 and 12) from the Swiss Survey of Children and Youth (COCON), a nationally representative study of social-emotional development. At time 1 (T1), 1,273 children (49% girls; M_age = 6.17 years, SD = 0.22) participated alongside 1,199 primary caregivers (93% biological mothers) and 870 teachers. At time 2 (T2), 1,101 primary caregivers and 853 teachers provided data, and 1,022 caregivers and 734 teachers did so at time 3 (T3).

Measures

Sympathy

Caregivers and teachers rated children's sympathy (from 1 = not at all true to 6 = always true) using a widely used scale [24]. For analytical purposes, we only used the three items of the scale (i.e., “feels sorry for others,” “feels sorry for other children who are being teased,” and “feels sorry for other children who are sad or upset”) that were content-invariant across time points and informants. In addition to allowing for our proposed analyses (which are contingent on content invariance), these items captured the prototypical “feeling sorrow” component that is considered the core of sympathy [10]. Omega reliability coefficients were 0.663 (95% CI [0.610, 0.716]) at T1, 0.800 (95% CI [0.767, 0.833]) at T2, and 0.768 (95% CI [0.726, 0.809]) at T3 for caregiver reports, and 0.908 (95%CI [0.893, 0.923]) at T1, 0.924 (95% CI [0.909, 0.940]) at T2, and 0.919 (95%CI [0.903, 0.935]) at T3 for teacher reports.

Results

Descriptive Statistics

As reported in Table 1, sympathy scores at the manifest level were always positively and statistically significantly correlated. Focusing on cross-informant correlations, caregivers and teachers only showed a small degree of convergence, both concurrently (rs ranged from 0.208 to 0.254) and over time (rs ranged from 0.134 to 0.207). As expected, boys were consistently rated as less sympathetic than girls.

TABLE 1

Table 1. Correlations, means, and standard deviations (SD) of sympathy.

STMS Results

First, we estimated an STMS model within each informant and ascertained the tenability of time-invariant factor loadings and intercepts by testing a series of increasingly restrictive measurement invariance assumptions (i.e., configural, metric, and scalar; [22]). We then compared these nested STMS models using the Δχ² test. However, because the Δχ² test is sensitive to sample size, we also considered changes in comparative-fit-index (ΔCFI) lower than 0.010 as indicative of measurement invariance between these nested models [25]. When equality constraints on factor loadings and item intercepts were not tenable, we tested less restrictive models by relaxing some parameter constraints in order to have, at least, partial scalar invariance (i.e., metric and scalar invariance in at least one item beyond the marker item; [26]). Second, we estimated a cross-informant STMS model combining the caregiver- and teacher-reported STMS models to evaluate their degree of convergence at the trait and state level. We also tested cross-informant measurement invariance to ensure that differences in children's sympathy scores from caregivers and teachers reflected true informant-based discrepancies. Finally, we explored possible mean-level differences in children's trait-level sympathy using an LDS model [16, 23].

To identify our latent variables, we fixed the factor loading of the marker item to 1 and its intercept to 0. We evaluated model fit according to standard criteria [27]. Specifically, we considered CFI and Tucker-Lewis-index (TLI) values >0.90, and root-mean-square-error-of-approximation (RMSEA) values <0.08 (with a 90% confidence interval; CI) as indicators of acceptable model fit [27]. We ran our analyses in Mplus 8 [28] and we accounted for missing data with full information maximum-likelihood estimation of the parameters (MLR)².

Caregiver Reports

As reported in Table 2, we established longitudinal partial scalar invariance for the STMS model according to the ΔCFI criterion. Only the factor loading (at the trait level) and intercept of the item “feels sorry for other children who are sad or upset” were relaxed to be different at T1. Interestingly, squared standardized loadings (see Table 3) indicated that approximately 23–38% of the variance of the items stemmed from trait-level variability (average trait consistency coefficient ≈31%) whereas only 16–24% reflected state-level variability (average occasion-specificity coefficient ≈ 20%; see [29, 30]). Hence, although a large part of the variability of the items was unexplained by the STMS model, caregiver reports mostly captured children's trait sympathetic tendencies rather than their occasion-specific sympathetic manifestations.

TABLE 2

Table 2. Measurement Invariance.

TABLE 3

Table 3. Factor loadings, intercepts, and variances from final STMS models.

Teacher Reports

We established full longitudinal scalar invariance for the STMS model involving teacher reports of children's sympathy, as the ΔCFI was lower than 0.01 at each step of the measurement invariance analysis (see Table 2). Unlike caregiver reports (see Table 3), squared standardized loadings of the items indicated that teachers mostly captured children's sympathy at the state level (variance ranging from 50 to 63%, average occasion-specificity coefficient ≈54%) rather than at the trait level (variance ranging from 23 to 27%, average trait consistency coefficient ≈25%).

Cross-Informant STMS

The STMS model with partial scalar invariance across informants³ (the factor loading and intercept of the caregiver-reported item “feels sorry for other children who are sad or upset” were not constrained to equality) showed a good fit to the data, ${\chi }_{(155)}^2$ = 309.825, scf = 1.112, p < 0.001, CFI = 0.973, TLI = 0.974, RMSEA = 0.028, 90% CI [0.024, 0.033], and was not statistically different [$\Delta {\chi }_{(4)}^2$ = 2.993, p = 0.559; ΔCFI = 0.000] from the partial metric invariance model, ${\chi }_{(151)}^2$ = 307.041, scf = 1.111, p < 0.001, CFI = 0.973, TLI = 0.973, RMSEA = 0.029, 90% CI [0.024, 0.033]. The partial metric invariance model, in turn, was not statistically different from the configural model [$\Delta {\chi }_{(4)}^2$ = 4.928, p = 0.295; ΔCFI = 0.000]. Hence, children's sympathy scores could be meaningfully compared across caregivers and teachers. As expected (see Figure 3), caregivers and teachers showed a different degree of rank-order convergence when children's sympathetic scores were disentangled at the trait and state levels. Specifically, caregivers and teachers reported a higher degree of cross-informant consistency at children's trait level of sympathy (r = 0.510, 95% CI [0.468, 0.549], p < 0.001), compared to their state level (r = 0.123, 95% CI [0.069, 0.177], p = 0.002 at each time point), with cross-informant agreements of 26 and 2%, respectively.

FIGURE 3

Figure 3. Combined Singletrait-multistate (STMS) Model of Children's Sympathy across Caregivers and Teachers. Latent variables indicate both trait (ξ) and state (ζ) components for each informant. Cross-informant trait consistency coefficient and cross-informant occasion-specific consistency coefficients were statistically significant (p < 0.01).

The presence of partial scalar invariance also allowed us to investigate absolute mean-level (dis)agreement across informants. Overall, caregivers (mean ξ _parent = 5.205, 95% CI [5.166, 5.243]) rated their children as more sympathetic than teachers did (mean ξ _teacher = 4.906, 95% CI [4.853, 4.959]). Constraining the two latent trait means to be equal across informants (${\chi }_{(156)}^2$ = 421.495, scf = 1.112, p < 0.001, CFI = 0.954, TLI = 0.955, RMSEA = 0.037, 90% CI [0.032, 0.041]) worsened the model fit of the partial scalar STMS model [$\Delta {\chi }_{(1)}^2$ = 125.445, p < 0.001; ΔCFI = 0.019], thereby revealing statistically significant differences at the mean-level perceptions of children's sympathy across informants. Hence, although parents and teachers showed a moderately high degree of convergence in ranking children relative to their peers based on their dispositional sympathy (e.g., both rated child A as generally more sympathetic than child B), they showed significant differences in capturing the exact mean level of each child's sympathy (e.g., caregiver ratings of children A and B could be 4.3 and 3.8, respectively, whereas teacher ratings of the same children could be 3.9 and 3.2, respectively).

To further investigate these absolute mean-level differences at the trait level, we used a LDS analysis [16, 19] in which we estimated a second-order latent difference factor (Δf) representing the difference between teachers and caregivers (ξ _teacher − ξ _caregiver). In the LDS model, ${\chi }_{(155)}^2$ = 309.825, scf = 1.112, p < 0.001, CFI = 0.973, TLI = 0.974, RMSEA = 0.028, 90% CI [0.024, 0.033], the mean (−0.299, p < 0.001) of Δf was statistically significant, indicating, on average, a lower mean value of teacher-reported sympathy compared to caregiver-reported sympathy. Specifically, using Cohen's guidelines [31], the latent mean-level difference between caregivers and teachers could be interpreted as a medium effect (Cohen's d = −0.561, 95% CI [−0.641, −0.481])⁴. The variance of Δf was also statistically different from zero (0.291, p < 0.001), highlighting significant inter-individual differences (i.e., caregivers and teachers perceived some children as more different than others). A final conditional LDS model, ${\chi }_{(171)}^2$ = 347.473, scf = 1.112, p < 0.001, CFI = 0.971, TLI = 0.971, RMSEA = 0.029, 90% CI [0.024, 0.033], revealed that children's gender (girls = 0, boys = 1) predicted the Δf (β = −0.490, p < 0.001, 95% CI [−0.574, −0.406]), suggesting that discrepancies between teachers and caregivers (ξ _teacher − ξ _caregiver) were stronger for boys than girls. Specifically, compared to girls, teachers rated boys much lower than caregivers did (Cohen's d = −1.125, 95% CI [−1.244, −1.006]).

Discussion

Understanding the nature of informant discrepancies has attracted the attention of many psychological researchers. This is because this diagnostic information yields potentially important implications when making decisions regarding the selection and implementation of intervention practices aimed at enhancing children's social-emotional development and wellbeing. According to meta-analytic findings, only a small-to-moderate degree of convergence (r = 0.28; [32]) exists between different types of reporters, such as caregivers and teachers, and this weak agreement tends to be even lower for less observable constructs, such as children's internal affective responses (r = 0.21; [33]). Hence, prominent developmental and clinical psychologists have emphasized the importance of a multi-informant approach to social-emotional and behavioral assessment because situation-specific effects may reveal meaningful variability in such constructs across contexts (e.g., home vs. school; [32, 33]).

In the present study, we highlighted how recent conceptualizations of LST theory [16] can inform children's multi-informant assessment by clearly indicating the level of analysis at which (dis)agreement between informants occurs. We showed that when trait- and state-level variability are distinguished within each informant, two types of relative (rank-order) consistency coefficients can be computed to reflect inter-rater agreement: (1) the trait consistency coefficient (i.e., time-unspecific cross-informant agreement at the trait level of the psychological attribute) and (2) the occasion-specific consistency coefficient (i.e., time-specific cross-informant agreement at the state level of the psychological attribute). To illustrate the advantages of separating these two indexes, we examined the level of (dis)agreement between caregivers and teachers in the evaluation of children's sympathy.

At the manifest level, we found that correlations of children's sympathy across informants were low (rs ranging from 0.13 to 0.25), reflecting a small amount of agreement between caregivers and teachers. This aligns with previous findings reporting only a small degree of convergence between caregivers and teachers in the assessment of children's sympathy [11]. This overall small effect could lead researchers to conclude that only minimal agreement exists between caregivers and teachers and, therefore, that children's sympathetic responses are highly variable across contexts. As a consequence, this high discrepancy may create problems in properly identifying children who may benefit from timely social-emotional interventions to promote their sympathy [34, 35].

Yet, our LST analysis revealed a more complex picture of cross-informant convergence. First, by establishing cross-informant measurement invariance (at the partial scalar level; [26]), we were able to confidently interpret the relations between caregivers' and teachers' evaluations as reflecting true (dis)agreement rather than methodological biases in their use of the scale. Second, we found that teachers' and caregivers' scores were differentially affected by occasional manifestations of children's sympathy: although both caregivers and teachers attributed a consistent amount of children's sympathetic responses to their dispositional, trait-like characteristics, teachers were more likely than caregivers to capture situational, state-like manifestations of children's sympathy. This difference could also be due to the fact that teachers were different across time (whereas caregivers, mostly mothers, did not change over the duration of the study).Third, cross-informant convergence was different when children's sympathy scores were decomposed into trait and state components. As expected, caregivers and teachers showed moderately high agreement (r = 0.510) in their ratings of children's dispositional tendency to feel sympathetic concern, yet fairly low agreement in their ratings of children's momentary manifestations of sympathy at each time point (r = 0.123). Thus, differently from the correlational results at the manifest level, we found that caregivers and teachers did agree in terms of identifying children who were, in general, more sympathetic than others. Although this result could be interpreted as further evidence of the relative stability (and visibility) of psychological traits across contexts (e.g., [36]), it may also indirectly reveal information about the inter-rater agreement concerning the causes of children's emotional responses. According to the Attribution Bias Context Model (ABC; [3]), the considerable cross-informant consistency at the trait-level could be related to the fact that informants—such as caregivers and teachers—tend to interpret children's social-emotional development and behaviors in terms of dispositional tendencies (i.e., child A is more sympathetic in general than child B; [3]). In line with this claim, our LST analysis indicated that both caregivers and teachers captured a considerable portion of the dispositional nature of children's sympathy (although teacher ratings were more state- than trait-sensitive). Hence, properly isolating agreement at the level at which both informants most attribute the causes of children's psychological functioning (i.e., the dispositional level) can thus result in relatively high convergence between them, even for a less manifest emotional response like sympathy and for caregivers and teachers who report from different contexts of observation. Interestingly, teachers and caregivers also showed a small, nearly negligible amount of agreement at the state level, reflecting the fluctuating, momentary deviations of children's sympathy from their general disposition. Hence, situational positive (or negative) spikes in sympathy seemed to have some marginal, time-specific consistency across contexts, which jointly affected caregiver and teacher reports of children's sympathy at each time point.

Although teachers and caregivers generally agreed in terms of identifying children who were more sympathetic than others, we also found that they moderately disagreed regarding the exact, “true” mean level of each child's dispositional sympathy. Specifically, teacher-reported latent scores were consistently lower than caregiver-reported latent scores. This may be because sympathy is not a highly visible emotional state at school. A child can feel concern for his/her classmates without displaying an obvious emotional response or engaging in immediate prosocial actions that can be clearly seen by the teacher (who is also responsible for numerous other students). From this perspective, parents have the benefit of one-on-one time that increases the chances of gaining insight into their child's sympathetic tendencies. In line with Funderburk et al. [14], it may also be the case that caregiver ratings are more positive than teacher ratings because of the strong emotional bond underlying the parent–child relationship. Moreover, caregivers and teachers may rely on different cues: they report from different contexts of observation characterized by distinct relationships and opportunities for social interaction which, in the end, provide them with different reference points to calibrate their assessments of children's sympathy (e.g., interactions with siblings vs. classmates). Realistically, the abovementioned factors could be jointly responsible for the overall lower dispositional scores of children's sympathy reported by teachers vs. caregivers.

Finally, we modeled and explained mean-level discrepancies at the trait level using a LDS framework [16] and found systematic, statistically significant variability in how much children were rated lower in sympathy by teachers vs. caregivers. Moreover, this variability was predicted by children's gender, such that boys' evaluations were consistently more discrepant (i.e., they were lower in teacher- vs. caregiver-reported dispositional sympathy). This finding may stem from gender-typed socialization practices, which could predispose boys to show less sympathy (especially at school where they interact—or at least have the opportunity to interact—more heavily with other peers and adults), thereby reinforcing teachers' stereotypical view of boys as much less sympathetic than girls [37]. In addition, boys may express their sympathetic concern in qualitatively different ways from girls (e.g., via non-verbal behaviors such as patting on the shoulder), which might not be easily captured by teachers in the classroom context. Hence, more work is needed to develop social-emotional instruments that include a variety of indicators that tap into both verbal and non-verbal aspects of sympathy-related responding.

Limitations

Despite its strengths, our current approach also has some limitations rooted in LST theory/methodology that may hinder its use for understanding informant discrepancies. First, the STMS requires the use of valid questionnaires that include content-invariant items across raters to establish cross-informant measurement invariance. Although there are some valid multi-informant assessment tools (e.g., The “Child Behavior Checklist”; [21]; the “Strengths and Difficulties Questionnaire”; [38]), numerous questionnaires used in the literature have been developed to capture the perspective of a specific informant (e.g., self-reports for self-efficacy scales), potentially limiting the use of our current approach for these constructs. Second, because some psychological attributes are more state-like than trait-like by nature (e.g., happiness), researchers should carefully plan appropriate time lags across measurement points to properly model trait and state variability (and to measure associated cross-informant convergence). Third, directly related to the previous point, the STMS assumes the presence of longitudinal data [16], which, very often, is not feasible for several reasons (e.g., time constraints, costs, etc.). Thus, in the absence of longitudinal data, we advise making the level of analysis at which raters should focus their evaluations clear to them (i.e., in the instructions for a particular questionnaire, specify if the rater should focus on how the child generally feels/behaves vs. how the child felt/behaved in the last day[s], week[s], or month[s]), thereby increasing the likelihood of convergence between different informants using the scale.

Conclusions and Future Directions

Although different informants likely capture unique and diverse aspects of children's social-emotional functioning, the extent of their disagreement might be erroneously exacerbated by a mismatch or confusion regarding the level (i.e., dispositional vs. situational) at which their assessments are focused. In the present study, we used LST analysis to disentangle these two levels of analysis and we showed how teachers and caregivers had a moderately high degree of convergence in how they evaluated children's dispositional sympathetic tendencies (which is perhaps even more surprising given that sympathy is a relatively difficult internal process to observe). We also highlighted the importance of considering absolute, mean levels of cross-informant (dis)agreement and gender differences thereof.

Finally, our findings may also offer some suggestions to help researchers develop better tools to assess essential dimensions of social-emotional functioning in childhood across different informants and contexts. For instance, future multi-informant assessments may benefit from including ad hoc open questions designed to capture important events (e.g., a specific sympathetic response or related behavior observed) that could account for occasion-specific cross-informant agreement. Moreover, future scales should clearly list the different reference points that can be used to compare children on the basis of psychological functioning (e.g., siblings, classmates, peers in general, etc.) in order to ease the convergence across informants, especially when they report from different contexts of observation (e.g., home vs. school).

Ethics Statement

The current study was conducted in Switzerland and consisted of non-invasive and unconstrained parent and teacher questionnaires. According to the regulations in the canton of Zurich in Switzerland (the so-called Regulations of the Ethics Commission for Psychological Research, 2011), there was no requirement for an ethics committee approval when the study was conducted. According to this regulation (article 5, paragraph 1), this study was exempted from requiring formal ethical approval. The study fully complies with the ethics guidelines given by this legal regulation (see article 8, paragraph 2). The regulation is based on the Ethical Principals of Psychologists and Code of Conduct (as outlined in the so-called Ethical Guidelines for Psychologists of the Swiss Society for Psychology, as amended on October 13, 2003) and the ethical standards of the American Psychological Association (APA). Written and informed consent was obtained from all research participants and from the parents/legal guardians of all non-adult participants. The data were analyzed anonymously.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Funding

This research was funded by the Swiss National Science Foundation and the Jacobs Foundation.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors would like to express their sincere thanks to the children, parents, and teachers for participating in the study. Moreover, the authors are grateful to all the interviewers and undergraduate students for their help in data collection and coding.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fams.2018.00008/full#supplementary-material

Footnotes

1. ^Throughout this paper, we utilize notations consistent with Geiser et al. [16].

2. ^With MLR estimation, the formula for Δχ² also includes the scaling correction factor (scf).

3. ^In this STMS model, we also constrained the covariances of the residual latent state factors over time to equality (ζ _caregiver with ζ _teacher at T1 = ζ _caregiver with ζ _teacher at T2 = ζ _caregiver with ζ _teacher at T3). The Mplus syntax for this model is reported in the Supplementary Material.

4. ^A latent mean score of zero of the Δf would have meant perfect, absolute mean-level agreement between caregivers and teachers in evaluating children's dispositional sympathy.

References

1. De Los Reyes A, Thomas SA, Goodman KL, Kundey SMA. Principles underlying the use of multiple informants' reports. Annu Rev Clin Psychol. (2013) 9:123–49. doi: 10.1146/annurev-clinpsy-050212-185617

PubMed Abstract | CrossRef Full Text | Google Scholar

2. De Los Reyes A, Kazdin AE. Measuring informant discrepancies in clinical child research. Psychol Assess (2004) 16:330–4. doi: 10.1037/1040-3590.16.3.330

PubMed Abstract | CrossRef Full Text | Google Scholar

3. De Los Reyes A, Kazdin AE. Informant discrepancies in the assessment of childhood psychopathology: a critical review, theoretical framework, and recommendations for further study. Psychol Bull. (2005) 131:483–509. doi: 10.1037/0033-2909.131.4.483

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Steyer R, Ferring D, Schmitt MJ. States and traits in psychological assessment. Eur J Psychol Assess. (1992) 8:79–98.

Google Scholar

5. Eisenberg N, Spinrad TL, Knafo A. Prosocial development. In: Lamb ME, Lerner RM, editors. Handbook of Child Psychology and Developmental Science, Vol. 3: Social, Emotional, and Personality Development. 7th edn. New York, NY: Wiley (2015). p. 610–56.

Google Scholar

6. Malti T, Sette S, Dys SP. Socioemotional responding: a perspective from developmental psychology. In: Scott R, Buchmann M, Kosslyn S, editors. Emerging Trends in the Social and Behavioral Sciences. Hoboken, NJ: John Wiley and Sons (2016). p. 1–15.

7. Malti T, Song J-H. Social-emotional development and aggression. In: Malti T, Rubin KH, editors. Handbook of Child and Adolescent Aggression: Emergence, Development, and Intervention. New York, NY: Guilford Press (2018).

8. Kimonis ER, Frick PJ, Muñoz LC, Aucoin KJ. Callous-unemotional traits and the emotional processing of distress cues in detained boys: testing the moderating role of aggression, exposure to community violence, and histories of abuse. Dev Psychopathol. (2008) 20:569–89. doi: 10.1017/S095457940800028X

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Eisenberg N, Spinrad T, Morris A. Empathy-related responding in children. In: Killen M, Smetana J, editors. Handbook of Moral Development. 2nd ed. New York, NY: Psychology Press (2014). p. 184–207.

10. Zuffianò A, Colasante T, Buchmann M, Malti T. The co-development of sympathy and overt aggression from middle childhood to early adolescence. Dev Psychol. (2017) 54:98–110. doi: 10.1037/dev0000417

CrossRef Full Text

11. Kienbaum J. The development of sympathy from 5 to 7 years: increase, decline or stability? A longitudinal study. Front Psychol. (2014) 5:468. doi: 10.3389/fpsyg.2014.00468

CrossRef Full Text | Google Scholar

12. Malti T, Eisenberg N, Kim H, Buchmann M. Developmental trajectories of sympathy, moral emotion attributions, and moral reasoning: the role of parental support. Soc Dev. (2013) 22:773–93. doi: 10.1111/sode.12031

CrossRef Full Text | Google Scholar

13. Murphy BC, Shepard SA, Eisenberg N, Fabes RA, Guthrie IK. Contemporaneous and longitudinal relations of young adolescents' dispositional sympathy to their emotionality, regulation, and social functioning. J Early Adolesc. (1999) 19:66–97.

Google Scholar

14. Funderburk BW, Eyberg SM, Rich BA., Behar L. Further psychometric evaluation of the Eyberg and Behar rating scales for parents and teachers of preschoolers. Early Educ Dev. (2003) 14:67–81. doi: 10.1207/s15566935eed1401_5

CrossRef Full Text

15. Stern JA, Cassidy J. Empathy from infancy to adolescence: an attachment perspective on the development of individual differences. Dev Rev. (2017) 47:1–22. doi: 10.1016/j.dr.2017.09.002

CrossRef Full Text | Google Scholar

16. Geiser C, Bishop J, Lockhart G. Collapsing factors in multitrait-multimethod models: examining consequences of a mismatch between measurement design and model. Front Psychol. (2015) 6:946. doi: 10.3389/fpsyg.2015.00946

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Millsap RE. Statistical Approaches to Measurement Invariance. New York, NY: Taylor and Francis Group (2011).

Google Scholar

18. Widaman KF, Ferrer E, Conger, RD. Factorial invariance within longitudinal structural equation models: measuring the same construct across time. Child Dev Perspect. (2010) 4:10–8. doi: 10.1111/j.1750-8606.2009.00110.x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. de Haan A, Prinzie P, Sentse M, Jongerling J. Latent difference score modeling: a flexible approach for studying informant discrepancies. Psychol Assess. (2017) doi: 10.1037/pas0000480. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Santor DA, Bagby RM, Joffe RT. Evaluating stability and change in personality and depression. J Pers Soc Psychol. (1997) 73:1354–62. doi: 10.1037/0022-3514.73.6.1354

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Achenbach TM, Rescorla LA. Manual for the ASEBA School-Age Forms and Profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, and Families (2001).

22. Vandenberg RJ, Lance CE. A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research. Organ. Res. Methods (2000) 3:4–70. doi: 10.1177/109442810031002

CrossRef Full Text | Google Scholar

23. McArdle JJ, Hamagami F. Latent difference score structural models for linear dynamic analyses with incomplete longitudinal data. In: Collins LM, Sayer AG, editors. New Methods for the Analysis of Change. Washington, DC: APA Press (2001). p. 139–75.

Google Scholar

24. Eisenberg N, Fabes RA, Murphy B, Karbon M, Smith M, Maszk P. The relations of children's dispositional empathy-related responding to their emotionality, regulation, and social functioning. Dev Psychol. (1996) 32:195–209. doi: 10.1037/0012-1649.32.2.195

CrossRef Full Text | Google Scholar

25. Cheung GW, Rensvold RB. Evaluating goodness-of-fit indices for testing measurement invariance. Struct Eq Model. (2002) 9:233–55. doi: 10.1207/S15328007SEM0902_5

CrossRef Full Text | Google Scholar

26. Byrne BM, Shavelson RJ, Muthén B. Testing for the equivalence of factor covariance and mean structures: the issue of partial measurement invariance. Psychol Bull. (1989) 105:456–66. doi: 10.1037/0033-2909.105.3.456

CrossRef Full Text | Google Scholar

27. Kline RB. Principles and Practices of Structural Equation Modeling. 3rd edn. New York, NY: Guilford (2010).

28. Muthén LK, Muthén BO. Mplus User's Guide. 8th edn. Los Angeles, CA: Muthén and Muthén (1998-2017).

29. Geiser C, Keller BT, Lockhart G. First- versus second-order latent growth curve models: some insights from latent state-trait theory. Struct Eq Model. (2013) 20:479–503. doi: 10.1080/10705511.2013.797832

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Geiser C, Hintz F, Burns G, Servera M. Latent variable modeling of person-situation data. In: Rauthmann JF, Sherman R, Funder DC, editors. The Oxford Handbook of Psychological Situations. New York, NY: Oxford University Press (2017). p. 1–57.

Google Scholar

31. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd edn. Hillsdale, NJ: Lawrence Erlbaum Associates (1988).

32. Achenbach TM, McConaughy SH, Howell CT. Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychol. Bull. (1987) 101:213–32. doi: 10.1037/0033-2909.101.2.213

PubMed Abstract | CrossRef Full Text | Google Scholar

33. De Los Reyes A, Augenstein TM, Aldao A, Thomas SA, Daruwala SE., Kline K, et al. Implementing psychophysiology in clinical assessments of adolescent social anxiety: use of rater judgments based on graphical representations of psychophysiology. J Clin Child Adolesc Psychol. (2015) 44:264–79. doi: 10.1080/15374416.2013.859080

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Durlak JA, Weissberg RP, Dymnicki AB, Taylor RD, Schellinger KB. The impact of enhancing students' social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev. (2011) 82:405–32. doi: 10.1111/j.1467-8624.2010.01564.x

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Malti T, Chaparro MP, Zuffianò A, Colasante T. School-based interventions to promote empathy-related responding in children and adolescents: a developmental analysis. J Clin Child Adolesc Psychol. (2016) 45:718–31. doi: 10.1080/15374416.2015.1121822

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Church AT, Katigbak MS, Reyes JAS, Salanga MGC, Miramontes LA, Adams, NB. Prediction and cross-situational consistency of daily behavior across cultures: testing trait and cultural psychology perspectives. J Res Pers. (2008) 42:1199–215. doi: 10.1016/j.jrp.2008.03.007

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Chaplin TM, Aldao A. Gender differences in emotion expression in children: a meta-analytic review. Psychol Bull. (2013) 139:735–65. doi: 10.1037/a0030737

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Goodman R. The strengths and difficulties questionnaire: a research note. J Child Psychol Psychiatry. (1997) 38:581–6. doi: 10.1111/j.1469-7610.1997.tb01545.x

PubMed Abstract | CrossRef Full Text | Google Scholar