Methodology Is Destiny: The Effect of Survey Prompts on Reported Levels of Giving and Volunteering
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Methodology Is Destiny Rooney et al. 10.1177/0899764004269312 Methodology Is Destiny: The Effect of Survey Prompts on Reported Levels of Giving and Volunteering Patrick Rooney Kathryn Steinberg Indiana University and Indiana University–Purdue University at Indianapolis Paul G. Schervish Boston College and Indiana University This article extends earlier methodological tests of giving and volunteering in Indiana to a large (N = 4,200) cross-sectional sample collected in the United States in the fall of 2001. The authors find that the results are consistent with those found in the earlier analyses, namely, that longer, more detailed prompts led respondents to recall giving and volunteering at higher incidence rates (proportion donating at all or volunteering at all) and at higher levels (dollars given or hours volunteered) than when compared to survey methodologies with fewer prompts. Keywords: giving; volunteering; measurement; methodology This article extends earlier work conducted by the Center on Philanthropy at Indiana University on the implications of different survey methodologies for levels of reported giving and volunteering. In our earlier research, we studied the methodological implications of question design for a sample of house- holds in Indiana (Rooney, Steinberg, & Schervish, 2001). Here we report the Note: The authors would like to thank Frank Walker and Walker Information for their expertise and assistance with the data gathering and Ronald Kessler and Elizabeth Martin for their help in reflecting on directions for further methodological inquiry. We are especially grateful to William Chin and Kiyoko Kamimura for their help with the econometrics. We also thank Jeff Small for his research assistance. Finally, we would like to thank the referees for several good suggestions and insights that significantly enhanced this article. Address correspondence to Patrick Rooney or Kathryn Steinberg at Center on Philanthropy at Indiana University, 550 W. North Street, Suite 301, Indianapolis, IN 46202; phone: 317-278-8909; fax: 317-684-8900; e-mail: rooney@iupui.edu, ksteinbe@iupui.edu Nonprofit and Voluntary Sector Quarterly, vol. 33, no. 4, December 2004 628-654 DOI: 10.1177/0899764004269312 © 2004 Association for Research on Nonprofit Organizations and Voluntary Action 628 Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 629 findings from a parallel methodological study from a large (N = 4,200), cross- sectional, national sample collected in the fall of 2001. The results from this national study are consistent with those from the earlier Indiana study, namely, that longer, more detailed prompts when compared to less detailed prompts result in respondents reporting higher participation rates and greater amounts of charitable giving and volunteering. Rooney et al. (2001) and Steinberg, Rooney, and Chin (2002) show that there are marked differences in reported giving and volunteering among several sets of randomly selected Indiana residents. Those who were asked the most detailed sets of questions about their giving patterns reported that they gave, on average, $1,000 more per household than respondents given a short survey, after controlling for differences in age, race, income, education, tax status, and marital status. Likewise, respondents who responded to longer, more detailed questionnaires about volunteering reported close to 200 more volunteer hours per year than those who responded to a short set of questions. These results were consistent whether using ordinary least squares (OLS), Tobit, or two- stage OLS (Heckman adjustment) regression techniques. Having established the methodological principle with Indiana—that more detailed questioning leads to higher reported incidence and amounts of giving and volunteering— the next question was whether the principle would hold true for the new and independent data composing a larger representative random sample of the U.S. population. METHODOLOGY In testing whether the methodological principle held true for the U.S. sam- ple, we compared and contrasted the findings from five different survey mod- ules. The AREA module replicated questions on giving and volunteering from Independent Sector (e.g., 1999) surveys that derive an overall level of giving and volunteering for a respondent’s household by totaling the amounts the respondent gives to particular areas of concern such as education, religion, health care, and so forth. The METHOD module replicated the measurement items from a survey conducted by Statistics Canada (Hall, McKeown, & Roberts, 2001), which included several prompts based on “method” of fund- raising contact such as direct mail, special events, and the like, or types of vol- unteering such as raking leaves, serving food, and so on. The Panel Study of Income Dynamics (PSID) survey module, designed by the Center on Philan- thropy and the University of Michigan’s PSID staff (for a longitudinal study called the Center on Philanthropy Panel Study or COPPS), contained several prompts based on key areas of giving and a screening question for whether someone volunteered followed by a question about the amount of volunteer- ing in the human services subsector. The fourth question set was the SHORT module, which aside from the standard demographic questions contained only the following questions: “Did you give last year?” “If so, how much?” Downloaded from nvs.sagepub.com by guest on February 27, 2015
630 Rooney et al. “Did you volunteer last year?” “If so, how much?” Finally, we fielded the METHOD/AREA module (based in part on O’Neill & Roberts, 2000) that obtains information on giving by prompting first by method of fund-raising contact, then by subsector of giving for each possible combination. For in- formation on volunteering, this module simply relied on the relevant ques- tions from the PSID module. To improve the response and completion rate for the METHOD/AREA module, which in some cases could take as long as 90 minutes, we offered 75% of the respondents the inducement of a long-distance calling card. The remaining 25% were not offered this inducement so that we could obtain information about whether such documents significantly affected response and completion rates. Our previous research (Rooney et al., 2001) indicated that inducements were particularly effective at increasing response rates in that module. To learn whether such inducements were equally needed or helpful for the other four modules, we offered the same incentive to approxi- mately 100 respondents taking those other modules. Table 1 summarizes the modules by the types of giving and volunteering prompts, the number of questions, and inducements. The appendix provides a more detailed descrip- tion and examples of the prompts for each module. The telephone interviews were conducted by Walker Information, head- quartered in Indianapolis. Callers used random digit dialing of households to obtain separate samples of at least 800 respondents for each module. Any given household participated in only one survey module. Approximately two thirds of the way through the data gathering, the Sep- tember 11 attack on America occurred. We suspended the telephone inter- views for a month immediately following the attacks. After 1 month, we resumed the telephone calls but added six tragedy-related questions to the beginning of each module asking about donations of money, blood, or other necessities, and/or volunteer time to the September 11 victims and to related causes. Because calling for AREA and SHORT modules had been completed prior to September 11, we added 100 additional respondents to these modules to ensure that each module had some respondents from both before and after the tragedy. In a separate analysis, Steinberg and Rooney (2002) investigate the correlates of the tragedy-related giving and whether there is any evidence that September 11 giving affected giving to other causes. It should be noted that in each module, we asked respondents about house- hold giving. With the exception of the survey of giving in Canada (Hall et al., 2001), this is the unit of inquiry that virtually all researchers have used when collecting data on giving. Household giving is of interest because it is more comparable to data reported on tax returns and therefore could, in theory, be verified with Internal Revenue Service (IRS) data. The household is also the unit that these surveys use to gather information on income and other socio- economic demographics. In calculating an annual amount of volunteer service, we asked respon- dents to designate how many hours per month they volunteered, and multi- Downloaded from nvs.sagepub.com by guest on February 27, 2015
Table 1. Modules—America Gives Demographic Tragedy VERY Questions Questions SHORT PSID AREA METHOD METHOD/AREA Sample size: total 4,200 1,304 900 800 900 800 800 Pre-9/11 2,896 — 800 512 800 576 208 Post-9/11 1,304 1,304 100 288 100 224 592 No. of questions: total 16 6 6 46 121 170 458 Giving — — 4 38 42 47 454 Volunteering — — 2 4 79 123 4 Types of prompts: Giving — Yes/no, Yes/no, Prompt by sub- Prompt by sub- Prompt by method Prompt by method amount amount sector of con- sector of con- of contact; 6 of contact, then (formal tribution tribution; 3 prompts for in- by subsector; 7 only) (formal only) prompts for formal giving prompts for in- informal giv- formal giving ing, 1 for polit- ical donations Volunteering — Yes/no, Yes/no, Yes/no, then by Prompt formal Prompt formal and Yes/no, then by 1 amount amount 1 subsector volunteering by informal volun- subsector (formal (formal subsector, 1 teering by me- (formal only) only) only) general prompt thod of contact Downloaded from nvs.sagepub.com by guest on February 27, 2015 for informal volunteering Inducements: None: total 3,202 — 805 698 805 694 200 Calling card: total 998 — 95 102 95 106 600 Pre-9/11 501 — 95 56 95 47 208 Post-9/11 497 — 0 46 0 59 392 Note: PSID = Panel Study of Income Dynamics. 631
632 Rooney et al. plied that figure by 12 to annualize it. Because previous studies (Hall, 2001; Independent Sector, 1999) found that the reliability of reporting on volunteer- ing by others in the household is quite low, we inquired only about the volun- teering done by the individual respondents. In this article, our focus is on the measurement of formal giving by house- holds and volunteering by individuals. Formal giving is the contributions of money and/or goods to organizations officially designated as charitable enti- ties (i.e., gifts that would qualify as a tax-deductible charitable contribution). Formal volunteering is the contribution of time to such charitable entities. Three of our modules (AREA, METHOD, and METHOD/AREA) include questions on informal giving—that is, contributions that are not collected through a particular organization or group. In addition, the AREA and METHOD modules query informal volunteering (volunteering on one’s own rather than through an organization). We will not emphasize the results of informal giving and volunteering in this article except to raise the method- ological question of whether measurement of giving and volunteering should focus only on formal gifts or also include informal donations of time, money, and goods. We also measure differences across demographic groups: age, income, race, educational attainment, gender, number of children, and num- ber of children in college. In analyzing the findings, we first used one-way ANOVAs in order to test whether the samples receiving each module were comparable demographi- cally and in their reported giving and volunteering. We then carried out sev- eral multivariate analyses (as suggested by O’Neill, 2001) to discern whether differences among the modules in mean levels of giving and volunteering can be explained simply by variations in sample characteristics or are due to the characteristics of the different modules. To do so, we explain donations of time and money in regression frameworks by including a set of dummy variables for the five modules, along with the demographic variables (listed above). If there are pure module effects, they will show up as significant coefficients for the module dummy variables. Unfortunately, the error terms in these regressions do not obey the classical assumptions that justify the exclusive use of OLS regressions. Donations can- not be negative, so the error term has a truncated distribution. In addition, giv- ing and volunteering data appear to have a nonnormal (heteroskedastic) error structure (e.g., Bradley, Holden, & McClelland, 1999; Rooney et al., 2001; Steinberg et al., 2002). Under these circumstances, OLS is biased and inconsis- tent. Tobit regression models, on the other hand, do not generate negative pre- dicted donations. Unfortunately, Tobits are not robust to nonnormal (heteroskedastic) errors. An additional problem with Tobit models is that they enforce a proportionality between a variable’s effect on the probability of giv- ing and the size of the donation for those who give. Another approach, a two- stage Heckman model, solves this latter problem but is not robust to non- normality. Because there is no commonly accepted ideal remedy for all of these problems, we conducted four different approaches (Tobit, Heckman Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 633 two-stage, OLS on the full sample, and OLS on positive donors only) in the hope that a consistent picture would emerge. We also ran probits to assess the effects of the various survey modules on the probability of donating or volun- teering at all. To simplify the presentation here, we focus our discussion on the Tobits and probits. A complete set of results, including the OLS and Heckman two-stage models, is available from the authors upon request. Our regression models use levels rather than logarithms for the continuous variables, for several reasons. First, it significantly simplifies the interpreta- tion of the results, which is helpful for a broad interdisciplinary readership. Second, it does not make any difference to the fit. Third, given the nature of the type of analysis, the large number of zeros (i.e., nondonors and/or non- volunteers) seems to lend itself to using level data with Tobit analyses. Finally, many of our main variables of interest are dichotomous in nature, so a log-log model would not have any impact on these results. The data set used in this article includes five modules, each with samples of 800 or 900 respondents. Each module includes questions about giving and volunteering. Hence, we have 4,200 respondents from the five modules that address both the time and treasure components of personal philanthropy. RESULTS Table 2 presents a comparison of demographic characteristics of each sam- ple. We discuss briefly only those sample characteristics that are statistically significantly different from the combined or total sample. Compared to the total sample means and proportions, the SHORT module contained signifi- cantly more Blacks (10.4% vs. 8.1% overall). The PSID had more couples (64.3% vs. 61.5% overall). The AREA module had slightly fewer households with incomes in excess of $120,000 (5.2% vs. 5.8%). The METHOD module had fewer couples (58.5% vs. 61.5%), a larger percentage with incomes of $40,000 or less (47.9% vs. 42.4% overall), a smaller percentage of high incomes (5.1% vs. 5.8%), and a concomitant smaller percentage who itemized their deduc- tions (45.9% vs. 49.8%). The METHOD/AREA module did not differ from the overall sample in any significant manner. In general, there are relatively few significant differences between the samples and these differences are rela- tively small, especially given the large sample sizes. Furthermore, these differ- ences can be controlled for statistically in our regressions. An interesting difficulty that we ran into was that there were small but sta- tistically significant differences across modules in the proportion of respon- dents reporting their income (χ2 = 9.504, p = .05). People were least likely to report their income in the AREA module (78.4%) and most likely to report income in the AREA/METHOD module (83.9%) (see Table 3). This is consis- tent with results from the earlier study done in Indiana (Rooney et al., 2001). Regressions with a variable for “income missing” were run to see if this affected our main results. This variable was small and insignificant and did Downloaded from nvs.sagepub.com by guest on February 27, 2015
634 Rooney et al. Table 2. Demographics of Total Sample by Module Total VERY METHOD/ Sample SHORT PSID AREA METHOD AREA Sample size 4,200 900 800 900 800 800 Female (%) 59.3 57.6 60 61 58.8 59.3 Couples (%) 61.5 60.9 64.3* 61.4 58.5* 62.3 White (%) 81.1 79.4 82.3 81 81.4 81.4 Black (%) 8.1 10.4* 7.1 7.6 7.5 7.5 Hispanic (%) 4.7 4.3 4.4 5.3 5.2 4.1 Asian (%) 2.1 2 2 2.4 1.8 2 Other minority (%) 4.1 3.8 4.1 3.7 4.2 5 Age: Mean 45.3 45.23 45.34 45.43 45.47 44.78 Median 44 44 44 44 44 44 Minimum 18 18 18 18 18 18 Maximum 93 91 87 92 90 93 Education (%): ≤ High school diploma 30.6 32.1 31.7 29 31.2 28.9 Some college 36.8 34.9 37 38.5 36.9 36.8 Bachelor’s degree 18.2 18.7 17.4 16.4 18 20.4 Graduate/professional school 14.4 14.3 13.8 16.1 13.9 13.9 Joint tests ** Income (%): $0-$40,000 42.4 39.6 39.2 42.6 47.9** 42.9 $40,000-$80,000 37.3 40.7 40.1 35.3 34.1 36.2 $80,000+ 20.3 19.7 20.7 22.1 18 20.9 Total reporting income 81.2** 82.4 80.4 78.4 81.1** 83.9 Joint tests % with income > $120,000 5.8 6.1 6.3 5.2** 5.1** 6.5 % who itemized deductions 49.8 52.3 50.1 52.5 45.9** 47.4 % of itemizers with donations 78.7 78.3 79.8 77.5 80.7 77.7 % with itemized gifts 34.3 36.1 35.1 35.7 31.9 32.3 Note: PSID = Panel Study of Income Dynamics. *p < .10. **p < .05. not have any material impact on the variables of interest, so it is not reported in detail here (copies of these results are available from the authors). In this next subsection, we will present the results for giving; we will address volunteering in a separate subsection. We start each of these two sub- sections by delineating the major differences and similarities between the var- ious sampling frames or modules. Then we present regression analyses, which allow us to test for differences in the methodologies while controlling for other factors such as income and educational attainment. BIVARIATE RESULTS ON GIVING In Table 3, we compare the simple means of formal giving for each of the modules, and we see some interesting similarities and differences. The mean Downloaded from nvs.sagepub.com by guest on February 27, 2015
Table 3. ANOVA Tests on Total Formal Giving (Excluding Outliers > $100,000) No. of Giving % a, b Independent Variable Mean N F (or t) p Questions Median Donors Independent Variable Mean n F p Module Income VERY SHORT 1,044.50 871 10.514 .001 4 200 78.0 $0-$20K 336.74 535 65.018 .000 PSID 1,407.23 800 0.346 .729 38 400 72.4 $20K-$40K 779.31 904 AREA 1,275.04 898 –0.804 .422 42 200 70.0 $40K-$60K 1,366.80 748 METHOD 1,141.78 798 –2.235 .026 47 286 92.0 $60K-$80K 1,632.06 522 METHOD/AREA 2,041.89 797 4.036 .000 454 610 94.7 $80K-$100K 2,347.57 295 Total (pooled) 1,373.45 4,164 10.514 .000 305 81.1 $100K-$120K 2,421.25 151 Gender $120K+ 5,029.91 241 Male 1,581.17 1,693 10.225 .001 Total 1,480.95 3,396 Female 1,231.14 2,471 Income (3 groups) Total 1,373.45 4,164 Low ($0-$40K) 614.77 1,439 139.957 .000 Marital (2 groups) Medium ($40K-$80K) 1,475.83 1,270 Singles 940.63 1,594 41.300 .000 High ($80K+) 3,304.73 687 Couples 1,651.53 2,546 Total 1,480.95 3,396 Total 1,377.82 4,140 Who decides donations Race (4 groups) Self 1,229.53 2,486 6.292 .000 Whites 1,476.12 3,320 4.503 .004 Spouse/partner 1,279.33 375 Downloaded from nvs.sagepub.com by guest on February 27, 2015 Blacks 997.69 331 Self and spouse 1,841.74 1,077 Hispanics 947.36 191 Other family member 1,020.91 132 Other minority 944.72 255 Other nonfamily 1,527.81 16 Total 1,379.74 4,097 Total 1,389.90 4,086 Race (2 groups) City size Whites 1,476.12 3,320 13.474 .000 Rural 1,188.48 1,493 5.159 .006 Other minority 967.93 777 Suburban 1,592.55 1,626 Total 1,379.74 4,097 Urban 1,404.50 932 Total 1,400.37 4,051 635 (continued)
636 Table 3. (continued) No. of Giving % a, b Independent Variable Mean N F (or t) p Questions Median Donors Independent Variable Mean n F p Education (2 groups) Region High school or less 636.32 1,270 85.210 .000 New England 1,396.29 206 0.710 .683 Postsecondary 1,708.10 2,874 Mid-Atlantic 1,188.15 523 Total 1,379.64 4,144 So. Atlantic 1,262.64 664 Education (4 groups) East No. Central 1,384.59 749 High school or less 636.32 1,270 61.869 .000 East So. Central 1,312.22 269 Some college 1,235.36 1,524 West No. Central 1,317.96 403 Bachelor’s 1,746.98 753 West So. Central 1,554.57 407 Graduate/ 2,865.86 597 Mountain 1,673.82 265 professional Pacific 1,436.84 678 school Total 1,373.45 4,164 Total 1,379.64 4,144 Flag 9/11 Religious attendance Before 9/11 1,268.91 2,866 8.343 .004 Not at all 741.86 847 47.399 .000 After 9/11 1,604.29 1,298 A few times/year 705.73 854 Total 1,373.45 4,164 1-2 times/month 1,003.37 601 Downloaded from nvs.sagepub.com by guest on February 27, 2015 Every week 1,906.70 1,420 > once a week 2,988.09 387 Total 1,386.71 4,109 Note: PSID = Panel Study of Income Dynamics. a. One-way ANOVAs (F tests) were used to test the hypothesis that several means were equal. b. t tests were used to test the hypothesis that individual model means were different from the total (pooled) mean.
Methodology Is Destiny 637 of the METHOD module ($1,142) and the METHOD/AREA mean of $2,042 were significantly higher than the total sample mean. The SHORT mean of $1,045 was significantly lower than that of the total sample. In addition, there is a very high significant correlation between the number of questions asked about giving and the mean amount reported per module (r2 = .952, p = .013). These results are prima facie evidence that the number of prompts and the length of the survey do matter in collecting data about individual philan- thropy. However, as we discussed above, there were some statistically signifi- cant differences in incomes across the various subsamples when compared to the overall sample. Hence, we will need to further refine this analysis with multivariate analyses. Table 3 also shows the median amounts donated and the percentage who reported making any donations for each module, along with mean amounts for various demographic groups. There is a high correlation between the num- ber of questions asked about giving and the median reported donation per module (r2 = .893, p = .041). The METHOD/AREA median of $610 was the highest, and this is the longest module. The correlation between number of questions and percentage reporting donations is also positive but is not statis- tically significant (r2 = .665, p = .221). Nevertheless, the longest module still yielded the highest rate of giving among respondents: 95% made donations in the METHOD/AREA module. These results are consistent with the work of Wilhelm (2002), who found that higher percentages of people reported dona- tions with methods surveys (e.g., method of fund-raising contact), but also that methods surveys primarily helped people recall small donations. Based on these important indicators of giving (mean and median dollars given and the percentage who gave anything), it does seem that a greater number of detailed prompts do stimulate greater recall. Of course, there is a danger that respondents report gifts that they did not actually give in an effort to conform to a perceived set of positive social expectations and/or to please or impress the interviewer. Although this is a problem in any type of survey research, it may be exacerbated by repeated questions about giving, which may convey the message to the respondent that it is expected or “normal” to give. Similarly, respondents may be embarrassed or bored if they repeatedly report no giving in surveys that ask about giving by many different areas and/or many different methods of contacts. These concerns are not readily resolved, but they do indicate the need to verify survey results through inde- pendent data sources whenever feasible. However, Havens and Schervish’s (2001) diary study, which used weekly prompting, found that 100% of their sample made donations to charities dur- ing the course of a year. In addition, they tested for the effects of repeated prompting by conducting the surveys during a 13th month. They found in surveying respondents during two consecutive Januaries that reported giving was not higher in the second January (the 13th month) than in the first January, which supports the hypothesis that extensive and repeated questioning does not inflate the responses of participants. They found that even after 4 or more Downloaded from nvs.sagepub.com by guest on February 27, 2015
638 Rooney et al. months of weekly questioning, respondents were remembering and recount- ing charitable giving they had been carrying out all along but had forgotten to mention or thought was not charitable giving. They conclude that their close look at charitable giving suggests, although does not prove, that having fewer prompts is more likely to lead to an understatement rather than an exagger- ation of charitable giving. It is worth noting that the use of incentives did not make a significant differ- ence in the reported amount of total formal giving (F[1, 4169] = 0.012, p = .914). This is consistent with the previous results in Indiana (Rooney et al., 2001), which found that inducement amounts, although positively related to response rate, did not make a significant difference in the reported amount of donations to charitable organizations. MULTIVARIATE RESULTS ON GIVING Now let us turn to the multivariate results in which we used OLS, Tobit, Heckman two-stage, and probit regression models to examine the marginal effects of the independent variables and to test the effects of differences in the modules more formally. Table 4 presents the main results of our Tobit and probit regression analyses (summarized in Table 5). These models assume that the various survey modules do not alter the coefficients of the other variables but that they shift the regression line up or down depending on the impact a survey module has on reported giving. Rooney, Mesch, Chin, and Steinberg (2003) found significant interaction effects between both race and gender and the various survey methodologies, so this must be treated as a simplifying assumption.1 Looking at the demographic variables in the regression models (i.e., age, race, gender, education, income, number of children, and number of children in college), we find nothing surprising or out of line with other studies. For example, the Tobit analysis indicates that respondents with some college edu- cation (or more) report household donations of $611 more than respondents with a high school education or less (the excluded category). Middle-income households ($40,000-$80,000) gave $699 more, and high-income households ($80,000+) gave $1,781 more than low-income households, holding every- thing else constant. The OLS and Heckman two-stage models produced simi- lar results, so a consistent picture emerges across all models: Giving increases significantly with increased income and education. Unlike the Indiana Gives study, America Gives found that race and age were significant in explaining differences in giving across the various mod- ules. These differences may be due to differences in sample sizes (885 vs. 4,200), as the coefficients in both studies are of similar magnitudes. The mar- ginal effect of age is small and is weakly significant (at best), so we think it can be safely ignored. The differences by race and gender require more attention than can be given in this article and are the focus of another paper (Rooney et al., 2003). America Gives also asked about the number of children in the Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 639 Table 4. Regressions on Giving Tobit Probability Probit Probability Variable Marginal Impact Value Marginal Impact Value Constant –1,374.31220 .000*** –.489047 .034** PSID 164.96357 .213 –.182103 .026** AREA 127.04729 .315 –.094559 .224 METHOD 377.11714 .003*** .787051 .000*** METHOD/AREA 954.33797 .000*** .954134 .000*** Age –15.44716 .299 .022430 .023** 2 Age 0.40164 .008*** –.00016 .112 Male 39.99896 .631 .225343 .000*** White 288.40536 .006*** .223871 .001*** Some college 610.95103 .000*** .430579 .000*** Medium income ($40K-$80K) 698.85896 .000*** .417821 .000*** High income ($80K+) 1,781.34590 .000*** .661742 .000*** No. children 127.97065 .000*** .045082 .084* No. children in college 352.04912 .000*** .016450 .811 Post 9/11 interview 72.17975 .472 .185087 .015** Sigma 2281.232107 .000*** n 3,355 3,355 2 Adjusted R .138 Log likelihood –27,698.2 –1,230.34 Note: PSID = Panel Study of Income Dynamics. Probit marginal impacts estimate the marginal probability of donating due to changes in each variable. Tobit marginal impacts estimate the change in donation amount due to changes in each variable. All coefficients for categorical vari- ables are relative to the values of the excluded category for that variable (females, minorities, high school or less, income $0-40K, pre-9/11 interview, VERY SHORT module). Statistical significance is determined for the coefficients on the latent index for donations, with respect to the latent indi- cator variable. Table with standard errors or t scores is available on request from the authors. *p < .10. **p < .05. ***p < .01. Table 5. Summary for Giving Variable Expected Sign Tobit Probit PSID + + –** AREA + + – METHOD + +*** +*** METHOD/AREA + +*** +*** Age N/A – +** 2 Age N/A +*** – Male + + –*** White + +*** +*** Some college + +*** +*** Medium income ($40K-$80K) + +*** +*** High income ($80K+) + +*** +*** No. of children – +*** +* No. of children in college – +*** + Post-9/11 interview 0 + +** Note: PSID = Panel Study of Income Dynamics. N/A = not applicable; 0 = no effect or unknown effect. *p < .10. **p < .05. ***p < .01. Downloaded from nvs.sagepub.com by guest on February 27, 2015
640 Rooney et al. family and the number of children in college (we assumed that these would both reduce the capacity for giving, as they would increase the internal famil- ial needs). Perhaps surprisingly, we found both the number of children and the number of children in college to be positive and significant explanatory variables. Of more interest to this methodological study is the analysis of the dummy variables for different modules. A rank ordering of the module coefficients in each regression model presents an interesting picture. In the Tobit model, the coefficient for the other modules is positive relative to the SHORT module (which was the excluded category). The AREA module is insignificantly dif- ferent from the SHORT module, but the METHOD module attains signifi- cance and predicts $377 more in donations relative to the SHORT module, holding everything else constant. The coefficients on the PSID module suggest that these households donated approximately $165 more, on average, than those answering the SHORT module, holding everything else constant, but this difference was not significant. Of even more interest is the fact that the METHOD/AREA module, which is the longest one, leads to reported giving of $954 more in the Tobit relative to the SHORT module (again holding every- thing else constant). (The results from the OLS and Heckman two-stage mod- els were generally consistent with the Tobit results.) This is a crucially impor- tant result, because it is almost identical to the main result of the Indiana Gives survey, which had a much smaller sample. Because the results from two very different samples at two different times were nearly identical, this is a strong indicator of the validity of the methodological result. These results suggest that the length and the number of prompts in the surveys do matter in terms of gathering data on dollars donated by households. To examine whether various modules have an impact on the probability of whether an individual or household donated at all, we look at the probit re- sults. These results indicate that holding everything else constant, the coeffi- cients for all modules behave the way we might expect: Longer, more detailed prompts also increase the probability of recalling having made any donations in the prior year. Controlling for all of the relevant demographic characteris- tics, the probability of reporting any donations is highest with the METHOD/ AREA module (95% more likely than those replying to the SHORT module), which had the most prompts for formal giving. Similarly, respondents to the METHOD module, which had the second most giving prompts, were 79% more likely to report making any donations than those replying to the SHORT module. The only surprises in the probit analysis are that the AREA module is not significantly different from the SHORT module and that respondents to the PSID module were 18% less likely to recall making any donations than were those with the SHORT—despite the fact that the PSID had many more prompts than does the SHORT module (38 vs. 4 prompts). Taken together, the results from the probit and Tobit once again point to the conclusion that sur- vey length and number of detailed prompts do matter in estimating house- hold giving. Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 641 BIVARIATE RESULTS ON VOLUNTEERING In Table 6, we compare the means, medians, and percentages of formal vol- unteering for each of the modules and the means for various demographic groups. (Note that this table represents the entire sample, excluding outliers of more than 2,000 hours per year [i.e., full-time volunteers]. We dropped the outliers because they were not evenly distributed across the modules. There- fore, a random event might skew the results of our testing of prompting and recall.) The means for the SHORT, PSID, and METHOD/AREA modules were significantly lower, and the mean for the METHOD module was significantly higher than the total sample mean. The means for the three shortest volunteer- ing modules were clustered between 55 and 67 hours per year (PSID = 65 vol- unteer hours per year; SHORT = 55 hours, and the METHOD/AREA = 67 hours; all three had median values of 0). The fact that the PSID and the METHOD/AREA modules used the exact same questions for volunteering and had almost identical mean values reinforces the notion that methodology matters. This suggests that the results are not due to sample sizes, or to the experience of callers, but that identical questions themselves produced very similar results across two large samples. It is worth noting that the highest mean value (164) comes from the METHOD module, which includes the highest number of questions measur- ing volunteering. The module with the second most prompts for volunteering was the AREA module, which had the second highest mean value (98 hours per year). There is a very high and significant correlation between the number of questions asked about volunteering and the mean reported volunteer hours per module (r2 = .96, p = .01). In addition, the percentage of respondents report- ing that they do any volunteer work is much higher for the longest module than the shorter ones. Collectively, these results offer prima facie substantia- tion to our thesis that the number of prompts related to volunteering does matter in collecting data about individual philanthropy. As with reported giving, the use of incentives did not make a significant difference in the reported amount of total formal volunteering. This is consis- tent with the previous results in Indiana (Steinberg et al., 2002), which found that inducement amounts did not make a significant difference in the amount of formal volunteering. MULTIVARIATE RESULTS ON VOLUNTEERING Next, we examine the multivariate results in which we used Tobit and probit regression models to test the marginal effects of the independent vari- ables and to assess the impact of differences in the modules more formally.2 (The probits determine the marginal probability of an individual volunteering at all given various characteristics.) Table 7 presents the main results of our regression analyses, summarized in Table 8. (Results from OLS and Heckman two-stage models are also available from the authors upon request.) Begin- Downloaded from nvs.sagepub.com by guest on February 27, 2015
642 Table 6. ANOVA Tests on Formal Volunteering (Excluding Outliers > 2,000 Hours) No. of Volunteerng % a, b Independent Variable Mean N F (or t) p Questions Median Volunteers Independent Variable Mean n F p Module Income VERY SHORT 54.54 893 –6.588 .000 2 0 42.4 $0-$20K 56.70 553 3.714 .001 PSID 64.95 800 –4.325 .000 4 0 45.6 $20K-$40K 96.00 902 AREA 97.95 892 1.249 .212 79 0 39.4 $40K-$60K 84.18 741 METHOD 163.85 780 6.014 .000 123 6 54.5 $60K-$80K 89.92 516 METHOD/AREA 66.63 800 –3.890 .000 4 0 41.6 $80K-$100K 118.01 293 Total (pooled) 88.63 4,165 33.058 .000 0 44.6 $100K-$120K 104.68 151 Gender $120K+ 118.98 245 Male 82.89 1,697 1.902 .168 Total 90.25 3,381 Female 92.58 2,468 Income (3 groups) Total 88.63 4,165 Low ($0-$40K) 81.40 1,435 5.643 .004 Marital (2 groups) Medium ($40K-$80K) 86.54 1,257 Singles 82.45 1,595 2.065 .151 High ($80K+) 115.43 689 Couples 92.69 2,544 Total 90.25 3,381 Total 88.74 4,139 Who decides Race (4 groups) Self N/A N/A N/A N/A Downloaded from nvs.sagepub.com by guest on February 27, 2015 Whites 90.67 3,318 0.385 .764 Spouse/partner N/A N/A Blacks 81.75 330 Self and spouse N/A N/A Hispanics 88.62 192 Other family member N/A N/A Other minority 77.90 254 Other nonfamily N/A N/A Total 89.06 4,094 Total N/A N/A Race (2 groups) City size Whites 90.67 3,318 0.902 .342 Rural 82.05 1,496 2.222 .109 Other minority 82.19 776 Suburban 89.71 1,626 Total 89.06 4,094 Urban 101.74 930 Total 89.64 4,052
Education (2 groups) Region High school or less 59.04 1,272 33.113 .000 New England 81.13 205 1.490 .155 Postsecondary 102.13 2,872 Mid-Atlantic 85.61 518 Total 88.90 4,144 So. Atlantic 75.33 668 Education (4 groups) East No. Central 80.59 748 High school or less 59.04 1,272 21.066 .000 East So. Central 79.50 271 Some college 81.41 1,524 West No. Central 97.15 402 Bachelor’s 118.99 751 Mountain 87.15 264 Graduate/ 133.79 597 West So. Central 93.82 409 professional Pacific 111.15 680 school Total 88.63 4,165 Total 88.90 4,144 Flag 9/11 Religious attendance Before 9/11 86.27 2,871 15.819 .000 Not at all 48.79 844 21.830 .000 After 9/11 93.87 1,294 A few times/year 68.91 854 Total 88.63 4,165 1-2 times/month 74.79 603 Every week 111.76 1,419 > once a week 153.84 389 Total 88.48 4,109 Note: PSID = Panel Study of Income Dynamics. N/A = not applicable. a. One-way ANOVAs (F tests) were used to test the hypothesis that several means were equal. b. t tests were used to test the hypothesis that individual model means were different from the total (pooled) mean. Downloaded from nvs.sagepub.com by guest on February 27, 2015 643
644 Rooney et al. Table 7. Regressions on Volunteering Tobit Probability Probit Probability Variable Marginal Impact Value Marginal Impact Value PSID 19.05337 .318 .072925 .305 AREA 43.17837 .019** –.066648 .331 METHOD 148.56956 .000*** .302158 .000*** METHOD/AREA –26.23220 .219 –.148902 .057* Age 3.07166 .164 .00771 .347 2 Age –0.01714 .449 –.00004 .590 Male –26.65009 .028** –.150414 .001*** Married –5.479759 .686 .00930 .855 White 27.2916 .076* .142134 .013** Some college 101.7348 .000*** .433981 .000*** Medium income ($40K-$80K) 23.39421 .112 .169616 .002*** High income ($80K+) 62.79343 .000*** .309819 .000*** No. of children 26.03154 .000*** .105829 .000*** No. of children in college 4.95296 .716 .030234 .588 Post-9/11 interview 53.02648 .000*** .198842 .000*** Sigma 295.6933 .000*** n 3,333 3,333 2 Adjusted R .068 Log likelihood –1,2042.2 Note: PSID = Panel Study of Income Dynamics. Probit marginal impacts estimate the marginal probability of donating due to changes in each variable. Tobit marginal impacts estimate the change in donation amount due to changes in each variable. All coefficients for categorical vari- ables are relative to the values of the excluded category for that variable (females, minorities, high school or less, income $0-40K, pre-9/11 interview, VERY SHORT module). Statistical significance is determined for the coefficients on the latent index for donations, with respect to the latent indi- cator variable. Table with standard errors or t scores is available on request from the authors. *p < .10. **p < .05. ***p < .01. Table 8. Summary for Volunteering Variable Expected Sign Tobit Probit PSID 0 + + AREA + +** – METHOD + +*** +*** METHOD/AREA 0 – –* Age N/A + + 2 Age N/A – – Male – –** –*** Married 0 – + White 0 +* +** Some college + +*** +*** Medium income ($40K-$80K) + + +*** High income ($80K+) + +*** +*** No. children + +*** +*** No. children in college – + + Post-9/11 interview 0 +*** +*** Note: PSID = Panel Study of Income Dynamics; N/A= not applicable; 0 = no effect or unknown effect. *p < .10. **p < .05. ***p < .01. Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 645 ning with the probits, we find that age and marital status do not seem to mat- ter; men are 15% less likely to volunteer, Whites are 14% more likely to volun- teer than non-Whites, those with some college education (or more) are 43% more likely to volunteer than those with a high school education or less, and there is a large increase in the probability of volunteering at all as one moves up the income ladder. As would be expected, because of the parental involvement with activities associated with school-aged children (e.g., coaching Little League, assisting in the classrooms, etc.), having children is associated with an 11% increase in the probability of volunteering, but having children in college has no significant effect. Having children in college may necessitate working more hours at a current job or taking a second job—either of which would reduce time avail- able for volunteering. The gender and race issues will be addressed in other work, as will the post-9/11 effect, but we should mention that there is a 20% increase in the probability of having volunteered in the past year among those interviewed in the aftermath of the attack on America. Looking at the differences in the number of volunteering hours reported using Tobit, we find that males volunteer almost 27 fewer hours than women, Whites volunteer just over 27 hours more than non-Whites (but this result only approaches significance with p = .076), some college is associated with an increase of almost 102 hours relative to those with a high school degree (or less), high-income earners volunteer almost 63 hours more than low-income earners, and those with more children volunteer 26 hours more. Those inter- viewed following the attack on America reported volunteering 53 hours more than those before the attack, holding everything else constant. With respect to the impact of the differences in survey methodologies on hours volunteered, the Tobit results are very consistent: When compared to the SHORT module (the omitted variable), the METHOD module, which has the most prompts about formal volunteering, is associated with 149 (Tobit) more volunteer hours in the previous year. The AREA module, which has the second-highest number of prompts for formal volunteering, also has a large and positive effect on the number of hours volunteered (43 using Tobit). Not surprisingly, the PSID and the METHOD/AREA, which have only four prompts compared to the two in the SHORT module, do not yield statis- tically significant differences in the number of hours volunteered. (OLS and Heckman two-stage results were generally consistent with the Tobits.) THE RELATIONSHIP BETWEEN VOLUNTEERING AND GIVING In addition to looking at giving or volunteering alone, we also examined the relationship between giving and volunteering. Previous research has found that volunteers are more likely than nonvolunteers to make charitable contributions (Hall et al., 2001; Independent Sector, 1999; Jalandoni & Hume, 2001; Steinberg et al., 2002). Likewise in this study, chi-square tests for differ- ences in the proportion between donors and nondonors among volunteers Downloaded from nvs.sagepub.com by guest on February 27, 2015
646 Rooney et al. Table 9. Relationship Between Giving and Volunteering Across Modules Total VERY METHOD/ Sample SHORT PSID AREA METHOD AREA n 4,200 900 800 900 800 800 No. of questions 21 62 138 186 473 Volunteers (%) No donations 11.1 10.1 17.8 21.2 4.4 2.7 Donations 88.9 89.9 82.2 78.8 95.6 97.3 Nonvolunteers (%) No donations 25.2 30.9 35.9 35.6 12.4 7.1 Donations 74.8 69.1 64.1 64.4 87.6 92.9 2 χ 133.981*** 52.724*** 32.355*** 21.025*** 17.271*** 7.441** Pearson correlations No. of questions and % of volunteers who donated .615 No. of questions and % of non- volunteers who donated .824* *p < .10. **p < .05. ***p < .01. and nonvolunteers found that volunteers were much more likely to be donors than nonvolunteers in the total sample, as well as in each of the modules (see Table 9). In fact, across all of the modules, nonvolunteers were between 2 and 3 times more likely to be nondonors than were those who had volunteered. CONCLUSIONS Our results have shown that whether using simple means or multivariate analyses, the longer and more detailed the module, the more likely a house- hold was to report making a charitable contribution of time or treasure and the higher the average level of its reported giving and volunteering. These results persist even after controlling for differences in age, educational attainment, income, household status, race, number of children, number of children in col- lege, and gender. We view these results, coupled with similar findings from our Indiana Gives comparison of survey methodology (Rooney et al., 2001) as further confirmation of Schervish and Havens’s (1998) methodological proposition that the more carefully a survey samples the full range of households by income, interviews the knowledgeable deci- sion makers in a household, furthers respondent recall, contracts trained Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 647 interviewers who ask complex questions patiently and accurately, and otherwise use rigorous field practices, the greater the amount of re- ported charitable giving. (p. 241) As we look to the future, one additional research question is to investigate what kinds of giving people tend to remember and forget without being prompted and whether the short or medium-length surveys capture most of the larger donations, so that users of those methodologies could be more con- fident about what they were likely to capture and to miss. There is also need for further research on other important methodological questions. For example, within households, what are the key factors that de- termine giving within a joint household: Are decisions usually made jointly? If not, who decides? Does the gender of the decision maker affect where the money goes once there is a decision to donate? Given that we cannot match survey samples with IRS records, is there a way that we can verify or validate our survey research? What, if any, are the true effects of race and gender when a complex array of other background characteristics, motivational determi- nants, and decision-making dynamics are taken into account? This discussion on survey methods raises a more general methodological question that scholars need to test in the field: How is the measurement error affected when surveys ask about individual versus household philanthropy? The key considerations are greater knowledge and recall of one’s own philan- thropy, but many respondents may recall how much they listed as itemized deductions, which would typically be a joint total. (Even married persons fil- ing separately would be likely to pool their contributions and list them under one person, as this would minimize their combined tax burden.) The innovations in survey methodology we have introduced for the study of giving and volunteering suggest an array of additional research issues to be pursued in the future. Among the issues that methodologists have addressed in regard to other topics, but have not been taken up in regard to giving and volunteering, include the impact of extended prompts, the kind of interview method in which the cues are embedded, and the potential for extended prompting to induce socially desirable responses. Much other research con- curs with our findings that using prompts (or cues) and cueing mechanisms to aid information retrieval does, in fact, improve respondent recall and reduces response bias and measurement error (Belli, 1998; DeMaio & Rothgeb, 1996; Moore, Stinson, & Welniak, 1999; Shum & Rips, 1999). This literature provides empirical support for the fact that improved cueing produces more valid re- sponses. At the same time, it suggests additional cueing techniques that merit testing in regard to giving and volunteering. Recently, Belli, Shay, and Stafford (2001) have demonstrated the potential advantages of event-history calendars with a flexible mode of interviewing that uses cues as part of a narrative form of inquiry rather than simply as addi- tional close-ended questions. Introducing and evaluating different kinds of Downloaded from nvs.sagepub.com by guest on February 27, 2015
648 Rooney et al. cues and the event-history approach to assist recall needs to be complemented by additional work on the long-recognized problem of social desirability (DeMaio, 1984). This is especially important for longer surveys such as ours in which the use of prompts requires most respondents to frequently, if not re- peatedly, answer in the negative about whether and how much they give and volunteer. Although there is some evidence that survey respondents tend to tell the “truth” when they know the answers, there is always the possibility of some memory error (Bradburn & Sudman, 1988). Furthermore, Wentland and Smith (1993) report that based on a meta-analysis of survey responses to ques- tions for which there are actual numbers available for comparisons to the sur- vey replies, the group averages tend to be fairly accurate. They report that although individual replies may underestimate or overestimate the actual results, these differences tend to net out in the group averages. They also sug- gest that inaccuracies due to socially desirable responses may not be a prob- lem in this type of research. Ideally, we would be able to match survey responses with IRS records for those who itemize their charitable deductions and compare. However, even this would not prove the accuracy of the survey replies because for different reasons, some tax filers may overstate their deductions and others may under- state them. As an experiment, we tried this approach with a small sample in a pretest for an upcoming study. Unfortunately, we found very few who both had their IRS 1040 Schedule A’s available and were willing to retrieve them and share the results with us. Perhaps others might be able to develop a research design that might test this method with greater success. Finally, all the issues and directions for research we have addressed in this article need to be taken up as well in regard to the panoply of informal giv- ing and volunteering. It is clear from closer examination of remittances by immigrants and the daily practices of all groups, regardless of ethnicity or income, there is a dramatic amount of financial and in-kind assistance of indi- viduals both in and outside of the immediate family. Equally substantial is the generous network of unpaid assistance that people carry out amidst their daily round to meet the needs of others. The same methodological princi- ple that calls for careful and detailed investigation of formal giving and vol- unteering applies to these realms of informal caring behavior equally, if not more so. Our results, in conjunction with our findings from the smaller Indiana Gives study, strongly support the principle that when it comes to estimating charitable giving and/or volunteering, methodology is destiny. The more individuals are prompted in detail, the more likely they are to recall having given any gifts or to have volunteered at all, and the more they report their total giving and their hours of volunteer time to be. Although there is no incontrovertible evidence to establish this conclusion, the growing prepon- derance of evidence makes it thoroughly persuasive. Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 649 Appendix Examples of Module Prompts VERY SHORT Module: Giving (yes/no, amount): In the past 12 months, have you made any financial contributions to a charitable or nonprofit organization or group? (If yes): How much? Volunteering: (yes/no, amount): In the past 12 months, have you done any volunteering for an organization or group? (If yes): How many hours total? PSID Module: Giving (Prompt by subsector of contribution, formal only): During the year 2000 did you or anyone in your household make donations • For religious purposes or spiritual development; • To an organization that served a combination of purposes (United Way, United Jewish Appeal, Catholic Charities, local community foundation); • To organizations that help people in need of food, shelter, or other basic necessities; • To health care or medical research organizations; • Toward educational purposes (not including tuition); • To other organizations: youth and family services; arts, culture, and ethnic aware- ness; improving neighborhoods or communities; environment; international aid or world peace? (If yes to any of the above): asked dollar amount first for each, provided categories if respondent did not know or refused. Volunteering (yes/no, then prompt by one subsector, formal only): During the year 2000, did you yourself do any volunteer work through organiza- tions that totaled 10 hours or more? (If yes): asked total hours first, provided categories if respondent did not know or refused. (If yes): How many hours of that volunteer time was spent helping people in need of food, shelter, or other basic necessities? Downloaded from nvs.sagepub.com by guest on February 27, 2015
650 Rooney et al. AREA Module: Giving (Prompt by subsector of contribution, informal and political donations): In which, if any, of the following fields have you and the members of your family or household contributed some money or other property in the past year? • Health • Education • Religious organizations • Human services • Etc. (If yes to any of the above): asked dollar amount for each. During the past 12 months, did you or the members of your family or household give money, food, or clothing to the homeless or street people, a needy neighbor, or an- other needy person? (If yes): How much? Did you provide any financial assistance to or for relatives (including children and parents) who do not live with you? (If yes): How much? How much, if anything, have you and the members of your family contributed to political organizations in the past year? Volunteering (Prompt formal volunteering by subsector, one prompt for informal): In which, if any, of the following areas have you done some volunteer work in the past 12 months? • Health • Education • Religious organizations • Human services • Etc. • Informal—alone—not for pay; Other (If yes to any of the above): asked number of hours in past month and past week. METHOD Module: Giving (Prompt by method of contact [formal], several prompts for informal): In the past 12 months, have you and your household made a donation to charitable and nonprofit organizations by Downloaded from nvs.sagepub.com by guest on February 27, 2015
Methodology Is Destiny 651 • Responding to a request through the mail; • Paying to attend a charity event; • Using payroll deductions; • Sponsoring someone in an event such as a walk-a-thon; • Etc.? (If yes to any of the above): asked dollar amount for each. In the past 12 months, did you give any money directly to the homeless or street people? (If yes): How much? Did you give any money to relatives, including children and parents, who do not live with you? (If yes): How much? Did you give any money to other individuals not already mentioned? (If yes): How much? Volunteering (Prompt formal and informal volunteering by method of contact): In the past year, did you do any of the following volunteer activities through a group or organization: • Canvassing, campaigning, or fundraising; • Serve on a board or committee; • Organize or supervise events; • Provide counseling or health care; • Etc.? (If yes to any of the above): asked about months and weeks volunteered, regularity, hours per week and month, extra hours. In the past 12 months, did you help anyone on your own, not through a particular organization, by • Cooking or cleaning • Shopping, driving • Teaching or coaching • Etc.? METHOD/AREA Module: Giving (Prompt by method of contact, then by subsector for formal giving; several prompts for informal giving): Downloaded from nvs.sagepub.com by guest on February 27, 2015
You can also read