Sample and design considerations in post-disaster mental health needs assessment tracking surveys
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
International Journal of Methods in Psychiatric Research
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Published online in Wiley InterScience
(www.interscience.wiley.com) DOI: 10.1002/mpr.269
Sample and design considerations in post-
disaster mental health needs assessment
tracking surveys
RONALD C. KESSLER,1 TERENCE M. KEANE,2,† ROBERT J. URSANO,3,† ALI MOKDAD,4
ALAN M. ZASLAVSKY1
1 Department of Health Care Policy, Harvard Medical School, Boston, MA, USA
2 National Center for PTSD, VA Boston Healthcare System, Boston, MA, USA
3 Department of Psychiatry and Center for the Study of Traumatic Stress, Uniformed Services University,
Bethesda, MD, USA
4 Institute for Health Metrics and Evaluation, University of Washington, Seattle, WA, USA
Abstract
Although needs assessment surveys are carried out after many large natural and man-made disasters, synthesis of findings
across these surveys and disaster situations about patterns and correlates of need is hampered by inconsistencies in study
designs and measures. Recognizing this problem, the US Substance Abuse and Mental Health Services Administration
(SAMHSA) assembled a task force in 2004 to develop a model study design and interview schedule for use in post-disaster
needs assessment surveys. The US National Institute of Mental Health subsequently approved a plan to establish a center
to implement post-disaster mental health needs assessment surveys in the future using an integrated series of measures
and designs of the sort proposed by the SAMHSA task force. A wide range of measurement, design, and analysis issues
will arise in developing this center. Given that the least widely discussed of these issues concerns study design, the current
report focuses on the most important sampling and design issues proposed for this center based on our experiences with
the SAMHSA task force, subsequent Katrina surveys, and earlier work in other disaster situations. Copyright © 2008
John Wiley & Sons, Ltd.
Key words: disaster, epidemiology, needs assessment survey, PTSD
Introduction (SAMHSA) in 2004 assembled a task force to develop
Although mental health needs assessment surveys are a model study design and interview schedule for use in
carried out after many large natural (Ironson et al., post-disaster mental health needs assessment surveys. It
1997; Kohn et al., 2005) and man-made (Gidron, 2002; was thought that such a protocol would both lead to
North et al., 2004) disasters, synthesis of findings about greater consistency than currently exists across such
patterns and correlates of post-disaster psychopathol- surveys and reduce the sometimes substantial delays
ogy is hampered by inconsistencies in study design and due to instrument development that occur in launch-
measures (Brewin et al., 2000; Galea et al., 2005; Norris, ing these surveys.
2005). Recognizing this problem, the US Substance The interview schedule developed by this task force
Abuse and Mental Health Services Administration was pre-tested among victims of the Florida hurricanes
of 2004. After revisions based on the results of cogni-
†
The contributions of Terence M. Keane and Robert J. Ursano tive interviews carried out with these pre-test respon-
were prepared as part of their official duties as US Government dents, a pilot survey of the revised interview schedule
employees. was carried out in late 2005 in three samples of people
Copyright © 2008 John Wiley & Sons, LtdPost-disaster needs assessment surveys S7
who were exposed to a natural or man-made disorder implementation of conventional telephone surveys and
in the previous two years (a train crash and resulting that impedes the travel of field interviewers to carry out
toxic chemical spill in a small town in South Carolina; face-to-face surveys. In the case of Hurricane Katrina,
a plant explosion in a small town in Illinois; and a series there was the additional complication that a massive
of tornados in several small towns in the Midwest). flood led to the evacuation and wide geographic disper-
Further instrument revisions were made based on sion of the population of New Orleans.
debriefing interviews with a sub-sample of the respon- The existence of a center for disaster surveys will
dents in this pilot survey, a clinical validation study of create opportunities to address these practical chal-
the post-traumatic stress disorder (PTSD) screening lenges in the US as well as to expand the conventional
questions in the pilot survey, and quantitative analyses role of needs assessment surveys by developing ongoing
of survey responses to confirm that the interview collaborations with government disaster-preparedness
schedule generated substantively plausible results. agencies and relief agencies. Several such opportunities
An expanded version of this revised instrument was for expansion exist. For example, the US federal gov-
then used in a series of mental health needs assessment ernment uses the mass media to disseminate informa-
tracking surveys among victims of Hurricane Katrina tion aimed at increasing knowledge and changing
in the US Gulf Coast (Kessler et al., 2006). These attitudes and behavior (KAB) of populations both
surveys posed a number of sampling and design chal- before disasters (i.e. disaster preparedness) and after
lenges related to the special circumstances of Hurricane disasters (i.e. disaster response). Needs assessment
Katrina that are discussed later. They also highlighted tracking surveys can be used to provide feedback to the
the fact that the subsequently much-discussed deficien- message development teams involved in these KAB
cies in federal disaster preparedness apply as much to social marketing public information campaigns (Flay
disaster needs assessment surveys as to other areas of et al., 1989). This kind of collaboration would require
disaster response. Based on this realization, the US coordination, as the survey team needs to be aware of
National Institute of Mental Health (NIMH) has the messages being disseminated by the message devel-
established a center that will implement post-disaster opment team in order to build relevant questions about
mental health needs assessment surveys in the future these messages into the needs assessment surveys.
using an integrated series of measures and designs. A A good example of such coordination is the current
wide range of measurement, design, and analysis issues collaboration between our Hurricane Katrina Com-
will arise in implementing these surveys. Given that munity Advisory Group (CAG; www.Hurricane
the least widely discussed of these issues concerns study Katrina.med.harvard.edu) and the American Red Cross
design, the current paper focuses on the most impor- (ARC) in tracking awareness and response to the new
tant sampling and design issues likely to be faced by ARC Access to Care (ATC) Program, a program
this center based on our experiences with the SAMHSA designed to help low-income victims of Hurricane
task force, subsequent Katrina surveys, and earlier work Katrina pay for emotional support services, such as
in other disaster situations. The issues considered are mental health treatment and substance abuse treat-
those that apply to surveys carried out in the US and, ment. The ongoing CAG tracking surveys are monitor-
by extension, in other developed countries. Many of ing awareness of the ATC Program, attitudes about the
the considerations discussed here would be rather program, and barriers to taking advantage of the
different in less developed countries. program. Analysis of the CAG data is providing infor-
mation to the ARC about population segments with
Challenges and opportunities low awareness of the ATC Program, media habits of
The major challenges in designing disaster-related these population segments that might be useful in
needs assessment surveys concern implementation. developing new program dissemination strategies, and
With regard to sampling, it is usually necessary to create information about barriers to using the program among
an appropriate sampling frame very quickly so that eligible community members who are aware of the
survey results can be used to make timely planning program but have not used it to pinpoint potentially
decisions. A complicating factor in many disaster situ- useful expansions of ARC outreach efforts.
ations even in developed countries is that infrastruc- This example is a rather obvious one, as public
ture damage creates logistical problems that hamper health marketing campaigns often use market tracking
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprS8 Kessler et al.
surveys to evaluate campaign success (e.g. Subar et al., to evaluate need for mental health treatment of these
1995). The only innovation is the use of mental health workers and their families during that time period
needs assessment surveys to take on the conventional would be from an administrative list sampling frame
role of market tracking surveys. Other opportunities with home contact information for all such workers.
to expand the conventional role of needs assessment Once the Environmental Protection Agency makes an
surveys are less obvious, though, although equally evaluation that the building is safe for workers to return,
important. A number of these are discussed later in the though, the affected workers (although not their fami-
section on design consideration. Before this, though, lies) would become highly clustered geographically (i.e.
we turn to the important matter of sampling. at their place of work), making it possible efficiently to
carry out mental health needs assessment surveys on
Sampling considerations site.
Another mixed case is the situation where a man-
General population sampling made disaster occurs at a place that involves both resi-
The difficulties associated with selecting a representa- dents of the area in which the disaster occurred and
tive sample of disaster survivors differ depending on people who were passing through the area at the time of
whether the disaster is or is not defined in terms of the disaster. A good example is the 2005 train crash at
geography. In the case of natural disasters (e.g. torna- a depot in the middle of the small town of Graniteville,
dos, hurricanes) or man-made disasters that have a South Carolina that released toxic chemicals into the
geographic epicenter (e.g. the Oklahoma City bombing), local environment, leading to injury, death, and toxic
it makes most sense to think in terms of area probabil- exposure among the passengers and crew of the train
ity household sampling as the main basis for sample and to risk of toxic exposure, evacuation, and commu-
selection. There are inevitable practical problems with nity disruption among residents of the community in
this form of sampling that can be exacerbated in situa- which the crash occurred (US Environmental Protec-
tions of mass evacuation. As described later; multiple- tion Agency, 2005). In a situation of this sort, the resi-
frame sampling (Skinner and Rao, 1996) can be used dents of the community would be geographically
to decrease coverage problems in situations of this sort. clustered while the surviving passengers and crew of
In the case of disasters that do not have a geographic the train would not be geographically clustered.
epicenter (e.g. a plane crash), in comparison, the use of We faced an especially complex situation with regard
list samples is a necessity unless the researchers have to sampling in assembling the Hurricane Katrina CAG.
the resources to engage in large-scale mass screening, A small proportion of the population, presumably rep-
using multiplicity sampling (Kalton and Anderson, resenting the most high-risk pre-hurricane residents of
1986) whenever possible to increase the efficiency of the areas most hard hit by the storm and resulting flood
the screening exercise. In any of these cases, frame in New Orleans, were living in evacuation centers
biases have to be taken into consideration. Land line (ECs) and later Federal Emergency Management
telephone frames, in particular, might under-represent Agency (FEMA)-sponsored hotel rooms, trailers, and
the most disadvantaged segments of the population even luxury liners. Many other pre-hurricane residents
(Brick et al., 2006), making it particularly useful to of the New Orleans Metropolitan Area were scattered
implement a multiple-frame sampling approach that throughout the country, largely living with relatives,
enriches the less restrictive frame for high-risk cases, but also in communities that had established ECs and
possibly by over-sampling Census blocks with low rates subsequently created community living situations in
of land line telephone penetration or high rates of which a certain number of needy families from New
poverty. Orleans were, in effect, adopted by the community.
Some studies will involve both geographically clus- The vast majority of pre-hurricane residents of the
tered and dispersed cases. For example, the workers in other areas in Alabama, Louisiana, and Mississippi that
a government building exposed to a terrorist attack were affected by the hurricane remained living either
with anthrax would be geographically dispersed during in their pre-hurricane households or in the surrounding
the initial time period when the building was evacuated community in which they lived before the hurricane as
and workers were sent home prior to a thorough evalu- they went about repairing the damage caused to their
ation of building contamination. The most feasible way homes and communities. Telephone lines were down
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprPost-disaster needs assessment surveys S9
in many parts of the affected areas for a considerably FEMA for assistance and also provided post-hurricane
longer time than is typical in US natural disasters. In contact information comparable to the ARC list infor-
addition, physical movement was made difficult by mation. As one would predict, considerable overlap
infrastructure damage and difficulty finding gasoline existed in the entries on these two lists, but the more
for cars. Conventional household enumeration was surprising finding was that a substantial number of
made difficult in some areas by the fact that many families applied only to one of the two. There were also
pre-hurricane homes no longer existed. a number of families that fraudulently applied on mul-
At the same time, we had several important resources tiple occasions and at different locations to the same
available to us that we used in building a multiple- agency. We corrected for these multiple counts in
frame sampling strategy that combined information sampling from these lists.
from a number of restricted frames to assemble the It is also noteworthy that a great many hurricane
sample of people who participated in the CAG. One evacuees registered with one or more of the “safe lists”
rather unexpected resource was the use of random digit set up on the internet by the television news channels,
dialing (RDD). It seems counterintuitive that RDD CNN and MSNBC, the ARC, and others. These lists
could be used to study Katrina survivors in light of the allowed people separated from their loved ones during
fact that the vast majority of the New Orleans popula- the hurricane or aftermath to let it be known that they
tion was forced to evacuate their homes after the storm were alive and to record their whereabouts in the hopes
and the fact that many people who lived in other areas of reconnecting with their loved ones. Google subse-
affected by the hurricane had non-working land lines quently integrated all the names recorded on all the
because of damage to telephone infrastructure. However, internet safe lists into a single consolidated list that
the main telephone provider in the hurricane area, Bell contained over 400 000 names. We made extensive use
South, forwarded phone calls made into the hurricane of this consolidated list in piloting the baseline CAG
area to new numbers (either land line numbers or cell interview. However, this pilot testing led to the discov-
phone numbers) outside the area that were registered ery that virtually all people on the safe lists were also
by the owners of the pre-hurricane numbers. As a result on the more inclusive ARC and FEMA lists of people
of this service, we were able to call an RDD sample of who applied for assistance. As a result, we did not use
phone numbers selected from 1+ telephone banks the safe lists in our final sample selection for the
working in New Orleans prior to the hurricane and to CAG.
connect with many displaced pre-hurricane New By the time the baseline CAG survey was fielded,
Orleans residents in temporary residences all across all the Katrina ECs had been closed and only a small
the country. number of evacuees were still housed in FEMA-
A second important resource was the availability of supported hotel rooms. This made it relatively easy to
extensive ARC and FEMA lists of people who regis- screen a representative sample of hotels selected from
tered for assistance. Of the over four million adult the Donnelly commercial sampling frame to find hotels
residents of the area defined by FEMA as affected by housing evacuees, to use information provided by hotel
Katrina (4 137 000 adult residents in the 2000 Census), managers to select a sample of rooms with probabilities
a majority applied for assistance to one or both of the proportional to size from these hotels, and to include
two major agencies that maintained comprehensive the respondents interviewed in this way as a supple-
applicant lists. We were in the fortunate position of mental sample. Not surprisingly, though, this exercise
having access to both of these lists. In order to reduce showed that virtually all hotel evacuees were included
overlap with the RDD frame, we restricted our use of with valid contact information on the FEMA relief list
these lists to cell phone exchanges and to land line that we were using as one of the main sample frames.
exchanges in areas outside of the RDD sampling area. As with respondents sampled from each of the other
Over 1.4 million families representing more than 2.3 frames, information was included about this overlap
million adults applied to the ARC for assistance and and used in making weighting adjustments in the
provided post-hurricane contact information that consolidated CAG sample.
included new residential addresses, telephone numbers The availability of these different frames allowed us
(often cell phones), and email addresses. An even larger to use relatively inexpensive telephone administration
number of families (roughly 2.4 million) applied to to reach the great majority of people who were living
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprS10 Kessler et al.
in the areas affected by Katrina before the hurricane. household informants would tell us about the where-
As noted earlier, we reduced overlap between the two abouts of such evacuees.
main frames by restricting our use of the ARC and We screened a nationally representative sample of
FEMA lists to cell phone exchanges and to land line 20 000 listed telephone numbers to investigate the
exchanges in areas outside of the RDD sampling area. validity of these assumptions, a random half using IVR
In addition, we collected data from every respondent in and the other half using live interviewers. We found a
the entire sample that allowed us to determine whether hit rate closer to one in 1000 in the households ran-
they had a non-zero probability of selection in each domized to be screened by live interviewers, with the
frame. For example, we asked respondents in the RDD number of evacuees in these households typically quite
sample if they applied to the ARC and to FEMA for large (4–7). This presumably reflects differential prefer-
assistance. This information made it possible for us to ences for relocation destinations of evacuees with and
use capture–recapture methods (Fisher et al., 1994) to without families. We found that the hit rate was much
estimate the size of each population segment defined by smaller in the households randomized to be screened
the multivariate profiles of their existence or non- by IVR. It is possible that this disadvantage of IVR
existence in each frame and to use these estimates of could have been corrected if we had pursued additional
size to develop weights that were used to combine iterations of alternative IVR scripts. We terminated the
these segments into an equal-probability sample of the exercise before these iterations, though, based on the
population. finding: that all evacuees in telephone households with
Concerns could be raised about the under-represen- listed phone numbers outside the hurricane area had
tation of three population segments in the frames applied either to the ARC or to FEMA for assistance
discussed up to now: evacuees who lived outside the with traceable contact information. This means that
hurricane area, were reachable by RDD, but who were these people were already part of our primary sample
not included on either the ARC or FEMA lists (either frames, making it unnecessary to screen for them in a
because they did not apply or because they did not supplemental national RDD sample.
provide traceable telephone contact information); other The most feasible way to reach the remaining groups
evacuees who lived outside the hurricane area who that are under-represented in the frames discussed
could not be reached by telephone (whether or not they earlier (i.e. evacuees who could not be reached by tele-
applied for ARC or FEMA assistance); and residents of phone) using probability sampling would have been to
the affected area who remained in the area but could use a survey field staff to carry out face-to-face inter-
not be contacted by telephone (because they did not views on an area probability sample of households and
have a working land line that could be reached by RDD group quarters. We did not do this in our survey of
and they either did not have a cell phone or did not Katrina survivors due to financial constraints. If we had
apply to the ARC or FEMA and provide a cell phone done so, it would have been important to include infor-
contact number). We attempted to reach the first of mation that allowed us to determine whether each
these three groups (i.e. evacuees who lived outside the respondent sampled from this frame also had a proba-
hurricane area, were reachable by RDD, but who were bility of selection in the list samples and the RDD
not included in either the ARC or FEMA lists) by sample. With regard to design considerations, a sample
experimenting with the use of a national RDD sample of this sort that focused on people living in the area
that employed multiplicity methods (i.e. asking for affected by the hurricane would be based on a conven-
evacuees among current household residents and among tional multi-stage clustered area probability sampling
first-degree relatives of a randomly selected informant design.
in each household) either with live telephone inter- Logistical complications would exist in sample selec-
viewers or interactive voice response (IVR) messages tion, as the Census measures of size used to select
with follow-up live telephone interviewers. Based on sampling segments (i.e. blocks in urbanized areas and
data from the ARC and FEMA lists about geographic block-equivalents in rural areas) would be much less
evacuation patterns, we anticipated that approximately accurate than normal because of housing destruction.
one in every 500 households in the US outside of the Block listing would also be more complex than usual
hurricane area would contain one or more hurricane in that the normal landmarks used to define sample
evacuees and that some additional number of segments would in some cases be destroyed, possibly
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprPost-disaster needs assessment surveys S11
making it necessary to work with knowledgeable local Geographic propinquity need not be a defining feature
informants (e.g. mail delivery workers) to help define of these groups. The families and close friends of the
segment boundaries. It might also be efficient to select people killed in an airplane crash, for example, would
larger segments than in a usual household survey to be a high-risk group for needs assessment that is widely
allow for the likelihood of housing unit destruction and dispersed in terms of geography. In the case of natural
to invest more heavily in block listing than usual. disasters, there are some other high-risk groups that
Logistical complications would also exist in interviewer might be expected to be more consistent across situa-
travel and housing and because of infrastructure tions, such as residents of nursing homes and people
damage. While making fieldwork more difficult, though, with physical disabilities who would have a difficult
none of these problems would be insurmountable. time evacuating.
An argument could be made that even non- One of the most important of these high-risk groups
probability sampling would be useful situations where after Hurricane Katrina consisted of people with pre-
probability sampling is prohibitively expensive so long hurricane severe-persistent mental illness (SPMI) whose
as the sampling was based on characteristics identified medical records were temporarily lost in the storm,
as reflecting high exposure to disaster-related stressors whose local pharmacies were destroyed, and who were
(e.g. areas that were directly hit by a tornado or areas unable to refill their antipsychotic medications. This
that were not reconnected to services after a natural group represents an extreme case of the much larger
disaster), as such an approach could provide useful group of people with pre-existing chronic conditions
information about the range of exposures and psycho- who were found in assessments of EC residents often to
logical reactions to the disaster. Quotas on the basis of have unmet need for maintenance medications to treat
a cross-classification of basic socio-demographic vari- their chronic conditions (Brodie et al., 2006). An exac-
ables could be imposed in such a case in order to guar- erbating factor is that the Strategic National Stockpile
antee breadth of coverage. of emergency medications (Centers for Disease Control
and Prevention, 2005) and short-term deployments of
High-risk population sampling emergency medical personnel in the Public Health
Initial needs assessment surveys of Hurricane Katrina Security and Bioterrorism Preparedness and Response
survivors focused on high-risk populations, including Act (Rosenbaum, 2006) both failed to anticipate this
pre-hurricane residents of New Orleans who remained problem by providing ready access to desperately-needed
in their homes shortly after the hurricane (Centers for medications for SPMI and other extreme chronic con-
Disease Control and Prevention, 2006a), people staying ditions. Once the problem was recognized, emergency
in evacuation centers (Centers for Disease Control and mental health service planners made special efforts to
Prevention, 2006b), and people residing in FEMA- obtain psychotropic medications for emergency medical
sponsored trailers or hotel rooms (Abramson and clinics as well as to recruit psychopharmacology experts
Garfield, 2006). First responders also are a high-risk to provide appropriate medications to people with
population of importance that has been the focus of SPMI who sought care in these clinics.
considerable research attention (Ben-Ezra et al., 2006; In the course of these planning activities, questions
Fullerton et al., 2004). Although these populations arose about the magnitude and distribution of unmet
make up only a small percentage of all the people who needs for services of the pre-hurricane SPMI population.
were affected by Katrina, their distinct geographic Needless to say, people with SPMI make up such a small
characteristics and their presumably high level of expo- part of the general population that we were unable to
sure to hurricane-related stressors make them impor- make reliable statements about the special needs of
tant targets for needs assessment. people with SPMI based on the CAG sample. Assess-
Such high-risk populations can be expected to vary ments could, of course, be made of unmet demand for
widely across disaster situations. The workers in a gov- treatment of SPMI based on systematic epidemiologic
ernment office building that was the target of an surveillance systems set up in emergency health clinics.
anthrax attack along with their families might be a However, we know that information on demand for
high-risk group in one disaster situation, while the resi- services often fails to give an accurate assessment of
dents of a geographic area close to a toxic chemical spill need for services, which is why general population needs
might be a high-risk group in another disaster situation. assessment surveys are of such great importance.
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprS12 Kessler et al.
In the case of comparatively rare high-risk popula- (CIPSEA; www.eia.doe.gov/oss/CIPSEA.pdf) call for
tions, the only practical option for needs assessment is increased data sharing among statistical units of federal
to gain access to a list sample that can be used as a agencies and for a correspondingly more extensive con-
sampling frame for tracing. It might sometimes be pos- fidentiality umbrella over shared data.
sible to merge multiple list samples to refine sampling
or to answer certain critical policy questions regarding Design considerations
high-risk populations. For example, a comprehensive
list existed of all nursing home residents in the areas Panel versus trend study designs to monitor change
affected by Katrina that could be linked to the National Our Hurricane Katrina tracking surveys use a panel
Death Index (NDI) in order to address concerns that design (i.e. the same respondents interviewed repeat-
the relocation was associated with a substantial increase edly over time) rather than a trend design (i.e. a new
in mortality of nursing home residents, although this sample of respondents selected in each interview) to
would involve substantial delays in light of the fact that monitor change. The panel design is preferable to the
posting in the NDI sometimes does not occur until as trend design when the main purpose of tracking is to
much as a year after death. Linkage of this sort could use baseline information about risk to predict the sub-
be done across multiple administrative data systems to sequent onset of some adverse outcome that might be
generate very useful data, especially when done in con- the subject of preventive intervention. There is consid-
junction with follow-up surveys. It would be possible, erable interest in the literature on PTSD, for example,
for example, to use linked income tax records and mor- in the extent to which baseline information obtained
tality records to track the mortality experience of pre- shortly after a disaster (the ‘peritraumatic’ time period)
hurricane residents of the areas affected by Hurricane can help pinpoint which disaster victims will or will
Katrina who either subsequently returned to their pre- not subsequently develop PTSD (e.g. Shalev and
hurricane residence or who moved to a different part Freedman, 2005; Simeon et al., 2005). Panel data are
of the country. needed to investigate such individual differences.
Similarly, it would be possible to link pre-disaster However, the panel design is inferior to the trend design
medical-pharmacy claims data of members of large when the purpose of the study is to monitor aggregate
health plans in areas affected by a disaster with trends, as the problems of sample reactivity and attrition
post-disaster claims data, income tax data, and NDI cumulate in a panel design but not in a trend design.
mortality data to track the associations of pre-disaster The decision to use a panel design in the Katrina
morbidity with subsequent geographic mobility, health surveys was based largely on the high costs and com-
care utilization, and mortality. Targeted tracking surveys plexity of selecting the baseline sample. We did not
then could be used to investigate the determinants of have enough funds to select a new sample each time
substantially reduced health care utilization among we carried out a subsequent wave of data collection. We
people with evidence of high pre-disaster need for treat- attempted to deal with the attrition problem, which we
ment. The main impediment to this kind of integrated expected to be higher than in most other panel surveys
analysis is lack of coordination among the agencies and because of the instability of the housing situations of
organizations that maintain the many different admin- many baseline respondents, in a number of ways. First,
istrative data systems that would be relevant to such we made it clear to respondents in the initial recruit-
undertakings. Legal constraints on sharing identifying ment process that we planned to follow them over time
information are important considerations here along to track the course of adjustment to the disaster and
with organizational inertia and structural disincentives we asked for their commitment to stay with the project
to collaborate in inter-organizational initiatives. An over a period of several years. Previous research has
inter-agency task force in the US federal government shown that commitment probes of this sort lead to
is currently grappling with these complex issues in an significant improvements in respondent participation
effort to develop a workable plan for the use of admin- (Oksenberg et al., 1979).
istrative databases in these ways in response to future In conjunction with the commitment probe, we
disasters. In addition, legislation and regulations associ- characterized the sample to participants as a ‘consumer
ated with the US federal government’s Confidential advisory group’ in an effort to build commitment to
Information Protection and Statistical Efficiency Act the ongoing enterprise and letting respondents know
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprPost-disaster needs assessment surveys S13
that they were community advisors whose views were to report their experiences over the past 30 days. The
valued by the project team and the policy-makers who first two of these designs are examples of the continu-
were the primary audience for study findings. Based on ous time tracking design, one in which the researcher
concerns about problems tracking the movements attempts to capture information across the entire time
of respondents, we provided each respondent with a interval since the disaster. The third design (i.e. six-
plastic identification card similar to a credit card that month intervals between data collection waves with
contained the project 800 number and web address. We 30-day recall questions), in comparison, is an example
asked respondents to use this card to contact us when- of a ‘snapshot’ design, one in which the researcher
ever they moved to give us their new contact informa- attempts to collect data only in a sample of time inter-
tion. We also gathered contact information for three vals rather than to capture information about experi-
people who were geographically stable that would know ences over the entire time interval since the disaster.
the whereabouts of each respondent if the respondent The decision as to whether the continuous time
moved and we were unable to trace them. Finally, we design or the snapshot design is preferable depends on
sent respondents mailings of study results every six a number of substantive and logistical considerations
months in order to maintain rapport and to obtain mail that can vary from one study to the next. The most
address correction information when respondents commonly used design in post-disaster needs assess-
moved and left a forwarding address. This set of ment surveys is a mixed design in which the time inter-
approaches has been very successful in allowing us to val between waves of data collection is fairly long (6–12
track the baseline sample with over 90% success over months), some information is collected in a continu-
subsequent waves. ous-time framework (e.g. retrospective questions about
While the strategies described in the last paragraph the persistence of PTSD over the entire time interval
have the potential to maximize continued participation since the last survey), while other information is col-
of baseline CAG members in subsequent interviews, lected in a snapshot framework (e.g. questions about
they also have the potential to bias results by changing current needs for services). However, this is unlikely to
the cognitive schemas that respondents use in answer- be the optimal design for addressing the research ques-
ing survey questions. One way to assess the magnitude tions that these studies are typically designed to address.
of this problem is to carry out a trend survey in parallel The mixed design is the right one, as needs assessment
with the panel survey to see the extent to which aggre- surveys always have multiple goals and it is important
gate estimates differ in the two samples. We had origi- to build in the flexibility to include questions that focus
nally intended to do this in the CAG, but financial on diverse time intervals. However, the long-time inter-
constraints made it impossible to implement a parallel vals that typically exist between waves are sub-optimal,
trend component of the design. More generally, though, as they make it likely that recall bias will be magnified
some version of a mixed panel-trend design would gen- and that potentially important short-term trends will
erally be the preferred design in post-disaster needs be missed.
assessment tracking when the complexities of sampling Based on these considerations, a strong argument
are not so great that this approach is prohibitively could be made for a continuous tracking design using
expensive. The mixed panel-trend is a preferred design the mixed panel-trend approach described in the last
because we will usually be interested both in aggregate sub-section. A variety of mixed panel-trend designs
trends and in individual-level change. exist (Kish, 1987). One of the most appealing is the
rolling panel design, in which new trend survey respon-
Sampling time dents are recruited on a regular basis (e.g. in monthly
An important design consideration in longitudinal samples) and followed over a specified series of panel
tracking studies is the time interval of assessment. waves that overlap in time with new trend surveys. This
Some tracking surveys are carried out every month and is the design used, for example, in the Bureau of Justice
ask respondents to report their experiences over the Statistics ongoing National Crime Victimization Survey
past 30 days. Other tracking surveys are carried out (NCVS; www.icpsr.umich.edu/NACJD/NCVS), where
every six months and ask respondents to report their monthly surveys include samples of people who are
experiences over a six-month recall period. Others still interviewed for the first through sixth times with
are carried out every six months and ask respondents six-month follow-ups between waves of interviewing.
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprS14 Kessler et al.
Random effects regression analysis can be used to esti- Environmental Protection Agency (EPA) tests docu-
mate the impact of non-response bias in the panel mented that fears of toxic chemical exposure were
component of the data on estimates of trends by taking unfounded) both through the investigation of time
into consideration systematic variation in trend series in point prevalence of mental disorders and
estimates across the sub-samples (Verbeke and through the inclusion of new public opinion questions
Molenberghs, 2001). on weekly or monthly waves of the survey that ask
Given that the tracking period for post-disaster explicitly about awareness of and reactions to the
needs assessment surveys is typically rather short (no mini-interventions.
more than several years), a useful variant on the rolling It is important to recognize that the notion of ‘con-
panel design would be to begin with a rather large tinuous’ time sampling is a misnomer, as retrospection
baseline sample interviewed as soon after the disaster is always needed in longitudinal data collection even
as possible in order to assess peritraumatic stress reac- when the time interval between waves is very short.
tions and to obtain rapid response information about Recall bias can easily creep into retrospective reports,
need that can be provided quickly to service planners. especially with regard to reports of emotional experi-
In addition, smaller trend samples could be selected on ences. Indeed, methodological research has shown that
a weekly or monthly basis for a period of six months or bias can be found in emotion reports even over a recall
so in order to provide fine-grained tracking information period as short as 24 hours (Diener and Seligman,
on aggregate patterns of persistence or remission of 2004). Researchers interested in reducing this bias have
symptoms. Fine-grained tracking could be especially developed the method of Ecological Momentary Assess-
useful when carried out in conjunction with monitor- ment (EMA) (Stone et al., 1999). EMA uses beepers
ing of mass media messages and treatment recruitment programmed to go off at random times in the day and
efforts in order to provide information about the effects diaries to have respondents record moment-in-time
of social marketing interventions on KAB. Respon- feelings across a sample of moments and days. An EMA
dents in the baseline interviews could then be re- trend study, for example, might recruit a separate
interviewed after the initial six-month trend period in random sample of disaster victims each week for one
a panel design that might have a six-month time year and ask them to complete moment-in-time assess-
interval between waves. ments at five randomly selected moments on each of
The panel component could be carried out with the the seven days of the week. EMA assessment can some-
full baseline sample in a rolling panel framework (e.g. times be a very useful adjunct to more conventional
respondents initially interviewed in Month 1 re- panel data collection (e.g. deVries, 1987; Wang et al.,
interviewed in Month 7, those initially interviewed in 2004). When EMA is considered too molecular, a daily
Month 2 re-interviewed in Month 8, those initially diary can be used instead, with respondents are asked
interviewed in Month 6 re-interviewed in Month 12) to record the experiences of their day before they go to
to collect information continuously each month, pos- bed each evening over the course of a one-week or two-
sibly including a small trend component (i.e. a small week diary period (e.g. Chepenik et al., 2006; Henker
representative sample of new respondents interviewed et al., 2002).
each month in Months 7+). Or the panel interviews
could be carried out only in a probability sub-sample of Before–after designs
baseline respondents that over-samples those with An important limitation of virtually all disaster needs
baseline indicators of long-term risk (e.g. retrospectively assessment surveys is that respondents are only inter-
reported pre-disaster history of psychopathology, viewed after the disaster, making it impossible to make
extreme peritraumatic stress reactions, high exposure direct before–after comparisons that could estimate the
to disaster-related stressors). impact of the disaster on the prevalence of mental dis-
This sort of mixed design would maximize flexibility orders in the population. There are some exceptions to
in addressing a wide range of substantive issues and this general problem. For example, the Epidemiological
would allow for the rapid assessment of population Catchment Area (ECA) Study in St Louis (Regier
response to mini-interventions (e.g. an announcement et al., 1984) was carried out shortly before the 1985
that special funds have been allocated by the federal flood, dioxin exposure scare, and subsequent mass evac-
government for disaster relief; an announcement that uation of Times Beach, a small town on the outskirts
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprPost-disaster needs assessment surveys S15
of the St Louis Metropolitan Area that was in the ECA government office complex that killed 168 people.
sample. This created an opportunity to carry out a Given the size of the three surveys described earlier and
before–after comparison of mental health associated the size of Oklahoma City, a sample of roughly 5000
with the Times Beach disaster. But situations of this adult residents of Oklahoma City would have been
sort are rare. The much more typical situation is for interviewed in one of these surveys in the 12 months
studies of the mental health impact of disorders to be before the terrorist attack if all three surveys had been
carried out only after the fact. Information about pre- in place in the mid-1990s. A sample as large as this
disaster psychopathology is collected retrospectively. would create a very stable baseline for assessing the
Two practical approaches exist to introduce before– mental health effects of the terrorist attack. In the case
after information on a more routine basis into post- of smaller disaster areas, such at Graniteville, South
disaster needs assessment surveys. The first is to use Carolina (population 7112), we could combine infor-
tracking information from ongoing government health mation from three surveys in similar communities
surveys to construct an appropriate post hoc pre-disaster collected over the prior 12 months to construct an
comparison group. Three major ongoing national approximate pre-disaster comparison group. Or we could
surveys exist that could be used in this way: the US combine data from interviews with residents of areas in
National Health Interview Survey (NHIS; www. the vicinity of the disaster site collected over a decade
cdc.gov/nchs/about/major/nhis/his.sample.htm), which or more before the disaster with post-disaster interviews
carries out face-to-face interviews weekly with a nation- in the affected area and use interrupted time series anal-
ally representative sample that includes approximately ysis (McDowell et al., 1980) to estimate the effect of the
43,000 households each year; the CDC Behavioral Risk disaster on the mental health of residents.
Factor Surveillance Survey (BRFSS; www.cdc.gov/ There are bureaucratic impediments to carrying out
brfsssabout.htm), which carries out weekly telephone this type of analysis in that the government agencies
interviews with a sample in each of the 50 United that administer the three ongoing surveys have restric-
States, the District of Columbia, Puerto Rico, the US tions on making information available to researchers
Virgin Islands, and Guam that includes more than about small area geographic characteristics of individ-
350,000 interviews each year; and the SAMHSA ual respondents. Even more important, the agencies are
National Survey on Drug Use and Health (NSDUH; slow in releasing the survey data for public use, making
www.oas.samhsa.gov/redesigningNHSDA.pdf), which it impossible to obtain pre-disaster data in a time frame
carries out annual face-to-face interviews with a nation- that would be useful for disaster response planning
ally representative sample of approximately 70,000 purposes. These impediments made it impossible for us
respondents with an over-sample of the most populous to use data from any of these surveys in pre-post analy-
states and of youth. Importantly, all three of these ses of the mental health effects of Hurricane Katrina
surveys include a version of the K-6 scale of psychologi- even though we estimate that more than 6000 residents
cal distress (Kessler et al., 2002; Kessler et al., 2003), the of the areas affected by Katrina were respondents in
core global screening measure of the Diagnostic and one of these three surveys in the 12 months before the
Statistical Manual of Mental Health, fourth edition hurricane. Efforts have been made recently to decrease
(DSM-IV) anxiety-mood disorders that we use in our the time delays in producing usable data files from these
model post-disaster mental health needs assessment surveys. We hope that the creation of the NIMH center
tracking survey. for post-disaster mental health needs assessment surveys
This truly massive resource of baseline information, will help cartelize these efforts and make it possible to
with roughly one out of every 600 adults in the entire use these surveys to create pre-disaster comparison
US being interviewed in one of these surveys each year, groups that can be used in needs assessment studies
could be used to provide baseline information to assess of future disasters.
the effects of disasters on the mental health of local The availability of before–after data can be very
populations by selecting sub-samples appropriate for useful in addressing an important question about need
comparison with targeted disaster populations. To illus- that we noted in the introduction: that the socio-
trate the potential of this approach, consider the case demographic correlates of need for treatment found in
of Oklahoma City (3 450 000 residents in the 2000 post-disaster surveys might have existed before the
Census), the site of a 1995 terrorist attack on a US disaster, in which case they could be unrelated to the
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprS16 Kessler et al.
disaster. An illustration of such an analysis is our use 6000 residents of the areas affected by Hurricane
of data collected in the 2001–2003 National Comorbid- Katrina were respondents in one of the three major
ity Survey Replication (NCS-R) (Kessler and Merikan- government surveys that collect K-6 information in the
gas, 2004) among respondents in the two Census 12 months before the hurricane. It might have been
Divisions subsequently affected by Hurricane Katrina difficult to trace all these people by trying to contact
to approximate a before–after comparison of the preva- them at their pre-hurricane addresses and searching for
lence of serious mental illness (Kessler et al., 2006). The them on safe lists and ARC-FEMA lists, but the degree
K-6 was used to screen for 30-day DSM-IV anxiety and of tracing success would in itself have been useful to
mood disorders in both the NCS-R and the baseline know along with the substantively useful information
CAG survey. Based on previous K-6 validation (Kessler that would have been obtained from the individual-
et al., 2003), scores on the 0–24 scale in the range 13–24 level pre-post comparisons of K-6 scores and pre-
were classified probable serious mental illness (SMI). disaster predictors of individual-level changes in these
A variety of socio-demographic correlates of SMI scores. Although we are unaware of any previous use of
were assessed in a comparable way in the two surveys. this design to evaluate the effects of disasters, we plan
The estimated prevalence of SMI was found to be dra- to use this design as part of our collaboration with the
matically higher in the CAG than the NCS-R. Socio- BRFSS in future post-disaster mental health needs
demographic variation in this between-survey difference assessment studies.
was assessed by pooling the data in the two surveys into
a single data analysis file and estimating logistic regres- Surveying help-seekers
sion equations to predict SMI from a 0–1 dummy vari- The ARC and FEMA lists of people who apply for
able (0 = the NCS-R, 1 = CAG), the socio-demographic assistance are made up entirely of people who sought
variables, and interactions between the survey dummy help. Help-seekers presumably differ from other resi-
and the socio-demographic variables. A great many dents of disaster populations in a number of ways,
significant socio-demographic correlates of SMI were including both in the extent of their need for help (i.e.
found in the CAG, such as female gender, low educa- the extent to which they experienced property loss in
tion, and pre-hurricane unemployment. However, all of the disaster) and in the extent to which they are moti-
these associations were also found in the NCS-R and vated and capable of making an application. Although
none of the associations was significantly stronger in we might expect to find a meaningful number of victims
the CAG than the NCS-R. This is consistent with the with high need who did not seek help due to extreme
view that the adverse mental health effects of Katrina physical restrictions (e.g. housebound in a wheelchair),
were equally distributed across broad segments of the possibly in conjunction with extreme social isolation
population (in the sense that rates of SMI increased and communications problems (e.g. no access to a tele-
proportionately in each group) despite the fact that phone, unable to speak English, blind or deaf), relief
SMI was significantly more common in some socio- agencies make efforts to find such people through a
demographic segments of the CAG sample. variety of community outreach and household screen-
We noted earlier that there are two practical ing programs. Based on this fact, it is not unreasonable
approaches to introduce before–after information on a to think that fairly representative data on demand for
routine basis into post-disaster needs assessment surveys. services could be obtained by sampling people who
The first one, which we just reviewed, is to use informa- applied for relief even though the sample might not
tion from ongoing government health surveys to con- represent all people with need for services.
struct an approximate pre-disaster comparison group In the case of need for treatment of mental disorders,
for pre-post trend analysis. The second is to use the a very important set of post-disaster help-seekers con-
same sort of data for panel analysis. The latter is often sists of those who call the various mental health crisis
referred to as a ‘follow-back’ design (Castle et al., 2004; hotlines that are typically established by local and
Seeman et al., 1989). In this approach, respondents who national mental health associations. The largest and
participated in a government survey some time prior to most important of these is the National Suicide Preven-
the disaster could be traced and re-interviewed after tion Lifeline, the only national suicide prevention and
the disaster to provide individual-level pre-post infor- intervention telephone line sponsored by the federal
mation. As noted earlier, we estimate that more than government (www.suicidepreventionlifeline.org). The
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprPost-disaster needs assessment surveys S17
Lifeline was launched in December 2004 to link callers paragraph, we plan to develop a similar system that will
to staff in more than 120 mental health crisis centers carry out CAHPS-like follow-up surveys with patients
around the country. SAMHSA used the lifeline crisis who are referred to post-disaster mental health services.
phone number as the hub for mental health referrals It is important for these surveys to be very inexpensive
during the aftermath of Hurricane Katrina and it is because the goal would be to give all patients a chance
likely to do so again in future mass disasters. Follow-up to respond so as to obtain countable information for as
needs assessment surveys with callers of Lifeline and many service providers as possible. As a result, patients
other crisis hotlines could be useful components of who have an email address will be surveyed using inex-
larger post-disaster mental health needs assessment pensive web survey technology (Schonlau et al., 2002),
efforts in at least three important ways. while other patients will be interviewed using inexpen-
First, an important under-studied issue concerns sive IVR technology.
patterns and determinants of unmet need for treatment Third, there is considerable uncertainly about the
of mental health problems after disasters (Boscarino most appropriate interventions to use in treating the
et al., 2005; Stuber and Galea, 2005). A useful way to emotional problems of disaster victims (Watson and
study this issue would be to carry out follow-up inter- Shalev, 2005). This uncertainty is due in no small part
views with callers of mental health hotlines that were of the difficulties involved in carrying out controlled
given a referral for treatment. The information obtained treatment studies in disaster situations. A potentially
in these interviews about modifiable barriers to treat- useful way to address this problem would be to build in
ment could be organized using existing conceptual randomization of referrals of help-seekers to different
frameworks (Rogler and Cortes, 1993) that might treatment settings and types in conjunction with the
provide insights into potential values modifications in follow-up interviews described in the last two para-
the referral process. We know of no previous research graphs. This approach could be used to evaluate a
of this sort carried out with callers to post-disaster highly specified treatment approach that is experimen-
mental health referral lines. However, we have estab- tally provided to a small probability sub-sample of help-
lished collaborations with the ARC and with Mental seekers in comparison to the usual care provided to all
Health America (MHA; formerly known as the other disaster victims. Alternatively, all help-seekers
National Mental Health Association) as well as with a could be randomized across the range of seemingly
number of MHA affiliates, including the National appropriate treatment settings available in a given
Suicide Prevention Lifeline, to implement this type of disaster situation and follow-up questionnaires of the
study as part of a larger plan for the proposed NIMH sort described in the last paragraph could be used to
center for post-disaster needs assessment tracking. determine whether effectiveness varies significantly
Second, relatively little is known about the quality across these settings both in the aggregate and for
of care provided to patients referred by crisis hotlines patients with particular characteristics. With regard to
after disasters to local mental health treatment centers. the latter, the large numbers of patients included in the
This quality control problem could be addressed, at randomization in a major disaster would make it possi-
least in part, by carrying out systematic follow-up inter- ble to determine whether overall treatment effective-
views that assess patient satisfaction. Surveys of this ness could be improved by some type of patient-program
sort are now a routine part of many treatment quality matching.
assurance programs, the most notable example being
the Consumer Assessment of Healthcare Providers Overview
and Systems (CAHPS) program (www.chaps.ahrq.gov/ As noted in the introduction, a wide range of measure-
default.asp), which now includes a behavioral health ment, design, and analysis issues present themselves in
care component. Publicizing the ‘report cards’ gener- planning a consistent approach to the implementation
ated by the results of these surveys in conjunction and of post-disaster mental health needs assessment surveys.
other quality indicators has been shown to influence We focused here only on design issues due to the fact
consumer choice of health plans (Jin and Sorensen, that these have been much less widely discussed in the
2006; Oetjen et al., 2006) which, in turn, is hoped to literature than measurement or analysis issues. It needs
influence health plan performance. As part of the to be recognized, though, that consistency of mea-
proposed collaboration with MHA noted in the last surement has to be the first step in the process of
Int. J. Methods Psychiatr. Res. 17(S2): S6–S20 (2008)
Copyright © 2008 John Wiley & Sons, Ltd DOI: 10.1002/mprYou can also read
