 |
 |
|
|
National Youth Anti-Drug Media Campaign
Phase 1 (Report No. 1)
Appendix A: Supplemental Information on Methodology
Selection of Target and Comparison Sites
The evaluation of Phase I of the ONDCP Youth Anti-Drug Media Campaign (the Media Campaign) involved 12 target sites and 12 comparison sites. The sites selected for the Campaign were metropolitan statistical areas (MSAs), as defined by the U.S. Census Bureau. An MSA is defined as a city with 50,000 or more inhabitants or an urban area of at least 50,000 inhabitants and a total metropolitan population of at least 100,000 (75,000 in New England). Target and comparison sites were classified as small, medium, or large on the basis of the size of the MSA so that sites could be paired according to population size. Small sites had populations of less than 200,000 people, medium sites had from 200,000 to 500,000, and large sites had 500,000 or more.
Target sites and comparison sites were selected and matched as described in Chapter 2. Although the site-specific data included in Chapter 4 indicate that target-comparison pairs were not always highly congruent on all demographic variables, the comparison sites were reasonable well matched for the purpose of understanding change in communities that were exposed to the Media Campaign intervention. The only predetermined difference between the target and comparison sites was presumed to be the presence or absence of paid anti-drug advertisements.
Pairing of sites was based mainly on the size of each site's MSA and predominant ethnic group. The percentages reported in Chapter 2 for ethnic categories in the sites refer to the specific ethnic group population for whom the target and comparison sites were matched. In sites such as Boise, Sioux City, and Portland, there was no significant representation of any ethnic groups; thus, percentages are reported for the white population.
In the analysis for the Phase I final report, some comparison sites will have to serve as controls for more than one target site to compensate for the three comparison sites that were dropped from the in-school survey study. The pre-post findings from the quantitative research, while not presented in this report, will be available in the final report. Detailed methods regarding the collection and analysis of quantitative data will be covered in the final report as well.
Qualitative Data
Qualitative data were obtained from baseline and intermediate site visits to all communities. The data obtained were gathered through youth and parent focus groups in both urban and non-urban areas and key informant interviews with members of the community. As well, additional contextual information was collected through observations of community anti-drug activities and meetings. Prior to conducting the site visits, letters were mailed from ONDCP to the Community Anti-Drug Coalitions of America (CADCA) representative (if there was one) in each of the target and comparison sites and to the State Prevention Coordinators. The purpose of these letters was to ask for cooperation with the study and to explain the purpose of the visit so that community members would have a better understanding of why ONDCP was collecting data in their community. Exhibit A-1 is a copy of the letter that was sent to local communities.
The guidelines used by site visitors to identify key informants when planning and scheduling site visits relied heavily on input from the CADCA representative and the State Prevention Coordinator. Site visitors conducted research on each of the local communities before scheduling the visit or making initial contact with local representatives. The local Chamber of Commerce and the Internet were key sources of information for learning more about each community.
Field staff needed to be highly flexible in collecting data in both target and comparison sites. Sometimes visits could be scheduled only a few weeks before going on site, and coordinating several focus groups and key informant interviews during the same week was challenging. On at least four of the intermediate site visits, severe weather (including tornadoes) caused the focus group to be cancelled at the last minute. This required the site visitors to coordinate replacement focus groups while they were still on site.
Focus Groups
Site visitors conducted eight focus groups during each site visit. Four focus groups were conducted in the central city or urban area and four were conducted in non-urban areas. Site visitors worked with a local coordinator to recruit youth and parent participants for the focus groups. Youth focus groups consisted of elementary school (grades 4-6), middle school (grades 7-9), and high school (grades 10-12) students. An effort was made to keep the groups homogeneous with regard to age (e.g., not putting 7th graders and 9th graders in the same group). Coordinators from various community agencies (e.g., Boys and Girls Clubs, churches, and the YMCA or YWCA) were instrumental in helping to arrange the focus groups. They contacted youth about participating and sent information to their parents explaining the purpose of the focus groups. Youth were sometimes recruited from local schools with the understanding that a school would not be contacted if it was participating in the in-school survey component of the research (to avoid predisposing youth's responses). High school youth were offered a stipend of $10 for their participation or a gift certificate valued at $10; local coordinators made the determination of whether to provide cash or gift certificates. Parents who participated were offered a stipend of $25. Local coordinators also were paid an honorarium for their time and effort in coordinating the focus groups. Refreshments were provided at all of the focus groups. Many of the youth focus groups were convened in the afternoon, when school was out. Parent focus groups were often held in the evening.
Communities differed on issues such as (1) where best to hold focus groups (e.g., Boys and Girls Club facilities, school buildings, or rented conference rooms); (2) whether passive parental permission was sufficient or active permission was required for youth to participate; (3) what kind of form to use when active parental permission was necessary and how best to obtain parental permission; and (4) what type of gift certificates would be appropriate if gift certificates rather than cash stipends were provided to focus group youth.
During the process of organizing focus groups, the evaluation contractor provided background information about the focus groups to the local coordinator. Exhibit A-2 is a sample information letter for local contacts to use as a template. Exhibit A-3 provides a summary of information used to help individuals and agencies who were organizing the focus groups.
Most procedures for organizing focus groups were consistent across sites to ensure comparability between sites and to avoid bias in the followup data. For example, each focus group included 8-10 participants; different participants were selected for the initial focus group and for each followup group; focus group youth and parents were not drawn from those participating in the survey component to avoid data contamination; developmental issues were taken into account when scheduling youth focus groups (e.g., 4th-6th grade focus groups generally did not include both boys and girls in the same group); and stipends were provided to each parent and high school student who participated.
Interviews With Key Informants
Key informant interviews were conducted in each of the sites to identify important contextual information that might affect the impact evaluation and to obtain information on local community members' response to the Media Campaign. Site team members interviewed the key individuals who were identified during the initial site visit preparation as being knowledgeable about the community's drug problems, about local programs and initiatives for combating the problems, and about other contextual information that might affect the impact evaluation (e.g., knowing about any major drug-related local events or news). Additional informants were identified through the local contacts that site visitors established. Site visitors were given preliminary information regarding ONDCP paid ads that were being run in the target sites so they would understand what the informants meant when they mentioned seeing a particular PDFA ad.
The goals of the key informant interviews were to expand on and confirm information that will be available from survey results at the completion of the Campaign; to corroborate information gathered from focus groups; to explore themes that emerged through discussions with other community members; to obtain community leaders' responses to the Campaign ads and their content (i.e., to determine if the prevention community viewed the Campaign as supportive of their local efforts or if the Campaign provided any early opportunities for sites to bolster existing programs); to learn about media informants' responses to the Campaign and the factors that influenced their decisions to run specific ads; and to obtain information on any changes in contextual conditions that would explain changes in youth's and parents' perceptions and attitudes.
Media Tracking
The procedures for monitoring and tracking exposure to anti-drug messages in the target and comparison sites involved systematically collecting data on television-aired PSAs and print media anti-drug coverage in editorials and news articles. Media monitoring included network affiliates for the three major national television networks (ABC, CBS, and NBC), WBN (Time-Warner national cable), CNN, FOX, UNIVISION (including Spanish-language programming), TBS, UPN, IND, and TELEMUNDO (Spanish cable). Electronic monitoring was available in all sites except Boise, Sioux City, Tucson, Eugene, and Duluth. Specific details regarding the monitoring and tracking activities are discussed in Chapter 2.
Data Collection Instruments and Procedures
This section includes a summary of the data collection instruments and procedures used to gather qualitative information for this report.
Key Informant Discussion Guides
Semistructured discussion guides were used to structure the interviews with key informants. The guides were open-ended to allow for additional probes to be included in the questioning. Interviews generally were scheduled prior to the site visit, although some interviews took place during the week of the site visit. Key informant interviews typically lasted 30-60 minutes and usually took place at the interviewee's place of employment, although some interviews were done at a neutral site (such as a restaurant) or over the telephone. Interviews were not tape-recorded but were documented through careful notetaking by the interviewer. Site visitors obtained useful documents or reports whenever possible at the time of the interview.
Interview notes were reviewed at the end of each day to clarify the responses noted and to add any other notes the interviewer was unable to write down during the interview. This early review of interview notes helped ensure the accuracy of the information recorded prior to the more intensive data analysis that followed.
Exhibit A-4 is the key informant discussion guide.
Focus Group Questions
Separate focus group questions were developed for elementary school students, parents, and older youth (middle school and high school students). The Campaign focus groups consisted of elementary school students (grades 4-6), middle or junior high school students (grades 7-9), high school students (grades 10-12), or parents. Each baseline and intermediate site visit typically involved eight focus groups, including two each from each of these four participant categories. Furthermore, within each participant category, urban and non-urban focus groups were identified. Focus groups were often conducted under the auspices of Boys and Girls Clubs, churches, and YMCAs and YWCAs.
Focus groups generally lasted 30-90 minutes and involved 6-10 participants, a moderator who ensured that all participants stayed on topic and participated, and a notetaker who was responsible for taking notes, setting up and operating the tape-recorder, creating a seating chart, and distributing refreshments and stipends. All focus group sessions were tape-recorded for later analysis (i.e., tapes were later used to generate written transcripts that were used in the data analysis to validate findings gleaned through analysis of focus group data summarized in site visit reports). Focus group questions are listed in Exhibit A-5.
Document Review
Documents such as newsletters, newspaper articles, analytic reports, and statistical summaries that were obtained during site visits typically came from key informants or were obtained from agencies or libraries. Such written and archival materials were either promptly reviewed and evaluated for relevance by site visitors or were categorized, filed, and cited in a succinct reference list for later analysis. Such analysis occurred continuously and at all different stages of the research process so that any preliminary insights would be helpful in interpreting the data and so that analysts would not be overwhelmed with the large volume of printed materials.
Observations
Site visits provided opportunities for staff to observe relevant community- or Campaign-related events and activities such as town hall meetings, speeches, ribbon-cutting ceremonies, and advertisement production. Observing such events complemented the evaluation contractor's primary activities of interviewing, conducting focus groups, and collecting archival and printed materials.
Preparation of Qualitative Database
The qualitative database consists of a combination of (1) electronic and hard-copy text files (i.e., baseline and intermediate site visit reports for all 24 sites as well as two focus group transcripts for each of the 24 sites); (2) audiotapes for the six focus groups conducted during each site visit that were not transcribed; (3) site visitors' field notes about their own observations as well information gleaned from archival and written materials including local records, police or government reports, town meeting minutes, hospital records, newspapers, community fliers or posters, billboards, and human services agency publications; (4) primary source archival materials (e.g., newspaper clippings and town meeting minutes); and (5) processed analytic files (code reports) from NUD*IST 4.0 software.
Another category of data are the coding runs (or reports) composed of text units (from multiple reports or transcripts) that are associated with a given coding category. Any one of these reports may contain dozens or hundreds of text units that were used for analyzing community processes at the sites. Such reports are used for the qualitative content analysis, for which text was organized according to a hierarchical code structure that included all major content and thematic areas.
Focus Group Transcripts
Upon completion of each site visit, tape-recordings of the focus groups were transcribed to obtain a verbatim record of the focus group discussion. The time spent transcribing was dependent upon such factors as (1) the quality of the tape and the clarity of the participants' voices; (2) the skill, experience, and typing speed of the transcriber; (3) distracting background noises on the tapes; (4) participants' use of slang or hard-to-understand accents; (5) the number of focus group participants (i.e., 5 different voices are easier to track than 12); and (6) whether the session was conducted in English or in Spanish.
The baseline transcripts included in the analysis of data for this report consisted of 24 focus group transcripts (two per site) from a group of 12 sites. Of the 24transcripts analyzed, 12 were drawn from 6 of the target sites and 12 were drawn from 6 of the comparison sites. There were 7 parent groups, 7 high school groups, 6 middle school groups, and 4 elementary school groups.
Site Visit Reports
All reports were written to a standardized topical outline. Separate outlines were developed for the baseline and intermediate reports. Written reports and electronic reports then were used for analysis. Electronic copies of reports were prepared for NUD*IST software analysis by following specified formatting requirements (e.g., inserting headers, keeping lines to no more than 72characters, and breaking paragraphs into text units).
Data Analysis Because of the volume of data collected on site, it was imperative that staff conduct preliminary data analyses at the time of site visit. For example, during site visits, site visit team members met daily to discuss the day's proceedings, write or type up notes, and formulate tentative analyses or hypotheses. Such analysis was done promptly at the time of data collection when the field experience was still fresh.
As for focus groups, the moderator and notetaker spent 15-30 min. debriefing immediately after the conclusion of each focus group and (1) writing up preliminary analyses; (2) identifying key issues, notable quotations, first impressions, trends, patterns, or problems that occurred during the focus group; (3) noting nonverbal activity that occurred; (4) comparing/contrasting the focus group to previous focus groups; (5) noting unexpected or unanticipated findings; and (6) determining if any of the questions needed to be revised or eliminated. This manner of debriefing was essential for informing later interpretation of the focus group data.
In addition, portions of the qualitative database that included any identifiers were stripped of information that might jeopardize the confidentiality of community respondents such as key informants or focus group participants. For example, a phrase such as "Detective John Smith, head of the Denver Police Department drug unit," would be changed to "a Denver law enforcement official."
Software
A qualitative analysis software package called NUD*IST (Version 4.0) was used to analyze data collected through site visits. NUD*IST (Non-Numerical Unstructured Data Indexing Searching and Theorizing) aids the process of sorting, aggregating, storing, and analyzing text data. Data were typed in Microsoft Word prior to being imported into NUD*IST.
An advantage of NUD*IST is that its nodes (i.e., code categories) can be referenced to specific texts or text units without having to do any laborious data entry (of codes) or cutting and pasting or time-consuming reaggregating of texts. Rather, NUD*IST accomplishes coding by simply referencing a particular node to a specific text unit.
Coding
The coding of qualitative data for the present report involved a series of steps which are summarized below.
Step 1: Create a logical, hierarchical code/node structure -- A structure was developed that encompassed all of the major content and thematic areas for the data analysis (see Exhibit A-6, Node Structure for Analysis of Qualitative Text Data). Under this structure, each separate content or thematic area had its own unique, numbered place in the hierarchy. For example, "Explicit Anti-Drug Interventions" had the identifying number 3.2 and was a subset of "Community Drug Context," which had the number 3. The purpose of this code structure was to create unique topical/thematic niches where text units from the analyzed documents (baseline focus group transcripts and baseline and intermediate site visit reports) were coded and aggregated.
Step 2: Define all codes/nodes -- Definitions of all codes were developed so that all analysts would have the same understanding of the meaning of each code. This allowed for a consistent and standardized approach to interpreting the text documents.
Step 3: Make analytic assignments -- Research analysts were assigned pairs of target and comparison sites along with the reports and transcripts from those sites. Analysts were not assigned to analyze data for sites where they had been site visitors to avoid introducing any bias into their analysis.
Step 4: Inter-coder reliability --
All analysts were assigned the same series of nine pages --
three pages from a baseline report, three from a baseline transcript, and three from an intermediate report. Each analyst was asked to use the Hierarchical Code Structure to code all relevant text. The analysts then met to compare coding results, discuss discrepancies, and find ways to improve the process.
Step 5: Coding of hard copy --
All site visit reports (24 baseline and 24intermediate) and a sampling of focus group transcripts (24 baseline) were distributed to their respective analysts for coding. The team met twice a week to monitor the process. Analysts recorded on the hard copies (a) how too-long paragraphs should be broken up into shorter paragraphs (i.e., text units) and (b) how text units should be coded (e.g., the code number, such as "3.1.1," was placed in the right-hand margin adjacent to the relevant text unit). Single text units could be assigned more than one code, if applicable. Group meetings allowed discussions of which text units were being assigned multiple codes and which codes were most relevant.
Step 6: Transferral of coding to NUD*IST files --
After all site visit reports and transcript files were imported into NUD*IST, three analysts were assigned the task of accessing NUD*IST and coding text units with the appropriate code(s) included in the node structure. Exhibit A-7 provides a summary of the number of documents and text units that were coded to various nodes.
Step 7: Generating code reports --
Once the process of coding text units was completed, coding reports were generated. These reports consisted of aggregated collections of all the text units from all reports and transcripts that had been coded at a particular node. These reports --
from 2 to 40 pages long --
were printed out and served as the principal database for analysis.
Step 8: Content analysis/writing --
Analysts were assigned to write different sections of the intermediate report, and each section was tied specifically to certain codes that comprised the node structure. Unique code reports (and often times several different code reports) were used for writing each of the report Sections. Thus, analysis of the contents of one or more code reports generated the narrative that formed a specific section of the intermediate report. Analysts reviewed the content analysis of each of the major sections and provided feedback on conclusions drawn from the data.
