NCASAA Evaluation Project

TABLE OF CONTENTS
Page
NCASAA EVALUATION PROJECT
1. INTRODUCTION ...............................................................................................1
2. RESEARCH QUESTIONS..................................................................................2
2.1 Volunteer Characteristics, Training, and Activities ...................................2
2.2 Services, Permanency Plans, and Placement .............................................2
2.3 Comparisons Between Children With and Without CASAs ......................3
3. DATA COLLECTION.........................................................................................3
4. PRELIMINARY FINDINGS ...............................................................................6
4.1 Children’s Demographic Characteristics ...................................................6
4.2 CASA Volunteers.....................................................................................8
4.3 CASA Assignment and Activities.............................................................8
5. NEXT STEPS....................................................................................................11
Caliber Associates 1
NCASAA EVALUATION PROJECT
1. INTRODUCTION
Each year child welfare agencies investigate between 2.5 and 3 million allegations of
child abuse/and or neglect, of which cases will be substantiated for one third of children.
Working with these children are 70,000 Court Appointed Special Advocate (CASAs) who help
children as they progress through the child welfare system and represent their best interests in
court. The current study represents an exciting opportunity to examine the short and longer-term
impacts of CASAs on children and families in contact with the child welfare system and to
expand knowledge of CASA programs and services. With the support of the Packard
Foundation, the National CASA Association (NCASAA) and Caliber Associates will combine
data collected through NCASAA’s management information system (COMET) with data
collected through the National Survey of Child and Adolescent Well-being (NSCAW), a
federally sponsored national survey of children and their families.
Sponsored by the Children’s Bureau, NSCAW is collecting data from 5,500 children,
parents, caregivers, caseworkers, and teachers over a four-year period that began in 1999. These
data will provide information on the characteristics and backgrounds of children and families in
contact with the child welfare system, the services they receive, and the short- and longer-term
outcomes of these services. NSCAW is the first national, longitudinal study of this scope and
will contribute important information to inform child welfare policies and practice.
Combining COMET and NSCAW data will allow us to examine impacts and outcomes
of CASA representation on key child welfare outcomes, including:
Child and family functioning
Types of placements and services children experience in the child welfare system,
including timing and appropriateness
Placements and services for specific sub-groups of children (e.g., toddlers, children of
color).
Combining the two data sets will allow analysis of CASA effects overall and by age, race,
placement type, history of child welfare system involvement, and other factors. The merging of
these data will occur in late August when the NSCAW data, which is being collected and cleaned
at Research Triangle Institute in North Carolina, become available. In this interim report, we
present the research questions the final report will address, the data collection process to date,
and preliminary findings based on COMET data submitted in the summer of 2002.
NCASAA Evaluation Project
The measure of excellence 2
2. RESEARCH QUESTIONS
Merging COMET and NSCAW data will allow us to answer a number of questions about
CASA volunteers and the children they serve. In addition to providing descriptive statistics on
children, volunteers, case activities, trainings, and court activities, we will also compare
characteristics of and outcomes for children who have and do not have CASAs. Below are some
of the questions this study seeks to answer.
2.1 Volunteer Characteristics, Training, and Activities
What is the relationship between volunteer characteristics (e.g., education,
employment status) and the types of activities CASAs engage in during the course of
cases?
What types of training do volunteers receive? How much training do volunteers
receive?
Is there are a relationship between CASA training and child or case outcomes?
How long after the court orders a CASA assigned does the CASA program assign a
volunteer to the case?
What is the average number of hours CASAs spend per case per month?
How does the number of hours spent vary by characteristics of the case (e.g., length
of time in placement, number of previous placements, type of abuse, age of the
child)?
2.2 Services, Permanency Plans, and Placement
What types of services are recommended for children?
How do service recommendations vary by child or volunteer demographics, type of
abuse, volunteer training, volunteer activities, and court activities?
To what extent are services completed as ordered?
How does the amount of time spent per case per month differ in cases in which
services are and are not completed?
What types of permanency plans are developed and ordered for children? How do
these plans vary by type of abuse, number of prior placements, child’s age, child’s
race?
To what extent are CASAs’ recommendations accepted by the court?
NCASAA Evaluation Project
Caliber Associates 3
Does the number of CASA recommendations accepted differ based on the number of
activities per case, the hours spent on the case, or the training of the volunteer?
How does the final placement of a child differ based on characteristics of the case
(e.g., child’s age, type of abuse, race, number of previous placements), number of
volunteer hours spent, percent of volunteer recommendations accepted?
2.3 Comparisons Between Children With and Without CASAs
What differences are there between children with and without a CASA (e.g.,
demographics, type of abuse, number of prior placements, number/type of risk and
protective factors)?
What differences are there in the number or type of services received by children with
and without CASAs?
What differences are there between children with and without CASAs in the extent to
which services needed are actually received?
How do children with and without CASAs compare on developmental measures over
time?
3. DATA COLLECTION
The data needed to answer these research questions will come from the merger of
COMET and NSCAW data sets. NSCAW data are being collected on a cohort of approximately
5,500 children who came in contact with the child welfare system between October 1999 and the
end of 2000. Children in the NSCAW sample represent 100 sites across the country. Data were
gathered on child and caregiver characteristics, developmental issues, risk factors, and services
received. Data collection took place in several waves: within a few weeks of a CPS
investigations (wave 1), at 12 months following the investigation (wave 2), and at 18 months
following the investigation (wave 3). A final wave of data collection will take place at 36
months following the investigation. The final report for this study will be based on analyses of
data collected during the first 3 data collection periods. Because NSCAW data will not be
available until later this summer, this interim report relies on COMET data submitted to date.
Through COMET, CASA programs are providing both program-level and case-level
data, including demographic characteristics of CASAs, training and experience, specific
activities and services provided to the child, and court events. Data are being collected from
approximately 25 CASA programs across the United States and sent electronically to Caliber
Associates. A portion of the 25 CASA programs submitting COMET data are also NSCAW
NCASAA Evaluation Project
The measure of excellence 4
sites. In the sites in which there is overlap, COMET data will be linked with NSCAW data using
unique child identifiers.
COMET data collection has occurred in several rounds. Most recently, 21 sites
submitted data in June and July of 2003. Of these, 17 sites also submitted data in 2002. The
final COMET data set will include the most recent available data for each site. Having sites
submit data in rounds allows Caliber staff to monitor data quality and reflects an ongoing effort
on the part of the NCASAA to work with sites to improve the completeness of their data.
Data from the sites are extracted from the COMET system using an extraction program,
and the resulting files are posted on a Web site, from which they are downloaded. The complete
data set from each site includes ten distinct files that Caliber staff merge into a single file,
organized by child. Once the merge process has been completed for each site, the individual site
files are merged to create the COMET database. Preliminary findings presented in this report are
based on analysis of the 2002 COMET data that includes information on 2,354 children from 17
CASA sites.
Creation of the database required development of syntax or command language that
facilitated organizing, by child, each site’s multiple data files. Some files contained one record
for each child, but other files were organized by variables such as volunteer activities. In this
case, files that included information about a CASA’s activities on a case needed to be linked
with files that contained information on the demographics of the child s/he was serving. Using
syntax, one record was created for each child that contained all relevant information on the
child’s demographic characteristics, case activities, the characteristics of the volunteer assigned
to the child, and other information specific to that child’s case.
One of the challenges in merging the data from individual CASA sites was the wide
range of values used for key variables. Because for a number of variables sites did not have a
predetermined list of categories from which to choose, they had wide latitude to define and enter
information. For example, in the CASA activities field in the COMET database, sites reported
up to 30 different types of CASA activities. These variations necessitated consolidating
categories and creating new values for some variables.
In consultation with NCASAA, the Caliber team created new values for initial allegation
(abuse type), language, employment, race/ethnicity (both child and CASA volunteer), location of
final placement, perpetrator relationship, services ordered for the child, training topic, type of
permanency plan, education, and CASA activities. Values were selected based on review of data
submitted by a sample of eight sites in the summer of 2002. Those values that were most
common across sites were selected. Syntax was used to recode values into consistent categories
NCASAA Evaluation Project
Caliber Associates 5
that were common across sites. However, for some variables, a number of the values could not
be fit into one of the categories and had to be assigned a value of “other.” For example, ‘CASA
sworn in’ and ‘PR/Fundraising event’ were entries in the CASA activities field that were recoded
as “other.” Because each site used different categories or values for each of these 11 variables,
syntax tailored to each site had to be developed.
Throughout the data submission process, data are checked for accuracy. Child and
volunteer identification numbers are checked to ensure that there are no duplicates. When
duplicates are identified, sites are asked to resubmit their data. Other data checks include reports
on the recoding of the 11 variables discussed above, which allows identification of any errors in
the recoding process. In addition to this check, the syntax for each site generates a report for
each of the eleven variables that are recoded. This ensures that each value has been coded into
one of the specified categories.
In June of 2003, sites were asked to provide data on cases through May of 2003. These
cases represent those children who were assigned a CASA between October 1999 and December
2000, the same timeframe being used for the NSCAW sample. Exhibit 1 below lists the status of
sites’ data submissions as of July 10, 2003. We anticipate that there may be additional sites
submitting data.
EXHIBIT 1
COMET DATA SUBMISSIONS
# Site (City, State) August 2002 Submission May 2003 Submission
1 Akron, Ohio Yes No
2 Birmingham, Alabama Yes No
3 Dallas, Texas Yes Yes
4 Davenport, Iowa Yes Yes
5 Decatur, Georgia Yes Yes
6 Des Moines, Iowa Yes Yes
7 Farmington, New Mexico Yes Yes
8 Greenville, South Carolina Yes Yes
9 Lafayette, Louisiana Yes No
10 Oregon City, Oregon Yes Yes
11 Roanoke, Virginia Yes Yes
12 Rochester, New York Yes No
13 Salt Lake City, Utah Yes Yes
14 Sioux Falls, South Dakota Yes Yes
15 St. Louis, Missouri Yes No
16 Wenatchee, Washington Yes No
17 Westbath, Maine Yes Yes
18 Fremont, Nebraska No Yes
19 Grand Rapids, Michigan No Yes
20 Topeka, Kansas No Yes
21 Monck’s Corner, South Carolina No Yes
NCASAA Evaluation Project
The measure of excellence 6
During the next four to six weeks, remaining submissions will be processed and the final
database will be checked to make sure that the data are as complete and accurate as possible. In
August 2003, the third wave of the NSCAW data will be finalized and will be ready to match
with the COMET data. At that time, the COMET data file will be sent to RTI to be matched
with NSCAW data. While the match is occurring at RTI, analysis of the merged data set will be
conducted by Caliber staff in preparation for the final report in November 2003.
4. PRELIMINARY FINDINGS
Several preliminary analyses were conducted with the COMET data submitted in 2002 by
17 CASA sites (n=2,354 children). The purpose of these analyses was to provide initial
descriptive statistics on the child, volunteers, type of maltreatment, and CASA activities and to
assess the quality and completeness of the sites’ data submissions. Only cases with complete
data on the variables of interest were included in the preliminary findings presented below.
4.1 Children’s Demographic Characteristics
Gender and Age
Male and female children are represented in the COMET sample in nearly equal
proportions. Fifty-one percent of children are male and 49 percent are female. The average age
of children in the sample is 6.6 years of age (SD 5.1). Of the 2,300 children with age reported,
children under one year old comprise approximately 15 percent of the sample, children aged one
to five account for 32 percent, children aged six to eleven account for 35 percent, and children
over 11 years old account for 18 percent (Exhibit 2).
EXHIBIT 2
PERCENTAGE DISTRIBUTION OF CHILDREN BY AGE
18 15
32
35
0
10
20
30
40
50
60
70
80
90
100
<1 Year 1 to 5 Years 6 to 11 Years 12 and Older
Age Group
Percentage
NCASAA Evaluation Project
Caliber Associates 7
Race/Ethnicity
As Exhibit 3 shows, the majority of children in the sample are White (44%), followed by
African American (30%), bi-racial (6%), Hispanic/Latino (4%), and American Indian/Alaska
Native (3%). Thirteen percent of children in the sample are included in the ‘other’ category.
The size of the ‘other’ category reflects the lack of consistency across sites in the way race was
reported. The only two race categories common to all sites were African American and White.
EXHIBIT 3
PROPORTION OF CHILDREN BY RACE
Caucasian
44%
African
American
30%
American
Indian/
Alaska Native
3%
Hispanic/
Latino
4%
Biracial
6%
Other
13%
Child Maltreatment Type
As shown in Exhibit 4, neglect is the most common type of maltreatment experienced by
children in the COMET sample (71%). Eighteen percent of children experienced physical abuse,
5 percent experienced sexual abuse, and 3 percent experienced emotional maltreatment. A large
number of children were victims of ‘other’ types of abuse (27%). This category includes abuse
types that could not be clearly categorized as neglect, physical abuse, sexual abuse, or emotional
maltreatment due to inconsistencies in reporting across sites. It should be noted that 5 percent of
the children in the sample had petitions alleging two or more types of abuse, resulting in
percentages adding to over 100 percent.
NCASAA Evaluation Project
The measure of excellence 8
EXHIBIT 4
PROPORTION OF CHILDREN BY MALTREATMENT TYPE (N=1,758)
18%
5% 3%
27%
71%
0%
20%
40%
60%
80%
100%
Neglect Physical Abuse Sexual Abuse Emotional
Maltreatment
Other
Maltreatment Type
Proportion of All Children
t
4.2 CASA Volunteers
The volunteers assigned to children in the COMET sample are a homogenous group in
terms of their sex, race, and language, but differ with respect to education and employment
status.
Volunteer Sex and Race
Approximately 80 percent of volunteers are female and 89 percent are White. One
percent of those who reported their language indicated that they speak a language other than
English.
Volunteer Employment Status and Education
Volunteers varied in their employment status and education. More than half of the
volunteers (63%) worked in addition to their responsibilities as a CASA. One in twenty (5%)
were students, and more than one in four (29%) were not employed or were retired. Almost all
(99%) volunteers had at least a high school education or the equivalent. Approximately 89
percent of volunteers had some college, had completed college, or had a graduate degree.
4.3 CASA Assignment and Activities
The first step in a CASA’s involvement with a child’s case is assignment of the CASA by
a judge. In this sample, on average, the judicial assignment of a CASA was made three months
after the petition alleging abuse or neglect was filed. An average of five days elapsed between
NCASAA Evaluation Project
Caliber Associates 9
the time the court assigned a child to the CASA program and the time the CASA program
assigned a volunteer to the case.
In this sample, CASAs were found to spend an average of 2.1 hours to 5 hours per month
on each child’s case (Exhibit 5). Volunteer time is divided among several types of activities,
including contact with the CASA supervisor/coordinator, the child’s caseworker, foster parents,
relatives, mother or father, teachers, other service providers, and the child. Other activities for
which amount of time spent was reported include report writing, travel, and attendance at court
proceedings.
EXHIBIT 5
AVERAGE NUMBER OF HOURS CASA VOLUNTEERS SPENT ON ALL
ACTIVITIES PER MONTH PER CHILD
(N=596 CLOSED CASES)
0
1
2
3
4
5
6
7
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
Number of Months from January 2000 to July 2002
Average Number of Hours per Child
One factor that may influence the amount of time spent on a case is the age of the child
being served. After dividing the children into groups based on age (<1, 1 to 5, 6 to 11, and 12
and older) we calculated the average amount of time spent on all recorded activities per month.
The results indicate that the amount of time spent per month varies somewhat by the age of the
child being served. The youngest children, those less than a year old, had the highest average
number of hours spent per case per month (3.1 hours). Those children ages 1 to 5 had a slightly
lower average (2.9 hours), while those 6 to 11 had a still lower average of 2.6 hours per month
spent on all activities. For those children 12 and older, the average was 2.9 hours per case per
month. None of these differences were, however, statistically significant.
NCASAA Evaluation Project
The measure of excellence 10
EXHIBIT 6
MEAN NUMBER OF HOURS SPENT ON CASES PER MONTH BY AGE
GROUP
(N=596)
3.1
2.9
2.6
2.9
0
1
2
3
4
<1 1 to 5 6 to 11 12 and older
Age Group
Mean Number of Hours
Across sites there were a fair number of specific types of activities categorized as “other”
due to inconsistencies in reporting. In order to get a better understanding of how volunteers’
time was spent, we selected a site that had few activities categorized as “other.” In this site, as
shown in Exhibit 7, the largest percentage on average of reported volunteer time was spent in
contact with the child (37%). Four additional categories each accounted, on average, for
13 percent of volunteers’ time on cases: time spent in court, contact with collaterals, report
writing, and contact with supervisor. Other unspecified activities accounted for 11 percent of
recorded activity on cases, and slightly less than 1 percent of recorded activity involved contact
with the child’s biological parents.
NCASAA Evaluation Project
Caliber Associates 11
EXHIBIT 7
AVERAGE PERCENTAGE OF TIME SPENT ON SELECTED
ACTIVITIES FOR CLOSED CASES
(ST. LOUIS, MO N=104)
Biological Parents
1%
Report Writing
13%
In Court
13%
Collaterals
13%
Supervisor
13%
Other Activities
11%
With Child
37%
CASAs can be instrumental in helping children get the services they need. We explored
the types of services that had been ordered and found that the most frequently reported services
ordered for children in the COMET sample include counseling, education, and medical services.
Among these services, counseling was the service ordered most frequently across all types of
maltreatment cases. As Exhibit 8 shows, counseling was ordered for 14 percent of children
reported for neglect, 13 percent of children who had been physically abused, and 23 percent of
children suffering from multiple types of maltreatment.
NCASAA Evaluation Project
EXHIBIT 8
SERVICES ORDERED BY TYPE OF MALTREATMENT
0
20
40
60
80
100
Neglect (n=921) Physical (n=111) Multiple (n=373)
Type of Maltreatment
Percentage
Counseling Education Medical
5. NEXT STEPS
During the next 6-8 weeks the Caliber team will continue to clean and add to the COMET
database. As soon as the NSCAW data from waves 1-3 are finalized, we will send the COMET
files to Research Triangle Institute to be merged with the NSCAW data. Analysis of the data
will take place in September and October, and the final report will be delivered in November
2003.