View in EDS

Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints

Saved in:
Bibliographic Details
Title: Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints
Language: English
Authors: Bamberger, Michael, Rugh, Jim, Church, Mary
Source: American Journal of Evaluation. 2004 25(1):5-37.
Availability: Sage Publications, 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243 (Toll Free); Fax: 800-583-2665 (Toll Free).
Peer Reviewed: Y
Page Count: 33
Publication Date: 2004
Document Type: Journal Articles
Reports - Descriptive
Descriptors: Foreign Countries, Program Effectiveness, Evaluators, Control Groups, Program Evaluation, Evaluation Methods, Case Studies, Data Collection, Budgeting, Time
DOI: 10.1177/109821400402500102
ISSN: 1098-2140
Abstract: The paper discusses two common scenarios in which evaluators must conduct impact evaluations when working under budget, time, or data constraints. Under the first scenario the evaluator is not called in until the project is already well advanced, and there is a tight deadline for completing the evaluation, frequently combined with a limited budget and without access to baseline data. Under the second scenario the evaluator is called in early, but for budget, political, or methodological reasons it is not possible to collect baseline data on a control group and sometimes not even on the project population. As a result of these constraints, many of the basic principles of impact evaluation design (comparable pretest-posttest design, control group, instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology, etc.) are often sacrificed. We describe the "Shoestring Evaluation" approach which is being developed to provide tools for ensuring the highest quality evaluation possible under constraints of limited budget, time and data availability. While most of the data collection and analysis techniques will be familiar to experienced evaluators, what is new is the combination of these techniques into an integrated six-step approach which covers: (1) planning and scoping the evaluation, (2-4) options for dealing with constraints related to costs, time and data availability (which could include reconstructing baseline conditions and control groups), (5) identifying the strengths and weaknesses (threats to validity and adequacy) of the evaluation design, and (6) taking measures to address the threats and strengthen the evaluation design and conclusions. When necessary, many of these corrective measures can be introduced at a very late stage, even when the draft evaluation report has already been produced. Although the approach was designed to assist evaluators working in developing countries, the principles are equally applicable for conducting evaluations under similar constraints in industrial nations. Case studies are included to illustrate the different Shoestring approaches, drawing from the authors' experience in Eastern Europe, Africa, Asia, and Latin America.
Abstractor: Author
Number of References: 42
Entry Date: 2005
Accession Number: EJ689219
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ689219
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Bamberger%2C+Michael%22">Bamberger, Michael</searchLink><br /><searchLink fieldCode="AR" term="%22Rugh%2C+Jim%22">Rugh, Jim</searchLink><br /><searchLink fieldCode="AR" term="%22Church%2C+Mary%22">Church, Mary</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22American+Journal+of+Evaluation%22"><i>American Journal of Evaluation</i></searchLink>. 2004 25(1):5-37.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Sage Publications, 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243 (Toll Free); Fax: 800-583-2665 (Toll Free).
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 33
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2004
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Control+Groups%22">Control Groups</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Case+Studies%22">Case Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Collection%22">Data Collection</searchLink><br /><searchLink fieldCode="DE" term="%22Budgeting%22">Budgeting</searchLink><br /><searchLink fieldCode="DE" term="%22Time%22">Time</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/109821400402500102
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1098-2140
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The paper discusses two common scenarios in which evaluators must conduct impact evaluations when working under budget, time, or data constraints. Under the first scenario the evaluator is not called in until the project is already well advanced, and there is a tight deadline for completing the evaluation, frequently combined with a limited budget and without access to baseline data. Under the second scenario the evaluator is called in early, but for budget, political, or methodological reasons it is not possible to collect baseline data on a control group and sometimes not even on the project population. As a result of these constraints, many of the basic principles of impact evaluation design (comparable pretest-posttest design, control group, instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology, etc.) are often sacrificed. We describe the "Shoestring Evaluation" approach which is being developed to provide tools for ensuring the highest quality evaluation possible under constraints of limited budget, time and data availability. While most of the data collection and analysis techniques will be familiar to experienced evaluators, what is new is the combination of these techniques into an integrated six-step approach which covers: (1) planning and scoping the evaluation, (2-4) options for dealing with constraints related to costs, time and data availability (which could include reconstructing baseline conditions and control groups), (5) identifying the strengths and weaknesses (threats to validity and adequacy) of the evaluation design, and (6) taking measures to address the threats and strengthen the evaluation design and conclusions. When necessary, many of these corrective measures can be introduced at a very late stage, even when the draft evaluation report has already been produced. Although the approach was designed to assist evaluators working in developing countries, the principles are equally applicable for conducting evaluations under similar constraints in industrial nations. Case studies are included to illustrate the different Shoestring approaches, drawing from the authors' experience in Eastern Europe, Africa, Asia, and Latin America.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: Author
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 42
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2005
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ689219
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ689219
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/109821400402500102
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 33
        StartPage: 5
    Subjects:
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Control Groups
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Case Studies
        Type: general
      – SubjectFull: Data Collection
        Type: general
      – SubjectFull: Budgeting
        Type: general
      – SubjectFull: Time
        Type: general
    Titles:
      – TitleFull: Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Bamberger, Michael
      – PersonEntity:
          Name:
            NameFull: Rugh, Jim
      – PersonEntity:
          Name:
            NameFull: Church, Mary
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2004
          Identifiers:
            – Type: issn-print
              Value: 1098-2140
          Numbering:
            – Type: volume
              Value: 25
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: American Journal of Evaluation
              Type: main
ResultId 1