case study protocol pdf

To read this content please select one of the options below:

Please note you do not have access to teaching notes, developing a robust case study protocol.

Management Research Review

ISSN : 2040-8269

Article publication date: 4 July 2023

Issue publication date: 11 January 2024

Case study research has been applied across numerous fields and provides an established methodology for exploring and understanding various research contexts. This paper aims to aid in developing methodological rigor by investigating the approaches of establishing validity and reliability.

Design/methodology/approach

Based on a systematic review of relevant literature, this paper catalogs the use of validity and reliability measures within academic publications between 2008 and 2018. The review analyzes case study research across 15 peer-reviewed journals (total of 1,372 articles) and highlights the application of validity and reliability measures.

The evidence of the systematic literature review suggests that validity measures appear well established and widely reported within case study–based research articles. However, measures and test procedures related to research reliability appear underrepresented within analyzed articles.

Originality/value

As shown by the presented results, there is a need for more significant reporting of the procedures used related to research reliability. Toward this, the features of a robust case study protocol are defined and discussed.

• Case study research
• Systematic literature review

Burnard, K.J. (2024), "Developing a robust case study protocol", Management Research Review , Vol. 47 No. 2, pp. 204-225. https://doi.org/10.1108/MRR-11-2021-0821

Emerald Publishing Limited

Copyright © 2023, Emerald Publishing Limited

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

How to Write a Study Protocol

• First Online: 02 February 2019

Cite this chapter

• Lukas B. Moser 8 , 9 &
• Michael T. Hirschmann 8 , 9  

2396 Accesses

This chapter aims to provide a guide for young trainees writing their first study protocol. It includes important aspects junior researchers should consider before getting started and preparing their first study protocol. After having read the chapter, the reader should have a good idea about what a study protocol is about and be able to answer the question why, when, and how a study protocol should be written. Finally, the reader will be prepared to master the very first step of conducting a successful study—writing a brief, concise, but comprehensive study protocol.

Study protocol examples of typical clinical scenarios further illustrate the approach to this mandatory and important part of a research project.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Al-Jundi A, Sakka S. Protocol writing in clinical research. J Clin Diagn Res. 2016;10:ZE10–3.

PubMed   PubMed Central   Google Scholar  

Babyak MA. What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med. 2004;66:411–21.

PubMed   Google Scholar  

Bando K, Sato T. Did you write a protocol before starting your project? Gen Thorac Cardiovasc Surg. 2015;63:71–7.

Article   Google Scholar  

Barletta JF. Conducting a successful residency research project. Am J Pharm Educ. 2008;72:92.

Brody BA, McCullough LB, Sharp RR. Consensus and controversy in clinical research ethics. JAMA. 2005;294:1411–4.

Article   CAS   Google Scholar  

Doran G. There’s a S.M.A.R.T. way to write management’s goals and objectives. Manag Rev. 1981;70:35–6.

Google Scholar  

Doran PC. Understanding dentures: instructions to patients. J Colo Dent Assoc. 1981;59:4–8.

CAS   PubMed   Google Scholar  

Fleming TR, DeMets DL. Surrogate end points in clinical trials: are we being misled? Ann Intern Med. 1996;125:605–13.

Fronteira I. How to design a (good) epidemiological observational study: epidemiological research protocol at a glance. Acta Medica Port. 2013;26:731–6.

Ghooi RB, Divekar D. Insurance in clinical research. Perspect Clin Res. 2014;5:145–50.

Gilmore SJ. Evaluating statistics in clinical trials: making the unintelligible intelligible. Australas J Dermatol. 2008;49:177–84; quiz 185–176.

Huang Y, Gilbert PB. Comparing biomarkers as principal surrogate endpoints. Biometrics. 2011;67:1442–51.

Kanji S. Turning your research idea into a proposal worth funding. Can J Hosp Pharm. 2015;68:458–64.

O’Brien K, Wright J. How to write a protocol. J Orthod. 2002;29:58–61.

Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995;123:A12–3.

Rosenthal R, Schafer J, Briel M, Bucher HC, Oertli D, Dell-Kuster S. How to write a surgical clinical research protocol: literature review and practical guide. Am J Surg. 2014;207:299–312.

Sade RM, Akins CW, Weisel RD. Managing conflicts of interest. J Thorac Cardiovasc Surg. 2015;149:971–2.

Skelly AC, Dettori JR, Brodt ED. Assessing bias: the importance of considering confounding. Evid Based Spine Care J. 2012;3:9–12.

Van Spall HG, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007;297:1233–40.

Vandenbroucke JP, von Elm E, Altman DG, Gotzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med. 2007;4:e297.

Vavken P, Culen G, Dorotka R. Management of confounding in controlled orthopaedic trials: a cross-sectional study. Clin Orthop Relat Res. 2008;466:985–9.

Warren MD. Aide-memoire for preparing a protocol. Br Med J. 1978;1:1195–6.

Download references

Author information

Authors and affiliations.

Department of Orthopaedic Surgery and Traumatology, Kantonsspital Baselland (Bruderholz, Liestal, Laufen), Bruderholz, Switzerland

Lukas B. Moser & Michael T. Hirschmann

University of Basel, Basel, Switzerland

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Michael T. Hirschmann .

Editor information

Editors and affiliations.

UPMC Rooney Sports Complex, University of Pittsburgh, Pittsburgh, PA, USA

Volker Musahl

Department of Orthopaedics, Sahlgrenska Academy, Gothenburg University, Sahlgrenska University Hospital, Gothenburg, Sweden

Jón Karlsson

Department of Orthopaedic Surgery and Traumatology, Kantonsspital Baselland (Bruderholz, Laufen und Liestal), Bruderholz, Switzerland

Michael T. Hirschmann

McMaster University, Hamilton, ON, Canada

Olufemi R. Ayeni

Hospital for Special Surgery, New York, NY, USA

Robert G. Marx

Department of Orthopaedic Surgery, NorthShore University HealthSystem, Evanston, IL, USA

Jason L. Koh

Institute for Medical Science in Sports, Osaka Health Science University, Osaka, Japan

Norimasa Nakamura

Rights and permissions

Reprints and permissions

Copyright information

© 2019 ISAKOS

About this chapter

Moser, L.B., Hirschmann, M.T. (2019). How to Write a Study Protocol. In: Musahl, V., et al. Basic Methods Handbook for Clinical Orthopaedic Research. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-58254-1_8

Download citation

DOI : https://doi.org/10.1007/978-3-662-58254-1_8

Published : 02 February 2019

Publisher Name : Springer, Berlin, Heidelberg

Print ISBN : 978-3-662-58253-4

Online ISBN : 978-3-662-58254-1

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Search Menu
• Author Guidelines
• Submission Site
• Open Access Options
• Self-Archiving Policy
• Reasons to Submit
• About Journal of Surgical Protocols and Research Methodologies
• Editorial Board
• Advertising & Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, contents of a research study protocol, conflict of interest statement, how to write a research study protocol.

• Article contents
• Figures & tables
• Supplementary Data

Julien Al Shakarchi, How to write a research study protocol, Journal of Surgical Protocols and Research Methodologies , Volume 2022, Issue 1, January 2022, snab008, https://doi.org/10.1093/jsprm/snab008

• Permissions Icon Permissions

A study protocol is an important document that specifies the research plan for a clinical study. Many funders such as the NHS Health Research Authority encourage researchers to publish their study protocols to create a record of the methodology and reduce duplication of research effort. In this paper, we will describe how to write a research study protocol.

A study protocol is an essential part of a research project. It describes the study in detail to allow all members of the team to know and adhere to the steps of the methodology. Most funders, such as the NHS Health Research Authority in the United Kingdom, encourage researchers to publish their study protocols to create a record of the methodology, help with publication of the study and reduce duplication of research effort. In this paper, we will explain how to write a research protocol by describing what should be included.

Introduction

The introduction is vital in setting the need for the planned research and the context of the current evidence. It should be supported by a background to the topic with appropriate references to the literature. A thorough review of the available evidence is expected to document the need for the planned research. This should be followed by a brief description of the study and the target population. A clear explanation for the rationale of the project is also expected to describe the research question and justify the need of the study.

Methods and analysis

A suitable study design and methodology should be chosen to reflect the aims of the research. This section should explain the study design: single centre or multicentre, retrospective or prospective, controlled or uncontrolled, randomised or not, and observational or experimental. Efforts should be made to explain why that particular design has been chosen. The studied population should be clearly defined with inclusion and exclusion criteria. These criteria will define the characteristics of the population the study is proposing to investigate and therefore outline the applicability to the reader. The size of the sample should be calculated with a power calculation if possible.

The protocol should describe the screening process about how, when and where patients will be recruited in the process. In the setting of a multicentre study, each participating unit should adhere to the same recruiting model or the differences should be described in the protocol. Informed consent must be obtained prior to any individual participating in the study. The protocol should fully describe the process of gaining informed consent that should include a patient information sheet and assessment of his or her capacity.

The intervention should be described in sufficient detail to allow an external individual or group to replicate the study. The differences in any changes of routine care should be explained. The primary and secondary outcomes should be clearly defined and an explanation of their clinical relevance is recommended. Data collection methods should be described in detail as well as where the data will be kept secured. Analysis of the data should be explained with clear statistical methods. There should also be plans on how any reported adverse events and other unintended effects of trial interventions or trial conduct will be reported, collected and managed.

Ethics and dissemination

A clear explanation of the risk and benefits to the participants should be included as well as addressing any specific ethical considerations. The protocol should clearly state the approvals the research has gained and the minimum expected would be ethical and local research approvals. For multicentre studies, the protocol should also include a statement of how the protocol is in line with requirements to gain approval to conduct the study at each proposed sites.

It is essential to comment on how personal information about potential and enrolled participants will be collected, shared and maintained in order to protect confidentiality. This part of the protocol should also state who owns the data arising from the study and for how long the data will be stored. It should explain that on completion of the study, the data will be analysed and a final study report will be written. We would advise to explain if there are any plans to notify the participants of the outcome of the study, either by provision of the publication or via another form of communication.

The authorship of any publication should have transparent and fair criteria, which should be described in this section of the protocol. By doing so, it will resolve any issues arising at the publication stage.

Funding statement

It is important to explain who are the sponsors and funders of the study. It should clarify the involvement and potential influence of any party. The sponsor is defined as the institution or organisation assuming overall responsibility for the study. Identification of the study sponsor provides transparency and accountability. The protocol should explicitly outline the roles and responsibilities of any funder(s) in study design, data analysis and interpretation, manuscript writing and dissemination of results. Any competing interests of the investigators should also be stated in this section.

A study protocol is an important document that specifies the research plan for a clinical study. It should be written in detail and researchers should aim to publish their study protocols as it is encouraged by many funders. The spirit 2013 statement provides a useful checklist on what should be included in a research protocol [ 1 ]. In this paper, we have explained a straightforward approach to writing a research study protocol.

None declared.

Chan   A-W , Tetzlaff   JM , Gøtzsche   PC , Altman   DG , Mann   H , Berlin   J , et al.    SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials . BMJ   2013 ; 346 : e7586 .

Google Scholar

• conflict of interest
• national health service (uk)

Email alerts

Citing articles via.

• Advertising and Corporate Services
• Journals Career Network
• JSPRM Twitter

Affiliations

• Online ISSN 2752-616X
• Copyright © 2024 Oxford University Press and JSCR Publishing Ltd
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Alternative Quality Management Systems for Highway Construction (2015)

Chapter: appendix c: case study protocol.

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

APPENDIX C: CASE STUDY PROTOCOL C.1 Overview of Case Study Background Information Delivering highway projects using alternative project delivery methods demands a shift in the traditional agency quality assurance (QA) and quality control (QC) programs to accommodate the faster pace of design and construction as well as the redistribution of responsibilities among project stakeholders. Thus, the objectives of this research are to:  Identify and understand alternative quality management systems  Develop guidelines for their use in highway construction projects Alternative project delivery in highway construction often requires the application of alternative quality management systems that emphasize contractor quality control and quality assurance. These new systems allow owners to have confidence through a verification of contractor quality system processes. They also permit state transportation agencies (STAs) to satisfy due diligence requirements for federal- aid highway projects. For example, the International Organization for Standardization (ISO) 9001 quality management system regulates quality management at all levels from material suppliers through the contractors to owners. It requires a formal project performance evaluation after completion and uses that information to publish contractor performance ratings, which can then be used for future contractor prequalification. The U. S. Army Corps of Engineers’ quality management system relies on detailed guide specifications and rigorous on-site testing by contractors. The Corps has used alternative project delivery on a routine basis for over thirty years on a wide variety of heavy civil projects that include roads and bridges, and as a result, furnishes an excellent analog from which to draw lessons learned and best practices that apply to highway design and construction. Research is needed to provide guidance on the use of alternative quality management systems for highway construction projects using alternative delivery methods. This research needs to address the major quality issues associated with these methods including accelerated project timelines and the change of the designer-of-record’s (DOR) contractual relationships with the owner to permit the required level of integration with the construction contractor. Issues of quality are further complicated by the addition of private funding of public projects in public-private partnerships (PPP). On these projects, an argument can be made that since the concessionaire is at risk for project performance the public agency has few if any reasons to involve itself in the quality management process. From current and past projects, there exists a limited but rapidly expanding body of experience associated with alternate methods of assuring quality. The purpose of this research is to bring together this relatively new body of experience and summarize it in one easily accessible reference treating the subject of QA in alternative projects. The case studies for this research will be used to learn how existing projects have managed quality in light of non-traditional delivery methods. After analyzing and comparing the various case studies, the information gathered will be condensed into working theories and used to modify what the authors of this research are calling the Integrated Quality Management Model (IQ2M) shown in Figure C1. The model was developed to be generic to all forms of project delivery and furnish a foundation for assigning quality management responsibilities between the owner, the designer, and the constructor. As such, it acts as a framework to structure the analysis of other 167

alternative project delivery quality systems that are common in highway construction and will be used in that manner in this research. Figure C1 – Integrated Quality Management Model (IQ2M) (adapted from Synthesis 376 (Gransberg and Molenaar 2008)) Relevant Definitions Across the highway construction and engineering industry, terms relating to quality often have multiple meanings that in some cases overlap with one another and in others supersede each other. To prevent confusion among several vital terms important to this study, the following definitions have been provided. These definitions are in accordance with the most recent issuance of the TRB Circular Glossary of Highway Quality Assurance Terms E-C137 and the NCHRP Synthesis 376. • Quality: (1) The degree of excellence of a product or service. (2) The degree to which a product or service satisfies the needs of a specific customer. (3) The degree to which a product or service conforms to a given requirement. • Quality Assurance (QA): All those planned and systematic actions necessary to provide confidence that a product or facility will perform satisfactorily in service. [QA addresses the overall problem of obtaining the quality of a service, product, or facility in the most efficient, economical, and satisfactory manner possible. Within this broad context, QA involves continued evaluation of the activities of planning, design, development of plans and specifications, advertising and awarding of contracts, construction, and maintenance, and the interactions of these activities.] TRB E-C074. 168

• Quality Control (QC): Also called process control. Those QA actions and considerations necessary to assess and adjust production and construction processes, so as to control the level of quality being produced in the end product. TRB E-C074. • Quality Management (QM): The overarching system of policies and procedures that govern the performance of QA and QC activities. The totality of the effort to ensure quality in design and/or construction. • Design-Bid-Build (DBB): A project delivery method where the design is completed either by in- house professional engineering staff or a design consultant before the construction contract is advertised. Also called the “traditional method.” • Design-Build (DB): A project delivery method where both the design and the construction of the project are simultaneously awarded to a single entity. • Construction Manager-General Contractor (CMGC): A project delivery method where the contractor is selected during the design process and makes input to the design via constructability, cost engineering, and value analysis reviews. Once the design is complete, the same entity builds the projects as the general contractor. CMGC assumes that the contractor will self-perform a significant amount of the construction work. • Construction Manager-at-Risk (CMR): A project delivery method similar to CMGC, but where the CM does not self-perform any of the construction work. • Public Private Partnership (P3): A project delivery method where the agency contracts with a concessionaire organization to design, build, finance and operate an infrastructure facility for a defined extended period of time. • Design deliverable: A product produced by the design-builder’s design team that is submitted for review to the agency (i.e. design packages, construction documents, etc.). • Construction deliverable: A product produced by the design-builder’s construction team that is submitted for review to the agency (shop drawings, product submittals, etc.). Statement of Purpose The primary research objectives and research questions for this project are as follows: Objectives • Document and categorize current practices and applications of Quality Management Systems (both traditional and alternative) in highway construction for all project delivery methods • Explore how highway construction projects of all project delivery methods are effectively applying alternate quality management systems. (developing and implementing quality management systems) • Identify benefits and limitations of the approaches 169

• Explore how to implement and apply quality management system for all methods of project delivery • Produce a guidebook that will match appropriate quality management systems to selected alternative delivery methods o Describes the quality systems in the I2QM model o Discusses the barriers to each system o Gives guidance for individual roles in development and adoption of alt. quality management models in their agency • Produce a research report that addresses the implications of adopting the guidelines and the barriers to implementation Research Questions 1. What is the fundamental definition of quality and what is the underlying purpose of a “quality program?” 2. How are projects using alternative delivery methods currently applying quality management systems? 3. What are the advantages and disadvantages to the contractor and the owner of alternative quality management systems relating to various project delivery alternatives? 4. What changes must be made to the baseline quality management system to adapt to evolving project delivery methods? Relevant Readings The protocol is based largely on the following documents and research reports.  NCHRP Project 10-83 Proposal  Coding Structure for NCHRP Project 10-83  TRB Circular E-C137 Glossary of Highway Quality Assurance Terms  NCHRP Synthesis 376  NCHRP Synthesis 40-02 170

C.2 Field Procedures Project Researchers The following is a list of the project investigators and their contact information.  Keith R. Molenaar, Ph.D. – Principal Investigator K. Stanton Lewis Chair and Associate Professor Construction Engineering and Management Program Department of Civil, Environmental, and Architectural Engineering University of Colorado at Boulder Campus Box 428, ECOT 643 Boulder, Colorado 80309-0428 Telephone: 303-735-4276 Facsimile: 303-492-7317 E-mail: [email protected]  Douglas D. Gransberg, Ph.D., P.E. – Co-Principal Investigator Donald and Sharon Greenwood Chair and Professor of Construction Engineering Construction Engineering and Management Program Department of Civil, Construction, & Environmental Engineering Iowa State University 456 Town Engineering Ames, IA 50011 Telephone: 515-294-4148 E-mail: [email protected]  David N. Sillars, Ph.D. – Co-Principal Investigator R.C Wilson Chair and Associate Professor Construction Engineering and Management Program Civil and Construction Engineering Oregon State University 220 Owen Hall Corvallis, Oregon 97331 Telephone: 541 737-8058 Fax: 541 737-3300 Email: [email protected]  Elizabeth R. Kraft Graduate Student Construction Engineering and Management Program Department of Civil, Environmental, and Architectural Engineering University of Colorado at Boulder Campus Box 428 Boulder, Colorado 80309-0428 171

Telephone: Facsimile: Email: [email protected]  Nickie West Graduate Student Construction Engineering and Management Program Department of Civil, Construction, & Environmental Engineering Iowa State University 456 Town Engineering Ames, IA 50011 Telephone: Facsimile: E-mail: [email protected]  Landon S. Harman Graduate Student Construction Engineering and Management Program Civil and Construction Engineering Oregon State University 220 Owen Hall Corvallis, Oregon 97331 Telephone: Email: [email protected] Case Study delegation Note: The information in this section will not be available until after potential case studies have been identified and selected for study. When this information is available, this section will list who will be contacting each case study, what the scope of their questions/role will be, and who will be following up to garner additional information or thank participants for their time and effort. Case Study Identification and Schedule Case study project selection criteria In the original research proposal, it was stated that “a concerted effort will be made to select case study projects from transportation agencies that have mature experience with at least two different project delivery methods.” A total of 6 – 12 case studies will be performed with at least two of the case studies coming from a non-STA (USACE or the FTA). Additionally 2 pilot studies will be conducted and reviewed prior to the remaining studies to validate the case study protocol and data coding and to ensure that the research objectives will be met by the data collected. Potential case studies will be identified in Task 1 of the research as a part of the initial survey that aims to identify alternative quality management systems currently in use. The results of this survey will be used to populate an initial list of potential case studies. Case study project selection protocol will involve three tiers of project information that must be present to move a potential case study project into the 172

final list of candidates. The case study candidates will be forwarded to the NCHRP Panel for approval. The tiers are as follows: 1. Project Factor Information: This tier seeks to create a uniform set of data points for every case study project to ensure that trends or disconnects found during analysis can be uniformly mapped across the entire case study population (Yin 2008). Examples of this information are project location, size, major type of construction, initial and final budget amounts, initial and final delivery periods, delivery method, and other factors as required. 2. Project Quality Factor Information: This tier seeks to explicitly define the precise details of the system used to manage quality across the case study project’s life cycle. Examples of this information are quality plans, quality organization composition, use of consultants for independent technical review or independent assurance/oversight, quality audits, division of quality management responsibility between the various stakeholder in the project and other factors as required. 3. Project Performance Information: This tier seeks to measure, if possible, the success of the quality management system employed in each project. It will use to the extent possible the stipulated performance metrics for scope, schedule, cost, quality and risk that were created for each case study project and will attempt to back-calculate performance metrics for common areas in all the projects to furnish a means of comparison and to identify and quantify the magnitude of the trends and disconnects in the case study project population. In addition to the above, a concerted effort will be made to select case study projects from transportation agencies that have mature experience with at least two different project delivery methods. The FHWA Report to Congress on Design-Build Effectiveness (2006) identified more than 30 state STAs that had been authorized to delivery design-build projects under SEP-14 authority. However, of that sample 12 had not completed a single design-build project, another eight had only completed one design-build project, and only six had finished more than five design-build projects. Thus, at that point in time, for this project delivery method the quality management systems of only six STAs could have been impacted by multiple project experiences. Since that time additional experience has been gained and NCHRP Synthesis 376 (2008) reported that four STAs had completed five to 10 design-build projects and nine had completed greater than 10. Thus, depending on the final criterion for design-build experience, as many as 13 STAs will have potential case study design-build projects that had quality management systems that were tailored for DB project delivery. That is not the case in construction management at risk. NCHRP Synthesis 40-02 (2010) reported that only Florida and Utah have completed more than a single construction management at risk project. Thus, case study projects with construction management at risk tailored quality management systems will have to come from those two STAs or from another mode of transportation such as transit. To summarize, the primary criterion for case study project selection will be the requirement to have come from an agency that has sufficient experience with a given project delivery method that the potential exists that the project was designed and built using a quality management system that was modified from the baseline design-bid-build system based on actual experience, using the USACE cyclical quality management system where quality management experience is fed back into the quality planning process to continuously improve the performance of the system itself. 173

This list of potential case studies created from the previously mentioned protocol will be supplemented by the Industry Advisory Board and the investigators’ industry contacts. The following three pieces of information will have been collected for each case study as a part of the survey in Task 1: (1) name and location of the project; (2) description of the project delivery, procurement, and contracting method in use; and (3) description of the quality management system. The goal for selecting the case studies will be to generate a cross section of cases that allow for analysis of the advantages and disadvantages of the quality management systems across the various project delivery method characteristics. To ensure that this goal is met, the following criteria will be placed on the case study selection will include: • Quality management systems for design and construction; • Quality assurance by all parties and independent auditors; • Project quality assurance including independent verification and independent acceptance; • Project delivery methods including design-build, construction manager-at-risk, and PPP; • Procurement methods including best value, A+B, and qualifications-based selection; and • Payment methods including incentives, lump sum, and guaranteed maximum price. Case study informant selection Once a case study project has been selected, several members of the team directly associated with creating and implementing the quality management plan will need to be interviewed. While many people are responsible for ensuring quality on a project during its lifecycle from conception through construction, we will seek to speak with – at a minimum – enough project team members to fully satisfy the research objectives and goals. This may include speaking with representatives from the owner’s, designer’s, and contractor’s project team to develop a full picture of the quality management systems utilized on a project and their relation to each other. Potential interviewees include the following:  Agency project manager, contracting manager, quality manager, etc.  Project design manager, construction manager, design quality manager, construction quality manager, etc.  Designer quality manager, project manager  Contractor preconstruction manager, quality manager, project manager  Third party quality assurance/quality control inspectors 174

Case Study Basic Data and Research Delegation Tables # Case Study Name Location Organization Contact (Information) 1 2 3 4 5 6 7 8 # Case Study Name Contacted? Lead Investigato r Interview Type Intervie w Date Follow-up Date 1 2 3 4 5 6 7 8 175

# Case Study Name Materials/Documents Received 1 2 3 4 5 6 7 8 C.3 Requested Documents The following is a list of documents that will be requested. However, successful completion of the research study does not require that each of the documents is collected. A subset of these documents will likely be collected depending upon the unique attributes of the project being studied. • Project RFP/RFQ • Project RFP/RFQ Response • Project Quality Management Plan • Project Design Quality Plan • Project Construction Quality Plan • Agency/Company Quality Plan • Project Quality Organizational Chart • Copy of the contract with the engineer/contractor/consultant • Organizational document that outlines its approach to quality assurance on project Case Study Questions This section seeks to formalize the interview questions asked of each case study to allow for easier comparison and analysis between case studies later on. Each interview will be unique and largely guided by the case study participants. To draw out the pertinent information, specific questions may be needed while some of the questions listed here may not be relevant. In modifying the protocol questions, generality should still be maintained so that the results can still be readily compared and categorized according to the coding structure. 176

Questionnaire The purpose of the questionnaire is to identify how state highway agencies (SHA) have implemented alternative QA programs and from that baseline, identify commonly used practices for dissemination and use by SHAs that intend to implement alternative procurement on future projects or alternative QA methods in their current program. The questionnaire consists of closed ended questions which will allow the researchers to perform a quantitative analysis. The data gathered by the questionnaire will be used to validate the case study findings. Ideally this questionnaire will be completed by the respondent prior to the interview. If it is not completed prior to the interview then it will be requested that the respondent complete it during the interview. The questionnaire is included in APPENDIX B. Background and Overall Quality Questions Background  Background information to keep track of who we spoke with on each project  Relevant prior experience used to potentially weight their opinions in later comparison 1. Name, occupation, employer 2. What are you current duties, especially related to QM, QA, QC? 3. Have you held any positions prior to your current position related to QM, QA, or QC? If so, please briefly list that information. 4. How long have you held your current position? 5. Have you worked on projects of different project delivery methods? If so specifically what project delivery methods do you have experience with? 6. How many years of experience do you with projects using baseline/DBB quality systems? 7. Name and location of the project for which you are answering project-specific questions Overall Understanding of Quality  Examines the overall understanding of quality and the informants attitudes towards quality  Establishes a baseline of traditional/DBB quality for comparison 8. How do you define project quality? 9. What is your understanding of the differences between quality management, quality assurance, and quality control? 10. In your experience, how has quality management been approached on a traditional/DBB project?  Project by project basis?  During the RFP process?  Dictated by owner agency? If so, how is it dictated?? Specifications, performance… 11. What are the critical elements/milestones of a quality management plan on a traditional/DBB project? 12. How are the quality roles and responsibilities divided on a traditional/DBB project? 13. In your experience, what quality systems/procedures have been successful on traditional/DBB projects? 177

14. What are some characteristics of a successful quality management plan, regardless of the project delivery method, contracting method, or procurement method? Organizational Questions  Provides the needed categories and statistics (from the coding structure) to later sort and group the enterprise level QM information we receive. As not everyone selected for an interview will have answered the survey, these questions are not redundant Please answer the following questions related to quality management at different phases of a project for the organization you work for as a whole. Please be as thorough as possible when discussing what quality management systems may exist and be utilized within your organization. 15. Of the projects your organization designs/builds/owns what percent of the projects are managed using the following delivery methods:  Design-bid-build (DBB): ______  Design-build (DB): _______  Construction Manager/General Contractor (CM/GC): _______  Public-Private-Partnership (PPP): _______  Other (please list): _______ 16. Of the projects your organization designs/builds/owns what percent of the projects are procured using the following procurement methods:  Cost: ______  Best-value: _______  Qualifications: _______  Design: _________ 17. Of the projects your organization designs/builds/owns what percent of the projects utilize the following contract payment methods:  Payment type: ________  Incentive type: ________ 18. If you work for a STA, what due diligence requirements must your organization meet in your state for quality assurance purposes on public projects?  Please list applicable laws, reporting requirements, or processes required in your state  Can you utilize new or alternative quality management practices that produce equal or superior quality projects to meet these requirements? If not, what legal barriers prevent you from doing so?  If so, what steps are required to do this? Has this been done before? Please describe if it has. 19. Does your organization perform any in-house design work?  On what percent of your projects do you perform in-house design?  Do you have formal quality assurance systems in place for this process?  If so, please describe this process. 20. Does your organization self-perform any construction work?  On what percent of your projects do you self-perform some amount of construction?  Do you have formal quality assurance systems in place for this process?  If so, what are they? 178

Design Phase QM  Examines design QM and is broken up into components according to the coding structure to allow for easier comparison between case studies later on  How is design quality management managed on all project delivery types whether design is kept in-house or out of house 21. Does your organization have any design quality assurance systems in place?  If so, please describe the systems, policies, procedures, techniques, or standards used to ensure the quality of the designs you produce or receive. Can you provide us with an electronic or hard copy of these systems, policies, procedures, techniques, or standards?  Have you found these systems to be effective at producing quality designs? If yes, what makes them superior to other systems you are familiar with or function well? If no, what changes would you suggest to increase their efficacy?  Are there any additional procedures that would further improve the quality of designs you produce or receive? If so, please describe them. 22. Does your organization have any design quality control systems in place?  If so, please describe the systems, policies, procedures, techniques, or standards used for quality control of the designs you produce or receive. Can you provide us with an electronic or hard copy of these systems, policies, procedures, techniques, or standards? 23. Does your organization utilize any form of peer review or 3rd party/independent design quality assurance?  If so, please describe how this is conducted and how the results are utilized to improve the quality of the original design.  If this process is formalized, can you provide us with any documents detailing it? Construction Phase QM  Examines construction QM from a STA’s point of view and is broken up into components according to the coding structure to allow for easier comparison between case studies later on  The primary focus of many QM plans and where the bulk of the information gathered at an enterprise level may be found 24. How does your organization perform construction quality assurance? Who has primary responsibility?  Does the process change based on the project type, contract style, or delivery method? If so, how is the process tailored? Who decides which method will be used?  Has your organization experimented with new quality assurance methods? Are you actively implementing any now? If so, what are they and how have you evaluated their efficacy?  Has your organization modified traditional quality assurance processes to make them applicable to projects with alternative delivery methods? If so, how have you modified them? 25. Who is responsible for construction quality control on projects your organization is a part of?  What is the process used for quality control? If this process changes due to project type, contract style, or delivery method, how does it change?  Do you still utilize traditional quality control methods such as control charts?  Has your organization modified traditional quality control processes to make them applicable to projects with alternative delivery methods? If so, how have you modified them? 179

 Has your organization experimented with new quality control methods? Are you actively implementing any now? If so, what are they and how have you evaluated their efficacy? 26. What role does independent assurance (IA) play in your projects?  Which parties – owner, designer, or builder – utilize IA to ensure a quality product? 27. How is owner verification testing used in your traditional projects?  What role does it have in your projects?  What statistical tests are used to verify the contractor’s results?  Who conducts the testing the owner or a third party?  Is it used on your non-traditional projects as well? If so, what function does it serve on those projects? 28. How is owner acceptance testing utilized on this project?  Is this seen as sufficient justification of quality construction to ensure final payment? How are the results used?  Who conducts the testing, the owner or a third party?  If it is used on non-traditional projects, what function does it serve? 29. Does your organization make use of or offer any form of post-construction quality assurance? Please describe this if you can.  Is this part of a warranty or of an operate-and-maintain contract? If not, what is it?  If you are engaged in contracts with warranties or operate-and-maintain clauses, how does this impact your quality management model before and during construction? Project Specific Questions Please answer the following questions for the specific project you were contacted about. While not every question may apply to your project, please be as thorough as possible. If you have additional information regarding the quality management process on you project that you would like to provide, please feel free to add that below or contact us directly. Procurement and delivery method  Used to categorize the case studies for later comparison, these identifying questions will help focus future analysis  Seeks to explore the relationship (if one exists) between the delivery method selected and the QM techniques used 30. What delivery method is being/was used for this project (DBB, DB, CM/GC, PPP, etc.)? 31. What procurement method (cost, best-value, qualifications based, design based, A + B, multiparameter bidding, etc.) was used to select the designer/builder/concessionaire on this project? 32. Was a prequalification process used? If so, please describe that process.  If a prequalification process was used, were interested parties required to submit a quality management plan (QMP)? Were they required to identify a dedicated quality manager? If you still have the QMPs from the initial solicitations, can you provide us with copies? 33. What requirements related to Quality were included in the RFP? 180

 Submittal of a QMP prior to award, qualifications of the quality staff on the project, submittal of a QMP after award, etc. 34. Were either the procurement or delivery methods selected partially or in whole because of their effect on quality management? If so, why were the particular methods selected? 35. What contract payment approach was selected for the project, payment or incentive?  What effect did this have/is this having on quality management on the project? Quality Management Plan  Determine roles and responsibilities for creating the overall quality management plan  Determine the purpose/requirements of the quality management plan  Determine how the Quality Management plan was developed and approved 36. How was the Quality Management Plan (QMP) created?  Was it a stock plan modified for the project? Was it created from scratch? 37. Who was responsible for creating the Quality Management Plan? 38. How was the overall quality management hierarchy created for this project?  What is the overall quality management hierarchy? Can you provide us with a copy of the org chart? 39. What were the Agency’s quality objectives/requirements for this project and how were these communicated? 40. Describe the QMP approval process. 41. How did the QMP differ from a traditional DBB project QMP?  Why did this project’s QMP need to be different from a traditional DBB? 42. How was the QMP development process different from a traditional DBB project?  Why was this projects QMP development process different from a traditional DBB QMP development? 43. How was this QMP and its development process successful?  What about the QMP and its development process could be improved for the next project? 44. What other management plans were required for this project and what were the quality roles and responsibilities associated?  Design, construction, environmental, traffic, etc… 45. Describe how the quality management on this project is different from a traditional DBB project  What procedures/processes/systems were different?  What project factors required the design quality management plan to be different from the traditional (project delivery, project complexity, funding), Design phase QM  Details the specific design QM techniques used on this project as broken down by the coding structure and gathers any additional information related to this process 181

 Seeks to understand the relationship between the techniques used on the project and those recommended at the agency-wide level  Seeks to understand how the design QM was developed and approved and requests the needed documents 46. How was the design Quality Management Plan (QMP) created and by whom?  What requirements was it based on? 47. What was your organization’s role/responsibilities in developing and implementing the design QMP on this project? 48. What was the project hierarchy (org chart) for this phase of the project? Who was responsible for design QA/QC on this project?  Can we get a copy of that chart? 49. What was the approval process for the Design QM? 50. How were the Agency’s design quality requirements communicated to the Designer?  By RFP, design guidelines, performance specifications, etc. 51. Describe how the design quality management on this project is different from a traditional DBB project  What procedures/processes/systems were different?  What project factors required the design quality management plan to be different from the traditional (project delivery, project complexity, funding), 52. What was the basic premise of the design QMP for this project?  Over the shoulder reviews, spot checks, design checks at certain milestones, etc. 53. Was quality assurance incorporated in the design phase of this project? If so, how was it implemented on this project?  Who was responsible for QA, the owner or designer?  Was a formal plan drafted detailing how design QA would be considered? If so, can you provide us with a copy of that report? 54. Were design quality control processes utilized on this project? If so, how?  Are there documents outlining these processes? If so, can you provide us with a copy? 55. Was a dedicated design quality manager assigned by the owner, designer, CM, or concessionaire? What were their responsibilities? What authority did they have to make changes to the design or QA processes? 56. Was a peer review or independent assurance component included in design phase quality management?  If so, what was its role and how did it impact the final design? 57. What problems were discovered and corrected through the design phase quality management process?  What problems were not discovered until the construction phase of the project? Could these have been avoided with a more robust design quality management process? If so, what changes would need to be made to your process to catch these problems in the future? 182

58. Was the design phase QMP used on this project a standardized system used on most or all of your organization’s projects? If not, was it tailored to meet the needs of this project from an existing process or created from scratch for this project?  How much time/what resources did creating the design QMP for this project require?  If an existing design QMP was modified for this project, can you provide copies of both for comparison? 59. How effective was the design QMP on this project?  How could it have been improved? Construction Phase QM  Details the specific construction QM techniques used on this project as broken down by the coding structure and gathers any additional information related to this process  Seeks to understand the relationship between the techniques used on the project and those recommended at the agency-wide level  Seeks to understand how the design QM was developed and approved and requests the needed documents 60. How was the Construction Quality Management Plan (QMP) created and by whom? 61. How much of the quality systems were developed during the RFP/contracting phase?  Was a quality plan required at RFP submittal? after award?  Was the quality plan developed collaboratively? 62. What was the approval process for the construction QMP? 63. How were the Agency’s design quality requirements communicated to those crafting the construction QMP?  By RFP, design guidelines, performance specifications, etc. 64. How were this project’s QM, QA, and QC systems different from a project using traditional DBB?  Roles/responsibilities, liabilities, etc.  What project factors required the design quality management plan to be different from the traditional (project delivery, project complexity, funding), 65. Was there a dedicated quality manager for this project? Who employed the manager, the owner, builder, designer, concessionaire, etc.?  What responsibility and authority to make changes did this manager have? 66. How was construction quality assurance put into place on this project? 67. What role did construction quality control play on this project? If a problem was discovered in the QC process, were changes made quickly enough to prevent future problems?  What facilitated or hindered this rapid communication?  How was QC implemented on this project? 68. Was owner verification testing used on this project to check the contractor’s QC process?  Who performed the testing, the owner or a 3rd party?  Were significant discrepancies discovered on this project? If so, did the QMP provide an effective way to deal with these? How? 69. Was owner acceptance testing included on this project before final payment? 183

 Who performed this testing?  Were significant issues discovered?  How were they handled? Were all parties satisfied with the outcome? 70. Was independent assurance included as part of the construction QMP on this project?  If so, what role did it play in quality management on the project? 71. What other features/systems were parts of the QMP during the construction phase of this project?  What additional features could have prevented quality issues from arising on this project? 72. How was QM in the construction phase affected by the contract delivery method?  By the procurement method? 73. Can you provide a copy of the construction QMP for this project?  Did you modify an existing QMP for the construction phase of this project or develop one from scratch? If you modified an existing plan, can you provide copies of both for comparison? 74. What challenges did the QMP on this project face and how were they overcome? 75. Overall, were you satisfied with the quality management of the construction phase of this project?  Which particular aspects of the QMP were you especially pleased with?  If not, what would you change regarding quality management for your next project? 76. Based on your experience, would any of the quality techniques/systems used on this project be beneficial if applied to a traditional/DBB project? Post-Construction QM  Explores whether a formal policy for QM relating to the post-construction period exists and if such a policy exists, seeks to understand its relationship to the QMP implemented during design and construction of the project 77. Did/does this project include any warranty or operate-and-maintain provisions in the contract? If no, skip the rest of this section. If yes, please answer the following questions.  How did the inclusion of these provisions affect the QMP during construction?  When compared with other projects NOT having these provisions, how did these provisions affect the overall quality of the project? Additional Questions  This section is designed to allow for further refinement of the interview (in the pilot study phase)  Offers the participant the opportunity to add any additional information he/she believes is relevant to our study that we may have overlooked or not asked about for lack of project specific knowledge 78. As a case study participant, how easy to understand did you find the interview questions? Were they straight forward? Did you struggle to understand the intent of some questions? In addition to any answers to these questions, please provide a rating from 1 to 10 (10=easy to understand, 1=so hard to understand it was difficult to finish). 184

 Which questions did you find especially difficult to understand or answer? 79. Did the interview seem repetitive to you? If so, what sections/questions seemed to contain overlapping material? 80. Was the interview burdensome to complete? Were you able to answer each question to the extent of your knowledge without fatigue? 81. Given our project objectives, what additional information can you provide that would help us to better understand alternative quality management? What other QM processes, procedures, or ideas are you aware of (for any phase of a project) that you have not shared with us thus far or have been unable to utilize first-hand? 82. Given our project objectives, are there additional questions that you feel we should be asking? If so, what questions would you suggest? 83. Do you know of any other co-workers or industry peers with a position related to quality management that would be interested in speaking with us? If so, can you provide their contact information? 84. Are you aware of any projects that have utilized or are utilizing excellent quality management procedures on a project with a non-traditional delivery method (DB, CM/GC, PPP, etc.)? If so, can you provide us with the name, location, and any other pertinent information for the project? C.4 Data Analysis • The complete data analysis plan is described in the project Work Plan. The main points of the analysis include: • Advantages and disadvantages to each system from the agency’s and the designer’s/constructor’s point of view; • Identification of trends and common finding between the lit review, survey and case studies; • Triangulate the common findings from these three sources of data to arrive at valid conclusions; • Case studies will be summarized individually in the lens of the IQ2M model; • Using literature review and survey information, compare key attributes of the baseline approach to key attributes in the IQ2M models. (design quality is a gap, but rigorous comparison of construction quality control, construction quality assurance and independent audit procedures will be made); • Individual findings will analyzed across the cases using pattern matching techniques; and • Comparison to baseline quality management system approaches. 185

C.5 Case Study Contact Flowchart #1 Phone •Call contacts for identified case studies and secure their agreement to act as a champion for the case study process • Set up a date and time for the case study interview during this contact #2 Letter • Send letter #1 from the case study protocol to the contact confirming their agreement to act as a champion •This letter should include a brief outline of our project goals and documents we will request #3 Letter • Send letter #2 from the protocol and the questionnaire to participant and ask them to fill it out before the interview if possible •This letter should also include a list of the documents we will be requesting, ask them to collect them ahead of time if possible #4 In-Person •Conduct the case study interview at the agreed time and date •Before leaving the interview, be sure that at a minimum: the questionnaire has been filled out and the needed documents have been collected #5 Letter • Send letter #3 from the protocol thanking the participant for their time and assistance and offer to share the results of the research with them when it is finished 186

C.6 Sample Letters Letter #1 MEMORANDUM DATE TO: Survey Participant FROM: Keith Molenaar Principal Investigator SUBJECT: NCHRP 10-83 Case Study Thank you for agreeing to participate in the NCHRP 10-83 Research Project case study concerning alternative quality management procedures for highway construction projects utilizing non-traditional contracts. We have enclosed some brief background information about the research project, its objectives, goals, and methods. We are currently scheduled to (meet with/call) you on (insert day/month) at (insert time) to conduct our interview. If for some reason this no longer works for you, please contact me as soon as possible to reschedule. Before our interview, we will send you a questionnaire related to the topics we would like to cover in the interview. Please review the questionnaire prior to the interview to become acquainted with the nature of the questions that we will be discussing. I’ve attached a brief outline of our research interests along with a list of documents related to the project that we would like to collect. If you have any questions, please feel free to contact me by telephone at 303-735-4276 or by email at [email protected]. Regards, Keith Molenaar Requested Documents Below is a list of documents we would like to collect regarding the project this case study is focusing on. While your project may not have all of the listed documents, we need to obtain a copy of any of the documents included on your project as well as any additional documents not listed that you believe are relevant to our research objectives as outlined below. While we would prefer (hard/electronic) copies, 187

either will suffice. Additionally, are there any online resources we can examine (FTP sites, project websites, etc.) before the interview to familiarize ourselves with the project? • Copy of the contract with the engineer/ contractor/ consultant • Organizational document that outlines its approach to quality assurance on project • Project RFP/RFQ • Project Quality Organizational Chart • Project RFP/RFQ response • Project Construction quality plan • Project Quality Management Plan • Agency/Company quality plan • Project design quality Plan • Project Background The NCHRP 10-83 research project has four primary research objectives. They are to: • Document and categorize current practices and applications of Quality Management Systems (both traditional and alternative) in highway construction for all project delivery methods • Explore how highway construction projects of all project delivery methods are effectively applying alternate quality management systems. (developing and implementing quality management systems) • Identify benefits and limitations of the approaches • Explore how to implement and apply quality management system for all methods of project delivery In the course of meeting those objectives, we will seek to answer these four questions: 1. What is the fundamental definition of quality and what is the underlying purpose of a “quality program?” 2. How are projects using alternative delivery methods currently applying quality management systems? 3. What are the advantages and disadvantages to the contractor and the owner of alternative quality management systems relating to various project delivery alternatives? 4. What changes must be made to the baseline quality management system to adapt to evolving project delivery methods? To answer these questions, we are performing several in-depth project case studies, covering both traditional and alternative delivery methods. The case studies will be used to learn how existing projects have managed quality in light of non-traditional delivery methods. After analyzing and comparing the various case studies, the information gathered will be condensed into working theories, used to modify what we are calling the Integrated Quality Management Model (IQ2M), and ultimately used to prepare a final research report for the NCHRP and a set of guidelines for when implementing certain AQM systems may be useful in light of other project characteristics. 188

Letter #2 MEMORANDUM April 17, 2015 TO: Survey Participant FROM: Keith Molenaar Principal Investigator SUBJECT: NCHRP 10-83 Case Study Questionnaire Thank you again for your assistance with this research effort investigating alternative quality management procedures for highway construction projects utilizing non-traditional contracts. Enclosed is a questionnaire that touches on many of the topics we would like to cover in our in-depth interview. The questionnaire will be used as a baseline to generate easily comparable data across all of the different case studies we will be performing. It will be a starting point for our discussion and we would greatly appreciate it if you would take the time to look it over and fill it out to the best of your ability before our scheduled interview on (insert day/month) at (insert time). Also enclosed is a list of the documentation we are seeking for this case-study. While we would prefer (hard/electronic) copies, either will suffice. If this information is available on a website for FTP site, please let us know so that we may familiarize ourselves with the information ahead of time. If you have any questions, please feel free to contact me by telephone at 303-735-4276 or by email at [email protected]. Regards, Keith Molenaar 189

Letter #3 MEMORANDUM April 17, 2015 TO: Survey Participant FROM: Keith Molenaar Principal Investigator SUBJECT: NCHRP 10-83 Case Study Follow-Up Thank you for your participation in the NCHRP 10-83 interview process. We recognize that this process is time consuming and very much appreciate your assistance in helping us better understand alternative quality systems in this industry. Your insight and experience with this project will be invaluable as we compare it with projects from across the country and try to develop a better understanding of which systems work well and which do not. The research report and guidelines will not be finished until the summer of 2012, but we would be happy to share the research results with you then if you would like. Again, thank you for your time! If you have any questions, please feel free to contact me by telephone at 303-735-4276 or by email at [email protected]. Regards, Keith Molenaar 190

C.7 Questionnaire INTRODUCTION/BACKGROUND: The purpose of this questionnaire is to identify how state highway agencies (SHA) have implemented alternative QA programs and from that baseline, identify commonly used practices for dissemination and use by SHAs that intend to implement alternative procurement on future projects or alternative QA methods in their current program. DEFINITIONS: The research will use TRB Circular E-C074, Glossary of Highway Quality Assurance Terms to standardize its terminology. The following are terms that must be carefully understood to properly complete this survey. Quality: (1) The degree of excellence of a product or service. (2) The degree to which a product or service satisfies the needs of a specific customer. (3) The degree to which a product or service conforms with a given requirement. Quality Assurance (QA): All those planned and systematic actions necessary to provide confidence that a product or facility will perform satisfactorily in service. [QA addresses the overall problem of obtaining the quality of a service, product, or facility in the most efficient, economical, and satisfactory manner possible. Within this broad context, QA involves continued evaluation of the activities of planning, design, development of plans and specifications, advertising and awarding of contracts, construction, and maintenance, and the interactions of these activities.] TRB E-C074. Quality Control (QC): Also called process control. Those QA actions and considerations necessary to assess and adjust production and construction processes, so as to control the level of quality being produced in the end product. TRB E-C074. Quality Management (QM): The overarching system of policies and procedures that govern the performance of QA and QC activities. The totality of the effort to ensure quality in design and/or construction. Design-Bid-Build (DBB): A project delivery method where the design is completed either by in-house professional engineering staff or a design consultant before the construction contract is advertised. Also called the “traditional method.” Design-Build (DB): A project delivery method where both the design and the construction of the project are simultaneously awarded to a single entity. Construction Manager-General Contractor (CMGC): A project delivery method where the contractor is selected during the design process and makes input to the design via constructability, cost engineering, and value analysis reviews. Once the design is complete, the same entity builds the projects as the general contractor. CMGC assumes that the contractor will self-perform a significant amount of the construction work. Construction Manager-at-Risk (CMR): A project delivery method similar to CMGC, but where the CM does not self-perform any of the construction work. Public Private Partnership (P3): A project delivery method where the agency contracts with a concessionaire organization to design, build, finance and operate an infrastructure facility for a defined extended period of time. 191

Design deliverable: A product produced by the design-builder’s design team that is submitted for review to the agency (i.e. design packages, construction documents, etc.). Construction deliverable: A product produced by the design-builder’s construction team that is submitted for review to the agency (shop drawings, product submittals, etc.). Requested Documents Below is a list of documents we would like to collect regarding the project this case study is focusing on. While your project may not have all of the listed documents, we need to obtain a copy of any of the documents included on your project as well as any additional documents not listed that you believe are relevant to our research objectives as outlined below. While we would prefer electronic copies if available, but either will suffice. • Copy of the contract with the engineer/ contractor/ consultant • Organizational document that outlines its approach to quality assurance on project • Project RFP/RFQ • Project Quality Organizational Chart • Project RFP/RFQ response • Project Construction quality plan • Project Quality Management Plan • Agency/Company quality plan • Project design quality Plan • Additionally, are there any online resources we can examine (FTP sites, project websites, etc.) before the interview to familiarize ourselves with the project? General Information -STA: 7. US state in which the respondent is employed: 8. Name of Agency: 9. Names and groups/sections of interviewees: 10. Please check the appropriate boxes for your agency’s project delivery program (PDM). Project Delivery Method Legislative/Legal Authority Number of years of experience with PDM DBB ☐NA; ☐Pilot projects only; ☐General authorization ☐NA; ☐1-5; ☐5-10;☐ > 10 CMGC ☐NA; ☐Pilot projects only; ☐General authorization ☐NA; ☐1-5; ☐5-10;☐ > 10 DB ☐NA; ☐Pilot projects only; ☐General authorization ☐NA; ☐1-5; ☐5-10;☐ > 10 P3 ☐NA; ☐Pilot projects only; ☐General authorization ☐NA; ☐1-5; ☐5-10;☐ > 10 Other ☐NA; ☐Pilot projects only; ☐General authorization ☐NA; ☐1-5; ☐5-10;☐ > 10 11. Are your Quality Management systems different between Project Delivery Methods? ☐Yes ☐No 192

12. What is the approximate proportion of in-house design versus outsourced design services? In-house design services - Click here to enter text.% Outsourced design services - Click here to enter text. % Administrative prequalification: A set of procedures and accompanying forms/documentation that must be followed by a designer or construction contractor to qualify to submit bids on construction projects using traditional project delivery. Performance based prequalification: A set of procedures and back-up documents that must be followed by a designer or construction contractor to qualify to submit a bid on a construction project based on quality, past performance, safety, specialized technical capability, project-specific work experience, key personnel, and other factors. 3. Does your agency use a prequalification program for design firms? ☐Yes ☐No If the answer to the previous question is YES, please answer for your agency’s program to prequalify design firms. Designer prequalification program factors Prequalification Type Administrative Performance Based Prequalification required for all projects ☐ ☐ Prequalification required for selected projects only ☐ ☐ Prequalification standards are the same for all projects ☐ ☐ Prequalification standards are different by project class ☐ ☐ 4. Does your agency use a prequalification program for construction contractors? ☐Yes ☐No If the answer to the previous question is YES, please answer for your agency’s construction contractor prequalification program. Construction prequalification program factors Prequalification Type Administrative Performance Based Prequalification required for all projects ☐ ☐ Prequalification required for selected projects only ☐ ☐ Prequalification standards are the same for all projects ☐ ☐ Prequalification standards are different by project class ☐ ☐ Case Study Project Information and Data 1. Project Name and location: 2. Project scope of work: 193

3. Original Total Awarded Value of project: $ Final Total Value of project: $ 4. Date preliminary design contract awarded: Date project advertised: 5. Date final design contract awarded: Date construction contract awarded: [Note: same if DB] 6. Original Project Delivery Period (including design) Final Project Delivery Period (including design) Explanatory notes: 7. Project delivery method used on this project: Design-Bid- Build CM-at- Risk Design- Build P3 Please explain what effect this choice had on the overall quality of the project: 8. Which of the following were reasons why your agency selected the delivery method used for this project? Check all that apply. Reduce/compress/accelerate project delivery period Establish project budget at an early stage of design development Get early construction contractor involvement Encourage innovation Facilitate Value Engineering Encourage price competition (bidding process) Compete different design solutions through the proposal process Redistribute risk Complex project requirements Flexibility needs during construction phase Reduce life cycle costs Provide mechanism for follow-on operations and/or maintenance 194

Innovative financing Other: Explain 9. Which of the above was the single most significant reason for the delivery method decision on this project? 10. Please explain the process that led you to the choice of the project delivery system for this project . Case Study Project Quality Management Policy/Procedures Information: The following questions will break up the quality management process into the following four phases: ♦ Procurement phase: Actions taken regarding the quality management process that are reflected in the agency’s contractor prequalification requirements and/or solicitation documentation such as in the Invitation for Bids (IFB), Request for Qualifications (RFQ) and the Request for Proposals (RFP). ♦ Design Phase (in-house): Actions taken after approval to start design work regarding ensuring the quality of the design deliverables as well as that the final design complies with contractual requirements. OR ♦ Design Phase (out-source): Actions taken after design contract award regarding ensuring the quality of the design deliverables as well as that the final design complies with contractual requirements. ♦ Construction Phase: Actions taken after contract award regarding the quality of the final constructed product to ensure that it complies with both the completed design and other contractual requirements. The research team understands that the term “Approval” has a variety of slightly different meanings from state to state. It is used here to indicate the process by which the agency indicates that it is satisfied with the quality of the design or construction deliverable and is willing to make payment for satisfactory completion of that task if asked. Procurement phase: 1. What procurement method was used to select the designer, builder, or concessionaire? (low bid, best-value, qualifications based selection, etc.) 195

2. Please answer for the case study project. If your process was conducted in more than one way, please answer for the most prevalent set of procedures. Do your project advertising/solicitation documents (i.e. IFB, RFQ, RFP, etc.) contain the following? Required proposal/ bid package submittal? If YES: Is it evaluated to make the award decision? If NO: Is it a required submittal after contract award? Yes No Yes No Yes No Qualifications of the Design Quality Manager ☐ ☐ ☐ ☐ ☐ ☐ Qualifications of the Construction Quality Manager ☐ ☐ ☐ ☐ ☐ ☐ Qualifications of other Quality Management Personnel (design reviewers, construction inspectors, technicians, etc.) ☐ ☐ ☐ ☐ ☐ ☐ Design quality management plan ☐ ☐ ☐ ☐ ☐ ☐ Design quality assurance plan ☐ ☐ ☐ ☐ ☐ ☐ Design quality control plan ☐ ☐ ☐ ☐ ☐ ☐ Construction quality management plan ☐ ☐ ☐ ☐ ☐ ☐ Construction quality assurance plan ☐ ☐ ☐ ☐ ☐ ☐ Construction quality control plan ☐ ☐ ☐ ☐ ☐ ☐ Quality management roles and responsibilities ☐ ☐ ☐ ☐ ☐ ☐ Design criteria checklists ☐ ☐ ☐ ☐ ☐ ☐ Construction testing matrix ☐ ☐ ☐ ☐ ☐ ☐ Quality-based incentive/disincentive features ☐ ☐ ☐ ☐ ☐ ☐ Warranties ☐ ☐ ☐ ☐ ☐ ☐ Optional warranties ☐ ☐ ☐ ☐ ☐ ☐ 3. Was your procurement method selected in part because of its effect on quality management? If so, what effect did the method you chose have on overall project quality? Design Phase: 4. Did this project have a formal design quality assurance program for any design performed in- house? ☐Yes ☐No 5. Did this project have a formal design quality assurance program for design performed by design consultants? ☐Yes ☐No 6. Did this project combine in-house design services with projects delivered by alternative methods (CMGC, DB, P3)? ☐Yes ☐No 196

For this project who performed the following design quality management tasks? (Check all that apply) Does not apply Agency design staff Agency project manage- ment staff Project design consult- ant Project con- struction staff in CMGC, DB, P3 Indepen- dent quality consultan t Other Please specify below Technical review of design deliverables ☐ ☐ ☐ ☐ ☐ ☐ ☐ Checking of design calculations ☐ ☐ ☐ ☐ ☐ ☐ ☐ Checking of quantities ☐ ☐ ☐ ☐ ☐ ☐ ☐ Acceptance of design deliverables ☐ ☐ ☐ ☐ ☐ ☐ ☐ Review of specifications ☐ ☐ ☐ ☐ ☐ ☐ ☐ Approval of final construction plans & other design documents ☐ ☐ ☐ ☐ ☐ ☐ ☐ Approval of progress payments for design progress ☐ ☐ ☐ ☐ ☐ ☐ ☐ Approval of post-award design QM/QA/QC plans ☐ ☐ ☐ ☐ ☐ ☐ ☐ Construction Phase: For this project who performed the following construction quality management tasks? (Check all that apply) Does not apply Agency design staff Agency project manage- ment staff Project design consult- ant Project con- struction staff in CMGC, DB, P3 Indepen- dent quality consultan t Other Please specify below Technical review of construction shop drawings ☐ ☐ ☐ ☐ ☐ ☐ ☐ Technical review of construction material submittals ☐ ☐ ☐ ☐ ☐ ☐ ☐ Checking of pay quantities ☐ ☐ ☐ ☐ ☐ ☐ ☐ Routine construction inspection ☐ ☐ ☐ ☐ ☐ ☐ ☐ Quality control testing ☐ ☐ ☐ ☐ ☐ ☐ ☐ Verification testing ☐ ☐ ☐ ☐ ☐ ☐ ☐ Acceptance testing ☐ ☐ ☐ ☐ ☐ ☐ ☐ Approval of progress payments for construction progress ☐ ☐ ☐ ☐ ☐ ☐ ☐ Approval of construction post-award QM/QA/QC plans ☐ ☐ ☐ ☐ ☐ ☐ ☐ Report of nonconforming work or punchlist. ☐ ☐ ☐ ☐ ☐ ☐ ☐ Quality Management Planning: Please answer the following questions about the project based on your experience. 12. Are the design QM plans used on this project different from the QM plans used on traditional design projects? 197

☐No ☐ Yes If yes, what is the major difference? Click here to enter text. 13. Are the construction QM plans used on this project different from the QM plans used on traditional DBB construction projects? ☐No ☐ Yes If yes, what is the major difference? Click here to enter text. 14. Did the agency mandate the use of standard agency specifications? ☐No ☐ Yes 15. Did the agency mandate the use of standard agency design details? ☐No ☐ Yes 16. Did the agency mandate the use of standard agency construction means and/or methods? ☐No ☐ Yes If no, what was required in their place? 17. Did the agency mandate a specific set of qualifications for the quality management staff of design consultants and construction contractors on this project? ☐No ☐ Yes If yes, what are those qualifications? Click here to enter text. 18. Did your agency utilize contractor quality assurance test results for acceptance on this project? ☐Yes. ☐No. Quality Management Procedures: 2. Do you think that the agency held the CMGC/design-builder/P3 concessionaire’s design/construction quality management staff to a higher standard of care than it sets for its internal staff? ☐Yes ☐No 198

Comments? Click here to enter text. 3. Does your organization have a document that outlines its approach to quality assurance on project? ☐Yes ☐No If yes, was it used on this project? If no, what was used in its place? 4. Does your agency use an approach to quality management that is substantially different than that used by other agencies and might be considered an “alternative QM system”? (i.e. statistical analysis of material test reports that eliminates the need for agency acceptance testing) ☐Yes ☐No Describe: 5. Which of the below best describes your agency’s approach to QA on this project? Interviewer: select appropriate delivery method DBB CMGC DB P3 ☐Design consultant primarily responsible for QA/Agency audits consultant program ☐Contractor primarily responsible for QA/Agency audits contractor program ☐Agency retains traditional QA roles ☐Agency retains an independent party to perform QA roles ☐Agency uses two or more of the above depending on the project ☐None of the above ☐Design consultant primarily responsible for QA/Agency audits consultant program ☐Contractor primarily responsible for QA/Agency audits designers and CMGC’s program ☐Agency retains traditional QA roles ☐Agency retains an independent party to perform QA roles ☐Agency uses two or more of the above depending on the project ☐None of the above ☐Design-builder primarily responsible for QA/Agency audits design-builder’s program ☐Agency retains traditional QA roles ☐Agency retains an independent party to perform QA roles ☐Agency uses two or more of the above depending on the project ☐None of the above ☐Concessionaire primarily responsible for QA/Agency audits concessionaire’s program ☐Agency retains traditional QA roles ☐Agency retains an independent party to perform QA roles ☐Agency uses two or more of the above depending on the project ☐None of the above If “None of the above” was selected, please describe the approach that was used instead: 6. What was the biggest quality challenge in the procurement phase? 199

7. What was the biggest quality challenge in the design phase? 8. What was the biggest quality challenge in the construction phase? 9. Please rate the following factors for their impact on the quality of this project: Factor Very High Impact High Impact Some Impact Slight Impact No Impact Qualifications of agency design staff ☐ ☐ ☐ ☐ ☐ Qualifications of agency project management staff ☐ ☐ ☐ ☐ ☐ Qualifications of agency construction staff ☐ ☐ ☐ ☐ ☐ Qualifications of the design consultant’s staff ☐ ☐ ☐ ☐ ☐ Design consultant’s past project experience ☐ ☐ ☐ ☐ ☐ Qualifications of the construction contractor’s staff ☐ ☐ ☐ ☐ ☐ Construction contractor’s past project experience ☐ ☐ ☐ ☐ ☐ Submittal of Quality management plans prior to work start ☐ ☐ ☐ ☐ ☐ Level of agency involvement in the QM process ☐ ☐ ☐ ☐ ☐ Use of agency specifications and/or design details ☐ ☐ ☐ ☐ ☐ Level of detail expressed in the procurement documents (IFB/RFQ/RFP) ☐ ☐ ☐ ☐ ☐ Use of manuals, standards and specifications developed for DBB type projects ☐ ☐ ☐ ☐ ☐ Allowing flexibility in choice of design standards and construction specifications ☐ ☐ ☐ ☐ ☐ Use of performance criteria/specifications ☐ ☐ ☐ ☐ ☐ Detailed design criteria ☐ ☐ ☐ ☐ ☐ Warranty provisions ☐ ☐ ☐ ☐ ☐ Incentive/disincentive provisions ☐ ☐ ☐ ☐ ☐ Follow-on maintenance provisions ☐ ☐ ☐ ☐ ☐ Innovative financing (PPP/concession) ☐ ☐ ☐ ☐ ☐ 200

C.8 Case Study Selection Matrix Case Study Selection Matrix Delivery Method Procurement Method # Case Study Name State Est. Value (millions) Pilot Design- Bid-Build Design- Build CM/GC PPP No Prequalifications Designer Prequalification Contractor Prequalification 1 Hastings River Bridge MN $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ 2 Willamette River Bridge OR $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ 3 $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ 4 $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ 5 $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ 6 $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ 7 $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ 8 $ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

TRB’s National Cooperative Highway Research Program (NCHRP) Web-Only Document 212: Alternative Quality Management Systems for Highway Construction documents the research process, data collection and analysis used to develop NCHRP Report 808: Guidebook on Alternative Quality Management Systems for Highway Construction .

READ FREE ONLINE

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.