CONSULTANT SELECTION TEAM (CST) SCORING & HOW CSTS ARE FORMED - 2018 PEPS Conference Raul Ortega, P.E.
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
CONSULTANT SELECTION TEAM (CST) SCORING & HOW CSTS ARE FORMED 2018 PEPS Conference Raul Ortega, P.E. November 27, 2018 2018 PEPS Conference November 27, 2018
Presentation Topics 1 How CST members are formed 2 How the scoring process works a State Process b Federal Process 3 How the scoring criteria is developed 4 What CST members look for in a good response 5 Questions and Discussion 2018 PEPS Conference November 27, 2018 2
How CST members are formed Districts/Divisions Districts/Divisions Fiscal Year Subject Matter contracting needs Procurement Plan Experts (SME’s) 2018 PEPS Conference November 27, 2018 4
CST requirements Selection Team • Lead CST - TxDOT PM consists of: • At least two other TxDOT SMEs • Engineering Contracts – Professional Engineer • Surveying Contracts – Professional Engineer or Requirements Registered Professional Land Surveyor • Architectural Contracts – Registered Architect 2018 PEPS Conference November 27, 2018 5
PEPS Procurement Engineer (PcE) as CST option •Optional role – serving as a CST member •subject matter expertise PEPS •if the CST only has two other Procurement members Engineer •if a CST member cannot complete the procurement (originally a CST of 3) 2018 PEPS Conference November 27, 2018 6
Non-CST support SMEs not on the CST help develop: • Non-listed categories (NLC) • Scope • Request for Qualifications (RFQ) interview questions • Request for Proposal (RFP) topics • Criteria 2018 PEPS Conference November 27, 2018 7
State Process – Accelerated and Streamlined (without Interviews) Evaluation Criteria Statement of Qualifications (SOQ) Evaluations: CST assigned Questions and Responses weight % (Q&R) Template CST assigned weight %. Prime Provider’s Past *Currently from TxDOT’s Performance Score CCIS Database. 2018 PEPS Conference November 27, 2018 9
How the scoring process works a. State Process with Interviews 2018 PEPS Conference November 27, 2018 10
State Process – Comprehensive and Streamlined (with Interviews) Evaluation Criteria SOQ Evaluations: CST assigned Questions and Responses weight % (Q&R) Template Prime Provider’s Past CST assigned weight %. Performance Score *Currently from TxDOT’s CCIS Database. (SOQ Phase) 2 Step Process Short-list based on SOQ scores CST assigned weight % Interview and Contract Guide for the Interview (ICG) Evaluation Criterion 2018 PEPS Conference November 27, 2018 11
How the scoring process works b. Federal Process 2018 PEPS Conference November 27, 2018 12
Federal Process Evaluation Criteria CST assigned Proposal Evaluations weight % for proposal CST assigned weight % Prime Provider’s Past *Currently from TxDOT’s Performance Score CCIS Database. Similar to the CST assigned weight % ICG - State 2 Step for the Interview Process Federal Selection Process Evaluation Criteria 2018 PEPS Conference November 27, 2018 13
How the scoring criteria is developed 2018 PEPS Conference November 27, 2018 14
Five steps for developing questions/topics 1. Develop goals & characteristics 2. Create questions/topics based on goals 3. Develop scoring criteria for questions/topics 4. Refine questions/topics & scoring criteria 5. Set weightings & length for each response 2018 PEPS Conference November 27, 2018 15
Develop scoring criteria Question/Interview Criteria Does not meet Meets TxDOT Significantly exceeds requirements requirements requirements 1 2 3 4 5 These responses These are responses that These are the “wow” factors indicate that the firm would satisfy the CST that “CST criteria for the 5’s may does not meet the the firm is qualified to do the work be include, but are not requirements for the question limited to…” What this A score of 2 or 4 is also a scoring option for the CST. means is that the CST may Based on the scoring criteria have not captured all of the and the CST’s subject matter “wow” factors expertise. 2018 PEPS Conference November 27, 2018 16
Example scoring criteria Provide an example of a project similar to the project advertised …. including project details…explanation of unexpected issues…description of resolution of issues… Does not meet requirements Meets TxDOT requirements Exceeds TxDOT requirements 1 3 5 Project discussed includes In addition to the items listed in most of the following: 3, response may include, but is not limited to the following: Details xxx Details Issues xxx xxx Issues Resolution xxx xxx Resolution xxx 2018 PEPS Conference November 27, 2018 17
Example scoring criteria – State Process Based on your team’s experiences with the advancement of bridge design tools and methodologies, discuss the pros and cons of these advancements. Does not meet requirements Meets TxDOT requirements Exceeds TxDOT requirements 1 Pros: 3 5 Design Tools: Pros: • Integration Design Tools: • Ease of revisions if needed • Keeping up with specifications/codes is • Easier to graphically or visually verify input correctness easier • Optimization of design • Design policy implementation • Help to reduce calculation mistakes • consistency Methods: • 3D visualizations for public • More accurately captures the behavior of complex structures Methods: • Modeling time dependent effects • Safety/reliability (ASD vs. LRFD) • Harmonic behavior for cable stay bridges Cons: • line element model vs. grillage model vs. FEM Design Tools: • stiffness of connections (column to cap) • Software adaptability to state design • point of fixity for foundations preferences Cons: • Time and training required for developing Design Tools: familiarization • Non-practical optimization of design • When a new version comes out - not being • May not get uniformity of the design of certain elements able to decipher what changed in the • Time and training required for developing familiarization software and how it affects your design • Not knowing how software is calculating certain elements – “black box” • Discrepancies in software versions, lack • Harder to hand check calculations of backward compatibility Methods: Methods • More complicated code • Understanding applicability • LLDFs • Calculations could be more prone to • Load combinations errors due to misinterpretations • Time of execution –duration (ex. Running a FEM) 2018 PEPS Conference November 27, 2018 18
Example Interview Criteria – State Process Interview Criteria Weight Criterion 1: Solid Understanding of multimodal transportation & potential environmental needs CST assigned weight % Criterion 2: Well-rounded public involvement and outreach experience CST assigned weight % Criterion 3: Clear understanding of data collection and reporting CST assigned weight % Total 100 % Presentation Topics Topic 1: The population of Texas is predicted to nearly triple in the next 25 years. Transportation officials are facing some critical decisions on addressing congestion, mobility, and safety concern issues. Provide your team’s plan and approach to problem solving solutions that address each of these 3 areas? Topic 2: Multimodal planning is an essential part of updating the TTP 2040 Plan. Assume you are presenting to the following groups: (a) Business executive group in a metropolitan location; (b) Home Owners association in a suburban community; (c) Small rural town residents; (d) LEP or Low SES residents in an urban setting. For each group, provide an explanation of multimodal planning and its importance. Topic 3: Assume your team has been tasked with created a GIS model to survey and map cultural resources important to tribes. What challenges and constraints would you anticipate? Outline the team members you would assign to this project and explain your team’s methodology to successfully deliver this assignment. 2018 PEPS Conference November 27, 2018 19
Example Interview Criteria – State Process Criterion 1: Solid Understanding of multimodal transportation planning and potential environmental needs or considerations (Topic 1 & any follow up questions) Does not meet requirements Meets TxDOT requirements Exceeds TxDOT requirements 1 3 5 Response displays sufficient understanding and In addition to items listed under score 3 - response may also knowledge of most of the following include but is not limited to the following • Provides an approach to address Congestion, • Approaches provided also: Mobility, and Safety that: • Recognizes that no one set or number of performance measures will fit multiple customer • Can relate project selection to groups or “market segments” & that performance performance outcomes. measures must evolve over time to be effective • Understands the intricacies • Demonstrates an understanding that performance is associated with multiple stakeholders not simply defined or measured by planning or and agencies engineering standards (e.g., cost-benefit, operations • Identifies and describes multimodal or life-cycle), & must include qualitative performance-based transportation measurements to account for the user’s planning issues transportation experience (e.g., desire for shorter • Understands planning, programming, travel times & mode choice to reduce delays and and environmental phases. improve experience). • Identifies issues that could impact the • Acknowledges the difficulties sustaining a process and provides potential performance measure program with competing solutions priorities Stresses & understands the collaborative • Able to provide standard best nature of each piece of the project development practices to minimize delays process in order to minimize delay. 2018 PEPS Conference November 27, 2018 20
Example Interview Criteria – State Process Criterion 2: Well-rounded public involvement and outreach experience (Topic 1, 2 & any follow up questions) Does not meet requirements Meets TxDOT requirements Exceeds TxDOT requirements 1 3 5 Response displays sufficient understanding and In addition to items listed under score 3 - response may also knowledge of most of the following include but is not limited to the following • Describes techniques for the effective • Demonstrates the potential of communicating to audiences communication of data/information needs at every level through excellent articulation of ideas and deadlines to ensure timely responses throughout presentation and responses to follow up questions from our transportation partners and providers • Displays very strong understanding of importance of communication • Ability to convey their ideas clearly and • Exemplifies strong communication skills effectively based on the audience. • Established program/technology for sharing time • Proposes methods of outreach to engage sensitive/critical information without delay bilingual and underserved communities and • Describes innovative public involvement techniques to stakeholders encourage stakeholder participation • Displays a proactive communicative approach • Discusses the importance of coordination among the team to mitigate public concerns • Recognizes the effects of issues on various and multiple • Good communicator (including presentation different tasks skills) • Recognizes the importance of the Public Involvement • Handles conflict or pressure with calm and coordination for sensitive issues focus • Brainstorming is evident throughout the development of responses • Addresses documentation and follow-up communication methods (meeting minutes…) 2018 PEPS Conference November 27, 2018 21
Example Interview Criteria – State Process Criterion 3: Clear understanding of data collection and reporting (Topic 1, 3 & any follow up questions) Does not meet requirements Meets TxDOT requirements Exceeds TxDOT requirements 1 3 5 Response displays sufficient understanding and In addition to items listed under score 3 - response may also knowledge of most of the following include but is not limited to the following • Describes effective methods for building • Provides innovative best practices that go beyond the professional trust necessary to obtain standards. proprietary data/information • Demonstrates innovative methods to extract and utilize data. • Provides customizable tools and reporting. • Discusses a tool to track project development • Includes training the user as an important component to and how to communicate that to internal and developing the data management tools external stakeholders. • Team actually developed a management information system • Demonstrates ability to gather, store, and or added value to an existing system for a transportation access data. agency • Demonstrates flexibility in being able to deliver • Team provided continued support for users after different tools based on client need implementation and initial training • Identified possible user requirements for a • Describes method(s) for requesting and obtaining new system and a method to ensure the data/information in multiple formats to accommodate entities requirements were included in the with limited human resources, technological resources or implementation technical skills • Describes or proposes possible techniques and tools to conduct training that promotes user-friendly aspects and enhances user skills 2018 PEPS Conference November 27, 2018 22
Example Criteria – Federal Process Evaluation Criteria Weight Technical Approach – project understanding, innovative CST assigned weight % concepts or alternatives, quality control procedures, staffing Project Manager’s Relevant Experience – similar or CST assigned weight % related projects, project management Key Staff’s Relevant Experience – similar projects CST assigned weight % Total 100 % Presentation Topics Topic 1: Your firm has been assigned to provide CEI services for an 3 mile highly urban section on a state highway corridor project. Describe your team’s specific experience in providing inspection, record keeping, and engineering support at the highest quality and responsiveness. You will be evaluated against: Technical Approach, Project Manager’s Relevant Experience, & Key Staff’s relevant experience. 2018 PEPS Conference November 27, 2018 23
Example Interview Criteria – Federal Process Topic 1: Your firm has been assigned to provide CEI services for an 3 mile highly urban section on a state highway corridor project. Describe your team’s specific experience in providing inspection, record keeping, and engineering support at the highest quality and responsiveness. You will be evaluated against: Technical Approach, Project Manager’s Relevant Experience, & Key Staff’s relevant experience. Does not meet requirements Meets TxDOT requirements Exceeds TxDOT requirements 1 3 5 • PM/Key Staff will attend kickoff meeting (Pre-Con). • PM/Key Staff to establish protocol with Client to keep • Technical approach shows an project on time and on budget for decision-making. exceptional or innovative strategy • PM will keep constant communication with client’s project by the PM for managing the team. development of the CEI project. • Inspectors to meet escalation protocol established through • PM is proactive and identifies the partnering process. issues in a timely manner to • PM will establish method of communication with district provide recommendations & (weekly progress reports, issue escalation…) solutions to avoid potential • Reporting methods to keep records updated issues. • Regular reporting of testing to keep deficiencies at a • PM keeps client’s leadership minimum. involved/informed on issues • Responsiveness of PM in a timely manner to issues that through duration of project. arise. • Team familiarity with area, • Key Staff to coordinate with adjacent projects (safety, traffic testing, challenges, and can control, barricades, closures) present options to best suit the area 2018 PEPS Conference November 27, 2018 24
What CST members look for in a good response 2018 PEPS Conference November 27, 2018 25
What CST members look for in a good response - SOQ or Proposal Recommendations Keep in mind that the CST are subject Take and show project ownership on matter experts samples provided Read the Question thoroughly Use your best examples and give credit Answer the questions being asked to firm that performed work under your team, as applicable Focus on readability Use space wisely Roles and responsibilities on previous work need to be clearly identified. Don’t Don’t waste space by marketing over-generalize Clearly identify the roles of the team Be clear, concise, specific, and members if asked technical Consider innovative techniques, challenges, and lessons learned 2018 PEPS Conference November 27, 2018 26
What CST members look for in a good response - Interview Recommendations Read the ICG thoroughly Focus on technical expertise - not on marketing material that will not Focus on the ICG format and be scored requirements Anticipate follow-up questions Focus on the interview topics from the CST. These questions are given in the ICG as well as the important to clarify and expand on evaluation criteria and weighting topics Focus on the interview delivery Expect tough competition Use time wisely Show team’s leadership between PM and task leads 2018 PEPS Conference November 27, 2018 27
Questions and Discussion 2018 PEPS Conference November 27, 2018 28
Contact Information Raul Ortega, P.E. TxDOT | PEPS Service Center for Divisions (915) 790-4411 Raul.Ortega@txdot.gov 2018 PEPS Conference November 27, 2018 29
Example Scoring Criteria
State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 Example Scoring Criteria (slide #18) Based on your team’s experiences with the advancement of bridge design tools and methodologies, discuss the pros and cons of these advancements. Does not meet Meets TxDOT Requirements Exceeds TxDOT Requirements requirements Pros: Pros: Design Tools: Design Tools: • Integration • Keeping up with specifications/codes is • Ease of revisions if needed easier • Easier to graphically or visually verify input • Design policy implementation correctness • Consistency • Optimization of design • 3D visualizations for public • Help to reduce calculation mistakes Methods: Methods: • Safety/reliability (ASD vs. LRFD) • More accurately captures the behavior of Cons: complex structures Design Tools: • Modeling time dependent effects • Harmonic behavior for cable stay bridges • Software adaptability to state design preferences • Line element model vs. grillage model vs. FEM • Time and training required for developing • Stiffness of connections (column to cap) familiarization • When a new version comes out - not being • Point of fixity for foundations able to decipher what changed in the Cons: software and how it affects your design Design Tools: • Discrepancies in software versions, lack of • Non-practical optimization of design backward compatibility • May not get uniformity of the design of Methods: certain elements • Understanding applicability • Time and training required for developing • Calculations could be more prone to errors familiarization due to misinterpretations State Procurement Process • Not knowing how software is calculating certain elements – “black box” PEPS 2nd Annual Conference • Harder | November to hand 27 – 28, 2018 check calculations Methods: • More complicated code • LLDFs • Load combinations • Time of execution –duration (ex. Running a FEM) These are ONLY examples and shall not be used, and will not be considered, for future responses for the solicitation of PEPS procurements.
State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 Example Interview Criteria (slide #19) Interview Criteria Weight Criterion 1: Solid Understanding of multimodal transportation & potential CST assigned weight % environmental needs Criterion 2: Well-rounded public involvement and outreach experience CST assigned weight % Criterion 3: Clear understanding of data collection and reporting CST assigned weight % Total 100 % Presentation Topics Topic 1: The population of Texas is predicted to nearly triple in the next 25 years. Transportation officials are facing some critical decisions on addressing congestion, mobility, and safety concern issues. Provide your team’s plan and approach to problem solving solutions that address each of these 3 areas? Topic 2: Multimodal planning is an essential part of updating the TTP 2040 Plan. Assume you are presenting to the following groups: (a) Business executive group in a metropolitan location; (b) Home Owners association in a suburban community; (c) Small rural town residents; (d) LEP or Low SES residents in an urban setting. For each group, provide an explanation of multimodal planning and its importance. Topic 3: Assume your team has been tasked with created a GIS model to survey and map cultural resources important to tribes. What challenges and constraints would you anticipate? Outline the team members you would assign to this project and explain your team’s methodology to successfully deliver this assignment. State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 These are ONLY examples and shall not be used, and will not be considered, for future responses for the solicitation of PEPS procurements.
State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 Example Interview Criteria (slide #20) Criterion 1: Solid understanding of multimodal transportation planning and potential environmental needs or considerations (Topic 1 and any follow up questions) Does not meet Meets TxDOT Requirements Exceeds TxDOT Requirements requirements In addition to items listed under score 3 - Response displays sufficient understanding and response may also include but is not limited to knowledge of most of the following the following • Provides an approach to address • Approaches provided also: Congestion, Mobility, and Safety that: o Recognizes that no one set or o Can relate project selection to number of performance measures performance outcomes. will fit multiple customer groups or o Understands the intricacies “market segments” & that associated with multiple performance measures must evolve stakeholders and agencies over time to be effective o Identifies and describes multimodal o Demonstrates an understanding performance-based transportation that performance is not simply planning issues defined or measured by planning or o Understands planning, engineering standards (e.g., cost- programming, and environmental benefit, operations or life-cycle), & phases. must include qualitative o Identifies issues that could impact measurements to account for the the process and provides potential user’s transportation experience solutions (e.g., desire for shorter travel o Able to provide standard best times& mode choice to reduce practices to minimize delays delays and improve experience). o Acknowledges the difficulties sustaining a performance measure program with competing priorities Stresses & understands the collaborative nature of each piece of the project development process in order to minimize delay. State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 These are ONLY examples and shall not be used, and will not be considered, for future responses for the solicitation of PEPS procurements.
State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 Example Interview Criteria (slide #21) Criterion 2: Well-rounded public involvement and outreach experience (Topic 1, 2 & any follow up questions) Does not meet Meets TxDOT Requirements Exceeds TxDOT Requirements requirements In addition to items listed under score 3 - Response displays sufficient understanding and response may also include but is not limited to knowledge of most of the following the following • Describes techniques for the effective • Demonstrates the potential of communication of data/information needs communicating to audiences at every level and deadlines to ensure timely responses through excellent articulation of ideas from our transportation partners and throughout presentation and responses to providers follow up questions • Ability to convey their ideas clearly and • Displays very strong understanding of effectively based on the audience. importance of communication • Proposes methods of outreach to engage • Exemplifies strong communication skills bilingual and underserved communities and • Established program/technology for sharing stakeholders time sensitive/critical information without • Displays a proactive communicative delay approach to mitigate public concerns • Describes innovative public involvement • Good communicator (including presentation techniques to encourage stakeholder skills) participation • Handles conflict or pressure with calm and • Discusses the importance of coordination focus among the team • Brainstorming is evident throughout the • Recognizes the effects of issues on various development of responses and multiple different tasks • Addresses documentation and follow-up • Recognizes the importance of the Public communication methods (meeting Involvement coordination for sensitive minutes…) issues State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 These are ONLY examples and shall not be used, and will not be considered, for future responses for the solicitation of PEPS procurements.
State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 Example Interview Criteria (slide #22) Criterion 3: Clear understanding of data collection and reporting (Topic 1, 3 & any follow up questions) Does not meet Meets TxDOT Requirements Exceeds TxDOT Requirements requirements In addition to items listed under score 3 - Response displays sufficient understanding and response may also include but is not limited to knowledge of most of the following the following • Describes effective methods for building • Provides innovative best practices that go professional trust necessary to obtain beyond the standards. proprietary data/information • Demonstrates innovative methods to extract • Discusses a tool to track project and utilize data. development and how to communicate that • Provides customizable tools and reporting. to internal and external stakeholders. • Includes training the user as an important • Demonstrates ability to gather, store, and component to developing the data access data. management tools • Demonstrates flexibility in being able to • Team actually developed a management deliver different tools based on client need information system or added value to an • Identified possible user requirements for a existing system for a transportation agency new system and a method to ensure the • Team provided continued support for users requirements were included in the after implementation and initial training implementation • Describes method(s) for requesting and • Describes or proposes possible techniques obtaining data/information in multiple and tools to conduct training that promotes formats to accommodate entities with user-friendly aspects and enhances user limited human resources, technological skills resources or technical skills State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 These are ONLY examples and shall not be used, and will not be considered, for future responses for the solicitation of PEPS procurements.
Federal Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 Example Criteria (slide #23) Evaluation Criteria Weight Technical Approach – project understanding, innovative concepts or CST assigned weight % alternatives, quality control procedures, staffing Project Manager’s Relevant Experience – similar or related projects, project CST assigned weight % management Key Staff’s Relevant Experience – similar projects CST assigned weight % Total 100 % Presentation Topics Topic 1: Your firm has been assigned to provide CEI services for an 3 mile highly urban section on a state highway corridor project. Describe your team’s specific experience in providing inspection, record keeping, and engineering support at the highest quality and responsiveness. You will be evaluated against: Technical Approach, Project Manager’s Relevant Experience, & Key Staff’s relevant experience. State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 These are ONLY examples and shall not be used, and will not be considered, for future responses for the solicitation of PEPS procurements.
Federal Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 Example Criteria (slide #24) Topic 1: Your firm has been assigned to provide CEI services for an 3 mile highly urban section on a state highway corridor project. Describe your team’s specific experience in providing inspection, record keeping, and engineering support at the highest quality and responsiveness. You will be evaluated against: Technical Approach, Project Manager’s Relevant Experience, & Key Staff’s relevant experience. Does not meet Meets TxDOT Requirements Exceeds TxDOT Requirements requirements • PM/Key Staff will attend kickoff meeting • Technical approach shows an exceptional or (Pre-Con). innovative strategy by the PM for managing • PM/Key Staff to establish protocol with the development of the CEI project. Client to keep project on time and on budget • PM is proactive and identifies issues in a for decision-making. timely manner to provide recommendations • PM will keep constant communication with & solutions to avoid potential issues. client’s project team. • PM keeps client’s leadership • Inspectors to meet escalation protocol involved/informed on issues through established through the partnering process. duration of project. • PM will establish method of communication • Team familiarity with area, testing, with district (weekly progress reports, issue challenges, and can present options to best escalation…) suit the area • Reporting methods to keep records updated • Regular reporting of testing to keep deficiencies at a minimum. • Responsiveness of PM in a timely manner to issues that arise. • Key Staff to coordinate with adjacent projects (safety, traffic control, barricades, closures) State Procurement Process PEPS 2nd Annual Conference | November 27 – 28, 2018 These are ONLY examples and shall not be used, and will not be considered, for future responses for the solicitation of PEPS procurements.
You can also read