NOTES

Topic: How do you assess the success of engagements: Are we doing our work well?

April 6, 2017

Participants: Burt Hering, Craig Kowalski, Ulea Lago, David Palmquist, Meghan Maloney, Abigail Oakes, Susan Loucks, Amber Cote, Jan Burrell, Kathy Leone (Marian and Doug), Julie Forman, Jane Greenman, Michael Towers, Doyle Eiler, David Palmquist, Darlyne Koretos, David Factor, Marissa Belau


This call intentionally focuses narrowly on ESC’s own work. Depending on interest, ESC-US may have a future Peer call – and maybe a session at the National Affiliate Conference (this fall in Boston Oct. 12-14) – to dig deeper into this topic and/or to address “impact” or “outcomes”.

  1. Craig – satisfaction survey – 4 question survey – Only 2/3 send back a reply. “Fatigue” causing it. Trying in-depth interviews with select clients who have been away for at least a year. Improving our project evaluations is part of our strategic plan.
  2. Julie – for years sent out surveys at end of engagement – 20 questions – more process oriented. Do similar survey with consultants – how did it go? How could it be better? We then do a comparison with client surveys. Client responses are popular with the consultants. Give us a quote we can use in our marketing and that we can pass on to volunteer. Response rate is 2/3. May add a qualitative element. Q. Who does the assessments? A. Whichever program manager worked on project or admin staff. Working on a system to make sure there are timely responses. Keep first relationship with program manager. Comment-Craig: we tried to do it from a staff role. We use Survey Monkey.
  3. Burt – 4 different ways: repeat business – 1/3 of clients; testimonials; SVP organizational tool at beginning of project and then at end. Corporate sponsor for that program. Surveys. Problems with timeliness – getting the assessments out . Leave surveys in the hands of the consultants (not necessarily the one who worked on the project) This year we started using outputs. How many people in the community do we affect – 200K people in a year. “what are the outcomes?”
    Q. What is the tool that SVP uses and is it free? A. SVP allows all nonprofits free use of its evaluation tool. 501 Commons has a similar tool that is easier to use and just as effective. See their website for more information. Called Elements of a Healthy Nonprofit. https://www.501commons.org/
  4. Kathy – put together action plan committee that’s leading strategic plan. Do survey at the end of the project but like the pre-assessment tools also. We look at a project 3 months out to see if client is interested in phase 2 of the project. Interested in the quality of the project. We collect information for our funders now. We have quantitative but not qualitative data. Q. How do you do surveys? A. Paper. Staff sends to nonprofit. Talking about using consultants over the phone.
  5. Abigail –We are in a unique position because we’re also a funder. Sometimes we wonder about the honesty of the feedback from clients since we are a funder and they may not want to offend us. Also use survey monkey. Talk with client. Follow up 3 times. Ask satisfaction as well as capacity building questions. Using data internally to help prepare consultants. Q. Do you have a sense of repeat business? A. Big pool of organizations – 10-15 projects at a time. 1-2 of those are repeat.
  6. Ulea – I will only talk about what we do differently. We use a 20-minute webinar to train consultants in the interview/evaluation process. We do an early project email survey which help guide the process.. I will email to see how things are going. Pokes at possible places where projects may go off the rail. Getting good feedback from the survey. Post project survey – Survey Monkey – done by consultants. Consultants tell the clients it’s coming. Hard to qualify impact. Did our project affect more than the team?
  7. David Palmquist – We are small and I work directly with every client.
  8. Jane Greenman – send out survey to clients twice a year asking for feedback and comments of projects we did throughout the year. Use Survey Monkey; also send survey after the project to get immediate feedback. Also check in with them to see how it’s going. 60% come from referrals or repeat business. Administered by staff as well as consultants. Phone calls to keep in touch.
  9. Meghan – do evaluations in 2 ways – Our Institutes distribute evaluations at end of session and then at end of the Institute. Also do at close of each consulting projects – 2-4 months. Consultant tells them it’s coming. Create some distance after project so they can reflect on it. Q. Who does the evaluations? A. Staff of the Institutes. Our project advisors do consultant evaluations; they were not part of the projects. Meghan also does them. Sometimes do in person. Personal touch. Adobe PDF form combination of qualitative and quantitative. Use for external communications, to assist staff and with grant writing. David uses data with consultants. Internally for consultants; learn how they can improve. Challenges – using Adobe form. Sometimes not the best way b/c consultant might not have Adobe. Another challenge is that sometimes leadership changes; also clients sometimes don’t respond. Takes lots of staff and consultant time.
  10. Amber – do extensive staged set of evaluations. We use Constant Contact to do survey. Sometimes assign interviews by consultants who have not been involved in the project. We also use consultants who worked on the project to call after one month and after 3 months to see how things are going. Have few consultants to do interview feedback with all of our clients and they have a conversation that’s anecdotal. End of year do one more big survey with all clients – organizational evaluation: “What would you like to see change and grow?” Share with consultants, board, funders, volunteer appreciation event. Q. How do you administer? Thru Constant Contact and consultants. Consultants do the lion’s share.
  11. Doyle – We ask one year from completion of project. Ask project leader to complete a one page set of questions and ask to what extent they have implemented the suggestions we make. Don’t always get full participation from project leaders. Get 50% or more feedback; Doyle sends out couple of reminders. Other thing we do — we give award annually that comes from projects completed in the last 4 years. We Identify 3 projects that staff finds worthy. The winner gets a plaque and comes to annual recognition event. We choose a project that has been effective in implementing suggestions made to them. Been doing for 7-8 years. Client used project to step up their game. Q. Who decides who gets the award? A. Used to be board ; now have operational committee. LA: none of our consultants are on the board. How would you handle that? A. Suggest Board member participate in the review to familiarize themselves with how you run your business.

From: Ulea Lago <ulago@escne.org>
To: Darlyne Koretos <dkoretos@cinci.rr.com>
Date: Thu, 6 Apr 2017 15:14:40 -0400
Subject: Survey vs Interview


Hi Darlyne,
I’ve attached a survey vs interview doc which shows the difference between the post project survey and post project interview. The second document is just the call questions- we send it to the consultant conducting the interview each time they are assigned a new client. Finally, I included the PowerPoint I used for the training webinar.

Hope this is helpful!

Ulea Grace Lago
Director of Consulting
ESC of New England
176 Federal Street
Ste 5-C, Boston, MA 02110
(MA)617-357-5550
(RI)401-783-3141
www.escne.org
ESC’s mission is to enhance the impact of nonprofit organizations through high quality, affordable consulting services, capacity building, leadership support, and programs provided by experienced professionals dedicated to serving the community.


SURVEY VS. INTERVIEW

Survey Monkey Survey

  1. Overall, how satisfied were you with the project you recently completed with ESC?

    • Extremely Satisfied
    • Somewhat Satisfied
    • Neither Satisfied nor Dissatisfied
    • Somewhat Dissatisfied
    • Extremely Dissatisfied
  2. How satisfied were you with the following aspects of the project?
    Extremely Satisfied, Somewhat Satisfied, Neither Satisfied nor Dissatisfied, Somewhat Dissatisfied, Extremely Dissatisfied, N/A

    • Work plan and process
    • Ease of working with consultants
    • Value you got from the assignment
    • Implement-ability of recommendations
  3. How IMPORTANT are the following attributes of consultants?
    Extremely Important, Somewhat Important, Neither Important nor Unimportant, Somewhat Unimportant, Extremely Unimportant, N/A

    • Ability to Meet Deadlines
    • Personal Interaction Skills
    • Responsiveness to Client Needs
    • Level of Experience
    • Functional Expertise (e.g. Marketing, Fundraising, etc.)
    • Sector Knowledge
    • Cultural Competency
    • Presentation Skills
  4. How SATISFIED were you with the following attributes of the ESC consultants you worked with?
    Extremely Satisfied, Somewhat Satisfied, Neither Satisfied nor Dissatisfied, Somewhat Dissatisfied, Extremely Dissatisfied, N/A

    • Ability to Meet Deadlines
    • Personal Interaction Skills
    • Responsiveness to Client Needs
    • Level of Experience
    • Functional Expertise (e.g. Marketing, Fundraising, etc.)
    • Sector Knowledge
    • Cultural Competency
    • Presentation Skills
  5. If the need arose, how likely are you to use ESC’s services again?

    • Extremely Likely
    • Somewhat Likely
    • Neither Likely nor Unlikely
    • Somewhat Unlikely
    • Extremely Unlikely
  6. Why did you choose ESC?

    • ESC Consulting Approach
    • Funder preferences
    • Price
    • Reputation of the organization
    • Expertise
    • Flexibility of ESC to work with timeline
    • Familiarity with specific consultants
    • Other (please specify)
  7. How likely are you to recommend ESC to another organization?

    • Extremely Likely
    • Somewhat Likely
    • Neither Likely nor Unlikely
    • Somewhat Unlikely
    • Extremely Unlikely
  8. Please include any additional comments you may have about ESC’s work with you on this project:
  9. Your Name/Organization (Optional):

PROJECT END EVALUATION CALL

  1. How would you rate the overall value you received from ESC’s work on this project?
    (instruct consultants on how to collect this)
  2. Think about the primary outcome you wanted to achieve through the consulting project. Please describe the outcome and provide any additional comments on the extent to which it has been achieved.
  3. What worked well? What was a challenge? What could ESC have done better?
  4. How likely are you to implement ESC’s recommendations? Why?
  5. What challenges do you now face?
  6. How else can ESC be of assistance?
  7. Would you feel comfortable acting as a reference for ESC?

PROJECT EVALUATION POWERPOINT

Click here to download the PowerPoint presentation