Skip to content

Posts tagged ‘Evaluation’

3
Mar

Thoughts on #Ext16 Evaluations

Extravaganza2016_Logo_ColorDear friends,

The E-Team leaders have spent the past week or two going over the details of the evaluations from Extravaganza 2016 in Anaheim. I wanted to follow up with what we learned from this year’s evals.

First, thank you very much to those who took the time to fill out the evaluation forms. The full evaluation report is (10 point font, Ariel) 46 pages long. That’s a ton of data. And we’re grateful. We’ve gone over every numerical answer, and every comment.

225 of you filled out the evaluation form, which represents a 33% return rate, which is a fantastic rate of return for an event like ours. It reminds all of us that people who participate in the Extravaganza really care about this event. It is important, and people want to see it continue to get better and better. It reminds us that tending the E is a trust that we take very seriously.

Looking at the results, clearly we had a great event together in Anaheim. I’ll summarize some of the data:

  • One of the key questions we look at is the first question asked: Would you recommend the Extravaganza to a friend? There are two ways that we examine this data.
    • The answers to this question form what’s called a “Net Promoter Score,” (NPS) which is a way to evaluate how people feel about this event. A NPS can range from -100 to +100. Anything above zero is considered good. Above 50 is considered excellent and above 70 is unbelievable. The E this year scored 77. We’re very, very happy with this score!
    • The other way is to look at the average score on a scale of 1-10. Ext16’s average score on this question was 9.34, which is the highest score we’ve ever had.

Clearly, this was a great event! Thank you to all of you who were a part of it, and helped make it that way!

  • Another very important question for us is “Was this event worth the money spent?” We know that this is a big investment for many, and the E team works hard to make sure that there is good value. Again, in 2016, we scored the highest in our history on this question.  The average response was 9.45. We’re glad that people felt this event was financially “worth it.”
  • All of the other major program areas were also evaluated, and we were very grateful for very good scores across the board. Another score that really stood out was the score for how well the Hyatt in Anaheim worked. Here, we also received the highest score on this question in our history. The Hyatt scored 9.4. And we agree. Our experience there was very, very good. We’re actually in conversation with them about bringing the E back there again at some point in the future.
  • The last numerical question is also one that we look at. It is “I will be back next year.” 173 responded: “Count me in!” 48 said “I might take a year or two off” and 4 said “No chance.” Of those 4, 3 indicated it was because of scheduling conflicts, or “I’m having a baby right before that.” We’ll accept that excuse!

The comments are always the most interesting parts of the evaluations to read. And again, we’re grateful for the time people took to share their thoughts. Some general themes emerged that were very helpful for the planning team to be made aware of.

  • People generally loved our worship and general sessions. We live right in the middle of the “music” opinions. Some people really love hymns. Some like contemporary worship music, and some really love camp-type songs. Likewise, some people liked the various version of the creed, the prayers and the words of institution…other people, not so much. We tried for a variety this year and it appears that generally that was effective, though some people clearly wanted more of one of the other styles. We’ll keep working on balancing the desires for the familiar and the new, the contemporary and the traditional.
  • Workshops were well received. As always, we have some that were stronger than others. Some comments were made that some of the workshop descriptions really didn’t align with what actually happened in the workshops. We’ll continue to work at this to help workshop leaders accurately describe what participants will experience.Our philosophy of workshops is simple: We hope to present the best, cutting edge thinking on children, youth and family ministry as possible. We also hope to continue to raise up new leadership for the mission of the church. And so we are willing to give people, some of whom are doing workshops for the very first time, an opportunity to lead. And we pass along the evaluations and feedback for them so that they can grow in their leadership.
  • People loved Anna Madsen. That was clear. There were many positive comments on her presentation.
  • Some people didn’t like that the E fell over Super Bowl weekend.   I get that. When we negotiate, we give the hotels a “window” of weekend between Martin Luther King Jr. weekend, and President’s weekend (not including the weekends.) Then we work to find the weekend that gives the best prices and the most value. We too would prefer that it were not Super Bowl weekend, (it winds up being a 6 hour time-suck in our schedule) but sometimes it makes too much financial sense to do it then. Next year, we’re not over Super Bowl weekend.
  • The issue of race and ethnicity continues to come up in evaluations. Some people thought that we had made good progress in increasing the level of diversity in the event, and others thought we have a long way to go. 11.2% of Extravaganza registrants self-identified as people of color, which is a dramatic increase from previous years. And the Network is working to increase the diversity of its leadership among all of our teams. And we’ve made some progress in this.Please know that this is an area of growth that the Network is committed to, and that we continue to make progress.
  • It was mentioned, and we know, that the way the Exhibit space is laid out wasn’t ideal. But it’s the best we could do given the hotel’s space constraints. We’ve already made it clear with the Hyatt that if we’re back there, we’d need a different space configuration to make the exhibit space flow better.
  • Even though it’s not directly “E” related, lots of people were very excited about the new MartinsList.org. So are we!

Those were the general themes we heard. Of course, there was much, much more. And as I said, we will go over and learn from every comment.

Thanks again for your willingness to share your thoughts, feelings and ideas about the E. And thanks for caring about this event, and ultimately this community, gathered together in the name of Jesus Christ.

Peace,
Todd

10
Mar

Extravaganza Evaluation Reflections

E2014RoomExtravaganza 2014 is in the history books.  The final hotel bill has been paid, and most of the random boxes of leftover stuff have been crammed back into the warehouse at the Network’s Global Headquarters (my basement).

And, we’ve been pouring through the Extravaganza evaluations.

Thank you to all of you who shared your thoughts, feelings and experiences from this year’s event.  44.8% of you took the time to fill out an Extravaganza evaluation form.  That’s a tremendous return rate.  I think it shows the deep level of care that people have for the Network and the Extravaganza.  It reminds me that the E isn’t just “something we go to,” rather it is a community to which we belong; it is renewal, it is education and it is networking.  On behalf of those of us who work on the project, we are very grateful for your commitment to the E, and especially to those who took the time to fill out the evaluation form.

This year, for the first time, we use a tool called a “Net Promoter Score” as an overall evaluation.  The possible range of Net Promoter Scores is -100 to +100.  Anything above a 0 is generally considered good.  Above 50 is considered very good; Above 70 is amazing.  The Extravaganza in 2014 scored a 78.

Another question we pay a lot of attention is the question of “On a scale of 1-10, was the Extravaganza worth the money spent?”  This year this question rated an 8.96, the highest score the event has ever received.  We feel very good about that! In 2013, the score for this question was 8.56.

As we’ve gone through the comments, there have been a few themes that have surfaced that I think deserve an acknowledgement, or a bit of a response.

  • A consistent theme was how much people appreciated the Hyatt Hotel.  It was a good property, in a good location, with great space.  E-goers really appreciated the meal “deals” and “discounts” that the hotel had put together.  That’s great!  Choosing a hotel property is always a bit nerve-wracking.  One of the things we’ll be able to do at future hotels now is to point to what the Hyatt in St. Louis did as a model for offering discounts.  Hopefully that will help to provide affordable meals in the near future.  It was also helpful that there were a lot of restaurants in close walking distance.  From a leadership perspective, we also really appreciated working with the hotel.  They did a great job.
  • Folks who were forced into the overflow hotel at the Drury Plaza loved it.  Folks at the Sheraton…wished that they were at the Hyatt.  Yep.  We know.  When we book space at the hotel, we are pretty conservative in our contracts because we con’t want to wind up “short” on the rooms we guarantee.  Then we’d incur stiff penalties.  And we’ve been on a growth trend in the last few years.  We are ratcheting our number of contracted rooms every year.  Hopefully we can avoid this in the future.
  • The main stage music had issues.  The concept of doing the “open-source” music remains one that we really liked.  It didn’t quite pull off like we’d hoped it would this year, though the musicians themselves did a very good job.  We haven’t given up on the concept necessarily, but if we do it again in the future, we’d probably do it differently.
  • 95% of people loved the “Ted Talk” format of the main stage speakers. About 5% really hated it.  There was no middle ground.  I think it’s interesting that two people can be sitting side by side at an event and have totally different experiences.
  • “There were too many changes from what was printed in the program book, especially with workshops.”  We will keep working on making this better.  Also, some of the workshop room sizes were too small.  We tried to guess which workshops would best fit in which rooms.  Sometimes we guess wrong.  In the past, we have toyed with the idea of people pre-registering for workshops, or a track of workshops, but generally people have told us that they prefer the flexibility and freedom of deciding “on the spot.”
  • “I wish more meals had been provided.”  Us too.  This is largely a budget issue.  The meals through the hotel are very expensive.  The hotel gives us the meeting space for free, based on a minimum amount of catering that we’ll buy.  So when we pay for a meal, we are really paying for the food, the service, and also also space.  That’s how the hotel makes it’s money.  So we could charge much more and provide more meals.  But we choose to charge less in registration fees, provide the meals we can and let you buy the other meals on your own at local restaurants.  It’s cheaper for you that way.
  • People loved the number and the variety of exhibitors who were at the event, and how the exhibit space “worked.”  We agree.  The space was really good.  We’re going to learn from this and work to emulate the vibe of that space every year, as much as we can.
  • A question that came up was “How do I volunteer to help with the E?”  Great question!  The Extravaganza has both E-Team members (planners) and E-Volunteers (doers.)  Every year new folks come in to both groups.  Often, people who have been E-volunteers, move into E-Team roles.  As specific needs come up on each team and the working groups, we pull new folks into these roles.  We’ll post requests for help as they come up on Facebook and on the website.  Watch for more information there.

The total evaluation report was 38 pages long.  So we’re not going to respond to every thought here.  But these were the themes that came up consistently throughout the evaluation forms.

The last question I want to highlight:  “I will be back next year.”    87% said “yes” and 13% said “I might take a year off.”  No one…not one person…said “no.”

Thanks again to all of you for being a part of the E!

Peace,
Todd

24
Apr

Evaluate

E13 Day 1 Katie-78Sincere thanks to all of you who filled out evaluations for Extravaganza 2013 this past January.  We’ve completed the process of collating the responses, writing up reports and doing analysis.  There was a ton of great data there.  The total, final, “full” report was 47 pages long  (small print..single space!)  Clearly, people care about the Network and this event, and for that we are grateful!

Generally, the feedback for the event was really positive.  Two of the questions that we look at, and which carry a lot of weight for us are:

  • “Overall, this event was (1=terrible; 10 = awesome).”  The response to this question was 8.55.  That’s a very solid response.  And;
  • “This event was worth the money spent (1=not even close; 10=totally).”  The response to this question was 8.56.  Again, we were pleased with this response.

There were lots of other questions, and lots of responses, including great and very insightful comments.  Please know that we will take all of this feedback seriously as the team reconfigures now and ramps up for the 2014 event in St. Louis.

There were also a few questions/comments that showed up throughout the responses that I think merit some response and explanation from the planning team.

The housing at the Hyatt sold out and a group had to move to the Hyatt House.  Those of you who had to stay there were inconvenienced.  We’ve never had this kind of issue before.  When we sign our contracts with hotels, we “guarantee” a certain number of rooms.  It has never been an issue before when we have gone over that mark.  Hotels have been able to accommodate us without a problem.  What we didn’t know was that there was a large, city-wide convention happening at the same time and that the remainder of the hotel was sold-out.  We had guaranteed 900 room nights.  We actually got the hotel to expand our room block to just over 1100 room nights.  But that still wound up leaving us short by around 150.  Of course, we would have sought out hotels just down the street, but they were sold out.  So while it was unfortunate, we were kind of stuck.  We apologize for the inconvenience!

This year the housing links will be live and ready to go on May 15.  We recommend that folks book early.  Last year, the only people who were displaced were those who tried to book their rooms after December 29th.

There was concern raised about the cost of the event.  Some people ask “why can’t we stay at a camp or retreat center.”  We’ve had this question raised before.  It’s a good question.  There are, actually, only 5-7 camp/retreat centers that have the capacity to host the Extravaganza.  And none of those are easily accessible to an airport or transit hub.  The cost of ground transportation would make the cost of the event much higher.  And we do negotiate really good rates from the hotels that host us.  I always say, “If I could find a Hampton Inn with rooms for $89 that have 12,000 sq. feet of ballroom space, 8-9000 sq. feet of exhibit space and 8-10 workshop rooms that can hold 50-100 people theater style, I’d book it!”  But I don’t think it exists.  We are very conscious of the funds people spend to be at the E.  We treat them like a sacred trust, and we work hard to be good stewards.

A minority of people struggled a bit with the worship services.  Reading the responses, it actually reminded me of the responses we often get when we talk about congregational worship.  What can be incredibly meaningful for one person might not even seem like worship to the person sitting next to them.  Some thought it was too contemporary…some thought it was too traditional.  All of your feedback is being carefully considered.

People generally liked having one speaker (John Roberto) for the 3 general sessions.  There were, again, a minority of people who didn’t like the strong emphasis John placed on technology.  I understand that.  It is where a lot of his time and attention went.  Comments like “I’m a youth and children’s minister, not a webmaster” were shared.  On one level I understand that, but on another level, I think part of the point was missed.  (Please understand, I’m just speaking for myself now)  You’re right that we’re not called to be webmasters.  However, we are called to communicate the truth and grace of the Gospel of Jesus Christ in the language and culture of the people.

The people with whom we work live in a digital culture, and use digital mediums to learn, to communicate, and to experience their world.  We have to be fluent in that language, and our role as people who curate resources has to move into the digital because that’s where people are.  This is not in lieu of face to face relationships, but rather alongside and enhancing relationships.  Watch the videos again online…there’s a lot of wisdom in what John says about the culture we’ve been called to minister within.

These were themes we heard that deserved to be acknowledged and responded to.  There were great comments, both critique and affirmation in the 47 pages of evals. Even though we can’t respond to them all, they have already influenced the planning for 2014.

Thank you for caring.  Thank you for your time in filling them out.  They are a great gift to the E-Team, the event, and to our community.

Peace,
Todd

16
Mar

Reflections on Evaluation

This past weekend, the Extravaganza team leaders met at the Hyatt in Anaheim, site of the 2013 Extravaganza, to process the

A General Session at E12

evaluations from the New Orleans event, and to do the organizational planning for ’13.

We spent quite a bit of time processing the evaluations, and I just wanted to share a couple of observations about your thoughts.

  • My first observation is that the overall rating of the event was an 8.56/10.  That’s a great number, and I think it reflects the good, hard work of the E-Team, and also the great community that gathers there.
  • My second observation relates to the sense of being kind of overwhelmed at the response.  This year’s Extravaganza evaluation process yielded a 48% return rate.  We had 307 total evaluations returned.  And the comments…wow…lots and lots of great comments. (I know because we read every one of them!)  Just as a point of reference, when the evaluations were compiled into a single report, and the comments were added (in 8pt. type, font = Ariel) the report totaled 49 pages long.

    What does this tell us?  People really care about this event and about the Network.  It is important to people.  They are invested. And for that, we are very, very grateful

    People come from very different places and experiences, and have different needs that manifest themselves in the evaluation.  Here’s what we heard:

    1.  People loved the worship style.  And, people didn’t connect with the worship style.
    2. People thought the schedule was too packed.  And, people thought the schedule was too loose.
    3. People thought the hotel was great.  And, people didn’t like the hotel.
    4. People connected with the theme.  And, people didn’t connect with the theme.

    Sometimes, I’d be looking at the evaluations and I’d wonder to myself  “were these people at the same event?”  Of course, they weren’t.  They were at their event.  And because everyone comes from a different ‘place’, every evaluation is going to be different.

  • People had mixed feelings about the cost.   What we found was that people thought that the Extravaganza itself is a good deal, and they appreciate the team’s efforts at saving money and keeping costs low.  The question of “was the event worth the money” scored an 8.59 out of 10.  Actually, a higher score than the overall event rating.  This tells us that generally people were satisfied.  At the same time, the cost of the hotel is high.  We knew going in this year’s event that the hotel costs would be a bit higher than normal.  We were committed to being in the city of the Gathering because our members have told us that this is of high value to them.  But to make that happen, we know that some of the associated costs will pop a bit.  We also know that our costs next year in ’13 are lower than they were in ’12.

    Interestingly, some folks questioned some of the specific event expenses.   They wondered if we really needed to pay for a glossy program book, or if we needed all of that sound and lights in the ballroom.  Great questions.  Here’s the answers:  if we just printed small, simple program books, we would have to pay for all of those costs.  By printing glossy, professional program books, we can sell ads in them, which offset the cost, and make for a better looking product.  The reality is that our expenses and our income for the program books came pretty close to offsetting each other.

    Sound and lights are provided by Big Dog Sound, who we have worked with for years.  Basically, we rent sound and lights ‘by the truckload.’  Tom of Big Dog, who is a Network member and a youth minister at  Bethel Lutheran Church in Cupertino, California, gives us a great deal.  And it really doesn’t cost us extra to have the quality, professional “stuff” rather than a simple sound system and fluorescent lights.

    So there are reasons for why we do what we do.  That doesn’t mean that we’ll stop being good stewards of your membership and registration dollars.  We treat them as if they were our own.  And we are very careful on what we spend.

There’s much more…and we’re still processing it.  Lots were said about speakers, music, workshops, hospitality, and everything else that is a part of the event.  But let me say that we are grateful for your thoughts…all 49 pages of them.  And we will take them seriously.  If you have any other comments, please post them as a comment here, or e-mail me at todd@elcaymnet.org.   And we’ve got great stuff in the hopper for 2013 in Anaheim.  Watch for some exciting announcements for speakers in the next few weeks.

Thank you for being the Network!
Todd