Art of Learning

Actions Speak Louder Than Words

This special issue is on Marketing Insights and Marketing Analytics with the goal to close the gap between marketing education and marketing practice. Let’s take this goal and flip it on its side andsee what happens when we apply data mining and business analytics to the study of marketing and business education. Consumer insights becomestudent insights and CRM becomes SRM.

We already collect substantial student data related to the assessment of learning as mandated by accreditation organizations. The data is intended to inform curriculum design and the student experience in order to improve learning and retention. We have become accustomed to gathering information from home-grown tests and case studies, standardized tests by third parties, exit interviews with the soon-to-graduate students, alumni surveys, employer surveys, and so forth.

To that, we also track course grades, GPA, progression towards a degree, participation in school activities, and much more. We have a rather comprehensive SRM. 

In many cases, the data represents survey data of one form or another.  While there is much to be gained in asking questions, our recent experience with data mining and business analytics tells us that actions speak louder than words. As the editors to this special issue have observed, marketing research has shifted away from “seeking answers to questions to seeking answers from data.“

What if we couldstudylearning in-situ? What if we had clickstream data like retail and consumer goods firms have? What could we discover?

Marketplace Simulationscollects this kind of data on every student that plays its simulations. By knowing what students do, when they do it, how long they do it, what other actions are associated with it, and what the outcome is, it is possible to discern many things about each student. As many have discovered with data mining and business analytics, the findings and consumer insights can be very surprising, and useful.

The origin of this data is quite interesting. For more than three decades, the author of Marketplace has observed thousands of students playing Marketplace simulations. He interacted with them in weekly executive briefings, classroom discussions, and chance hallway conversations. He frequently reviewed their decisions and outcomes within the software.

He assessed what they knew and did not know; what they did and did not do; and how they reacted to the simulated market, their own actions, the actions of the competitors, and whether everything played out as they expected. He also noted their reactions when things went well or poorly.  Did they attribute the outcomes to themselves, the actions of others, or some aspect of the simulation itself? Could they reconstruct what they did, when they did it, why they did it, and how it impacted their own capabilities, competitive advantage, and the market’s response? Did they always tell the truth, or want to.

He looked them in the eye, observed their body language, listened to their stories, explanations, and conclusions. He observed their emotional and intellectual engagement, what they were learning or not learning, how they were connecting the dots, and if there was a change in their knowledge, skills, and confidence. And, much more.

The goal was always the same, to improve the simulation design and experience to achieve higher and higher levels of learning and skill development. The insights gained resulted in a continuous string of changes from tiny details to major adjustments to next generation updates.

For a long time, this work was primarily qualitative in nature. But it was apparent that students could not tell you everything you might want to know. And, while we could see the final decision and outcome, we could not tell how they got there.

In order to better judge the efficacy of any element of the simulation experience, we started tracking everything thestudents did and the outcome of it.  The developers were looking for cause and effect so they could fine tune changes to achieve the desired learning. The collected information also proved useful to faculty wanting to provide better coaching to their students and be able to check claims that the students might make regarding their work.

We recently asked ourselves,can we repurpose this vast clickstream data to assess student competencies? Were there indicators that we could attribute to this competency or that one?

We began by looking at learning goals of several business schools and sets of entrepreneurial competencies proposed bynew venture scholars.This approach proved to be problematic as the goals, definitions, and explanations did not map well onto the available data. We started with the competency and then went looking for evidence of it in the data.

We decided to stop looking for evidence of this or that goal achievement or competency and started looking at strings of behaviors and how the behaviors affected related outcomes and subsequent behaviors. We asked ourselves what does this pattern or that pattern suggest about the player. This approach was much more fruitful.

Take for example the start of the simulation, a student has to make decisions without prior simulation knowledge and experience. Can the player draw inferences from unfamiliar information in order to make an unfamiliar decision with an unknown potential outcome?

Similarly, how does that student do when they learn how good or bad a prior decision was and can compare it tothe same decision that their competitors madeand see their performance outcomes?Does the student have the ability to draw meaningful inferences from this newly acquired performance information and use these inferences to improve the earlier decision? Can the player deduce the importance or impact of the available options on a potential outcome and then make a better decision emanating from these inferences?

Moving forward in time, is the player able to apply what was previously learned and extrapolate that knowledge when faced with new, but related choices?

If the player tries again and again to improve performance, what does that tell us about the student. If a student revises an earlier decision but it leads to a poorer outcome than the original, how do we catalogue that sequence?  And, what does the student do next? Do they leave the decision as it is or try again?

There are managerial and practical guidelines throughout the game instructions and help file. There is substantial operational data, competitor information, and performance data. Are they consulted?  When is it consulted?  How much is it consulted?

What is the pattern related to a single decision? A player can make a decision onthe first pass; the decision is made and the player moves on.Alternatively, the player could try out variations of that decision and check its impact on other decision areas. How is one student different than the other?

Do the behavior patterns indicate that a deeper logic was applied or that almost no effort was made? 

Do some students appear to give up while others redouble their efforts?  Are some slow to try new things while others move quickly?

What are we to conclude when a player makes a decision that is not suggested or required of them?There are a number of decisions that could improve performance but they would require the player to figure out their potential value and risk.

Is there a bigger pattern to a players’ actions that suggest they might be better following one career path or another?

It is clear that students have varying abilities to do the things they do. Many of the patterns are known to the author and developers, but they had not been studied in a systematic way or formally catalogued or labeled. We have also discovered behavior patterns that were unknown but have provided new insight into how students work and appear to respond to ever-changing circumstances.

As we now trace student actions and outcomes, we have begun to label the patterns as indicative of this or that competency or this or that shortcoming. Let’s consider three students in our sample that played the Introduction toMarketing simulation from Marketplace Simulations. 

We observed that the first student,Casey Largent,performed well, demonstrating very high marketing acumen.  She worked hard to understand her business situation and made unusually insightful decisions. Her effort to performance ratio indicated a high level of efficiency with sustained excellence throughout her play. Her learning agility and drive for result were excellent. At the same time, there appears to beroom to improve her inductive reasoning and perseverance. While Casey avoided costly mistakes, she did not take advantage of all the opportunities to grow her business.

Jeremiah Johnson was alsoa good player and showed some extraordinary moves in the simulation.  He struggled and underachieved in the early business quarters, trying many things before finding success. His analytical work, deductive reasoning, tenacity, perseverance, innovation, initiative, and resilience carried him through to a strong performance in the final quarters of play. Though quite successful in the end, he was not very efficient as a result of his many trials and errors. He was more aggressive than Casey, though not as thoughtful in certain decisions.

Tyler Moore did not show much interest in the simulation. He gave little attention to the data and made decisions quickly.If he had invested more time studying relevant information and evaluating his options, his understanding of the business and his decisionsmight have been better. It is clear that Tyler’s overall performance was not satisfactory andthere are many areas where he canimprove. As a bit of advice, Tyler may wish to keep in mind the old adage, Actions speak louder than words. His coworkers and bosses will see how he works, which will shape their assessment of his value as an employee.

Along these lines of advice, wewill offer someguidance to each student. The competencies and shortcomings are defined in such a wayas to help the student understand their meaning in a practical way.  We will also offer “nurture narratives” in order to help the student take the news in a positive way and work to improve themselves.

How Can the Competency Data Be Used?

We are developing data that will enable us to apply CRM thinking to education.  There is however, one important difference. With CRM, the goal is to get the customer to spend more money on the proprietor’s products and services. For example, a hair care proprietor wants you to not only purchase shampoo but also conditioner, styling gels, coloring, brushes, accessories, etc. The intent is to hold the customer’s interest and move the customer down the sales funnel where the company wants her to purchasemore products,more often, and in larger quantities. 

With education, the goal is to enhance learning, academic success, retention, and career opportunities. The difference in goals changes everything.  But, the principles developed in the consumer sector apply to the education market.

Take for example the promotional messages of SAS, one of the largest providers of data mining tools for business analytics. See Table 1. We have selected several promotional messages targeted at retail customers to try data mining. We took the same messages and substituted educational constructs for business constructsin order to understand how data mining can be very useful in education. In short, the case for data mining has already been made, we just have not had the opportunity to take it to its fullest yet.

Building off of this perspective, we currently envision four primary users and related uses for the competency data; students, educators, employers, and program administrators.  Let’s take a quick look at each segment and their potential uses.

Students

 

There are several ways where students could benefit from a competency report. First and foremost, it would help them to understand their strengths and weaknesses. From a credibility point of view, the assessments arederived from their own actions and outcomes which they can relate to and verify.

Many students want to enhance their competencies and professional skills in order to be successful in their career. Knowing their strengths can help to develop their confidence and can be highlighted in interviews.Understanding their weakness will also help them to work upon closing the gap between where they are and where they want to be.

There are also some special cases worth considering. For example, some students might come from non-traditional backgrounds and have difficulty doing the work of a business person.Unfortunately, a few might give up hope if they feel they are not prepared. The competency reports can help them to develop insight in how to improve their approach to the work to be done and work upon the areas which are important for their success.

Sometimes a harsh result could give reason to demotivate a student. For this reason, we are creating the nurture narratives to help a student to take the news in a positive way and work out an action plan to improve.

Finally, if a student is identified as Lazy,he might not be happy, but probably knows that heis not taking life and hisstudies seriously. Knowing that others, and not just in the simulation, can see the way he works and performs might motivate him to put more into his professional work.

Faculty

 

Many professors are interested in tools thatcan help them to assess, grade, counsel, and develop their students. Currently, they assess their students on the basis of personal interactions, class participation, and answers given orally or in writing to various assignments and tests. Faculty do not have much opportunity to observe a student’s work style, pattern, unique approach to handlinga particular situation, or how they go about making business decisions. The competency reports canprovide information which is not readily available otherwise.

We do not recommend that these competency reports be used to assess performance and determine grades.  The information is too personal; it relates more to how students do work and respond to business conditions, opportunities, challenges, successes, and setbacks – many of their own making.  There are other indicators for grading. Rather, the information should be used to help develop student talents, confidence, and career options.

In this respect, faculty often have opportunity to speak with students one on one and manydevelop a mentoring role. With the competency reports, instructors will be better able to council students on many aspects of their education, careers, and lives. The datacould help instructors to guide students in areas where they have apparent weaknesses, providingmore salient tips on how to improve. The information could also help with guidance on selecting a career path and job opportunities that might fit the student better; everyone has some strengths and weaknesses and insight into them might suggest a particular career that would fit the student better.

 

Employers

Employers spend a lot of timescreening candidates to determine if they have the right competencies and skills. The competency reports could help them recruit the best team members. The reportsare not prepared from a set of questions asked of the candidate but from the way the candidate responded or reacted to a variety of business situations without asking a single question. The results will be explained in simple charts and graphs which will help in analyzing the information. The reports could:

  1. Enable better short-listing of candidates (a first round decision).Recruiters could match evidenced competencies with the requirements of the job, which could save money and time in the selection process.
  2. Allow more directed conversations and questions during interviews.Recruiters could crosscheck (confirm or counter) the indicated competencies and shortcomings.
  3. Enable a better decision on the final selection.If the candidate’s replies match up with the competency assessments, then the recruiter can make a more confidentdecision.
  4. Indicate areas to improve for current employees (after participating in the simulation), enhancing their job skills and performance potential.

Administrators

 

Program administrators want to improve the quality of their programsand the success of their students. They also want to promote the strengths of their programs and the competencies of their graduates.

Is there a particular pattern to the competencies in evidence. How much of each originates in the students? How much is derived from the training program? It is the nature or nurture question.

If academic leaders are not satisfied with what is observed, how can it be changed?  What aspects of the curriculum can be modified to deliver the desired outcomes? Are there skill enhancement programs that could be added?

More beneficially, If there are particular competencies in the student body that provide graduates with a differential advantage, they can be promoted among recruiters.

One particularly important application of the competency reports would be in the onboarding of new students. It is at this time that students are vulnerable to discouragement and withdrawal. We previously suggested that students could develop self-improvement programs for themselves.Career counselorscould do the same for students at risk. If a student participated in a simulation in an introductory course in marketing or business, the results could be shared with the career counseling office.

In one-on-one meetings, a counselor could highlight a student’s natural talents (build self-image and confidence) and recruit them into personal development activities where they could work on improving themselves in areas where they have weaknesses.  As we have tried to do with the nurture narratives, the goal would be to help the student see and initiate a path to success and personal growth. They also need to realize there is nothing unusual in being better at some things than others. With good guidance, they can improve areas where they are not satisfied.

 

Much More Work to Do

We are in a very early development phase. The patterns are just beginning to emerge, as are their interpretation.

After a decision is processed, we can look at the performance outcome and judge the student’s success. And, we can do this across decision areas to see if the pattern repeats itself. While one decision could be a matter of luck or good guessing, repetitions across decision areas reduces that probability and increases the probability that something more systematic is going on.

We can already see in this early development thatevery student does not exhibit every competency or shortcoming we have discerned. It could be that the circumstances have not presented themselves or the student’s assessment took them down a different path than others. We also see an inverse relationship between competencies and shortcomings. More competency is associated with fewer shortcomings and more shortcomings are associated with less competency. However, Jeremiah demonstratedthat a person vigorously exploring the options can have both. As we dig deeper into more content areas across the entire simulation experience and more student subjects, we expect to see these initial competencies and shortcomings to be more fully explicated, as well as new ones discovered. There is much to be discovered, and perhaps, reinterpreted.

Finally, we realize that the observed competencies represent nothing more than an additional data point to achieve student insight. Additional data will be needed from other sources in order to obtain a more wholistic view of any student. An expanded data set that includes student insights from data mining and business analytics of in-situ learning will serve to create a more effective SRM.

Table 1: SAS Retail Messaging Translated into Educational Messaging

SAS Promotional Messages to Retail Prospects

Messages Rewritten for the Educational Market

“Large customer databases hold hidden customer insight that can help you improve relationships, optimize marketing campaigns and forecast sales. Through more accurate data models, retail companies can offer more targeted campaigns – and find the offer that makes the biggest impact on the customer.”

“Large student databases hold hidden student insight that can help you improve relationships, optimize school communications and forecast retention. Through more accurate data models, schools can offer more targeted courses – and find the curriculum that makes the biggest impact on the student.”

“… retailers must rethink how to use customer data and insights from analytics to improve customer engagement. 

“… schools must rethink how to use student data and insights from analytics to improve student engagement. 

“Personalize interactions to drive the best performance. And integrate the experience across all touch points throughout the customer journey.

“Personalize interactions to drive the best performance. And integrate the experience across all touch points throughout the student journey.

“…. fine-tune marketing interactions and connect customers to the merchandise they want.

“…. fine-tune school to student interactions and connect students to the learning they want.

“… consolidate the customer data … to create a complete view of the customer.

“… consolidate the student data … to create a complete view of the student.

“… use analytics to fully understand what products customers are looking for, what drives purchases, how and when to contact customers, and what the right offers are.

“… use analytics to fully understand what knowledge and skills students are looking for, what drives enrollment, how and when to contact students, and what the right offers are.

“From the customer perspective, better analytics means communications are now much more on the mark.”

“From the student perspective, better analytics means communications are now much more on the mark.”

“They also suggest that customer insights and real-time customer interaction optimization can help a retailer in the following ways:

“They also suggest that student insights and real-time student interaction optimization can help a school in the following ways:

“Enable analytically driven scoring, segmentation and decision making based on historical and current interactions through the use of advanced predictive analytics.

“Enable analytically driven scoring, segmentation and decision making based on historical and current interactions through the use of advanced predictive analytics.

“Combine with context to create meaningful, personalized real-time interactions that improve the customer experience at specific touch points.

“Combine with context to create meaningful, personalized real-time interactions that improve the student experience at specific touch points.

“Map analytics to each stage of the customer life cycle so you can deliver the right message to the right place at the right time, every time.

“Map analytics to each stage of the student life cycle so you can deliver the right content to the right place at the right time, every time.

“Discover how SAS® Customer Intelligence solutions use advanced analytics to help you understand customer uniqueness and build effective targeted marketing, enabling better real-time customer experiences.”

“Discover how SAS® Student Intelligence solutions use advanced analytics to help you understand student uniqueness and build effective targeted curricula, enabling better real-time student experiences.”

Leave a Comment

Your email address will not be published. Required fields are marked *