A culture of evidence

How big data in education will guide community college transformation

By Karen Stout, Ed.D., President and CEO, Achieving the Dream

NFL players have a chip on their shoulders. Literally. Starting with the 2015-16 season, the league embedded RFID (radio frequency identification) chips into players’ shoulder pads. The chips provide real-time access to vector data such as players’ position on the field, their velocity, direction of travel, and yardage covered during a play. And the information coming from the chips can be used in a number of ways to enhance the game. Media outlets can use that data for enhanced replay. At-home viewers can see the data come to life in an NFL app for Xbox One and Windows 10.

It seems there isn’t an aspect of our lives that isn’t touched by data, including our dreams.

In 2013, The New York Times wrote about digital technology that advances our ability to understand the statistically recurring patterns in our dreams. The article ends with a solid hypothesis – that dreams are meaningful to a considerable degree – and a question: How much more might an enhanced system of big data dream analysis teach us about people’s lives?

Thinking as educators, how much more might big data analysis teach us about students’ lives and further accelerate the work of Achieving the Dream? The mission of this national reform network is to improve college completion rates, especially for the most underprepared students. Our information systems can be powerful enablers toward this goal.

Technologies such as cloud computing, advanced business intelligence systems, and visual analytics have redefined what community colleges can do with data, how fast it can be done, and how readily it can be shared and applied. Administrative staff can get real-time and accessible reports, dashboards, and data visualizations to optimize services and processes. Students, advisors and instructors can get in-the-moment advice and alerts to improve performance. We have only begun to explore the possibilities.

However, the greatest data and analytics in the world won’t have any effect if they don’t fuel a decision or change something. That requires a culture where people understand, value, and demand fact-based decisions and strategies.

From instinct and intuition to data and analytics

In the 2003 book, Moneyball: The Art of Winning an Unfair Game, author Michael Lewis documents the data-driven strategies that produced the competitive 2002 Oakland Athletics baseball team. The central premise of Moneyball is that traditional statistics used to assess players, such as stolen bases, runs batted in, and batting average, are relics of a 19^th-century view of the game, and that the collective wisdom of baseball insiders is subjective and often flawed. Rigorous statistical analysis showed that other metrics, such as on-base percentage and hitting percentage, are better indicators of success. Luckily for the Oakland As, those qualities were cheaper to find on the open market than more traditionally valued qualities, such as speed and contact. This data-driven approach flew in the face of conventional wisdom and was met with skepticism. General Manager Billy Beane had heated discussions with his scouting team. They didn’t believe the numbers. Beane prevailed. His counterintuitive approach to player selection led to a team that was competitive with much larger market teams, such as the New York Yankees. Beane achieved that success with a payroll of $44 million, compared to the Yankees’ $135 million. And his value-sourced team made the playoffs in 2002 and 2003.

This is just one example of how analyzing all available data helped to surface the attributes that really mattered, identified players who had been undervalued by the market, and assembled a team that was destined for success. Isn’t this just what our nation’s community colleges are trying to do?

Anatomy of an analytical ecosystem

Whether you call it an “evidence-based,” “data-driven,” or “analytical” culture, it’s important to think in terms of an ecosystem rather than about institutional processes or a technology platform. In an ecosystem, all parts work together to keep the system in balance. Healthy ecosystems are strengthened and sustained by diversity. And they are controlled by both external and internal factors. A functioning analytical ecosystem must include the following elements:

• Mission/commitment – A strong, stated commitment to evidence-based decision making as an institutional value that has consistent buy-in, involvement, and investment from the board, presidential leadership, and senior management.
• Planning systems – Strategic and annual planning systems with clearly stated, measurable goals (long-term medium-term, and short-term) and aligned key performance indicators that reach deep into the organization.
• Support structure – An organizational structure/architecture that values and supports institutional research and information technology, coupled with strong systems for data management and governance.
• Training and development – Skills building to enable employees at all levels to gain the needed level of data awareness and analytical competence.
  
• Reporting and analysis – Tools and approaches such as Kaplan and Norton’s balanced scorecard to enable staff to develop unit- and course-based success cards and explore “what-if” scenarios.

As we work with community colleges to build the data capacity that serves this culture, four basic requirements become evident:

• Get more granular. Understand the importance of using disaggregated, longitudinal cohort data. This idea was revolutionary in its time but commonplace for most colleges now. In addition to looking at aggregate data, dig deep into data that describes subgroups of students.
• Get to the “why.” Data can very well tell us what happened. Achieving the Dream colleges also use focus groups to bring in the voice of students, faculty, and the community to clarify why something is happening.
• Uncover the obstacles. Find the underlying factors that impede student success. Determine what should be done differently to help students succeed, and address those needs through interventions or policy changes.
• Prove what works. Assess each intervention to determine its effectiveness. In our early experience, many interventions worked, and some were scaled. Overall, the work has informed our thinking and moved us to more “whole college” approaches and reforms such as pathways and advising redesign.

A matter of degrees

Sound data principles alone will not make data-driven decision making come alive or make a difference. “The numbers have no way of speaking for themselves; we speak for them,” wrote Nate Silver in his 2012 bestseller, The Signal and the Noise. Silver is the statistician who successfully called the outcomes in 49 of 50 states in the 2008 U.S. presidential election and was named one of “The World’s 100 Most Influential People” by Time in 2009.

The numbers need an advocate, a Billy Beane – ideally, a campus-wide legion of advocates – as well as the processes and channels to deliver data in consumable fashion to the point of action. Our colleges must have well-thought-out methods for getting faculty, administrators and senior leaders to engage with the data, believe in it, and use it to support decisions about institutional improvement and transformation.

I’m pleased to report that there is progress in that direction. According to Diana Oblinger, President Emerita of EDUCAUSE, the conversation in colleges has moved from “What is analytics?” to “How do we get started, and how do we use analytics well?"

Certainly, analytics capabilities in higher education are developmental. It will take time. It must be tended and nurtured---is not linear, and it is messy. And it is not an either-or proposition. Institutions vary in the degree to which their people, processes, technology, and culture are prepared for the challenges of data-driven transformation. This readiness can be conceptualized on a maturity index from analytically challenged to analytically inspired.

In Part 3 of this series, we’ll take a look at four dimensions of an analytical ecosystem: people, process, technology, and culture, and five levels of maturity along those dimensions. Markers will assess where colleges stand, and what it looks like to reach the next level.