!
(2016).!Teachers!as!producers!of!data!analytics:!A!case!study!of!a!teacher-focused!educational!data!science!progra m.!Journal(of(Learning(
Analytics,!3(3),!193–214.!http://dx.doi.org/10.18608/jla.2016.33.10!
ISSN!1929-7750!(online).!The!Journal!of!Learning!Analytics!works!under!a!Creative!Commons!License,!Attribution!-!NonCommercial-NoDerivs!3.0!Unported!(CC!BY-NC-ND!3.0)!
!
193(
Teachers as Producers of Data Analytics: A Case Study of a Teacher-
Focused Educational Data Science Program
Chase&McCoy&
Indiana!University,!Bloomington,!USA!
chamccoy@indiana.edu!
Patrick&C.&Shih&
Indiana!University,!Bloomington,!USA!
ABSTRACT:&Educational!data!science!(EDS)!is!an!emerging,! interdisciplinary!research!dom ain!that!
seeks!to!improve!educational!assessm en t,!teaching,!and!stud ent!learning!through!data!analytics.!
Teachers! have!been!portrayed!in!the!EDS!literature!as!users!of!pre-constructed!data!dashboa rds!
in!educatio n al! tec h n o lo gie s,! with ! little! co n sid e ra tio n ! give n ! to! the m ! as ! activ e! pr od u c e rs! of! da ta !
analytics.!This!article!presents!the!case!study!results!of!an!EDS!program!at!a!large!university!in!
Midwestern! U.S.A.! in! which! faculty! and! instructors! were! provided! with! access! to! institutional!
data!and! data! analytics!technologies!in! order! to!explore!questions!related!to!their!classroom!and!
departmental!environments.!Semi-structured!interviews!of!program!participants!were!conducted!
to!examine !th e!participants’!exp erien ces !as !p ractitio ne r!re search ers !in !E D S.!The!analysis!showed!
that! participants! were! motiv ated ! to! participate ! to! impro ve! their! learnin g! and ! educa tion al!
environments!through!data!analytics,!as!opposed!to!developing!a!research!agenda!in!EDS;!that!
participants! experienced! a! range! of! barriers! related! to! data! literacy;! and! that! participant!
community! support! in! addition! to! administrative! support! are! vital! to! teacher-fo cu se d! EDS!
programs.!This!study!adds!to!a!small!but!growing!body!of!research!in!EDS!that!considers!teachers!
as!producers!and!not!just!consumers!of!data!analytics.!
Keywords:& Educational! data! science,! learning! analytics,! academic! analytics,! data! literacy,!
practitioner!research!
1 INTRODUCTION
While! there! has!never!been! a! dearth! of! student!data! collected!by! universities,! the! last! decade! has! seen!
an!increase!in!the!types!of!data!captured!as!well!as!developments!in!analytics!capab ilities,!which!have!
enabled! new! avenues! for! data-driven! inquiry! into! the! classro om! environ m en t! a nd ! stu d en t! lea rning !
(Slade!&!Prinsloo,!2013).!In!this!d ata-driven!environment,!educa tion al!data!science!(EDS)!has!emerged !
as!a!multi-disciplinary!com munity!of!researchers!(Piety,!Hickey,!& !Bishop,!2014).!Due!to!the!increas ed !
ubiquity! of! educational! technologies,! such! as! learning! management! systems! (LMS s)! an d! MO OCs,!
educational! data!scientists!have!been!able!to!mo ve!beyond!institutional!assessment!and!into!classroom!
assessment!(Reyes,!2015).!The!communities!that!have!em erged!within!EDS!include!academic!analytics!
(Goldstein! &! K atz,! 2005),! educational! data! mining! (Baker! &! Ya cef,! 2009),! an d! learnin g ! analyti cs!
(Siemens,!20 12 ).!
!
(2016).!Teachers!as!producers!of!data!analytics:!A!case!study!of!a!teacher-focused!educational!data!science!progra m.!Journal(of(Learning(
Analytics,!3(3),!193–214.!http://dx.doi.org/10.18608/jla.2016.33.10!
ISSN!1929-7750!(online).!The!Journal!of!Learning!Analytics!works!under!a!Creative!Commons!License,!Attribution!-!NonCommercial-NoDerivs!3.0!Unported!(CC!BY-NC-ND!3.0)!
!
194(
The! research! in! EDS! has! primarily! focused! on! the! application! of! data! analytics! to! improve! student!
learning,! a c adem ic! performance,! and ! institutional! practices.! In! this! research,! the! teacher! has! been!
predominately! considered! as! a! user! of! technologies! that! display! analytics! about! the! learning!
environment!with!little!consideration!given!to!the!teacher!as!a!possible!producer!of!analytics.! Research!
supporting!this!view!has!addressed!how !teache rs!use!and!integrate!the!data!an alytics!from!dashbo ards!
and! other! similar! technologies! into! their! student! and! classroom! assessment! and! subsequent!
interventions! (Verbert! et! al.,! 2014).! In! th e! cu rren t! edu catio na l! clima te! tha t! pu she s! for! incre ase d!
institutio n al! a ss e ss m e n t,! E D S ! is ! in! t h e! p o s itio n ! to ! e m p o w e r! t ea c he rs ! n o t! o n ly! to! use! educational!
analytics!tools,!but!to!become! active! participants!in!EDS.! W hile!the! literature!on!teachers! as! participants!
in!EDS ! is! s ta rtin g ! to ! g ain ! tr ac tio n ! in ! th e !le a rn in g! a n a lytic s !c o mmunit y!(Clow ,!2014;!Ferguson! et!al.,!2014),!
to!date!there!have!been!few!empirical!studies!focusing!on!teachers’!motivations!to!participate!in!EDS!
and!on!the!barriers!they!face!while!eng a gin g !in !d at a!a n a lytic s .!
In! this! pap e r,! w e ! ad d r es s! th e se ! ga p s ! in! th e ! liter at u re ! th ro u g h ! th e ! res u lts ! of! a ! stu d y ! of! u n iv er sit y!
instructo rs !and!faculty !who !particip a te d!in!a!tea c he r-focuse d!EDS!program !that!positioned !teachers!not!
as!users!of!an alytics,!but!a s!active!participan t s!in!th e !produc tion!of!data !analytics. !Based!on!in te rv ie ws!
with!the!program’s!participants,!we! found!that!regardless!of! the! participants’!previous! data! analytics!
backgrounds,!all! had!experienced!some!type!o f!barrier!related!to!data!literacy.!D ata !literacy!of!teachers!
has! been! addressed! in! the! learning! analytics! literature! in! relation! to! the! ability! to! interpret! and! use!
analytics! for! assessment! (Pea,! 2014);! however,! our! findings! suggest! that! data! literacy! in! EDS! is! an!
intricate!w e b !of!interconnected!skills!necessary!for!identifying,!organizing,!analyzing,!and!summarizing!
data.!In!addressing!these!barriers!in! the! program,! participants!valued!administrative!support,!which!was!
integral! to! the! comple tio n ! of! the! participan t s’! projec ts .! However,! the! participants! stated! that! they!
wished! that!more! emphasis!had! been! placed! on! inter-project!collaboration.! Furthermore,! the! results!
suggest!that!th ese!types!of!prog ram s!sh ould!not!limit!teach er!inquiry!to!a!particular!EDS!commun ity,!as!
the! p articipants! were!motivated!to!use!data!to! learn!more!about!their!educational!environment!than!
they!were!to!ide ntify!w ith!o r!dev elop !a!rese arch !age nd a!in!a!p articu lar!ED S!co m m un ity.&Based!on!the!
study’s! results,!we!present!the!best!practices!for! teach er-fo cu sed !EDS!programs!and!argue!for!emphasis!
on!supporting!teacher’s!inquiry!interests!as!they!relate!to!the!different!EDS!communities;!data!literacy!
training! that! ad dre sses! n o t! on ly! da ta! inte rpre tatio n,! bu t! d ata! a cce ss! a nd ! an alysis;! an d! d ev elop ing ! a!
collaborative!participant!community.!
2 RELATED WORK
2.1 EDS Communities
Piety! et! al.! (2014)! have! identified! four! emerging! communities! in! EDS,! which! they! state! are!
academic/institutional! analytics,! learning! analytics/educational! data! mining,! learner!
analytics/personalization,! and! systematic! instructional! improvement.! The! academic! analytics! and!
systematic!instructional!improvem en t!com m un ities!are!similar!to!the!d ata!u se!pra ctices!associated!w ith!
institutio n al! research,! in!that!th e y! focus!on ! macro- le v el! analyses! of!the!u n its ! within!higher!education,!
!
(2016).!Teachers!as!producers!of!data!analytics:!A!case!study!of!a!teacher-focused!educational!data!science!progra m.!Journal(of(Learning(
Analytics,!3(3),!193–214.!http://dx.doi.org/10.18608/jla.2016.33.10!
ISSN!1929-7750!(online).!The!Journal!of!Learning!Analytics!works!under!a!Creative!Commons!License,!Attribution!-!NonCommercial-NoDerivs!3.0!Unported!(CC!BY-NC-ND!3.0)!
!
195(
such! as! colleges,! scho ols,! or! departments! (van! Barn ev eld,! Arn old ,! &! Cam p b ell,! 2012).! Fu rth erm ore,!
academics! analytics! focuses! on! organizational! concerns,! such! as! how! students! progress! through! the!
educational! system! regarding! retention! and! persistence! in! a! m ajor! or! program! (Piety! et! al.,! 2014).!
Learning! analytics! and! educational! data! mining,! on! the! other! hand,! take! a! micro-level! appro a ch ! t o !
understanding!the!learning !p roce ss!th rou gh !th e!“ the !m ea su rem en t,!co llection ,!an alysis,!a nd !rep o rting!of!
data!about!learners!and!their!contexts,!for!the!purpose!of!understanding!and!optimizing!learning!and!
the!environm e nts!in !wh ich !it!occu rs”!(Siemens!&!Lo ng,!2011,!p.!34).!Learner!an alytics,!while!still!focusing!
on! the! classroom! environment,! differs! from! le ar nin g ! analytic s! in! t h at ! rather! than! focusing ! on! the!
learning! pro c es s,! rese a rch e rs ! take ! a! mac ro -level! view! of! the! clas sro o m! to! understa n d ! the! diffe re n ce s !
among!students’!cognitive!traits!in!relation!to!academic!success!(Piety!et!al.,!2014).!
The!comm unities!of!learning!analytics!and!educationa l!data!mining!in!EDS!h a ve !quickly!develop ed !into!
academic!disciplines!with!their!o wn!journals!and!conferences&(Sieme ns ,!2013);!however,! in! practice!the!
systematic!implementation!of!EDS!outside!of!institutional!research!offices!and!educational!researchers!
has!faced!challenges.!Picciano!(2012)!notes!that!higher!education!lacks!qu alified!professionals,!o utside!
of!educational!data!scientists!and!institutional!researchers,!trained!to!use!and!understand!large!sets!of!
student! data.! R ega rdin g! learning! analytics,! Ferguson! et! al.! (201 4)! address! this! deficit! of! analytics!
professionals! by!suggesting!that! universities! should!begin!to!view!the!development!of!hum an!resources!
as! imperative! for! organization-wide! learning! analytics! adoption.! In! practice,! however,! Bichsel! (2012)!
suggests! that! w hile! u niversity! ad ministrators! support! analytics! initiatives,! the! costs! o f! such ! initiatives!
have!been!prohibitive.!Furthermore,!Bichsel!(2012)!concludes!that!“many!institutions!view!analytics!as!
an!expensive!endeavor,!rather!than!as!an!investment”!(p.!3).!The!costs!associated!with!analytics!is!not!in!
relation! to ! the!tech no logie s! associated! with!analy tics,! rather!the! costs!relate d! to!the! recruitment! and!
development!of!data!analytics!professionals!(Goldstein!&!Katz,!2005).!
2.2 Data Literacy and EDS
As!universities!and!colleges!begin!to!address!these!data!analytics!barriers!related!to!human!resources!
and!analytics!adoption!in!higher!education,!they!must!focus!their!attention!on!ensuring!that!institutional!
staff!possess! and! develop!the! skillsets!required! for!data! analytics! (Prinsloo,!Arch er,! Barnes,! Chetty,! &!
van! Zyl,! 2015).! In! addition! to! this! institutional! perspective,! Selwyn! (2015)! argues! that! educational!
researchers! too! are! facing! challeng es! regarding ! the ! development! of! the! necessary! skills! to! “engage!
effectively!and!insightfully!not!only!with!the!social! uses!of!data,! but!the! d ata!themselves”!(p.!14).!While!
the! data! analytics! capabilities! of! th ese ! two! groups,! institutional! staff! and! educational! researchers,! is!
certainly!important!to!add ress,!teachers! m ust!b egin! to!b e! considered! as! key! actors!in! ED S,!p articularly!in!
the! asse ssm e nt! of! teaching! a nd ! student! learning! (Crisp,! 2012).! With! the! develop ment! of! n e w !
assessment! techniques,! particularly! e-assessment,! teachers! are! hindered! by! the! lack! of! training!
opportunities,! which! results! in! low! levels! of! teachers’! computer! and! technical! literacy! required! to!
engage!effectively!with!these!techniques!(Guàrdia,!Crisp,!&!Alsina ,!2 0 1 6 ) .!
!
(2016).!Teachers!as!producers!of!data!analytics:!A!case!study!of!a!teacher-focused!educational!data!science!progra m.!Journal(of(Learning(
Analytics,!3(3),!193–214.!http://dx.doi.org/10.18608/jla.2016.33.10!
ISSN!1929-7750!(online).!The!Journal!of!Learning!Analytics!works!under!a!Creative!Commons!License,!Attribution!-!NonCommercial-NoDerivs!3.0!Unported!(CC!BY-NC-ND!3.0)!
!
196(
Pea!(2014)!argues!that!attention!must!be!paid!to!the!barriers!that!teachers!face!when!presented!w ith!
analytics!to!m ake!d ata-driven!decisions!about!the!classroom!environment.! This!is!important! to!consider!
since!EDS!is!not!only!a!concern!of!an alytics!professionals!and! academic!researchers,!but!also!teachers!
who!are!positioned!to! blend!data-driven!decision! making! w ith! hum an!judgement! to!impact! the!learning!
environment!(Siemens!&!Baker,!2012).!As!a!result,!unive rsities!and !colleg es!no t!only!nee d!to!focu s!on !
the! developm ent! of! analytics!professionals,!they! must! also!begin! to! address!the! teacher’s!role! in!the!
data!analytics!process!given!that!learning!analytics!and!learner!analytics!depend!“upon!the!ability!of!a!
teacher! to!qu ickly! make! sense! of!da ta! visualizations! presented! in! learning! dashb o a r ds”! (Pea,!2014,!p.!
40).!
Common! throughout! the! research! in! the! EDS! communities! is! the! role! of! the! teacher! in! the! analytics!
process.!Teachers!are!portrayed!as!users! of! an alytics!technologies,!such!as! data!dashboards.! Concerning!
these! data! dashboards,! teachers! are! provided! with! visualizations! of! aggregated! data! about! their!
students! and! their! performance! in! the! classroom! environment! (Verbert! et! al.,! 2014).! As! users! of!
analytics! technologies,!however,!teachers! still!m ust!possess!the!necessar y!data!literacy!skills!to!use!and!
understand!data! to!inform! their!decision!m aking! (Mandinach! &! Gummer,! 2013).!The! skills!associated!
with!data! literacy! include! those! necessary!to! identify,!collect,! organize,! analyze,!and! summarize!data!
(Mandin ach ,! H on ey,! L ight,! &! Brunner,! 2008).! The! research! on! teachers! in! EDS! to! date! has! primarily!
focused!on!the!latter!tw o!of!these!skills,!m e an ing !that!this!res ea rch!has!explore d!how !teachers !use!and!
interact!with!the!analytics!provide d !to!them!and!how!they!make!use!of!these!analytics!to!inform!their!
classroom!teaching! and! stude nt!a ssessm ent.!Exa mp les!of! such ! studies!include ! teachers’!ado ption! beliefs!
of! analytics! technologies! (A li,!Asadi,!Gašević,!Jovanović,!&!Hatala,!2013),!their!use!practices!of!real- tim e !
evaluation!tools!of! student! learning! (Choun ta!&!Avouris,!2014),! and!frameworks! for! developing! learning!
analytics!systems!for!teachers!(Dyckhoff,!Zielke,!Bü lt mann,!C h a tt i,!& !Sc h r o e d e r ,!2012).!The!research!is!
lacking!on !the !dat a!liter ac y!skills !ne ce ss a ry!for !teac h er s!us ing !analytics!in! the! classroom!environment.!In!
this!w ork ,!we!address!the!issues!related!to! da ta!literacy!by! e xam in ing!the!types!of! b arriers!that!teache rs!
face!when !given !the !stud en t!da ta!an d!to ols!re qu ired !to!co nd uc t!data !ana lytics .!
2.3 Teacher-Focused EDS
While!EDS!research!has!primarily!addressed!teachers!as!users! of! educational!data! analytics,!a!num ber! of!
programs! have!developed! in!EDS!that!have!positioned!teachers! not!only! as!users,! but!also!as!producers!
(Ferguson! et! al.,! 2014;! SLAM,! 2014).! T he! University! of! Michigan’s! Student! Learning! and! Analytics! at!
Michigan! (SLAM)! provided! faculty! in! the! program! with! access! to! student! data! and! training! through!
seminars! to! introd uce! them! to! learning ! a na lytics! tools! and! stu de nt! data! (NM C,! 2013).! The! program!
included ! not! only! faculty,! but!graduate! students!as! well,!and ! had!the! support! of!a! larger!institutional!
task!force,!the !Learning!Analytics!Task!Force!(SLAM,!2014).!In!another!program,!the!Data!Wranglers!at!
the! Open ! U nive rsity,! aca de m ic! staff! worked! in! co llabo ration ! w ith! th e! university’s! seven! academic!
faculties! to! particip ate! in ! a! learn ing ! an alytics ! initiative! (Clow,! 2014).! Through! this! program! the! Data!
Wranglers!conducted!analytics!using!a!variety!of!data!sources,!and!they!served!as!data!liaisons!to!the!
academic! departments! by! providing! the m! with! actionable! reports.! Furthermore,! the! Data! Wranglers!
!
(2016).!Teachers!as!producers!of!data!analytics:!A!case!study!of!a!teacher-focused!educational!data!science!progra m.!Journal(of(Learning(
Analytics,!3(3),!193–214.!http://dx.doi.org/10.18608/jla.2016.33.10!
ISSN!1929-7750!(online).!The!Journal!of!Learning!Analytics!works!under!a!Creative!Commons!License,!Attribution!-!NonCommercial-NoDerivs!3.0!Unported!(CC!BY-NC-ND!3.0)!
!
197(
were! encouraged! to! foster! an! interest! in! learning! analytics! and! to! develop! a! learning! analytics!
community!of!practice!in!th e!de pa rtm e nts!(C low ,!201 4;!Ferg uso n!e t!al.,!2014 ).!
In!this!study,!we!exam ine!a!teacher-foc us ed !EDS!program!similar!to!those!o u tlined !above!(Clow,!2014;!
Ferguson!et!al.,!2014;!SLAM,!2014)!to!understand!the!experiences!of!teachers!actively!participating!in!
EDS.!We!developed!three!research!questions!to!guide!this!qualitative!study:
RQ1:( W hat( factors( motivate( faculty( to( participate( voluntarily( in( teacher-focused ( EDS (
programs?(
RQ2:( What( types( of( data( access,( use,( and( interpretation( barriers( and( challenges( do(
faculty(encou n ter(in(tea ch er-focused(ED S (pro gra m s? (
RQ3:( In( what( ways( could( future(teacher-focused( ED S( p rog rams( provide( sup po rt( to( aid (
participants(in(the(educational(data(analytics(process?(
3 CASE STUDY BACKGROUND
At!a!large!Midwestern!university!in!the!U.S.A.,!a!teacher-focused!EDS!(T E D S)! pro gra m ! of! staff! an d !fa cult y!
was!created!to!enable!the!exploration!of!student!engagement,!retention,!and!success!through!the!use!
and!analyses!of!student!data.!The!program!had!the!support!from!the!university’s!centre!for!tea ch ing! an d!
learning,! the! research ! and! assessm e n t! office,! and! the! Office! of! the! Vice! Provost! for! Undergraduate!
Education,! and! w as! a! part! of! a! larger! campus-wide! initiative! to! further! student! success! through!
educational!data!science.!Given!that!this!was!the!TEDS!program’s!first!year!in!existence,!according!to!the!
program! director! the! administration! recognized! that! there! would! be! issues! related! to! the! faculty’s!
production!of!data!analytics!and!the!administration’s!data!support.!As!such,!this!first!year!of!TEDS!was!to!
serve!as!a!pilot!for!future!iterations!of!the!program.!
According!to!the!TEDS!program!director,!the!purpose !o f!th e!p ro gram ! w as! to! pro vide !a n! ou tlet!fo r!fa cu lty!
to!understa nd !a nd !us e!in stitution al!d ata !to!construct!data!analytics !pro jects !tha t!fo cus ed !on !a nsw e ring!
questions! related! to! student! success.! The! program! was! rooted! in! the! scholarship! of! teaching! and!
learning,! a n d ! more! spe cif ica lly! was! bas ed ! o n ! th e ! c on c e pt ! o f! fa cu lt y! le a rn in g! c o m munities .! A ! fa cu lt y!
learning!commun ity !is!com p rise d !of!cross -disciplinary! faculty!that!participate!in!a!program!dedicated! to!
improv in g!te ac h in g!an d !st ud e n t!lea rn ing!(Co x,! 2004 ).! The!TEDS! program!solicited!faculty!from!a!number!
of! disciplines! who! held! a! teaching! role! at! the! university,! which! included! tenured! and! tenure-track!
faculty,!te ac hin g!faculty,!and !university!staff.!Graduate!students!were!allowed!to!participate!so!long!as!
they! w ere!part!of! a!faculty-le d !proje ct .!The!pro jec ts !cho se n !for!TED S!we re !relate d!to!imp ro v ing !teac h in g!
and!learning!through!the!application!of!educational!data!analytics.!
Beginning!in! December!2014,! the! one-year! program! provided! participants!w ith! access! to! institutional!
student! data,! $2,000 ! in! resea rch! funds,! and! a! license! for! Tableau,! a! data! visualization! software.! The!
participants! in!the! program!were! required!to!attend!regular!meetings!with!a!program! directo r!from !the !
university’s!centre!fo r!teaching!and!learnin g.!The!m e eting s!provided!an!outlet!for!particip an ts!to!share !