Monthly Archives: January 2011

Where do we find good critiques of learning analytics?

Concepts need strong critiques. In discussing connectivism, for example, I’ve found critical comments from Plon Verhagen, Bill Kerr, Rita Kopp, Frances Bell, among others, to be very valuable.

Analytics in technology fields are growing in prominence, as revealed by distributed computing (Hadoop, MapReduce), techniques and methods (social network analysis, pattern detection, clustering, classification approaches, language analysis), tools (Gephi, R), open data (WorldBank, open government, OECD), and big data. These developments are increasing the attention devoted to analytics for use in business settings (business intelligence) and in learning and knowledge (the focus of our conference and this course).

Jacques Ellul’s Technological Society is the most effective critique I’ve encountered on the shortcomings (and dangers) of a technique-driven society. Technology instantiates technique. And technique rapidly encroaches in all areas of live (standards-based testing in education, citation analytics in higher education (see Thompson Reuters InCites initiates). If it can’t be measured, it doesn’t get attention or funding (yes, feel free to insert “that” Einstein quote about measurement and what counts into the comments).

Where are the critiques of analytics – particularly in relation to learning and education? This missing element was brought to the the forefront in our Friday wrap up conversation in Elluminate…and in a blog post by Martin Weller on the upcoming conference. Scott Leslie initiated a thread in the comments section dismissing learning analytics. I would prefer a substantive critique, rather than a general reaction to the concept, but Scott’s comment served to raise the profile (in my mind at least) of why we need to critique, not only explore favorably, learning and knowledge analytics. I’ve started a thread in the moodle forum – please drop in and voice your concerns with the concept of analytics as applied to learning.

Artifacts of sensemaking

Now that we are nearing the end of week 1 in LAK11, we’re starting to see a few attempts at making sense of the flow of activity in different forums. These sensemaking attempts include: blog posts, summary Moodle forum posts, images, analysis of discussion forum activity, social network analysis, etc. As we progress in the course, we’ll encounter numerous tools for playing with data and text. Creating and sharing artifacts of sensemaking is an important activity in open online courses.


Higher education generally homogenizes learners through pre-requisites or subject streams (programs). Most learners in a course will be at a roughly similar stage – or so the program structure suggests. In reality, learners are a diverse group, even in reasonably small classes. They come to a course with different beliefs, live experiences, knowledge, aspirations, and learning habits. The uniformity of university program tracks masks the differences of learners.

In an open course, participants aren’t filtered in the same way. Participants range from “absolutely new to the topic” to “have written many books on the topic”. As a result, filtering (or forming sub-networks/groups/discussion clusters) happens once the course is underway. The first few weeks are a bit tumultuous – it’s really a sociological and psychological process of identifying yourself to others and positioning yourself meaningfully in the conversation. It’s not unlike attending a conference or a large social gathering – we reveal aspects of ourselves/our knowledge, we offer tentative views/positions to see if they will resonate with others, we begin to connect with those who respond favorably, we gravitate toward those who we find interesting (but not so interesting that we feel no connection), and so on.

One of the primary ways of connecting with others in an open course is through creating and sharing artifacts of sensemaking. These artifacts are resources produced by individual learners (diagrams, summary posts, podcasts, videos) that reflect their attempts to make sense of the course from her/his perspective. Given the diversity of participants, each learner plays a dual teaching-learning role. When our learning is transparent, we become teachers. We have over 600 participants in this course, which means you will connect with others. You will find people at a similar stage of knowledge. You might even find people in your own community. Essentially, we form small sub-networks that connect (lattice-like) to other sub-networks. Novices engage with novices…but simultaneously, they move into expert networks when a topic warrants. This fluidity of interaction across novice-intermediate-expert networks is one of the main points of value in open courses. And one of the main differentiators from traditional courses.

Reflections on Open Courses: Curation, Ombuds, and Concierges

Part of the focus in LAK11 is to explore how we can better use data to make sense of complex topics such as:

  • How students interact with social and technological systems, information, and each other
  • Which patterns of activity on the part of the learner produce the best performance (still largely defined by grades)
  • How knowledge is “grown” as individuals interact with others
  • How individual learners develop their conceptual understanding of a topic
  • How teams solve complex problems (stages of development and group formation)
  • The tools and activities that are most effective in solving a particular problem in a particular context
  • How individual learners “eliminate” unneeded or irrelevant ideas and concepts
  • How learners orient themselves in complex environments – wayfinding and social sensemaking

This list could go on for quite a while – essentially, any activity that involves information exchange and communication in the process of solving a problem or expanding knowledge in a domain is worthy of analysis through the data trails that are left by individuals.

In this course, we will explore various methods for analyzing data produced by learners and numerous tools that aid that analysis. However, it’s worth considering some of the limitations of an algorithmically-defined world of education. Google search and automated news service are poster children of what an algorithm can achieve. The task for Google has not been easy – as soon as any service becomes popular, marketers and spammers aren’t far behind (Twitter/Quora are starting to experience this too, but since they control who access their sites, they are more successful at eliminating noise). Many of the companies that rely on Google for customers find a change in the (Google’s) algorithms for search can have a devastating impact on sales.

Google’s search algorithm has been ruined argues that:

What has happened is that Google’s ranking algorithm, like any trading algorithm, has lost its alpha. It no longer has lists to draw and, on its own, it no longer generates the same outperformance — in part because it is, for practical purposes, reverse-engineered, well-understood and operating in an adaptive content landscape. Search results in many categories are now honey pots embedded in ruined landscapes — traps for the unwary. It has turned search back into something like it was in the dying days of first-generation algorithmic search, like Excite and Altavista: results so polluted by spam that you often started looking at results only on the second or third page — the first page was a smoking hulk of algo-optimized awfulness.

What’s the solution? Well, a return to curation, of course. We trust people more than technology. I have a friend who is a pilot for a major airline. Apparently, it is possible – in certain airports around the world – for certain planes to land automatically without pilot involvement. Most people, however, would likely find it unnerving to get into a plane without a pilot. A fully automated flight?? Never! But…most of what happens during a flight is already automated. The huge number of adjustments that an airplane (autopilot) makes to compensate for turbulence are largely automated. Why do we still like to see a person in the cockpit?

When Google first announced their automated news site, many journalists and news site ridiculed the idea of non-curated news. Overall, it has worked fairly well. But over the last five years, social networks and social media have taken over the web. Google is driven by the mission to organize the world’s information. Facebook is driven by the mission to “help you connect and share with the people in your life”. The two companies are on a collision course: is the future informationally or socially based? Eventually, social bleeds into informational. And vice versa.

Political discussion is a great example of how this works: pundits within the US political spectrum often express surprise at how “the other party’s followers” defy logic as they follow Beck, Olbermann, Stewart, Limbaugh, Maddow, etc. We trust people we like, people with whom we feel a connection or shared concern/identity. Beck or Olbermann are curators – they present their views and spin existing stories within the framework of their beliefs. All social interactions are information. Many information interactions are social.

What does this have to do with LAK11?

Since Stephen Downes, Dave Cormier, and I, first started offering open online courses, we’ve used a variety of techniques to provide “temporary centres”. Social and technological networks don’t have a centre. When we learn in a classroom or in a learning management system (LMS), a central place exists where we can go for readings and discussions. When a course is distributed – such as LAK11 – across Google Groups, Netvibes, blogs, Moodle, Elluminate, Facebook, Twitter – we encounter the problem of how to create temporary centres that will help us to understand what’s happening in the course. I addressed this concern in Activity Streams: splicing information and social relations.

In the open courses I’ve taught with Stephen, we’ve used his OLDaily and gRSShopper software (in addition to Moodle, Twitter, Facebook, Second Life, etc). When participants start the course, they provide their blog feed and any updates on their blog, related to the course, is automatically included in the Daily. Similarly, Moodle discussion topics and the use of the course tag on Twitter are also included in the email. Stephen and I provide some commentary or facilitator posts to the Daily as well. Basically, for course participants, the Daily is a temporary centre, tying together the many strands of activity in the course. Additionally, and one of the biggest benefits of this model, a full archive of course activity – by date – is available for future analysis. Have a look at the archive of CCK08, CCK09, Critical Literacies, and PLENK10. These courses artifacts are ripe for analysis. We just need to decide what types of questions we want answered!

In LAK11, we’ve taken a different approach. We’ve retained similar course design elements to previous open online courses (OOCs – I’m starting to think that M=Massive part of MOOCs is misleading or even off-putting. Plus someone mentioned that in Catalan, MOOC means mucus :)). In LAK11 we have (course links can be accessed here):

  • The Daily (Google Groups)
  • Moodle Forums
  • Course blog
  • Elluminate sessions
  • Facebook groups
  • Conversations on Twitter/Diigo/Delicious – tied together with the LAK11 course tag

…and so on. For those participants who have taken an open course with me/Stephen/Dave in the past, this format will be familiar. What will be less familiar, and a model that Dave and I experimented with in our EdFutures course, is the lack of archive and integration of conversations in other spaces in the Daily email. There are two primary reasons for this:

  1. We want to demonstrate that if someone wants to offer an open online course, they do not need to run their own server or write their own software.
  2. We didn’t ask Stephen if we could run this course on his site

What we gain in our decision to run this course on various sites, using more or less accessible tools, is the demonstration that anyone with an interesting topic/idea and a willingness to experiment can open up a course for a broader audience. About half of the courses that we’ve run over the last three years have been in for-credit programs in a public university. The other half have been focused on professional development without formal university credit. We’re trying to model that open online courses are broadly accessible – for both learning and teaching.

What we lose – and I’m still uneasy about this trade off – is the integrated archive of activity in the course. I still send out a Daily email to the Google group. I aggregate blogs/delicious/diigo/twitter links and commentary on my netvibes page. The problem, however, is that netvibes is rather dumb. It just leaves the content on the page until something new is posted. If you’re tracking activity on Netvibes, you’ll likely encounter much of the same content until it has been updated with new content. Activity is not archived by date.

Making some improvements

With each offering of an OOC, we try to play with one aspect of the course format. In the second year of offering the connectivism and connective knowledge course, we decided to offer a series of “mini-conferences” within the course, in addition to improving the Daily with better distinctions between facilitator posts, Twitter course tag use, and blogs/Moodle. In PLENK, we were more focused on tracking learner activity: how many new participants joined each week, activity in the moodle forum, number of blogs being aggregated, attendees at the live sessions, and so on. For CCK11 (starting on Monday), we’re going to experiment with running the course without an LMS and using only gRSShopper for interaction. In LAK11, one of the key additions seems to be the role of “course ombud” that Dave Cormier is serving.

Dave is basically acting as a course curator. He takes elements of the course that resonate with him and shares his experiences with other participants. Tony Searl is starting to play a similar role by aggregating the course blogs and content based on his interests. Fascinating stuff. In open courses so far, we’ve tried to open up content and interaction. We’re starting to see people create their own Daily and sharing those artifacts with each other.

Back to where we started

Complexity cannot be understood solely through algorithms. Algorithms don’t instill within us the sense of trust that we require to make decisions and act meaningfully. Curation is an important component in the process. Through social distributed networks, as we each try and make sense of that part of the course that resonates with us, the multi-narrative begins to serve as a sensemaking agent. Curation is important – yes, it’s biased, yes it misses contributions, but it’s personal. (I addressed curation briefly in this paper (.pdf), in Curatorial Teaching, and in this short presentation).

While information is growing in abundance and tools and algorithms (data mining, visualization) are being developed as solutions, we can’t overlook the importance of wayfinding and sensemaking in social systems. As we progress in this course – especially next week when our topic is “big data” – I think we need to also focus on the human aspect of data, sensemaking, curation, and trust.

Course Syllabus

The course syllabus for the open online course Learning and Knowledge Analytics is now available:

If you haven’t done so yet, join this google group: LAK11. All updates will be sent to this group.

The moodle discussion forum (free to create an account):

Important links for LAK11

Bookmark this post, it includes important links for the course:

Netvibes Aggregation Page

Elluminate (for Live Sessions)

Moodle (for asynchronous interactions)

Google Groups (The Daily email)