A Litmus Test for Educational Technology

Posted on by Jack McShea

technology_244495

Even a cursory glance at the education press will shock (if not stun) the reader with the immense glut of high-tech proposals pointed in the direction of the classroom. Take your pick: the virtual classroom, distance learning, xMOOC, cMOOC, learning analytics, the flipped classroom, project-based learning, competency-based curricula, blended learning, eBooks, the LMS, social networking…. They all take a technological swipe at fixing some real or perceived problem with the way we educate each other.

Educational technologists are all too often cautiously taciturn when it comes to offering critiques or early warning signs of new technologies. Perhaps this is because they are customarily like expectant fathers, the very “last to know.” If you are lucky you might stumble onto a paper or blog post by a hoary-haired observer of the educational landscape who recounts a technology that promised some needed transformation but nevertheless fell woefully short. There are many in the annals of educational reform. Anybody remember the MORU? Anybody using television in the classroom?

Still, those working in the trenches of educational content creation and delivery have a nagging problem to address. Is there no simple test that can be applied to technologies to help fix a bearing on where they are headed? The task is not a simple one given the breadth of the scale. After all, the 2000-seat lecture theater and the pencil are both examples of educational technologies.

Thankfully one place to look is among the media theorists. Media are, by definition, things that extend us. Tools and prosthetic devices that enhance or amplify some ability or faculty are media. In this sense both the lecture and the pencil are media.

Media theorists Marshall McLuhan and Neil Postman both wrestled with the problem of analyzing media. Marshall and Eric McLuhan fashioned the Tetrads of Media Effects for this purpose while Postman, who saw himself as a media ecologist, chose a set of six questions to flush out the qualities and effects of technologies. It is to Postman that we direct this post with the hope that it provides an example of how we might clarify the characteristics and effects of educational media. The interested reader will no doubt want to apply the Tetrad too since both approaches bring their own insights.

The first three questions that Postman raises help to clarify the inherent nature of a technology. The tone of the questions is somewhat sociological in tenor, possibly because Postman, like McLuhan, saw media as acting on and transforming environments and culture. Following Postman then, the first three question are (feel free to insert your favorite educational technology here):

 

Question 1: What is the problem to which this technology is a solution?

 

Question 2: Whose problem is it? (Who benefits from it and who pays for it? They are often different parties.)

 

Question 3: Suppose we solve this problem and solve it decisively, what new problems might be created because we have solved the problem?

 

You might find it almost impossible to resist the temptation to insert xMOOC in the above as the technology of interest, but if you prefer a warm-up in the “rear view mirror” try PowerPoint instead.

The last three questions are designed to provide focus on the preceding reflections. They are intended to be independent of political ideology and agenda.

 

Question 4: Which people and what institutions might be most seriously harmed by a technological solution?

 

Question 5: What changes in language are being enforced by new technologies and what is being gained and lost by such changes? (Think of the use and meaning of the word debate relative to the Lincoln-Douglas debates of 1858 as compared to those of the last election.)

 

Question 6: What sort of people and institutions acquire special economic and political power because of technological change? (Exploitation of a technology always results in a realignment of economic and political power.)

 

Although not properly qualifying as a seventh question Postman urges us to keep a tight grip on this one:

 

“Am I using the technology or is it using me?”

 

 

References

Hayton, Darin, “The MORU as Precursor to the MOOC“

Marshall McLuhan Biography (Wikipedia)

McLuhan’s Tetrad of Media Effects (Wikipedia)

Neil Postman Biography (Wikipedia)

Six Questions for Understanding Media (YouTube; see above)

Daniel, Sir John,  ”Making Sense of MOOCs: Musings in a Maze of Myth, Paradox and Possibility“

Weston, Mark, E., “How Education Fails Technology (And What to Do About It)“

Khan-On-Khan

Posted on by Jack McShea

khan-on-khan

Salman Khan, founder of the Khan Academy, responds to a request for a meta-tutorial on how to make Khan tutorials. In it he describes four guiding principles for making “KSVs,” or Khan Style Videos, as they are now called.

Maintain a Conversational Tone
Talk the way you would talk to another human being – the way you would actually have a conversation with another human being. Avoid highly scripted, highly polished, emotionless style. In Khan’s opinion this is not what human beings want to hear.

“The listener is highly sensitive to what is going on in the speaker’s brain. If the speaker’s brain is not thinking in real time – if he is just reading something – the listener’s cue is why should I be thinking it through? Why should I care?”

If the speaker does not care about the topic and betrays as much through his speaking style and voice, it’s a big cue to the listener not to care as well.

In addition to a lack of affect in the presentation Khan cautions that negative affect is a danger as well. Never talk above or below the listener. Don’t be patronizing. Don’t pontificate. Be respectful of the student but don’t talk above him. Speak as though you see that the student has come to the topic for understanding and mastery and you both agree to work through it together.

Use Colors and Visuals Sparingly
Khan feels that hand-drawn pictures and diagrams (which he greatly favors) resonate better with the brain. In addition the presenter benefits from more output: It takes more time to do computer graphics for presentations ahead of time than it does to construct sketches on the fly.

Prepare Your Mind
Make sure the concepts and ideas are “distilled” in your mind. Speak directly from your mind and your heart (once prepared; noting the emotion again, which Khan takes as a positive element). Interestingly Khan notes that while he is not a fan of scripts as an end product the writing can help prepare the mind.

Keep It Short
Initially forced by YouTube, Khan considers about ten minutes to be the right length to articulate a concept. Given a large topic, chop it to small pieces and make a separate presentation for each piece.

Thanks to Anya Kamenetz and Fast Company for bringing this presentation to the fore.

References.

4 TIPS FOR CREATING A SAL KHAN-STYLE INSTRUCTION VIDEO…FROM SAL KHAN

Making a KSV (video)

Flipping the Classroom Along the Other Axis

Posted on by Jack McShea
HorseFeathers, Groucho Marx

“And I say to you gentlemen that this college is a failure. The trouble is we’re neglecting football for education.”—Professor Quincy Adams Wagstaff

These two stories on the seismic changes underway in education could be footnotes to the post “Witch Hunt or Reformation?” but they seem to stand well on their own.

Anya Kamenetz writing for Fast Company recounts a note from Coursera’s Daphne Koller on putting the student first, a trend that will no doubt gather into an avalanche. The enrollment figures alone for Coursera and Udacity make an interesting statement regarding the numbers of people gravitating to alternative forms of education.

“From their experience teaching 100,000 students in ‘massively open online courses,’ Two groups of Stanford professors founded two rival startups, Coursera and Udacity, in 2012. Udacity’s Peter Norvig and Sebastian Thrun created all their courses in-house, while Coursera’s Daphne Koller and Andrew Ng partnered with leading universities to present their best professors’ stuff across all disciplines.

‘Our cardinal rule, our touchstone was ‘what’s the best for the students,’ says Koller. ‘Stanford alone? Or multiple top universities? Computer Science or all subjects? The choice was clear cut.’

By the fourth quarter of the year, Coursera had 33 university partners, over 300 courses, and 1.6 million students; Udacity had 14 courses & just 112,000 students.”

In the second story of note Salman Khan of the Khan Academy went on record with his vision of a new university – a self-paced learning environment, based on a flipped classroom model, that fosters engagement and doing in conjunction with guidance from accepted masters in the subject area. Interestingly Khan embraces the portfolio as the metric of achievement, not the exam or credit-hour.

Alisha Azevedo writing for Wired Campus:

“In a chapter titled ‘What College Could Be Like,’ Mr. Khan conjures an image of a new campus in Silicon Valley where students would spend their days working on internships and projects with mentors, and would continue their education with self-paced learning similar to that of Khan Academy. The students would attend ungraded seminars at night on art and literature, and the faculty would consist of professionals the students would work with as well as traditional professors.”

Further on:

“Although students would not be graded in the imagined university he describes, they would compile a portfolio of their work and assessments from their mentors.

‘Existing campuses could move in this direction by de-emphasizing or eliminating lecture-based courses, having their students more engaged in research and co-ops in the broader world, and having more faculty with broad backgrounds who show a deep desire to mentor students,’ he writes.”

Peter Thiel gets a nod as well from Khan in the Wired Campus post.

 

REFERENCES.

Kamenetz, Anya, “Coursera Co-CEO Daphne Koller On Putting Students First“, Fast Company, 26 November 2012.

Azevedo, Alisha, “Khan Academy Founder Proposes a New Type of College“, Chronicle of Higher Education, Wired Campus, 29 November 2012.

 

Getting Out Of The Way In The Classroom

Posted on by Jack McShea

MOOCs (Massive Open Online Courses) and Inverted (or Flipped) Classrooms have been attracting a lot of attention in education and training circles, but two recent experiments performed far from the rarified heights of North American campuses and training centers, are causing many to stop and reassess what it means to learn and, more importantly, what it means to learn when we are devoid of formal structure. If you have ever spent time wondering how humans learn naturally or informally you might be obliged to spend some time musing over these findings. I refer specifically to the “Hole In The Wall” experiments of Sugata Mitra and to Nicholas Negroponte and the One Laptop Per Child’s adventure with solar-powered tablet computers in Ethiopia.

In the case if Mitra, Wikipedia provides a summary of the “Hole In The
Wall Experiments”:

“In an experiment conducted first in 1999, known as Hole in the Wall (HIW) experiments in children’s learning … a computer was placed in a kiosk created within a wall in a slum at Kalkaji, Delhi and children were allowed to use it freely. The experiment aimed at proving that kids could be taught by computers very easily without any formal training. Sugata termed this as Minimally Invasive Education (MIE). The experiment has since been repeated at many places, HIW has more than 23 kiosks in rural India. In 2004 the experiment was also carried on in Cambodia.” -Wikipedia

More to the point, perhaps:

“This work demonstrated that groups of children, irrespective of who or where they are, can learn to use computers and the Internet on their own using public computers in open spaces such as roads and playgrounds, even without knowing English.” -Wikipedia

Mitra has a couple talks at TED (see below) where he describes the experiment and some of the results. They are worthwhile viewing if for no other reason than the sidebar comments from Mitra on what the kids learned, the degree to which they took the experience and the feedback he got from them on the technology.

More recently Nicholas Negroponte and the One Laptop Per Child organization (OLPC) have published results of what they see as a promising experiment in Ethiopia where solar-powered tablet computers were delivered to remote villages, preloaded with programs, and left to uncover what kids do with them. The main point in this instance is that there is no teacher, curriculum, or syllabus, just some software and a device to run it. The goal is to see if illiterate children will/can use the device to learn to read.

The articles referenced below give background to the experiment and preliminary conclusions to what was observed. The gist of it is:

“Earlier this year, OLPC workers dropped off closed boxes containing the tablets, taped shut, with no instruction. ‘I thought the kids would play with the boxes. Within four minutes, one kid not only opened the box, found the on-off switch … powered it up. Within five days, they were using 47 apps per child, per day. Within two weeks, they were singing ABC songs in the village, and within five months, they had hacked Android,’ Negroponte said. ‘Some idiot in our organization or in the Media Lab had disabled the camera, and they figured out the camera, and had hacked Android.’ ” -MIT Technology Review

The results might seem astounding to anyone steeped in the dogmatic lock-step pedagogy of the classroom. Comparing results of earlier (some say failed) experiments by OLPC, a notable variation in this case is the conspicuous absence of a “teacher” in the process, leading many to wonder whether the secret rests in preparing the environment, creating opportunities to learn, and then simply getting out of the way:

“I believe the second experiment is working because nobody is there trying hard to figure out how the new technology should fit into the old model of teaching and learning.

And nobody is trying to frame the learning experience through superficial content that the kids just don’t care about.

…It’s letting the kids discover what’s in the boxes. And how to get it out of the boxes. And why the boxes even matter in the first place.

It’s setting a goal, establishing an environment to realize the goal, and trusting in the capacity of human potential. Student potential.

And sometimes, it’s just getting out of their way.” -Ben Grey, The Edge of Tomorrow

None of this calls for the elimination of teachers per se. It does comment on the overall design of a process many in teaching and training take as a given, and the role of the “teacher” in that process.

References.
Wikipedia: Sugata Mitra
TED: The Child-Driven Education
TED: How Kids Teach Themselves
Wikipedia: One Laptop Per Child
Wikipedia: Massive Open Online Course
NYT: The Year of the MOOC – Massive Open Online Courses are Multiplying at a Rapid Pace
Feldstein, M., “Everybody Wants to MOOC the World“
MIT Technology Review: “Given Tablets but No Teachers, Ethiopian Children Teach Themselves“
Doctorow, Cory, “Illiterate kids given sealed boxes with tablets figure out how to use, master, and hack them“
Wikipedia: Minimally Invasive Education
OLPC News: “Who is to Blame for OLPC Peru’s Failure? An OLPC Intern Viewpoint“
Grey, Ben, “We need to think very, very seriously about this“

Why Is the Most Viewed Talk on TED an Education Talk?

Posted on by Jack McShea

Next time you are wrestling with the black dog of educational design loneliness, you might take comfort in the fact that TED Blog has a list of the top 20 most-watched videos on the TED web site and the #1 spot is held by Sir Ken Robinson‘s talk on how schools kill creativity.

Given how we (consciously or otherwise) use our personal educational experiences as templates for those that we create for others – and the fact that we have all been to “school” many times over – it might be helpful if we kept an eye on what we might tacitly project through our courses, materials and presentations. That is, old tired depleted wine poured into new bottles.

Why is Ken’s Robinson’s talk so popular? Probably because we have a deep-seated belief that it works. In Robinson’s words:

“I have an interest in education — actually, what I find is everybody has an interest in education … partly because it’s education that’s meant to take us into this future that we can’t grasp.” – Sir Ken Robinson

Robinson’s talk is expertly delivered and provides many thoughtful points to muse over. See the full presentation at the TED web site: Ken Robinson says schools kill creativity

References.

The 20 most-watched TED Talks to date

Witch Hunt or Reformation?

Posted on by Jack McShea

It used to be said of teachers that they are the perfect products of the system. They might have technical concerns over a fine point of a course or doubts over the administration of a program but, in general, they are the true believers. It’s no wonder then that so many are starting to paint Salman Khan (the Khan Academy) with an apple and a serpent’s body, while posing Sebastian Thrun (Udacity) at the main gate of Stanford University ready to to nail his Ninety-five Theses to the door.

Prior to the rise of protestants like Khan and Thrun educators only had to occasionally fend off irate parents, bad student evaluations, intractable administrators, myopic government programs and gnostic home-schoolers. Now it seems they have to deal with other similarly perfect products of the system – renegade graduates from schools like MIT and Harvard who work in international finance and venture capital, some married to medical doctors or lawyers, others captains of industry – who are not only willing to voice their discontent about what is going on in classrooms but also do something about it. And what makes these folks more difficult to handle than the more normal educational John the Baptists that have come before is that they are well-funded, articulate, determined to make changes, not known to easily lose interest, and they have the full attention of the Media. They are known for their grit and for winning. And they represent the desires and interests of a staggeringly large percentage of educational consumers who feel that there is something very wrong with the ways things are getting done. Is it any wonder that so many orthodox educators are having such a hard time reading the signs and separating the medium from the message? Does the name Custer mean anything to educators today?

What we are seeing among reformers at present are the beginnings of a consumer-oriented reformation of the educational system lead by some of the most notable products of the system itself.

And by “educational system” one can take anything that involves learning, training and credentialing as subject to the ensuing shift so there is probably no place to hide.

Pro-consumer trends in education are growing and emerging in curious ways. Too many people who have been through schools, colleges, degree and training programs have emerged disappointed and dissatisfied. The experience is costly, time consuming, and often without personal or professional reward. Too much of what passes for education or training is merely a ritualized data dump, designed and delivered more for the convenience of the department, institution or instructor than for the benefit of the student. No meaningful design can be discerned, no compelling delivery can be expected, and engagement is not in the least a part of the program. Result: tedium, no real change, and the assertion by some that many degrees and professional certifications have become little more than educational indulgences sold to customers who spend the requisite time miming what was in another place and another time an authentic educational practice. Adding insult to injury, costs continue to climb even while training and educational producers attempt to resell the old wine in the new information-age bottles of online learning, virtual classrooms and massive all-you-can-eat open source course buffets. Bear in mind that some of these educational advances are brought to us by the same folks who championed computer-based training in the nineties and television as an educational medium before that. No matter. We’re all here for the show. Educational theater at its best.

Compounding the problem and accentuating the disconnect between educational producers and their “consumers” is the fact that learners come from an world of media and communications that undermine many of the basic assumptions of the nineteenth century classroom. They do not come from a world where there is a shortage of information. Furthermore, information is not sequestered in specialized silos of higher knowledge. The stuff is everywhere and the job of the instructional designer and teacher is to make it easier to get at and assimilate. This one simple gesture of service might be where Salman Khan excels the most and finds his greatest success. People like his concise and casual video tutorials. Many (perhaps most) use them voluntarily, spend time on the lessons and exercises, and report progress in their learning. The customer is voting with his attention. What should we as educational producers learn from this?

Learners have immediate needs and pragmatic goals in mind. They are not looking for long-winded linear expositions of subjects that start at postulates and end atop pedantic minarets of “higher” understanding. They want the chunk that fits today – right now – into the empty pane of their mosaic understanding. And it probably can’t or shouldn’t last for more than ten minutes, as Khan would attest. What this implies is that one of the first jobs of a teacher is curation. Teachers have to know the subject well both in terms of its theoretical underpinnings and history through to its applications. They need to “own the material.” And part of that owning is knowing which parts matter, how they fit and how to adjust the presentation to the audience.

Curation is key. Know what matters. Make it relevant. Make it meaningful. Make it fit.

The pedagogical emphasis is flipped in post-reformation classrooms (if classrooms exist at all). Control is handed to the student and there is no one right way to complete a course of learning. The nature of the reversal from the focus on the master to the that of the student (“sage on the stage” vs. “guide by the side”) is hardly new. One can find allusions and references to it in sources as wide ranging as the American mythologist Joseph Campbell and the media theorist Marshall McLuhan. By way of illustration, Campbell, a life-long teacher, noted that in the tradition of the East it is the student who seeks out the teacher with his questions and problems in hand, while in the West it is the teacher (the subject matter expert) who hands down the knowledge to the passive vessel and asks all the questions. No doubt a very different undertaking in either case.

Marshall McLuhan, a teacher and professor of literature, saw the “Orientalizing” of the West, and indeed the educational process, as a natural outgrowth of life in an electronic medium. The new learner does not enjoy the point of view and safe linearity of typography. The digital native inhabits an informational landscape more akin to an acoustic space that surrounds him isotropically, issuing updates at the speed of light. We are informational hunter-gathers roaming over expanses of data. Information overload, McLuhan observed, reverses into pattern recognition, another name for curation.

The other push coming from the educational reformation underway is the trend towards involvement. It will shortly become the new tacit standard for educational design.

This is right in step with what we know about life in the electronic world and the inclinations of the digital natives. Courses and presentations that focus on dumping disconnected facts are out. They’re old hat. The flipped or inverted classroom is in and showing signs of paying off.

Fostering involvement in the learning process means using new information while building new skills. Involvement means solving compelling problems and making things that have not existed. Doing and making are the new metrics of achievement. Knowledge and ideas are wonderful things. Knowledge and ideas transformed into a 3D object are better. We can all see it, feel it, touch it and assess its virtues (now and five or ten years later). Not so with most degrees and certifications from disparate sources.

This should come as a comfort to those who feel threatened or displaced by the likes of the Khan Academy or Udacity. The new protestants are going to move into project-based and experiential learning because they have to, but they are not there yet even if you consider the pioneering efforts of Peter Thiel and the Thiel Fellowship. It is, after all, where the highest order skills and thinking are displayed and they know it.

In the world of the academy the Ph.D. is the terminal degree in most disciplines. But in Silicon Valley and the world of the venture capitalist the terminal degree is the successful startup. Ideas, after all, are a dime a dozen. Implementation separates the great from the ordinary and is critical to success. Everything else is preparatory and sidebar to doing. This is right in line with the consumer-oriented drive towards relevance and real-world metrics. People want to do. And what’s more, in the process the consumer becomes producer in an act of complete involvement.

References.
The Trouble with Khan Academy, by Robert Talbert.

Khan Academy: the Teachers Strike Back, Slashdot

How well does Khan Academy teach?, by Valerie Strauss

Why Corporate Training is Broken And How to Fix It, by Jay Cross

Sebastian Thrun Aims to Revolutionize University Education With Udacity, by Peter Murray

How Would You Like A Graduate Degree For $100?, by George Anders

PayPal Co-Founder Offers Students Scholarships To Leave College, by Robin Young, Here and Now

Peter Thiel Has New Initiative To Pay Kids To “Stop Out Of School”, by Robin Young

Thiel Fellowship Pays 24 Talented Students $100,000 Not to Attend College, by Ben Wieder

The Trouble With Online Education, by Mark Edmundsen

The Flipped Classroom Infographic
A new method of teaching is turning the traditional classroom on its head.

Academically Adrift, by Richard Arum and Josipa Roksa

Academically Adrift, review by Scott Jaschik

Academically Adrift, review by University of Chicago Press Books

Higher Education?: How Colleges Are Wasting Our Money and Failing Our Kids–and What We Can Do About It, by Andrew Hacker and Claudia Dreifus

Contingent Education?: How Colleges Are Wasting Our Money and Failing Our Kids, review by Scientific American

How Colleges Are Wasting Our Money and Failing Our Kids — and What We Can Do About It, review at Education News by Jimmy Kilpatrick

From MIT to Stanford, college classes where a startup is the final exam, by Christina Farr

A Core Curriculum To Create Engaged Entrepreneurs, by Cathy Davidson

Paper vs Pixels – Do Ebooks Make Learning More Difficult?

Posted on by Jack McShea

Let’s be frank. E-readers and e-books are pretty neat. They can pack an entire library into a portable gadget allowing the reader free range over multiple tomes or ready access to the complete works of a particular author. Passages can be searched, annotated, high-lighted and shared with other readers. But immediate advantages aside, do you have a nagging impression that the experience of reading an e-book is qualitatively different than that from a paper book? If so, you’re in good company. According to an article at Time Healthland even notables such as Larry Page of Google are having doubts about the new technology. Common complaints about e-books include increased fatigue, lack of attention span (engagement) and reduced reading speed. Not a pretty list considering the high hopes the technology holds in education and training circles as a new medium for textbooks, workbooks, technical papers and course content. To make matters worse, decreased recall and increased time-to-learn are being added to the list as further broadsides against the general deployment of the new medium as an educational tool.

According to Time Healthland author Maia Szalavitz, cognitive psychologist Kate Garland at the University of Leicester in the UK has determined that print has some subtle advantages over pixels when it comes to learning new information. As reported by Szalavitz, Garland and her colleagues found:

  • More repetition was required with computer reading to impart the same information
  • Book readers seemed to digest the material more fully

Szalavitz goes on to say that Garland takes the process of factual recollection to be either instant recall (“knowing”) or search and retrieval (“remembering”). Subjects who learned via paper books seemed to “know” the material sooner than those who used electronic media.

“What we found was that people on paper started to ‘know’ the material more quickly over the passage of time. It took longer and [required] more repeated testing to get into that knowing state [with the computer reading, but] eventually the people who did it on the computer caught up with the people who [were reading] on paper.” – Dr Kate Garland as quoted by TimeHealthland

A possible explanation of why paper books offer an advantage might involve contextual and navigational cues that are missing from e-books. E-books provide fewer spatial and navigational landmarks to readers and in so doing might subtly affect recall. Author Szalavitz:

“Context and landmarks may actually be important to going from ‘remembering’ to ‘knowing.’ The more associations a particular memory can trigger, the more easily it tends to be recalled. Consequently, seemingly irrelevant factors like remembering whether you read something at the top or the bottom of page — or whether it was on the right or left hand side of a two-page spread or near a graphic — can help cement material in mind.”

Add to the discussion the vast array of sizes, shapes, screen types and software designs that come into play in the design of e-books and e-readers and it is easy to see that the research possibilities explode. Whether concerns over the effectiveness of e-readers and e-books applies to online and web-based training is not discussed but begs to be considered as well. Let’s hope that Dr Garland and others continue the work on the cognitive differences between print and pixels. Some quantitative comparisons of the technologies would certainly be welcome. No doubt the topic of e-books will need to be revisited from time to time as technologies and applications unfold and grow in popularity.

 

References.

Do E-Books Make It Harder to Remember What You Just Read?

http://healthland.time.com/2012/03/14/do-e-books-impair-memory/

Larry Page, My Wife’s Lament, and Reading on Books vs. Screens

More Time With More Screens

The Fordham Experiment

Flatlined During Class

Posted on by Jack McShea

This published finding may not be scientifically significant (N=1) for some applications, but if nothing else it does provoke a chuckle (and a tendency to draw general conclusions without enough supporting data simply because – let’s face it – we’ve all been there).

A study published in the IEEE Transactions on Biomedical Engineering reports data from a wearable sensor attached to a student for a week. The portable sensor records the electrodermal activity of the wearer.

“Changes in skin conductance at the surface, referred to as electrodermal activity (EDA), reflect activity within the sympathetic axis of the Autonomic Nervous System (ANS) and provide a sensitive and convenient measure of assessing alterations in sympathetic arousal associated with emotion, cognition, and attention.”

Note the times at which the student is at class (yellow underscore).

No doubt this demonstration will be adopted by proponents of the Inverted Classroom and other related high-engagement learning techniques as an illustration of why the traditional lecture or classroom should be avoided. It is gratifying to note that activity levels during labs (yellow-green) and while doing homework and study (pink and red) are elevated. But then again, upon further inspection the sleep cycle is pretty impressive too.

One can hope this is followed by further investigations and discussions on the physiological and psychological meaning of the EDA waveforms given that the primary purpose of the paper is to report on Poh, Swenson and Picard’s work on sensor development.

 

References.

Poh, M.Z., Swenson, N.C., Picard, R.W., “A Wearable Sensor for Unobtrusive, Long-term Assessment of Electrodermal Activity,” IEEE Transactions on Biomedical Engineering, vol.57, no.5, pp.1243-1252, May 2010. doi: 10.1109/TBME.2009.2038487

Download a PDF of the report here.

Ito, Joi, “A week of a student’s electrodermal activity“ http://joi.ito.com/weblog/2012/04/30/a-week-of-a-stu.html

 

 

Shut Up and Teach – Or – Why Science Says the Lecture Is a Bad Idea

Posted on by Jack McShea

The notion of replacing or limiting the venerable lecture has been visited in earlier posts (The Inverted Classroom and The Future of the Lecture) but it seems the topic is far from exhausted. Recent research in cognitive psychology published in the journal Science points to another dimension in the problem of lecturing, namely, that people (read: our brains) do not remember much of what they hear in lectures. This may come as obvious to many students and conference attendees alike but this time it’s coming from investigative scientists who have the numbers to prove it.

Backing up a bit, suppose you were asked to design and deliver a class or training session that had to maximize educational outcome – meaning, it had to work as a learning tool more to the benefit of the students than the teacher – no holds are barred, and you knew of a technique that resulted in an 80% improvement over the traditional lecture method. Would you use that method? More to the point, could you justify not using it? Well that is what Deslauriers, Schelew and Wieman found (see Science article below) when they compared the lecture with a more interactive class they designed to teach physics. All things being equal, if you supplant the lecture with a presentation that is designed to work more in accord with how most people learn, test scores go from 41% for the garden-variety lecture class to 74% for the interactive class. Pretty impressive stuff.

So what is the nature of the design of the interactive class? Put simply, research in cognitive psychology suggests that learners will get better results if they use what they have just been given right away. The theme: Deliver new information, play with it, use it to solve problems, evaluate mastery of the skills and concepts, repeat as needed. Deslauriers, Schelew and Wieman’s physics students were hit repeatedly with questions during class that they had to answer with clickers. Students frequently worked in groups where they were challenged to use their new knowledge to solve problems. Lastly, the students were evaluated in part using two class tests rather than the traditional single mid-term exam.

Let’s make it clear, pouring the old wine in a new bottle does not make it sweeter. Content matters. Doing homework in class and listening to lectures at night is not “flipping the classroom.” Recording lectures and putting them on YouTube or iTunes U is no solution:

“A University of Maryland study of undergraduates found that after a physics lecture by a well-regarded professor, almost no students could provide a specific answer to the question, ‘What was the lecture you just heard about?’ A Kansas State University study found that after watching a video of a highly rated physics lecture, most students still incorrectly answered questions on the material.” — David H. Freeman, Discover Magazine

Even in the best cases of well-thought-out well-designed interactive classes some likely criticisms remain. There is an issue with the Hawthorne Effect that needs to be retired, but personal experience suggests that these findings are not surprising or unusual, at least in kind. Another question that surfaces is whether this kind of interactive class lends itself to subjects like literature, philosophy, history or political science. What are the limits of the approach?

Finally, we have to ask why if there is so much evidence and personal experience against lectures do we persist in giving them? The answer might well be wrapped in four prominent qualities of the practice: 1) lecturing is easy and cheap to do; 2) we have been taught to accept bad lectures as normal (for well over a thousand years!); 3), they (certainly the live version) create an illusion of interactivity between the presenter and audience that is not supported in actual observation (see D. Clark below); and 4), they stand as proof by the presenter and/or the institution that the material has been covered and “delivered” to the audience.

Pragmatically, and for the reasons above, lectures inherently favor the presenter and the institution. Lectures originated in a time when books and information were both scarce and expensive and colleges needed to solve a problem of distribution. Closer to the modern era lectures appear to be supported by tacit agreement with the dubious notion that teaching and telling are the same thing:

“The problem is not with the lecture but with the idea that receiving information is the key part of learning.” — Dominik Lukeš

The notion that the lecture’s time has come is finally reaching the Academy. Educators like Graham Gibbs (see below) have been questioning its value for over thirty years. More recently university professors like Stanford University’s (formerly) Sebastian Thrun have had their own epiphanies on the matter:

“Mr. Thrun told the crowd his move [away from Stanford] was motivated in part by teaching practices that evolved too slowly to be effective. During the era when universities were born, ‘the lecture was the most effective way to convey information. We had the industrialization, we had the invention of celluloid, of digital media, and, miraculously, professors today teach exactly the same way they taught a thousand years ago,’ he said.” — Nick DeSantis, Wired Campus

Dr Wieman likewise has his own concerns about his colleagues and the future of the lecture in science instruction. As recorded by David Freeman of Discover Magazine:

“But scientists who teach have proven reluctant to toss out the lecture, never mind the evidence that it doesn’t work. ‘They say this is the way it’s always been done, and it was good enough for them, so it’s good enough for their students,’ Wieman says. Were this attitude to hold in medicine we would still be bloodletting, in physics we would be trying to reach the moon with very large rubber bands, and in economics we would still be suffering major worldwide financial crashes. (Well, physics and medicine are advancing, anyway.)” — David H. Freeman, Discover Magazine

What seems certain is that we are on the foothills of a major shift in what happens in the classroom. What develops in terms of the effects on corporate, college and military training remains to be seen. After all, it might not result in a single universal one-size-fits-all form. How this upheaval in teaching feeds into distance learning and web-based training is another discussion that almost certainly has to rear its head. The resultant form of the instructional process is anybody’s guess, but what is certain is that whatever it evolves into, whatever we see as the best fit for our instructional purpose, teaching well will remain hard work.

References.
Freeman, David, H., Impatient Futurist: Science Finds a Better Way to Teach Science

Louis Deslauriers, et al., Improved Learning in a Large-Enrollment Physics Class, Science 332, 862 (2011); DOI: 10.1126/science.1201783 (PDF)

Gibbs, G., “Twenty Terrible Reasons for Lecturing,” SCED Occasional Paper No. 8, Birmingham. 1981.

Clark, Donald, “Don’t Lecture Me” – ALT-C 2010.

Clark, Donald, “Lectures selling students short: evidence from ‘Science’ “

Lukeš, Dominik, “Putting lectures in their place with cautious optimism“

DeSantis, Nick, “Tenured Professor Departs Stanford U., Hoping to Teach 500,000 Students at Online Start-Up“

Deslauriers, Loius, Schelew, Ellen and Wieman, Carl, “Improved Learning in a Large-Enrollment Physics Class” Science 13 May 2011: Vol. 332 no. 6031 pp. 862-864

 

The Curvilinear Classroom – Is Linearity Optional?

Posted on by Jack McShea

AllThingsD Early Adopters ran a quote in their Voices section from an article at PCPro that reads like a page right out of Marshall McLuhan. Echoing McLuhan’s return of acoustic space and the role of the mosaic in everyday life, Dr Rosie Flewitt of the Open University comments on how the modern learner might be shifting from sequential linearity toward a simultaneous gestalt:

“E-learning experts argue that withholding computers at a young age could actually deprive children of modern communications skills. ‘One area of literacy that’s changing is the order in which things are presented – it isn’t linear, it’s organised spatially, and often some meaning is carried in the design, layout, images, sounds, movement, subtle changes in colour in a game – it’s all part of what literacy is in today’s world,’ says Flewitt. ‘These are fundamental changes to operational literacy, the biggest since the printing press.‘ ”

Naturally some question is left as to whether this effect is limited to young children as a group or if one can detect a tendency toward acoustic involvement among younger participants in college classrooms and corporate training centers. The main point, however, is that linearity might already be optional in the classroom, where new and different styles of presentation and involvement might be called for in order to better reach the audience.

To contrast Dr Flewitt’s comment on linear versus spatial literacy, consider this synopsis of McLuhan’s acoustic space by Library and Archives Canada:

“The key characteristic of acoustic space is that it engages multiple senses at the same time. It does not demand that objects be dissected to be understood; rather, the multiple parts co-exist simultaneously. To understand acoustic space, you must perceive all of it, not focus on one part. In other words, acoustic space demands that you apprehend figure and ground simultaneously, that the senses work together. McLuhan believed that oral cultures existed in acoustic space since their primary mode of communicating was speech.”

In this interview with Nina Sutton, Mcluhan explains the rise and dominance of visual space from the phonetic alphabet forward: McLuhan on Acoustic Space.

As a sidebar it is interesting to note that McLuhan eventually dropped the use of the term Global Village from his work preferring the term Global Theatre instead. Apparently Global Village goes back to the advent of radio while the notion of the Global Theatre is more a part of Sputnik, television and modern global communications.

References.

AllThingsD: Early Adopters

PCPro: How Much Tech Can Children Take?

Library and Archives Canada: Old Messengers, New Media: The Legacy of Innis and McLuhan

McLuhan, Marshall. The Gutenberg Galaxy. Toronto: University of Toronto Press, 2011.

The Playboy Interview: Marshall McLuhan, Playboy Magazine (©1969, 1994) by Playboy. Download here in PDF: (mcluhan-playboy).

 

Maybe This Is What’s Missing – Or – It’s Only Work If You’d Rather Be Doing Something Else

Posted on by Jack McShea

Source: MIT

Substitute (instructional design | course development | teaching | writing | learning) where you see “research” and “physics” in the excerpts below.

 

“But when it came time to do some research. I couldn’t get to work. I was a little tired; I was not interested; I couldn’t do research!… And then I thought to myself, “You know, what they think of you is so fantastic, it’s impossible to live up to it. You have no responsibility to live up to it!”… Then I had another thought; Physics disgusts me a little bit now, but I used to enjoy doing physics. Why did I enjoy it? I used to play with it. I used to do whatever I felt like doing – it didn’t have to do with whether it was important for the development of nuclear physics…. So I get this new attitude… I’m going to play with physics, whenever I want to, without worrying about any importance whatsoever. Withing a week I was in the cafeteria and some guy, fooling around, throws a plate in the air…. I had nothing to do, so I start to figure out the motion of the rotating plate… And before I knew it (it was a very short time) I was ‘playing’ – working, really – with the same old problem that I loved so much, that I had stopped working on when I went to Los Alamos; my thesis-type problems; all those old-fashioned wonderful things. It was effortless. It was easy to play with these things. It was like uncorking a bottle: Everything flowed out effortlessly…. There was no importance to what I was doing, but ultimately there was. The diagrams and the whole business that I got the Nobel Prize for came from that piddling around with the wobbling plate.” – Richard Feynman, excerpts from Surely You’re Joking, Mr Feynman

 

There. How does that make you feel? More importantly, how does that make you feel about your work? Worth a try, isn’t it? After all, if Feynman had not realized the connection between play and learning he might have stayed in funk much longer and, perhaps, missed the opportunity to experience the pleasure in discovering something really interesting. (I am tempted to end the last sentence with something like “…that ultimately led to a Nobel Prize in Physics,” but I know enough about Richard Feynman to avoid that one. He would be quick to say that nothing he did was ever about winning a prize.)

 

Attribution

Miki at PythonWise

References

Surely You’re Joking, Mr. Feynman (PDF)

Surely You’re Joking, Mr. Feynman! (Adventures of a Curious Character)

The Pleasure of Finding Things Out: The Best Short Works of Richard P. Feynman

The Feynman Lectures on Physics

Six Easy Pieces: Essentials of Physics Explained by Its Most Brilliant Teacher

Related Posts

Cargo Cult Science and Education

 

 

The Eye of the Beholder – Why We Prefer Rounded Corners Over Sharp Edges

Posted on by Jack McShea

Rounded rectangles are everywhere. You might think the reason they are so ubiquitous is because web and product designers’ minds are being controlled by an alien graphical design style ray that shows little chance of letting go. Or, maybe not. Beauty, in the case of the rounded rectangle, might be in the eye of the beholder – literally.

Apparently the visual system favors rectangles with rounded corners, making layouts, interfaces and presentation graphics easier to view and take in. Having a hard time believing that rounded corners make a difference, try this. Look at the images below. Which is easier to look at?

Attribution: uxmovement.com

The reason the circle appears more agreeable is because we are wired to prefer round to sharp edges (and by extension round to sharp things). Keith Lang at UI&Us quotes researcher Jürg Nänni on the eye-brain’s peculiar penchant for roundness:

A rectangle with sharp edges takes indeed a little bit more cognitive visible effort than for example an ellipse of the same size. Our ‘fovea-eye’ is even faster in recording a circle. Edges involve additional neuronal image tools. The process is therefore slowed down. – Professor Jürg Nänni as quoted by Keith Lang (see below)

Anthony Tseng at UX Movement presents two other examples where rounded corners aid and abet the perception of graphical information. The box diagram is a common graphical type used in organization charts and process diagrams. Note the differences between the rectangular and rounded lines. The curves add flow to the procession through the diagram.

Attribution: FMC Visualization Guidelines

In a second example Anthony Tseng shows how rounded corners not only guide the eyes but also act on the attention of the viewer. In what might be a great tip for instructional designers and artists notice how the use of the corner radius acts to focus attention on what is inside the boxes.

 

Attribution: Anthony Tseng

Rounded corners also make effective content containers. This is because the rounded corners point inward towards the center of the rectangle. This puts the focus on the contents inside the rectangle. – Anthony Tseng at uxmovement.com

 

Still wondering why we see so many rounded rectangles in objects around us?

Attribution: UI&Us

 

References

Tseng, Anthony, “Why Rounded Corners are Easier on the Eyes“

Lang, Keith, “Realizations of Rounded Rectangles“

FMC Visualization Guidelines

At a Loss for Words – The Future of the Lecture Might Be in Less Talk

Posted on by Jack McShea

Silentium - Latin for "Shut Up & Pay Attention"

A recent study from researchers Louis Deslauriers, Ellen Schelew and Nobel Laureate Carl Wieman suggests that the Methuselah of instructional technologies, the venerable broadcast lecture, might finally be showing signs of going the way of geocentricity and the four humors. Applying methods taken from the theory of “deliberate practice” by psychologist Anders Ericsson, the research team introduced a more interactive, discussion-based and assessment-oriented approach to a physics class that strongly implies major improvements to science and engineering instruction in general.

The setting for the study involves two groups of electromagnetics students (control: 267; test: 271) wherein both were given the same learning objectives and enjoyed the same pedagogical approach (but not the same instructors) for the first 11 weeks of instruction. On week 12, Deslauriers and Schelew (both of whom have limited teaching experience) jumped into the fray and according to the BPS Research Digest lead the test group utilizing “…discussions in small groups, group tasks, quizzes on pre-class reading, clicker questions (each student answers questions using an electronic device that feeds their answers back to the teacher), and instructor feedback.” And, what is especially important to note here: there was no formal lecturing. According to the researchers the object of the game was:

“…to have the students spend all their time in class engaged in deliberate practice at ‘thinking scientifically’ in the form of making and testing predictions and arguments about the relevant topics, solving problems, and critiquing their own reasoning and that of others.”

In contrast to the test group, the control group went on learning the same material in the normal (typically passive) fashion epitomized by classroom lectures for probably the last 900 years. The students, however, apparently noticed a difference. As quoted in the BPS review:

“Student engagement (measured by trained observers) and attendance in the control group was unchanged in week 12 compared with earlier weeks. In the intervention group, attendance rose by 20 per cent and engagement nearly doubled.“

The critic or cynic might assert that the presenters were putting on a better show in the test case. What about student performance? On the first day of class after week 12 both groups were tested on what they had learned the previous week. In addition, as part of the preparation for the test, both groups were given all the materials used by the intervention group, i.e., the clicker questions, group activities and problem sets, and exercise solutions. The results are as striking as the jump in student engagement:

The non-lecture intervention group averaged 74 percent correct while the control group averaged 41 percent. Factoring out random guessing, the intervention group did twice as well as the traditional lecture students (the effect size being on the order of 2.5 standard deviations!). Not to be downplayed, student reviews rated the non-lecture approach very positively. Ninety percent said they enjoyed the process.

Jeffrey Mervis writing for the AAAS ScienceNow magazine quotes Wieman as saying:

“ ‘It’s almost certainly the case that lectures have been ineffective for centuries. But now we’ve figured out a better way to teach’ that makes students an active participant in the process, Wieman says. Cognitive scientists have found that ‘learning only happens when you have this intense engagement,’ he adds. ‘It seems to be a property of the human brain.’ ” – Jeffrey Mervis, A Better Way to Teach?

Given the novelty of the technique and the overt nature of the study there has been some criticism of the results based on the Hawthorne Effect. The research team discounts this criticism on the basis that the intervention only occupied a small percentage of the students’ overall daily learning activities. Drilling a little deeper, psychology professor Daniel Willingham (as recounted in Carey below) cautioned that the study might not have been designed well enough to discern which of the factors introduced in the new classroom style account for the gains in student performance and to what degree.

In what might be one of the clearest victories for proponents of the Inverted Classroom the research team is optimistic of the result and reckons it can be generalized to a wide range of post-secondary courses. No doubt further studies can be expected. The study in question is supported by a $12 million dollar program to investigate new methods to enhance science instruction using research-backed methods.

References.

Deslauriers, L., Schelew, E., and Wieman, C. (2011). Improved Learning in a Large-Enrollment Physics Class. Science, 332 (6031), 862-864 DOI: 10.1126/science.1201783

Carey, Benedict (2011). Less Talk, More Action: Improving Science Learning
http://www.nytimes.com/2011/05/13/science/13teach.html

Mervis, Jeffrey (2011). A Better Way to Teach?
http://news.sciencemag.org/sciencenow/2011/05/a-better-way-to-teach.html

Dwyer, Liz (2011). Research Proves College Lectures Need to Go the Way of the Dinosaur
http://www.good.is/post/research-proves-college-lectures-need-to-go-the-way-of-the-dinosaur/

Expert Performance and Deliberate Practice
http://www.psy.fsu.edu/faculty/ericsson/ericsson.exp.perf.html

The Inverted Classroom
http://www.hg2s.com/blog/2009/11/14/the-inverted-classroom/

Is the Internet Changing the Ways We Learn?

Posted on by Jack McShea

Is the Internet Changing the Way We Learn?

I like the infographic (see below) “How the Internet is Revolutionizing Education.” It presents an interesting timeline of developments in educational delivery and provides a handy reminder of some things that I’ve forgotten with regard to trends and current industry buzz. And yes, education in all its forms is an industry and has its buzz.

First, looking at the history of distance learning and non-traditional (i.e., non-lecture/classroom) modes of content delivery, writers rarely admit anything that comes before television. Frankly, I never see radio mentioned. Maybe that’s because to most, television is the first “modern” technology. But, that said, there is a long tradition (Boston, 1728) of correspondence education that rises through the Victorian Era (University of London, 1858) that seems important in laying the groundwork of several notable non-residential K-12 programs and even, I suspect, to the acceptance of modern online universities. Perhaps not surprisingly, the British Open University is the first school on the graphic to enter the fray in the early 1970s using television as its primary mode of disseminating lectures to the masses. Funny how television never really materialized as a great training tool. In retrospect, is that surprising?

This is the first of my jogged memories from the chart:

The UO (as it is known) makes perfect sense to me but when it was tried in the US it failed miserably (not so in the UK). You can read about it here. For those of you wondering, the flip side is also true in my case: Schools like the University of Phoenix (as they currently exist) do not make perfect sense to me and yet they are thriving in the US (scroll down the chart a bit), educational bubble notwithstanding. So much for my role as an industry pundit. In ancient times I would have been stoned to death.

Overall this graphic fosters a meme that I consider somewhat dubious: the Internet is changing the way we learn. I think you have to be careful with this one. First and foremost it is probably not the case that we are learning any differently than our forefathers. It is probably the case that we utilize new and different methods for obtaining information, gathering the rudiments of new skills and assessing our mastery of a subject or topic. But beyond that the need for engagement, practice, recall and synthesis seem to be standard among members of our species. As an example of some old wine in a new bottle masquerading as a new instructional form witness the Khan Academy. Is the actual process of learning  – that is, the embedding of new knowledge or skills – any different here? Does it have to be to be important? This brings me to the second reminder:

Given the explosion in alternate forms of content delivery, I don’t know anyone who would go to a traditional college or classroom as a first choice.

Isn’t that odd? I have to confess, if I had to bone up on linear algebra or differential equations, say, I’d go to the Khan Academy (note mathematics as the example) or the Open Courseware Consortium, not to the local college. What’s that tell you about: a) my prior experience at university, b) the reputation of quality of American higher/continued education, c) the role of technology in my lifestyle, d) my lifestyle, e) the cost and accessibility of higher/continued education in America, f) the fact that too many of us have (had to) become consumer-oriented with regard to our learning (in contrast to our “certification”). Take your pick.

Interestingly, given the apparent rise of e-learning since 1999, you would think that we have a viable alternative to instructor-lead training in online web-based tutorials. On the whole nothing could be farther from the truth. Sadly, even though traditional classroom instruction might be foundering as designers search for compelling new forms to save live presentations, it’s hard to find evidence that e-learning as a genre ever succeeded in a big way. Taken as a whole completion rates for online courses are and have been deplorable, levels of engagement minimal (possibly explaining the previous point), and even when they are completed online courses don’t often meet educational objectives except in the most superficial ways when compared to control groups. E-learning does have some notable traits that distinguish it in the pantheon of educational delivery methods: it is a cost-effective way to broadcast information to a population that might have geographical and temporal constraints; and, it does drive consumers to virtual and brick-and-mortar classrooms when provided as an option.

Frankly, if I were saddled with the task of saving e-learning, I would go to YouTube.

Maybe what that says is that even though Television failed as an educational panacea in the early days of distance learning, Son of Television is back, bigger and better than ever before. But does any of this change what I have to do to learn linear algebra? Enough said.

How the Internet is Revolutionizing Education

Via: OnlineEducation.net

It’s All Up From Here – The Worst PowerPoint Slides of 2011

Posted on by Jack McShea
The Infocus 'What Not To Present' Contest

Infocus 'What Not To Present' Contest

It’s not clear whether the presentation experts at InFocus Labs have opened a Pandora’s Box with this event, but their “What Not To Present“ contest apparently overwhelmed even their most staid judges in terms of popular response and the degree to which things can sink and still be considered acceptable. The response from the field was both daunting and gratifying:

“Our ‘What Not to Present’ contest was epic! Many thanks to all of you kind folks that submitted entries and spread the word about it. Many amazingly horrendous slides were sent in from all around the world. We laughed. We cried. We cringed.”

Naturally once the floodgates were opened the selection of a winner was not at all an easy task.

“We randomly chose our top 3 winners, but then quickly realized that we had to do more. So we are giving away ANOTHER projector to the slide we thought was the most horrendous. We passed the ugliness around the InFocus offices and to many of our partners pandering for votes – and we have a winner!”

Prizes generously include InFocus projectors and accessories.

Suspense mounting? Here’s the First Place winner from the random selection round:

First Place Random Round

See yourself in that slide? Me too. Kind of makes me cringe. Hopefully we all have slides like that locked securely in our pasts.

 

But that’s not the point of the “What Not to Present” competition. The good folks at InFocus must surely be sick and tired of their excellent products being associated with – one might even say equated to – the kind of visual flotsam that populated this contest. And they’d like it to stop. So, in an ongoing effort to assist in cultivating our design and presentation senses they are going to offer ongoing therapy to the readers of their blog wherein experts Garr Reynolds and Ellen Finkelstein will offer free advice on how to make presentations attract our attentions for all the right reasons. So, stay tuned. In the meantime, and capturing a feeling right at home on these pages, Ellen Finkelstein offers a few tips on how to avoid being submitted as a contestant in next year’s “What Not to Present” contest: Have Compassion on Your Audience!

 

Now, ready for this year’s Grand Prize Winner of the worldwide InFocus “What Not to Present” competition? Here you go.

 

IT Modernization Roadmap to the depths of hell

 

Further Reading

PowerPoint Overload – Two Pounds of Sausage in a One Pound Bag

“The Cognitive Style of PowerPoint: Pitching Out Corrupts Within”

“PowerPoint Does Rocket Science–and Better Techniques for Technical Reports”

Design for social change?