Tag Archive for research methods

Pizza Box Maps and Game Ephemera

April 3, 2012 Games 2 comments

I just found out about the Play Generated Map and Document Archive, which organizes and preserves role-playing game ephemera like character sheets, maps, and campaign notes. (So, you know, more or less what it says on the tin.)

From the site:

PlaGMaDA’s mission is to preserve, present, and interpret play generated cultural artifacts, namely manuscripts and drawings created to communicate a shared imaginative space.  The Archive will solicit, collect, describe, and publicly display these documents so as to demonstrate their relevance, presenting them as both a historical record of a revolutionary period of experimental play and as aesthetic objects in their own right.  By fostering discussion and educating the public, it is hoped that the folkways which generate these documents can be encouraged and preserved for future generations.

When it comes to role-playing games, I’m not just a researcher – I’m also a player. This project makes me deeply happy on both fronts.

As a researcher, I’m quite interested in the question of how we get good data about the experience of role-playing. When I conduct my own research, I rely on direct observation and interviews, but I also look at precisely this kind of ephemera to understand how the group works and what they jointly agree to pay attention to. I’ve looked at emails, game websites, maps, character sheets, game-related fiction, art, and more – and I’ve found they all illuminate what actually goes on at the table, not to mention being valuable to analyze in and of themselves.

Until now, there hasn’t been a great centralized resource for this material. I’ve been acquiring it on an ad-hoc basis through personal connections with groups, and developing my own system for categorizing and analyzing it. This, obviously, only goes so far. I want big data, dammit!

As a player, I find myself struggling to document and archive my play experiences. In our group’s long-term games, we generally keep a world-building wiki and write up session notes after each time we meet, but that has limitations. One of them is that it doesn’t include precisely this sort of ephemera. For example, at the end of our last long-term game, we moved from sketching maps on the backs of pizza boxes to using a whiteboard laid across the table. We made the change for a variety of reasons – one group member got a free whiteboard, we had to run an epic combat or two – but there’s no record of it except in our group’s heads.

I’ve got other documentation problems, too. I basically don’t document our short-term or one-shot games; there’s a significant barrier to recording and explaining what happens, especially since short games often include people outside our core group who are less committed to the preservation of play experiences. Plus, I’ve been playing one-on-one games with my husband for more than a decade; we’ll casually drop into and out of play as part of the fabric of our lives, and that certainly doesn’t lend itself to recording without an enormous investment of time and effort.

I’m not sure PlaGMaDa solves all my player-side problems, but it certainly helps.

If your group produces neat material, you should submit it to PlaGMaDa – and if it’s really neat, you should also drop me a line so I can make sure to have a look at it!

Which Test?

May 25, 2010 Creativity, Think Tank One comment

As a researcher, I’ve wanted a site like whichtest.info for ages. By answering a few simple questions about your data sample, the site helps you figure out which statistical methods to apply. It’s not a replacement for stats classes, but it’s an incredibly helpful supplement. (Or, if you’re like me, it’s a good way to confirm that you’re actually doing what you meant to!)

It’s also reminded me that there’s lots of easy, relatively low-tech tools for thinking that no one’s built yet, because no one’s seen the money in it – or has gotten sufficiently annoyed to roll their own. Just last week I had to pull out my stats textbook to check something and got really irritated by how long it took me. Next time I’m annoyed at something in my daily life, I’ll pay attention!

Valve’s Play-Testing Methodologies

April 30, 2010 Games, Lit No comments

I was incredibly excited to come across this presentation on Valve’s approach to play-testing.  For those who don’t want to page through the whole PDF, they go over the advantages and limitations of many different methods of gathering data: observation, talk-alouds, surveys, in-game data collection, physiological measures and more.  Each method is good at getting at certain kinds of data, and you have to think about the game design problem you’re trying to solve when you choose which one to use.

It doesn’t surprise me that Valve does a great job with play-testing; their games are super-polished!  What did surprise me, though, was how similar their analysis was to what I learned in my research methods classes.  There’s a lot of talk about how far apart academia and industry are, but this particular area seems very closely aligned.

I was, of course, also reading with an eye to the data collection we’re doing for Lit.  As we start play-testing our first digital prototypes, there’s that moment of “Oh, man!  We’re a bunch of academics.  What the hell are we doing?”  The answer, it seems, is learning from the best that’s out there.  Rock.

Headlamp and Online Data

November 13, 2009 Advance, Think Tank No comments

When I was thinking about Advance, I decided that I wanted to automate a lot of my data collection. I made this decision for a variety of reasons, some theoretical (if I keep the data collection low-key, it won’t interfere with player behavior) and some practical (more time spent on R&D, less on entering and coding data). Most important, I wanted to be able to distribute my game online and reach a broad population, but still be able to collect sophisticated and subtle data.

Turns out this is actually hard. Who knew?

I’m not surprised that automated online data collection has its own set of challenges, but I’m a bit surprised by what some of those challenges have been. I keep running into fairly simple things I want to do that aren’t well-supported by existing tools. Counterbalancing presentation of tasks. Randomizing subject assignment to research condition. Conditional pre- and post-test support. Complex tasks combined with surveys. Some tools have some of these features, but I haven’t found any that have all of them.

Enter Headlamp Research, which is designing tools for people to do research online. They haven’t released their toolset yet, but I just took their “What do we need to be doing?” survey and was really impressed. If nothing else, they’re asking all the right questions.

They’re not relevant for the work I’m doing on Advance, because I’m rolling my own tools into the game itself – but by the time I’m ready to begin a new study, it seems like they’ll have some really powerful tools available, plus a user population already in place. Self-selection effects could be problematic (after all, you’re only testing the kinds of people who sign up to do research online!) but I find their approach really inspiring.

Take Our Surveys!

October 14, 2009 Lit No comments

As part of our design research for Lit, we’re exploring what game features make people feel “rushed” (excited/stimulated) and relaxed.  We’ve put together two brief surveys, one for each emotional state.

Take the rush survey.

Take the relax survey.

Thanks in advance for your help!

Reorientation to Research

March 25, 2009 Advance No comments

One of the hardest things about this project, for me, has been balancing its many threads: reading, research, coding, writing, design.  Each of these could easily be a full-time job, so how do I balance them?  Not only do I have to figure out how to allocate my own time, I also have to provide leadership for the other people on the project.  No two of us have the same skills, nor the same amount of time to commit, nor the same things we want to get out of the project.  This makes it an even harder problem.

This is not something I’m particularly good at, and it’s been a big challenge so far.  While I’ve done a good job of setting aside lots of time to work on this project – I do something for it every single day, come hell or high water! – I tend to get very focused on one aspect of it at a time.  When I’m doing development, I just want to keep developing.  Same if I’m doing design, or reading articles, or writing.  This is not great for making sure all the parts of the project move ahead together!

To address this, I’ve decided to start evaluating each area of the project in the context of our upcoming deadlines.  For example, I’m giving a talk in research practicum next month.  I sat down to make an outline and realized that, while we’ve got a lot of design and development progress to report, we don’t have as many new articles to discuss as I’d like.  

Okay, so I’d identified a challenge, but I wasn’t sure how to address it.  Doing more reading and research is crucial for us, but we can’t afford to take time away from design and development if we want to be able to run a pilot this summer. 

Pazit, however, is an experienced project manager, and she had some really good suggestions about how to handle this.  “Do more research” isn’t exactly an actionable goal – but I have a big list of references I want to read and I know how I want to incorporate them into the project.  She suggested that we each agree to read one article a week from now until next month’s presentation, and discuss how they fit our work during our group meetings.  That’s something we can totally do alongside our existing commitments to the project!  I think the discussion is also going to be key, because I think a lot of good ideas (and an even longer reading list!) will come out of our conversations.

This is my first time leading a research team, so I’m always on the alert for learning experiences like this one.  I’m really excited about the research progress we’ll be making over the next few weeks, and about my new techniques for managing a multi-threaded project as well!