Article Index

C2Someone asked, “What makes a good roleplaying questionnaire?” My immediate answer would have been: one where an NPC questions a PC, or a PC analyzes NPCs' answers. But I also wonder: good for whom, or good for what? Showing a sample would be pointless, so let us have an article, or a series. Even this first installment of our longer series will be a monster spanning at least four or five pages. Look left at the mad cultist. He could fit into the Ministry of Silly Walks: it takes an unusual gait to keep that barbed wire from slicing his thighs. But he trusts that if the spikes point outward and he brings his knee rather forcefully into contact with one of your sensitive spots, it will hurt you more. The questionnaire in his hand is another game, directing your thoughts toward a subject in the hope you too become a mad cultist. Would a social scientist praise his manipulative questionnaire? But if the party examines questionnaire results during an investigation to narrow down suspects, the adventure needs a good questionnaire, not a manipulative one. Seeing many of these gets you accustomed to questionnaires and teaches you to ask good questions and use the technical tools. Then you can ask better questions about roleplaying too. Let us move from general explanation toward specifics.

Most of us roleplayers are not scientists, especially not sociologists. I hope my coauthor Sol, who is both, writes the continuation and adds more considerations, with recommended reading adding still more. But you will not learn questionnaire design from this alone. The topic is currently prominent partly because of M.A.G.U.S., though there was an FRPG questionnaire too. One worse than the next. Yet we put these articles in the Call of Cthulhu category. I considered Shadowrun and rejected it because its tools differ. Primarily, we want to explore not what you can ask roleplayers, but two important subjects: how questionnaires and related research can appear within game worlds and how to handle them as a GM; and situations where a questionnaire becomes a tool for running the game rather than an opinion poll, for example when assigning roles within the party. 

The first basic point is that the questioner's goals matter. That sentence immediately needs explanation: an explanation that seems obvious, unnecessarily explicit, and self-evident while reading it. Yet when making a questionnaire without reading it, those responsible usually forget all these obvious things. They thereby give up many benefits, never considering that a questionnaire or vote could achieve more than one goal. Imagine a familiar but primitive situation, starting with a simple poll. A picture shows five characters, perhaps roughly drawn. The picture and post ask which we prefer playing: Like if all are equally good, Love for the priest, Angry for the tough militant paladin, Sad for the monk, Wow for the believer, and Laugh for the atheist mocking faith. 

Questions like this appear often, and yield considerable data. This is only one question, not a completed questionnaire, but we have many and people answer each in turn. Facebook makes the data needed to link responses difficult to obtain, but if regular followers use our app, everything can be linked for them. What does the questioner gain or lose? Some questions never reach particular respondents, and not everyone's answers can be linked. On the other hand, Facebook records respondents' interactions with the post and shows that page's posts more frequently. Alongside analyzable answers, this is often a goal, perhaps the more important one. The approach is light and playful, with many ways to engage people. A serious social-media expert can achieve a great deal using Like statistics, visits, app-use data, and information from Facebook.

Besides answers and comments, it measures which posts you interact with. Two questions daily for 365 days can build quite a database. A large publisher with an online service and access to all your reaction data through Facebook registration has a winning combination. Best of all, it even measures functionally illiterate people's answers! Unlike your character, though, you will not have such an app or access to everything through the Graph API, so you do not start on equal terms. An economic power linked to certain political sides might, having stated that it will not sell your personal data while allowing aggregated anonymous statistics. That is the better case. In any event, correlations between keyword combinations and reactions—likes, shares, keywords in comments, and other Facebook reactions—are measured too. Some companies' privacy policies are less friendly. You can likewise forget interviewers and telephone interviews. Nor are you likely to filter responses by spending recorded in a loyalty-program database. An in-character questionnaire and an out-of-character questionnaire about the game are entirely different.

As we might guess, someone beginning a consultation with that database does not seek to learn our opinions. They knew them long ago; their policy aims to maximize votes. The questionnaire has two goals: prompting interaction, and making you feel the expected decision reached here is your own, building loyalty to a brand. The target group consists of sympathetic people susceptible to this influence. Many roleplaying questionnaires in the world are advertising like this too. When the questioner genuinely wants to know something, matters become harder. Let me introduce that with an excerpt from a conversation in my story Are You Afraid of Good? 

“Apparently Dezső and János know each other from here too. Dezső recommended the concert to János. Judging by his plans, he thought he was a punk.”

“True...”

“Besides, two people's shared preferences for a meeting like this differ from what their individual preferences suggest.”

“How so?”

“You said you usually go to McDonald's, and the Like is there too. I rarely eat fast food; if I need a sandwich on the move, it is McDonald's or Subway. For conversation, I go to restaurants. We both choose McDonald's more often than KFC, so it is common ground. Right?”

“Right.”

“But I think McDonald's is no good for talking: crowded, busy, uncomfortable. The food is dreadful too, unless you need an immediate stopgap. You would have considered many restaurants expensive. We were not looking for something both of us like and use in some circumstances, but something neither objected to in this particular situation. You suggested KFC because drinks are refillable, it is cheap and nearby, and there are power sockets if we need to charge anything, plus free Wi-Fi. You could also have said much of the food tastes good.”

“Yes.”

“Next time, we will think of it ourselves, and perhaps their next promotion will appear on your Facebook wall with a Like.”

“You win...”

Vivian and her companion are investigating, and the question of how two people know each other arises. Likes, attendance at shared events: things Facebook reveals. The conversation shows that knowing people's primary preferences often is not enough. Asked which fast-food restaurant they visit most, the girls' first shared answer would be McDonald's, yet they did not arrange to meet there. Ask what criteria guide their choice of venues generally and places to talk specifically, and their answers reveal they will meet at KFC. Or make the question specific: what would they choose in such-and-such a situation?

Something being obvious to you because you know what you mean to ask does not mean respondents know. Your sense of which answers you want or do not want does not mean respondents want to give those answers. Run a few investigative adventures where questionnaire responses help build profiles and approach a solution. Once players see bad questionnaires' answers misleading their characters, they will not regard criticism of similar flaws as empty bashing. A questionnaire is a joint activity, however. It works when respondents cooperate and receive something too. As the restaurant conversation shows, finding a solution good for both parties is not always easy.

Sol is a scientist and sociologist. Someone suggested his professional analysis “missed” the consistency-check questions, despite their being basic. They warrant attention in two cases: when very bad or very good. “Clever Magnus, smart Magnus, lovely Magnus, he found the A key on the keyboard” is not professional analysis. Leave that fine work to people around the author. Finding the A key is basic too, just like a consistency-check question; praising it is a grossly personal jab. Sol is not a psychologist or marketer, however. If a questionnaire aims to make you habitually seek the topic and questionnaire on Fridays, associate it with positive experiences, provide feedback, and sustain that dialogue later in development, the worst outcome is respondents associating your question and brand with negative experiences. Manipulating them, or heaven forbid deceiving them. Making an agreement and leaving them waiting in vain. Looking unprofessional and reinforcing that impression is another mistake. Regularity is needed. Negative feedback about quality can make them stop following your other news too. Disappointment breeds anger. 

A pollster also wants you to talk to them next time, preferably maintaining a database of easily reached respondents. They build loyalty to themselves rather than the topic. Most questionnaire stories have three participants: respondents, pollster, and client. All three sides' preferences must be met. Many companies would not undertake Hungary's national consultation. Invitations to complete questionnaires often appear on websites and Facebook groups, where steering the group's discussion can be a goal. If someone else recommends a questionnaire on their site, Facebook page, or elsewhere, however, negative opinions may attach to that promoter too. A fourth party needing satisfaction appears. 

A questionnaire often directs a community or individual toward a subject. A cultist might use it to open your mind to their teachings. Repeated questionnaires, hidden or less-hidden manipulation, and conditioned reflexes or addiction built through reinforcement and feedback are things we would expect of a cultist. Gathering information raises problems: are we only counting camps favoring individual answers, or also interested in means, medians, standard deviations, and other measures offering a fuller picture? The typical “who would you vote for if the election were today?” has no mean or median. Results differ between researchers and certainly do not mirror the coming election. These considerations often arise in in-character questionnaire design and matter in questionnaires about roleplaying too. Now for something that must be handled differently in and out of character. We said that familiar question does not reflect the approaching election. Why?

Because the election is not today. We do not know who will run together. “I like this party, but not if it allies with that one” preferences do not appear when alliances are not being discussed. The election is not today. Party preference measured in a representative sample means nothing by itself. It scarcely accounts for constituency candidates, independents, and many other details. There is insufficient information to estimate seats. But it communicates wonderfully, yields a graph, and Index publishes the results as news. The result simply is not relevant. In the game, however, the questionnaire's goal is not having masses of NPCs read its results. 

Likewise, ask roleplayers and none is a powergaming brat, especially if names are attached. We either trust a survey's anonymity or do not; many surveys are not anonymous anyway. In political polling too, many people conceal preferences. Without “don't know/prefer not to answer,” they certainly will not vote according to their real preferences. A government supporter fears the loud opposition crowd; an opposition supporter fears retaliation by those in power. That is why the consistency checks mentioned earlier exist. In-character questionnaires rarely need them, because nobody wants to waste extra time. Honesty becomes a matter of agreement or a test roll. But what is the question's point? What information does anyone derive? On its own, precisely as much as from the M.A.G.U.S. questionnaire condemned on every forum.

If, among people not voting for a right-wing party in this round, you examine rejection of DK, Jobbik, and the governing parties, you already get an answer about whether an alliance with DK is worthwhile. Rejection can be measured on a ten-point scale, yielding means, medians, standard deviations, and everything a statistician needs. Index does not receive this data and would hardly want to write an article from it. Where and by what means to ask also matters, as do whom and why.

“I want to make a questionnaire about this subject” and “I want to use the answers somehow” are not enough, for either an in-character or out-of-character questionnaire.