Article Index

C2Someone asked, “What makes a good roleplaying questionnaire?” My immediate answer would have been: one where an NPC questions a PC, or a PC analyzes NPCs' answers. But I also wonder: good for whom, or good for what? Showing a sample would be pointless, so let us have an article, or a series. Even this first installment of our longer series will be a monster spanning at least four or five pages. Look left at the mad cultist. He could fit into the Ministry of Silly Walks: it takes an unusual gait to keep that barbed wire from slicing his thighs. But he trusts that if the spikes point outward and he brings his knee rather forcefully into contact with one of your sensitive spots, it will hurt you more. The questionnaire in his hand is another game, directing your thoughts toward a subject in the hope you too become a mad cultist. Would a social scientist praise his manipulative questionnaire? But if the party examines questionnaire results during an investigation to narrow down suspects, the adventure needs a good questionnaire, not a manipulative one. Seeing many of these gets you accustomed to questionnaires and teaches you to ask good questions and use the technical tools. Then you can ask better questions about roleplaying too. Let us move from general explanation toward specifics.

Most of us roleplayers are not scientists, especially not sociologists. I hope my coauthor Sol, who is both, writes the continuation and adds more considerations, with recommended reading adding still more. But you will not learn questionnaire design from this alone. The topic is currently prominent partly because of M.A.G.U.S., though there was an FRPG questionnaire too. One worse than the next. Yet we put these articles in the Call of Cthulhu category. I considered Shadowrun and rejected it because its tools differ. Primarily, we want to explore not what you can ask roleplayers, but two important subjects: how questionnaires and related research can appear within game worlds and how to handle them as a GM; and situations where a questionnaire becomes a tool for running the game rather than an opinion poll, for example when assigning roles within the party. 

The first basic point is that the questioner's goals matter. That sentence immediately needs explanation: an explanation that seems obvious, unnecessarily explicit, and self-evident while reading it. Yet when making a questionnaire without reading it, those responsible usually forget all these obvious things. They thereby give up many benefits, never considering that a questionnaire or vote could achieve more than one goal. Imagine a familiar but primitive situation, starting with a simple poll. A picture shows five characters, perhaps roughly drawn. The picture and post ask which we prefer playing: Like if all are equally good, Love for the priest, Angry for the tough militant paladin, Sad for the monk, Wow for the believer, and Laugh for the atheist mocking faith. 

Questions like this appear often, and yield considerable data. This is only one question, not a completed questionnaire, but we have many and people answer each in turn. Facebook makes the data needed to link responses difficult to obtain, but if regular followers use our app, everything can be linked for them. What does the questioner gain or lose? Some questions never reach particular respondents, and not everyone's answers can be linked. On the other hand, Facebook records respondents' interactions with the post and shows that page's posts more frequently. Alongside analyzable answers, this is often a goal, perhaps the more important one. The approach is light and playful, with many ways to engage people. A serious social-media expert can achieve a great deal using Like statistics, visits, app-use data, and information from Facebook.

Besides answers and comments, it measures which posts you interact with. Two questions daily for 365 days can build quite a database. A large publisher with an online service and access to all your reaction data through Facebook registration has a winning combination. Best of all, it even measures functionally illiterate people's answers! Unlike your character, though, you will not have such an app or access to everything through the Graph API, so you do not start on equal terms. An economic power linked to certain political sides might, having stated that it will not sell your personal data while allowing aggregated anonymous statistics. That is the better case. In any event, correlations between keyword combinations and reactions—likes, shares, keywords in comments, and other Facebook reactions—are measured too. Some companies' privacy policies are less friendly. You can likewise forget interviewers and telephone interviews. Nor are you likely to filter responses by spending recorded in a loyalty-program database. An in-character questionnaire and an out-of-character questionnaire about the game are entirely different.

As we might guess, someone beginning a consultation with that database does not seek to learn our opinions. They knew them long ago; their policy aims to maximize votes. The questionnaire has two goals: prompting interaction, and making you feel the expected decision reached here is your own, building loyalty to a brand. The target group consists of sympathetic people susceptible to this influence. Many roleplaying questionnaires in the world are advertising like this too. When the questioner genuinely wants to know something, matters become harder. Let me introduce that with an excerpt from a conversation in my story Are You Afraid of Good? 

“Apparently Dezső and János know each other from here too. Dezső recommended the concert to János. Judging by his plans, he thought he was a punk.”

“True...”

“Besides, two people's shared preferences for a meeting like this differ from what their individual preferences suggest.”

“How so?”

“You said you usually go to McDonald's, and the Like is there too. I rarely eat fast food; if I need a sandwich on the move, it is McDonald's or Subway. For conversation, I go to restaurants. We both choose McDonald's more often than KFC, so it is common ground. Right?”

“Right.”

“But I think McDonald's is no good for talking: crowded, busy, uncomfortable. The food is dreadful too, unless you need an immediate stopgap. You would have considered many restaurants expensive. We were not looking for something both of us like and use in some circumstances, but something neither objected to in this particular situation. You suggested KFC because drinks are refillable, it is cheap and nearby, and there are power sockets if we need to charge anything, plus free Wi-Fi. You could also have said much of the food tastes good.”

“Yes.”

“Next time, we will think of it ourselves, and perhaps their next promotion will appear on your Facebook wall with a Like.”

“You win...”

Vivian and her companion are investigating, and the question of how two people know each other arises. Likes, attendance at shared events: things Facebook reveals. The conversation shows that knowing people's primary preferences often is not enough. Asked which fast-food restaurant they visit most, the girls' first shared answer would be McDonald's, yet they did not arrange to meet there. Ask what criteria guide their choice of venues generally and places to talk specifically, and their answers reveal they will meet at KFC. Or make the question specific: what would they choose in such-and-such a situation?

Something being obvious to you because you know what you mean to ask does not mean respondents know. Your sense of which answers you want or do not want does not mean respondents want to give those answers. Run a few investigative adventures where questionnaire responses help build profiles and approach a solution. Once players see bad questionnaires' answers misleading their characters, they will not regard criticism of similar flaws as empty bashing. A questionnaire is a joint activity, however. It works when respondents cooperate and receive something too. As the restaurant conversation shows, finding a solution good for both parties is not always easy.

Sol is a scientist and sociologist. Someone suggested his professional analysis “missed” the consistency-check questions, despite their being basic. They warrant attention in two cases: when very bad or very good. “Clever Magnus, smart Magnus, lovely Magnus, he found the A key on the keyboard” is not professional analysis. Leave that fine work to people around the author. Finding the A key is basic too, just like a consistency-check question; praising it is a grossly personal jab. Sol is not a psychologist or marketer, however. If a questionnaire aims to make you habitually seek the topic and questionnaire on Fridays, associate it with positive experiences, provide feedback, and sustain that dialogue later in development, the worst outcome is respondents associating your question and brand with negative experiences. Manipulating them, or heaven forbid deceiving them. Making an agreement and leaving them waiting in vain. Looking unprofessional and reinforcing that impression is another mistake. Regularity is needed. Negative feedback about quality can make them stop following your other news too. Disappointment breeds anger. 

A pollster also wants you to talk to them next time, preferably maintaining a database of easily reached respondents. They build loyalty to themselves rather than the topic. Most questionnaire stories have three participants: respondents, pollster, and client. All three sides' preferences must be met. Many companies would not undertake Hungary's national consultation. Invitations to complete questionnaires often appear on websites and Facebook groups, where steering the group's discussion can be a goal. If someone else recommends a questionnaire on their site, Facebook page, or elsewhere, however, negative opinions may attach to that promoter too. A fourth party needing satisfaction appears. 

A questionnaire often directs a community or individual toward a subject. A cultist might use it to open your mind to their teachings. Repeated questionnaires, hidden or less-hidden manipulation, and conditioned reflexes or addiction built through reinforcement and feedback are things we would expect of a cultist. Gathering information raises problems: are we only counting camps favoring individual answers, or also interested in means, medians, standard deviations, and other measures offering a fuller picture? The typical “who would you vote for if the election were today?” has no mean or median. Results differ between researchers and certainly do not mirror the coming election. These considerations often arise in in-character questionnaire design and matter in questionnaires about roleplaying too. Now for something that must be handled differently in and out of character. We said that familiar question does not reflect the approaching election. Why?

Because the election is not today. We do not know who will run together. “I like this party, but not if it allies with that one” preferences do not appear when alliances are not being discussed. The election is not today. Party preference measured in a representative sample means nothing by itself. It scarcely accounts for constituency candidates, independents, and many other details. There is insufficient information to estimate seats. But it communicates wonderfully, yields a graph, and Index publishes the results as news. The result simply is not relevant. In the game, however, the questionnaire's goal is not having masses of NPCs read its results. 

Likewise, ask roleplayers and none is a powergaming brat, especially if names are attached. We either trust a survey's anonymity or do not; many surveys are not anonymous anyway. In political polling too, many people conceal preferences. Without “don't know/prefer not to answer,” they certainly will not vote according to their real preferences. A government supporter fears the loud opposition crowd; an opposition supporter fears retaliation by those in power. That is why the consistency checks mentioned earlier exist. In-character questionnaires rarely need them, because nobody wants to waste extra time. Honesty becomes a matter of agreement or a test roll. But what is the question's point? What information does anyone derive? On its own, precisely as much as from the M.A.G.U.S. questionnaire condemned on every forum.

If, among people not voting for a right-wing party in this round, you examine rejection of DK, Jobbik, and the governing parties, you already get an answer about whether an alliance with DK is worthwhile. Rejection can be measured on a ten-point scale, yielding means, medians, standard deviations, and everything a statistician needs. Index does not receive this data and would hardly want to write an article from it. Where and by what means to ask also matters, as do whom and why.

“I want to make a questionnaire about this subject” and “I want to use the answers somehow” are not enough, for either an in-character or out-of-character questionnaire. 


We are used to demographic questions asking our age, where we live, about family, children, and other details. Many know these allow responses to be filtered. I would add that they can filter the questions too. Sorry to return to the topical M.A.G.U.S. questionnaire, but perhaps unfortunately everyone has encountered it. Several kinds of people could speak on the subject: current M.A.G.U.S. players; people playing on Ynev with another system; former players who might return for a better system; readers and fans who would try playing; other roleplayers who hear about the questionnaire; and even people tired of the M.A.G.U.S. wars overwhelming every roleplaying group, who do not want to see it but could be won over by a good system and service. All six groups could complete a good questionnaire. The real one did not consider that. Yet you want to separate their answers, because the larger group is heterogeneous and divisible along many dimensions. When, as a GM, you have PCs complete a questionnaire intending to use their answers later in adventures, asking them to write their names may not occur to you. You can instead ask about traits uniquely identifying the characters.

Suppose the party has two female and three male characters. Your first demographic question asks the character's sex. Lo and behold, among five anonymous forms, one marked male can belong to only three characters. Asking about education eliminates at least one of those three and separates the two women too. Perhaps skin color distinguishes the remaining two men. You retain those five forms separately, while during an investigation they receive about 250 additional generated responses and must work out which belong to the killer and victims. Demographic questions initially help filter by profile. 

Once they know the killer's political leaning, a consistency-check question examining that no longer measures something; it becomes a filter, like a demographic question. In an American setting, for example, they might seek a white, not particularly educated Republican man who, asked what discrimination affects him, probably mentioned the higher price and difficulty of finding large clothes and shoes, since clues suggest a large man. For our bald friend, a shampoo opinion saying only that he does not use shampoo becomes another demographic question.

Questions about regular haunts, organizing activities, and daily routines tell you where you are most likely to find the perpetrator, perhaps to recognize them from a photograph or composite sketch.

You can find common ground in the victims' lives too. Finding it brings you closer to the hunting ground where your suspect operates, making the questionnaire a genuinely useful adventure tool. The party will first smile when their own completed forms also appear in the adventure's database, and again when some answers return two or three, or even four or five adventures later. Perhaps six months later. An answer may enter an adventure's story, or an NPC may produce their questionnaire and approach them because “several questionnaires say you often visit where this happened.” Another investigator finds them just as they once found someone through questionnaires.

If “roleplaying questionnaire” means an in-character questionnaire assembled by an NPC and completed by PCs, an important consideration immediately arises: what you present must be believable. Or sometimes precisely its implausibility matters, since this is Call of Cthulhu. Remember what I wrote on the previous page?

Sol is a scientist and sociologist. Someone suggested his professional analysis “missed” the consistency-check questions, despite their being basic. They warrant attention in two cases: when very bad or very good. “Clever Magnus, smart Magnus, lovely Magnus, he found the A key on the keyboard” is not professional analysis. Leave that fine work to people around the author. Finding the A key is basic too, just like a consistency-check question; praising it is a grossly personal jab.

You are running Call of Cthulhu, perhaps in present-day Hungary, where those M.A.G.U.S. questionnaires exist in the game world. Suppose the party investigates cultists operating among roleplayers, perhaps largely LARPers: first having people act things out, then exploiting what they learned for the game and their opened minds. During the investigation through roleplaying circles, numerous questionnaires first appear and characters complete them. Then one appears where the noteworthy thing is not clever Magnus finding B and C too, but questions allowing means, medians, and standard deviations; demographics; and a few clever interactive questions distinguishing current, former, and potential players. You would immediately ask what replaced this person and look for a suitable monster.

When running horror or writing a comparable story, players must be able to notice changes and unusual things. That includes the questionnaire maker's behavior, but also responses. For people whose answers have recently shifted suspiciously, suggesting possible cultist danger, ask how they answered the basic demographic questions. Where do they live, how old are they, what sex, what income, what education? It becomes easy to identify which factory has something dangerous spreading through it. You filter by answers usually measured, then measure and build statistics and geographical profiles from questions ordinarily used merely as demographic data.

A few skill checks, Know and Idea rolls, and similar things make this approach work too. 


You will not like what comes next. In many cases, a questionnaire asks for information that a service provider could use to identify you. Or it asks for information useful for social engineering. Often, of course, genuine market research, opinion polling, or substantial scientific work is taking place. In other cases, however, it may involve abuse, spearphishing, or some other dirty trick. The best case is that there is no dirty trick: the questionnaire only costs you time, and you receive something in return. Or at least a promise, a little illusion. Sometimes the questionnaire costs you aggravation as well as time. We have seen examples of that too. And then, of course, there are the truly dirty tricks.

One questionnaire does not reveal too much dangerous information about you. But what about five? And what if they can see your Facebook profile too? Questionnaires are often used for more than asking a community's opinion. The situation above is interesting because the same kind of questionnaire can paint a “standardized” picture of individual characters' backgrounds. Or at least of what their players think about their characters. If I had to show a sample questionnaire now and then explain what was arranged how, and why, I would start with a questionnaire that was bad in many respects. Not just bad, but downright awful, immediately raising the question: why? Because I would deliberately restrict the possible answers, deliberately manipulate. The questionnaire's answers would distribute points according to the character creation rules, producing a character who fit the adventure. To do that, however, I would have to exclude answers that really did not fit the adventure.

And this would already be crucial when choosing the right platform. The program must be able to calculate: that is a basic requirement. There must be read-only fields. There must also be...

Imagine a series of questionnaires. They appear to concern transport habits. The first few batches of questions concern your present transport habits. They ask about many service providers, then make an interesting little shift. They try to compare these with your childhood transport information. Let us say the respondents live in Budapest. A district alone is not much information. But the kind of vehicle you used to reach a metro station then, and the kind you use now, is already important information. Approximately how many stops you traveled, how long you traveled, whether you did, what proportion of the route you covered: these still examine transport information. So do questions measuring your journey to school instead of to the metro. But the first set tells them with fairly good accuracy where you lived, while the second helps them work out where to look for the school. They may know the year from demographic information too.

How long does it take, searching Facebook and asking former school staff and other pupils a few questions, to find out “what was the first-grade homeroom teacher called in year X?” Perhaps for several first-grade classes. This can easily supply the answers to two secret questions. If they ask you later, perhaps on another platform, questions about banking habits and many other things can follow. Naturally, in each case they know your IP address and numerous details needed to identify your computer. And they look for a nearby Wi-Fi network they can easily join, so the attempt does not even come from an unfamiliar location. The browser's user agent and many other identifiers may be familiar, since these can be copied. 

Connections between questions in a questionnaire, or connections between several linked questionnaires, can both be used to extract information you do not want to disclose. Indeed, with a little luck, filling out a questionnaire can reveal something you otherwise would not tell them simply because you do not know the answer for certain yourself. The problem is the questionnaire's length. And many people do not welcome having their private lives intruded upon and analyzed. This is another reason they reject questionnaires. Of course, that is only true up to a point. There is another mechanism.

The simplest situation is when the reward for completing the questionnaire is neither tangible, nor a promise, nor some result, but the knowledge that you did not spoil anything good; you even helped by spending those five minutes on it. And anyone who says the whole thing is bad and was not even worth that can go screw themselves. After all, they have spoiled the good feeling that even completing the most awful questionnaire gave you. The next situation is when you receive a promise or an illusion in exchange for the questionnaire: this will improve something you care about too. For D&D's fifth edition, every playtest packet came with a questionnaire, and we filled them out; we even provided feedback on the forum. But many of us do not play the finished game. The enthusiasm generated on the basis described here can very easily turn into disappointment: all it takes is for them to call something final that you intensely dislike, when they did not even ask about the most important arguments and considerations.

Of course, someone lectured me that the thing I complained about no longer worked that way in the final PHB. They simply failed to understand why this did not interest me. Yet it is very simple: supplements will not have a test version, something that sweeps away the supposedly final solution. I would like to believe the questionnaire and testing served a purpose. But it is damned easy to feel disappointed when the questioner fails to notice one thing: Charm Person was not the only iconic thing in D&D; many races also had various traits and limitations that justified those traits. When these slowly disappear, and the explanation amounts to “well, there are many worlds, so we cannot include the limitation,” I have reason to be disappointed. Because, excuse me, this is someone saying that a longsword costs the same in every city in every world, that abilities are universal too, that the same spells were discovered everywhere—and yet the iconic things that matter to me are precisely what is missing.

And here I should point out that the D&D questionnaire was merely a simple example. With a local café too, it is natural for a questionnaire's main promise to be that the product will become better for us, that we will benefit. Of course, there too, one person wants a vegan café with strongly feminist politics, another wants Hooters, so someone will be disappointed. But a questionnaire put out on the premises can certainly be used for profiling too. Unlike the D&D questionnaire, however, this may often be on paper; asking for a few minutes that way is completely different from an online questionnaire linked in local Facebook groups. It is no accident that offline questionnaires often come with an interviewer. 

Most people say you cannot ask everything. They are right. You cannot, should not, and above all need not ask everything. People will give you five minutes out of goodwill. But fifteen minutes usually needs a promise behind it, and half an hour or an hour needs a substantial reward. Yet even a moderately amateur practitioner will say: a question in a questionnaire can easily control which questions appear later, which ones we must answer. If, instead of first asking which spell is iconic and making me check Charm Person, they ask what features of (A)D&D made me play it, offering the world, storyline, spells, special races, and so on, they will probably not subsequently ask me how iconic Charm Person was. But they will learn about the time it took ten to fifteen adventures to get around a racial limitation because our dwarf had made a sacred vow to become a wizard if the god of magic healed his friend; or how iconic it was when the elf girl, thanks to her wild wood-elf heritage, could lift more than the heroic knight could move, though her strength did not show. And how much, how very much we laughed when someone wanted to help her. Her weapons did not even take advantage of her strength. But she made artistic weapons. 

Asking only a few questions on the grounds that “there are many possible answers, it took a lot of typing” is utterly embarrassing. We do know where that mistake was made, do we not? A good questionnaire needs branches and a relatively large number of questions. Enough to connect answers that say little individually but become revealing together. In offline questionnaires, the interviewer often handled these branches. Their most important characteristic, of course, was that they did not influence the respondent. At least in theory. In many cases, however, if someone assumes the interviewer has very different opinions from theirs, the wish to meet the interviewer's expectations may still arise. And that brings us to the next part of the subject.


When I suggested that Sol write the next article, it was not primarily about his professional knowledge. We are not trying to write a tutorial, merely to draw attention to a few things. To explore a subject from a roleplayer's perspective. But the truth is that even so, the most interesting stories about questionnaires require a level of knowledge that only he possesses among the site's team. We are preparing an article, perhaps a series, about the period of Hungary's transition from communism for Call of Cthulhu players. But it is equally true that today's young players may have missed many political or scientific subjects from the first decade of our millennium too. Information technology also developed during this time: in the late 1980s, ordinary people scarcely had internet access, let alone questionnaire software. But it was not just the tools that developed; questionnaire designers found solutions to ever more new problems.

If every pollster's seat projection for an election was completely wrong, there were reasons for that, and methodology evolved. Somehow they had to find out the concealed answers too, and answers the respondents themselves did not know. The polling organizations, staffed predominantly by professionals one and all, were badly wrong, and there was plenty to correct. Many replaced their entire methodology for projecting seats. The point came when they had to say: the world has changed. A questionnaire, a research methodology that worked brilliantly yesterday had failed, not just here but in numerous other places around the world. The internet, the replacement of tools, and social changes all appeared in succession. Indeed, television voting offers an interesting detour into the territory between questionnaires and Facebook polls.

One defining moment in the rivalry between TV2 and RTL Klub came when both stations considered buying Big Brother. RTL Klub, however, decided to emphasize luxury instead, include filming at outside locations, and, of course, approach the business behind the entire series differently. It would not matter if TV2 won the battle by buying the original and achieving higher viewing figures, provided RTL Klub won the war. The goal was more direct revenue, which would mean more money to buy and produce programs later. And they would even avoid the royalties. Somehow or other, Való Világ beat TV2 in the ratings battle too. Much of that required the market research behind developing the format. But people had to be persuaded not only to follow the online broadcast, but to vote too. 

They needed people for whom viewers would vote practically without limit, even by setting government offices' telephone exchanges to vote automatically and thus defrauding their workplaces; yet those people also had to be divisive enough to keep viewers glued to the screen, whether because they loved them or hated them and wanted to be well informed about the show. Serious market research preceded casting. It had to keep competitors from recognizing the strategic goals, and extract answers from respondents who publicly said they did not watch such things but sat down to them every evening anyway. Recognizing these people's unrecognized and concealed preferences made it possible to know what kinds of people and situations the show needed, how to beat TV2. The rest is history. 

This was the point when the view that TV2 could fuck up even the best format, while RTL Klub could beat them with less impressive material, became particularly conspicuous. Numerous experiences reinforced it, and by the time TV2 acquired the label of government mouthpiece, it had little chance of recovering. Because voting cost money, however, it measured not so much popularity as the size of solvent demand. Counted not per person, but per forint. Yet they also needed to see what kinds of people and situations increased revenue during an eviction vote, and the voting results, along with independent measurements and studies, influenced what happened in the editing room and what tasks contestants received. And, of course, the advice whispered to them by the editors. 

Today, everything revolves around the fact that you or I can have a questionnaire. There is a free plan for it. Only the resulting survey will usually not be good. We often hear “the survey is not representative” and see a questionnaire failing to reach its most important target audience. We speak Hungarian; everyone knows how to ask questions. What could go wrong? Except that many of the cases above show that professionals sometimes get it wrong too. And most roleplayers making questionnaires on this basis are not even professionals. And we see the spectacular successes.

What these successes share is attention to the public mood throughout the research. They did not merely throw together a generally good questionnaire: they considered the circumstances in which it would ask its questions, and whom it would ask. And right now, it makes sense to ask different things of someone playing the Hungarian-language version of Call of Cthulhu, someone using the d20-based version, and someone whose foundation is the newest English-language core rulebook. Indeed, we ask different questions again when targeting roleplayers who do not yet play Call of Cthulhu. Or fans of Lovecraft's work who do not yet play roleplaying games. 

Opinion polls miss the mark in numerous cases. They did not really predict Trump's victory either. Yet there were plenty of parameters measurable by questionnaire that would have shown it early. A social phenomenon we know well because we are part of it. Pillars of Eternity being censored because of SJWs? A questionable professional joining the Mass Effect team because of SJWs? A favorite comedian refusing to perform in many places because of SJWs? These problems appear online and spread quickly enough. This increased the chances of victory for Trump, who firmly opposed political correctness and SJWs. The voting bloc this could reach was stable; his competitors ground one another down. People could debate how to interpret his scandalous words, but every statement still sent a clear message to groups opposed to PC. 

Failing to measure what matters, what could be decisive, is one of the greatest mistakes. Yet many universities and other intellectual institutions have never been ideologically neutral. Where PC seems a fundamental value and the SJW attitude natural, people will not measure this, because of their own convictions. The true enemy of truth is much less the lie than the conviction that prevents the facts from coming to light. Lies, after all, can be disproved. Conviction is far harder to correct, especially when the researcher brings it into the research themselves. 


There are excellent books on the subject. Sol will surely recommend them. You will laugh a great deal while reading one, yet learn to make questionnaires along the way; then, reading the other, you will learn to evaluate them while laughing too. But we cannot replace two books with articles. And a few articles cannot replace the books, let alone a university education. Numerous prestigious universities offer free online courses. Some will teach questionnaires in the context of sociology, others from the perspective of market research and marketing. For a scientist, accurate measurement will often matter most; for a marketer, it will be how the questionnaire helps build brand loyalty, what message it sends, how it mobilizes the community. These are different considerations. Few questionnaires meet every criterion. 

There are, however, plenty of good questionnaires. Many reward you for completing them, and you can see the tricks I discussed. You see that many things must be rated in ways that allow means and medians; they can connect numerous questions, filter, investigate relationships, and use any number of tools. Install a few questionnaire programs and you will find help and documentation in many places; quite a few questionnaire services offer examples with good explanations. Most people, however, asked for a roleplaying questionnaire. Forgetting that, off the top of my head, there are at least six entirely different types in this area:

  • Questionnaires used by psychologists or sociologists researching the subject
  • Questionnaires intended for market research
  • Questionnaires made for advertising purposes
  • Questionnaires providing a crutch for game developers
  • GM and character creation aids
  • In-character questionnaires

No single example covers every category. 

One book does not cover universities' relevant curricula, but it is enough for what you need as a roleplayer. Even though a book written for marketers, one analyzing the connection between questionnaires and spearphishing from a security perspective, and one a sociologist would recommend say entirely different things. Each is sufficient for what you need. Many books will not be up to date. But they impart the knowledge. If you want genuinely usable knowledge, you should read at least a book's worth. Once you acquire it, you do not need the examples we provide. And if you do not read the book, you cannot make use of the examples.

But the truth is that often, questionnaire expertise is not what is needed. Taking just one example from the M.A.G.U.S. questionnaires, I want to show why many of the problems cannot be remedied as simply as showing you a few examples with explanations. I am choosing an example from here precisely because the problem with this particular question is so conspicuous. It would be very difficult to find such a widely known questionnaire containing a question this bad. Besides, those who requested the article because of current events would expect such an example. The question is surprisingly simple: “Which do you think is more important?” Unlike many other questions, it has no explanation attached. There are two choices, between which votes were divided roughly equally. We can choose between “A balanced rules system” and “Classes receive the disproportionately larger number of skills needed for their distinctive flavor (head-hunter, bard).” 

Why the Question Is Bad

The first thing that occurs to me is simple: it does not specify in what respect the answer should be more important. In other words, the question is ambiguous.

  • I may think that numerous colorful classes lead to easily produced supplements and are important from a business perspective. 
  • I may think a balanced system matters for marketing. 
  • Something may not really be important, but may help make the game enjoyable.
  • Something may, in my view, make a beginner GM's work easier and therefore matter. 
  • Some things matter to me, and others, as an experienced site owner, I believe matter to the community.

And if the question is ambiguous, the answer received at this point will be ambiguous too.

On the other hand, there is a deeper problem. Class diversity and a balanced system—even game balance—can coexist. Are we choosing which of two possibilities to implement, or which to emphasize? The question says nothing about whether the space between the two options is a “golden mean” or a case of “falling between two stools.” 

Why the Question Type Is Bad

Let us begin with the obvious. Even if I linked you only this page of the article and you have not read the beginning, you will notice that for some people one answer is “a little more important,” while for others the other is “much more important.” But if you receive no information about the degree of importance, omit the reason for that importance too, and also omit the extent to which the more important consideration should prevail, there is little chance of building consensus on this basis. While the two answers currently differ in popularity by two votes out of 612, roughly 0.32%, distortion from the chosen question type can shift the response as much as 20% away from respondents' intentions at that moment.

But the problem with the question type does not end there. Although I seem to remember previously seeing a completely bad question about compensating for or enhancing advantages, the questionnaire has certainly been edited since then, and that question is now missing. Yet even with such a checking question, it was unsuitable for refining the answer. Class flavor is one thing, but there are many other things to sacrifice for balance. And I would like to point out that the early statements suggest there will be no class system.

Nor does it reveal how much we are willing to give up for slightly better balance when we know all along that the balance will certainly not be quite right. 

Why the First Answer Is Bad

A balanced rules system? Could someone just tell me whether the author means game balance? The balance between risk and reward, which may eventually create balance within the party too? Or that the rules system's declared purpose is to serve several play styles, and that the needs defined by those styles are balanced? Some people take their own interpretation for granted. Others point out that it depends on context we do not know. That another person would find a different meaning self-evident.

Game balance as an interpretation is problematic to begin with. Call of Cthulhu appeared long before M.A.G.U.S., and we know the question: which is worth more, 1% in Library Use or 1% in some combat skill? The system cannot tell us, because it does not know how often we will roll each. Creating game balance at the system level is a hopeless undertaking. Should I assume the questioner has not noticed this yet and must mean game balance? A balanced system, however, makes me think precisely of how Call of Cthulhu tries to provide equal support for many kinds of adventures and situations. Its system is balanced even though the rolls and many other things mean it does not seek game balance created or enforced by the system. 

Why the Second Answer Is Bad

It is easy to say that some people strongly identify with this answer, so it could even be a good answer. Indeed, it could be the better of the two. But one fundamental problem is that it ignores the fact that skills are not the only things making classes special. The question does not define what we mean by a class. Yet different systems use many different concepts of class. You can say that because the planned system does not use classes, this definition is unnecessary, but that does not rescue the question. 

Although I could offer other definitions of class, the best approach for this question is that a class represents a package of knowledge the character would not have time to acquire later within a game's normal time span. This would entail many years of largely uninterrupted study. Abilities you receive as part of a class but cannot learn independently as skills—because they require too much—are special abilities. In this case, class diversity means the “jack of all trades, master of none” concept: a diverse and varied package of skills received in the absence of powerful special abilities. 

Rank, a role within a church, and any number of other things also influence game balance, although the system does not quantify these values. Reducing the question of balance to skills, and excluding advantages that are often much stronger but unquantified—noble rank, a position within a church, and so forth—would be a mistake. Let us instead put the question this way: most such characters lack a sufficiently good education, so are they entitled to all those skills merely because they are weaker in other respects? A balanced system supporting many play styles—we are not talking about game balance—can build on differences in power within the party. 

Missing Answer Choices

As we might suspect, for some people what matters is neither “class diversity” nor some kind of balance, but, say, portraying certain characters faithfully to the world. Well, look at that: this direction is entirely absent from the choices. The various kinds of balance include game balance, story balance, “risk versus reward” balance, and a balance of “needs served,” as well as that peculiar “balance of rank” in which the characters playing together are those who would “belong in the same party” even under the world's normal workings.

We cannot choose many things that occur to us. Not even a neutral answer such as “they are about equally important.” 

And I will be a bastard about it: someone who survives many adventures without going insane and even improves a little—rolling checks—is powerful. Beside a well-played but often cautious character, the latest in a succession of new characters is less so. All right, their dice rolls may be lucky or unlucky. But on average, the possibility of differentiated rewards in a long-running game matters more than balance enforced by the system. And the d20 system, which strives strongly for balance and builds on balanced conflicts, did not become particularly popular among either Call of Cthulhu or M.A.G.U.S. fans.

Every flaw above is terribly obvious. I could produce a corrected questionnaire. Not necessarily professional work. I would merely avoid the mistakes described above. Of course, those are not all the mistakes, only the most obvious. You would receive the corrected questionnaire, with an explanation. A quarter as much as this story-filled article contains. But it would not help you avoid the other mistakes. There are enough good questionnaires. Yet if someone does not have the sense, after the first criticism, to have their next questionnaire reviewed before publication... To have everything ready beforehand, checked in as many rounds as necessary—publishers have editors, after all, and can ask others to test the work too—then it is questionable how much advice requiring considerably more judgment they could keep in mind.

There are many games in which the word “realistic” does not mean invoking it to whine for extra damage in an unrealistic combat system shaped to feel like chess, while demanding nerfs for someone else in the name of balance. “Realistic” makes me think of not narrating things far wilder than CSI's miracle gadgets in a Call of Cthulhu adventure set in the present or recent past, where nothing supernatural is involved. But it also means not introducing a camera-equipped smartphone into an adventure set in either the 1920s or the early 1990s—in Hungary, say, in the latter case. In many cases you can ask. And should. Because these games have depth to their worlds, and sometimes a sociologist gives advice, perhaps about questionnaires; sometimes an IT specialist explains how to run a hacking scene; and sometimes a camera operator explains how to run the filming of a reality show. 

There would be plenty to discuss. Whether someone deserves credit for finding the letter A while writing their questionnaire, or whether work this slapdash is embarrassing from a professional developer, is a natural subject in M.A.G.U.S. groups. Sadly, this is what the future of M.A.G.U.S. revolves around. But other roleplaying groups are not necessarily interested. All right: questionnaires are useful to them too, and the topic has become timely and is generating plenty of activity. But they are not interested in “oh, do not pick on him, he found the letter A, what a clever boy.” The umpteenth amateurish move has no news value either. It can go alongside the related news item. 

And despite the topicality, how to handle questionnaires in, say, a Call of Cthulhu adventure is much more interesting than whether a lead developer has reached the most basic of basics. More than two decades have passed since the transition from communism. Questionnaires and opinion polls have been part of our lives ever since. But not only ours: they are part of the lives of many of our Call of Cthulhu, Laundry, WoD, or CoD characters too, if those characters live in this environment. Questionnaires provide any number of interesting stories and subjects. I will say it yet again: who wants to defend the developer tinkering with their favorite system because he can find the letter “A” is still not one of them.

In the continuation of our article, Sol can describe interesting cases and examples along with their lessons. Then, at the end, we can offer an example of how a questionnaire becomes part of a series of adventures.