Article Index

There are excellent books on the subject. Sol will surely recommend them. You will laugh a great deal while reading one, yet learn to make questionnaires along the way; then, reading the other, you will learn to evaluate them while laughing too. But we cannot replace two books with articles. And a few articles cannot replace the books, let alone a university education. Numerous prestigious universities offer free online courses. Some will teach questionnaires in the context of sociology, others from the perspective of market research and marketing. For a scientist, accurate measurement will often matter most; for a marketer, it will be how the questionnaire helps build brand loyalty, what message it sends, how it mobilizes the community. These are different considerations. Few questionnaires meet every criterion. 

There are, however, plenty of good questionnaires. Many reward you for completing them, and you can see the tricks I discussed. You see that many things must be rated in ways that allow means and medians; they can connect numerous questions, filter, investigate relationships, and use any number of tools. Install a few questionnaire programs and you will find help and documentation in many places; quite a few questionnaire services offer examples with good explanations. Most people, however, asked for a roleplaying questionnaire. Forgetting that, off the top of my head, there are at least six entirely different types in this area:

  • Questionnaires used by psychologists or sociologists researching the subject
  • Questionnaires intended for market research
  • Questionnaires made for advertising purposes
  • Questionnaires providing a crutch for game developers
  • GM and character creation aids
  • In-character questionnaires

No single example covers every category. 

One book does not cover universities' relevant curricula, but it is enough for what you need as a roleplayer. Even though a book written for marketers, one analyzing the connection between questionnaires and spearphishing from a security perspective, and one a sociologist would recommend say entirely different things. Each is sufficient for what you need. Many books will not be up to date. But they impart the knowledge. If you want genuinely usable knowledge, you should read at least a book's worth. Once you acquire it, you do not need the examples we provide. And if you do not read the book, you cannot make use of the examples.

But the truth is that often, questionnaire expertise is not what is needed. Taking just one example from the M.A.G.U.S. questionnaires, I want to show why many of the problems cannot be remedied as simply as showing you a few examples with explanations. I am choosing an example from here precisely because the problem with this particular question is so conspicuous. It would be very difficult to find such a widely known questionnaire containing a question this bad. Besides, those who requested the article because of current events would expect such an example. The question is surprisingly simple: “Which do you think is more important?” Unlike many other questions, it has no explanation attached. There are two choices, between which votes were divided roughly equally. We can choose between “A balanced rules system” and “Classes receive the disproportionately larger number of skills needed for their distinctive flavor (head-hunter, bard).” 

Why the Question Is Bad

The first thing that occurs to me is simple: it does not specify in what respect the answer should be more important. In other words, the question is ambiguous.

  • I may think that numerous colorful classes lead to easily produced supplements and are important from a business perspective. 
  • I may think a balanced system matters for marketing. 
  • Something may not really be important, but may help make the game enjoyable.
  • Something may, in my view, make a beginner GM's work easier and therefore matter. 
  • Some things matter to me, and others, as an experienced site owner, I believe matter to the community.

And if the question is ambiguous, the answer received at this point will be ambiguous too.

On the other hand, there is a deeper problem. Class diversity and a balanced system—even game balance—can coexist. Are we choosing which of two possibilities to implement, or which to emphasize? The question says nothing about whether the space between the two options is a “golden mean” or a case of “falling between two stools.” 

Why the Question Type Is Bad

Let us begin with the obvious. Even if I linked you only this page of the article and you have not read the beginning, you will notice that for some people one answer is “a little more important,” while for others the other is “much more important.” But if you receive no information about the degree of importance, omit the reason for that importance too, and also omit the extent to which the more important consideration should prevail, there is little chance of building consensus on this basis. While the two answers currently differ in popularity by two votes out of 612, roughly 0.32%, distortion from the chosen question type can shift the response as much as 20% away from respondents' intentions at that moment.

But the problem with the question type does not end there. Although I seem to remember previously seeing a completely bad question about compensating for or enhancing advantages, the questionnaire has certainly been edited since then, and that question is now missing. Yet even with such a checking question, it was unsuitable for refining the answer. Class flavor is one thing, but there are many other things to sacrifice for balance. And I would like to point out that the early statements suggest there will be no class system.

Nor does it reveal how much we are willing to give up for slightly better balance when we know all along that the balance will certainly not be quite right. 

Why the First Answer Is Bad

A balanced rules system? Could someone just tell me whether the author means game balance? The balance between risk and reward, which may eventually create balance within the party too? Or that the rules system's declared purpose is to serve several play styles, and that the needs defined by those styles are balanced? Some people take their own interpretation for granted. Others point out that it depends on context we do not know. That another person would find a different meaning self-evident.

Game balance as an interpretation is problematic to begin with. Call of Cthulhu appeared long before M.A.G.U.S., and we know the question: which is worth more, 1% in Library Use or 1% in some combat skill? The system cannot tell us, because it does not know how often we will roll each. Creating game balance at the system level is a hopeless undertaking. Should I assume the questioner has not noticed this yet and must mean game balance? A balanced system, however, makes me think precisely of how Call of Cthulhu tries to provide equal support for many kinds of adventures and situations. Its system is balanced even though the rolls and many other things mean it does not seek game balance created or enforced by the system. 

Why the Second Answer Is Bad

It is easy to say that some people strongly identify with this answer, so it could even be a good answer. Indeed, it could be the better of the two. But one fundamental problem is that it ignores the fact that skills are not the only things making classes special. The question does not define what we mean by a class. Yet different systems use many different concepts of class. You can say that because the planned system does not use classes, this definition is unnecessary, but that does not rescue the question. 

Although I could offer other definitions of class, the best approach for this question is that a class represents a package of knowledge the character would not have time to acquire later within a game's normal time span. This would entail many years of largely uninterrupted study. Abilities you receive as part of a class but cannot learn independently as skills—because they require too much—are special abilities. In this case, class diversity means the “jack of all trades, master of none” concept: a diverse and varied package of skills received in the absence of powerful special abilities. 

Rank, a role within a church, and any number of other things also influence game balance, although the system does not quantify these values. Reducing the question of balance to skills, and excluding advantages that are often much stronger but unquantified—noble rank, a position within a church, and so forth—would be a mistake. Let us instead put the question this way: most such characters lack a sufficiently good education, so are they entitled to all those skills merely because they are weaker in other respects? A balanced system supporting many play styles—we are not talking about game balance—can build on differences in power within the party. 

Missing Answer Choices

As we might suspect, for some people what matters is neither “class diversity” nor some kind of balance, but, say, portraying certain characters faithfully to the world. Well, look at that: this direction is entirely absent from the choices. The various kinds of balance include game balance, story balance, “risk versus reward” balance, and a balance of “needs served,” as well as that peculiar “balance of rank” in which the characters playing together are those who would “belong in the same party” even under the world's normal workings.

We cannot choose many things that occur to us. Not even a neutral answer such as “they are about equally important.” 

And I will be a bastard about it: someone who survives many adventures without going insane and even improves a little—rolling checks—is powerful. Beside a well-played but often cautious character, the latest in a succession of new characters is less so. All right, their dice rolls may be lucky or unlucky. But on average, the possibility of differentiated rewards in a long-running game matters more than balance enforced by the system. And the d20 system, which strives strongly for balance and builds on balanced conflicts, did not become particularly popular among either Call of Cthulhu or M.A.G.U.S. fans.

Every flaw above is terribly obvious. I could produce a corrected questionnaire. Not necessarily professional work. I would merely avoid the mistakes described above. Of course, those are not all the mistakes, only the most obvious. You would receive the corrected questionnaire, with an explanation. A quarter as much as this story-filled article contains. But it would not help you avoid the other mistakes. There are enough good questionnaires. Yet if someone does not have the sense, after the first criticism, to have their next questionnaire reviewed before publication... To have everything ready beforehand, checked in as many rounds as necessary—publishers have editors, after all, and can ask others to test the work too—then it is questionable how much advice requiring considerably more judgment they could keep in mind.

There are many games in which the word “realistic” does not mean invoking it to whine for extra damage in an unrealistic combat system shaped to feel like chess, while demanding nerfs for someone else in the name of balance. “Realistic” makes me think of not narrating things far wilder than CSI's miracle gadgets in a Call of Cthulhu adventure set in the present or recent past, where nothing supernatural is involved. But it also means not introducing a camera-equipped smartphone into an adventure set in either the 1920s or the early 1990s—in Hungary, say, in the latter case. In many cases you can ask. And should. Because these games have depth to their worlds, and sometimes a sociologist gives advice, perhaps about questionnaires; sometimes an IT specialist explains how to run a hacking scene; and sometimes a camera operator explains how to run the filming of a reality show. 

There would be plenty to discuss. Whether someone deserves credit for finding the letter A while writing their questionnaire, or whether work this slapdash is embarrassing from a professional developer, is a natural subject in M.A.G.U.S. groups. Sadly, this is what the future of M.A.G.U.S. revolves around. But other roleplaying groups are not necessarily interested. All right: questionnaires are useful to them too, and the topic has become timely and is generating plenty of activity. But they are not interested in “oh, do not pick on him, he found the letter A, what a clever boy.” The umpteenth amateurish move has no news value either. It can go alongside the related news item. 

And despite the topicality, how to handle questionnaires in, say, a Call of Cthulhu adventure is much more interesting than whether a lead developer has reached the most basic of basics. More than two decades have passed since the transition from communism. Questionnaires and opinion polls have been part of our lives ever since. But not only ours: they are part of the lives of many of our Call of Cthulhu, Laundry, WoD, or CoD characters too, if those characters live in this environment. Questionnaires provide any number of interesting stories and subjects. I will say it yet again: who wants to defend the developer tinkering with their favorite system because he can find the letter “A” is still not one of them.

In the continuation of our article, Sol can describe interesting cases and examples along with their lessons. Then, at the end, we can offer an example of how a questionnaire becomes part of a series of adventures.