Custom validation and scoring of the Creative Achievement Questionnaire in Qualtrics

The Creative Achievement Questionnaire (CAQ) is an instrument developed by Shelley Carson to measure a person's creative achievement. I came across it years ago when doing some pilot studies on the impact of creative achievement on learning Computer Science. In fact, for years I have been pointing nascent scholars to the paper by Carson et al. as an exemplar of careful attention to reliability and validity in social science.

A colleague and I thought it would be interesting to revisit some research questions around creativity and computer science education. I'll leave a discussion of the full study design for another time. For today, I want to focus on how I was able to adapt the CAQ for delivery in Qualtrics. Last time I used the CAQ, we deployed it on paper and scored by hand, which was fine for a small number of participants; now, we want to scale it upward. Additionally, we want each participant to know their CAQ score at the end. It took several hours of working with Qualtrics to figure out a way to do this.

The CAQ has several sections that look like this:

A. Visual Arts (painting, sculpture)
_0. I have no training or recognized talent in this area.
_1. I have taken lessons in this area.
_2. People have commented on my talent in this area.
_3. I have won a prize or prizes at a juried art show.
_4. I have had a showing of my work in a gallery.
_5. I have sold a piece of my work.
_6. My work has been critiqued in local publications.
*_7. My work has been critiqued in national publications.

The idea is that you mark the numbers that are applicable to you. Scoring a section in the simple case is easy: simply sum up the numbers of the items. For example, if you have taken painting lessons (1) and people have commented on your painting talent (2) then your score for this section is 3. If that's all there were to it, it would be simple to use Qualtrics' embedded scoring system, which allows for a numeric value to go with each option. The trick, however, is that asterisk on the last line: that means that you have to mark the space with the number of times this has happened and then multiply that by seven. So, if you have had your work critiqued in national publications twice, that contributes 14 to your CAQ score.

I put assembled a survey that looks like this:

The editor made it easy to add the text field to a checkbox, but what I need to be able to do is figure out what number was in the textbox and what numeric entry it is in the list of options, and then also to store that product—along with the sum of the other values—into an embedded variable.

I found the documentation for the Javascript API to be a bit opaque, in part because it assumes you already know their internal vocabulary for surveys. Matt Bloomfield's Medium article helped me to make some more sense out of how custom scripting in Qualtrics works. Using Chrome's developer tools, I was able to poke around in the preview view of the survey and find that the text field above has HTML like this:

<input aria-describedby="QR~QID7~8~VALIDATION" class="TextEntryBox InputText QR-QID7-8-TEXT QWatchTimer" data-runtime-textvalue="runtime.Choices.8.Text" id="QR~QID7~8~TEXT" name="QR~QID7~8~TEXT" style="width: 150px;" title="My work has been critiqued in national publications this many times:" type="text" value="" />

That id looks interesting, doesn't it? Its parent looks like this:

<label for="QR~QID7~8" id="QID7-8-label" class="MultipleAnswer" data-runtime-class-q-checked="runtime.Choices.8.Selected"><span>My work has been critiqued in national publications this many times:</span></label>

It appears that the text field shares an id prefix with its parent, and that I can identify a text field by looking for the suffix "~TEXT". Additional experimentation confirmed that the number just before that is the sequence number of the option within the question: this particular item is the eighth option, since CAQ items are numbered in true computer science fashion as 0–7.

After much experimentation, I was able to come up with one reusable Javascript function to score each section of the CAQ. It makes use of the fact that jQuery is built in but unconventionally invoked through a variable named, appropriately, jQuery. This allows me to search for an id by suffix, which I did not previously know was possible. This function was added to the "header" section of Qualtrics Look and Feel for the survey to ensure that it was included on each page.

 function scoreCaqSection(section, context) {  
  var selections = context.getSelectedChoices();  
  var score = 0;  
  for (var i = 0; i < selections.length; i++) {  
      var questionId = context.getQuestionInfo().QuestionID;  
      var selector = '[id$="' + questionId + '~' + selections[i] + '~TEXT"]';  
      // Check if there is a text entry associated with this question  
      var textField = jQuery(selector);  
      if (textField.val()) {  
       score += (selections[i] - 1) * textField.val();  
      } else {  
       score += selections[i] - 1;  
      }   
  }   
  Qualtrics.SurveyEngine.setEmbeddedData('CAQ.'+section, score); console.log('Score for ' + section + ': ' + score);  
 }</script>  

The last line of the script is setting an embedded data variable, so by the end of the 10 parts of the CAQ, there will be embedded variables called CAQ.A, CAQ.B, etc., that hold the CAQ component scores. Each individual portion of the CAQ can therefore include an simple custom script like this:

 Qualtrics.SurveyEngine.addOnPageSubmit(function(type) {  
      scoreCaqSection('A', this);  
 });  

The call indicates what section of the CAQ is being scored, which allows for proper storage of the partial score, as well as the question object itself. The final page of the survey then can use this awkward embedded expression computation to get the final score:

$e{e://Field/CAQ.A + e://Field/CAQ.B + e://Field/CAQ.C + e://Field/CAQ.D + e://Field/CAQ.E + e://Field/CAQ.F + e://Field/CAQ.G + e://Field/CAQ.H + e://Field/CAQ.I + e://Field/CAQ.J}

Yes, I did work for a little while on automating that expression to contain a loop instead of hardcoded values, but I realized I was spending an order of magnitude more time on that than by just typing them all in.

The final piece of the puzzle was to ensure that what the users enter into the text fields is actually a number. Qualtrics provides custom validation options, but I was surprised that it does not have a built in option to check that a field is a number. I needed to combine this with the idea that a number needed to be in the field only if the option was checked. Qualtrics will ensure that if you type in a box, the option is checked, but not the other way around. The custom validation for part A therefore looks like this:

This is checking that either the option is checked and there is a valid number in the field, or the option is not selected. Most of the cases are fairly simple, but the portion with multiple starred items requires more individual steps. We currently have a GA stress-testing our Qualtrics implementation to ensure that the validation and computation are correct, and this frees us up to work some more on the IRB protocol.

This was my first experience with Qualtrics, since I usually go for the much simpler Google Forms when I need to collect some data. It clearly has some more advanced features, and the custom scripting option is powerful, although it was slow going for me to make sense out of it. There's relatively little chatter in the forums I could find about adding custom Javascript, and the perfunctory nature of the API documentation makes me think that this is not a frequently-used feature by Qualtrics' customers. Since it took me several hours of work to get this far, I wanted to take a few minutes this morning to write up what I have and what I learned. Perhaps then, next time future me needs to remember how to do this, I can simply end up back here and figure out what I used to know. It wouldn't be the first time my blog has helped me with that.