Analysis of the Results of a Card Sorting Test: A Detailed Approach

What is card sorting, and why did I use it?

Card sorting is a technique used to understand how people organize and group information. It is a type of test that helps design teams decide how to structure the information architecture of a website or app. We can use this technique to understand how much users already know about your product or service. Card sorting also reveals their mental models, helping shape the best way to organize the information architecture.

I used this methodology to design the menu for the new technical panel of Aruba’s Cloud and Dedicated Server services. I chose to use card sorting because I wanted to group the products and services in a way that made sense to my users, and I wanted to understand how users perceive the concepts related to Cloud and Dedicated Server products and services.

How the test was conducted

I conducted the test remotely and unmoderated to save time and costs. I recruited users through TestingTime and used OptimalWorkshop to prepare the test. Recruiting users with such specific skills was not easy, as it became evident that many struggled with the task.

I opted for open card sorting: I provided participants with the complete list of Aruba’s Cloud and Dedicated Server services and asked them to create categories and group the services. This approach helped us understand:

How customers comprehend and analyze the information
Where people expect to find information
Whether there are different groups of users who think differently about the information

A total of about 90 users participated in the test, with 85 recruited from TestingTime and the others from Aruba’s Enterprise clients. The recruitment criteria included users who were Cloud service users (providing some examples) and, if possible, were developers or system administrators.

Analysis of the results

Once the test was completed, it was essential to carefully analyze the results to extract meaningful insights and make the right design decisions. I obviously ended up with a lot of data, but until I analyzed and synthesized it, it didn’t provide many insights into how users perceived the content.

‍

Here are the key steps I followed to conduct the analysis of the card sorting results:

Data collection
I made sure to have all the data collected during the card sorting test: the categories created by participants, specific groupings of the cards, and any additional comments or observations.
Data organization
Initially, I downloaded the raw data from OptimalWorkshop in .csv format. I entered the names of the services in the rows and the categories created by the customers in the columns on an Excel file. I then transcribed the users’ choices into the file.
Assignment frequency
After transcribing the users’ choices into the file, I color-coded the cells based on how many times an item was placed in a category. I initially visualized the results by considering all the categories created by users. Then, I tried to group similar categories, standardizing the results.
I used dark blue for very clear choices, light blue for fairly clear choices, and gray for uncertain choices.
Inter-participant consistency
Next, I evaluated the consistency among participants in grouping the items. I looked for common patterns or trends among the various participants. At the same time, I paid attention to significant discrepancies that could indicate ambiguity in the content of the items or the logic of the categories.
Qualitative analysis
In addition to the quantitative analysis of the data, I also considered the qualitative aspects of the participants’ responses. Where present, I included comments, questions, or observations that provide context or explanations for why they organized the cards in a certain way.
Pattern identification
I looked for recurring patterns or themes in the categories created by the participants. To do this, I used both the raw data and the graphs generated by OptimalWorkshop. In particular, the Similarity Matrix provided a fairly clear idea of the items that users considered related.
Iteration and reflection
I used the results of the analysis to redesign the technical panel menu. Along with the project team (especially the product managers), we found that the categories identified by the users were generally in line with what we had imagined.
Result validation
As mentioned earlier, I validated the analysis results by involving other team members. We ensured that the conclusions drawn were supported by the data and that any proposed changes were relevant and effective.

‍

‍

Conclusions

In summary, analyzing the results of a card sorting test requires a balanced approach between quantitative and qualitative analysis, and between “manual” analysis and tool-assisted analysis. It is crucial to critically reflect on emerging patterns and their implications for design. By using these steps as a guide, valuable insights can be gained to improve the usability and overall experience of the product or service being designed.

References