A cross-cultural user evaluation of product recommender interfaces
Summary (4 min read)
1. INTRODUCTION
- Online systems that help users select the most preferential item from a large electronic catalog are known as product search and recommender systems.
- Studies show that customer trust is positively associated with customers’ intentions to transact, purchase a product, and return to the website [9].
- The authors have primarily studied trust-building by the different design dimensions of explanation interfaces, given explanations’ potential benefits to improve users’ confidence about recommendations and their acceptance of the system [10,18].
- In order to accelerate users’ decision process by saving their information-searching effort in reviewing all recommended items, the authors have proposed a so called preference-based organization technique.
1.1 Summary of Previous Studies
- A carefully conducted user survey (53 subjects) first showed some interesting observations regarding the influence of explanations on trust building and the effectiveness of the organization-based recommender interface [4].
- Moreover, the organized view of recommendations was largely favored than the traditional “why”based list view, since it was perceived to more likely accelerate the process of product comparison and choice making.
- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
1.2 Contribution of Our Current Work
- The previous two experiments pointed out promising benefits of the organization interface regarding its trust-inspiring ability.
- They motivated us to further evaluate the interface’s practical performance in a more realistic and interactive system where it serves as the computation and explanation of personalized recommendations according to users’ preferences (rather than based on products’ general popularity).
- In addition, the authors were interested in identifying whether people from different categories of cultural backgrounds (i.e., oriental and western cultures) would all react actively to the organizationbased system.
- As for upper-level competence perceptions, perceived ease of use and perceived usefulness, the two primary determining elements of convincing users to accept a technology [6], were included, besides decision confidence, perceived effort and satisfaction.
- This paper is hence organized as follows: section 2 and 3 describes the organization-based interface and its function in an implemented prototype system; section 4 introduces the cross-cultural user evaluation’s design and experimental procedure; section 5 presents results from the study; and section 6 concludes the paper’s work.
2. ORGANIZATION-BASED
- The organization interface has been developed to compute and categorize recommended products, and use the category title (e.g. “these products have cheaper price and longer battery life, but slower processor speed and heavier weight”) as the explanation of multiple products .
- To derive effective principles for this interface design, the authors tested 13 paper prototypes by means of pilot studies and user interviews, and finally concluded five design principles.
- Briefly speaking, the algorithm contains three main steps: Step 1: the user preferences over all products are represented as a weighted additive form of value functions according to the multiattribute utility theory (MAUT) [11].
- Each tradeoff vector is a set of (attribute, tradeoff) pairs, where tradeoff indicates the improved (denoted as ↑) or compromised (↓) property of the product’s attribute value compared to the same attribute of the top candidate.
- The authors select ones with higher tradeoff utilities (i.e., gains against losses relative to the top candidate and user preferences) in consideration of both category titles and their associated products.
3. PROTOTYPE SYSTEM
- The authors implemented the organization interface in a product recommender system, which is in particular to assist users in searing for high-involvement products (e.g., notebooks, digital cameras, and cars) for which people will be willing to spend considerable effort in locating a desired choice, in order to avoid any financial damage or emotional burden.
- A typical interaction procedure with the system can be as follows.
- Among these products, the user can either choose one as her final choice, or select a neartarget and click “Better Features” to view recommended products with some better values than the selected one.
- Specifically, the weight of improved attribute(s) that appears in the examined category title will be increased and the weight of compromised one(s) be decreased.
- The user could choose to optimize any attributes’ values (e.g., $100 cheaper) and accept compromise(s) on one or more less important attributes, which revisions will be directly reflected in her preference model.
4.1 Cultural Difference
- It is commonly recognized that elements of a user interface appropriate for one culture may not be appropriate for another.
- Barber and Badre [2] claimed that Americans prefer websites with a white background, while Japanese dislike the white and Chinese favor the red background.
- People are deeply influenced by the cultural values and norms they hold.
- The most typical classification is Oriental vs. Western cultures.
- In their experiment, the participants were mainly coming from two nations respectively representing the two different cultures: China (oriental culture) and Switzerland (western culture).
4.2 Participants and Materials
- In total, 120 participants volunteered to take part in the experiment.
- Another 60 subjects are mainly students in their university, and 41 of them are Swiss and the others are from European countries nearby like France, Italy and Germany.
- Another system differs from it only in respect of the recommendation display.
- Users can also freely specify and revise preferences, examine products’ detailed specifications, and in-depth compare neartargets in a consideration set.
- They were both developed with two product catalogs: 64 digital cameras each constrained by 8 main attributes (manufacturer, price, resolution, optical zoom, etc), and 55 tablet PCs by 10 main attributes (manufacturer, price, processor speed, weight, etc).
4.3 Evaluation Criteria
- The measured variables used in previous user studies (e.g., perceived effort, return intention) [17] were extended to include more subjective aspects, which are essentially related to the competence-based trust model the authors have established for recommender systems [4].
- The model consists of three main constructs: system-design features, competence-inspired trust, and trust-induced behavioral intentions.
- Besides, the authors included questions about decision confidence, cognitive effort, and satisfaction.
- Most of them came from existing literatures where they have been repeatedly shown to exhibit strong content validity [12].
- Except for these subjective criteria, the authors also measured participants’ objective decision accuracy and effort.
4.4 Experiment Design and Procedure
- A 22 full-factorial between-group experiment design was used.
- (oriental culture, western culture) and (ORG, LIST), also known as The manipulated factors are.
- At the beginning, the participant was required to fill in a prequestionnaire about her/his personal information and subjective opinions on the priority order of different factors in influencing her/his general trust formation in an e-commerce website.
- Then s/he was asked to use the assigned system to locate a product that s/he most preferred and would purchase if given the opportunity.
- After the choice was made, the participant was asked to answer post-study questions related to all of the measured subjective variables.
4.5 Hypotheses
- Regarding the culture difference, the authors postulated that it would not have significant influence on users’ decision behavior in either ORG or LIST.
- That is, people would react similarly to the system no matter which cultural background s/he is from.
- The ORG system was further hypothesized to outperform LIST, especially in terms of subjective constructs related to user trust, owing to the replacement of the list view of recommendations with the organized view.
5.1 Objective Measures
- The authors first measured users’ objective performance in the two systems (see Table 3).
- The authors respectively compared the results between two groups of people from the same cultural background but used different systems, two groups of people using the same system but from different cultures, and the overall comparison of ORG and LIST taking into account of all study participants.
- The between-group analyses were done by the Student t-test assuming unequal variances, with estimated power of 86% under the assumption of “large” effect size, which power indicates a high likelihood of detecting a significant effect provided one exists.
- All of the differences are not significant.
- The overall interaction cycles consumed in ORG is higher than in LIST, but the difference does not reach to a significant level.
5.2 Subjective Measures
- The authors further examined whether the cultural background would influence users’ subjective perceptions with the system, and which system would perform better respecting these subjective aspects.
- Analysis of all users’ responses indicates that ORG obtained positively higher scores on all of them, 6 of which are significantly better (see Table 4).
- More concretely, the participants using ORG on average expressed significantly higher perceived recommendation quality, higher perceived ease of use, higher perceived usefulness, lower perceived effort, higher satisfaction and higher intention to save effort in repeated visits, compared to the rates of another group with LIST.
- As for the other two system-design features, the two systems did not exhibit significant differences, which might be because they both provide explanations (for recommendation transparency) and preference revision tools (for user-control).
- All of the results hence infer that oriental subjects’ reaction to ORG was indeed more positively stronger than western users’, which is primarily reflected on their perceived recommendation quality, decision confidence and cognitive effort.
5.3 Other Results
- In the pre-questionnaire, the authors asked each participant to rate a set of statements about the relative importance of factors influencing their perception of an e-commerce website’s general trustworthiness, their intention to purchase a product on the website and intention to repeatedly visit it for products’ information.
- Table 5 shows the priority order of these factors for each question from both oriental and western subjects.
- All average scores are beyond the medium level (“moderately important”).
- For the trustworthiness perception, the priority order of the five factors is the same between two groups of users: the website’s integrity (e.g., product quality, security, delivery service, etc) is the most important, followed by its reputation, price info, and competences in helping users find ideal products and providing good recommendations.
- As a matter of fact, the most important factor leading to users’ return intention is that the website can help them effectively find a product they really like.
6. CONCLUSION
- The authors presented a user study that evaluated the organization-based recommender system in a cross-cultural experiment setup.
- In-depth analysis concerning cultural impacts further shows that some of these significant phenomena were observably stronger among oriental participants, implying that oriental users will likely be more actively reacting to the organization interface once it replaces the traditional list view.
- Another implication is for the user evaluation of recommender systems.
- The authors believe that other researchers will profit from their evaluation methods when they conduct similar types of experiments.
Did you find this useful? Give us your feedback
Citations
9 citations
Cites background from "A cross-cultural user evaluation of..."
...Namely, only differences in the evaluation of recommendation interfaces [3], preferences for certain items [22], and attitudes towards mobile recommenders [4] have been studied....
[...]
9 citations
8 citations
7 citations
6 citations
References
40,975 citations
"A cross-cultural user evaluation of..." refers background in this paper
...As for upper-level competence perceptions, perceived ease of use and perceived usefulness, the two primary determining elements of convincing users to accept a technology [6], were included, besides decision confidence, perceived effort and satisfaction....
[...]
...The overall competence is composed of two crucial variables: perceived ease of use and perceived usefulness, which have been determined as the primary factors of persuading users to accept and use a technology [6]....
[...]
40,720 citations
15,645 citations
"A cross-cultural user evaluation of..." refers methods in this paper
...Step 3: all of the tradeoff vectors are then organized into different categories by utilizing an association rule mining tool [1] to discover the recurring subsets of (attribute, tradeoff) pairs among them....
[...]
8,895 citations
2,758 citations
"A cross-cultural user evaluation of..." refers background in this paper
..., the integrity and benevolence aspects of trust formation) [8], and less from the website’s competence such as its decision agent’s ability in providing good recommendations and explaining its results....
[...]