Evaluation for item recommendation

[paoloRais]

Hello and thanks for this very interesting piece of software.

I am trying to use it for item recommendation based on positive-only input (interactions between a user and an item).
I have some questions related to the format of the input and the algorithm:

1. How should I encode the ratings?
   I set all the ratings as 1, obtaining a compact format input like:
   user, item, rating, location
   1, 2, 1, home
   2, 1, 1, home
   1, 3, 1, work
   ...
   Does the algorithm take care explicitly of the user-item negative interactions (i.e. when a user did not interact with an item)?
2. I would be interesting in obtaining top K recommendations where the candidate items list contains also the items that have already been interacted with (i.e. 'rated'). This is to obtain higher accuracy, since it seems that my users tend to interact again with items they have previously experienced.
   Is it possible to obtain this behaviour?
3. (related to 2.) I am using a test set in a separate file to assess precision. My test file contains items that have already been rated by a user (possibly in the same context too). This seems to generate an error, since by running CARSKit I obtain:

INFO ] 2016-04-12 17:41:59,227 -- WorkingPath: /home/paolo/raisDataScience/recommender_system/CARSkit/CARSKit.Workspace/
[INFO ] 2016-04-12 17:41:59,241 -- Your original rating data path: /home/paolo/raisDataScience/recommender_system/CARSkit/train.csv
[INFO ] 2016-04-12 17:41:59,241 -- Current working path: /home/paolo/raisDataScience/recommender_system/CARSkit/CARSKit.Workspace/
[WARN ] 2016-04-12 17:41:59,246 -- You rating data is in Compact format. CARSKit is working on transformation on the data format...
[INFO ] 2016-04-12 17:42:02,694 -- Data transformaton completed (from Compact to Binary format). See new rating file: /home/paolo/raisDataScience/recommender_system/CARSkit/CARSKit.Workspace/ratings_binary.txt
[INFO ] 2016-04-12 17:42:02,726 -- Dataset: ...ARSKit.Workspace/ratings_binary.txt
[INFO ] 2016-04-12 17:42:02,732 -- DataPath: /home/paolo/raisDataScience/recommender_system/CARSkit/CARSKit.Workspace/ratings_binary.txt
[INFO ] 2016-04-12 17:42:05,780 -- Rating data set has been successfully loaded.
[INFO ] 2016-04-12 17:42:05,878 --
/*****************************************************************************************************
*

• Dataset: /home/paolo/raisDataScience/recommender_system/CARSkit/CARSKit.Workspace/ratings_binary.txt
• User amount: 17113
• Item amount: 78
• Rate amount: 207493
• Context dimensions: 1 (location)
• Context conditions: 25 (location: 25)
• Context situations: 24
• Contextual Data density: 15.5447%
• Scale distribution: [1.0 x 207493]
• Average value of all ratings: 1.000000
• Standard deviation of all ratings: 0.000000
• Mode of all rating values: 1.000000
• Median of all rating values: 1.000000
  *
  *****************************************************************************************************/
  [INFO ] 2016-04-12 17:42:05,878 -- Dataset: ...ARSKit.Workspace/ratings_binary.txt
  [INFO ] 2016-04-12 17:42:05,878 -- DataPath: /home/paolo/raisDataScience/recommender_system/CARSkit/CARSKit.Workspace/ratings_binary.txt
  [INFO ] 2016-04-12 17:42:08,577 -- Rating data set has been successfully loaded.
  [INFO ] 2016-04-12 17:42:08,579 -- With Setup: test-set -f /home/paolo/raisDataScience/recommender_system/CARSkit/test.csv
  [INFO ] 2016-04-12 17:42:08,580 -- Dataset: ...recommender_system/CARSkit/test.csv
  [INFO ] 2016-04-12 17:42:08,580 -- DataPath: /home/paolo/raisDataScience/recommender_system/CARSkit/test.csv
  [ERROR] 2016-04-12 17:42:08,580 -- value already present: 0
  java.lang.IllegalArgumentException: value already present: 0
  at com.google.common.collect.HashBiMap.put(HashBiMap.java:238)
  at com.google.common.collect.HashBiMap.put(HashBiMap.java:215)
  at carskit.data.processor.DataDAO.readData(DataDAO.java:169)
  at carskit.main.CARSKit.runAlgorithm(CARSKit.java:317)
  at carskit.main.CARSKit.execute(CARSKit.java:115)
  at carskit.main.CARSKit.main(CARSKit.java:87)

Just to add all the information, my config file is:

dataset.ratings.wins=C:\Users\irecs\Desktop\Data\music\ratings.txt
dataset.ratings.lins=/home/paolo/raisDataScience/recommender_system/CARSkit/train.csv

dataset.social.wins=-1
dataset.social.lins=1

ratings.setup=-threshold -1 -datatransformation 1

recommender=CAMF_C

evaluation.setup=test-set -f /home/paolo/raisDataScience/recommender_system/CARSkit/test.csv
item.ranking=on -topN 10

output.setup=-folder CARSKit.Workspace -verbose on, off --to-clipboard --to-file results_all.txt

guava.cache.spec=maximumSize=200,expireAfterAccess=2m

num.factors=10
num.max.iter=120

learn.rate=2e-10 -max -1 -bold-driver

reg.lambda=0.001 -u 0.001 -i 0.001 -b 0.001 -s 0.001 -c 0.001
pgm.setup=-alpha 2 -beta 0.5 -burn-in 300 -sample-lag 10 -interval 100

similarity=PCC
num.shrinkage=-1

num.neighbors=10

AoBPR=-lambda 0.3
BUCM=-gamma 0.5
BHfree=-k 10 -l 10 -gamma 0.2 -sigma 0.01
FISM=-rho 100 -alpha 0.5
Hybrid=-lambda 0.5
LDCC=-ku 20 -kv 19 -au 1 -av 1 -beta 1
PD=-sigma 2.5
PRankD=-alpha 20
RankALS=-sw on
RSTE=-alpha 0.4
SLIM=-l1 1 -l2 5 -k 50
CSLIM_C=-lw1 1 -lw2 5 -lc1 1 -lc2 5 -k 20 -als 0
CSLIM_CUCI=-lw1 1 -lw2 5 -lc1 1 -lc2 5 10 -1 -als 0
CSLIM_CI=-lw1 1 -lw2 5 -lc1 1 -lc2 5 -k 20 -als 0
CSLIM_CU=-lw1 1 -lw2 5 -lc1 1 -lc2 5 -k 10 -als 0
GCSLIM_CC=-lw1 1 -lw2 5 -lc1 1 -lc2 5 -k -1 -als 0
CSLIM_ICS=-lw1 1 -lw2 5 -k -1 -als 0
CSLIM_LCS=-lw1 1 -lw2 5 -k -1 -als 0
CSLIM_MCS=-lw1 1 -lw2 5 -k -1 -als 0
GCSLIM_ICS=-lw1 1 -lw2 5 -k -1 -als 0
GCSLIM_LCS=-lw1 1 -lw2 5 -k -1 -als 0
GCSLIM_MCS=-lw1 1 -lw2 5 -k -1 -als 0
FM=-lw 0.01 -lf 0.02

[irecsys]

Hello, thanks for your interests!

1). You can use binary values. For example, 1 as user has interactions with item, and 0 as no interactions. it is viewed as ratings or probibilities for users to have interactions on items

your data format is correct, but you should also have negative feedbacks in your data.

2). well, it depends on the setting. For example, if a user did not interact with an item in a spcific context, e.g., home, then he or she is possible to interact with it again in another context, like cinema. But, in our evaluations, we assume users will NOT interact with a same item within a same context in the past. You can change the evalRankings() by yourself to achieve your own goals.

3). My answers in part 1 and 2 can help you understand the outputs. In your data, the only rating is 1.0. In terms of the error below, you have duplicated interaction records, which is related to the answer in part 2.

2016-04-12 17:42:08,580 -- value already present: 0 java.lang.IllegalArgumentException: value already present: 0 at com.google.common.collect.HashBiMap.put(HashBiMap.java:238) at com.google.common.collect.HashBiMap.put(HashBiMap.java:215) at carskit.data.processor.DataDAO.readData(DataDAO.java:169) at carskit.main.CARSKit.runAlgorithm(CARSKit.java:317) at carskit.main.CARSKit.execute(CARSKit.java:115) at carskit.main.CARSKit.main(CARSKit.java:87)

[paoloRais]

Thanks for your reply!

I changed the input file and managed to make it work. It seems that there is some issue in the automatic conversion between a compact input and a binary input. For example, the compact version of the attached input files does not work (I get an error while reading the test file), while the (equivalent) binary input version works fine.
compact_test_toy.txt
compact_train_toy.txt
test_toy.txt
train_toy.txt

Following your advice, I modified evalRankings() by removing the part that excludes the item that have already been ranked. It seems to work fine so far, I am running some test.

[irecsys]

Thanks for your information. I will double check the data convertion.

[paoloRais]

Hello again,

After some tinkering I am a bit unsure about the best strategy to adapt for generating the top-k recommendations using positive-only binary input for the rating.

Checking the literature it appears to me that all of the context-aware algorithms are designed for the case where the input contains ratings. The case of positive-only binary input is trickier: I can't just set the rating to 0 for the user-item-context combinations for which I didn't observe an interaction, since it would penalise those items, assuming that a lack of interaction is a negative rating; in addition, the input file containing all the possible non-observed combinations would be enormous.
This situation is thus quite different from binarized ratings, in which the rating 0 has a genuine meaning.

I already have reasonable non-context-aware baselines (SLIM and BPR) that work correctly in this case. In your opinion, what can I do to get reasonable context-aware top-k items?

[irecsys]

Hello, the design in CARSKit actually follows the original desing in LibRec. So, there is a setting to set rating threshold so that you can bin the profiles to relevant and irrelevant ones. Well, in this case, you can view them as probabilities, for example, what is the probablity the user will like the items in specific contexts. In recommender systems, we'd like to recommender the items with predicted rating larger than a threshold, i.e., we do not want to recommend a book to a user, while this user may simply rate the book by a two star.

I agree with you that the design is not that good. In my view, the best way is to only use the rating threshold in evaluation process. For example, we do not bin the profiles at the beginning, and simplyh use rating threshold when we recommend items to the users -- predicted rating less than the threshold will not be pushed to the user. If you like this idea, you can download a copy of the source codes and then change them accordingly.

I will do some experiments and evaluations to examine which way is better. If the later one is better, I will update the library accordingly. Thanks for your comments and sorry for my late response.

[irecsys]

Hello, I have changed the evaluation for topN recommendation.

If a rating threshold is set for topN recommendation, the threshold will not be used initially to bin the ratings to 0s and 1s. Instead, the threshold is only adopted in the evalRanking() process, where only the items with a larger predicted rating (than threshold) will be recommended. Thanks for your advice and suggestions!

[ajohannsdottir]

Just to clarify, I have positive-only input also, do I need to explicitly create data transactions with 0 rating or will the model automatically do that for me ?

[irecsys]

Hello, I think so, since there must be negative feedbacks to make it work

[SKPalu]

Has anyone on this list used CARSKit in a distributed environment? If so, can you give a pointer of how to do this please? I'm thinking if CARSKit can be used in Spark.