Update benchmarks with FuxiCTR_v1.2.2

LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2021 BARS
+   Copyright (C) 2022 The BARS Project.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,21 +1,20 @@
-# BARS 
+# BARS
 
-Towards Open Benchmarking for Recommender Systems: https://openbenchmark.github.io
+A First Step Towards Open Benchmarking for Recommender Systems: https://openbenchmark.github.io
 
-Despite the significant progress made in both research and practice of recommender systems over the past two decades, there is a lack of a widely-recognized benchmarking suite in this field. This not only increases the difficulty in reproducing existing studies, but also incurs inconsistent experimental results among them, which largely limit the practical value and potential impact of research in this field. In this project, we present our initiative project aimed for open benchamrking for recommender systems. The benchmarking project allows anyone to easily follow and contribute, and hopefully drive more solid and reproducible research on recommender systems.
+Despite the significant progress made in both research and practice of recommender systems over the past two decades, there is a lack of a widely-recognized benchmark in this field. This not only increases the difficulty in reproducing existing studies, but also incurs inconsistent experimental results among them, which largely limit the practical value and potential impact of research in this field. In this project, we make our initiative efforts towards open benchamrking for recommender systems. The BARS benchmark project allows anyone to easily follow and contribute, and hopefully drive more solid and reproducible research on recommender systems.
 
 The BARS benchmark currently covers the following two tasks. 
 
-+ [CTR Prediction](./ctr_prediction)
-+ [Candidate Item Matching](./candidate_matching)
-
++ [BarsCTR: A Benchmark for CTR Prediction](./ctr_prediction)
++ [BarsMatch: A Benchmark for Candidate Item Matching](./candidate_matching)
 
 ## Contributing
-We welcome any contribution that could help improve the BARS benchmark. Check the [start guide on how to contribute](https://github.com/openbenchmark/BARS/blob/master/CONTRIBUTING.md).
 
+We welcome any contribution that could help improve the BARS benchmark. Check the [start guide on how to contribute](https://github.com/openbenchmark/BARS/blob/master/CONTRIBUTING.md).
 
 ## Discussion
+
 If you have any questions or feedback about the BARS benchamrk, please [start a discussion here](https://github.com/openbenchmark/BARS/discussions/new), or join our [WeChat group](https://gitee.com/xpai/Images/raw/master/1637915312191.jpg).
 
 ![Scan WeChat QR](./images/wechat.jpg)
-

candidate_matching/README.md

@@ -1,14 +1,14 @@
-# BARS-Matching Benchmark
+# BarsMatch Benchmark
 
-Open Benchmarking for Candidate Item Matching https://openbenchmark.github.io/candidate-matching
+BarsMatch: A Benchmark for Candidate Matching https://openbenchmark.github.io/candidate-matching
 
 The repo contains four parts:
 
-+ [Paper List](./papers.json)
-+ [Dataset Splits](./datasets)
-+ [Benchmark Models](./benchmarks)
-+ [Benchmark Results](./leaderboard)
++ [Paper List of Candidate Matching](./papers.json)
++ [Benchmark Datasets](./datasets)
++ [Benchmark Settings](./benchmarks)
++ [Benchmark Leaderboards](./leaderboard)
 
 ## Contributing
-We welcome any contribution to help improve the benchmark. Check the [guide on how to contribute](https://github.com/xue-pai/Open-Match-Benchmark/blob/master/CONTRIBUTING.md).
 
+We welcome any contribution to help improve the benchmark. Check the [guide on how to contribute](https://github.com/xue-pai/Open-Match-Benchmark/blob/master/CONTRIBUTING.md).

ctr_prediction/README.md

@@ -1,11 +1,12 @@
-# BARS-CTR Benchmark
+# BarsCTR Benchmark
 
-Open Benchmarking for CTR Prediction https://openbenchmark.github.io/ctr-prediction
+BarsCTR: A Benchmark for CTR Prediction https://openbenchmark.github.io/ctr-prediction
 
 The repo contains four parts:
 
-+ [Paper List](./papers.json)
-+ [Dataset Splits](./datasets)
-+ [Benchmark Models](./benchmarks)
-+ [Benchmark Results](./leaderboard)
++ [Paper List of CTR Prediction](./papers.json)
++ [Benchmark Datasets](./datasets)
++ [Benchmark Settings](./benchmarks)
++ [Benchmark Leaderboards](./leaderboard)
+
 

ctr_prediction/benchmarks/AFM/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03.csv

@@ -0,0 +1,8 @@
+ 20220126-192306,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_002_4a58edb9 --gpu 1,[exp_id] AFM_avazu_x1_002_4a58edb9,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.738503 - logloss: 0.401095,[test] AUC: 0.757359 - logloss: 0.370485
+ 20220126-194525,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_008_18a26b69 --gpu 3,[exp_id] AFM_avazu_x1_008_18a26b69,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.738329 - logloss: 0.401989,[test] AUC: 0.756950 - logloss: 0.371083
+ 20220126-194519,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_004_9556c70e --gpu 3,[exp_id] AFM_avazu_x1_004_9556c70e,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.738392 - logloss: 0.401394,[test] AUC: 0.756554 - logloss: 0.371064
+ 20220126-193728,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_003_13adef36 --gpu 2,[exp_id] AFM_avazu_x1_003_13adef36,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.739158 - logloss: 0.400023,[test] AUC: 0.756514 - logloss: 0.371021
+ 20220126-191609,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_006_1569fc76 --gpu 1,[exp_id] AFM_avazu_x1_006_1569fc76,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.737004 - logloss: 0.401815,[test] AUC: 0.756193 - logloss: 0.371003
+ 20220126-192213,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_005_2c338e74 --gpu 0,[exp_id] AFM_avazu_x1_005_2c338e74,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.738300 - logloss: 0.401358,[test] AUC: 0.756059 - logloss: 0.371229
+ 20220126-193039,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_007_014385a7 --gpu 2,[exp_id] AFM_avazu_x1_007_014385a7,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.737730 - logloss: 0.401879,[test] AUC: 0.755318 - logloss: 0.371964
+ 20220126-190109,[command] python run_expid.py --version pytorch --config Avazu/AFM_avazu_x1/AFM_avazu_x1_tuner_config_03 --expid AFM_avazu_x1_001_73860ac7 --gpu 0,[exp_id] AFM_avazu_x1_001_73860ac7,[dataset_id] avazu_x1_3fb65689,[train] N.A.,[val] AUC: 0.737708 - logloss: 0.401403,[test] AUC: 0.754699 - logloss: 0.371728

ctr_prediction/benchmarks/AFM/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02.csv

@@ -0,0 +1,9 @@
+ 20220127-152904,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_004_954c0ecc --gpu 3,[exp_id] AFM_criteo_x1_004_954c0ecc,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.804069 - logloss: 0.447414,[test] AUC: 0.804436 - logloss: 0.446977
+ 20220127-152129,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_005_d8ea9c7d --gpu 4,[exp_id] AFM_criteo_x1_005_d8ea9c7d,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.803815 - logloss: 0.447517,[test] AUC: 0.804089 - logloss: 0.447154
+ 20220127-144735,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_008_6df419fc --gpu 7,[exp_id] AFM_criteo_x1_008_6df419fc,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.803721 - logloss: 0.447630,[test] AUC: 0.804026 - logloss: 0.447262
+ 20220127-141400,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_007_4138fd26 --gpu 6,[exp_id] AFM_criteo_x1_007_4138fd26,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.803214 - logloss: 0.448275,[test] AUC: 0.803638 - logloss: 0.447759
+ 20220127-231218,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_009_c044ff29 --gpu 0,[exp_id] AFM_criteo_x1_009_c044ff29,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.802828 - logloss: 0.448389,[test] AUC: 0.803153 - logloss: 0.447985
+ 20220127-160557,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_006_089b8bc5 --gpu 5,[exp_id] AFM_criteo_x1_006_089b8bc5,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.802620 - logloss: 0.448513,[test] AUC: 0.802938 - logloss: 0.448099
+ 20220128-004229,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_001_710faa0a --gpu 0,[exp_id] AFM_criteo_x1_001_710faa0a,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.802390 - logloss: 0.448821,[test] AUC: 0.802615 - logloss: 0.448479
+ 20220127-173323,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_002_8ec85047 --gpu 1,[exp_id] AFM_criteo_x1_002_8ec85047,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.801091 - logloss: 0.449804,[test] AUC: 0.801306 - logloss: 0.449471
+ 20220127-174824,[command] python run_expid.py --version pytorch --config Criteo/AFM_criteo_x1/AFM_criteo_x1_tuner_config_02 --expid AFM_criteo_x1_003_53db00dc --gpu 2,[exp_id] AFM_criteo_x1_003_53db00dc,[dataset_id] criteo_x1_7b681156,[train] N.A.,[val] AUC: 0.799616 - logloss: 0.451056,[test] AUC: 0.799837 - logloss: 0.450732

ctr_prediction/benchmarks/AFM/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02.csv

@@ -0,0 +1,18 @@
+ 20220126-214533,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_011_2d591f68 --gpu 0,[exp_id] AFM_frappe_x1_011_2d591f68,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.971434 - logloss: 0.218243,[test] AUC: 0.969733 - logloss: 0.226424
+ 20220126-214539,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_012_242dd9fd --gpu 0,[exp_id] AFM_frappe_x1_012_242dd9fd,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.969669 - logloss: 0.212333,[test] AUC: 0.967725 - logloss: 0.223535
+ 20220126-214457,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_010_d58884d3 --gpu 1,[exp_id] AFM_frappe_x1_010_d58884d3,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.967273 - logloss: 0.232043,[test] AUC: 0.966885 - logloss: 0.232084
+ 20220126-215042,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_017_a771093b --gpu 0,[exp_id] AFM_frappe_x1_017_a771093b,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.963643 - logloss: 0.219164,[test] AUC: 0.962184 - logloss: 0.225581
+ 20220126-213927,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_005_32d99fe9 --gpu 0,[exp_id] AFM_frappe_x1_005_32d99fe9,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.962525 - logloss: 0.234119,[test] AUC: 0.961885 - logloss: 0.235727
+ 20220126-214236,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_008_35889f27 --gpu 0,[exp_id] AFM_frappe_x1_008_35889f27,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.962589 - logloss: 0.230594,[test] AUC: 0.961529 - logloss: 0.234464
+ 20220126-214040,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_001_a874e8ba --gpu 0,[exp_id] AFM_frappe_x1_001_a874e8ba,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.962155 - logloss: 0.238215,[test] AUC: 0.961165 - logloss: 0.242596
+ 20220126-215122,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_018_c86fc45c --gpu 0,[exp_id] AFM_frappe_x1_018_c86fc45c,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.962642 - logloss: 0.227133,[test] AUC: 0.960748 - logloss: 0.232967
+ 20220126-215025,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_014_9895c7ee --gpu 0,[exp_id] AFM_frappe_x1_014_9895c7ee,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.962579 - logloss: 0.232125,[test] AUC: 0.960552 - logloss: 0.238697
+ 20220126-214416,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_009_19945476 --gpu 1,[exp_id] AFM_frappe_x1_009_19945476,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.961751 - logloss: 0.232026,[test] AUC: 0.959939 - logloss: 0.237911
+ 20220126-215006,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_015_ae17c0a8 --gpu 1,[exp_id] AFM_frappe_x1_015_ae17c0a8,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.959339 - logloss: 0.241750,[test] AUC: 0.957667 - logloss: 0.246371
+ 20220126-214812,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_013_5671dff3 --gpu 1,[exp_id] AFM_frappe_x1_013_5671dff3,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.959136 - logloss: 0.239104,[test] AUC: 0.956986 - logloss: 0.247043
+ 20220126-214911,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_016_26a3b9ce --gpu 1,[exp_id] AFM_frappe_x1_016_26a3b9ce,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.957464 - logloss: 0.241843,[test] AUC: 0.956351 - logloss: 0.244413
+ 20220126-213854,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_002_17267821 --gpu 1,[exp_id] AFM_frappe_x1_002_17267821,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.957650 - logloss: 0.247867,[test] AUC: 0.956021 - logloss: 0.251908
+ 20220126-213927,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_007_5105e46b --gpu 1,[exp_id] AFM_frappe_x1_007_5105e46b,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.957627 - logloss: 0.239914,[test] AUC: 0.955993 - logloss: 0.243861
+ 20220126-213606,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_004_70fd3ee1 --gpu 1,[exp_id] AFM_frappe_x1_004_70fd3ee1,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.950848 - logloss: 0.257810,[test] AUC: 0.949729 - logloss: 0.260169
+ 20220126-213734,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_003_d58825c0 --gpu 0,[exp_id] AFM_frappe_x1_003_d58825c0,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.948334 - logloss: 0.264631,[test] AUC: 0.947404 - logloss: 0.266915
+ 20220126-214103,[command] python run_expid.py --version pytorch --config Frappe/AFM_frappe_x1/AFM_frappe_x1_tuner_config_02 --expid AFM_frappe_x1_006_29f71b8f --gpu 1,[exp_id] AFM_frappe_x1_006_29f71b8f,[dataset_id] frappe_x1_04e961e9,[train] N.A.,[val] AUC: 0.946413 - logloss: 0.268847,[test] AUC: 0.945738 - logloss: 0.270422

ctr_prediction/benchmarks/AFM/AFM_movielenslatest_x1/AFM_movielenslatest_x1_tuner_config_04.csv

@@ -0,0 +1,4 @@
+ 20220128-121755,[command] python run_expid.py --version pytorch --config Movielens/AFM_movielenslatest_x1/AFM_movielenslatest_x1_tuner_config_04 --expid AFM_movielenslatest_x1_002_76325415 --gpu 1,[exp_id] AFM_movielenslatest_x1_002_76325415,[dataset_id] movielenslatest_x1_cd32d937,[train] N.A.,[val] AUC: 0.947478 - logloss: 0.265064,[test] AUC: 0.947193 - logloss: 0.265332
+ 20220128-121731,[command] python run_expid.py --version pytorch --config Movielens/AFM_movielenslatest_x1/AFM_movielenslatest_x1_tuner_config_04 --expid AFM_movielenslatest_x1_001_256a62c3 --gpu 0,[exp_id] AFM_movielenslatest_x1_001_256a62c3,[dataset_id] movielenslatest_x1_cd32d937,[train] N.A.,[val] AUC: 0.947218 - logloss: 0.265006,[test] AUC: 0.947040 - logloss: 0.264900
+ 20220128-121850,[command] python run_expid.py --version pytorch --config Movielens/AFM_movielenslatest_x1/AFM_movielenslatest_x1_tuner_config_04 --expid AFM_movielenslatest_x1_004_7d0f0534 --gpu 1,[exp_id] AFM_movielenslatest_x1_004_7d0f0534,[dataset_id] movielenslatest_x1_cd32d937,[train] N.A.,[val] AUC: 0.945872 - logloss: 0.268186,[test] AUC: 0.945811 - logloss: 0.267917
+ 20220128-121725,[command] python run_expid.py --version pytorch --config Movielens/AFM_movielenslatest_x1/AFM_movielenslatest_x1_tuner_config_04 --expid AFM_movielenslatest_x1_003_fffe0eb5 --gpu 0,[exp_id] AFM_movielenslatest_x1_003_fffe0eb5,[dataset_id] movielenslatest_x1_cd32d937,[train] N.A.,[val] AUC: 0.944607 - logloss: 0.273424,[test] AUC: 0.944349 - logloss: 0.273750