Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for many CRAN mirrors #64

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Conversation

louisaslett
Copy link

This PR is to address problems 1 and 2 described in issue #63.

I will update this PR thread with the revised results of running the test given in the above issue.

@louisaslett
Copy link
Author

This first commit address problem 1 in issue #63.

Prior to this fix, we have:

> getCRANmirrors()$URL[79]
[1] "https://www.stats.bris.ac.uk/R/"
> chooseCRANmirror(ind = 79)
> foghorn:::cran_url()
[1] "https://www.stats.bris.ac.uk"

After fixing we have:

> getCRANmirrors()$URL[79]
[1] "https://www.stats.bris.ac.uk/R/"
> chooseCRANmirror(ind = 79)
> foghorn:::cran_url()
[1] "https://www.stats.bris.ac.uk/R"

This resolves the test in issue #63 for a large number of mirrors, but not all:

> for(i in 1:nrow(getCRANmirrors())) {
+   chooseCRANmirror(ind = i)
+   tryCatch({
+     cran_details("mlmc")
+     cran_results("[email protected]")
+     n_cran_flavors()
+     cli::cli_alert_success(paste0("Success for mirror ", i, " => ", getOption("repos")[[1]]))
+   },
+   error = \(e) cli::cli_alert_warning(paste0("Failed for mirror ", i, " => ", getOption("repos")[[1]])))
+ }
✔ Success for mirror 1 => https://cloud.r-project.org
! Failed for mirror 2 => http://mirror.fcaglp.unlp.edu.ar/CRAN
✔ Success for mirror 3 => https://cran.csiro.au
! Failed for mirror 4 => https://mirror.aarnet.edu.au/pub/CRAN
✔ Success for mirror 5 => https://cran.ms.unimelb.edu.au
✔ Success for mirror 6 => https://cran.wu.ac.at
✔ Success for mirror 7 => https://www.freestatistics.org/cran
! Failed for mirror 8 => https://ftp.belnet.be/mirror/CRAN
! Failed for mirror 9 => https://cran-r.c3sl.ufpr.br
! Failed for mirror 10 => https://vps.fmvz.usp.br/CRAN
! Failed for mirror 11 => https://brieger.esalq.usp.br/CRAN
✔ Success for mirror 12 => https://ftp.uni-sofia.bg/CRAN
! Failed for mirror 13 => https://muug.ca/mirror/cran
! Failed for mirror 14 => https://mirror.csclub.uwaterloo.ca/CRAN
✔ Success for mirror 15 => https://cran.mirror.rafal.ca
! Failed for mirror 16 => https://cran.dcc.uchile.cl
! Failed for mirror 17 => https://mirrors.tuna.tsinghua.edu.cn/CRAN
! Failed for mirror 18 => https://mirrors.bfsu.edu.cn/CRAN
! Failed for mirror 19 => https://mirrors.pku.edu.cn/CRAN
✔ Success for mirror 20 => https://mirrors.ustc.edu.cn/CRAN
! Failed for mirror 21 => https://mirrors.zju.edu.cn/CRAN
✔ Success for mirror 22 => https://mirror-hk.koddos.net/CRAN
! Failed for mirror 23 => https://mirrors.e-ducation.cn/CRAN
! Failed for mirror 24 => https://mirrors.qlu.edu.cn/CRAN
! Failed for mirror 25 => https://mirrors.nju.edu.cn/CRAN
! Failed for mirror 26 => https://mirrors.sjtug.sjtu.edu.cn/cran
! Failed for mirror 27 => https://mirrors.sustech.edu.cn/CRAN
! Failed for mirror 28 => https://mirrors.hust.edu.cn/CRAN
! Failed for mirror 29 => https://mirrors.nwafu.edu.cn/cran
! Failed for mirror 30 => https://www.icesi.edu.co/CRAN
✔ Success for mirror 31 => https://mirror.uned.ac.cr/cran
! Failed for mirror 32 => https://mirror.library.ucy.ac.cy/cran
! Failed for mirror 33 => https://mirrors.nic.cz/R
! Failed for mirror 34 => https://mirrors.dotsrc.org/cran
! Failed for mirror 35 => https://cran.asia
! Failed for mirror 36 => https://mirror.cedia.org.ec/CRAN
! Failed for mirror 37 => https://pbil.univ-lyon1.fr/CRAN
✔ Success for mirror 38 => https://mirror.ibcp.fr/pub/CRAN
✔ Success for mirror 39 => https://cran.biotools.fr
! Failed for mirror 40 => https://cran.irsn.fr
✔ Success for mirror 41 => https://mirror.howtolearnalanguage.info
! Failed for mirror 42 => https://ftp.fau.de/cran
! Failed for mirror 43 => https://cran.datenrettung360.de
! Failed for mirror 44 => https://ftp.gwdg.de/pub/misc/cran
! Failed for mirror 45 => https://mirror.dogado.de/cran
✔ Success for mirror 46 => https://cran.uni-muenster.de
! Failed for mirror 47 => https://mirror.clientvps.com/CRAN
! Failed for mirror 48 => https://mirror.kamp.de/cran
! Failed for mirror 49 => https://ftp.cc.uoc.gr/mirrors/CRAN
✔ Success for mirror 50 => https://cran.hafro.is
! Failed for mirror 51 => https://cran.icts.res.in
! Failed for mirror 52 => https://mirror.niser.ac.in/cran
! Failed for mirror 53 => https://cran.usk.ac.id
! Failed for mirror 54 => https://cran.um.ac.ir
! Failed for mirror 55 => https://cran.mirror.garr.it/CRAN
! Failed for mirror 56 => https://cran.stat.unipd.it
✔ Success for mirror 57 => https://ftp.yz.yamagata-u.ac.jp/pub/cran
✔ Success for mirror 58 => https://cran.yu.ac.kr
✔ Success for mirror 59 => https://cran.itam.mx
! Failed for mirror 60 => https://www.est.colpos.mx
! Failed for mirror 61 => https://mirror.marwan.ma/cran
! Failed for mirror 62 => https://mirrors.evoluso.com/CRAN
! Failed for mirror 63 => https://mirror.lyrahosting.com/CRAN
! Failed for mirror 64 => https://cran.stat.auckland.ac.nz
✔ Success for mirror 65 => https://cran.uib.no
✔ Success for mirror 66 => https://cran.mi2.ai
✔ Success for mirror 67 => https://cran.radicaldevelop.com
! Failed for mirror 68 => https://mirror.truenetwork.ru/CRAN
! Failed for mirror 69 => https://cran.mirror.ac.za
! Failed for mirror 70 => https://ftp.cixug.es/CRAN
! Failed for mirror 71 => https://cran.rediris.es
✔ Success for mirror 72 => https://mirror.accum.se/mirror/CRAN
✔ Success for mirror 73 => https://stat.ethz.ch/CRAN
! Failed for mirror 74 => https://mirror.metanet.ch/cran
✔ Success for mirror 75 => https://cran.csie.ntu.edu.tw
! Failed for mirror 76 => http://mirrors.psu.ac.th/pub/cran
✔ Success for mirror 77 => https://cran.pau.edu.tr
! Failed for mirror 78 => https://cran.gedik.edu.tr
✔ Success for mirror 79 => https://www.stats.bris.ac.uk/R
✔ Success for mirror 80 => https://cran.ma.imperial.ac.uk
! Failed for mirror 81 => https://anorien.csc.warwick.ac.uk/CRAN
✔ Success for mirror 82 => https://mirror.las.iastate.edu/CRAN
! Failed for mirror 83 => http://ftp.ussg.iu.edu/CRAN
✔ Success for mirror 84 => https://repo.miserver.it.umich.edu/cran
✔ Success for mirror 85 => https://cran.wustl.edu
! Failed for mirror 86 => https://archive.linux.duke.edu/cran
✔ Success for mirror 87 => https://cran.case.edu
✔ Success for mirror 88 => https://ftp.osuosl.org/pub/cran
! Failed for mirror 89 => https://lib.stat.cmu.edu/R/CRAN
! Failed for mirror 90 => https://cran.mirrors.hoobly.com
! Failed for mirror 91 => https://mirrors.nics.utk.edu/cran
! Failed for mirror 92 => https://mirror.chpc.utah.edu/pub/cran
! Failed for mirror 93 => https://cran.nyuad.nyu.edu
✔ Success for mirror 94 => https://espejito.fder.edu.uy/cran
✔ Success for mirror 95 => https://mirrors.cicku.me/cran

Coverage is now up from ~19% to ~36%.

@louisaslett
Copy link
Author

This second commit address problem 2 in issue #63.

Prior to this fix, we have:

> getCRANmirrors()$URL[35]
[1] "https://cran.asia/"
> chooseCRANmirror(ind = 35)
> foghorn:::url_pkg("mlmc")
[1] "https://cran.asia/package=mlmc"

However, clicking this link reveals the mirror does not redirect such URLs: https://cran.asia/package=mlmc.
Since this is a package which can be updated if the CRAN URL layout changes, I believe it should be considered safe to link direct to the package URL.

After fixing we have:

> getCRANmirrors()$URL[35]
[1] "https://cran.asia/"
> chooseCRANmirror(ind = 35)
> foghorn:::cran_url()
[1] "https://cran.asia/web/packages/mlmc/index.html"

This full package URL is supported by the mirror, try clicking: https://cran.asia/web/packages/mlmc/index.html

This resolves the test in issue #63 for nearly all mirrors:

> for(i in 1:nrow(getCRANmirrors())) {
+   chooseCRANmirror(ind = i)
+   tryCatch({
+     cran_details("mlmc")
+     cran_results("[email protected]")
+     n_cran_flavors()
+     cli::cli_alert_success(paste0("Success for mirror ", i, " => ", getOption("repos")[[1]]))
+   },
+   error = \(e) cli::cli_alert_warning(paste0("Failed for mirror ", i, " => ", getOption("repos")[[1]])))
+ }
✔ Success for mirror 1 => https://cloud.r-project.org
! Failed for mirror 2 => http://mirror.fcaglp.unlp.edu.ar/CRAN
✔ Success for mirror 3 => https://cran.csiro.au
✔ Success for mirror 4 => https://mirror.aarnet.edu.au/pub/CRAN
✔ Success for mirror 5 => https://cran.ms.unimelb.edu.au
✔ Success for mirror 6 => https://cran.wu.ac.at
✔ Success for mirror 7 => https://www.freestatistics.org/cran
✔ Success for mirror 8 => https://ftp.belnet.be/mirror/CRAN
✔ Success for mirror 9 => https://cran-r.c3sl.ufpr.br
✔ Success for mirror 10 => https://vps.fmvz.usp.br/CRAN
✔ Success for mirror 11 => https://brieger.esalq.usp.br/CRAN
✔ Success for mirror 12 => https://ftp.uni-sofia.bg/CRAN
✔ Success for mirror 13 => https://muug.ca/mirror/cran
✔ Success for mirror 14 => https://mirror.csclub.uwaterloo.ca/CRAN
✔ Success for mirror 15 => https://cran.mirror.rafal.ca
✔ Success for mirror 16 => https://cran.dcc.uchile.cl
✔ Success for mirror 17 => https://mirrors.tuna.tsinghua.edu.cn/CRAN
✔ Success for mirror 18 => https://mirrors.bfsu.edu.cn/CRAN
✔ Success for mirror 19 => https://mirrors.pku.edu.cn/CRAN
✔ Success for mirror 20 => https://mirrors.ustc.edu.cn/CRAN
✔ Success for mirror 21 => https://mirrors.zju.edu.cn/CRAN
✔ Success for mirror 22 => https://mirror-hk.koddos.net/CRAN
✔ Success for mirror 23 => https://mirrors.e-ducation.cn/CRAN
✔ Success for mirror 24 => https://mirrors.qlu.edu.cn/CRAN
✔ Success for mirror 25 => https://mirrors.nju.edu.cn/CRAN
✔ Success for mirror 26 => https://mirrors.sjtug.sjtu.edu.cn/cran
✔ Success for mirror 27 => https://mirrors.sustech.edu.cn/CRAN
✔ Success for mirror 28 => https://mirrors.hust.edu.cn/CRAN
✔ Success for mirror 29 => https://mirrors.nwafu.edu.cn/cran
✔ Success for mirror 30 => https://www.icesi.edu.co/CRAN
✔ Success for mirror 31 => https://mirror.uned.ac.cr/cran
✔ Success for mirror 32 => https://mirror.library.ucy.ac.cy/cran
✔ Success for mirror 33 => https://mirrors.nic.cz/R
✔ Success for mirror 34 => https://mirrors.dotsrc.org/cran
✔ Success for mirror 35 => https://cran.asia
✔ Success for mirror 36 => https://mirror.cedia.org.ec/CRAN
✔ Success for mirror 37 => https://pbil.univ-lyon1.fr/CRAN
✔ Success for mirror 38 => https://mirror.ibcp.fr/pub/CRAN
✔ Success for mirror 39 => https://cran.biotools.fr
✔ Success for mirror 40 => https://cran.irsn.fr
✔ Success for mirror 41 => https://mirror.howtolearnalanguage.info
✔ Success for mirror 42 => https://ftp.fau.de/cran
✔ Success for mirror 43 => https://cran.datenrettung360.de
✔ Success for mirror 44 => https://ftp.gwdg.de/pub/misc/cran
✔ Success for mirror 45 => https://mirror.dogado.de/cran
✔ Success for mirror 46 => https://cran.uni-muenster.de
✔ Success for mirror 47 => https://mirror.clientvps.com/CRAN
✔ Success for mirror 48 => https://mirror.kamp.de/cran
✔ Success for mirror 49 => https://ftp.cc.uoc.gr/mirrors/CRAN
✔ Success for mirror 50 => https://cran.hafro.is
✔ Success for mirror 51 => https://cran.icts.res.in
✔ Success for mirror 52 => https://mirror.niser.ac.in/cran
✔ Success for mirror 53 => https://cran.usk.ac.id
✔ Success for mirror 54 => https://cran.um.ac.ir
✔ Success for mirror 55 => https://cran.mirror.garr.it/CRAN
! Failed for mirror 56 => https://cran.stat.unipd.it
✔ Success for mirror 57 => https://ftp.yz.yamagata-u.ac.jp/pub/cran
✔ Success for mirror 58 => https://cran.yu.ac.kr
✔ Success for mirror 59 => https://cran.itam.mx
✔ Success for mirror 60 => https://www.est.colpos.mx
✔ Success for mirror 61 => https://mirror.marwan.ma/cran
✔ Success for mirror 62 => https://mirrors.evoluso.com/CRAN
✔ Success for mirror 63 => https://mirror.lyrahosting.com/CRAN
✔ Success for mirror 64 => https://cran.stat.auckland.ac.nz
✔ Success for mirror 65 => https://cran.uib.no
✔ Success for mirror 66 => https://cran.mi2.ai
✔ Success for mirror 67 => https://cran.radicaldevelop.com
✔ Success for mirror 68 => https://mirror.truenetwork.ru/CRAN
✔ Success for mirror 69 => https://cran.mirror.ac.za
✔ Success for mirror 70 => https://ftp.cixug.es/CRAN
✔ Success for mirror 71 => https://cran.rediris.es
✔ Success for mirror 72 => https://mirror.accum.se/mirror/CRAN
✔ Success for mirror 73 => https://stat.ethz.ch/CRAN
✔ Success for mirror 74 => https://mirror.metanet.ch/cran
✔ Success for mirror 75 => https://cran.csie.ntu.edu.tw
! Failed for mirror 76 => http://mirrors.psu.ac.th/pub/cran
✔ Success for mirror 77 => https://cran.pau.edu.tr
✔ Success for mirror 78 => https://cran.gedik.edu.tr
✔ Success for mirror 79 => https://www.stats.bris.ac.uk/R
✔ Success for mirror 80 => https://cran.ma.imperial.ac.uk
✔ Success for mirror 81 => https://anorien.csc.warwick.ac.uk/CRAN
✔ Success for mirror 82 => https://mirror.las.iastate.edu/CRAN
✔ Success for mirror 83 => http://ftp.ussg.iu.edu/CRAN
✔ Success for mirror 84 => https://repo.miserver.it.umich.edu/cran
✔ Success for mirror 85 => https://cran.wustl.edu
✔ Success for mirror 86 => https://archive.linux.duke.edu/cran
✔ Success for mirror 87 => https://cran.case.edu
✔ Success for mirror 88 => https://ftp.osuosl.org/pub/cran
✔ Success for mirror 89 => https://lib.stat.cmu.edu/R/CRAN
✔ Success for mirror 90 => https://cran.mirrors.hoobly.com
✔ Success for mirror 91 => https://mirrors.nics.utk.edu/cran
✔ Success for mirror 92 => https://mirror.chpc.utah.edu/pub/cran
✔ Success for mirror 93 => https://cran.nyuad.nyu.edu
✔ Success for mirror 94 => https://espejito.fder.edu.uy/cran
✔ Success for mirror 95 => https://mirrors.cicku.me/cran

Coverage is now up from ~19% (pre this PR) to ~97% following both commits.

Investigations into the mirrors still failing revealed:

This means foghorn's forced use of SSL is the only real remaining problem from issue #63 and that only affects 2 mirrors (the third simply having an expired certificate). Therefore, if the maintainers are happy then I think merging this PR would resolve the issue I opened!

Hope that helps,

Louis

@fmichonneau
Copy link
Owner

Thanks for the fixes @louisaslett!

I'm surprised that mirrors don't support the canonical URL format required by CRAN, but your fix makes sense.

It would be great to add a test to make sure we account for these mirrors. Is it something you'd be able to do?

Thanks again!

@louisaslett
Copy link
Author

I'm surprised that mirrors don't support the canonical URL format required by CRAN, but your fix makes sense.

Agreed, I was surprised too. Since writing the issue+PR, I spotted that the canonical URL format is listed under "Optional server configuration" on the CRAN Mirror FAQ, so it seems less than half of mirrors do this optional step sadly.

It would be great to add a test to make sure we account for these mirrors. Is it something you'd be able to do?

I'd be happy to have a go at putting some test together, though I'm not completely sure I understand. Do you mean have the url_pkg() function check if the canonical format is supported and only fall back to the hardcoded package path if it is not? If not, could you elaborate a little more on what you had in mind?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants