Skip to content

0.3.8 - log level, progress bar, refresh OPUS and ELRC; stats

Compare
Choose a tag to compare
@thammegowda thammegowda released this 25 Nov 03:31
· 56 commits to master since this release
v0.3.8
d04438d
  • CLI arg --log-level with default set to WARNING
  • progressbar can be disabled from CLI --no-pbar; default is enabled --pbar
  • mtdata stats --quick does HTTP HEAD and shows content length; e.g. mtdata stats --quick Statmt-commoncrawl-wmt19-fra-deu
  • python -m mtdata.scripts.recipe_stats to read stats from output directory
  • Security fix with tar extract | Thanks @TrellixVulnTeam
  • Added NLLB datasets prepared by AllenAI | Thanks @AlexUmnov
  • Opus and ELRC datasets update | Thanks @ZenBel
  • default for fail_on_error is set to true; returns non zero exit code on error. set --no-fail flag to ignore errors while mtdata get command

Contributors

ZenBel, AlexUmnov, and TrellixVulnTeam
Assets 2
Loading