Code

Karpnv/cc#35

Code

Open

karpnv wants to merge 137 commits intomainNVIDIA/NeMo-speech-data-processor:mainfrom karpnv/ccNVIDIA/NeMo-speech-data-processor:karpnv/ccCopy head branch name to clipboard

+4,645-45

Commits

Commits on Sep 12, 2023

• commoncrawl

  Show description for 40d271e
  karpnv

  committed
  40d271e

  Copy full SHA for 40d271e

• batch

  Show description for c4ea5e4
  karpnv

  committed
  c4ea5e4

  Copy full SHA for c4ea5e4

• rm filter

  Show description for 4ebc195
  karpnv

  committed
  4ebc195

  Copy full SHA for 4ebc195

Commits on Sep 25, 2023

• add caption

  Show description for d9b3473
  karpnv

  committed
  d9b3473

  Copy full SHA for d9b3473

• proxy_fields

  Show description for 9a74b30
  karpnv

  committed
  9a74b30

  Copy full SHA for 9a74b30

• duration_threshold

  Show description for f2c8f2b
  karpnv

  committed
  f2c8f2b

  Copy full SHA for f2c8f2b

• big

  Show description for 199bc22
  karpnv

  committed
  199bc22

  Copy full SHA for 199bc22

• small

  Show description for f450f42
  karpnv

  committed
  f450f42

  Copy full SHA for f450f42

Commits on Sep 29, 2023

• yaml

  Show description for 1952828
  karpnv

  committed
  1952828

  Copy full SHA for 1952828

• FfmpegConvert

  Show description for c9614f8
  karpnv

  committed
  c9614f8

  Copy full SHA for c9614f8

Commits on Oct 9, 2023

• ASR_HF

  Show description for e911070
  karpnv

  committed
  e911070

  Copy full SHA for e911070

• args

  Show description for 1097672
  karpnv

  committed
  1097672

  Copy full SHA for 1097672

Commits on Oct 20, 2023

• duration_key

  Show description for d90cd61
  karpnv

  committed
  d90cd61

  Copy full SHA for d90cd61

• nfa

  Show description for d3973c8
  karpnv

  committed
  d3973c8

  Copy full SHA for d3973c8

• source_audio

  Show description for 6170682
  karpnv

  committed
  6170682

  Copy full SHA for 6170682

Commits on Oct 23, 2023

• dsalign

  Show description for bf5ada0
  karpnv

  committed
  bf5ada0

  Copy full SHA for bf5ada0

Commits on Oct 26, 2023

• audio_duration

  Show description for 075a08a
  karpnv

  committed
  075a08a

  Copy full SHA for 075a08a

• EvalBandwidth and AlignerSubprocess

  Show description for e065608
  karpnv

  committed
  e065608

  Copy full SHA for e065608

Commits on Nov 2, 2023

• split CreateInitialManifestCC

  Show description for dd9f260
  karpnv

  committed
  dd9f260

  Copy full SHA for dd9f260

• split CreateInitialManifestCC

  Show description for 1282ffb
  karpnv

  committed
  1282ffb

  Copy full SHA for 1282ffb

• key_field

  Show description for c1396ad
  karpnv

  committed
  c1396ad

  Copy full SHA for c1396ad

Commits on Nov 8, 2023

• offline_diar_infer

  Show description for fbee380
  karpnv

  committed
  fbee380

  Copy full SHA for fbee380

• arm

  Show description for 7fc4c1e
  karpnv

  committed
  7fc4c1e

  Copy full SHA for 7fc4c1e

Commits on Nov 9, 2023

• duplicates

  Show description for 7661d8e
  karpnv

  committed
  7661d8e

  Copy full SHA for 7661d8e

• drop_text_duplicates

  Show description for 718a812
  karpnv

  committed
  718a812

  Copy full SHA for 718a812

• mcv

  Show description for 8bfdfc9
  karpnv

  committed
  8bfdfc9

  Copy full SHA for 8bfdfc9

Commits on Nov 10, 2023

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  ad51e5d

  Copy full SHA for ad51e5d

• split

  Show description for 6b4a9a6
  karpnv

  committed
  6b4a9a6

  Copy full SHA for 6b4a9a6

• it nl eu

  Show description for 860ed6a
  karpnv

  committed
  860ed6a

  Copy full SHA for 860ed6a

Commits on Nov 13, 2023

• TrainDevTestSplitCC

  Show description for 17953c4
  karpnv

  committed
  17953c4

  Copy full SHA for 17953c4

• merge

  Show description for b6f16c0
  karpnv

  committed
  b6f16c0

  Copy full SHA for b6f16c0

Commits on Nov 15, 2023

• en split

  Show description for b69bfc1
  karpnv

  committed
  b69bfc1

  Copy full SHA for b69bfc1

• Merge branch 'karpnv/cc' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  6942e61

  Copy full SHA for 6942e61

Commits on Nov 16, 2023

• rm pandas

  Show description for 3813285
  karpnv

  committed
  3813285

  Copy full SHA for 3813285

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into armenian

  karpnv

  committed
  5979d5d

  Copy full SHA for 5979d5d

Commits on Nov 17, 2023

• text processing for MCV PR

  Show description for 5d30d6a
  karpnv

  committed
  5d30d6a

  Copy full SHA for 5d30d6a

• path

  Show description for aa21b87
  karpnv

  committed
  aa21b87

  Copy full SHA for aa21b87

Commits on Nov 21, 2023

• RandomPart

  Show description for 9d5e195
  karpnv

  committed
  9d5e195

  Copy full SHA for 9d5e195

Commits on Nov 22, 2023

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  ad10221

  Copy full SHA for ad10221

Commits on Nov 24, 2023

• random_state

  Show description for 5b7700f
  karpnv

  committed
  5b7700f

  Copy full SHA for 5b7700f

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into armenian

  karpnv

  committed
  c724601

  Copy full SHA for c724601

• docstring

  Show description for 2473855
  karpnv

  committed
  2473855

  Copy full SHA for 2473855

Commits on Nov 28, 2023

• split common processors

  Show description for 96dfaed
  karpnv

  committed
  96dfaed

  Copy full SHA for 96dfaed

• langs

  Show description for 424edf7
  karpnv

  committed
  424edf7

  Copy full SHA for 424edf7

• audio_books

  Show description for 0e2ca51
  karpnv

  committed
  0e2ca51

  Copy full SHA for 0e2ca51

• mv

  Show description for 293648b
  karpnv

  committed
  293648b

  Copy full SHA for 293648b

Commits on Dec 1, 2023

• mv todata_to_data

  Show description for 970b9e7
  karpnv

  committed
  970b9e7

  Copy full SHA for 970b9e7

• mv torch

  Show description for f8c5961
  karpnv

  committed
  f8c5961

  Copy full SHA for f8c5961

Commits on Dec 5, 2023

• PR comments

  Show description for a412180
  karpnv

  committed
  a412180

  Copy full SHA for a412180

• paths

  Show description for 2208d03
  karpnv

  committed
  2208d03

  Copy full SHA for 2208d03

• rename

  Show description for 460cbbb
  karpnv

  committed
  460cbbb

  Copy full SHA for 460cbbb

• import

  Show description for f3cebd2
  karpnv

  committed
  f3cebd2

  Copy full SHA for f3cebd2

• docs

  Show description for 9a8d4f2
  karpnv

  committed
  9a8d4f2

  Copy full SHA for 9a8d4f2

Commits on Dec 6, 2023

• subprocess

  Show description for c3ba8c9
  karpnv

  committed
  c3ba8c9

  Copy full SHA for c3ba8c9

Commits on Dec 11, 2023

• Subprocess

  Show description for 21005a2
  karpnv

  committed
  21005a2

  Copy full SHA for 21005a2

• fix docs

  Show description for 9995909
  karpnv

  committed
  9995909

  Copy full SHA for 9995909

• CreateInitialManifestByExt doc

  Show description for 8cfdf39
  karpnv

  committed
  8cfdf39

  Copy full SHA for 8cfdf39

• drop_abs_path

  Show description for af7ca03
  karpnv

  committed
  af7ca03

  Copy full SHA for af7ca03

• Merge branch 'karpnv/cc' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  289a52f

  Copy full SHA for 289a52f

Commits on Dec 12, 2023

• add lang

  Show description for c12b732
  karpnv

  committed
  c12b732

  Copy full SHA for c12b732

Commits on Dec 14, 2023

• deps

  Show description for f310b01
  karpnv

  committed
  f310b01

  Copy full SHA for f310b01

• PreserveByValue

  Show description for 7b7df73
  karpnv

  committed
  7b7df73

  Copy full SHA for 7b7df73

• GetSourceFolder

  Show description for a688b8a
  karpnv

  committed
  a688b8a

  Copy full SHA for a688b8a

• drop Attributes

  Show description for c196b50
  karpnv

  committed
  c196b50

  Copy full SHA for c196b50

• args

  Show description for 4f22ff2
  karpnv

  committed
  4f22ff2

  Copy full SHA for 4f22ff2

• rm methods

  Show description for 9a9831c
  karpnv

  committed
  9a9831c

  Copy full SHA for 9a9831c

• rm Note

  Show description for f869773
  karpnv

  committed
  f869773

  Copy full SHA for f869773

Commits on Dec 15, 2023

• more fixes

  Show description for 4224052
  karpnv

  committed
  4224052

  Copy full SHA for 4224052

Commits on Dec 18, 2023

• header

  Show description for 8289f82
  karpnv

  committed
  8289f82

  Copy full SHA for 8289f82

• ASRWhisper

  Show description for bd42a6c
  karpnv

  committed
  bd42a6c

  Copy full SHA for bd42a6c

Commits on Jan 9, 2024

• AudioLid args

  Show description for b2c1f0d
  karpnv

  committed
  b2c1f0d

  Copy full SHA for b2c1f0d

• merge

  Show description for a4d1173
  karpnv

  committed
  a4d1173

  Copy full SHA for a4d1173

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  8537977

  Copy full SHA for 8537977

Commits on Jan 16, 2024

• GetSpecificFiles CopyFiles

  Show description for 7935e44
  karpnv

  committed
  7935e44

  Copy full SHA for 7935e44

• Merge branch 'karpnv/cc' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  cdf330a

  Copy full SHA for cdf330a

Commits on Jan 18, 2024

• separate dev test

  Show description for 39ec1a3
  karpnv

  committed
  39ec1a3

  Copy full SHA for 39ec1a3

Commits on Jan 22, 2024

• rm

  Show description for 980eeb7
  karpnv

  committed
  980eeb7

  Copy full SHA for 980eeb7

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into armenian

  karpnv

  committed
  f39d1d3

  Copy full SHA for f39d1d3

• black

  Show description for 165c295
  karpnv

  committed
  165c295

  Copy full SHA for 165c295

• self.torch_dtype

  Show description for 8cd5896
  karpnv

  committed
  8cd5896

  Copy full SHA for 8cd5896

• mv to cv

  Show description for 43ff82d
  karpnv

  committed
  43ff82d

  Copy full SHA for 43ff82d

• mv configs

  Show description for a24e6c9
  karpnv

  committed
  a24e6c9

  Copy full SHA for a24e6c9

Commits on Jan 23, 2024

• rename

  Show description for 157de3a
  karpnv

  committed
  157de3a

  Copy full SHA for 157de3a

Commits on Jan 26, 2024

• ManifestToUtf8

  Show description for af69829
  karpnv

  committed
  af69829

  Copy full SHA for af69829

Commits on Feb 8, 2024

• black

  Show description for 4c8a210
  karpnv

  committed
  4c8a210

  Copy full SHA for 4c8a210

• not in

  Show description for b1b45bc
  karpnv

  committed
  b1b45bc

  Copy full SHA for b1b45bc

Commits on Feb 9, 2024

• black

  Show description for fc30b34
  karpnv

  committed
  fc30b34

  Copy full SHA for fc30b34

• add ASRWhisper

  Show description for ee1c52e
  karpnv

  committed
  ee1c52e

  Copy full SHA for ee1c52e

• requirements

  Show description for 3dcc2f7
  karpnv

  committed
  3dcc2f7

  Copy full SHA for 3dcc2f7

• test audio_books.yaml

  Show description for 61c8fe7
  karpnv

  committed
  61c8fe7

  Copy full SHA for 61c8fe7

Commits on Feb 26, 2024

• add docs

  Show description for f7182f2
  karpnv

  committed
  f7182f2

  Copy full SHA for f7182f2

• black

  Show description for 2d5ee5b
  karpnv

  committed
  2d5ee5b

  Copy full SHA for 2d5ee5b

Commits on Feb 27, 2024

• lanID

  Show description for 81716ba
  karpnv

  committed
  81716ba

  Copy full SHA for 81716ba

• srt

  Show description for 33b4f62
  karpnv

  committed
  33b4f62

  Copy full SHA for 33b4f62

• load_manifest

  Show description for e4ebfa7
  karpnv

  committed
  e4ebfa7

  Copy full SHA for e4ebfa7

Commits on Feb 28, 2024

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into armenian

  karpnv

  committed
  dadf773

  Copy full SHA for dadf773

Commits on Feb 29, 2024

• docs

  Show description for ab7e1d9
  karpnv

  committed
  ab7e1d9

  Copy full SHA for ab7e1d9

• black

  Show description for e63f250
  karpnv

  committed
  e63f250

  Copy full SHA for e63f250

Commits on Mar 1, 2024

• key

  Show description for 714a7d1
  karpnv

  committed
  714a7d1

  Copy full SHA for 714a7d1

Commits on Mar 15, 2024

• nemo file

  Show description for fcee183
  karpnv

  committed
  fcee183

  Copy full SHA for fcee183

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  c076199

  Copy full SHA for c076199

• black

  Show description for 464c44b
  karpnv

  committed
  464c44b

  Copy full SHA for 464c44b

• key style

  Show description for 5efdbcd
  karpnv

  committed
  5efdbcd

  Copy full SHA for 5efdbcd

• rm PreserveByValue

  Show description for 981d5bd
  karpnv

  committed
  981d5bd

  Copy full SHA for 981d5bd

• black

  Show description for a1e3fab
  karpnv

  committed
  a1e3fab

  Copy full SHA for a1e3fab

Commits on Mar 19, 2024

• rm operator

  Show description for ab8c685
  karpnv

  committed
  ab8c685

  Copy full SHA for ab8c685

• batch_size > 1

  Show description for f31f7d1
  karpnv

  committed
  f31f7d1

  Copy full SHA for f31f7d1

• Merge branch 'armenian' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  b734db4

  Copy full SHA for b734db4

• German Youtube with new processors (#49)

  Show description for 02b35a8
  ssh-meister

  authored
  02b35a8

  Copy full SHA for 02b35a8

• black

  Show description for f862b2a
  karpnv

  committed
  f862b2a

  Copy full SHA for f862b2a

• merge

  Show description for c1ee056
  karpnv

  committed
  c1ee056

  Copy full SHA for c1ee056

Commits on Mar 20, 2024

• black

  Show description for e2fe178
  karpnv

  committed
  e2fe178

  Copy full SHA for e2fe178

• proxy

  Show description for 8f99da0
  karpnv

  committed
  8f99da0

  Copy full SHA for 8f99da0

Commits on Mar 21, 2024

• New processors for calculating metrics (#50)

  Show description for df15c33
  ssh-meister

  authored
  df15c33

  Copy full SHA for df15c33

Commits on May 9, 2024

• beamsearch

  Show description for 3434b7c
  karpnv

  committed
  3434b7c

  Copy full SHA for 3434b7c

• yaml

  Show description for 082d168
  karpnv

  committed
  082d168

  Copy full SHA for 082d168

Commits on May 14, 2024

• chunk_manifest

  Show description for c6fe2a5
  karpnv

  committed
  c6fe2a5

  Copy full SHA for c6fe2a5

Commits on May 16, 2024

• get_capitalisation_from_target

  Show description for e68d3fe
  karpnv

  committed
  e68d3fe

  Copy full SHA for e68d3fe

Commits on May 17, 2024

• ConcatManifests

  Show description for 0b34b9f
  karpnv

  committed
  0b34b9f

  Copy full SHA for 0b34b9f

Commits on May 21, 2024

• utf8

  Show description for 4aeb88c
  karpnv

  committed
  4aeb88c

  Copy full SHA for 4aeb88c

Commits on May 27, 2024

• shell bool

  Show description for 421bad6
  karpnv

  committed
  421bad6

  Copy full SHA for 421bad6

Commits on May 29, 2024

• LangIdWhisper

  Show description for a3e56d7
  karpnv

  committed
  a3e56d7

  Copy full SHA for a3e56d7

Commits on May 31, 2024

• black

  Show description for 52c8552
  karpnv

  committed
  52c8552

  Copy full SHA for 52c8552

• Merge branch 'karpnv/cc' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  3614723

  Copy full SHA for 3614723

Commits on Jun 1, 2024

• Updated LangIDWhisper processor (#62)

  Show description for dc64941
  

  ssh-meister

  and

  sashameister

  authored
  dc64941

  Copy full SHA for dc64941

Commits on Jun 13, 2024

• resolve conflicts

  Show description for d16ff14
  karpnv

  committed
  d16ff14

  Copy full SHA for d16ff14

Commits on Jun 27, 2024

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  47a6875

  Copy full SHA for 47a6875

Commits on Sep 4, 2024

• kenlm_path fix

  Show description for bb28efc
  karpnv

  committed
  bb28efc

  Copy full SHA for bb28efc

• merge main

  Show description for 7163a88
  karpnv

  committed
  7163a88

  Copy full SHA for 7163a88

Commits on Sep 27, 2024

• add ApplyLlama3 and pnc pipeline

  Show description for c71f558
  karpnv

  committed
  c71f558

  Copy full SHA for c71f558

Commits on Oct 14, 2024

• Merge branch 'main' of github.com:NVIDIA/NeMo-speech-data-processor into karpnv/cc

  karpnv

  committed
  4441a2a

  Copy full SHA for 4441a2a

Commits on Nov 24, 2024

• black

  Show description for 1b8c189
  karpnv

  committed
  1b8c189

  Copy full SHA for 1b8c189

• rm yt'

  Show description for 3810d3b
  karpnv

  committed
  3810d3b

  Copy full SHA for 3810d3b

• rm sdp/processors/datasets/yt

  Show description for f02f37a
  karpnv

  committed
  f02f37a

  Copy full SHA for f02f37a

• whitespace

  Show description for 9d492e9
  karpnv

  committed
  9d492e9

  Copy full SHA for 9d492e9

• rm llm

  Show description for 82f58e0
  karpnv

  committed
  82f58e0

  Copy full SHA for 82f58e0

• rm extra langs

  Show description for e9bb5db
  karpnv

  committed
  e9bb5db

  Copy full SHA for e9bb5db