-
Notifications
You must be signed in to change notification settings - Fork 7
/
Copy pathREADME.html
655 lines (641 loc) · 57.4 KB
/
README.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
<p><img src="https://img.shields.io/docker/automated/tomkellygenetics/universc?label=docker%20local%20build" alt="Docker Manual build" /> <img src="https://img.shields.io/docker/cloud/automated/tomkellygenetics/universc?label=docker%20cloud%20build" alt="Docker Cloud Build" /> <img src="https://img.shields.io/docker/cloud/build/tomkellygenetics/universc?label=cloud%20build" alt="Docker Cloud Status" /> <img src="https://img.shields.io/docker/stars/tomkellygenetics/universc" alt="Docker Stars" /> <img src="https://img.shields.io/docker/pulls/tomkellygenetics/universc" alt="Docker Pulls" /></p>
<p><embed src="https://img.shields.io/docker/v/tomkellygenetics/universc/1.0.3" /> <img src="https://img.shields.io/microbadger/layers/tomkellygenetics/universc/latest?label=%[email protected]%22" alt="MicroBadger Layers (latest)" /> <embed src="https://img.shields.io/docker/image-size/tomkellygenetics/universc/1.0.3?label=%22image%[email protected]" /> <img src="https://img.shields.io/docker/v/tomkellygenetics/universc/latest" alt="Docker Image Version (latest by date)" /> <img src="https://img.shields.io/microbadger/layers/tomkellygenetics/universc/latest" alt="MicroBadger Layers (latest)" /> <img src="https://img.shields.io/docker/image-size/tomkellygenetics/universc/latest" alt="Docker Image Size (latest)" /></p>
<p><img src="https://img.shields.io/github/checks-status/minoda-lab/universc/master?label=GitHub%20checks" alt="GitHub branch checks state" /> <img src="https://img.shields.io/github/release-date/minoda-lab/universc" alt="GitHub Release Date" /> <img src="https://img.shields.io/github/last-commit/minoda-lab/universc/master" alt="GitHub last commit (branch)" /> <img src="https://img.shields.io/github/issues/minoda-lab/universc" alt="GitHub issues" /> <img src="https://img.shields.io/github/issues-pr/minoda-lab/universc" alt="GitHub pull requests" /></p>
<p><a href="http://hits.dwyl.com/minoda-lab/universc"><img src="http://hits.dwyl.com/minoda-lab/universc.svg" alt="GitHub Views" /></a> <a href="http://hits.dwyl.com/tomkellygenetics/universc"><img src="http://hits.dwyl.com/tomkellygenetics/universc.svg" alt="GitHub Views" /></a> <img src="https://img.shields.io/github/search/minoda-lab/universc/master" alt="GitHub search hit counter" /> <img src="https://img.shields.io/github/forks/minoda-lab/universc?style=social" alt="GitHub forks" /> <img src="https://img.shields.io/github/stars/minoda-lab/universc?style=social" alt="GitHub Repo stars" /> <img src="https://img.shields.io/github/watchers/minoda-lab/universc?style=social" alt="GitHub watchers" /></p>
<p><img src="https://img.shields.io/github/languages/code-size/minoda-lab/universc" alt="GitHub code size in bytes" /> <img src="https://img.shields.io/github/repo-size/minoda-lab/universc" alt="GitHub repo size" /> <img src="https://img.shields.io/github/languages/top/minoda-lab/universc" alt="GitHub top language" /> <img src="https://img.shields.io/github/languages/count/minoda-lab/universc" alt="GitHub language count" /></p>
<p><img src="https://img.shields.io/github/downloads/minoda-lab/universc/total?label=GitHub%20downloads" alt="GitHub all releases" /> <img src="https://img.shields.io/github/v/release/minoda-lab/universc?label=GitHub%20release" alt="GitHub release (latest by date)" /> <img src="https://img.shields.io/github/downloads/minoda-lab/universc/1.0.3/total" alt="GitHub release (latest by date)" /> <img src="https://img.shields.io/github/downloads/minoda-lab/universc/1.0.3/total" alt="GitHub release (by tag)" /></p>
<p><img src="https://github.com/minoda-lab/universc/workflows/CI%20to%20Docker%20hub/badge.svg" alt="Docker CI" /> <img src="https://github.com/minoda-lab/universc/workflows/Docker%20compose%20build/badge.svg" alt="Docker compose" /> <img src="https://github.com/minoda-lab/universc/workflows/Docker%20container%20tests/badge.svg" alt="Actions Build" /> <img src="https://github.com/minoda-lab/universc/workflows/Docker%20build%20image/badge.svg" alt="Actions Call" /></p>
<p><img src="https://github.com/minoda-lab/universc/workflows/Run%20all%20tests%20in%20Docker/badge.svg" alt="Actions Tests" /> <img src="https://github.com/minoda-lab/universc/workflows/Test%2010x%20Genomics/badge.svg" alt="Test 10x Genomics" /> <img src="https://github.com/minoda-lab/universc/workflows/Test%20DropSeq%20%2F%20Nadia/badge.svg" alt="Test DropSeq" /> <img src="https://github.com/minoda-lab/universc/workflows/Test%20ICELL8/badge.svg" alt="Test ICELL8" /> <img src="https://github.com/minoda-lab/universc/workflows/Test%20SCI%2DSeq/badge.svg" alt="Test SCI-Seq" /> <img src="https://github.com/minoda-lab/universc/workflows/Test%20inDrops%20v3/badge.svg" alt="Test inDrops v3" /> <img src="https://github.com/minoda-lab/universc/workflows/Test%20Smart%2DSeq3/badge.svg" alt="Test Smart-Seq3" /></p>
<h1 id="universc">UniverSC</h1>
<p><strong>Single-cell processing across technologies</strong></p>
<hr />
<p><strong>Summary</strong></p>
<p>Single-cell RNA-sequencing analysis to quantify RNA molecules in individual cells has become popular owing to the large amount of information one can obtain from each experiment. UniverSC is a universal single-cell processing tool that supports any UMI-based platform. Our command-line tool enables consistent and comprehensive integration, comparison, and evaluation across data generated from a wide range of platforms. Here we provide a guide to install and use this tool to process single-cell RNA-Seq data from FASTQ format.</p>
<p><strong>Package</strong></p>
<p>UniverSC version 1.1.0</p>
<p><strong>Maintainers</strong></p>
<p>Tom Kelly<sup>†</sup> (RIKEN IMS) and Kai Battenberg<sup>†</sup> (RIKEN CSRS/IMS)</p>
<p>† These authors contributed equally to this work</p>
<p>Contact: <first name>.<family name>[at]riken.jp</p>
<hr />
<p><strong>Disclaimer</strong>: we are third party developers not affiliated with 10X Genomics or any other vendor of single-cell technologies. We are releasing this code on an open-source <a href="#licensing">license</a> which calls Cell Ranger™ as an external dependency.</p>
<hr />
<h2 id="getting-started">Getting Started</h2>
<h3 id="advanced-users">Advanced users</h3>
<p>If you have <code>cellranger</code> already installed, then all you need to do is clone or download this git repository. You can then run the script in this directory or add it your <code>PATH</code>. See the <a href="#quick-start">Quick Start</a> guide below.</p>
<p>If you wish to install <code>cellranger</code> and configure this script to run on a Linux environment, we provide details on <a href="#installation">installation</a> below. Note that <code>launch_universc.sh</code> requires write-access a Cell Ranger installation so it needs to be installed in a user's "home" directory on a server. No admin powers needed!</p>
<p>Note that <code>cellranger</code> installations that are pre-compiled on Linux will not run on Mac or Windows. Note that Mac OS and some Linux distributions also have different version of sed and rename. It is possible to compile an open-source version of Cell Ranger but it is tricky to install the dependencies so we recommend using our docker <a href="#Docker">image</a> if you wish to do this.</p>
<h3 id="beginners">Beginners</h3>
<p>If you are a beginner bioinformatician or wish to run this on a local computer (Mac or Windows), no problem! We provide a "docker" image containing everything needed to run it without installing the software needed. All you need to do is install <a href="https://docs.docker.com/desktop/">docker</a> and follow our guide to use the <a href="#Docker">image</a>. This comes bundled with all the compatible versions needed to run it.</p>
<p>Note that you need to run the shell commands given in a unix-like command-line interface (the "Terminal" application on Mac or Linux systems). Many shells are supported but we recommend the "bash" shell for beginners (this is the default on most systems). Windows 10 includes a <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">subsystem</a> to run <code>bash</code>. If this is too complicated, you can open a Linux environment (Ubuntu) in docker by following our instructions. Then you can enter bash commands into the terminal opened by docker.</p>
<p>If you run into problems installing or running <code>launch_universc.sh</code> please don't hesistate to contact us via email or GitHub.</p>
<h2 id="purpose">Purpose</h2>
<p>We've developed a bash script that will run Cell Ranger on FASTQ files for these technologies. See below for details on how to use it.</p>
<p>If you use this tool, please <a href="#Citation">cite</a> to acknowledge the efforts of the authors. You can report problems and request new features to the maintainers with and <a href="#Issues">issue</a> on GitHub. Details on how to <a href="#Install">install</a> and <a href="#Usage">run</a> are provided below. Please see the <a href="#Help">help</a> and <a href="#Examples">examples</a> to try solve your problem before submitting an issue.</p>
<p>Details on the <a href="#Docker">Docker image</a> are given below. We recommend using Docker unless you have a server environment with Cell Ranger installed already.</p>
<h3 id="supported-technologies">Supported Technologies</h3>
<p>In principle, any technology with a cell barcode and unique molecular identifier (UMI) can be supported.</p>
<p>The following technologies have been tested to ensure that they give the expected results: 10x Genomics, Nadia (DropSeq), ICELL8 version 3</p>
<p>We provide the following preset configurations for convenience based on published data and configurations used by other pipelines (e.g, DropSeqPipe and Kallisto/Bustools). To add further support for other technologies or troubleshoot problems, please submit an Issue to the GitHub repository: <a href="https://github.com/TomKellyGenetics/universc/issues">TomKellyGenetics/universc</a> as described in <a href="#Issues">Bug Reports</a> below.</p>
<p>Some changes to the Cell Ranger install are required to run other technologies. Therefore we provide settings for 10x Genomics which restores settings for the Chromium instrument. We therefore recommend using 'convert' for processing all data from different technologies as the tool manages these changes. Please note that on a single install of Cell Ranger, multiple technologies or multiple samples of the same technology with different whitelist barcodes cannot be run cannot be run simultaneousely (the tool will also check for this to avoid causing problems with existing runs). Multiple samples of the same technology with the same barcode whitelist can be run simultaneously.</p>
<p>If you are using <code>UniverSC</code> you should also do so to run 10x Genomics data. If you wish to restore Cell Ranger to default settings, see the <a href="#Uninstalling">installation</a> or <a href="#Debugging">troubleshooting</a> sections below.</p>
<h4 id="pre-set-configurations">Pre-set configurations</h4>
<ul>
<li>10x Genomics (version automatically detected): 10x, chromium
<ul>
<li>10x Genomics version 2 (16 bp barcode, 10 bp UMI): 10x-v2, chromium-v2</li>
<li>10x Genomics version 3 (16 bp barcode, 12 bp UMI): 10x-v3, chromium-v3</li>
</ul></li>
<li>BD Rhapsody (27 bp barcode, 8 bp UMI): bd-rhapsody</li>
<li>CEL-Seq
<ul>
<li>CEL-Seq (8 bp barcode, 4 bp UMI): celseq</li>
<li>CEL-Seq2 (6 bp UMI, 6 bp barcode): celseq2</li>
</ul></li>
<li>Drop-Seq (12 bp barcode, 8 bp UMI): dropseq</li>
<li>ICELL8
<ul>
<li>ICELL8 version 2 (11 bp barcode, No UMI): icell8-non-umi, icell8-v2</li>
<li>ICELL8 version 3 (11 bp barcode, 14 bp UMI): icell8 or custom</li>
</ul></li>
<li>inDrops
<ul>
<li>inDrops version 1 (19 bp barcode, 6 bp UMI): indrops-v1, 1cellbio-v1</li>
<li>inDrops version 2 (19 bp barcode, 6 bp UMI): indrops-v2, 1cellbio-v2</li>
<li>inDrops version 3 (16 bp barcode, 6 bp UMI): indrops-v3, 1cellbio-v3</li>
</ul></li>
<li>MARS-Seq
<ul>
<li>MARS-Seq (6 bp barcode, 10 bp UMI): marsseq, marsseq-v1</li>
<li>MARS-Seq2 (7 bp barcode, 8 bp UMI): marsseq2, marsseq-v2<br />
</li>
</ul></li>
<li>Microwell-Seq (18 bp barcode, 6 bp UMI): microwell</li>
<li>Nadia (12 bp barcode, 8 bp UMI): nadia, dropseq</li>
<li>Quartz-Seq
<ul>
<li>QuartzSeq (6 bp index, no UMI): quartz-seq</li>
<li>Quartz-Seq2 (14 bp barcode, 8 bp UMI): quartzseq2-384</li>
<li>Quartz-Seq2 (15 bp barcode, 8 bp UMI): quartzseq2-1536</li>
</ul></li>
<li>RamDA-Seq (6 bp index, no UMI): ramda-seq</li>
<li>Single-cell combinatorial indexing (SCI-RNA-Seq)</li>
<li>SCI-Seq 2-level indexing (30 bp barcode, 8 bp UMI): sciseq2</li>
<li>SCI-Seq 3-level indexing (40 bp barcode, 8 bp UMI): sciseq3</li>
<li>SCIFI-Seq (27 bp barcode, 8 bp UMI): scifiseq</li>
<li>SCRB-Seq (6 bp barcode, 10 bp UMI): scrbseq, mcscrbseq</li>
<li>SeqWell (12 bp barcode, 8 bp UMI): plexwell, seqwell, seqwells3</li>
<li>SPLiT-Seq (10 bp UMI, 24 bp barcode): splitseq</li>
<li>Smart-seq</li>
<li>Smart-Seq (16 bp barcode, No UMI): smartseq</li>
<li>Smart-Seq2 (16 bp barcode, No UMI): smartseq2</li>
<li>Smart-Seq2-UMI, Smart-seq3 (16 bp barcode, 8 bp UMI): smartseq3</li>
<li>STRT-Seq
<ul>
<li>STRT-Seq (6 bp barcode, no UMI)</li>
<li>STRT-Seq-C1 (8 bp barode, 5 bp UMI)</li>
<li>STRT-Seq-2i (13 bp barcode, 6 bp UMI)</li>
</ul></li>
<li>SureCell (18 bp barcode, 8 bp UMI): surecell, ddseq, biorad</li>
</ul>
<h4 id="chemistry-settings-available">Chemistry settings available</h4>
<p>All technologies support 3′ single-cell RNA-Seq. Barcode adjustments and whitelists are changed automatically. For 5′ single-cell RNA-Seq, this is only supported for 10x Genomics version 2 chemistry, ICELL8, Smart-Seq, and STRT-Seq. For 10x Genomics, this is detected automatically but can be configured with the <code>--chemistry</code> argument. For other technologies, the template switching oligonucleotide is automatically converted to the match the 10x sequence.</p>
<h4 id="support-for-umi-based-and-non-umi-technologies">Support for UMI-based and non-UMI technologies</h4>
<p>By default, UMIs are supported where available so with the following exceptions for non-UMI technologies: ICELL8 v2, RamDA-Seq, Quartz-Seq, Smart-Seq, Smart-Seq2. While using UMI is recommended we provide a mock UMI for counting reads for these technologies (and data from previous versions).</p>
<p>Other techniques can be forced to replace the UMI with a mock sequence for counting reads only with <code>--non-umi</code> or <code>--read-only</code> arguments. Forcing non-UMI techniques is <em>not recommended</em> unless you are integrating non-UMI and UMI-based technologies. It is not necessary to specific <code>--non-umi</code> for non-UMI techniques as these will be used automatically when applicable. For ICELL8 and Smart-Seq where both non-UMI (icell8-v2, smartseq2) and UMI-based (icell8-v3, smartseq3) techniques are available it is possible to specify which to use.</p>
<h4 id="single-and-dual-indexed-technologies">Single and dual indexed technologies</h4>
<p>Where needed the cell barcode can be detected in the index I1 or I2 file. Single indexes are supported for STRT-Seq, Quartz-Seq, and RamDA-Seq. Dual indexes are supported for inDrops-v3, SCI-RNA-Seq, scifi-seq, and Smart-Seq. Combinatorial indexing technologies have linkers between barcodes removed automatically to match the barcode whitelist.</p>
<h4 id="demultiplexing-for-dual-indexing">Demultiplexing for dual-indexing</h4>
<p>For dual-indexed technologies such as inDrops-v3, Sci-Seq, SmartSeq3 it is advised to use "bcl2fastq" before calling UniverSC:</p>
<pre><code> /usr/local/bin/bcl2fastq -v --runfolder-dir "/path/to/illumina/bcls" --output-dir "./Data/Intensities/BaseCalls"\
--sample-sheet "/path/to/SampleSheet.csv" --create-fastq-for-index-reads\
--use-bases-mask Y26n,I8n,I8n,Y50n --mask-short-adapter-reads 0\
--minimum-trimmed-read-length 0</code></pre>
<p>Please adjust the lengths for <code>--use-bases-mask</code> accordingly for read 1, index 1 (i7), index 2 (i5), and read 2. Ensure that <code>--create-fastq-for-index-read</code> is used where possible. If a sequencing facility has demultiplexed the samples for you without this, UniverSC will attempt to extract index sequences from FASTQ headers in read 1. Using <code>--no-lane-splitting</code> is optional as UniverSC can process an arbirtary number of lanes.</p>
<p>There is no need to specify index sequences in the same sheet for cell barcodes, using "NNNNNNNN" will match all samples and the cell barcodes will be distinguished by the single-cell processing pipeline. Index sequences should only be used to demultiplex samples and replicates (not cells).</p>
<h4 id="custom-inputs">Custom inputs</h4>
<p>Custom inputs are also supported by giving the name "custom" and length of barcode and UMI separated by a "_" character.</p>
<p>e.g. Custom (16bp barcode, 10bp UMI): <code>custom_16_10</code></p>
<p>Custom barcode files are also supported for preset technologies. These are particularly useful for well-based technologies to demutliplex based on the wells.</p>
<p>Note that custom inputs do not remove linker or adapter sequences for combinatorial indexng technologies. These must be removed from the Read 1 file before running UniverSC. To request a preset technology setting instead, please submit a feature request on GitHub as described below.</p>
<h2 id="release">Release</h2>
<p>This tool will be released open-source (see <a href="#licensing">legal stuff</a> below). We welcome any feedback on it and any contributions to improve it. Hopefully it will save people time by making it easier to compare technologies.</p>
<p>We have tested it on several technologies but we need users like you to let us know how we can improve it. We hope that it will save you time by handing tedious parts of data formatting so that you can focus on the results.</p>
<h3 id="citation">Citation <span id="Citation"><span></h3>
<p>A submission to a journal and biorXiv is in progress. Please cite these when they are available. Currently, the package can be cited as follows:</p>
<p>Kelly, S.T., Battenberg, Hetherington, N.A., K., Hayashi, K., and Minoda, A. (2021) UniverSC: a flexible cross-platform single-cell data processing pipeline. bioRxiv 2021.01.19.427209; doi: <a href="https://doi.org/10.1101/2021.01.19.427209" class="uri">https://doi.org/10.1101/2021.01.19.427209</a> package version 1.0.3. <a href="https://github.com/minoda-lab/universc" class="uri">https://github.com/minoda-lab/universc</a></p>
<pre><code>@article {Kelly2021.01.19.427209,
author = {Kelly, S. Thomas and Battenberg, Kai and Hetherington, Nicola A. and Hayashi, Makoto and Minoda, Aki},
title = {{UniverSC}: a flexible cross-platform single-cell data processing pipeline},
elocation-id = {2021.01.19.427209},
year = {2021},
doi = {10.1101/2021.01.19.427209},
publisher = {Cold Spring Harbor Laboratory},
abstract = {Single-cell RNA-sequencing analysis to quantify RNA molecules in individual cells has become popular owing to the large amount of information one can obtain from each experiment. We have developed UniverSC (https://github.com/minoda-lab/universc), a universal single-cell processing tool that supports any UMI-based platform. Our command-line tool enables consistent and comprehensive integration, comparison, and evaluation across data generated from a wide range of platforms.Competing Interest StatementThe authors have declared no competing interest.},
eprint = {https://www.biorxiv.org/content/early/2021/01/19/2021.01.19.427209.full.pdf},
journal = {{bioRxiv}},
note = {package version 1.1.0},
URL = {https://github.com/minoda-lab/universc},
}
</code></pre>
<pre><code>@Manual{,
title = {{UniverSC}: a flexible cross-platform single-cell data processing pipeline},
author = {S. Thomas Kelly, Kai Battenberg, Nicola A. Hetherington, Makoto Hayashi, and Aki Minoda},
year = {2021},
note = {package version 1.1.0},
url = {https://github.com/minoda-lab/universc},
}</code></pre>
<h3 id="bug-reports">Bug Reports <span id="Issues"><span></h3>
<h4 id="reporting-issues">Reporting issues</h4>
<p>To add further support for other technologies or troubleshoot problems, please submit an Issue to the GitHub repository: https://github.com/TomKellyGenetics/universc/issues</p>
<p>Please submit <a href="https://github.com/minoda-lab/universc/issues">issues</a> on GitHub to report problems or suggest features. <a href="https://github.com/minoda-lab/universc/pulls">Pull requests</a> to the <code>dev</code> branch on GitHub are also welcome to add features or correct problems. Please see the <a href="CONTRIBUTING.md">contributor guide</a> for more details.</p>
<h3 id="requesting-new-technologies">Requesting new technologies</h3>
<p>Where possible, please provide an minimal example of the first few lines of each FASTQ file for testing purposes.</p>
<p>It is also helpful to describe the technology, such as:</p>
<ul>
<li>length of barcode</li>
<li>length of UMI</li>
<li>which reads they're on</li>
<li>whether there is a known barcode whitelist available</li>
<li>whether adapters or linkers are required</li>
<li>whether a preprint, publication, or company specifications are available</li>
</ul>
<p>Technologies that may be difficult to support are those with:</p>
<ul>
<li>barcodes longer than 16bp</li>
<li>barcodes with phase blocks or varying length</li>
<li>UMIs longer than 12bp</li>
<li>technologies that do not have UMI</li>
<li>combinatorial indexing</li>
<li>dual indexing</li>
</ul>
<p>Please bear this in mind when submitting requests. We will consider to add further technologies but it could take significant resources to add support for techniques with these designs. Note that updates to the tool have added support for several examples of these.</p>
<h2 id="installation">Installation <span id="installation"><span></h2>
<p>This script requires Cell Ranger to be installed and exported to the PATH (version 3.0.0 or higher recommended). The script itself is exectuable and does not require installation to run but you can put it in your PATH or bin of your Cell Ranger install if you wish to do so. We provide scripts to do this for your convenience.</p>
<p>See the details below on how set up Cell Ranger and launch_universc.sh.</p>
<h4 id="download-universc">Download UniverSC</h4>
<p>To download UniverSC open a terminal prompt and enter the following commands.</p>
<pre><code>cd $HOME/Downloads
git clone https://github.com/TomKellyGenetics/universc.git
cd universc</code></pre>
<h3 id="quick-start">Quick Start</h3>
<p>If you already have Cell Ranger installed, then you can run the script without installing it.</p>
<pre><code>bash launch_universc.sh</code></pre>
<p>You can call it in another directory by giving the path to the script.</p>
<pre><code>cd $/HOME/my_project
bash $HOME/Downloads/universc/launch_universc.sh</code></pre>
<p>See the details below on how to install Cell Ranger and launch_universc.sh add them to the PATH so that <code>launch_universc.sh</code> can be run from any directory.</p>
<h3 id="runnning-in-a-git-repository">Runnning in a git repository</h3>
<p>If you are running code in a git repository you can add UniverSC as a submodule.</p>
<pre><code>cd $/HOME/my_git_repo
git submodule add https://github.com/TomKellyGenetics/universc.git
bash universc/launch_universc.sh</code></pre>
<h3 id="system-requirements">System Requirements</h3>
<p>In principle, the script can run on any Unix systems with Cell Ranger installed. You can check whether Cell Ranger is already availble by running:</p>
<pre><code>whereis cellranger</code></pre>
<p>You can see which Cell Ranger installation will run as follows:</p>
<pre><code>which cellranger
cellranger count --version</code></pre>
<p>If Cell Ranger is already installed on your system, you can add it to your $PATH as follows:</p>
<pre><code>export PATH=/home/username/path/to/cellranger-x.x.x:$PATH </code></pre>
<h4 id="installing-dependencies">Installing dependencies</h4>
<p>If Cell Ranger is not installed on your system, you must install it before running launch_universc.sh.</p>
<p>Please see the <a href="https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger">manual for Cell Ranger</a> on the 10x Genomics website for more details on how to use it. We provide support for passing various options to Cell Ranger and sensible defaults for each technology.</p>
<p>This script is compatible with the installation of Cell Ranger that you can <a href="https://support.10xgenomics.com/single-cell-gene-expression/software/downloads/latest">download</a> from the 10x Genomics website and gives the same output formats.</p>
<p>However, we recommend to use the open-source release of Cell Ranger on GitHub. This is release on an MIT License and is not subject to the 10x Genomics End User License Agreement.</p>
<p>We provide open-source repositories with minor updates for compatibility with current versions of dependencies.</p>
<p>The code is available here:</p>
<p><a href="https://github.com/TomKellyGenetics/cellranger/releases" class="uri">https://github.com/TomKellyGenetics/cellranger/releases</a></p>
<p>We also provide Docker images for Cell Ranger versions 2.0.2, 2.1.0, 2.1.1, and 3.0.2:</p>
<p><a href="https://github.com/TomKellyGenetics/cellranger_clean/packages" class="uri">https://github.com/TomKellyGenetics/cellranger_clean/packages</a></p>
<p><a href="https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags" class="uri">https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags</a></p>
<h4 id="cluster-mode-configuration">Cluster Mode configuration</h4>
<h4 id="software-requirements">Software Requirements</h4>
<p>These have been pre-installed in the Docker image described above.</p>
<p>A full example of installation is available in the <a href="https://github.com/TomKellyGenetics/cellranger">GitHub repository</a> and on <a href="https://hub.docker.com/r/tomkellygenetics/cellranger_clean/dockerfile">DockerHub</a>.</p>
<ul>
<li>Python 2.7.13</li>
<li>rust 1.28.0</li>
<li>clang 6.0</li>
<li>go 1.11</li>
<li>node 8.11.4</li>
<li>Cython 0.28.0</li>
<li>STAR 2.5.1b</li>
<li>bcl2fastq 2.19.1.403</li>
<li>tsne 0.15</li>
</ul>
<p>The following additional shell utilities are required. Mac OS and most Linux distributions come with these pre-installed.</p>
<ul>
<li>make 3.81</li>
<li>git 2.20.1</li>
<li>sed (GNU sed) 4.4 (gsed)</li>
<li>tar 2.8.3</li>
<li>rename 0.20 (perl-rename)</li>
<li>perl 5.26.1</li>
<li>rsync 2.6.9</li>
</ul>
<p>Note that rename is installed by default on Mac, Ubuntu and Debian but a different version must be used on other Linux distrubutions.</p>
<p>CentOS and Fedora:</p>
<pre><code>sudo yum install prename</code></pre>
<pre><code>sudo dnf install prename</code></pre>
<p>Red Hat Linux:</p>
<pre><code>sudo rpm install prename</code></pre>
<p>Arch Linux:</p>
<pre><code>yay perl-rename</code></pre>
<h5 id="recommended-software">Recommended software</h5>
<ul>
<li>git-lfs 2.10.0</li>
</ul>
<h4 id="hardware-requirements">Hardware requirements</h4>
<ul>
<li>8-core Intel or AMD processor (16 cores recommended)</li>
<li>64GB RAM (128GB recommended)</li>
<li>1TB free disk space</li>
<li>64-bit CentOS/RedHat 6.0 or Ubuntu 12.04</li>
</ul>
<h4 id="ensuring-write-access-to-cell-ranger">Ensuring write-access to Cell Ranger</h4>
<p>The conversion process requires write-access to to the Cell Ranger install directory so an install on your user directory is recommended.</p>
<p>You can check where Cell Ranger is installed with:</p>
<pre><code>which cellranger</code></pre>
<p>If calling the script gives the help menu, launch_universc.sh has sucessfully run with access to the directories that it needs. It will give an error message if the Cell Ranger directory is not writeable.</p>
<pre><code>bash launch_universc.sh</code></pre>
<p>This script requires Cell Ranger (version 3.0.0 or higher recommended) to be installed and have write-access to the Cell Ranger install directory, so an install on your user directory is recommended. This script also requires Cell Ranger to be exported to the PATH. The script itself is exectuable and does not require installation to run but you can put it in your PATH or bin of your Cell Ranger install if you wish to do so.</p>
<p>This script will run in bash on any OS (but it has only been tested on Linux Debian). Running Cell Ranger with this configuration requires a lot of memory (40Gb) so running on server is recommended. SGE job modes are supported to run Cell Ranger with multiple threads.</p>
<p>This is required because launch_universc.sh will make changes to the Cell Ranger install to ensure compatibility with the technology running. A local install in you user home directory is needed to make these changes. This ensures that these changes do not affect jobs run by other users and allows launch_universc.sh to change the whitelist and source code as needed.</p>
<p>These changes are reversible but mean that only one technology can be run at the same time. You can restore original configurations with:</p>
<pre><code>bash launch_universc.sh -t "10x" --setup</code></pre>
<h5 id="local-install">Local install</h5>
<p>If Cell Ranger is not already installed we recommend installing it in a directory that you have write access to such as <code>$HOME/local</code>.</p>
<h5 id="importing-an-installed-version-of-cell-ranger">Importing an installed version of Cell Ranger</h5>
<p>If Cell Ranger has been installed by a system administrator, you will only have read-access to that installation. You can still use rather than installing a new version but you will need to copy it to your home directory and add this version to your PATH.</p>
<pre><code>mkdir -p $HOME/local
cd ~/local
installed_version=$(echo $(which cellranger) | rev | cut -d"/" -f2- | rev)
cp -rv $installed_version .
installed_directory=$(echo $(which cellranger) | rev | cut -d"/" -f2 | rev)
cd $installed_directory
new_version=$(pwd)
#remove previous version from PATH
export PATH=$(echo $PATH | sed "s;$installed_version:;;g")
#add new version to PATH
eval $(echo export PATH=$new_version:\$PATH)
cd ..</code></pre>
<p>You should be able to see that the locally installed version can be called as follows:</p>
<pre><code>echo $PATH
which cellranger
bash launch_universc.sh</code></pre>
<h4 id="installing-launch_universc.sh-to-the-path">Installing launch_universc.sh to the PATH</h4>
<h5 id="running-the-script">Running the script</h5>
<p>Adding the script to the PATH is not absolutely neccessary, it can be called as follows from the directory that it is downloaded in.</p>
<pre><code>cd $HOME/Downloads
git clone https://github.com/TomKellyGenetics/universc.git
cd universc
bash launch_universc.sh</code></pre>
<h5 id="automated-configuration">Automated configuration</h5>
<p>We provide a Makefile with all necessary configurations to automatically check whether launch_universc.sh is installed correctly.</p>
<p>You can specify any directory to install as a "prefix". This will create a directory "$HOME/local/universc-0.3" where the files needed will be stored.</p>
<pre><code>cd $HOME/Downloads
git clone https://github.com/TomKellyGenetics/universc.git
make
make install prefix=$HOME/local</code></pre>
<p>It is also possible to add the current working directory as the installed directory.</p>
<pre><code>make install prefix="."</code></pre>
<p>In this case <em>do not</em> delete the installed directory after you install it or the script will fail to run.</p>
<p>By default it will be installed in a root directory with read-only access. This requires administrator priviledges. Note that the manual can onlly be installed with root priviledges.</p>
<pre><code>sudo make install
sudo make manual</code></pre>
<p>You can verify that launch_universc.sh has been added to the PATH.</p>
<pre><code>echo $PATH
which launch_universc.sh</code></pre>
<p>You can then run <code>launch_universc.sh</code> from any working directory.</p>
<h5 id="updating">Updating</h5>
<p>If launch_universc.sh is already installed and you wish to update it, first you need to pull the changes from GitHub from the universc directory.</p>
<pre><code>cd universc
git pull origin master
make</code></pre>
<p>Then you can update to the directory of your choice using the same options for <code>--prefix</code> as to install.</p>
<pre><code>make install prefix=$HOME/local</code></pre>
<p>This is remove previous versions of launch_universc.sh and install the latest version.</p>
<p>The manual can be updated with:</p>
<pre><code>sudo make manual</code></pre>
<h5 id="uninstalling">Uninstalling</h5>
<p>Before uninstalling UniverSC please ensure that any versions of Cell Ranger used are restored to their default configuration:</p>
<pre><code>export PATH=/Users/tom/Downloads/cellranger-x.y.z:$PATH
bash launch_universc.sh -t "10x" --setup</code></pre>
<p>We provide an automated script to reverse the changes above.</p>
<pre><code>make uninstall</code></pre>
<p>This is will automatically detect the installation of launch_universc.sh.</p>
<p>If multiple versions of Cell Ranger are present, you can specify which to remove with.</p>
<pre><code>make remove prefix=$HOME/local</code></pre>
<h4 id="custom-shell">Custom shell</h4>
<p>Make will install the script with <code>bash</code> by default but alternative shells are supported. You will need to run the install script or run it with your shell of choice.</p>
<pre><code>sh launch_universc.sh
sh inst/INSTALL --prefix $HOME/local
launch_universc.sh</code></pre>
<pre><code>ksh launch_universc.sh
ksh inst/INSTALL --prefix $HOME/local
launch_universc.sh</code></pre>
<pre><code>zsh launch_universc.sh
zsh inst/INSTALL --prefix $HOME/local
launch_universc.sh</code></pre>
<pre><code>fish launch_universc.sh
fish inst/INSTALL --prefix $HOME/local
launch_universc.sh</code></pre>
<p>The help menu should reflect the shell used to run it.</p>
<p>To update, similarly run the <code>inst/UPGRADE</code> script with your chosen shell.</p>
<h3 id="docker-image">Docker image <span id="Docker"><span></h3>
<p>We provide a docker image with all software needed to run UniverSC.</p>
<p>This requires "docker" to be installed and a valid DockerHub account.</p>
<p>You can check whether docker is available by running:</p>
<pre><code>which docker
docker run hello-world </code></pre>
<p>This may require you to login to your account.</p>
<pre><code>docker login -u "myusername"</code></pre>
<p>If you cannot run docker on a remote server, contact your systems administrator.</p>
<h4 id="pulling-from-remote-dockerhub-repository">Pulling from remote DockerHub repository</h4>
<p>We provide a docker image for UniverSC version 1.1.0.</p>
<p>You can import it if you have docker installed.</p>
<pre><code>docker pull tomkellygenetics/universc:latest</code></pre>
<p>Then you can run convert with:</p>
<pre><code>run -it tomkellygenetics/universc:latest launch_universc.sh</code></pre>
<p>You can open a shell in the docker image with:</p>
<pre><code>run -it tomkellygenetics/universc:latest /bin/bash</code></pre>
<pre><code>run -it tomkellygenetics/universc:latest /bin/zsh </code></pre>
<p>Either of these shells are supported.</p>
<h5 id="building-the-docker-image-locally">Building the Docker image locally</h5>
<p>The Dockerfile is provided in the repository so it can be built from source. This will build a Docker image with the latest version of universc provided that updates to dependencies on GitHub are still compatible.</p>
<pre><code>git clone https://github.com/TomKellyGenetics/universc.git
docker build -t universc:latest . </code></pre>
<p>Please bear mind that it can take considerable time to install all necessary dependencies. A stable internet connection is required.</p>
<h3 id="manual-configuration">Manual configuration</h3>
<p>You can manually add the script here to the PATH, for example:</p>
<pre><code>PATH=$HOME/Downloads/universc:$PATH</code></pre>
<p>This means that the directory where the script is can be found from the shell.</p>
<pre><code>echo $PATH
cd ~
launch_universc.sh</code></pre>
<p>Add the following line to the <code>~/.bashrc</code> file and use <code>source ~/.bashrc</code> to load a new session. This means that you do not need to add the sript to the PATH in future sessions.</p>
<pre><code>export PATH=$HOME/Downloads/universc:$PATH</code></pre>
<h3 id="setting-up-cell-ranger-references">Setting up Cell Ranger references</h3>
<p>This repository comes with almost all necessary files to run test jobs. Test data and Cell Ranger references are available with Git large file storage (LFS).</p>
<p>To install git LFS run:</p>
<pre><code>curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install</code></pre>
<p>To import large files from Github change to the "universc" directory and run:</p>
<pre><code>git lfs pull origin</code></pre>
<p>This provides almost all files required. The STAR index and reference need to be generated or imported from an existing reference. The following code detects whether the references are available in an existing cellranger installation.</p>
<pre><code>cellrangerversion=`cellranger count --version | head -n 2 | tail -n 1 | cut -f2 -d'(' | cut -f1 -d')'`
cellrangerpath=`which cellranger`
# set up cellranger reference
if [[ ! -f test/cellranger_reference/cellranger-tiny-ref/3.0.0/star/SA ]] && [[ -f $(dirname $cellrangerpath)/cellranger-tiny-ref/3.0.0/star/SA ]]; then
rsync $(dirname $cellrangerpath)/cellranger-tiny-ref/3.0.0/star/SA test/cellranger_reference/cellranger-tiny-ref/3.0.0/star/SA
fi
if [[ ! -f test/cellranger_reference/cellranger-tiny-ref/1.2.0/star/SA ]] && [[ -f $(dirname $cellrangerpath)/cellranger-tiny-ref/1.2.0/star/SA ]]; then
rsync $(dirname $cellrangerpath)/cellranger-tiny-ref/1.2.0/star/SA test/cellranger_reference/cellranger-tiny-ref/1.2.0/star/SA
fi</code></pre>
<p>This creates a reference for Cell Ranger here:</p>
<ul>
<li><p>test/cellranger_reference/cellranger-tiny-ref/1.2.0</p></li>
<li><p>test/cellranger_reference/cellranger-tiny-ref/3.0.0</p></li>
</ul>
<h4 id="automated-references">Automated references</h4>
<p>You can reset the references with the automated settings here:</p>
<pre><code>cd test/cellranger_reference/cellranger-tiny-ref/
make clean
make reference
cd ../../..</code></pre>
<h4 id="custom-cell-ranger-references">Custom Cell Ranger references</h4>
<p>It is also possible to generate a custom reference for any genome provided you have a FASTQ genome reference file and a GTF/GFF3 annotation file. Please ensure that the chromosomes match between the FASTA headers and the chromosome column (1st) of the GTF/GFF3 file.</p>
<p>The <code>gffread</code> function includes with the <a href="http://cole-trapnell-lab.github.io/cufflinks/file_formats/#the-gffread-utility">cufflinks</a> utility can convert to gtf. For example:</p>
<pre><code>gffread test/cellranger_reference/cellranger-tiny-ref/genes-1.2.0.gff3 -T -o test/cellranger_reference/cellranger-tiny-ref/genes-1.2.0.gtf </code></pre>
<p>To generate new references we first remove the references imported.</p>
<pre><code>rm -rf test/cellranger_reference/cellranger-tiny-ref/1.2.0 test/cellranger_reference/cellranger-tiny-ref/3.0.0</code></pre>
<p>We then generate references from the FASTA and GTF files as shown in the following examples:</p>
<pre><code>cellranger mkref --genome=test/cellranger_reference/cellranger-tiny-ref/1.2.0 \
--fasta=test/cellranger_reference/cellranger-tiny-ref/genome-1.2.0.fa \
--genes=test/cellranger_reference/cellranger-tiny-ref/ genes-1.2.0.gtf
cellranger mkref --genome=test/cellranger_reference/cellranger-tiny-ref/3.0.0 \
--fasta=test/cellranger_reference/cellranger-tiny-ref/genome-3.0.0.fa \
--genes=test/cellranger_reference/cellranger-tiny-ref/ genes-3.0.0.gtf</code></pre>
<p>See the Cell Ranger manuals for more <a href="https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/advanced/references">details on references</a>.</p>
<h2 id="usage">Usage <span id="Usage"><span></h2>
<p>The script will:</p>
<ul>
<li>give you a help guide</li>
</ul>
<p><code>bash launch_universc.sh -h</code></p>
<ul>
<li><p>convert R1 files so that barcodes and UMIs are where they're expected to be for 10x (this can take some time for larger files)</p></li>
<li><p>runs Cell Ranger with the same parameters as for 10x and treats samples exactly the same</p></li>
<li><p>the barcode whitelists are changed and some checks on barcodes disabled (requires a writeable install of Cell Ranger in your user directory)</p></li>
<li><p>it can run Cell Ranger in parallel in SGE mode on the server if you use <code>--jobmode "sge"</code> and set up an <code>sge.template</code> file</p></li>
<li><p>it can also restore the original Cell Ranger settings for running 10x samples</p></li>
</ul>
<p><code>bash launch_universc.sh --setup --technology "10x"</code></p>
<h3 id="valid-barcodes">Valid barcodes</h3>
<p>Please note that this script alters the barcode whitelist. Known ICELL8 barcodes are supported but this is not possible with Nadia or DropSeq chemistry so 100% valid barcodes will be returned.</p>
<h3 id="manual">Manual <span id="Help"><span></h3>
<h4 id="locally-install-manual">Locally install manual</h4>
<p>You can display a manual from the locally installed UniverSC directory with:</p>
<pre><code> man man/launch_universc.sh </code></pre>
<p>Note that the working directory must be <code>universc</code> or the full path to the man directory must be given.</p>
<h4 id="installing-the-manual-with-root-priviliges">Installing the manual with root priviliges:</h4>
<h5 id="automated-configuration-1">Automated Configuration</h5>
<p>We provide an automated script to install the manual.</p>
<pre><code>sudo make manual</code></pre>
<p>You can remove the manual with:</p>
<pre><code>sudo make manual-clean</code></pre>
<h5 id="configuration">Configuration</h5>
<p>The manual can be installed as follows on Mac and Linux.</p>
<pre><code># add manual directory to PATH if not already found
## check config for Linux
if [[ -f /etc/manpath.config ]]
then CONFIG="/etc/manpath.config"
fi
## check config for Mac
if [[ -f /etc/manpaths ]]
then CONFIG="/etc/manpaths"
fi
if [[ ! -z $CONFIG ]]
then MANDIR=`tail -n 1 ${CONFIG}`
else if [[ ! -z $MANPATH ]]
then
SHELL_RC=`echo ~/.${0}rc`
echo "export MANPATH=/usr/local/man" >> $SHELL_RC
MANDIR=`echo ${MANPATH} | cut -d: -f1`
fi
fi
sudo mkdir -p ${MANDIR}/man1
cp man/launch_universc.sh man/launch_universc.sh.1
sudo mv man/launch_universc.sh.1 ${MANDIR}/man1/launch_universc.sh.1
gzip ${MANDIR}/man1/launch_universc.sh.1</code></pre>
<p>Alternatively the man can be installed with:</p>
<pre><code>cp man/launch_universc.sh man/launch_universc.sh.1
sudo install -g 0 -o 0 -m 0644 man/launch_universc.sh.1 ${MANDIR}/man1</code></pre>
<p>The manual can then be called from any directory as follows:</p>
<pre><code>man launch_universc.sh</code></pre>
<h4 id="help-menu">Help menu</h4>
<p>You can access the following help menu with <code>launch_universc.sh --help</code> in the terminal.</p>
<pre><code>Usage:
bash launch_universc.sh --testrun -t THECHNOLOGY
bash launch_universc.sh -t TECHNOLOGY --setup
bash launch_universc.sh -R1 FILE1 -R2 FILE2 -t TECHNOLOGY -i ID -r REFERENCE [--option OPT]
bash launch_universc.sh -R1 READ1_LANE1 READ1_LANE2 -R2 READ2_LANE1 READ2_LANE2 -t TECHNOLOGY -i ID -r REFERENCE [--option OPT]
bash launch_universc.sh -f SAMPLE_LANE -t TECHNOLOGY -i ID -r REFERENCE [--option OPT]
bash launch_universc.sh -f SAMPLE_LANE1 SAMPLE_LANE2 -t TECHNOLOGY -i ID -r REFERENCE [--option OPT]
bash launch_universc.sh -v
bash launch_universc.sh -h
Convert sequencing data (FASTQ) from Nadia or ICELL8 platforms for compatibility with 10x Genomics and run cellranger count
Mandatory arguments to long options are mandatory for short options too.
--testrun Initiates a test trun with the test dataset
-R1, --read1 FILE Read 1 FASTQ file to pass to Cell Ranger (cell barcodes and umi)
-R2, --read2 FILE Read 2 FASTQ file to pass to Cell Ranger
-I1, --index1 FILE Index (I1) FASTQ file to pass to Cell Ranger (OPTIONAL)
-I2, --index2 FILE Index (I2) FASTQ file to pass to Cell Ranger (OPTIONAL and EXPERIMENTAL)
-f, --file NAME Path and the name of FASTQ files to pass to Cell Ranger (prefix before R1 or R2)
e.g. /path/to/files/Example_S1_L001
-i, --id ID A unique run id, used to name output folder
-d, --description TEXT Sample description to embed in output files.
-r, --reference DIR Path of directory containing 10x-compatible reference.
-t, --technology PLATFORM Name of technology used to generate data.
Supported technologies:
10x Genomics (version automatically detected): 10x, chromium
10x Genomics version 2 (16 bp barcode, 10 bp UMI): 10x-v2, chromium-v2
10x Genomics version 3 (16 bp barcode, 12 bp UMI): 10x-v3, chromium-v3
BD Rhapsody (27 bp barcode, 8 bp UMI): bd-rhapsody
CEL-Seq (8 bp barcode, 4 bp UMI): celseq
CEL-Seq2 (6 bp UMI, 6 bp barcode): celseq2
Drop-Seq (12 bp barcode, 8 bp UMI): dropseq
ICELL8 version 2 (11 bp barcode, No UMI): icell8-non-umi, icell8-v2
ICELL8 version 3 (11 bp barcode, 14 bp UMI): icell8 or custom
inDrops version 1 (19 bp barcode, 6 bp UMI): indrops-v1, 1cellbio-v1
inDrops version 2 (19 bp barcode, 6 bp UMI): indrops-v2, 1cellbio-v2
inDrops version 3 (16 bp barcode, 6 bp UMI): indrops-v3, 1cellbio-v3
Nadia (12 bp barcode, 8 bp UMI): nadia, dropseq
MARS-Seq (6 bp barcode, 10 bp UMI): marsseq, marsseq-v1
MARS-Seq2 (7 bp barcode, 8 bp UMI): marsseq2, marsseq-v2
Microwell-Seq (18 bp barcode, 6 bp UMI): microwell
QuartzSeq (6 bp index, no UMI): quartz-seq
Quartz-Seq2 (14 bp barcode, 8 bp UMI): quartzseq2-384
Quartz-Seq2 (15 bp barcode, 8 bp UMI): quartzseq2-1536
RamDA-Seq (6 bp index, no UMI): ramda-seq
SCI-Seq 2-level indexing (30 bp barcode, 8 bp UMI): sciseq2
SCI-Seq 3-level indexing (40 bp barcode, 8 bp UMI): sciseq3
SCIFI-Seq (27 bp barcode, 8 bp UMI
SCRB-Seq (6 bp barcode, 10 bp UMI): scrbseq, mcscrbseq
SeqWell (12 bp barcode, 8 bp UMI): plexwell, seqwell, seqwells3
Smart-seq (16 bp barcode, No UMI): smartseq
Smart-seq2 (16 bp barcode, No UMI): smartseq2
Smart-seq2-UMI, Smart-seq3 (16 bp barcode, 8 bp UMI): smartseq3
SPLiT-Seq (10 bp UMI, 24 bp barcode): splitseq
STRT-Seq (6 bp barcode, no UMI)
STRT-Seq-C1 (8 bp barode, 5 bp UMI)
STRT-Seq-2i (13 bp barcode, 6 bp UMI)
SureCell (18 bp barcode, 8 bp UMI): surecell, ddseq, biorad
Custom inputs are also supported by giving the name "custom" and length of barcode and UMI separated by "_"
e.g. Custom (16 bp barcode, 10 bp UMI): custom_16_10
-b, --barcodefile FILE Custom barcode list in plain text (with each line containing a barcode)
-c, --chemistry CHEM Assay configuration, autodetection is not possible for converted files: SC3Pv2 (default), SC5P-PE, or SC5P-R2
5′ scRNA-Seq ('SC5P-PE') is available only for 10x Genomics, ICELL8, SmartSeq, and STRT-Seq technologies
All other technologies default to 3′ scRNA-Seq parameters. Only 10x Genomics and ICELL8 allow choosing which to use.
-n, --force-cells NUM Force pipeline to use this number of cells, bypassing the cell detection algorithm.
-j, --jobmode MODE Job manager to use. Valid options: local (default), sge, lsf, or a .template file
--localcores NUM Set max cores the pipeline may request at one time.
Only applies when --jobmode=local.
--localmem NUM Set max GB the pipeline may request at one time.
Only applies when --jobmode=local.
--mempercore NUM Set max GB each job may use at one time.
Only applies in cluster jobmodes.
-p, --per-cell-data Generates a file with basic run statistics along with per-cell data
--non-umi or --read-only Force counting reads by adding or replacing UMI with a mock sequence
--setup Set up whitelists for compatibility with new technology and exit
--as-is Skips the FASTQ file conversion if the file already exists
-h, --help Display this help and exit
-v, --version Output version information and exit
--verbose Print additional outputs for debugging
For each fastq file, follow the naming convention below:
<SampleName>_<SampleNumber>_<LaneNumber>_<ReadNumber>_001.fastq
e.g. EXAMPLE_S1_L001_R1_001.fastq
Example_S4_L002_R2_001.fastq.gz
For custom barcode and umi length, follow the format below:
custom_<barcode>_<UMI>
e.g. custom_16_10 (which is the same as 10x)
Files will be renamed if they do not follow this format. File extension will be detected automatically.
</code></pre>
<h3 id="examples">Examples <span id="Examples"><span></h3>
<h4 id="running-cell-ranger">Running Cell Ranger</h4>
<pre><code>cellranger testrun --id="tiny-test"</code></pre>
<pre><code># open gzip files from test data
gunzip -fk universc/test/shared/cellranger-tiny-fastq/3.0.0/*fastq.gz
gunzip -fk cellranger-3.0.2.9001/cellranger-cs/3.0.2.9001/lib/python/cellranger/barcodes/3M-february-2018.txt.gz
# Cell Ranger call
cellranger count --id="tiny-count-v3" \
--fastqs="cellranger-3.0.2.9001/cellranger-tiny-fastq/3.0.0/" --sample="tinygex" \
--transcriptome="cellranger-3.0.2.9001/cellranger-tiny-ref/3.0.0"</code></pre>
<h4 id="running-launch_universc.sh-on-10x-data">Running launch_universc.sh on 10x data</h4>
<pre><code># call convert on 10x with multiple lanes
bash /universc/launch_universc.sh --id "test-10x-v3" --technology "10x" \
--reference "/universc/test/cellranger_reference/cellranger-tiny-ref/3.0.0" \
--file "/universc/test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L001" \
"/universc/test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L002"</code></pre>
<h4 id="running-launch_universc.sh-on-dropseq-data">Running launch_universc.sh on DropSeq data</h4>
<p>Obtain DropSeq data from public database:</p>
<pre><code>wget https://www.ncbi.nlm.nih.gov/geo/download/\?acc\=GSM1629192\&format\=file\&file\=GSM1629192%5FPure%5FHumanMouse%2Ebam
mv index.html\?acc=GSM1629192\&format=file\&file=GSM1629192%5FPure%5FHumanMouse%2Ebam GSM162919.bam
samtools sort -n GSM162919.bam > GSM162919.qsort
samtools view GSM162919.qsort HUMAN_21:9825832-48085036 > GSM162919.qsort2
samtools sort -O BAM GSM162919.bam > GSM162919.sort.bam
samtools index GSM162919.sort.bam
samtools view GSM162919.sort.bam HUMAN_21:9825832-48085036 > GSM162919.chr21.bam
samtools view -O BAM GSM162919.sort.bam HUMAN_21:9825832-48085036 > GSM162919.chr21.sort.bam
samtools sort -n GSM162919.chr21.sort.bam -o GSM162919.chr21.qsort.bam
bedtools bamtofastq -i GSM162919.chr21.qsort.bam -fq GSM1629192_chr21_R1.fastq
mv GSM1629192_chr21_R1.fastq GSM1629192_chr21_R2.fastq
fastq-dump -F --split-files SRR1873277
fastq_pair GSM1629192_chr21_R2.fastq SRR1873277_1.fastq
head -n 117060 SRR1873277_1.fastq.paired.fq 117060 > SRR1873277_1.fastq.paired.fq
head -n 117060 GSM1629192_chr21_R2.fastq.paired.fq > GSM1629192_chr21_R2.fastq.paired.fq
cp SRR1873277_1.fastq.paired.fq GSM1629192_chr21_R2.fastq.paired.fq ~/repos/universc/test/shared/dropseq-test
cp SRR1873277_1.fastq.paired.fq GSM1629192_chr21_R2.fastq.paired.fq ~/repos/universc/test/shared/dropseq-test
mv SRR1873277_1.fastq.paired.fq SRR1873277_R1.fastq
mv GSM1629192_chr21_R2.fastq.paired.fq universc/test/shared/dropseq-test/SRR1873277_R2.fastq
mv GSM1629192_chr21_R2.fastq.paired.fq universc/test/shared/dropseq-test/SRR1873277_R2.fastq</code></pre>
<p>Run UniverSC:</p>
<pre><code>bash universc/launch_universc.sh -t "DropSeq" --setup
# call on dropseq with files
bash universc/launch_universc.sh --id "test-dropseq" --technology "nadia" \
--reference "universc/test/cellranger_reference/cellranger-tiny-ref/3.0.0" \
--read1 "universc/test/shared/dropseq-test/SRR1873277_S1_L001_R1_001" \
--read2 "universc/test/shared/dropseq-test/SRR1873277_S1_L001_R2_001" </code></pre>
<h4 id="running-launch_universc.sh-on-icell8-data">Running launch_universc.sh on ICELL8 data</h4>
<pre><code># call on icell8 files with custom whitelist and non-standard file names
bash launch_universc.sh --setup -t "icell8" --barcodefile "test/shared/icell8-test/BarcodeList.txt"
bash launch_universc.sh --id "test-icell8-custom" --technology "iCell8" \
--reference "test/cellranger_reference/cellranger-tiny-ref/3.0.0" \
--read1 "test/shared/icell8-test/iCELL8_01_S1_L001_R1_001.fastq" "test/shared/icell8-test/iCELL8_01_S1_L002_R1_001.fastq" \
--read2 "test/shared/icell8-test/iCELL8_01_S1_L001_R2_001.fastq" "test/shared/icell8-test/iCELL8_01_S1_L002_R2_001.fastq" \
--barcodefile "test/shared/icell8-test/BarcodeList.txt" \
--jobmode "sge"</code></pre>
<h3 id="debugging">Debugging</h3>
<p>We've made considerable efforts to ensure you don't run into problems. However, it may be necessary from time to time to troubleshoot issues calling UniverSC. For other technologies, various changes to Cell Ranger are made in a reversible fashion. If you run into problems you can restore Cell Ranger to default parameters:</p>
<pre><code>bash launch_universc.sh -t "10x" --setup</code></pre>
<p>Then you can call <code>launch_universc.sh</code> as above or configure Cell Ranger for your technology of choice such as :</p>
<pre><code>bash launch_universc.sh --setup -t "icell8" --barcodefile "test/shared/icell8-test/BarcodeList.txt"</code></pre>
<p>Set up calls are particularly useful to set up the whitelist in advance of running multiple samples simultaneously, provided they are the same technology.</p>
<p>It is also possible that your Cell Ranger installation will be "locked" by UniverSC. This is intentional to prevent different technologies running simultaneously. When running Cell Ranger, we need to ensure that the barcode whitelist corresponds to the technology that is running and cannot be changed until existing runs will finish.</p>
<p>However, this means that in the case of an error or if a job is "killed", then the lock file will not be cleared. You can do this manually as follows:</p>
<pre><code>cellrangerversion=`cellranger count --version | head -n 2 | tail -n 1 | cut -f2 -d'(' | cut -f1 -d')'`
cellrangerpath=`which cellranger`
rm ${cellrangerpath}-cs/${cellrangerversion}/lib/python/cellranger/barcodes/.lock</code></pre>
<p>When doing this <em>please ensure that no other instances are running</em> for Cell Ranger convert.</p>
<p>You can also see the current configuration of UniverSC for each Cell Ranger install as follows:</p>
<pre><code>cellrangerversion=`cellranger count --version | head -n 2 | tail -n 1 | cut -f2 -d'(' | cut -f1 -d')'`
cellrangerpath=`which cellranger`
cat ${cellrangerpath}-cs/${cellrangerversion}/lib/python/cellranger/barcodes/.lastcalled</code></pre>
<p>These columns show the barcode length, UMI length, and barcode whitelist of the last technology used by UniverSC. Please <em>do not remove this file</em> unless the last technology used is 10x Genomics.</p>
<h2 id="licensing">Licensing <span id="Licensing"><span></h2>
<p>This package is provided open-source on a GPL-3 license. This means that you are free to use and modify this code provided that they also contain this license.</p>
<p>Please note that we are third-party developers releasing it for use by users like ourselves. We are not affiliated with 10x Genomics, Dolomite Bio, Takara Bio, or any other vendor of single-cell technologies. This software is not supported by 10x Genomics and only changes data formats so that other technologies can be used with the Cell Ranger pipeline.</p>
<p>Cell Ranger (version 2.0.2, 2.1.0, 2.1.0, and 3.0.2) has been released open source on and MIT license on GitHub. We use this version of Cell Ranger for testing and running our tools. Note that the code that generates the 'cloupe' files is not included in this release. The Cloupe browser uses files generated by proprietary closed-source software and is subject to the 10x Genomics End-User License Agreement which does not allow use with data generated from other platforms.</p>
<p>Therefore 'launch_universc.sh' does not support Cloupe files and you should not use them with technologies other than 10x Genomics.</p>