Tags · jchenpku/samtools

1.10

samtools release 1.10:

Changes affecting the whole of samtools, or multiple sub-commands:

* Samtools now uses the new HTSlib header API. As this adds more
checks for invalid headers, it is possible that some illegal files
will now be rejected when they would have been allowed by earlier
versions. (samtools#998)

Examples of problems that will now be rejected include @sq lines
with no SN: tag, and @rg or @pg lines with no ID: tag.

* samtools sub-commands will now add '@pg' header lines to output
sam/bam/cram files. To disable this, use the '--no-PG' option.
(samtools#1087; samtools#1097)

* samtools now supports alignment records with reference positions
greater than 2 gigabases. This allows samtools to process
alignments for species which have large chromosomes, like axolotl
and lungfish. Note that due to file format limitations, data
with large reference positions must use the SAM format. (samtools#1107;
samtools#1117)

* Improved the efficiency of reading and writing SAM format data by
2 fold (single thread). This is further improved by the ability to
use multiple threads, as previously done with BAM and CRAM.

* samtools can now write BGZF-compressed SAM format. To enable
this, either save files with a '.sam.gz' suffix, or use
'--output-fmt sam.gz'.

* samtools can now index BGZF-compressed SAM files.

* The region parsing code has been improved to handle colons in
reference names. Strings can be disambiguated by the use of
braces, so for example when reference sequences called "chr1" and
"chr1:100-200" are both present, the regions "{chr1}:100-200" and
"{chr1:100-200}" unambiguously indicate which reference is being
used. (samtools#864)

* samtools flags, flagstats, idxstats and stats now have aliases
flag, flagstat, idxstat and stat. (samtools#934)

* A new global '--write-index' option has been added. This allows
output sam.gz/bam/cram files to be indexed while they are being
written out. This should work with addreplacerg, depad, markdup,
merge, sort, split, and view. (samtools#1062)

* A global '--verbosity' option has been added to enable/disable
debugging output. (samtools#1124, thanks to John Marshall)

* It is now possible to have data and index files stored in
different locations. There are two ways to tell samtools
where to find the index:

1. Samtools bedcov, depth, merge, mpileup, stats, tview, and
view accept a new option (-X). When this is used, each input
sam/bam/cram listed on the command line should have a
corresponding index file.

Note that all the data files should be listed first, followed
by all the index files. (samtools#978, thanks to Mingfei Shao)

2. A delimiter '##idx##' can be appended to the data file name
followed by the index file name. This can be used both for
input files and outputs when indexing on-the-fly.

* HTSlib (and therefore SAMtools) now uses version 4 signatures by
default for its s3:// plug-in. It can also write to S3 buckets,
as long as version 4 signatures are in use. See HTSlib's NEWS
file and htslib-s3-plugin manual page for more information.

* HTSlib (and therefore SAMtools) no longer considers a
zero-length file to be a valid SAM file. This has been
changed so that pipelines such as `somecmd | samtools ...`
with `somecmd` aborting before outputting anything will now
propagate the error to the second command.

* The samtools manual page has been split up into one for each
sub-command. The main samtools.1 manual page now lists the
sub-commands and describes the common global options. (samtools#894)

* The meaning of decode_md, store_md and store_nm in the fmt-option
section of the samtools.1 man page has been clarified. (samtools#898,
thanks to Evan Benn)

* Fixed numerous memory leaks. (samtools#892)

* Fixed incorrect macro definition on Windows. (samtools#950)

* bedcov, phase, misc/ace2sam and misc/wgsim now check for
failure to open files. (samtools#1013, thanks to Julie Blommaert and
John Marshall)

Changes affecting specific sub-commands:

* A new "coverage" sub-command has been added. This prints a
tabular format of the average coverage and percent coverage for
each reference sequence, as well as number of aligned reads,
average mapping quality and base quality. It can also (with the
'-m' option) plot a histogram of coverage across the genome.
(samtools#992, thanks to Florian Breitwieser)

* samtools calmd:

- Reference bases in MD: tags are now converted to upper case.
(samtools#981, samtools#988)

* samtools depth:

- Add new options to write a header to the output (-H) and to
direct the output to a file (-o). (samtools#937, thanks to Pierre
Lindenbaum)

- New options '-g' and '-G' can be used to filter reads. (samtools#953)

- Fix memory leak when failing to set CRAM options. (samtools#985, thanks
to Florian Breitwieser)

- Fix bug when using region filters where the '-a' option did not
work for regions with no coverage. (samtools#1113; samtools#1112 reported by
Paweł Sztromwasser)

* samtools fasta and fastq:

- '-1 FILE -2 FILE' with the same filename now works properly.
(samtools#1042)

- '-o FILE' is added as a synonym for '-1 FILE -2 FILE'. (samtools#1042)

- The '-F' option now defaults to 0x900 (SECONDARY,SUPPLEMENTARY).
Previously secondary and supplementary records were filtered
internally in a way that could not be turned off. (samtools#1042; samtools#939
reported by @finswimmer)

- Allow reading from a pipe without an explicit '-' on the command
line. (samtools#1042; samtools#874 reported by John Marshall)

- Turn on multi-threading for bgzf compressed output files. (samtools#908)

- Fixed bug where the samtools fastq -i would output incorrect
information in the Casava tags for dual-index reads. It
also now prints the tags for dual indices in the same way as
bcl2fastq, using a '+' sign between the two parts of the
index. (samtools#1059; samtools#1047 reported by Denis Loginov)

* samtools flagstat:

- Samtools flagstat can now optionally write its output in JSON
format or as a tab-separated values file. (samtools#1106, thanks to
Vivek Rai).

* samtools markdup:

- It can optionally tag optical duplicates (reads following
Illumina naming conventions only). The is enabled with the
'-d' option, which sets the distance for duplicates to be
considered as optical. (samtools#1091; samtools#1103; samtools#1121; samtools#1128; samtools#1134)

- The report stats (-s) option now outputs counts for optical and
non-primary (supplementary / secondary) duplicates. It also
reports the Picard "estimate library size" statistic. A new
'-f' option can be used to save the statistics in a given file.
(samtools#1091)

- The rules for calling duplicates can be changed using the new
--mode option. This mainly changes the position associated with
each read in a pair. '--mode t' (the default) is the existing
behaviour where the position used is that of the outermost
template base associated with the read. Alternatively '--mode s'
always uses the first unclipped sequence base. In practice,
this only makes a difference for read pairs where the two reads
are aligned in the same direction. (samtools#1091)

- A new '-c' option can be used to clear any existing duplicate
tags. (samtools#1091)

- A new '--include-fails' option makes markdup include QC-failed
reads. (samtools#1091)

- Fixed buffer overflow in temporary file writer when writing a
mixture of long and short alignment records. (samtools#911; samtools#909)

* samtools mpileup:

- mpileup can now process alignments including CIGAR P (pad)
operators correctly. They will now also produce the correct
output for alignments where insertions are immediately followed
by deletions, or deletions by insertions. Note that due to
limitations in HTSlib, they are still unable to output sequences
that have been inserted before the first aligned base of a read.
(samtools#847; samtools#842 reported by Tiffany Delhomme. See also htslib issue
samtools#59 and pull request samtools#699).

- In samtools mpileup, a deletion or pad on the reverse strand is
now marked with a different character ('#') than the one used on
a forward strand ('*'), if the '--reverse-del' option is used.
(samtools#1070)

- New option '--output-extra' can be used to add columns for user
selected alignment fields or aux tags. (samtools#1073)

- Fixed double-counting of overlapping bases in alignment records
with deletions or reference skips longer than twice the insert
size. (samtools#989; samtools#987 reported by @dariomel)

- Improved manual page with documentation about what each output
column means. (samtools#1055, thanks to John Marshall)

* samtools quickcheck:

- Add unmapped (-u) option, which disables the check for @sq lines
in the header. (samtools#920, thanks to Shane McCarthy)

* samtools reheader:

- A new option '-c' allows the input header to be passed to a
given command. Samtools then takes the output of this command
and uses it as the replacement header. (samtools#1007)

- Make it clear in help message that reheader --in-place only
works on CRAM files. (samtools#921, thanks to Julian Gehring)

- Refuse to in-place reheader BAM files, instead of unexpectedly
writing a BAM file to stdout. (samtools#935)

* samtools split:

- In samtools split, the '-u' option no longer accepts an extra
file name from which a replacement header was read. The two
file names were separated using a colon, which caused problems
on Windows and prevented the use of URLs. A new '-h' option has
been added to allow the replacement header file to be specified
in its own option. (samtools#961)

- Fixed bug where samtools split would crash if it read a SAM
header that contained an @rg line with no ID tag. (samtools#954,
reported by @blue-bird1)

* samtools stats:

- stats will now compute base compositions for BC, CR, OX and RX
tags, and quality histograms for QT, CY, BZ and QX tags. (samtools#904)

- New stats FTC and LTC showing total number of nucleotides for
first and last fragments. (samtools#946)

- The rules for classifying reads as "first" or "last" fragment
have been tightened up. (samtools#949)

- Fixed bug where stats could over-estimate coverage when using
the target-regions option or when a region was specified on the
command-line. (samtools#1027; samtools#1025, reported by Miguel Machado; samtools#1029,
reported by Jody Phelan).

- Fixed error in stats GCD percentile depth calculation when the
depth to be reported fell between two bins. It would report the
depth entirely from the lower bin instead of taking a weighted
average of the two. (samtools#1048)

- Better catching and reporting of out of memory conditions.
(samtools#984; samtools#982, reported by Jukka Matilainen)

- Improved manual page. (samtools#927)

* samtools tview:

- tview can now display alignments including CIGAR P operators, D
followed by I and I followed by D correctly. See mpileup above
for more information. (samtools#847; samtools#734, reported by Ryan Lorig-Roach)

- The "go to position" text entry box has been made wider. (samtools#968,
thanks to John Marshall)

- Fixed samtools tview -s option which was not filtering reads
correctly. It now only shows reads from the requested sample or
read group. (samtools#1089)

* samtools view:

- New options '-d' and '-D' to only output alignments which
have a tag with a given type and value. (samtools#1001, thanks to
Gert Hulselmans)

* misc/plot-bamstats script:

- Fixed merge (-m) option. (samtools#923, samtools#924 both thanks to Marcus D
Sherman)

- Made the quality heatmap work with gnuplot version 5.2.7 and
later. (samtools#1068; samtools#1065 reported by Martin Mokrejš)

- Fixed --do-ref-stats bug where fasta header lines would be
counted as part of the sequence when the --targets option was
used. (samtools#1120, thanks to Neil Goodgame)

* Removed the misc/varfilter.py Python script, as it takes
consensus-pileup as input, which was removed from samtools in
release 0.1.17 in 2011. (samtools#1125)

Dec 6, 2019
76877ea
zip
tar.gz

1.9

samtools release 1.9:

 * Samtools mpileup VCF and BCF output is now deprecated.  It is
   still functional, but will warn.  Please use bcftools mpileup
   instead. (samtools#884)

 * Samtools mpileup now handles the '-d' max_depth option
   differently.  There is no longer an enforced minimum, and '-d
   0' is interpreted as limitless (no maximum - warning this may
   be slow).  The default per-file depth is now 8000, which
   matches the value mpileup used to use when processing a single
   sample.  To get the previous default behaviour use the higher
   of 8000 divided by the number of samples across all input
   files, or 250. (samtools#859)

 * Samtools stats new features:

   - The '--remove-overlaps' option discounts overlapping portions
     of templates when computing coverage and mapped base counting.
     (samtools#855)



   - When a target file is in use, the number of bases inside the
     target is printed and the percentage of target bases with
     coverage above a given threshold specified by the
     '--cov-threshold' option. (samtools#855)

   - Split base composition and length statistics by first and last
     reads. (samtools#814, samtools#816)

 * Samtools faidx new features:

   - Now takes long options. (samtools#509, thanks to Pierre Lindenbaum)

   - Now warns about zero-length and truncated sequences due to the
     requested range being beyond the end of the sequence. (samtools#834)



   - Gets a new option (--continue) that allows it to carry on when a
     requested sequence was not in the index. (samtools#834)

   - It is now possible to supply the list of regions to output in a
     text file using the new '--region-file' option. (samtools#840)

   - New '-i' option to make faidx return the reverse complement of
     the regions requested. (samtools#878)

   - faidx now works on FASTQ (returning FASTA) and added a new fqidx
     command to index and return FASTQ. (samtools#852)

 * Samtools collate now has a fast option '-f' that only operates on
   primary pairs, dropping secondary and supplementary.  It tries to
   write pairs to the final output file as soon as both reads have
   been found. (samtools#818)

 * Samtools bedcov gets a new '-j' option to make it ignore deletions
   (D) and reference skips (N) when computing coverage. (samtools#843)

 * Small speed up to samtools coordinate sort, by converting it to
   use radix sort. (samtools#835, thanks to Zhuravleva Aleksandra)

 * Samtools idxstats now works on SAM and CRAM files, however this
   isn't fast due to some information lacking from indices. (samtools#832)

 * Compression levels may now be specified with the level=N
   output-fmt-option.  E.g. with -O bam,level=3.

 * Various documentation improvements.

 * Bug-fixes:

   - Improved error reporting in several places. (samtools#827, samtools#834, samtools#877,
     cd7197)

   - Various test improvements.

   - Fixed failures in the multi-region iterator (view -M) when
     regions provided via BED files include overlaps (samtools#819,
     reported by Dave Larson).

   - Samtools stats now counts '=' and 'X' CIGAR operators when
     counting mapped bases. (samtools#855)

   - Samtools stats has fixes for insert size filtering (-m, -i).
     (samtools#845; samtools#697 reported by Soumitra Pal)

   - Samtools stats -F now longer negates an earlier -d option.
     (samtools#830)

   - Fix samtools stats crash when using a target region. (samtools#875,
     reported by John Marshall)

   - Samtools sort now keeps to a single thread when the -@ option is
     absent. Previously it would spawn a writer thread, which could
     cause the CPU usage to go slightly over 100%. (samtools#833, reported by
     Matthias Bernt)

   - Fixed samtools phase '-A' option which was incorrectly defined
     to take a parameter. (samtools#850; samtools#846 reported by Dianne Velasco)

   - Fixed compilation problems when using C_INCLUDE_PATH. (samtools#870;
     samtools#817 reported by Robert Boissy)

   - Fixed --version when built from a Git repository. (samtools#844, thanks
     to John Marshall)

   - Use noenhanced mode for title in plot-bamstats.  Prevents
     unwanted interpretation of characters like underscore in
     gnuplot version 5. (samtools#829, thanks to M. Zapukhlyak)

   - blast2sam.pl now reports perfect match hits (no indels or
     mismatches). (samtools#873, thanks to Nils Homer)

   - Fixed bug in fasta and fastq subcommands where stdout would not
     be flushed correctly if the -0 option was used.

   - Fixed invalid memory access in mpileup and depth on alignment
     records where the sequence is absent.

Jul 18, 2018
b24d812
zip
tar.gz

1.8

samtools release 1.8:

 * samtools calmd now has a quiet mode.  This can be enabled by
   passing `-Q` to calmd. (Thanks to Colin Davenport)

 * In samtools depth `-d 0` will effectively remove the depth limit.
   (samtools#764)

 * Improvements made to samtools collate's interface and
   documentation.  It is now possible to specify an output file
   name using `-o`, instead of deriving it from the prefix used
   for temporary files.  The prefix itself is now optional if
   `-o` or `-O` (to stdout) is used. (samtools#780)

 * Bug-fixes:

   - Make samtools addreplacerg choose output format by file
     extension. (samtools#767; reported by Argy Megalios)

   - Merge tests now work on ungzipped data, allowing tests to be run
     against different deflate libraries.

   - samtools markdup error messages about missing tags have been
     updated with the suggestion that samtools fixmate is run
     beforehand. (samtools#765; reported by Yudong Cai)

   - Enables the `--reference` option for samtools fastq.  Now works
     like other programs when a reference sequence is needed for CRAM
     files. (samtools#791, reported by Milana Kaljevic)

Apr 3, 2018
61e9762
zip
tar.gz

1.7

Samtools release 1.7: markdup supplementaries, multi-region iterator

* HTSlib, and so samtools, now support BAMs which include CIGARs with more
  than 65535 operations as per HTS-Specs 18th November (dab57f4 and 2f915a8).

* samtools quickcheck will now write a warning to stderr if it finds
  any problems.  These messages can be suppressed with a new `-q` option.

* samtools markdup can now mark supplementary alignments of reads where
  the primary alignment is found to be a duplicate.  Supplementary marking
  can be turned on by passing the `-S` option to markdup.  When this
  option is enabled, all the alignment data will be written to a temporary
  file so that supplementary alignments that occur before a duplicated
  primary can be correctly marked in the final output.  The location
  of this temporary file can be influenced using the new `-T` option.

* samtools view now supports HTSlib's new multi-region iterator.
  This can be enabled by passing the `-M` option to view.  When using
  this option:

  - The BED filter (`-L` option) will use the index to skip through the file
  - Reads from overlapping regions will only be output once

* samtools bedcov will now ignore BED comment and header lines (samtools#571; thanks
  to Daniel Baker).

* samtools collate now updates the @hd SO: and GO: tags, and sort will
  remove a GO: tag if present.  (samtools#757; reported by Imran Haque).

* Bug-fixes:

 - maq2sam now checks for input files that end early.  (samtools#751; patch supplied
   by Alexandre Rebert of the Mayhem team, via Andreas Tille from Debian.)

 - Fixed incorrect check when looking up header tags that could lead
   to a crash in samtools stats. (samtools#208; thanks to Dave Larson.)

 - Fixed bug in samtools fastq `-O` option where it would fail if
   the OQ tag in the input file had an unexpected type. (samtools#758;
   reported by Taejeong Bae)

 - The MD5 calculations in samtools dict and md5fa did not handle
   non-alphabetic characters in the same way as the CRAM MD5 function.
   They have now been updated to match. (samtools#704; reported by Chris Norman).

 - Fix possible infinite loop in samtools targetcut.

 - Building bam_tview_curses should no longer fail if a curses header file
   cannot be found.

Jan 26, 2018
6d79411
zip
tar.gz

1.6

Release 1.6: New markdup command; sort improvements

* Added new markdup sub-command and '-m' option for fixmate.  Used together,
  they allow duplicates to be marked and optionally removed.  This
  fixes a number of problems with the old 'rmdup' sub-command, for
  example samtools issue samtools#497.  'rmdup' is kept for backwards compatibility
  but 'markdup' should be used in preference.

* Sort is now much better at keeping within the requested memory limit.  It
  should also be slightly faster and need fewer temporary files when the file
  to be sorted does not fit in memory.  (samtools#593; thanks to Nathan Weeks.)

* Sort no longer rewrites the header when merging from files.  It can also
  now merge from memory, so fewer temporary files need to be written and
  it is better at sorting in parallel when everything fits in memory.

* Both sort and merge now resolve ties when merging based on the position
  in the input file(s).  This makes them fully stable for all ordering
  options.  (Previously position sort was stable, but name and by tag
  sorts were not).

* New --output-qname option for mpileup.

* Support for building on Windows using msys2/mingw64 or cygwin has
  been improved.

Sep 28, 2017
f4dc22a
zip
tar.gz

1.5

Release 1.5 [Solstice Release] (21st June 2017)

--------------------

* Samtools fastq now has a -i option to create a fastq file from an index
  tag, and a -T option (similar to -t) to add user specified aux tags to
  the fastq header line.

* Samtools fastq can now create compressed fastq files, by giving the
  output filenames an extention of .gq, .bgz, or .bgzf

* Samtools sort has a -t TAG option, that allows records to be sorted by
  the value of the specified aux tag, then by position or name.  Merge
  gets a similar option, allowing files sorted this way to be merged.
  (samtools#675; thanks to Patrick Marks of 10xgenomics).

Jun 20, 2017
f510fb1
zip
tar.gz

1.4.1

Release 1.4.1 (8th May 2017)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Added options to fastq to create fastq files from BC (or other)
  tags.

* Samtools view has gained a -G <flags> option to exclude on all bits
  set.  For example to discard reads where neither end has been
  mapped use "-G 12".

* Samtools cat has a -b <fofn> option to ease concatenation of many
  files.

* Added misc/samtools_tab_completion for bash auto-completion of
  samtools sub-commands. (samtools#560)

* Samtools tview now has J and K keys for verticale movement by 20
  lines. (samtools#257)

* Various compilation / portability improvements.

* Fixed issue with more than 65536 CIGAR operations and SAM/CRAM files.
(samtools#667)

May 8, 2017
8f8600a
zip
tar.gz

1.4

Release 1.4 (13 March 2017)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Noteworthy changes in samtools:

* Fixed Issue samtools#345 - out-by-one error in insert-size in samtools stats

* bam_split now add a @pg header to the bam file

* Added mate cigar tag support to fixmate

* Multi-threading is now supported for decoding BAM and CRAM (as well
  as the previously supported encoding).  Most commands that read BAM
  or CRAM have gained an -@ or --threads arguments, providing a
  significant speed bonus.  For commands that both read and write
  files the threads are shared between decoding and encoding tasks.

* Added -a option to samtools mpileup to show all locations, including
  sites with zero depth; repeating the option as -aa or -a -a additionally
  shows reference sequences without any reads mapped to them (samtools#496).

* The mpileup text output no longer contains empty columns at zero coverage
  positions.  Previously it would output "...0\t\t..." in some circumstances
  (zero coverage due to being below a minumum base quality); this has been
  fixed to output as "...0\t*\t*..." with placeholder '*' characters as in
  other zero coverage circumstances (see PR samtools#537).

* To stop it from creating too many temporary files, samtools sort
  will now not run unless its per-thread memory limit (-m) is set to
  at least 1 megabyte (samtools#547).

* The misc/plot-bamstats script now has a -l / --log-y option to change
  various graphs to display their Y axis log-scaled.  Currently this
  affects the Insert Size graph (PR samtools#589; thanks to Anton Kratz).

* Fixmate will now also add and update MC (mate CIGAR) tags.

Mar 13, 2017
0a859b1
zip
tar.gz

1.3.1

Samtools release 1.3.1: bug fixes, notably sort temporary files

* The sort command creates any needed temporary files alongside the final
output file (similarly to the pre-1.3 behaviour), and now aborts when
it detects a collision with another sort invocation's temporary files.

When the -T PREFIX option specified is a directory (or when sorting to
standard output), a random component is now added to temporary filenames
to try to avoid collisions (samtools#432, samtools#523, samtools#529, samtools#535, PR samtools#530).

* All samtools commands now check for I/O errors more carefully, especially
when writing output files (samtools#111, samtools#253, samtools#470, PR samtools#467).

* Build fixes for 32-bit systems; be sure to run configure on such systems
to enable large file support and access to 2GiB+ files.

* The fasta/fastq/bam2fq command no longer ignores reads when the -s option
is used (samtools#532).

* The fastq -O option no longer crashes on reads that do not have an OQ tag
field (samtools#517).

* The merge and sort commands now handle (unusual) BAM files that have no
textual @sq headers (samtools#548, samtools#550).

* Sorting files containing @co headers no longer duplicates the comment
headers, which previously happened on large sorts for which temporary
files were needed (samtools#563).

* The rmdup and view -l commands no longer crash on @rg headers that do not
have a LB field (samtools#538).

* Fixed miscellaneous issues samtools#128, samtools#130, samtools#131, samtools#489, and samtools#514.

Apr 22, 2016
897c002
zip
tar.gz

PreviousNext

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1.10-31

1.10

1.9

1.8

1.7

1.6

1.5

1.4.1

1.4

1.3.1

Tags: jchenpku/samtools