0.6 RC2

[vreuter]

Close #129
Close #130
Close #133
Close #136
Close #137
Close #139
Close #134
Close #132

• Update requirements.
• Separate some logical concerns within Project constructor and Sample subclass seeking.
• Clean up messaging about sample run skip reasons.
• Remove outdated example files.
• Require setuptools that's now generally assumed for install
• Faster and more informative failure for nonexistent pipelines_dir interface file

[nsheff]

I confirm that this solves the missing interface problem in my test.

[nsheff]

But this did not force a new reliance on pandas 0.20 when I had pandas 0.19 installed, so the requirement thing is not working yet.

[nsheff]

My commits fix #134 for me. Previously, an old version of pandas was satisfying it because the version number was getting stripped off when we read in the requirements file. @vreuter please double-check f58fc74

[nsheff]

Addresses #133

[vreuter]

When you get the chance @afrendeiro can you rerun the case that caused #129 with looper --dbg ... and see what the debugging output indicates regarding what's happening as the Project searches for the Sample subclasses?

[afrendeiro]

Okay, I've pulled the latest 0.6-rc2 and ran looper with --dbg.
A few issues with this:

1. for a successful install setup.py needs to stop looking for the script files that were removed;
2. why argparse.SUPPRESS on the log options? Good for simplicity but prevents non-devs from seeing the logs...

Here's a sample from a looper --dbg check (highlighted important bits):

looper:787 [INFO] > Command: check (Looper version: 0.6.0-rc2) 
...
models:632 [WARNING] > Looper v0.6 suggests switching from pipelines_dir to  pipeline_interfaces. See docs for details. 
...
models:905 [DEBUG] > Adding sample sheet 
models:915 [DEBUG] > Creating samples from annotation sheet 
models:1162 [DEBUG] > **Successfully imported pipelines **
models:999 [DEBUG] > merge: **sample** 'ATAC-seq_HAP1_ACTB_r1_2471_10'; lane=3 
models:999 [DEBUG] > merge: sample 'ATAC-seq_HAP1_ACTB_r1_2471_10'; data_source=/scratch/lab_bsf/samples/BSF_0279_HG7MMBBXX/BSF_0279_HG7MMBBXX_3_sample
...

So looper is still finding my pipeline_dirs but making generic Sample objects. I don't understand.
Do you guys have any pipeline with __library__ to test this?

One more issue with this update: instantiating a Project no longer calls add_sample_sheet internally.
Wasn't the consensus on this that they would be tied? For looper CLI usage there's no difference since looper will call that but for interactive use this again implies a change...

[nsheff]

do you have a module called 'pipelines' installed, and does this module have a matching sample subclass? This is where it will be looking (not in the pipeline itself), by my reading...

[nsheff]

yeah, the current logic is not the way I thought it worked. this is going to take some reworking I think. It's not doing what it used to somehow...

@afrendeiro: do you want it to be pulling the child class from a global 'pipelines' install (bad way imo because it relies on the magic name 'pipelines'), or do you want it to pull child class from the python file that defines the pipeline (preferable in my opinion).

Right now it's doing the former and not the latter as far as I can tell; I think we should switch to the latter and exclude the former.

found problems

[vreuter]

I deliberately pulled two functions from the tail of the Project constructor with 49f1e5f, because I was unaware of previous discussion about this. In general, I much prefer to separate things, but if there's consensus / I'm outvoted to include the calls at the end of the constructor logic, that's OK.

I see two main disadvantages--only one of which has unknown/potentially significant extent, the other is known and bounded--and one advantage, of inclusion in the constructor. The significant disadvantage is that this complicates inheriting Project. If a potential client developer wanted to derive from Project but have additional constructor logic before those calls at the end, he or she would need to modify the constructor itself rather than just overriding the functions that would be called at its end / post-construction. While probably not a frequent use case, this feels like a more substantial issue. The minor one is that this complicates testability, either requiring mocking or eroding test logic independence. The advantage is as mentioned, that it reduces needed function calls when using a Project in isolation, outside of the CLI.

[vreuter]

Regarding the Sample subclasses, the logic could also get messed up if the import search finds a valid pipelines before a globally installed package. This sometimes happens when within the directory from which the import is attempted there's a module (or another package) with the same name as the targeted, globally-installed package.

[nsheff]

@vreuter How close are we to merging this?

[vreuter]

I'm set with it now. Did you have other cases you wanted to run?

[nsheff]

I'll give it a try with some of the ones I'm working on now. I'm mostly just itching to move on

[vreuter]

Are you making more changes?

[vreuter]

One note on the Sample subclasses that I noticed during this round of changes...it seems like some of the Sample subclass constructors assume that the input data is a Series. This became apparent when I'd represented a generic Sample as an ordinary dict and then passed that data through to the subtype constructor(s) within looper.run. This works fine, having Series be the base type that's passed, but in general I think it may make more sense to have the Samplesubclass constructors also operate on a normal dictionary and/or an Iterable of two-tuples since they're more basic/builtin types. This could lower the barrier to someone using a Sample, eliminating a need for familiarity with pandas.