publishDir pattern specification conflicts with set-val-file in output channel

[fedemantica]

Bug report

Expected behavior and actual behavior

When using publishDir with a pattern specification containing a variable, the pattern is expected to be recognized.
This works well with nextflow version 20.04.1, but with nextflow version 21.10.6 or newer (and maybe with some previous versions, I am not sure where the problem arose) the matching outputs are not copied to the specified directory.
In particular, we only have this problem when the output channel contains a set-val-file specification (otherwise everything works as expected).

Steps to reproduce the problem

This snippet shoud reproduce the problem:

#!/usr/bin/env nextflow

Channel.from("Hs2_Mm2-test", "Hs2_Dme", "Mm2_Dme").toList().flatMap().set{input_test}

process test {
tag {"${comp_id}"}
stageInMode = 'copy'
publishDir "${params.output}", mode: "copy", pattern: "${comp_id}/species_test.txt"

input:
    val(comp_id) from input_test

output:
    file("${comp_id}/species_test.txt")
    set val(comp_id), file("${comp_id}/") into out_channel

script:
    """
     mkdir ${comp_id}
     echo "This is a test for intermediate files" > ${comp_id}/species_test.txt
     """
}

Program output

The program does not fail nor reports any warnings/errors. The files are just not copied in the output directory (specified in the command line)

N E X T F L O W  ~  version 21.10.6
Launching `nextflow_test.nf` [shrivelled_cajal] - revision: b7e777a5ae
executor >  crg (3)
[b4/eea357] process > test (Mm2_Dme) [100%] 3 of 3 ✔

Environment

• Nextflow version: 21.20.6 or newer (maybe even some previous versions, released after version 20.04.1)
• Java version: 11.0.1
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)

[pditommaso]

Try the latest version 22.10.6

[fedemantica]

I get the same problem/output:

N E X T F L O W  ~  version 22.10.6
Launching `nextflow_test.nf` [tiny_brown] DSL1 - revision: b7e777a5ae
executor >  crg (3)
[40/78bd5c] process > test (Hs2_Dme)      [100%] 3 of 3 ✔

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bentsherman]

I think problem is that the publish pattern depends on comp_id which is not available during process definition. It is a task-specific input so it can be used in the script block or in a closure.

The publish pattern does not support closures, so I tried to put the entire publishDir in a closure:

publishDir { [
    [ path: "${params.output}", mode: "copy", pattern: "${comp_id}/species_test.txt" ]
] }

This actually doesn't work because ProcessConfig doesn't handle a closure input correctly:

nextflow/modules/nextflow/src/main/groovy/nextflow/script/ProcessConfig.groovy

Lines 875 to 886 in a60ef72

	ProcessConfig publishDir( target ) {
	if( target instanceof List ) {
	for( Object item : target ) { publishDir(item) }
	}
	else if( target instanceof Map ) {
	publishDir( target as Map )
	}
	else {
	publishDir([path: target])
	}
	return this
	}

So I added some logic to support closures:

        else if( target instanceof Closure ) {
            configProperties.put('publishDir', target)
        }

But it still didn't work 🤷‍♂️

Not sure what else to try