passing multiple csv files to a collection runner in newman

[harishshet222]

How to pass multiple csv files in newman run?

newman run CollectionRunner.json --global-var IP_PORT="http://127.0.0.1:8088/" -r htmlextra --reporter-htmlextra-title "Result Report" --reporter-htmlextra-export /home/reports/HARISH.html --iteration-data /home/dataset1.csv

I want to pass multiple data sets to the collection, I have dataset1,dataset2,dataset3,dataset4 in my directory. all should gets executed for the collectionRunner and the RESULT REPORT should contain all the result,

I tried something like this, newman run CollectionRunner.json --global-var IP_PORT="http://127.0.0.1:8088/" -r htmlextra --reporter-htmlextra-title "Result Report" --reporter-htmlextra-export /home/reports/Result.html --iteration-data /home/dataset1.csv --iteration-data /home/dataset2.csv --iteration-data /home/dataset3.csv

But its not working

[adrijshikhar]

@codenirvana @shamasis I went through the code. I found that if we pass one file in with --iteration-data flag, the data after it is passed and checked whether it is an array or string. If it is not an array it is considered as the location of a file, and its data is extracted in iterationData function in lib/run/options.js. If it is an array, it is returned as it is, may be because it is considered as the iteration data and does not need to be parsed. When multiple files are specified using --iteration-data flag, location is trimmed and the last file is considered as the source. I tried to fix this by passing memoize function to extract the location of the files as an array, as done for the folder. I was successful in doing that but the check in iterationData function in lib/run/options.js, checks the location variable for array and returns it as it is, because it is supposed to be string from which data is to be extracted or an array of JSON object which is data itself, and the data from the location array is never extracted. What I am proposing is to modify the check and let it parse the location array with previous functionality and extract data from each location and append it to the single iterationData variable. What say?

[adrijshikhar]

I would like to work on this issue.

[Gaurav-Punjabi]

Is anyone still working on this issue? I'd like to contribute.

[adrijshikhar]

Yes, i am. There is a darft PR linked to the issue.

[shamasis]

Hello all.

Let’s deep dive into the workflow and usage of this feature. How would multiple data files work in general as a feature? How would that add value as a general feature and when should users start using this.

Maybe @harishshet222 can elaborate on the use case of adding multiple data files and how we can build this as a general feature for everyone.

[IshaanDesai45]

I would like to work on the issue , how to get started??

[adrijshikhar]

Already working on it, and there is a draft pr pending.

[ustccq]

Hi, all.
I need a feature that, to point to a directory, or multi files,
make requests in collection subfolder A execution times corresponding to the row count of file A who giving the parameters needed in subfolder A.
And requests in subfolder B executed times corresponding to its data file B.

I hope it will just be like what I can do with JMeter. Just one data file really can't cover most complex usage scenarios.

[Haniket]

is this issue still open....if so I would like to work on it.

[Lakshmi-PriyaS]

Is there any implementation for this requirement? It is difficult to pass different data in pre-requisite every time for each test case in a collection. It would be better if we have a single file to have all data with sub folder name, so that based on sub folder name it can pick the data from csv file.

[Lampei]

@shamasis You had asked for an example of usage of this feature...I just now ran into one.
I currently have my data files setup as a dev/qa/stg etc as the values being passed in those environment could be different (IDs, etc)
For those values that do not change though, I would like to have one data file that can be used across environments.
e.g. my-api.multi.json my-api.dev.json
This way, I'm reducing the duplication necessary across my data files, and keeping the environment specific data files very small.

[MarcoDeJesus]

Thinking on a JSON format data file, As we already know that the iterationData property accepts an array of javascript objects, as in:

newman.run({ collection: require('./sample-collection.json'), iterationData: [{ "var": "data", "var_beta": "other_val" }] )}

What about creating a function that accepts an many datasets as required and returns a mixed json data files turned into an array and then pass it to iterationData property? like:

const iterationDataMixer = (...args) => {
  let mixedDataInArray = []
  let jsDataObject = {};
      
  // First Object in the array to determine the number of iterations to run
  for (let i = 0; i < args[0].length; i++) {
    jsDataObject = {}

    for (let j = 0; j < args.length; j++) {
      jsDataObject = Object.assign(jsDataObject, args[j][i]);
    }
    mixedDataInArray.push(jsDataObject);
  }

  return mixedDataInArray;
}

module.exports = {
  iterationDataMixer
}

Finally just pass the new mixed data array to your iterationData property:

let firstDataFile = [{ "var": "data", "var_beta": "other_val" }];
let secondDataFile =  [{ "var": "data", "var_beta": "other_val" }];
let thirdDataFile =  [{ "var": "data", "var_beta": "other_val" }];

newman.run({
    collection: require('./sample-collection.json'),
    iterationData: iterationDataMixer(firstDataFile, secondDataFile, thirdDataFile)
)}

Consider that you need to have data files with the same array sizes