Performance idea: find duplicate samples more efficiently

[nsheff]

Raised in #432 (comment)

I actually think the most important performance-related problem is not actually storing the samples in two ways, but in the way duplicate sample name are identified and merged, which is extremely inefficient.

This looks like an N^2 approach, since I'm not sure but I bet pythons List.count() function has to go through all the items in the list:

peppy/peppy/project.py

Lines 637 to 649 in cac87fb

	def _get_duplicated_sample_ids(self, sample_names_list: List) -> set:
	return set(
	[
	sample_id
	for sample_id in track(
	sample_names_list,
	description="Detecting duplicate sample names",
	disable=not (self.is_sample_table_large and self.progressbar),
	console=Console(file=sys.stderr),
	)
	if sample_names_list.count(sample_id) > 1
	]
	)

and then samples are looped through again here:

peppy/peppy/project.py

Line 590 in cac87fb

) = self._get_duplicated_and_not_duplicated_samples(

and I think other times. So there's some algorithmic issues. This should be able to be accomplished in 1 linear pass through the sample objects. Can probably be done very quickly using pandas, but even if using sample objects just a single loop should probably work.

Basically just fixing the counting to go through and count once would probably be a huge speed benefit, and should be really simple to implement.

[nsheff]

@neil-phan this is what we discussed

[nsheff]

This is now released on 0.35.5.