how to use background point

[KingerFzb]

I try to let input_labels be 0 or -1,but it doesn't work
input_labels = np.array([[[1,0/-1]]],dtype=np.float32)

[kexul]

From the code, I'm afraid that background point is not supported:

An integer tensor of shape [B] where each element is 1,2 or 3.

EfficientSAM/efficient_sam/efficient_sam_decoder.py

Lines 55 to 93 in c9408a7

	def _embed_points(
	self,
	points: torch.Tensor,
	labels: torch.Tensor,
	) -> torch.Tensor:
	"""Embeds point prompts."""
	points = points + 0.5 # Shift to center of pixel
	point_embedding = self.pe_layer.forward_with_coords(
	points, self.input_image_size
	)
	invalid_label_ids = torch.eq(labels, -1)[:,:,None]
	point_label_ids = torch.eq(labels, 1)[:,:,None]
	topleft_label_ids = torch.eq(labels, 2)[:,:,None]
	bottomright_label_ids = torch.eq(labels, 3)[:,:,None]
	point_embedding = point_embedding + self.invalid_points.weight[:,None,:] * invalid_label_ids
	point_embedding = point_embedding + self.point_embeddings.weight[:,None,:] * point_label_ids
	point_embedding = point_embedding + self.bbox_top_left_embeddings.weight[:,None,:] * topleft_label_ids
	point_embedding = point_embedding + self.bbox_bottom_right_embeddings.weight[:,None,:] * bottomright_label_ids
	return point_embedding
	def forward(
	self,
	coords,
	labels,
	) -> torch.Tensor:
	"""
	Embeds different types of prompts, returning both sparse and dense
	embeddings.
	Arguments:
	points: A tensor of shape [B, 2]
	labels: An integer tensor of shape [B] where each element is 1,2 or 3.
	Returns:
	torch.Tensor: sparse embeddings for the points and boxes, with shape
	BxNx(embed_dim), where N is determined by the number of input points
	and boxes.
	"""
	return self._embed_points(coords, labels)

@yformer Any comments?

[yformer]

Hi @KingerFzb @kexul, thanks for your interest in our work!

We did not support background point for this version.

We have updated EfficientSAM for unified image and video segmentation. Can you give it a try?

Please check https://github.com/yformer/EfficientTAM

[kexul]

We did not support background point for this version.

Thanks for your update! I'll try! 😄

[kexul]

Hi @KingerFzb @kexul, thanks for your interest in our work!

We did not support background point for this version.

We have updated EfficientSAM for unified image and video segmentation. Can you give it a try?

Please check https://github.com/yformer/EfficientTAM

Hi @yformer , Is the model released? Can't wait to try it locally!