Home

Awesome

Conect Segment-Anything with CLIP

We aim to classify the output masks of segment-anything with the off-the-shelf CLIP models. The cropped image corresponding to each mask is sent to the CLIP model. <img src="https://github.com/PengtaoJiang/SAM-CLIP/blob/main/imgs/pipeline.png" width="100%" height="50%">

Other Nice Works

Editing-Related Works

  1. sail-sg/EditAnything
  2. IDEA-Research/Grounded-Segment-Anything
  3. geekyutao/Inpaint-Anything
  4. Luodian/RelateAnything

Nerf-Related Works

  1. ashawkey/Segment-Anything-NeRF
  2. Anything-of-anything/Anything-3D
  3. Jun-CEN/SegmentAnyRGBD
  4. Pointcept/SegmentAnything3D

Segmentation-Related Works

  1. maxi-w/CLIP-SAM
  2. Curt-Park/segment-anything-with-clip
  3. kadirnar/segment-anything-video
  4. fudan-zvg/Semantic-Segment-Anything
  5. continue-revolution/sd-webui-segment-anything
  6. RockeyCoss/Prompt-Segment-Anything
  7. ttengwang/Caption-Anything
  8. ngthanhtin/owlvit_segment_anything
  9. lang-segment-anything
  10. helblazer811/RefSAM
  11. Hedlen/awesome-segment-anything
  12. ziqi-jin/finetune-anythin
  13. ylqi/Count-Anything
  14. xmed-lab/CLIP_Surgery
  15. RockeyCoss/Prompt-Segment-Anything
  16. segments-ai/panoptic-segment-anything
  17. Cheems-Seminar/grounded-segment-any-parts
  18. aim-uofa/Matcher
  19. SysCV/sam-hq
  20. CASIA-IVA-Lab/FastSAM
  21. ChaoningZhang/MobileSAM
  22. JamesQFreeman/Sam_LoRA
  23. UX-Decoder/Semantic-SAM
  24. cskyl/SAM_WSSS
  25. ggsDing/SAM-CD
  26. yformer/EfficientSAM
  27. XiaRho/SEMat

Labelling-Related Works

  1. vietanhdev/anylabeling
  2. anuragxel/salt

Tracking-Related Works

  1. gaomingqi/track-anything
  2. z-x-yang/Segment-and-Track-Anything
  3. achalddave/segment-any-moving

Medical-Related Works

  1. bowang-lab/medsam
  2. hitachinsk/SAMed
  3. cchen-cc/MA-SAM
  4. OpenGVLab/SAM-Med2D

Todo

  1. We plan to connect segment-anything with MaskCLIP.
  2. We plan to finetune on the COCO and LVIS datasets.

Run Demo

Download the sam_vit_h_4b8939.pth model from the SAM repository and put it at ./SAM-CLIP/. Follow the instructions to install segment-anything and clip packages using the following command.

cd SAM-CLIP; pip install -e .
pip install git+https://github.com/openai/CLIP.git

Then run the following script:

sh run.sh

Example

Input an example image and a point (250, 250) to the SAM model. The input image and output three masks as follows:

<center><img src="https://github.com/PengtaoJiang/SAM-CLIP/blob/main/imgs/ADE_val_00000001.jpg" width="50%" height="50%"></center>

The three masks and corresponding predicted category are as follows:

<center> <img src="https://github.com/PengtaoJiang/SAM-CLIP/blob/main/outs/ADE_val_00000001/outs.png" width="100%" height="50%"> </center> <center> <img src="https://github.com/PengtaoJiang/SAM-CLIP/blob/main/outs/ADE_val_00000001/logits.png" width="100%" height="50%"> </center>

You can change the point location at L273-274 of scripts/amp_points.py.

## input points 
input_points_list = [[250, 250]]
label_list = [1]