Mastering Picture and Video Segmentation with SAM 2

Introduction

This information will stroll you thru what  Section Something Mannequin 2  is, the way it works, and the way you’ll put it to use to portion objects in photos and movies. It presents state-of-the-art execution and adaptableness in fragmenting objects into photos, making it an vital useful resource for a assortment of laptop imaginative and prescient purposes. This straight factors to supplying a nitty-gritty, step-by-step walkthrough for establishing and using SAM 2 to carry out image division. By taking this direct, it is possible for you to to provide division covers for photos using each field and level prompts.

Studying Aims

• Describe the important thing options and purposes of the Section Something Mannequin 2 SAM 2 in picture and video segmentation.
• Efficiently configure a CUDA-enabled surroundings, set up vital dependencies, and clone the Section Something Mannequin 2  repository for picture segmentation duties.
• Apply SAM 2 to generate segmentation masks for photographs utilizing each field and level prompts and visualize the outcomes successfully.
• Consider how SAM 2 can revolutionize photograph and video modifying by enabling real-time segmentation, automating advanced duties, and democratizing content material creation for a broader viewers.

This text was printed as part of the Information Science Blogathon.

Conditions

A while just lately you start, assure you’ve received a CUDA-enabled GPU for faster dealing with. Additionally, confirm that you’ve got Python put in in your machine. This information assumes you’ve some fundamental data of Python and picture processing ideas.

What’s SAM 2?

 Section Something Mannequin 2  is an progressed instrument for image division created by Fb AI Inquire about (Affordable). On July twenty ninth, 2024, Meta AI discharged SAM 2, an progressed image and video division institution present. SAM 2 empowers purchasers to produce focuses or packing containers in an image or video to create division covers for specific objects.

Click on right here to entry it.

Key Options of SAM 2

• Superior Masks Era: SAM 2 generates high-quality segmentation masks primarily based on consumer inputs, resembling factors or bounding packing containers.
• Flexibility: The mannequin helps each picture and video segmentation.
• Velocity and Effectivity: With CUDA help, SAM 2 can carry out segmentation duties quickly, making it appropriate for real-time purposes.

Core Elements of SAM 2

• Picture Encoder: Encodes the enter picture for processing.
• Immediate Encoder: Converts user-provided factors or packing containers right into a format the mannequin can use.
• Masks Decoder: Generates the ultimate segmentation masks primarily based on the encoded inputs.

Functions of SAM 2

Allow us to now look into the purposes of SAM 2 beneath:

• Picture and Video Enhancing: SAM 2 permits for exact object segmentation, enabling detailed edits and artistic results in images and movies.
• Autonomous Autos: In autonomous driving, SAM 2 can be utilized to establish and observe objects like pedestrians, autos, and highway indicators in real-time.
• Medical Imaging: SAM 2 can help in segmenting anatomical constructions in medical photographs, aiding in diagnostics and remedy planning.

What’s Picture Segmentation?

Picture segmentation is a pc imaginative and prescient approach that entails dividing a picture into a number of segments or areas to simplify its evaluation. Every phase represents a unique object or a part of an object inside the picture, making it simpler to establish and analyze particular parts.

Sorts of Picture Segmentation

• Semantic Segmentation: Classifies every pixel right into a predefined class.
• Occasion Segmentation: Differentiates between completely different cases of the identical object class.
• Panoptic Segmentation: Combines semantic and occasion segmentation.

Setting Up and Using SAM 2 for Picture Segmentation

We’ll information you thru the method of establishing the Section Something Mannequin 2 (SAM 2) in your surroundings and using its highly effective capabilities for exact picture segmentation duties. From guaranteeing your GPU is able to configuring the mannequin and making use of it to actual photographs, every step might be coated intimately that can assist you harness the complete potential of SAM 2.

Step 1: Verify GPU Availability and Set Up the Atmosphere

First, let’s make sure that your surroundings is correctly arrange, beginning with checking for GPU availability and setting the present working listing.

# Verify GPU availability and CUDA model
!nvidia-smi
!nvcc --version

# Import vital modules
import os

# Set the present working listing
HOME = os.getcwd()
print("HOME:", HOME)

Rationalization

• !nvidia-smi and !nvcc –model: These instructions test in case your framework incorporates a CUDA-enabled GPU and present the CUDA type.
• os.getcwd(): This work will get the present working catalog, which could be utilized for overseeing report methods.

Step 2: Clone the SAM 2 Repository and Set up Dependencies

Subsequent, we have to clone the SAM 2 repository from GitHub and set up the required dependencies.

# Clone the SAM 2 repository
!git clone https://github.com/facebookresearch/segment-anything-2.git

# Change to the repository listing
%cd segment-anything-2

# Set up the SAM 2 bundle
!pip set up -e .

# Set up further packages
!pip set up supervision jupyter_bbox_widget

Rationalization

• !git clone: Clones the SAM 2 repository to your native machine.
• %cd: Adjustments the listing to the cloned repository.
• !pip set up -e .: Installs the SAM 2 bundle in editable mode.
• !pip set up supervision jupyter_bbox_widget: Installs further packages required for visualization and bounding field widget help.

Step 3: Obtain Mannequin Checkpoints

Mannequin checkpoints are important, as they comprise the educated parameters of SAM 2. We are going to obtain a number of checkpoints for various mannequin sizes.

# Create a listing for checkpoints
!mkdir -p checkpoints

# Obtain the mannequin checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_tiny.pt -P checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_small.pt -P checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_base_plus.pt -P checkpoints
!wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt -P checkpoints

Rationalization

• !mkdir -p checkpoints: Creates a listing for storing mannequin checkpoints.
• !wget -q … -P checkpoints: Downloads the mannequin checkpoints into the checkpoints listing. Totally different checkpoints characterize fashions of various sizes and capabilities.

Step 4: Obtain Pattern Pictures

For demonstration functions, we’ll use some pattern photographs. You may as well use your photographs by following related steps.

# Create a listing for knowledge
!mkdir -p knowledge

# Obtain pattern photographs
!wget -q https://media.roboflow.com/notebooks/examples/canine.jpeg -P knowledge
!wget -q https://media.roboflow.com/notebooks/examples/dog-2.jpeg -P knowledge
!wget -q https://media.roboflow.com/notebooks/examples/dog-3.jpeg -P knowledge
!wget -q https://media.roboflow.com/notebooks/examples/dog-4.jpeg -P knowledge

Rationalization

• !mkdir -p knowledge: Creates a listing for storing pattern photographs.
• !wget -q … -P knowledge: Downloads the pattern photographs into the info listing.

Step 5: Set Up the SAM 2 Mannequin and Load an Picture

Now, we are going to arrange the SAM 2 mannequin, load a picture, and put together it for segmentation.

import cv2
import torch
import numpy as np
import supervision as sv

from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

# Allow CUDA if accessible
torch.autocast(device_type="cuda", dtype=torch.bfloat16).__enter__()

if torch.cuda.get_device_properties(0).main >= 8:
    torch.backends.cuda.matmul.allow_tf32 = True
    torch.backends.cudnn.allow_tf32 = True

# Set the gadget to CUDA
DEVICE = torch.gadget('cuda' if torch.cuda.is_available() else 'cpu')

# Outline the mannequin checkpoint and configuration
CHECKPOINT = "checkpoints/sam2_hiera_large.pt"
CONFIG = "sam2_hiera_l.yaml"

# Construct the SAM 2 mannequin
sam2_model = build_sam2(CONFIG, CHECKPOINT, gadget=DEVICE, apply_postprocessing=False)

# Create the automated masks generator
mask_generator = SAM2AutomaticMaskGenerator(sam2_model)

# Load a picture for segmentation
IMAGE_PATH = "/content material/WhatsApp Picture 2024-08-02 at 14.17.11_2b223e01.jpg"
image_bgr = cv2.imread(IMAGE_PATH)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)

# Generate segmentation masks
sam2_result = mask_generator.generate(image_rgb)

Rationalization

• CUDA Setup: Allows CUDA for quicker processing and units the gadget to GPU if accessible.
• Mannequin Setup: Builds the SAM 2 mannequin utilizing the desired configuration and checkpoint.
• Picture Loading: Masses and converts the pattern picture to RGB format.
• Masks Era: Makes use of the automated masks generator to generate segmentation masks for the loaded picture.

Step 6: Visualize the Segmentation Masks

We are going to now visualize the segmentation masks generated by SAM 2.

# Annotate the masks on the picture
mask_annotator = sv.MaskAnnotator(color_lookup=sv.ColorLookup.INDEX)
detections = sv.Detections.from_sam(sam_result=sam2_result)
annotated_image = mask_annotator.annotate(scene=image_bgr.copy(), detections=detections)

# Plot the unique and segmented photographs facet by facet
sv.plot_images_grid(
    photographs=[image_bgr, annotated_image],
    grid_size=(1, 2),
    titles=['source image', 'segmented image']
)

# Extract and plot particular person masks
masks = [
    mask['segmentation']
    for masks in sorted(sam2_result, key=lambda x: x['area'], reverse=True)
]

sv.plot_images_grid(
    photographs=masks[:16],
    grid_size=(4, 4),
    dimension=(12, 12)
)

Rationalization:

• Masks Annotation: Annotates the segmentation masks on the unique picture.
• Visualization: Plots the unique and segmented photographs facet by facet and in addition plots particular person masks.

Step7: Use Field Prompts for Segmentation

Field prompts permit us to specify areas of curiosity within the picture for segmentation.

# Outline the SAM 2 Picture Predictor
predictor = SAM2ImagePredictor(sam2_model)

# Reload the picture
image_bgr = cv2.imread(IMAGE_PATH)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)

# Encode the picture for bounding field enter
import base64

def encode_image(filepath):
    with open(filepath, 'rb') as f:
        image_bytes = f.learn()
    encoded = str(base64.b64encode(image_bytes), 'utf-8')
    return "knowledge:picture/jpg;base64,"+encoded

# Allow customized widget supervisor in Colab
IS_COLAB = True

if IS_COLAB:
    from google.colab import output
    output.enable_custom_widget_manager()

from jupyter_bbox_widget import BBoxWidget

# Create a bounding field widget
widget = BBoxWidget()
widget.picture = encode_image(IMAGE_PATH)

# Show the widget
widget

Rationalization

• Picture Predictor: Defines the SAM 2 picture predictor.
• Picture Encoding: Encodes the picture to be used with the bounding field widget.
• Widget Setup: Units up a bounding field widget for specifying areas of curiosity.

Step8: Get Bounding Containers and Carry out Segmentation

After specifying the bounding packing containers, we will use them to generate segmentation masks.

# Get the bounding packing containers from the widget
packing containers = widget.bboxes
packing containers = np.array([
    [
        box['x'],
        field['y'],
        field['x'] + field['width'],
        field['y'] + field['height']
    ] for field in packing containers
])

[{'x': 457, 'y': 341, 'width': 0, 'height': 0, 'label': ''},
 {'x': 205, 'y': 79, 'width': 0, 'height': 1, 'label': ''}]

# Set the picture within the predictor
predictor.set_image(image_rgb)

# Generate masks utilizing the bounding packing containers
masks, scores, logits = predictor.predict(
    field=packing containers,
    multimask_output=False
)

# Convert masks to binary format
masks = np.squeeze(masks)

# Annotate and visualize the masks
box_annotator = sv.BoxAnnotator(coloration=sv.Shade.white())
mask_annotator = sv.MaskAnnotator(color_lookup=sv.ColorLookup.INDEX)

detections = sv.Detections(
    xyxy=packing containers,
    masks=masks.astype(bool)
)

source_image = box_annotator.annotate(scene=image_bgr.copy(), detections=detections)
segmented_image = mask_annotator.annotate(scene=image_bgr.copy(), detections=detections)

# Plot the annotated photographs
sv.plot_images_grid(
    photographs=[source_image, segmented_image],
    grid_size=(1, 2),
    titles=['source image', 'segmented image']
)

Rationalization

• Bounding Containers: Retrieves the bounding packing containers specified utilizing the widget.
• Masks Era: Makes use of the bounding packing containers to generate segmentation masks.
• Visualization: Annotates and visualizes the masks on the unique picture.

Step9: Use Level Prompts for Segmentation

Level prompts permit us to specify particular person factors of curiosity for segmentation.

# Create level prompts primarily based on bounding packing containers
input_point = np.array([
    [
        box['x'] + (field['width'] // 2),
        field['y'] + (field['height'] // 2)
    ] for field in widget.bboxes
])
input_label = np.array([1] * len(input_point))

# Generate masks utilizing the purpose prompts
masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True
)

# Convert masks to binary format
masks = np.squeeze(masks)

# Annotate and visualize the masks
point_annotator = sv.PointAnnotator(color_lookup=sv.ColorLookup.INDEX)
mask_annotator = sv.MaskAnnotator(color_lookup=sv.ColorLookup.INDEX)

detections = sv.Detections(
    xyxy=sv.mask_to_xyxy(masks=masks),
    masks=masks.astype(bool)
)

source_image = point_annotator.annotate(scene=image_bgr.copy(), detections=detections)
segmented_image = mask_annotator.annotate(scene=image_bgr.copy(), detections=detections)

# Plot the annotated photographs
sv.plot_images_grid(
    photographs=[source_image, segmented_image],
    grid_size=(1, 2),
    titles=['source image', 'segmented image']
)

Rationalization

• Level Prompts: Creates level prompts primarily based on the bounding packing containers.
• Masks Era: Makes use of the purpose prompts to generate segmentation masks.
• Visualization: Annotates and visualizes the masks on the unique picture.

Key Factors to Keep in mind When Working SAM 2

Allow us to now look into few vital key factors beneath:

Revolutionizing Picture and Video Enhancing

• Potential to remodel the photograph and video modifying business.
• Future enhancements could embrace improved precision, decrease computational necessities, and superior AI integration.

Actual-Time Segmentation and Enhancing

• Evolution might result in real-time segmentation and modifying capabilities.
• Permits seamless alterations in movies and pictures with minimal effort.

Artistic Potentialities for All

• Opens up new inventive prospects for each professionals and amateurs.
• Simplifies the manipulation of visible content material, the creation of beautiful results, and the manufacturing of high-quality media.

Automating Complicated Duties

• Automates intricate segmentation duties.
• Considerably accelerates workflows, making subtle modifying extra accessible and environment friendly.

Democratizing Content material Creation

• Makes high-level modifying instruments accessible to a broader viewers.
• Empowers storytellers and evokes innovation throughout numerous sectors, together with leisure, promoting, and training.

Impression on VFX Business

• Enhances visible results (VFX) manufacturing by streamlining advanced processes.
• Reduces the effort and time required for creating intricate VFX, enabling extra bold tasks and enhancing total high quality.

Spectacular Potential of SAM 2

The Section Something Mannequin 2 (SAM 2) stands poised to revolutionize the fields of photograph and video modifying by introducing vital developments in precision and computational effectivity. By integrating superior AI capabilities, SAM 2 will allow extra intuitive consumer interactions and real-time segmentation and modifying, permitting seamless alterations with minimal effort. This groundbreaking expertise guarantees to democratize content material creation, empowering each professionals and amateurs to govern visible content material, create beautiful results, and produce high-quality media with ease.

As SAM 2 automates advanced segmentation duties, it’s going to speed up workflows and make subtle modifying accessible to a wider viewers. This transformation will encourage innovation throughout numerous industries, from leisure and promoting to training. Within the realm of visible results (VFX), SAM 2 will streamline intricate processes, lowering the effort and time wanted to create elaborate VFX. This may allow extra bold tasks, elevate the standard of visible storytelling, and open up new inventive prospects within the VFX world.

Conclusion

By following this information, you’ve realized methods to arrange and use the Section Something Mannequin 2 (SAM 2) for picture segmentation utilizing each field and level prompts. SAM 2 offers highly effective and versatile instruments for segmenting objects in photographs, making it a beneficial asset for numerous laptop imaginative and prescient duties. Be happy to experiment along with your photographs and discover the capabilities of SAM 2 additional.

Key Takeaways

• SAM 2 is a complicated device developed by Meta AI that allows exact and versatile picture and video segmentation utilizing each field and level prompts.
• The mannequin can considerably improve photograph and video modifying by automating advanced segmentation duties, making it extra accessible and environment friendly.
• Establishing SAM 2 requires a CUDA-enabled GPU and a fundamental understanding of Python and picture processing ideas.
• SAM 2’s capabilities open new prospects for each professionals and amateurs in content material creation, providing real-time segmentation and artistic management.
• The mannequin has the potential to remodel numerous industries, together with visible results, leisure, promoting, and training, by democratizing high-level modifying instruments.

Often Requested Questions

The media proven on this article just isn’t owned by Analytics Vidhya and is used on the Creator’s discretion.