Search before asking
Description
First of all, thank you for the great work on the SAM3 integration. While experimenting with the new SAM3VideoSemanticPredictor, I found that the underlying tracker and temporal memory are actually much more flexible than the current public API suggests.
Currently, the predictor is designed to operate on videos. If the input is a list of images (numpy.ndarray), each image is treated independently and temporal memory is not preserved.
However, after studying the implementation and experimenting with the predictor internals, I found that the model itself does not require an actual video file. It only requires an object exposing the expected video dataset interface (frame, frames, mode, source_type, etc.).
This means that temporal tracking can already work perfectly on an arbitrary sequence of images without ever creating a video.
Motivation:
Many real-world applications naturally produce ordered image sequences rather than video files, for example:
- Industrial inspection
- Robotics
- Medical imaging
- Satellite imagery
- Scientific imaging
- Time-lapse acquisitions
- Multi-camera acquisition pipelines
In these scenarios, converting images into a temporary video introduces unnecessary complexity, additional I/O, and potentially unwanted compression.
Proposed API:
It would be extremely useful to expose a public API similar to:
predictor = SAM3VideoSemanticPredictor(...)
results = predictor.predict_frames(
frames, # list[np.ndarray]
text=["person","tree",]
)
where:
- frames is an ordered list of NumPy images.
- The internal tracker preserves temporal memory exactly as it does for video inference.
- No temporary video file is created.
- No additional encoding or decoding step is required.
- Why I believe this is feasible
While debugging the predictor, I found that the temporal memory is maintained through the existing inference_state and tracker_inference_states.
After initializing the predictor through setup_source(), replacing the dataset with a minimal video-like dataset exposing the required interface allows the tracker to propagate memory correctly across independent NumPy frames.
This suggests that the limitation is currently in the public API rather than in the model itself.
Benefits:
- Adding support for sequential image inference would provide:
- No temporary video creation
- No disk I/O
- No video encoding
- No compression artifacts
- Lower latency
- Better integration into image-processing pipelines
- Cleaner API for many industrial and scientific applications
I wanted to share this because I believe the capability is already almost entirely implemented internally. Exposing it through a dedicated public method (or supporting a sequence of NumPy images directly) could make the video predictor much more flexible without requiring major architectural changes.
Thank you for considering this feature.
sam3_frame_sequence_example.py
Use case
No response
Additional
No response
Are you willing to submit a PR?
Search before asking
Description
First of all, thank you for the great work on the SAM3 integration. While experimenting with the new SAM3VideoSemanticPredictor, I found that the underlying tracker and temporal memory are actually much more flexible than the current public API suggests.
Currently, the predictor is designed to operate on videos. If the input is a list of images (numpy.ndarray), each image is treated independently and temporal memory is not preserved.
However, after studying the implementation and experimenting with the predictor internals, I found that the model itself does not require an actual video file. It only requires an object exposing the expected video dataset interface (frame, frames, mode, source_type, etc.).
This means that temporal tracking can already work perfectly on an arbitrary sequence of images without ever creating a video.
Motivation:
Many real-world applications naturally produce ordered image sequences rather than video files, for example:
In these scenarios, converting images into a temporary video introduces unnecessary complexity, additional I/O, and potentially unwanted compression.
Proposed API:
It would be extremely useful to expose a public API similar to:
where:
While debugging the predictor, I found that the temporal memory is maintained through the existing inference_state and tracker_inference_states.
After initializing the predictor through setup_source(), replacing the dataset with a minimal video-like dataset exposing the required interface allows the tracker to propagate memory correctly across independent NumPy frames.
This suggests that the limitation is currently in the public API rather than in the model itself.
Benefits:
I wanted to share this because I believe the capability is already almost entirely implemented internally. Exposing it through a dedicated public method (or supporting a sequence of NumPy images directly) could make the video predictor much more flexible without requiring major architectural changes.
Thank you for considering this feature.
sam3_frame_sequence_example.py
Use case
No response
Additional
No response
Are you willing to submit a PR?