GstAnalyticsSegmentationMtd
This type of metadata holds information on which pixels belongs to a region of the image representing a type of object.
It supports two types of segmentation, semantic or instance:
- Semantic: All objects of the same type have the same id
- Instance: Each instance of an object has a different id
The results of the segmentation are stored in a GstBuffer that has a GstVideoMeta associated with it. This buffer is stored in the GstAnalyticsSegmentationMtd using gst_analytics_relation_meta_add_segmentation_mtd(). The GstBuffer containing the segmentation mask is image-like but the color values are arbitrary values, referred by region-id in this API, without meaning beyond specifying that two pixels in the original image with the same values in their corresponding mask value belong to the same region.
To further describe a region, the GstAnalyticsSegmentationMtd can be associated with other GstAnalyticsMtd. Since region ids are generated by the segmentation process itself and are not always sequential, we use a map of indexes to region ids starting with 0 without discontinuity which facilitate N-to-N mapping with other GstAnalyticsMtd. For example it can be associated with GstAnalyticsClsMtd to describe the class of object matching the pixels of a segmented region.
Example: Associate Instance Segmentation with Classification
In the following example the segmentation process will fill segmask with values of 0 for background, 12 for the first region which correspond to a to a strawberry, 7 for the second region that also correspond to a strawberry in the image and 31 for the third region that correspond to a leaf in the image. region_ids is fill during segmentation post-processing
region_ids: |region-index | region-id |
| 0 | 0 | 
| 1 | 12 | 
| 2 | 7 | 
| 3 | 31 | 
region_count = 4
  GstAnalyticsSegmentationMtd segmtd;
  GstAnalyticsClassificationMtd clsmtd;
  GstBuffer *segmask, *img;
  guint *region_ids;
  gsize region_count, class_count;
  gfloat *class_confidence;
  GQuark *classes;
  ... (segmentation filling segmask based on img)
  gst_analytics_relation_meta_add_segmentation_mtd (rmeta, segmask,
    GST_SEGMENTATION_TYPE_INSTANCE, region_count, region_ids, &segmtd);
  class_count = region_count;
  ... (class-index must match and correspond to region-index)
  classes [0]  = g_quark_from_string ("background");
  classes [1]  = g_quark_from_string ("strawberry");
  classes [2] = g_quark_from_string ("strawberry");
  classes [3] = g_quark_from_string ("leaf");
  ... (set confidence level for each class associated with a region
  ... where -1.0 mean undefined.)
  class_confidence [0] = -1.0;
  class_confidence [1] = 0.6;
  class_confidence [2] = 0.9;
  class_confidence [3] = 0.8;
  gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count,
    class_confidence, classes, &clsmtd);
  gst_analytics_relation_meta_set_relation (rmeta,
   GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id);
Example: Associate Semantic Segmentation with Classification Assuming the same context as for Instance Segmentation above but instead a semantic segmentation is performed, therefore region-id-12 and region-id-7 are now represented by the same region-id-12
region_ids: (here |region-index | region-id |
| 0 | 0 | 
| 1 | 12 | 
| 2 | 31 | 
Code remain the same except that we set all confidence level to undefined (-1.0).
  ... (class-index must match and correspond to region-index)
  classes [0]  = g_quark_from_string ("background");
  classes [1]  = g_quark_from_string ("strawberry");
  classes [2] = g_quark_from_string ("leaf");
  ... (set confidence level for each class associated with a region
  ... where -1.0 mean undefined.)
  class_confidence [0] = -1.0;
  class_confidence [1] = -1.0;
  class_confidence [2] = -1.0;
  gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count,
    class_confidence, classes, &clsmtd);
  gst_analytics_relation_meta_set_relation (rmeta,
   GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id);
Example: Retrieving class associated with a segmentation region-id-12 This the typical case for an overlay as we visit the segmentation mask we we find region-id values
gsize idx;
gst_analytics_segmentation_mtd_get_region_index (&segmtd, &idx, 12);
gst_analytics_relation_meta_get_direct_related (rmeta, segmtd.id,
  GST_ANALYTICS_REL_TYPE_RELATE_TO, gst_analytics_cls_mtd_get_type (),
  NULL, &clsmtd);
 GQuark region_class = gst_analytics_cls_mtd_get_quark (&segmtd, idx)
 ...
GstAnalyticsSegmentationMtd
Handle containing data required to use gst_analytics_segmentation_mtd APIs. This type is generally expected to be allocated on the stack.
Members
id
		(guint)
		–
	Instance identifier
meta
		(GstAnalyticsRelationMeta *)
		–
	Instance of GstAnalyticsRelationMeta where the analytics-metadata identified by id is stored.
Since : 1.26
GstAnalytics.SegmentationMtd
Handle containing data required to use gst_analytics_segmentation_mtd APIs. This type is generally expected to be allocated on the stack.
Members
id
		(Number)
		–
	Instance identifier
meta
		(GstAnalytics.RelationMeta)
		–
	Instance of GstAnalytics.RelationMeta where the analytics-metadata identified by id is stored.
Since : 1.26
GstAnalytics.SegmentationMtd
Handle containing data required to use gst_analytics_segmentation_mtd APIs. This type is generally expected to be allocated on the stack.
Members
id
		(int)
		–
	Instance identifier
meta
		(GstAnalytics.RelationMeta)
		–
	Instance of GstAnalytics.RelationMeta where the analytics-metadata identified by id is stored.
Since : 1.26
Methods
gst_analytics_segmentation_mtd_get_mask
GstBuffer * gst_analytics_segmentation_mtd_get_mask (const GstAnalyticsSegmentationMtd * handle, gint * masks_loc_x, gint * masks_loc_y, guint * masks_loc_w, guint * masks_loc_h)
Get segmentation mask data.
Parameters:
handle
–
Instance
masks_loc_x
(
[out][nullable])
–
Left coordinate of the rectangle corresponding to the mask in the image.
masks_loc_y
(
[out][nullable])
–
Top coordinate of the rectangle corresponding to the mask in the image.
masks_loc_w
(
[out][nullable])
–
Width of the rectangle corresponding to the mask in the image.
masks_loc_h
(
[out][nullable])
–
Height of the rectangle corresponding to the mask in the image.
Segmentation mask data stored in a GstBuffer
Since : 1.26
GstAnalytics.SegmentationMtd.prototype.get_mask
function GstAnalytics.SegmentationMtd.prototype.get_mask(): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_mask'
}
Get segmentation mask data.
Parameters:
Instance
Returns a tuple made of:
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Since : 1.26
GstAnalytics.SegmentationMtd.get_mask
def GstAnalytics.SegmentationMtd.get_mask (self):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_mask'
Get segmentation mask data.
Parameters:
Instance
Returns a tuple made of:
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Segmentation mask data stored in a Gst.Buffer
Since : 1.26
gst_analytics_segmentation_mtd_get_region_count
gsize gst_analytics_segmentation_mtd_get_region_count (const GstAnalyticsSegmentationMtd * handle)
Get the regions count.
Parameters:
handle
–
Instance
Number of regions segmented
Since : 1.26
GstAnalytics.SegmentationMtd.prototype.get_region_count
function GstAnalytics.SegmentationMtd.prototype.get_region_count(): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_count'
}
Get the regions count.
Parameters:
Instance
Number of regions segmented
Since : 1.26
GstAnalytics.SegmentationMtd.get_region_count
def GstAnalytics.SegmentationMtd.get_region_count (self):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_region_count'
Get the regions count.
Parameters:
Instance
Number of regions segmented
Since : 1.26
gst_analytics_segmentation_mtd_get_region_id
guint gst_analytics_segmentation_mtd_get_region_id (const GstAnalyticsSegmentationMtd * handle, gsize index)
Get id of the region corresponding to index, which should be smaller than the return value of gst_analytics_segmentation_mtd_get_region_count
Parameters:
handle
–
Instance
index
–
Region index
The region ID
Since : 1.26
GstAnalytics.SegmentationMtd.prototype.get_region_id
function GstAnalytics.SegmentationMtd.prototype.get_region_id(index: Number): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_id'
}
Get id of the region corresponding to index, which should be smaller than the return value of GstAnalytics.SegmentationMtd.prototype.get_region_count
The region ID
Since : 1.26
GstAnalytics.SegmentationMtd.get_region_id
def GstAnalytics.SegmentationMtd.get_region_id (self, index):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_region_id'
Get id of the region corresponding to index, which should be smaller than the return value of GstAnalytics.SegmentationMtd.get_region_count
The region ID
Since : 1.26
gst_analytics_segmentation_mtd_get_region_index
gboolean gst_analytics_segmentation_mtd_get_region_index (const GstAnalyticsSegmentationMtd * handle, gsize * index, guint id)
Get region index of the region identified by id.
Parameters:
handle
–
Instance
index
(
[out][not nullable])
–
Region index
id
–
Region id
TRUE if a region with id exist, otherwise FALSE
Since : 1.26
GstAnalytics.SegmentationMtd.prototype.get_region_index
function GstAnalytics.SegmentationMtd.prototype.get_region_index(id: Number): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_index'
}
Get region index of the region identified by id.
Returns a tuple made of:
TRUE if a region with id exist, otherwise FALSE
TRUE if a region with id exist, otherwise FALSE
Since : 1.26
GstAnalytics.SegmentationMtd.get_region_index
def GstAnalytics.SegmentationMtd.get_region_index (self, id):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_region_index'
Get region index of the region identified by id.
Returns a tuple made of:
TRUE if a region with id exist, otherwise FALSE
TRUE if a region with id exist, otherwise FALSE
Since : 1.26
Functions
gst_analytics_segmentation_mtd_get_mtd_type
GstAnalyticsMtdType gst_analytics_segmentation_mtd_get_mtd_type ()
Get an instance of GstAnalyticsMtdType that represent segmentation metadata type.
A GstAnalyticsMtdType type
Since : 1.26
GstAnalytics.SegmentationMtd.prototype.get_mtd_type
function GstAnalytics.SegmentationMtd.prototype.get_mtd_type(): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_mtd_type'
}
Get an instance of Number that represent segmentation metadata type.
Since : 1.26
Enumerations
GstSegmentationType
Enum value describing supported segmentation type
Members
GST_SEGMENTATION_TYPE_SEMANTIC
		(0)
		–
	Segmentation where the belonging of each pixel to a class of objects is identified.
GST_SEGMENTATION_TYPE_INSTANCE
		(1)
		–
	Segmentation where the belonging of each pixel to instance of an object is identified.
Since : 1.26
GstAnalytics.SegmentationType
Enum value describing supported segmentation type
Members
GstAnalytics.SegmentationType.SEMANTIC
		(0)
		–
	Segmentation where the belonging of each pixel to a class of objects is identified.
GstAnalytics.SegmentationType.INSTANCE
		(1)
		–
	Segmentation where the belonging of each pixel to instance of an object is identified.
Since : 1.26
GstAnalytics.SegmentationType
Enum value describing supported segmentation type
Members
GstAnalytics.SegmentationType.SEMANTIC
		(0)
		–
	Segmentation where the belonging of each pixel to a class of objects is identified.
GstAnalytics.SegmentationType.INSTANCE
		(1)
		–
	Segmentation where the belonging of each pixel to instance of an object is identified.
Since : 1.26
The results of the search are