d3d.benchmarks

This module contains benchmark utilities

class d3d.benchmarks.DetectionEvalStats

Bases: object

Detection stats summary of a evaluation step

acc_angular

‘unordered_map[int,vector[float]]’

Type: acc_angular

acc_box

‘unordered_map[int,vector[float]]’

Type: acc_box

acc_dist

‘unordered_map[int,vector[float]]’

Type: acc_dist

acc_iou

‘unordered_map[int,vector[float]]’

Type: acc_iou

acc_var

‘unordered_map[int,vector[float]]’

Type: acc_var

as_object(self)

fn

‘unordered_map[int,vector[int]]’

Type: fn

fp

‘unordered_map[int,vector[int]]’

Type: fp

ndt

‘unordered_map[int,vector[int]]’

Type: ndt

ngt

‘unordered_map[int,int]’

Type: ngt

tp

‘unordered_map[int,vector[int]]’

Type: tp

class d3d.benchmarks.DetectionEvaluator(classes, min_overlaps, int pr_sample_count=40, float min_score=0, unicode pr_sample_scale=u'log10')

Bases: object

Benchmark for object detection

Object detection benchmark. Targets association is done by score sorting.

Parameters

classes – Object classes to consider
min_overlaps – Min overlaps per class for two boxes being considered as overlap. If single value is provided, all class will use the same overlap threshold
min_score – Min score for precision-recall samples
pr_sample_count – Number of precision-recall sample points (expect for p=1,r=0 and p=0,r=1)
pr_sample_scale – PR sample type, {lin: linspace, log: logspace 1~10, logX: logspace 1~X}

acc_angular(self, float score=NAN)

acc_box(self, float score=NAN)

acc_dist(self, float score=NAN)

acc_iou(self, float score=NAN)

add_stats(self, DetectionEvalStats stats) → void: Add statistics from calc_stats into database

ap(self): Calculate (mean) average precision

calc_stats(self, Target3DArray gt_boxes, Target3DArray dt_boxes, TransformSet calib=None) → DetectionEvalStats

dt_count(self, float score=NAN)

fn(self, float score=NAN): Return false negative count. If score is not specified, return the median value

fp(self, float score=NAN): Return false positive count. If score is not specified, return the median value

fscore(self, float score=NAN, float beta=1, bool return_all=False)

get_stats(self) → DetectionEvalStats: Summarize current state of the benchmark counters

gt_count(self)

precision(self, float score=NAN, bool return_all=False)

recall(self, float score=NAN, bool return_all=False)

reset(self) → void

summary(self, float score_thres=0.8, bool verbose=False): Print default summary (into returned string)

tp(self, float score=NAN): Return true positive count. If score is not specified, return the median value

class d3d.benchmarks.SegmentationEvaluator(classes, background=0, min_points=0)

Bases: object

Benchmark for semgentation

Parameters

classes – classes to be considered during evaluation, other classes are all considered as background
background – class to be considered as background class
min_points – minimum number of points when calculating segments in panoptic evaluation

add_stats(self, SegmentationStats stats) → void

calc_stats(self, ndarray gt_labels, ndarray pred_labels, ndarray gt_ids=None, ndarray pred_ids=None) → SegmentationStats: Please make sure the id are 0 if the label is in stuff category

fn(self, bool instance=False)

fp(self, bool instance=False)

get_stats(self) → SegmentationStats: Summarize current state of the benchmark counters

iou(self, bool instance=False)

pq(self): Panoptic Quality (PQ) in panoptic segmentation

reset(self) → void

rq(self): Recognition Quality (RQ) in panoptic segmentation

sq(self): Segmentation Quality (SQ) in panoptic segmentation

summary(self)

tp(self, bool instance=False)

class d3d.benchmarks.SegmentationStats

Bases: object

Tracking stats summary of a data frame

as_object(self)

cumiou

‘unordered_map[uint8_t,float]’ Summation of IoU of TP segments in instance segmentation

Type: cumiou

fn

‘unordered_map[uint8_t,int]’ Number of false negative data points in semantic segmentation

Type: fn

fp

‘unordered_map[uint8_t,int]’ Number of false positive data points in semantic segmentation

Type: fp

ifn

‘unordered_map[uint8_t,int]’ Number of false negative data segments in instance segmentation

Type: ifn

ifp

‘unordered_map[uint8_t,int]’ Number of false positives data segments in instance segmentation

Type: ifp

itp

‘unordered_map[uint8_t,int]’ Number of true negative data segments in instance segmentation

Type: itp

tp

‘unordered_map[uint8_t,int]’ Number of true negative data points in semantic segmentation

Type: tp

class d3d.benchmarks.TrackingEvalStats

Bases: d3d.benchmarks.DetectionEvalStats

Tracking stats summary of a evaluation step

as_object(self)

fragments

‘unordered_map[int,vector[int]]’ Number of ground-truth trajectory matched to different tracked tracjetories

Type: fragments

id_switches

‘unordered_map[int,vector[int]]’ Number of tracked trajectory matched to different ground-truth trajectories

Type: id_switches

ndt_ids

‘unordered_map[int,vector[unordered_map[uint64_t,int]]]’ Frame count of all proposal targets (represented by their IDs)

Type: ndt_ids

ngt_ids

‘unordered_map[int,unordered_map[uint64_t,int]]’ Frame count of all ground-truth targets (represented by their IDs)

Type: ngt_ids

ngt_tracked

‘unordered_map[int,vector[unordered_map[uint64_t,int]]]’ Frame count of ground-truth targets being tracked

Type: ngt_tracked

class d3d.benchmarks.TrackingEvaluator(classes, min_overlaps, int pr_sample_count=40, float min_score=0, unicode pr_sample_scale=u'log10')

Bases: d3d.benchmarks.DetectionEvaluator

Benchmark for object tracking

Object tracking benchmark. Targets association is done by score sorting.

Parameters

classes – Object classes to consider
min_overlaps – Min overlaps per class for two boxes being considered as overlap. If single value is provided, all class will use the same overlap threshold
min_score – Min score for precision-recall samples
pr_sample_count – Number of precision-recall sample points (expect for p=1,r=0 and p=0,r=1)
pr_sample_scale – PR sample type, {lin: linspace, log: logspace 1~10, logX: logspace 1~X}

add_stats(self, DetectionEvalStats stats) → void

calc_stats(self, Target3DArray gt_boxes, Target3DArray dt_boxes, TransformSet calib=None) → TrackingEvalStats

fragments(self, float score=NAN): Return fragments count. If score is not specified, return the median value

get_stats(self) → TrackingEvalStats: Summarize current state of the benchmark counters

gt_traj_count(self): Return total ground-truth trajectory count. gt() will return total bounding box count

id_switches(self, float score=NAN): Return ID switch count. If score is not specified, return the median value

lost_ratio(self, float score=NAN, float frame_ratio_threshold=0.2, bool return_all=False)

Return the ratio of mostly lost trajectories.

Parameters: frame_ratio_threshold – The threshold of ratio of tracked frames over total frames. A trajectory with lower tracked frames ratio will be counted as mostly tracked

mota(self, float score=NAN): Return the MOTA metric defined by the CLEAR MOT metrics. For MOTP equivalents, see acc_* properties

reset(self) → void

summary(self, float score_thres=0.8, float tracked_ratio_thres=0.8, float lost_ratio_thres=0.2, unicode note=None, bool verbose=False): Print default summary (into returned string)

tracked_ratio(self, float score=NAN, float frame_ratio_threshold=0.8, bool return_all=False)

Return the ratio of mostly tracked trajectories.

Parameters: frame_ratio_threshold – The threshold of ratio of tracked frames over total frames. A trajectory with higher tracked frames ratio will be counted as mostly tracked