Particularly important for safety critical applications is the possibility to provide uncertainties together with the models’ predictions. Most of the readers have probably put the most random objects inside their vehicles (e.g. furniture and washing machine, instruments, food and beverages, animals). It is hence paramount to somehow account for these random events a deployed system could be encountered with. Hence, we decided to create additional images for the vehicle interior which could be used to assess a model’s reliability. For each of the 3 seat positions in the vehicle interior rear bench the model should classify which object is occupying it, with empty being one possible choice. We created two training datasets for the Volkswagen Sharan vehicle, a new synthetic vehicle not used before, using adult passengers only (4384 sceneries and 8 classes) and one using adults, child seats and infant seats (3515 samples and 64 classes). We created fine-grained test sets to asses the reliability on several difficulty levels:
- only unseen adults (2617 sceneries),
- only unseen child and infant seats (490 sceneries),
- unseen adults and unseen child and infant seats (896 sceneries),
- unknown random everyday objects (\eg dog, plants, bags, washing machine, instruments, tv, skateboard, paintings – 1622 sceneries),
- unseen adults and unknown everyday objects (1421 sceneries),
- unseen adults, unseen child and infant seats and unknown everyday objects (1676 sceneries).
Unseen means that the test set uses new and different adults, child and infant seats not used during training. Besides the uncertainty estimation within the same vehicle interior, one can use images from unseen vehicle interiors from SVIRO (or any of the other extensions) to further test the models reliability on the same task, but in novel environments, i.e. vehicle interiors. One could also train on the SVIRO vehicles together with its everyday objects and check whether the model can generalize to the fine-grained test sets mentioned above. The ground truth labelling is the same as for SVIRO.
Example images with everyday objects are shown here:
Download all training and test splits
