VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes

Name:

VISAT (Visual Attributes) Traffic Sign Recognition Dataset

Description:

We present VISAT, a novel open dataset and benchmarking suite for evaluating model robustness in the task of traffic sign recognition with the presence of visual attributes. Built upon the Mapillary Traffic Sign Dataset (MTSD), our dataset introduces two benchmarks that respectively emphasize robustness against adversarial attacks and distribution shifts. For our adversarial attack benchmark, we employ the state-of-the-art Projected Gradient Descent (PGD) method to generate adversarial inputs and evaluate their impact on popular models. Additionally, we investigate the effect of adversarial attacks on attribute-specific multi-task learning (MTL) networks, revealing spurious correlations among MTL tasks. The MTL networks leverage visual attributes (color, shape, symbol, and text) that we have created for each traffic sign in our dataset. For our distribution shift benchmark, we use ImageNet-C's realistic data corruption and natural variation techniques to perform evaluations on the robustness of both base and MTL models. Moreover, we further explore spurious correlations among MTL tasks through synthetic alterations of traffic sign colors using color quantization techniques. Our experiments focus on two major backbones, ResNet-152 and ViT-B/32, and compare the performance between base and MTL models. The VISAT dataset and benchmarking framework contribute to the understanding of model robustness for traffic sign recognition, shedding light on the challenges posed by adversarial attacks and distribution shifts.

Homepage:

http://rtsl-edge.cs.illinois.edu/visat/

Downloads:

http://rtsl-edge.cs.illinois.edu/visat/downloads/

Note: The VISAT Dataset Tools and VISAT Benchmark Tools codebases are currently released as direct downloads in the above link. In the future, however, we indeed plan to move the codebase release to a more permanent and version-control environment such as GitHub.

Tasks:

Traffic Sign Recognition

Splits:

Training (80%, 242263 instances), Validation (10%, 30283 instances), Testing (10%, 30283 instances)

Statement of Responsibility:

We, the authors of the VISAT dataset and benchmarks, bear all responsibility in case of violation of rights due to the development and release of the VISAT project.

Licensing:

The Mapillary Traffic Sign Dataset (MTSD), upon which our dataset is built, is released under the Creative Commons Attribution NonCommercial ShareAlike (CC BY-NC-SA) license. Therefore, due to the ShareAlike term of the license, we are required to release our VISAT dataset and benchmarks under the same license, i.e., the Creative Commons Attribution NonCommercial ShareAlike (CC BY-NC-SA) license, as that of the MTSD. A copy of the license for the VISAT dataset and benchmarks can be accessed at the VISAT website. Prior to accessing our dataset and benchmarks, all users must read, understand, and follow the terms of the license.

Maintenance:

The release of the VISAT dataset and benchmarks is hosted on one of our laboratory servers within the University of Illinois network. This website hosted by our server is entirely open and uncredentialized. The University of Illinois Technology Services will ensure the continuous functioning of our laboratory server hardware, while our project members will maintain all software on our server required by this website and its downloads page. Therefore, the release of the VISAT dataset and benchmarks shall remain available to the public indefinitely. If, for any reason, we elect to relocate the release of the VISAT dataset and benchmarks to other hosting websites or platforms, we will maintain public access to this website, where we would then indicate the location and access of the new hosting platform.

For all questions, please contact Simon at jundayu2@illinois.edu.

We present the labeling of all VISAT dataset visual attributes in the following table and here. There are in total 401 original classes as listed in the table below. There are in total 20 color attributes, 13 shape attributes, 216 symbol attributes, and 91 text attributes, as listed here.