When people witness a harmful event, they can judge who is responsible and to what degree. Such judgments depend on multiple factors, including what happened, whether the agent could have prevented the outcome, and whether they were aware that their behavior could cause harm. This information is either observed or inferred from perceptual inputs, often based on details that are present in naturalistic videos but are difficult to convey in word descriptions (e.g. if a pedestrian crossing the road was hit by a car, how suddenly did the pedestrian start crossing?). This study investigates how humans generate rich representations of responsibility from visual input using a controlled set of naturalistic traffic videos sampled to achieve wide variation along relevant dimensions. From these videos, we extract a variety of event features describing entities and their relations, studying the extent to which different features contribute to responsibility judgments in realistic settings.