Metrics module
The metrics module relies on the CARLA recorder to facilitate the easy calculus and monitoring of any kind of parameter. After the scenario is finished, the recorder stores the simulation information. Users can then define their own metrics, and check the corresponding results with no need to play the simulation again and again.
This section covers an explanation of the different elements that form the module, a step by step tutorial of how to create a new metric, and a reference of the functions that query the recording and define the metrics.
Structure of the module
Similarly to the main ScenarioRunner module, the Metrics module is run using a main script, metrics_manager.py
, and the rest of the information is contained inside a folder structure. Here is a brief introduction to the main script.
metrics_manager.py
— The main script of the module. Run this to show the results of the set of metrics. The script has the usualhost
andport
arguments, and some more to set the metrics and recording to be used.host
(string) – IP address where a CARLA simulation is running. Default is(127.0.0.1)
.port
(int) – TCP port where the CARLA simulation is running. Default are2000
and2001
.metrics
— Path to the metrics to be used.log
— Path to the.log
file containing the recording (relative to the environment variableSCENARIO_RUNNER_ROOT
).criteria
(optional) — Path to a JSON file with the criteria of the scenario.
The rest of the elements that shape the module can be found in the srunner/metrics
folder. These folder has been divided in three subfolders.
-
srunner/metrics/data
— Stores information about the scenarios. By default, it has six files, which are part of the examples metric. -
srunner/metrics/examples
— Contains some example metrics, and the metric containing the base classBaseMetric
to create new metrics.basic_metric.py
– Contains the base classBaseMetric
. All the metrics are inherited from it.criteria_filter.py
– Returns a JSON with the most important attributes of the criteria.distance_between_vehicles.py
– Returns the distance betwen two vehicles. Useful to show how to query the recording.distance_to_lane_center.py
– Calculates the distance between the vehicle location and the center of the lane. Useful to show how to access the map API information..
-
srunner/metrics/tools
— Contains two key scripts that allow to query the recording.metrics_parser.py
– Transforms the string provided by the recording to a dictionary.metrics_log.py
– Provides with several functions to query the dictionary created withmetrics_parser.py
. These functions are the easiest way to access information of a scenario. They listed in a reference in the last segment of this page.
How to use the metrics module
1. Record a scenario
The metrics module needs for a recording of a simulation in order to work. Otherwise, it has no data to make the calculations of the metrics.
Use the record
argument. Add the path where the information should be saved should be saved. The path must be relative to the environment variable SCENARIO_RUNNER_ROOT
.
python scenario_runner.py --scenario <scenario_name> --record <path/to/save/the/recorder/file>
By recording the scenario two files will be created in the desired path. These files will later be used as the log
and criteria
arguments in the metrics_manager.py
.
1. A CARLA recording (.log
) — Contains simulation data per frame. To know more about this, read the recorder docs.
2. A criteria file (.json) — The criteria of the scenario parsed as a dictionary, and stored in a JSON file. The keys of the dictionary are the names of the criteria. The values are the attributes of each criteria.
Note
Only the JSON serializable attributes will be parsed, the rest will be ignored.
By default, both files are named after the scenario that is run. If the scenario is named example.py
, the files will be example.log
and example.json
.
2. Define the metrics
It is time to create the desired metrics that will be used for the scenario. The possibilities are endless, but for the sake of this tutorial, the metric described in srunner/metrics/examples/distance_between_vehicles.py
will be used as an example. Let's dig a little bit into the code itself.
class DistanceBetweenVehicles(BasicMetric):
"""
Metric class DistanceBetweenVehicles
"""
def _create_metric(self, town_map, log, criteria):
"""
Implementation of the metric. This is an example to show how to use the recorder,
accessed via the log.
"""
# Get the ID of the two vehicles
ego_id = log.get_ego_vehicle_id()
adv_id = log.get_actor_ids_with_role_name("scenario")[0] # Could have also used its type_id
dist_list = []
frames_list = []
# Get the frames both actors were alive
start_ego, end_ego = log.get_actor_alive_frames(ego_id)
start_adv, end_adv = log.get_actor_alive_frames(adv_id)
start = max(start_ego, start_adv)
end = min(end_ego, end_adv)
# Get the distance between the two
for i in range(start, end):
# Get the transforms
ego_location = log.get_actor_transform(ego_id, i).location
adv_location = log.get_actor_transform(adv_id, i).location
# Filter some points for a better graph
if adv_location.z < -10:
continue
dist_v = ego_location - adv_location
dist = math.sqrt(dist_v.x * dist_v.x + dist_v.y * dist_v.y + dist_v.z * dist_v.z)
dist_list.append(dist)
frames_list.append(i)
# Use matplotlib to show the results
plt.plot(frames_list, dist_list)
plt.ylabel('Distance [m]')
plt.xlabel('Frame number')
plt.title('Distance between the ego vehicle and the adversary over time')
plt.show()
2.1. Name the metric. First of all, as it was previously mentioned, all metrics are childs of the BasicMetric
class.
class DistanceBetweenVehicles(BasicMetric):
2.2. Name the main method. A metric only requires one method in order to run, _create_metric()
, which has three arguments.
town_map
— Instance of the carla.Map() where the scenario took place in.log
— Instance to theMetricsLog
class, with all the functions needed to access the recorder dictionary.criteria
— A JSOn with the criteria dictionary. The file provided when recording a scenario that is later provided to themetrics_manager.py
.
def _create_metric(self, town_map, log, criteria):
2.3. Implement the metric. The code will vary depending on the metric itself.
The example metric DistanceBetweenVehicles
calculates the distance between the ego vehicle, and the car it follows (adversary). To do so, it needs the id
of the two vehicles. These can be retrieved from the log using the get_ego_vehicle_id()
and get_actor_ids_with_role_name("scenario")[0]
functions.
# Get the ID of the two vehicles
ego_id = log.get_ego_vehicle_id()
adv_id = log.get_actor_ids_with_role_name("scenario")[0] # Could have also used its type_id
The id
are used to retrieve the frames where the vehicles were alive using get_actor_alive_frames(actor_id)
.
# Get the frames both actors were alive
start_ego, end_ego = log.get_actor_alive_frames(ego_id)
start_adv, end_adv = log.get_actor_alive_frames(adv_id)
start = max(start_ego, start_adv)
end = min(end_ego, end_adv)
Now everything is ready to loop through those frames, get their transforms, and calculate the distance. To get the transform, get_actor_transform(actor_id, frame)
is used.
dist_list = []
frames_list = []
...
# Get the distance between the two
for i in range(start, end):
# Get the transforms
ego_location = log.get_actor_transform(ego_id, i).location
adv_location = log.get_actor_transform(adv_id, i).location
# Filter some points for a better graph
if adv_location.z < -10:
continue
dist_v = ego_location - adv_location
dist = math.sqrt(dist_v.x * dist_v.x + dist_v.y * dist_v.y + dist_v.z * dist_v.z)
dist_list.append(dist)
Note
The vertical condition of the adversary is to only take into account the adversary when it is driving normally.
Lastly, use matplotlib to define which should be the output of the metric when running metrics_manager.py
.
# Use matplotlib to show the results
plt.plot(frames_list, dist_list)
plt.ylabel('Distance [m]')
plt.xlabel('Frame number')
plt.title('Distance between the ego vehicle and the adversary over time')
plt.show()
3. Run the metrics manager
Finally it is time to used the information retrieve from the scenario, and the metrics that have been defined to return some metrics information. Let's run the module using the example metric that has been used so far, and an example log named after it.
python metrics_manager.py --metric srunner/metrics/examples/distance_between_vehicles.py --log srunner/metrics/data/DistanceBetweenVehicles.log
Warning
A simulation must be running. Otherwise, the module will not be able to acces the map API.
This will create a new window with the results plotted. The script will not finish until the ouput window is closed.
Recording queries reference
When defining a metric, all the information about the scenario is accessed via the MetricsLog
class (the log
argument at the _create_metric()
function). This class is located at srunner/metrics/tools/metrics_log.py
, and this reference is a following of the functions contained in it.
Generic actor data
-
get_ego_vehicle_id(self)
Returns theid
of the ego vehicle.- Return — int
-
get_actor_ids_with_role_name(self, role_name)
Returns a list of actorid
that match the givenrole_name
.- Return — list
- Parameters
role_name
(str) —role_name
of the actor.
-
get_actor_ids_with_type_id(self, type_id)
Returns a list of actorid
that match the giventype_id
, according to fnmatch standard.- Return — list
- Parameters
type_id
(str) —type_id
of the actor.
-
get_actor_attributes(self, actor_id)
Returns a dictionary with all the attributes of an actor.- Return — dict
- Parameters
actor_id
(int) —id
of the actor.
-
get_actor_bounding_box(self, actor_id)
Returns the bounding box of the specified actor.- Return — carla.BoundingBox
- Parameters
actor_id
(int) —id
of the actor.
-
get_traffic_light_trigger_volume(self, traffic_light_id)
Returns the trigger volume of the specified traffic light.- Return — carla.BoundingBox
- Parameters
traffic_light_id
(int) —id
of the traffic light.
-
get_actor_alive_frames(self, actor_id)
Returns a tuple with the first and last frame an actor was alive. Note that frames start at 1, not 0.- Return — tuple
- Parameters
actor_id
(int) —id
of the actor.
Generic simulation data
-
get_collisions(self,actor_id)
Returns a list of dictionaries with two keys.frame
is the frame number of the collision, andother_id
, a list of the ids the actor collided with at that frame.- Return — list
- Parameters
actor_id
(int) —id
of the actor.
-
get_total_frame_count(self)
Returns an int with the total amount of frames the simulation lasted.- Return — int
-
get_elapsed_time(self, frame)
Returns a float with the elapsed time of a specific frame.- Return — float
- Parameters
frame
(int) — Frame number.
-
get_delta_time(self, frame)
Returns an float with the delta time of a specific frame.- Return — float
- Parameters
frame
(int) — Frame number.
-
get_platform_time(self, frame)
Returns a float with the platform time of a specific frame.- Return — float
- Parameters
frame
(int) — Frame number.
Actor accelerations
-
get_actor_acceleration(self, actor_id, frame)
Returns the acceleration of the actor at a given frame. Returns None if the actorid
doesn't exist, the actor has no acceleration, or the actor wasn't alive at that frame.- Return — carla.Vector3D
- Parameters
actor_id
(int) —id
of the actor.frame
(int) — Frame number.
-
get_all_actor_accelerations(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the accelerations of the actor at the frame interval. By default, the frame interval comprises all the recording.- Return — list(carla.Vector3D)
- Parameters
actor_id
(int) —id
of the actor.first_frame
(int) — Initial frame of the interval. By default, the start of the simulation.last_frame
(int) — Last frame of the interval. By default, the end of the simulation.
-
get_actor_accelerations_at_frame(self, frame, actor_list=None)
Returns a dict where the keys are the frame number, and the values are the carla.Vector3D of the actor at the given frame. By default, all actors are considered but if actor_list is passed, only actors in the list will be checked.- Return — list(carla.Vector3D)
- Parameters
frame
(int) — Frame number.actor_list
(int) — List of actorid
.
Actor angular velocities
-
get_actor_angular_velocity(self, actor_id, frame)
Returns the angular velocity of the actor at a given frame. Returns None if the actor id doesn't exist, the actor has no angular velocity, or the actor wasn't alive at that frame.- Return — carla.Vector3D
- Parameters
actor_id
(int) —id
of the actor.frame
(int) — Frame number.
-
get_all_actor_angular_velocities(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the angular velocities of the actor at the frame interval. By default, the frame interval comprises all the recording.- Return — list(carla.Vector3D)
- Parameters
actor_id
(int) —id
of the actor.first_frame
(int) — Initial frame of the interval. By default, the start of the simulation.last_frame
(int) — Last frame of the interval. By default, the end of the simulation.
-
get_actor_angular_velocities_at_frame(self, frame, actor_list=None)
Returns a dictionary where the keys are the frame number, and the values are the carla.Vector3D of the actor at the given frame. By default, all actors are considered but ifactor_list
is passed, only actors in the list will be checked.- Return — list(carla.Vector3D)
- Parameters
frame
(int) — frame number.actor_list
(int) — List of actor ids.
Actor controls
-
get_vehicle_control(self, vehicle_id, frame)
Returns the control of a vehicle at a given frame. Themanual_gear_shift
attribute will always be False.- Return — carla.VehicleCotnrol
- Parameters
vehicle_id
(int) —id
of the vehicle.frame
(int) — Frame number.
-
get_vehicle_physics_control(self, vehicle_id, frame)
Returns the physics control of a vehicle at a given frame.- Return — carla.VehiclePhysicsControl
- Parameters
vehicle_id
(int) —id
of the vehicle.frame
(int) — Frame number.
-
get_walker_speed(self, walker_id, frame)
Returns the speed of a walker at a given frame.- Return — carla.Vector3D
- Parameters
walker_id
(int) —id
of the walker.frame
(int) — Frame number.
Actor transforms
-
get_actor_transform(self, actor_id, frame)
Returns the transform of the actor at a given frame. Returns None if the actor id doesn't exist, the actor has no transform, or the actor wasn't alive at that frame.- Return — carla.Transform
- Parameters
actor_id
(int) —id
of the actor.frame
(int) — Frame number.
-
get_all_actor_transforms(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the transforms of the actor at the frame interval. By default, the frame interval comprises all the recording.- Return — list(carla.Transform)
- Parameters
actor_id
(int) —id
of the actor.first_frame
(int) — Initial frame of the interval. By default, the start of the simulation.last_frame
(int) — Last frame of the interval. By default, the end of the simulation.
-
get_actor_transforms_at_frame(self, frame, actor_list=None)
Returns a dictionary where the keys are the frame number, and the values are the carla.Transform of the actor at the given frame. By default, all actors are considered but ifactor_list
is passed, only actors in the list will be checked.- Return — list(carla.Transform)
- Parameters
frame
(int) — Frame number.actor_list
(int) — List of actorid
.
Actor velocities
-
get_actor_velocity(self, actor_id, frame)
Returns the velocity of the actor at a given frame. Returns None if the actor id doesn't exist, the actor has no velocity, or the actor wasn't alive at that frame.- Return — carla.Vector3D
- Parameters
actor_id
(int) —id
of the actor.frame
(int) — Frame number.
-
get_all_actor_velocities(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the velocities of the actor at the frame interval. By default, the frame interval comprises all the recording.- Return — list(carla.Vector3D)
- Parameters
actor_id
(int) —id
of the actor.first_frame
(int) — Initial frame of the interval. By default, the start of the simulation.last_frame
(int) — Last frame of the interval. By default, the end of the simulation.
-
get_actor_velocities_at_frame(self, frame, actor_list=None)
Returns a dict where the keys are the frame number, and the values are the carla.Vector3D of the actor at the given frame. By default, all actors are considered but if actor_list is passed, only actors in the list will be checked.- Return — list(carla.Vector3D
- Parameters
frame
(int) — Frame number.actor_list
(int) — List of actorid
.
Scene lights
- get_scene_light_state(self, light, vehicle_id, frame)
Returns the state of a scene light for a given frame. The light state group will always be carla.LightGroup.None.- Return — carla.LightState
- Parameters
light_id
(int) —id
of the scene light.frame
(int) — Frame number.
Traffic lights
-
get_traffic_light_state(self, traffic_light_id, frame)
Returns the state of a traffic light at a given frame.- Return — carla.TrafficLightState.
- Parameters
traffic_light_id
(int) —id
of the traffic light.frame
(int) — Frame number.
-
is_traffic_light_frozen(self, traffic_light_id, frame)
Returns whether or not a traffic light is frozen at a given frame.- Return — bool
- Parameters
traffic_light_id
(int) —id
of the traffic light.frame
(int) — Frame number.
-
get_traffic_light_elapsed_time(self, traffic_light_id, frame)
Returns the elapsed time of a traffic light at a given frame.- Return — float
- Parameters
traffic_light_id
(int) —id
of the traffic light.frame
(int) — Frame number.
-
get_traffic_light_state_time(self, traffic_light_id, state, frame)
Returns the maximum time of a specific state of a traffic light at a given frame.- Return — float
- Parameters
traffic_light_id
(int) —id
of the traffic light.state
(carla.TrafficLightState) —id
of the actor.frame
(int) — Frame number.
Vehicle lights
-
get_vehicle_lights(self, vehicle_id, frame)
Returns a list with the active lights of a specific vehicle at a given frame.- Return — list(carla.VehicleLightState)
- Parameters
vehicle_id
(int) —id
of the vehicle.frame
(int) — Frame number.
-
is_vehicle_light_active(self, light, vehicle_id, frame)
Checks whether or not a given vehicle light is active for a vehicle at a specific frame.- Return — bool
- Parameters
light
(carla.VehicleLightState) — The light state to be compared with the vehicle.vehicle_id
(int) —id
of the vehicle.frame
(int) — Frame number.