Added ObjectNav task definition, dataset, metrics, goal sensor (faceb…

…ookresearch#284) Added ObjectNav task definition, dataset, metrics, goal sensor. ObjectGoal Nav Task definition: metrics, goals, sensors with documentations. ObjectGoal Nav Dataset definition and loading and link for downloading. Topdown target objects, view points, bounding boxes visualization Compatible with Benchmark API Added ObjectNav DDPPO config for a baseline Made PointNavDataset general enough to be inherited by other Dataset types Refactored RLEnv to use measures for reward and success to speed up distance calculations Added mechanism for Measures (Metrics) classes to be depend on other measure. That increased speed and calculation and decrease copy pasta. Speed improvements 2x: from 8 FPS to 15.5 FPS, compare to 40 FPS for PointGoal with same setup. made SPL dependent on DistanceToGoal added metrics calc to reset method for SPL and DistanceToGoal
mfkiwl · Feb 7, 2020 · f5e29c6 · f5e29c6
1 parent 69d6a74
commit f5e29c6
Show file tree

Hide file tree

Showing 32 changed files with 928 additions and 133 deletions.
diff --git a/README.md b/README.md
@@ -159,6 +159,7 @@ Download the Habitat related Gibson dataset following the instructions [here](ht
 | --- | --- | --- | --- | --- | --- |
 | [Point goal navigation](https://arxiv.org/abs/1807.06757) | Gibson | [pointnav_gibson_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/pointnav/gibson/v1/pointnav_gibson_v1.zip) | `data/datasets/pointnav/gibson/v1/` |  [`datasets/pointnav/gibson.yaml`](configs/datasets/pointnav/gibson.yaml) | 385 MB |
 | [Point goal navigation](https://arxiv.org/abs/1807.06757) | MatterPort3D | [pointnav_mp3d_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/pointnav/mp3d/v1/pointnav_mp3d_v1.zip) | `data/datasets/pointnav/mp3d/v1/` | [`datasets/pointnav/mp3d.yaml`](configs/datasets/pointnav/mp3d.yaml) | 400 MB |
+| Object goal navigation | MatterPort3D | [objectnav_mp3d_v0.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/objectnav/m3d/v0/objectnav_mp3d_v0.zip) | `data/datasets/objectnav/mp3d/v0/` | [`datasets/objectnav/mp3d.yaml`](configs/datasets/objectnav/mp3d.yaml) | 4 GB |
 | [Embodied Question Answering](https://embodiedqa.org/) | MatterPort3D | [eqa_mp3d_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/eqa/mp3d/v1/eqa_mp3d_v1.zip) | `data/datasets/eqa/mp3d/v1/` | [`datasets/eqa/mp3d.yaml`](configs/datasets/eqa/mp3d.yaml) | 44 MB |
 | [Visual Language Navigation](https://bringmeaspoon.org/) | MatterPort3D | [vln_r2r_mp3d_v1.zip](https://dl.fbaipublicfiles.com/habitat/data/datasets/vln/mp3d/r2r/v1/vln_r2r_mp3d_v1.zip) | `data/datasets/vln/mp3d/r2r/v1` | [`datasets/vln/mp3d_r2r.yaml`](configs/datasets/vln/mp3d_r2r.yaml) | 2.7 MB |
 

diff --git a/configs/datasets/objectnav/mp3d.yaml b/configs/datasets/objectnav/mp3d.yaml
@@ -0,0 +1,4 @@
+DATASET:
+  TYPE: ObjectNav-v1
+  SPLIT: train
+  DATA_PATH: data/datasets/objectnav/mp3d/v0/{split}/{split}.json.gz
diff --git a/configs/tasks/obj_nav_mp3d.yaml b/configs/tasks/obj_nav_mp3d.yaml
@@ -0,0 +1,47 @@
+ENVIRONMENT:
+  MAX_EPISODE_STEPS: 500
+SIMULATOR:
+  TURN_ANGLE: 30
+  TILT_ANGLE: 30
+  AGENT_0:
+    SENSORS: ['RGB_SENSOR', 'DEPTH_SENSOR']
+    HEIGHT: 0.88
+    RADIUS: 0.2
+  HABITAT_SIM_V0:
+    GPU_DEVICE_ID: 0
+  SEMANTIC_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    POSITION: [0, 0.88, 0]
+  RGB_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    POSITION: [0, 0.88, 0]
+  DEPTH_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    MIN_DEPTH: 0.5
+    MAX_DEPTH: 5.0
+    POSITION: [0, 0.88, 0]
+TASK:
+  TYPE: ObjectNav-v1
+  SUCCESS_DISTANCE: 0.1
+
+  SENSORS: ['OBJECTGOAL_SENSOR']
+  GOAL_SENSOR_UUID: objectgoal
+
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
+  SPL:
+    TYPE: SPL
+    DISTANCE_TO: VIEW_POINTS
+    SUCCESS_DISTANCE: 0.2
+
+DATASET:
+  TYPE: ObjectNav-v1
+  SPLIT: val
+  CONTENT_SCENES: []
+  DATA_PATH: "data/datasets/objectnav/mp3d/v1/{split}/{split}.json.gz"
+  SCENES_DIR: "data/scene_datasets/"
diff --git a/configs/tasks/objectnav_mp3d.yaml b/configs/tasks/objectnav_mp3d.yaml
@@ -0,0 +1,51 @@
+ENVIRONMENT:
+  MAX_EPISODE_STEPS: 500
+SIMULATOR:
+  TURN_ANGLE: 30
+  TILT_ANGLE: 30
+  ACTION_SPACE_CONFIG: "v1"
+  AGENT_0:
+    SENSORS: ['RGB_SENSOR', 'DEPTH_SENSOR']
+    HEIGHT: 0.88
+    RADIUS: 0.2
+  HABITAT_SIM_V0:
+    GPU_DEVICE_ID: 0
+  SEMANTIC_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    POSITION: [0, 0.88, 0]
+  RGB_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    POSITION: [0, 0.88, 0]
+  DEPTH_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    MIN_DEPTH: 0.5
+    MAX_DEPTH: 5.0
+    POSITION: [0, 0.88, 0]
+TASK:
+  TYPE: ObjectNav-v1
+  POSSIBLE_ACTIONS: ["STOP", "MOVE_FORWARD", "TURN_LEFT", "TURN_RIGHT", "LOOK_UP", "LOOK_DOWN"]
+  SUCCESS_DISTANCE: 0.1
+
+  SENSORS: ['OBJECTGOAL_SENSOR', 'COMPASS_SENSOR', 'GPS_SENSOR']
+  GOAL_SENSOR_UUID: objectgoal
+
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
+  SPL:
+    TYPE: SPL
+    DISTANCE_TO: VIEW_POINTS
+    SUCCESS_DISTANCE: 0.2
+  DISTANCE_TO_GOAL:
+    DISTANCE_TO: VIEW_POINTS
+
+DATASET:
+  TYPE: ObjectNav-v1
+  SPLIT: val
+  CONTENT_SCENES: []
+  DATA_PATH: "data/datasets/objectnav/mp3d/v0/{split}/{split}.json.gz"
+  SCENES_DIR: "data/scene_datasets/"
diff --git a/configs/tasks/pointnav.yaml b/configs/tasks/pointnav.yaml
@@ -21,7 +21,7 @@ TASK:
     DIMENSIONALITY: 2
   GOAL_SENSOR_UUID: pointgoal_with_gps_compass
 
-  MEASUREMENTS: ['SPL']
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
   SPL:
     TYPE: SPL
     SUCCESS_DISTANCE: 0.2
diff --git a/configs/tasks/pointnav_gibson.yaml b/configs/tasks/pointnav_gibson.yaml
@@ -21,7 +21,7 @@ TASK:
     DIMENSIONALITY: 2
   GOAL_SENSOR_UUID: pointgoal_with_gps_compass
 
-  MEASUREMENTS: ['SPL']
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
   SPL:
     TYPE: SPL
     SUCCESS_DISTANCE: 0.2

diff --git a/configs/tasks/pointnav_mp3d.yaml b/configs/tasks/pointnav_mp3d.yaml
@@ -21,7 +21,7 @@ TASK:
     DIMENSIONALITY: 2
   GOAL_SENSOR_UUID: pointgoal_with_gps_compass
 
-  MEASUREMENTS: ['SPL']
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
   SPL:
     TYPE: SPL
     SUCCESS_DISTANCE: 0.2

diff --git a/configs/tasks/pointnav_rgbd.yaml b/configs/tasks/pointnav_rgbd.yaml
@@ -21,7 +21,7 @@ TASK:
     DIMENSIONALITY: 2
   GOAL_SENSOR_UUID: pointgoal_with_gps_compass
 
-  MEASUREMENTS: ['SPL']
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
   SPL:
     TYPE: SPL
     SUCCESS_DISTANCE: 0.2
diff --git a/configs/tasks/vln_r2r.yaml b/configs/tasks/vln_r2r.yaml
@@ -21,7 +21,7 @@ TASK:
   SENSORS: ['INSTRUCTION_SENSOR']
   INSTRUCTION_SENSOR_UUID: instruction
   POSSIBLE_ACTIONS: ['STOP', 'MOVE_FORWARD', 'TURN_LEFT', 'TURN_RIGHT']
-  MEASUREMENTS: ['SPL']
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
   SPL:
     TYPE: SPL
     SUCCESS_DISTANCE: 3.0

diff --git a/configs/test/habitat_all_sensors_test.yaml b/configs/test/habitat_all_sensors_test.yaml
@@ -25,7 +25,7 @@ TASK:
     DIMENSIONALITY: 2
   GOAL_SENSOR_UUID: pointgoal_with_gps_compass
 
-  MEASUREMENTS: ['SPL']
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
   SPL:
     TYPE: SPL
     SUCCESS_DISTANCE: 0.2
diff --git a/configs/test/habitat_mp3d_object_nav_test.yaml b/configs/test/habitat_mp3d_object_nav_test.yaml
@@ -0,0 +1,51 @@
+ENVIRONMENT:
+  MAX_EPISODE_STEPS: 500
+SIMULATOR:
+  TURN_ANGLE: 30
+  TILT_ANGLE: 30
+  ACTION_SPACE_CONFIG: "v1"
+  AGENT_0:
+    SENSORS: ['RGB_SENSOR', 'DEPTH_SENSOR']
+    HEIGHT: 0.88
+    RADIUS: 0.2
+  HABITAT_SIM_V0:
+    GPU_DEVICE_ID: 0
+  SEMANTIC_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    POSITION: [0, 0.88, 0]
+  RGB_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    POSITION: [0, 0.88, 0]
+  DEPTH_SENSOR:
+    WIDTH: 640
+    HEIGHT: 480
+    HFOV: 79
+    MIN_DEPTH: 0.5
+    MAX_DEPTH: 5.0
+    POSITION: [0, 0.88, 0]
+TASK:
+  TYPE: ObjectNav-v1
+  POSSIBLE_ACTIONS: ["STOP", "MOVE_FORWARD", "TURN_LEFT", "TURN_RIGHT", "LOOK_UP", "LOOK_DOWN"]
+  SUCCESS_DISTANCE: 0.1
+
+  SENSORS: ['OBJECTGOAL_SENSOR', 'COMPASS_SENSOR', 'GPS_SENSOR']
+  GOAL_SENSOR_UUID: objectgoal
+
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
+  SPL:
+    TYPE: SPL
+    DISTANCE_TO: VIEW_POINTS
+    SUCCESS_DISTANCE: 0.2
+  DISTANCE_TO_GOAL:
+    DISTANCE_TO: VIEW_POINTS
+
+DATASET:
+  TYPE: ObjectNav-v1
+  SPLIT: mini_val
+  CONTENT_SCENES: []
+  DATA_PATH: "data/datasets/objectnav/mp3d/v0/{split}/{split}.json.gz"
+  SCENES_DIR: "data/scene_datasets/"
diff --git a/configs/test/habitat_r2r_vln_test.yaml b/configs/test/habitat_r2r_vln_test.yaml
@@ -21,7 +21,7 @@ TASK:
     DIMENSIONALITY: 2
   GOAL_SENSOR_UUID: pointgoal_with_gps_compass
   INSTRUCTION_SENSOR_UUID: instruction
-  MEASUREMENTS: ['SPL']
+  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SPL']
   SPL:
     TYPE: SPL
     SUCCESS_DISTANCE: 3.0

diff --git a/examples/shortest_path_follower_example.py b/examples/shortest_path_follower_example.py
@@ -95,7 +95,7 @@ def shortest_path_example(mode):
             observations, reward, done, info = env.step(best_action)
             im = observations["rgb"]
             top_down_map = draw_top_down_map(
-                info, observations["heading"], im.shape[0]
+                info, observations["heading"][0], im.shape[0]
             )
             output_im = np.concatenate((im, top_down_map), axis=1)
             images.append(output_im)

diff --git a/habitat/config/default.py b/habitat/config/default.py
@@ -94,6 +94,13 @@ def __init__(self, *args, **kwargs):
     "PointGoalWithGPSCompassSensor"
 )
 # -----------------------------------------------------------------------------
+# OBJECTGOAL SENSOR
+# -----------------------------------------------------------------------------
+_C.TASK.OBJECTGOAL_SENSOR = CN()
+_C.TASK.OBJECTGOAL_SENSOR.TYPE = "ObjectGoalSensor"
+_C.TASK.OBJECTGOAL_SENSOR.GOAL_SPEC = "TASK_CATEGORY_ID"
+_C.TASK.OBJECTGOAL_SENSOR.GOAL_SPEC_MAX_VAL = 50
+# -----------------------------------------------------------------------------
 # HEADING SENSOR
 # -----------------------------------------------------------------------------
 _C.TASK.HEADING_SENSOR = CN()
@@ -130,13 +137,17 @@ def __init__(self, *args, **kwargs):
 _C.TASK.TOP_DOWN_MAP.MAP_PADDING = 3
 _C.TASK.TOP_DOWN_MAP.NUM_TOPDOWN_MAP_SAMPLE_POINTS = 20000
 _C.TASK.TOP_DOWN_MAP.MAP_RESOLUTION = 1250
-_C.TASK.TOP_DOWN_MAP.DRAW_SOURCE_AND_TARGET = True
+_C.TASK.TOP_DOWN_MAP.DRAW_SOURCE = True
 _C.TASK.TOP_DOWN_MAP.DRAW_BORDER = True
 _C.TASK.TOP_DOWN_MAP.DRAW_SHORTEST_PATH = True
 _C.TASK.TOP_DOWN_MAP.FOG_OF_WAR = CN()
 _C.TASK.TOP_DOWN_MAP.FOG_OF_WAR.DRAW = True
 _C.TASK.TOP_DOWN_MAP.FOG_OF_WAR.VISIBILITY_DIST = 5.0
 _C.TASK.TOP_DOWN_MAP.FOG_OF_WAR.FOV = 90
+_C.TASK.TOP_DOWN_MAP.DRAW_VIEW_POINTS = True
+_C.TASK.TOP_DOWN_MAP.DRAW_GOAL_POSITIONS = True
+# Axes aligned bounding boxes
+_C.TASK.TOP_DOWN_MAP.DRAW_GOAL_AABBS = True
 # -----------------------------------------------------------------------------
 # COLLISIONS MEASUREMENT
 # -----------------------------------------------------------------------------
@@ -173,6 +184,7 @@ def __init__(self, *args, **kwargs):
 # -----------------------------------------------------------------------------
 _C.TASK.DISTANCE_TO_GOAL = CN()
 _C.TASK.DISTANCE_TO_GOAL.TYPE = "DistanceToGoal"
+_C.TASK.DISTANCE_TO_GOAL.DISTANCE_TO = "POINT"
 # -----------------------------------------------------------------------------
 # # ANSWER_ACCURACY MEASUREMENT
 # -----------------------------------------------------------------------------
@@ -210,8 +222,8 @@ def __init__(self, *args, **kwargs):
 # -----------------------------------------------------------------------------
 _C.SIMULATOR.DEPTH_SENSOR = SIMULATOR_SENSOR.clone()
 _C.SIMULATOR.DEPTH_SENSOR.TYPE = "HabitatSimDepthSensor"
-_C.SIMULATOR.DEPTH_SENSOR.MIN_DEPTH = 0
-_C.SIMULATOR.DEPTH_SENSOR.MAX_DEPTH = 10
+_C.SIMULATOR.DEPTH_SENSOR.MIN_DEPTH = 0.0
+_C.SIMULATOR.DEPTH_SENSOR.MAX_DEPTH = 10.0
 _C.SIMULATOR.DEPTH_SENSOR.NORMALIZE_DEPTH = True
 # -----------------------------------------------------------------------------
 # SEMANTIC SENSOR

diff --git a/habitat/core/embodied_task.py b/habitat/core/embodied_task.py
@@ -8,7 +8,7 @@
 """
 
 from collections import OrderedDict
-from typing import Any, Dict, Iterable, Optional, Type, Union
+from typing import Any, Dict, Iterable, List, Optional, Type, Union
 
 import numpy as np
 
@@ -175,6 +175,31 @@ def get_metrics(self) -> Metrics:
         """
         return Metrics(self.measures)
 
+    def _get_measure_index(self, measure_name):
+        return list(self.measures.keys()).index(measure_name)
+
+    def check_measure_dependencies(
+        self, measure_name: str, dependencies: List[str]
+    ):
+        r"""Checks if dependencies measures are enabled and calculatethat the measure
+        :param measure_name: a name of the measure for which has dependencies.
+        :param dependencies: a list of a measure names that are required by
+        the measure.
+        :return:
+        """
+        measure_index = self._get_measure_index(measure_name)
+        for dependency_measure in dependencies:
+            assert (
+                dependency_measure in self.measures
+            ), f"""{measure_name} measure requires {dependency_measure}
+                listed in tje measures list in the config."""
+
+        for dependency_measure in dependencies:
+            assert measure_index > self._get_measure_index(
+                dependency_measure
+            ), f"""{measure_name} measure requires be listed after {dependency_measure}
+                in tje measures list in the config."""
+
 
 class EmbodiedTask:
     r"""Base class for embodied tasks. ``EmbodiedTask`` holds definition of