Heracles

Setup

Pull the Neo4j database image:

docker pull neo4j:5.25.1

Run the database:

docker run -d \
   --restart always \
   --publish=7474:7474 --publish=7687:7687 \
   --env=NEO4J_AUTH=neo4j/neo4j_pw \
   neo4j:5.25.1

Clone and install spark_dsg:

sudo apt install libzmqpp-dev nlohmann-json3-dev ninja-build
git clone [email protected]:MIT-SPARK/Spark-DSG.git
cd Spark-DSG
pip install .

Clone and install this repo:

[email protected]:npolshakova/heracles.git
cd heracles
pip install .

If you run into installation problems, check .github/workflows/ci-action.yml for an example, as that successfully installs and runs the library for CI.

If you don't have a scene graph to test with, you can use a large example scene graph here. This scene graph has 2D places and objects, and 3D places (that don't make much sense), but no regions or rooms.

The script examples/dsg_test.py is a small example of loading an existing scene graph file into a graph database and running some simple queries. You can run it in interaction mode (ipython3 -i heracles/examples/dsg_test.py) and then try executing some of the other example queries below.

Useful Queries

Print distinct object classes in scene graph

db.query("MATCH (n: Object) RETURN DISTINCT n.class as class""")

Print distinct object classes and counts

db.query("MATCH (n: Object) RETURN DISTINCT n.class as class, COUNT(*) as count""")

Print center point of objects of specific class

db.query("MATCH (n: Object {class: 'tree'}) RETURN n.center as center""")

Filter for objects of a certain type within a bounding box

db.query("""WITH point({x: -100, y: 16, z: -100}) AS lowerLeft,
                 point({x: -90, y: 22, z:100}) AS upperRight
            MATCH (t: Object {class: "tree"})
            WHERE point.withinBBox(t.center, lowerLeft, upperRight)
            RETURN t as result""")

Filter for all objects of a certain type within 30 meters of given point

db.query(""" MATCH (t: Object {class: "tree"})
             WITH point.distance(point({x: -100, y:16, z:0}), t.center) as d, t
             WHERE d < 30
             RETURN t as obj, d as distance""")

Get all 2D places within 5 hops of given place

For graph-based reasoning with larger neighborhoods, there are some special built-in graph search algorithms that will be more performant that these general cypher queries. For example, for connected-component queries, you probably want to use the graph-data-science plugin

db.query(""" MATCH (p: MeshPlace {nodeSymbol: "P(1832)"})
             MATCH path=(p)-[:MESH_PLACE_CONNECTED *1..5]->(c: MeshPlace)
             RETURN DISTINCT c.nodeSymbol as ns
             ORDER BY c.nodeSymbol
         """)

Create a Room

Potentially useful for directly modifying data with LLM-generated prompt. For a more robust but less general way of adding nodes see graph_interface.py.

db.query("""WITH point({x: -95, y: 15, z: 0}) as center
            CREATE (:Room {nodeSymbol: "R(1)", center: center,  class: "test_room"})""")

Connect place near room to room

db.query("""WITH point({x: -120, y: 0, z: -100}) AS lowerLeft,
                 point({x: -70, y: 30, z:100}) AS upperRight
            MATCH (p: Place)
            MATCH (r: Room {nodeSymbol: "R(1)"})
            WHERE point.withinBBox(p.center, lowerLeft, upperRight)
            CREATE (r)-[:CONTAINS]->(p)
            """)

Query for objects in Room

Based on explicit structure of Room -> Place -> Object

db.query("""MATCH (r: Room {nodeSymbol: "R(1)"})-[:CONTAINS]->(p: Place)-[:CONTAINS]->(o: Object)
            RETURN o""")

Based on implicit / transitive structure of hierarchical relationships:

db.query("""MATCH (r: Room {nodeSymbol: "R(1)"})-[:CONTAINS*]->(o: Object)
            RETURN o""")

Development Notes

Converting between spark_dsg and Neo4J schema

Currently there is a manual step of turning each spark_dsg node into a dictionary of its attributes, which is then used to fill in the graph database schema. It would be slightly cleaner if this dict was automatically generated by calling vars() or __dict__ on the spark_dsg node, but those objects are lacking a __dict__, I assume because of pybind11-related reasons. I don't really want to deal with adding that functionality to our pybind11 spark_dsg interface, so for now we have this intermediate dictionary "representation"