[use IK in record_demos.py] Instead of using teleoperation, IK is used to collect data

[YongcanZhou]

Has anyone modified the remote control data collection script to use IK to achieve the movement of the robot arm to complete the data collection instead of using the keyboard or VR?

My modifications are as follows. But the robot's movement is always not as expected.

    if "Franka" in args_cli.task:
        # === Initialize IK controller and entity configuration here ===
        from isaaclab.controllers import DifferentialIKController, DifferentialIKControllerCfg
        from isaaclab.managers import SceneEntityCfg

        global diff_ik_controller, robot_entity_cfg, ee_jacobi_idx

        # 1. Configure robot_entity_cfg
        robot_entity_cfg = SceneEntityCfg("robot", joint_names=["panda_joint.*"], body_names=["panda_link7"])
        robot_entity_cfg.resolve(env.scene)
        
        # 2. Get the body index of the Jacobian matrix
        ee_jacobi_idx = robot_entity_cfg.body_ids[0] - 1 if env.scene["robot"].is_fixed_base else robot_entity_cfg.body_ids[0]
        # 3. Initialize IK Controller
        diff_ik_cfg = DifferentialIKControllerCfg(command_type="pose", use_relative_mode=False, ik_method="dls")
        diff_ik_controller = DifferentialIKController(diff_ik_cfg, num_envs=env.num_envs, device=env.device)

        cube_1 = env.scene["cube_1"]
        # Get the world coordinates of cube_1
        target_pos_w = cube_1.data.root_pos_w  # (num_envs, 3)
        target_quat_w = cube_1.data.root_quat_w  # (num_envs, 4)
        # Get the world coordinates of the robot root
        root_pose_w = env.scene["robot"].data.root_state_w[:, 0:7]  # (num_envs, 7)

        # Convert to the target in the root coordinate system
        target_pos_b, target_quat_b = subtract_frame_transforms(
            root_pose_w[:, 0:3], root_pose_w[:, 3:7], target_pos_w, target_quat_w
        )  # (num_envs, 3), (num_envs, 4)

        target_com = torch.cat([target_pos_b, target_quat_b], dim=1)  # (num_envs, 7)
        ik_commands = torch.zeros(env.num_envs, diff_ik_controller.action_dim, device=env.device)
        ik_commands[:] = target_com[:]
        diff_ik_controller.set_command(ik_commands)

    with contextlib.suppress(KeyboardInterrupt) and torch.inference_mode():
        while simulation_app.is_running():
            # get data from teleop device
            teleop_data = teleop_interface.advance()

            # perform action on environment
            if running_recording_instance:
                # compute actions based on environment
                if "Franka" in args_cli.task:
                    actions = get_ik_action(env)
                else:
                    actions = pre_process_actions(teleop_data, env.num_envs, env.device)
                

def get_ik_action(env):
    # Get the target end position and posture (with cube_1 as the target)
    cube_1 = env.scene["cube_1"]
    target_pos = cube_1.data.root_pos_w  # (num_envs, 3)
    target_quat = cube_1.data.root_quat_w  # (num_envs, 4)

    robot = env.scene["robot"]
    # Get current simulation data
    jacobian = robot.root_physx_view.get_jacobians()[:, ee_jacobi_idx, :, robot_entity_cfg.joint_ids]
    ee_pose_w = robot.data.body_state_w[:, robot_entity_cfg.body_ids[0], 0:7]
    root_pose_w = robot.data.root_state_w[:, 0:7]
    joint_pos = robot.data.joint_pos[:, robot_entity_cfg.joint_ids]
    # Calculate the relative pose of the target in the root coordinate system
    ee_pos_b, ee_quat_b = subtract_frame_transforms(
        root_pose_w[:, 0:3], root_pose_w[:, 3:7], ee_pose_w[:, 0:3], ee_pose_w[:, 3:7]
    )
    # IK
    joint_pos_des = diff_ik_controller.compute(ee_pos_b, ee_quat_b, jacobian, joint_pos)
    action = joint_pos_des
    robot.set_joint_position_target(joint_pos_des, joint_ids=robot_entity_cfg.joint_ids)
    return action

[YongcanZhou]

arm_fail.mp4

[RandomOakForest]

Thank you for posting this, and adding the video. I'll move this to the Discussions section for the team and others to follow up.

[RandomOakForest]

Following up, to modify your remote control data collection script to use inverse kinematics (IK) for robot arm movement (replacing keyboard or VR), it’s crucial to ensure correct usage of the DifferentialIKController, correct frame conversions, and correct selection of the end-effector link. Here are key points you need to check and recommended modifications:

---

1. Use the Correct End Effector

For the Franka arm in Isaac Lab, the proper end-effector link is usually "panda_hand", not "panda_link7". Using "panda_link7" may cause frame offsets or undesired behavior.

Example:

robot_entity_cfg = SceneEntityCfg(
    "robot",
    joint_names=["panda_joint.*"],
    body_names=["panda_hand"]
)
robot_entity_cfg.resolve(env.scene)

---

2. Consistent Frame Transformations

All positions and orientations for the IK controller must be expressed in the robot’s base frame (not world). When using a target (e.g., cube), convert its world pose into the robot base frame using subtract_frame_transforms—which you’re doing correctly.

---

3. Obtain the Right Jacobian Index

Match the example logic:

if env.scene["robot"].is_fixed_base:
    ee_jacobi_idx = robot_entity_cfg.body_ids[^1_0] - 1
else:
    ee_jacobi_idx = robot_entity_cfg.body_ids

This matches the controller’s expectation for the Franka robot—a fixed base arm.

---

4. Correct Data Extraction for IK

• robot.data.body_state_w[:, body_id, 0:7] — for end-effector pose in world
• robot.data.root_state_w[:, 0:7] — for base pose in world

Example for conversion:

ee_pos_b, ee_quat_b = subtract_frame_transforms(
    root_pose_w[:, 0:3], root_pose_w[:, 3:7], ee_pose_w[:, 0:3], ee_pose_w[:, 3:7]
)

---

5. Controller Setup and Usage

Follow the official IK tutorial and match the buffer setup. Here’s a safe and reproducible structure closely matching the working script:¹²

# Controller configuration (run once for setup)
diff_ik_cfg = DifferentialIKControllerCfg(command_type="pose", use_relative_mode=False, ik_method="dls")
diff_ik_controller = DifferentialIKController(diff_ik_cfg, num_envs=env.num_envs, device=env.device)

# Per-step updates (in your control loop)
# Get current quantities
jacobian = robot.root_physx_view.get_jacobians()[:, ee_jacobi_idx, :, robot_entity_cfg.joint_ids]
ee_pose_w = robot.data.body_state_w[:, robot_entity_cfg.body_ids, 0:7]
root_pose_w = robot.data.root_state_w[:, 0:7]
joint_pos = robot.data.joint_pos[:, robot_entity_cfg.joint_ids]

# Compute end effector pose in robot base frame
ee_pos_b, ee_quat_b = subtract_frame_transforms(
    root_pose_w[:, 0:3], root_pose_w[:, 3:7], ee_pose_w[:, 0:3], ee_pose_w[:, 3:7]
)

# Compute IK (target pose comes from your application)
ik_target_pos_b, ik_target_quat_b = ... # Convert target (cube) from world to base frame here

ik_commands = torch.cat([ik_target_pos_b, ik_target_quat_b], dim=1)
diff_ik_controller.set_command(ik_commands)

joint_pos_des = diff_ik_controller.compute(ee_pos_b, ee_quat_b, jacobian, joint_pos)
robot.set_joint_position_target(joint_pos_des, joint_ids=robot_entity_cfg.joint_ids)

---

6. Common Issues (and Fixes)

• End-effector drifts or fails to reach target:
  • Check that "panda_hand" is used (not "panda_link7").
  • Ensure all quantities are batched if using multiple environments.
• Joint limits exceeded:
  • The IK solution may violate joint limits. Consider resetting to feasible starting states, or implement joint clamping if needed.³
• Target pose unreachable:
  • If your target (cube) is outside the robot’s reachable workspace, the IK will fail.

---

7. Full Example Reference

The Isaac Lab tutorial (run_diff_ik.py) provides a clean, working example using all best practices.²¹

---

Summary of Key Edits for Your Code:

• Change body_names=["panda_link7"] to body_names=["panda_hand"]
• Carefully convert all target poses and current EE poses into the robot’s base frame before IK
• Match the Jacobian, joint and pose indexing exactly as in the reference code
• Always batch operations when using multiple environments

Footnotes

1. https://isaac-sim.github.io/IsaacLab/main/source/tutorials/05_controllers/run_diff_ik.html ↩ ↩²

2. https://github.com/isaac-sim/IsaacLab/blob/main/scripts/tutorials/05_controllers/run_diff_ik.py ↩ ↩²

3. https://github.com/isaac-sim/IsaacLab/issues/2038 ↩