C2F-ARM under performs than in the paper

[ZXP-S-works]

Hi Stephen,

C2F-ARM is an ingenious method and I tried to replicate the learning curves in the paper. However, I failed in achieving the same performance as the paper for tasks "stack_wine" and "phone_on_base".

In the paper Fig. 3, for the "phone_on_base" task C2F-ARM achieved ~60 return at step 5e3 and for the "stack_wine" task C2F-ARM achieved ~70 return at step 5e3. However, when I run the code, I got ~25 return at step 5e3 for "phone_on_base" and ~15 return at step 5e3 which deviate from the std of seeds in the paper. Could you give me some instructions on how to replicate the results in the paper?

Thank you for your time!

Best,
XP

Here is the learning curve I got:

Here is some information about the setup I have:
Collecting demos (RLBench):
--save_path=***/ARM/data/
--tasks=stack_wine or phone_on_base
--image_size=128,128
--renderer=opengl
--episodes_per_task=10 (I used 10 since C2F-ARM only needs 10)
--variations=1
--processes=1

Run the training code:
(python3.7)
method=C2FARM
rlbench.task=stack_wine or phone_on_base
rlbench.demo_path=***/Baselines/ARM/data
framework.gpu=0
framework.training_iterations=5000 or 15000(I used a smaller max iterations)
framework.logdir=./arm_test

Workstation:
Linux 20.04 LTS
C2F-ARM Git commit Initial QTE commit.
rlbench 1.2.0
torch 1.13.1
torchaudio 0.13.1
torchsummary 1.5.1
torchvision 0.14.1

[Xiangyu-Lee]

Hello, how do I run the code to get the image? Do I run the launch.py file? I am a beginner, and I'd appreciate it if you saw it.

[ZXP-S-works]

Hi xiao,

To generate these learning curves you basically train the model with flag framework.tensorboard_logging=True. The script will log the data for you. When you want view it, you can run tensorboard --logdir=PATH_TO_YOUR_LOG --samples_per_plugin=images=500 --port=6006 in the terminal.

Best,
XP

[Xiangyu-Lee]

I am very grateful for your reply, which provides me with very useful advice.