Unable to Achieve High-Resolution Results as Shown in README Examples

[brioanimationdev]

Hi,

I’m having trouble achieving the same high-resolution results as shown in the README examples. I followed the tutorial steps provided in the README closely and recorded my video using an iPhone 13 Mini. Below are the detailed steps I took:

Steps Followed:

1. Uploaded an 8-second video of my hand (light conditions were good).
2. Split the video into frames.
3. Created frame_list_test.txt and frame_list_train.txt based on the frames.
4. Ran RobustVideoMatting to generate the mask data.
5. Ran Mediapipe to extract the keypoint data.
6. Ran Depth-Anything-V2 to generate the depth maps.
7. Ran InterWild to generate the mano_params data.
8. Created the cam_param.json file using the following script:

   import math
   import json
   
   # Frame dimensions
   img_width = 720
   img_height = 1080
   
   # Principal point (center of the image)
   princpt_x = img_width / 2
   princpt_y = img_height / 2
   
   # Horizontal Field of View (HFOV) in degrees
   HFOV_deg = 73
   HFOV_rad = math.radians(HFOV_deg)
   
   # Compute focal length in pixels
   focal_length_px = (img_width / 2) / math.tan(HFOV_rad / 2)
   
   # Create camera parameters dictionary
   cam_param = {
       "focal": [focal_length_px, focal_length_px],  # Assuming fx = fy
       "princpt": [princpt_x, princpt_y]
   }
   
   # Save to cam_param.json
   cam_param_path = '/content/UHM/data/Custom/data/YourSubjectName/0/cam_param.json'
   with open(cam_param_path, 'w') as f:
       json.dump(cam_param, f, indent=4)

9. Placed and arranged all the models as per the README instructions.
10. Ran the training script. After training completed, I received 10 snapshot files (snapshot_10.pth through snapshot_99.pth).
11. Tested with snapshot 99 using the following command:

    %cd /content/UHM/Avatar/main/
    !python test.py --subject_id TomerSubject1 --test_epoch 99

12. Ran the test script again with the texture flag:

    %cd /content/UHM/Avatar/main/
    !python test.py --subject_id TomerSubject1 --test_epoch 99 --use_tex

13. Finally, ran the test script with the fit_pose_to_test flag:

    %cd /content/UHM/Avatar/main/
    !python test.py --subject_id TomerSubject1 --test_epoch 99 --use_tex --fit_pose_to_test

Issue:

Despite following all these steps, the resulting videos/meshes are not as accurate or detailed as I had hoped. I’ve attached all the data I used for reference. Could you please review it and help me figure out what I might be doing wrong? Any tips to achieve better results would be greatly appreciated!
my results + data:
https://drive.google.com/drive/folders/1CFGv7VChYxkTQOsNI_FevL7HMAB9cc6V?usp=sharing

Thanks in advance,
Brio

[mks0601]

Hi, you should not use monocular depth estimator like depth anything as they provide depth maps with scale and translation ambiguity. If you cannot get depth maps from your camera, you can just ignore the depth map loss and try again.

[brioanimationdev]

Thanks for your quick reply!

I agree that the new results without the depth data appear better, though I wouldn't say they are significantly more accurate.

To illustrate, I’m adding two examples:

• video_tex: Uses only the texture flag.
• video_tex_pose: Uses both texture and fit pose.

While I'm working to understand how to further improve the results, I also wanted to ask about the black glitches between frames in the fit pose. Could you shed some light on why this might be happening?

Thanks again for your help!

links to videos:
https://drive.google.com/drive/folders/1pZLxAmbH4DpeMTEkF-QMWFZnhfMbZdb0?usp=sharing

[mks0601]

I'm not sure what you mean by 'uses only the texture flag' and 'uses both texture and fit pose', but why the leftmost column has some ghost artifacts? The leftmost column should be the raw data without any processing, so it should be clean

[brioanimationdev]

Thanks for the quick replay !

I meant that for the 'video_tex', I used the following command:
%cd /content/UHM/Avatar/main/ !python test.py --subject_id TomerSubject1 --test_epoch 99 --use_tex

And for the 'video_tex_pose', I used this command:
!python test.py --subject_id TomerSubject1 --test_epoch 99 --use_tex --fit_pose_to_test

From my understanding, the --fit_pose_to_test option should produce better results for my resulting mesh based on the frame_list_test.txt file I created earlier. Am I correct?"

Do you have any other suggestions for achieving better mesh results? Have you used any depth models, or do you have a method for selecting the best snapshot?

[mks0601]

Got it. Could you clarify what's happening in the leftmost column? there is a ghost artifacts

[brioanimationdev]

I first used a regualr video of me record my hand on a gray wall.
Than I used the RVM for remove the original bg and add a white one.
Than I used the RVM for creating the masks on the new frames with the white bg.
The artifacts you see is the shadow on the wall behined the hand.

[mks0601]

I'm not clear about your original video. Could you share it with me?

[brioanimationdev]

Thanks for your help ! I am really appreciate it.
Sure, here is my original video:
https://github.com/user-attachments/assets/5958e12d-32be-4c91-a677-9f5dfd17f54d

[mks0601]

you'd better change your capture place as the shadow behind could make the RVM work bad as in your ghost artifact.