TACO: Temporal Consensus Optimization for Continual Neural Mapping

Xunlan Zhou^*, Hongrui Zhao^*, Negar Mehr

^*Equal contribution

    

Overview

This repository contains the official implementation of TACO, a replay-free framework for continual neural mapping. TACO treats historical neural implicit map snapshots as temporal neighbors and optimizes the current map with an importance-weighted temporal consensus objective. This lets the map preserve reliable geometry from the past while adapting regions that have changed, without storing or replaying previous RGB-D observations.

The current codebase builds on a Co-SLAM-style neural implicit mapping backbone and includes:

• TACO temporal consensus optimization.
• Importance estimation from output sensitivity on hash-grid parameters.
• Masked, importance-weighted consensus with historical model snapshots.
• Baselines and ablations including KR, EWC, MAS, CNM, and UNIKD.
• Official configs for Replica and ScanNet RGB-D sequences.

Installation

1. Create the environment

git clone https://github.com/labicon/TACO.git
cd TACO
conda create -n taco python=3.10
conda activate taco

Install PyTorch. The code has been tested with CUDA 11.8 on Ubuntu/Linux systems.

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu118

Install the remaining Python packages used for mapping, mesh extraction, visualization, and configuration.

pip install PyMCubes==0.1.6 open3d==0.18.0 trimesh==4.5.3 opencv-python==4.11.0.86 matplotlib==3.10.0 pyyaml==6.0.2 networkx seaborn psutil tensorboard tqdm scipy pyrender imageio

If OpenCV raises a Qt xcb plugin error on your machine, replace opencv-python with:

pip install opencv-python-headless==4.11.0.86

2. Build tiny-cuda-nn

Make sure CMake is 3.21 or newer and GCC is 11 or newer.

pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

3. Build PyTorch3D

pip install -U iopath
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

Datasets

Replica

Download the Replica RGB-D sequences generated by the iMAP/NICE-SLAM pipeline into data/Replica.

bash scripts/download_replica.sh

For quick testing, the original NICE-SLAM authors provide a small Office 1 download: https://drive.google.com/drive/folders/1211QzrvfZGgiNk7H0t-NhpuUz6vv2MNN?usp=sharing

ScanNet

Follow the ScanNet data access instructions at http://www.scan-net.org/ and extract RGB/depth frames from .sens files with the official ScanNet reader.

Running TACO

Configuration files live under configs/. Scene-specific YAML files inherit shared defaults through the inherit_from field.

Important options:

• data.datadir: input RGB-D sequence.
• data.output and data.exp_name: output directory.
• data.load_path: pretrained decoder checkpoint, used when multi_agents.fix_decoder: True.
• mapping.enable_replay: set to False for replay-free TACO.
• training.temporal_consensus.enabled: enables TACO temporal consensus.
• training.temporal_consensus.K: number of historical snapshots kept for consensus.
• training.temporal_consensus.frames_per_task: snapshot update interval.
• training.temporal_consensus.inject_every: temporal consensus injection interval.
• training.temporal_consensus.mask_threshold: masks unreliable low-importance parameters.
• multi_agents.distributed_algorithm: use AUQ_CADMM for the current importance-weighted consensus implementation.
• multi_agents.fix_decoder: set to True to optimize the hash-grid representation with a fixed decoder.

Run a Replica Office 1 experiment:

python main.py --config configs/Replica/office1.yaml

The default configs/Replica/office1.yaml is configured as a single-agent TACO run with replay disabled, temporal consensus enabled, and a fixed decoder loaded from fixed_decoder.pt.

Decoder Pretraining

If you want to train a decoder checkpoint before running mapping with multi_agents.fix_decoder: True, use:

python train_decoder.py --config configs/Replica/office1.yaml

The best decoder checkpoint is saved as best_decoder.pt under the configured output directory. Update data.load_path in your mapping config to use that checkpoint.

Visualization

Visualize a run:

python visualizer.py --config configs/Replica/office1.yaml

Evaluation

TACO evaluates reconstruction quality in two steps: first cull the reconstructed mesh using the observed camera frustums, then compare the culled mesh with the ground-truth mesh. The wrapper below runs both steps:

python evaluate.py \
  --config configs/Replica/office1.yaml \
  --input_mesh output/Replica/office1/TACO/agent_0/mesh_track1999.ply \
  --gt_mesh data/Replica/office1/office1_mesh.ply \
  --gt_pose

By default, evaluate.py writes a culled mesh next to the input mesh with a _cull_occlusion.ply suffix and then reports 3D accuracy, completion, and completion ratio. If you want to use estimated poses from a checkpoint instead of dataset ground-truth poses, replace --gt_pose with:

--ckpt_path output/Replica/office1/TACO/agent_0/checkpoint1999.pt

For ScanNet, use the matching ScanNet config, reconstructed mesh, and ground-truth mesh:

python evaluate.py \
  --config configs/scannet/scene0000.yaml \
  --input_mesh output/scannet/scene0000_00/TACO/agent_0/mesh_track5577.ply \
  --gt_mesh data/scannet/scene0000_00/scene0000_00_vh_clean_2.ply \
  --gt_pose

The underlying scripts are still available for custom workflows:

• cull_mesh.py: mesh culling.
• eval_recon.py: reconstruction metrics.
• tools/eval_ate.py: camera trajectory evaluation.

Outputs

Each run writes results to:

<data.output>/<data.exp_name>/

Typical outputs include:

• config.json: resolved run configuration.
• agent_*/checkpoint*.pt: model and pose checkpoints.
• agent_*/mesh_track*.ply: reconstructed meshes.
• agent_*/logs/: TensorBoard logs.
• agent_*/mem_usage.csv: CPU/GPU memory logs.
• graph_data.json: communication/dropout history when graph communication is used.

Acknowledgements

This codebase builds on neural implicit mapping components from Co-SLAM. We thank the Co-SLAM authors and the broader open-source mapping community for their contributions.

Citation

If you find this code useful, please cite:

@misc{zhou2026tacotemporalconsensusoptimization,
      title={TACO: Temporal Consensus Optimization for Continual Neural Mapping}, 
      author={Xunlan Zhou and Hongrui Zhao and Negar Mehr},
      year={2026},
      eprint={2602.04516},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2602.04516}, 
}