Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts

Mohammad Sadil Khan* · Sankalp Sinha* · Talha Uddin Sheikh · Didier Stricker · Sk Aziz Ali · Muhammad Zeshan Afzal

*equal contributions

NeurIPS 2024 (Spotlight 🤩)

⚙️ Installation

🌍 Environment

• 🐧 Linux
• 🐍 Python >=3.9

📦 Dependencies

$ conda env create --file environment.yml

✅ Todo List

• Release Data Preparation Code
• Release Training Code
• Release Inference Code

📊 Data Preparation

Download the DeepCAD data from here.

Generate Vector Representation from DeepCAD Json

You can also download the processed cad vec from here.

$ cd CadSeqProc
$  python3 json2vec.py --input_dir $DEEPCAD_JSON --split_json $TRAIN_TEST_VAL_JSON --output_dir $OUTPUT_DIR --max_workers $WORKERS --padding --deduplicate

Download the text annotations from here.

🚀 Training

In the Cad_VLM/config/trainer.yaml, provide the following path.

Required Updates in yaml

• cache_dir: The directory to load model weights from Huggingface.
• cad_seq_dir: The root directory that contains the ground truth CAD vector.
• prompt_path: Path for the text annotation.
• split_filepath: Json file containing the UIDs for train, test or validation.
• log_dir: Directory for saving logs, outputs, checkpoints.
• checkpoint_path (Optional): For resuming training after some epochs.

$ cd Cad_VLM
$ python3 train.py --config_path config/trainer.yaml

🤖 Inference

For Test Dataset

In the Cad_VLM/config/inference.yaml, provide the following path. Download the checkpoint for v1.0 here.

Required Updates in yaml

• cache_dir: The directory to load model weights from Huggingface.
• cad_seq_dir: The root directory that contains the ground truth CAD vector.
• prompt_path: Path for the text annotation.
• split_filepath: Json file containing the UIDs for train, test or validation.
• log_dir: Directory for saving logs, outputs, checkpoints.
• checkpoint_path: The path to model weights.

$ cd Cad_VLM
$ python3 test.py --config_path config/inference.yaml

Run Evaluation

$ cd Evaluation
$ python3 eval_seq.py --input_path ./output.pkl --output_dir ./output

For Random Text Prompts

In the Cad_VLM/config/inference_user_input.yaml, provide the following path.

Required Updates in yaml

• cache_dir: The directory to load model weights from Huggingface.
• cad_seq_dir: The root directory that contains the ground truth CAD vector.
• prompt_path: Path for the text annotation.
• split_filepath: Json file containing the UIDs for train, test or validation.
• log_dir: Directory for saving logs, outputs, checkpoints.
• checkpoint_path: The path to model weights.
• prompt_file (Optional): For single prompt ignore it, for multiple prompts provide a txt file.

For single prompt

$ cd Cad_VLM
$ python3 test_user_input.py --config_path config/inference.yaml --prompt "A rectangular prism with a hole in the middle."

For Multiple prompts

$ cd Cad_VLM
$ python3 test_user_input.py --config_path config/inference.yaml

💻 Run Demo

$ cd App
$ gradio app.py

👥 Contributors

Our project owes its success to the invaluable contributions of these remarkable individuals. We extend our heartfelt gratitude for their dedication and support.

✍🏻 Acknowledgement

We thank the authors of DeepCAD and SkexGen and acknowledge the use of their code.

📜 Citation

If you use this dataset in your work, please consider citing the following publications.

@inproceedings{khan2024textcad,
title={Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts},
author={Mohammad Sadil Khan and Sankalp Sinha and Sheikh Talha Uddin and Didier Stricker and Sk Aziz Ali and Muhammad Zeshan Afzal},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=5k9XeHIK3L}
}

@InProceedings{Khan_2024_CVPR,
author = {Khan, Mohammad Sadil and Dupont, Elona and Ali, Sk Aziz and Cherenkova, Kseniya and Kacem, Anis and Aouada, Djamila},
title = {CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {4713-4722}
}