Compositional Generalization in Semantic Parsing

Code and details for reproducing results for the paper ["Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures"] (https://arxiv.org/abs/2007.08970)

Instructions for most experiments

This section describes how most experiments of the paper can be reproduced. For CGPS and T5 see the separate sections further down.

Prerequisites

If you want to get started quickly you can use the tf-1-15-cu100 image from deeplearning-platform-release which comes with some of the right dependencies preinstalled. The following command starts a VM with this image and a V100 GPU on Google Cloud:

gcloud config set project ${YOUR_PROJECT}
gcloud config set compute/zone europe-west4-a

VM_NAME=run-cfq
gcloud compute instances create $VM_NAME \
  --image-project=deeplearning-platform-release \
  --image-family=tf-1-15-cu100 \
  --machine-type=n1-standard-8 \
  --boot-disk-size=120GB \
  --maintenance-policy=TERMINATE \
  --accelerator="type=nvidia-tesla-v100,count=1" \
  --metadata="install-nvidia-driver=True"

Next, on that machine, get code with:

sudo apt-get install subversion -y
svn export https://github.com/google-research/google-research/trunk/cfq

You can then install the necessary Python dependencies using:

python3.7 -m pip install -r cfq/requirements.txt --user
# Also make sure the installed binaries are in the path.
export PATH="$HOME/.local/bin:$PATH"

Running the experiment

This command will download the dataset, preprocess it, train a model and finally evaluate it and report it's accuracy:

python3.7 -m cfq.run_experiment \
  --dataset=scan --split=mcd1 \
  --model=evolved_transformer --hparams_set=cfq_evolved_transformer \
  --train_steps=100000

Customize the invocation arguments for the experiment that you want to run.

Model / hyperparameter configurations used in the paper are:

model	hparams_set
lstm_seq2seq_attention	cfq_lstm_attention_multi
transformer	cfq_transformer
universal_transformer	cfq_universal_transformer
evolved_transformer	cfq_evolved_transformer
shuffle_network	shuffle_network_baseline

Instructions for CGPS experiments

This command will download the dataset, preprocess it, train a model and finally evaluate it and report it's accuracy:

python3.7 -m CGPS.main --flagfile=CGPS/experiments/cfq_mcd1/flags.txt

T5 fine-tuning instructions

Below are instructions for fine-tuning a T5-small model on the MCD1 split of CFQ. These instructions can easily be modified to fine-tune T5 on any split of both the SCAN and CFQ dataset.

Before you begin

Before starting this tutorial, check that your Google Cloud project is correctly set up. For more information, see Set up an account and a Cloud TPU project.

We recommend using CTPU to create a VM and a TPU device. See the quickstart for more details.

NOTE Please make sure your VM has at least 30Gb of RAM, otherwise preprocessing the CFQ dataset will fail.

The T5 Github page contains further instructions for setting up the TPU device using CTPU.

Installing dependencies

Once you are logged into your Google Cloud VM, install T5 for GCP.

pip3 install t5[gcp]

Then download the custom T5 tasks from this repository in order to run fine-tuning on SCAN and CFQ, and make sure the tasks can be used by the T5 binary by putting them into a directory and adding it to PYTHONPATH.

GR_REPO="https://raw.githubusercontent.com/google-research/google-research"
wget "${GR_REPO}/master/cfq_pt_vs_sa/cfq_scan_tasks.py"
mkdir t5_cfq_scan
mv cfq_scan_tasks.py t5_cfq_scan
export PYTHONPATH=$PYTHONPATH:$(pwd)

Fine-tuning.

The preprocessing in cfq_scan_tasks.py uses tf.py_function, which doesn't work on headless TPU devices. Therefore we cache the datasets first using T5's cache_tasks_main.py.

First download the dataset and split to your local disc using TFDS.

python3 -c "import tensorflow_datasets as tfds; tfds.load('cfq/mcd1')"

The dataset will be cached to ~/tensorflow_datasets by default. Next we cache the dataset.

TFDS_LOCAL_PATH=$USER/tensorflow_datasets
# Change this to a directory in a Cloud bucket.
CACHE_DIR=gs://<YOUR_CLOUD_BUCKET>/cache

# Clone the repository locally.
git clone https://github.com/google-research/text-to-text-transfer-transformer.git

python3 text-to-text-transfer-transformer/t5/data/cache_tasks_main.py \
  --tasks=cfq_mcd1 \
  --output_cache_dir=$CACHE_DIR \
  --module_import=t5_cfq_scan.cfq_scan_tasks \
  --tasks_additional_cache_dirs=$TFDS_LOCAL_PATH

Make sure the following environment variables are set:

• TPU_NAME: The name of your TPU device.

• PROJECT: The name of your Google Cloud project.

• ZONE: Your project zone.

• MODEL_DIR: Location where the model will be saved (in a Cloud bucket).

• DATA_DIR: Location where the dataset data will be saved.

• TPU_SIZE: Size of your TPU (e.g., 2x2).

• CACHE_DIR: Google Cloud directory where the dataset is cached.

We used the following settings for model parallelism and TPU topologies when doing our experiments.

T5 size	model parallelism	TPU topology
small	1	4x4
base	2	8x4
large	8	8x16
3B	8	8x16
11B	8	8x16

Then run the following command to start fine-tuning:

DATASET=cfq
SPLIT=mcd1
T5_SIZE=small
t5_mesh_transformer  \
  --tpu="${TPU_NAME}" \
  --gcp_project="${PROJECT}" \
  --tpu_zone="${ZONE}" \
  --model_dir="${MODEL_DIR}" \
  --t5_tfds_data_dir="${DATA_DIR}" \
  --gin_file="dataset.gin" \
  --gin_param="utils.tpu_mesh_shape.model_parallelism = 1" \
  --gin_param="utils.tpu_mesh_shape.tpu_topology = '${TPU_SIZE}'" \
  --gin_file="gs://t5-data/pretrained_models/${T5_SIZE}/operative_config.gin" \
  --gin_param="MIXTURE_NAME = '${DATASET}_${SPLIT}'" \
  --module_import=t5_cfq_scan.cfq_scan_tasks \
  --additional_task_cache_dirs="${CACHE_DIR}" \
  --gin_param="mesh_train_dataset_fn.use_cached = True"

Evaluation

Assuming the model-finetuned for 262,144 steps (the default), and all bash flags mentioned above are set, evaluation can be run with the following command:

t5_mesh_transformer \
  --tpu="${TPU_NAME}" \
  --gcp_project="${PROJECT}" \
  --tpu_zone="${ZONE}" \
  --model_dir="${MODEL_DIR}" \
  --gin_file="${MODEL_DIR}/operative_config.gin" \
  --t5_tfds_data_dir=${DATA_DIR} \
  --gin_file="eval.gin" \
  --gin_file="beam_search.gin" \
  --gin_param="run.dataset_split = 'validation'" \
  --gin_param="utils.tpu_mesh_shape.tpu_topology = '${TPU_SIZE}'" \
  --gin_param="eval_checkpoint_step = 262144" \
  --gin_param="MIXTURE_NAME = 'cfq_${SPLIT}'" \
  --module_import=t5_cfq_scan.cfq_scan_tasks