Skip to content

Commit

Permalink
Update teaser image
Browse files Browse the repository at this point in the history
  • Loading branch information
JialianW committed Dec 15, 2022
1 parent 156c6e1 commit 33ae34a
Show file tree
Hide file tree
Showing 4 changed files with 11 additions and 7 deletions.
18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,6 @@ describes them with any style of free-form texts it was trained with, e.g., clas

Please follow [Installation instructions](docs/INSTALL.md).

## GRiT with ChatGPT

The [TWC team](https://github.com/taskswithcode) recently tried to feed the GRiT's dense captioning outputs
into ChatGPT to describe the scene as shown [here](https://twitter.com/imisra_/status/1602817378290905088?s=20&t=ROmjQqRIkbmFsBK69USRew).
The result looks amazing which I paste it below:
<p align="center"> <img src='docs/chatgpt_grit.jpg' align="center"> </p>

## Object Understanding Demo - One Model Two tasks

[Download the GRiT model](https://datarelease.blob.core.windows.net/grit/models/grit_b_densecap_objectdet.pth) or use the following commend to download:
Expand Down Expand Up @@ -49,6 +42,17 @@ Output images will be saved under the `visualization` folder, which looks like:

You can also try the Colab demo provided by the [TWC team](https://github.com/taskswithcode): [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/taskswithcode/GriT/blob/master/TWCGRiT.ipynb)

## ChatGPT with GRiT
We fed GRiT's dense captioning outputs (box location and description) into ChatGPT to have it
describe the scene. It turns out we can get a very good scene description, and we can also require
ChatGPT to describe the object relative positions. One example is shown below:

**GRiT's dense captioning outputs**
<p align="center"> <img src='docs/densecap.jpg' align="center"> </p>

**We request ChatGPT to describe the scene based GRiT's dense captioning outputs**
<p align="center"> <img src='docs/chatgpt.png' align="center"> </p>

## Benchmark Inference and Evaluation
Please follow [dataset preparation instructions](datasets/DATASETS.md) to download datasets.

Expand Down
Binary file added docs/chatgpt.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/chatgpt_grit.jpg
Binary file not shown.
Binary file added docs/densecap.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 33ae34a

Please sign in to comment.