Feature request: add layer for darts lookup table?

[jeongukjae]

In some cases, it can be more efficient and memory-efficient than hashtable in tensorflow.

It should be great if darts lookup table has following methods

• build:
  • Arguments: strings tensors(key), int tensors(values), and default value
• common prefix search:
  • Arguments: string tensors
  • Returns: ragged tensor for searched results, and results' value
• exact match search:
  • Arguments: string tensors
  • Returns: searched value, or default value for unknown strings

[cantonios]

Is this a tf.text-specific request, or should it be filed against tensorflow?

Do you have a link for "darts lookup table"?

[jeongukjae]

Ah, sorry, I didn't specify the details. It's tensorflow-text specific request.

Darts is double array trie and we can use it like lookup table. You can check the basic interface here: https://github.com/s-yata/darts-clone/blob/master/doc/en/Interface.md#dictionary-class. Additionally, tensorflow text already has a dependency of darts-clone (used in wordpiece tokenizer, darts-clone is cloned repository of darts)

text/WORKSPACE

Lines 37 to 45 in b32645f

	http_archive(
	name = "darts_clone",
	build_file = "//third_party/darts_clone:BUILD.bzl",
	sha256 = "c97f55d05c98da6fcaf7f9ecc6a6dc6bc5b18b8564465f77abff8879d446491c",
	strip_prefix = "darts-clone-e40ce4627526985a7767444b6ed6893ab6ff8983",
	urls = [
	"https://github.com/s-yata/darts-clone/archive/e40ce4627526985a7767444b6ed6893ab6ff8983.zip",
	],
	)

Double array trie is performant and efficient data structure to store lots of strings and paired values, so it can be useful to train/serve with lots of vocabs. (like tens of milliions vocabs in the single model. it can be hard to use hash table because of the memory burden)

So I'm suggesting implementing the basic methods of the darts-clone's interface.