
This third_party additional enables the standard tokenization algorithms for text-based ML models for TFLite. The support library will enable the new language detection model and multiple other features under design and implementation. TFLite Chrome Browser Eng Review: https://groups.google.com/a/google.com/g/chrome-eng-review/c/jNnAeHXw8_0 Bug: 1171995 Change-Id: Ida60ca763f8d151b3e7dcb833b4108793e75c971 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2658595 Reviewed-by: Alex Gough <ajgo@chromium.org> Reviewed-by: Dirk Pranke <dpranke@google.com> Reviewed-by: John Abd-El-Malek <jam@chromium.org> Reviewed-by: Sophie Chang <sophiechang@chromium.org> Commit-Queue: Michael Crouse <mcrouse@chromium.org> Cr-Commit-Position: refs/heads/master@{#851273}
2.8 KiB
2.8 KiB
All symbols in TensorFlow Text
Primary symbols
text
text.BertTokenizer
text.Detokenizer
text.Reduction
text.SentencepieceTokenizer
text.SplitMergeTokenizer
text.Tokenizer
text.TokenizerWithOffsets
text.UnicodeCharTokenizer
text.UnicodeScriptTokenizer
text.WhitespaceTokenizer
text.WordShape
text.WordpieceTokenizer
text.case_fold_utf8
text.coerce_to_structurally_valid_utf8
text.gather_with_default
text.greedy_constrained_sequence
text.keras
text.keras.layers
text.keras.layers.ToDense
text.max_spanning_tree
text.max_spanning_tree_gradient
text.metrics
text.metrics.rouge_l
text.ngrams
text.normalize_utf8
text.pad_along_dimension
text.regex_split
text.regex_split_with_offsets
text.sentence_fragments
text.sliding_window
text.span_alignment
text.span_overlaps
text.viterbi_constrained_sequence
text.wordshape