RNN-T by julianmack · Pull Request #7 · MyrtleSoftware/myrtlespeech

julianmack · 2019-11-11T12:43:55Z

Adding Transducer (including RNNT model, transducer_loss and transducer_decoders). Also added CER calculation.

Note that, in order to organise the speech_to_text builder by loss it was necessary to alter the code order (to consider loss before model) meaning that the following files appear to have many more changes than they actually do:

src/myrtlespeech/builders/speech_to_text.py
tests/protos/test_speech_to_text.py

julianmack

Not ready for review
edit: out of date. PR now ready

julianmack · 2019-11-13T15:52:29Z

+    try:
+        stt: Union[SpeechToText, None] = None
+        stt = build(stt_cfg)
+    except AttributeError:
+        warnings.warn(
+            "This test has been (partially) disabled. TODO: remove this \
+            exception catching."
+        )
+    if stt is not None:
+        assert isinstance(stt, SpeechToText)
+        assert isinstance(stt, SeqToSeq)
+        warnings.warn("SpeechToText only built and not checked if correct")


@julianmack - remove this hack? This prevents failing tests when st draws a crossover ctc/rnnt loss and network

julianmack · 2019-11-13T15:52:51Z

+    try:
+        model, epochs, train_loader, eval_loader = build(task_cfg)
+        if model is not None:
+            assert isinstance(model, torch.nn.Module)
+        else:
+            warnings.warn(
+                "Not checking if model is returned. Remove above `if` \
+            statement once tests/protos/test_speech_to_text.py has had exception handling removed"
+            )
+
+        assert isinstance(epochs, int)
+        assert isinstance(train_loader, torch.utils.data.DataLoader)
+        assert isinstance(eval_loader, torch.utils.data.DataLoader)
+        warnings.warn("TaskConfig only built and not checked if correct")
+    except ValueError as e:
+        if str(e) == "unsupported model None":
+            warnings.warn(f"Caught error {e}.")
+        else:
+            raise e


@julianmack - remove this hack? This prevents failing tests when st draws a crossover ctc/rnnt loss and network

julianmack

Still to do:

Refactor decoder
Add a few tests (rnn w. hidden state and rnn_t loss in particular)
Remove hacks that prevent stt test failing when there is an rnnt network and ctc loss.

samgd

Great to have this working! The largest review theme is to generalise the transduction loss/decoding from being specific to an RNN. Yet to review the builders or tests under the assumption that they will change significantly when updating based on the comments.

samgd

Yet to review the docs, tests, and builders under the assumption that they may change significantly based on comments.

samgd · 2019-12-03T11:03:24Z

    cd .. && \
    rm -rf apex

+# install warp-transducer
+ENV CXX=/usr/bin/g++-6
+ENV CC=/usr/bin/gcc-6
+RUN make deps/warp-transducer


Can both warp-transducer and apex be installed via the Makefile so the Makefile becomes the single point of call for dependency installation?

It can but in the docs it states that:

The Dockerfile installs NVIDIA Apex, used for mixed precision, using a Python-only build and will omit some Apex features and performance improvements.

and I assumed that this was because CI doesn't like the proper apex build.

This might have been the wrong assumption - should we be using the cpp build for CI?

julianmack · 2019-12-05T14:00:42Z

  - pip=19.2.2=py37_0
  - pluggy=0.12.0=py_0
-  - pre_commit=1.17.0=py37_0
+  - pre_commit=1.11.2=0


I have worked out what the pre-commit problem was. The pre-commit update from version 1.11.2 -> 1.12 broke the reorder_python_imports integration.
So I've conda installed a more up-to-date version (which updates a few other core packages hence the other edits here)

…invalid combinations

…on throws error

…etwork lengths

julianmack commented Nov 11, 2019

View reviewed changes

Comment thread docs/source/install.rst

julianmack commented Nov 11, 2019

View reviewed changes

julianmack added the not ready WIP pull request not ready to be merged yet label Nov 12, 2019

julianmack force-pushed the rnnt branch 2 times, most recently from ec3838d to 4610468 Compare November 13, 2019 14:47

julianmack commented Nov 13, 2019

View reviewed changes

julianmack removed the not ready WIP pull request not ready to be merged yet label Nov 13, 2019

julianmack requested a review from samgd November 13, 2019 15:55

julianmack force-pushed the rnnt branch 2 times, most recently from b6de02d to b5bb579 Compare November 21, 2019 17:34

julianmack commented Nov 21, 2019

View reviewed changes

Comment thread Dockerfile Outdated

julianmack force-pushed the rnnt branch from 1d0b79e to 3b62fde Compare November 22, 2019 10:23

julianmack commented Nov 22, 2019

View reviewed changes

Comment thread tests/builders/test_rnn_t.py Outdated

julianmack marked this pull request as ready for review November 25, 2019 15:06

samgd suggested changes Nov 26, 2019

View reviewed changes

julianmack requested a review from samgd November 28, 2019 14:13

samgd suggested changes Dec 5, 2019

View reviewed changes

julianmack commented Dec 5, 2019

View reviewed changes

julianmack force-pushed the rnnt branch from 29afc29 to e401f21 Compare December 5, 2019 15:46

samgd suggested changes Dec 9, 2019

View reviewed changes

Comment thread Dockerfile Outdated

Comment thread src/myrtlespeech/model/transducer.py

julianmack force-pushed the rnnt branch from 8e4c992 to a0a1900 Compare December 9, 2019 16:09

julianmack requested a review from samgd December 10, 2019 11:33

julianmack force-pushed the rnnt branch 2 times, most recently from e7b3438 to edc1eb3 Compare December 11, 2019 14:34

julianmack changed the title ~~Rnnt~~ RNN-T Dec 16, 2019

julianmack force-pushed the rnnt branch from c5c5a16 to f0ba9fb Compare December 18, 2019 16:15

julianmack mentioned this pull request Jan 15, 2020

Learning-rate warmup + polynomial schedule #22

Draft

julianmack added the blocked label Jan 15, 2020

julianmack added 20 commits February 12, 2020 17:26

Added cmake instruction to installation

cae83c7

Added type annotation to preprocessing

98e0db4

Updated whitespace

6ab5b94

Updated rnn-t config

c259fa7

Updated whitespace

8bc1020

Removed hack in test/protos/speech_to_text

4ea1871

Small test changes

ffe1898

Removed hypothesis deadline

30d1911

Updated test docstrings

6bbc6df

refactored tests to use tensors() util to create inputs

07018de

Removed hypothesis deadline

67a1914

Reorganised speech_to_text builder to build loss first and check for …

d130f5b

…invalid combinations

Added tests to check that invalid loss + model/post_process combinati…

ea3f627

…on throws error

Refactored tests to reduce test-suite time

6fe2088

Updated ValueError message

78a7f44

Small fixes

528ee9f

Fixed duplicate import error

b713b17

Checkout rnn and fc files to master since these are up-to-date

64c2de4

Reverted rnn tests to fix rebase error

c96d894

Fixed failing tests after rebase

c14c7ee

julianmack force-pushed the rnnt branch from f097186 to c14c7ee Compare February 12, 2020 18:21

julianmack added 9 commits February 12, 2020 18:30

Updated incorrect function typing

38382d5

Small changes to prediction rnn_t including adding +1 to prediction n…

9e40e27

…etwork lengths

Fixed rnn hidden state API change failing tests

3f29625

Small changes for redability

98606ad

Updated transducer API to return hidden state. rnn_t not yet updated

a72e4ed

Transducer now accepts and returns hidden states

a6f57c5

Pushed rnn hidden states out of joint_network into Transducer class

20b23d6

Added typing to transducer post_process

7c16ce1

Reverted whitespace change

e710e09

Uh oh!

Conversation

julianmack commented Nov 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

julianmack left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julianmack Nov 13, 2019

Choose a reason for hiding this comment

Uh oh!

julianmack Nov 13, 2019

Choose a reason for hiding this comment

Uh oh!

julianmack left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

samgd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samgd left a comment

Choose a reason for hiding this comment

Uh oh!

samgd Dec 3, 2019

Choose a reason for hiding this comment

Uh oh!

julianmack Dec 5, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julianmack Dec 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

julianmack commented Nov 11, 2019 •

edited

Loading

julianmack left a comment •

edited

Loading

julianmack Dec 5, 2019 •

edited

Loading