save fastspeech2 pb model error

[zcy618]

Dear friends:
I use below code to save h5 file as PB model:

import yaml
import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf

from tensorflow_tts.inference import AutoConfig
from tensorflow_tts.inference import TFAutoModel
from tensorflow_tts.inference import AutoProcessor

processor = AutoProcessor.from_pretrained(
pretrained_path="./tensorflow_tts/processor/pretrained/baker_mapper.json"
)

input_text = "我如此爱你"
input_ids = processor.text_to_sequence(input_text, inference=True)

config = AutoConfig.from_pretrained("./examples/fastspeech2/conf/fastspeech2.baker.v2.yaml")
fastspeech2 = TFAutoModel.from_pretrained(
config=config,
pretrained_path=None, #"./fastspeech2-200k.h5",
is_build=False, # don't build model if you want to save it to pb. (TF related bug)
name="fastspeech2"
)

mel_before, mel_after, duration_outputs, _, _ = fastspeech2.inference(
input_ids=tf.expand_dims(tf.convert_to_tensor(input_ids, dtype=tf.int32), 0),
speaker_ids=tf.convert_to_tensor([0], dtype=tf.int32),
speed_ratios=tf.convert_to_tensor([1.0], dtype=tf.float32),
f0_ratios =tf.convert_to_tensor([1.0], dtype=tf.float32),
energy_ratios =tf.convert_to_tensor([1.0], dtype=tf.float32),
)

fastspeech2.load_weights("./fastspeech2-200k.h5")

save model into pb and do inference. Note that signatures should be a tf.function with input_signatures.

tf.saved_model.save(fastspeech2, "./test_saved", signatures=fastspeech2.inference)

but when I run it, I meet below error:

2020-12-07 18:05:14.061198: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
phoneme seq: sil ^ uo3 #0 r u2 #0 c ii3 #0 ^ ai4 #0 n i3 sil
2020-12-07 18:05:17.312790: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-12-07 18:05:18.367197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:04:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.898GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2020-12-07 18:05:18.368208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:82:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.898GHz coreCount: 20 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 298.32GiB/s
2020-12-07 18:05:18.368258: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-07 18:05:18.370493: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-12-07 18:05:18.372277: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-12-07 18:05:18.372643: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-12-07 18:05:18.374855: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-12-07 18:05:18.376082: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-12-07 18:05:18.380322: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-12-07 18:05:18.383254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2020-12-07 18:05:18.383772: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-07 18:05:18.415520: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2294725000 Hz
2020-12-07 18:05:18.418169: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55cba6596650 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-07 18:05:18.418198: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-12-07 18:05:18.901355: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55cba63c9810 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-07 18:05:18.901420: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1
2020-12-07 18:05:18.901441: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): GeForce GTX 1080, Compute Capability 6.1
2020-12-07 18:05:18.913007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:04:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.898GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2020-12-07 18:05:18.915151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:82:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.898GHz coreCount: 20 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 298.32GiB/s
2020-12-07 18:05:18.915275: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-07 18:05:18.915372: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-12-07 18:05:18.915432: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-12-07 18:05:18.915488: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-12-07 18:05:18.915549: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-12-07 18:05:18.915591: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-12-07 18:05:18.915633: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-12-07 18:05:18.919310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2020-12-07 18:05:18.919376: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-07 18:05:19.989349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-07 18:05:19.989414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 1
2020-12-07 18:05:19.989424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N N
2020-12-07 18:05:19.989430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 1: N N
2020-12-07 18:05:19.991845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7420 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:04:00.0, compute capability: 6.1)
2020-12-07 18:05:19.993056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7428 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080, pci bus id: 0000:82:00.0, compute capability: 6.1)
2020-12-07 18:05:28.063430: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-12-07 18:05:28.267447: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
Traceback (most recent call last):
File "savetopb.py", line 35, in
fastspeech2.load_weights("./fastspeech2-200k.h5")
File "/home/chenyuz/Desktop/TensorFlowTTS/venv/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 2200, in load_weights
'Unable to load weights saved in HDF5 format into a subclassed '
ValueError: Unable to load weights saved in HDF5 format into a subclassed Model which has not created its variables yet. Call the Model first, then load the weights.

Could you help to check what may I miss please?
Thanks

[dathudeptrai]

@zcy618 try tensorflow 2.2.0.

cc: @monatis I think you already fix this issue, right ? .

[monatis]

Hi @dathudeptrai and @zcy618!
I was able to export Tacotron 2 to saved_model format by modifying call() method of the model class, and the same can be used for Fastspeech 2. My change is on a separate branch on my fork, and I published a notebook here that demonstrates e2e export to, loading from and inference with saved_model. Basically, teacher forcing in call() method caused problems, so I modified it on another branch in such a way as to accept a single input, and checked out that branch when exporting.

[zcy618]

hi dathudeptrai and monatis:
after try tf2.2, this issue is gone, and I did not try monatis's code.

and now I have two questions:
1, if I keep two tf version2.3 and 2.2, it seems a little trouble for me, so I want to know if I could use tf2.2 to train the whole project.
2, I use tf2.3 to generate the h5 file, and use tf2.2 to generate pb file, so I want to know if there will be some problems on inference?
Thanks.

[dathudeptrai]

hi dathudeptrai and monatis:
after try tf2.2, this issue is gone, and I did not try monatis's code.

and now I have two questions:
1, if I keep two tf version2.3 and 2.2, it seems a little trouble for me, so I want to know if I could use tf2.2 to train the whole project.
2, I use tf2.3 to generate the h5 file, and use tf2.2 to generate pb file, so I want to know if there will be some problems on inference?
Thanks.

Training with tensorflow 2.3, 2.4 and use 2.2 for freeze model is fine. I will fix the bug in Tf 2.3, 2.4.

[JunjunCui]

@zcy618 hi, I met the same problem. May I ask, how did you solve it?

[dathudeptrai]

@JunjunCui see #446