[딥러닝] Chapter 09-3. LSTM과 GRU 셀

● Chpater 09의 학습목표

1. 텍스트와 시계열 데이터 같은 순차 데이터에 잘 맞는 순환 신경망의 개념과 구성 요소에 대해 배웁니다.
2. 케라스 API로 기본적인 순환 신경망에서 고급 순환 신경망을 만들어 영화 감성평을 분류하는 작업에 적용해 봅니다.
3. 순환 신경망에서 발생하는 문제점과 이를 극복하기 위한 해결책을 살펴봅니다.

● 학습목표: 순환 신경망에서의 핵심 기술인 LSTM과 GRU 셀을 사용한 모델을 만들어 봅니다.

 

● 키워드: LSTM, 셀 상태, GRU

 

● 지난 시간 - 순차 데이터와 순환 신경망의 구현

 

LSTM의 구조 및 설명

# IMDB 데이터 가져오기
# 500 가지 숫자로

from tensorflow.keras.datasets import imdb

from sklearn.model_selection import train_test_split

(train_input, train_target), (test_input, test_target) = \
imdb.load_data(num_words = 500)

train_input, val_input, train_target, val_target = train_test_split(
    train_input, train_target, test_size= 0.2, random_state= 42)

 

# 100 개로 패딩하기
from tensorflow.keras.utils import pad_sequences

train_seq = pad_sequences(train_input, maxlen = 100)
val_seq = pad_sequences(val_input, maxlen = 100)

 

# 순환 신경망 만들기
from tensorflow import keras

model = keras.Sequential()
model.add(keras.layers.Embedding(500, 16, input_length= 100))
model.add(keras.layers.LSTM(8))
model.add(keras.layers.Dense(1, activation = 'sigmoid'))

model.summary()

 

SimpleRNN model에서 input 16개와 뉴런 8개일때 model parameter가 200개 였습니다. LSTM에는 작은 셀이 4개가 들어 있으므로 4배인 800의 값을 가집니다.

 

 

 

# 모델 훈련하기
# Epochs = 100
# Early stopping
# save best moel

rmsprop = keras.optimizers.RMSprop(learning_rate = 1e-4)
model.compile(optimizer = rmsprop, loss = 'binary_crossentropy', metrics = ['accuracy'])

checkpoint_cb = keras.callbacks.ModelCheckpoint('best-LSTM-model.h5', save_best_only = True)
earlystopping_cb = keras.callbacks.EarlyStopping(patience = 3, restore_best_weights = True)

History = model.fit(train_seq, train_target, epochs = 100, batch_size = 64,
                    validation_data = (val_seq, val_target),
                    callbacks = [checkpoint_cb, earlystopping_cb])

WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.RMSprop` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.RMSprop`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.RMSprop`.
Epoch 1/100
313/313 [==============================] - 11s 25ms/step - loss: 0.6925 - accuracy: 0.5250 - val_loss: 0.6922 - val_accuracy: 0.5388
Epoch 2/100
  7/313 [..............................] - ETA: 6s - loss: 0.6926 - accuracy: 0.5290/opt/homebrew/lib/python3.11/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
313/313 [==============================] - 9s 29ms/step - loss: 0.6906 - accuracy: 0.5725 - val_loss: 0.6897 - val_accuracy: 0.5782
Epoch 3/100
313/313 [==============================] - 9s 30ms/step - loss: 0.6866 - accuracy: 0.6170 - val_loss: 0.6839 - val_accuracy: 0.6294
Epoch 4/100
313/313 [==============================] - 8s 26ms/step - loss: 0.6766 - accuracy: 0.6534 - val_loss: 0.6688 - val_accuracy: 0.6614
Epoch 5/100
313/313 [==============================] - 8s 26ms/step - loss: 0.6457 - accuracy: 0.6966 - val_loss: 0.6141 - val_accuracy: 0.7190
Epoch 6/100
313/313 [==============================] - 11s 35ms/step - loss: 0.5673 - accuracy: 0.7391 - val_loss: 0.5514 - val_accuracy: 0.7380
Epoch 7/100
313/313 [==============================] - 8s 26ms/step - loss: 0.5363 - accuracy: 0.7484 - val_loss: 0.5361 - val_accuracy: 0.7464
Epoch 8/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5169 - accuracy: 0.7600 - val_loss: 0.5152 - val_accuracy: 0.7644
Epoch 9/100
313/313 [==============================] - 10s 33ms/step - loss: 0.5006 - accuracy: 0.7746 - val_loss: 0.5019 - val_accuracy: 0.7706
Epoch 10/100
313/313 [==============================] - 11s 34ms/step - loss: 0.4867 - accuracy: 0.7798 - val_loss: 0.4893 - val_accuracy: 0.7804
Epoch 11/100
313/313 [==============================] - 9s 29ms/step - loss: 0.4748 - accuracy: 0.7868 - val_loss: 0.4801 - val_accuracy: 0.7820
Epoch 12/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4640 - accuracy: 0.7922 - val_loss: 0.4719 - val_accuracy: 0.7840
Epoch 13/100
313/313 [==============================] - 9s 29ms/step - loss: 0.4552 - accuracy: 0.7964 - val_loss: 0.4634 - val_accuracy: 0.7892
Epoch 14/100
313/313 [==============================] - 9s 28ms/step - loss: 0.4479 - accuracy: 0.8009 - val_loss: 0.4587 - val_accuracy: 0.7896
Epoch 15/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4417 - accuracy: 0.8033 - val_loss: 0.4543 - val_accuracy: 0.7932
Epoch 16/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4364 - accuracy: 0.8071 - val_loss: 0.4494 - val_accuracy: 0.7944
Epoch 17/100
313/313 [==============================] - 9s 28ms/step - loss: 0.4320 - accuracy: 0.8095 - val_loss: 0.4470 - val_accuracy: 0.7942
Epoch 18/100
313/313 [==============================] - 8s 27ms/step - loss: 0.4285 - accuracy: 0.8091 - val_loss: 0.4432 - val_accuracy: 0.7990
Epoch 19/100
313/313 [==============================] - 9s 30ms/step - loss: 0.4252 - accuracy: 0.8120 - val_loss: 0.4428 - val_accuracy: 0.7994
Epoch 20/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4227 - accuracy: 0.8120 - val_loss: 0.4396 - val_accuracy: 0.8006
Epoch 21/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4205 - accuracy: 0.8111 - val_loss: 0.4385 - val_accuracy: 0.7984
Epoch 22/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4187 - accuracy: 0.8123 - val_loss: 0.4396 - val_accuracy: 0.7996
Epoch 23/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4171 - accuracy: 0.8121 - val_loss: 0.4411 - val_accuracy: 0.7928
Epoch 24/100
313/313 [==============================] - 8s 27ms/step - loss: 0.4159 - accuracy: 0.8126 - val_loss: 0.4351 - val_accuracy: 0.7994
Epoch 25/100
313/313 [==============================] - 9s 27ms/step - loss: 0.4146 - accuracy: 0.8143 - val_loss: 0.4358 - val_accuracy: 0.8018
Epoch 26/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4134 - accuracy: 0.8144 - val_loss: 0.4337 - val_accuracy: 0.8024
Epoch 27/100
313/313 [==============================] - 9s 29ms/step - loss: 0.4124 - accuracy: 0.8152 - val_loss: 0.4351 - val_accuracy: 0.8042
Epoch 28/100
313/313 [==============================] - 10s 33ms/step - loss: 0.4115 - accuracy: 0.8144 - val_loss: 0.4339 - val_accuracy: 0.8020
Epoch 29/100
313/313 [==============================] - 11s 34ms/step - loss: 0.4107 - accuracy: 0.8146 - val_loss: 0.4344 - val_accuracy: 0.8010

 

# 손실 그래프 그려보기

import matplotlib.pyplot as plt

plt.title('Loss by Epochs, Epochs = 100, Earlystopping, Optimizer = RMSprop')

plt.xlabel('Epochs', fontsize = 14)
plt.ylabel('Loss', fontsize = 14)

plt.grid(True, which = 'both', linestyle = '--', color = 'gray')

plt.plot(History.history['loss'], color = 'gray', linewidth = 2.5)
plt.plot(History.history['val_loss'], color = 'blue', linewidth = 2.5)

plt.legend(['Train set', 'Val set'])
plt.show()

 

RNN에서  Dropout 적용

# Dropout을 적용해서 model 만들기

model2 = keras.Sequential()
model2.add(keras.layers.Embedding(500, 16, input_length = 100))
model2.add(keras.layers.LSTM(8, dropout= 0.3))
model2.add(keras.layers.Dense(1, activation = 'sigmoid'))

# Drop out을 적용해서 model 학습
rmsprop = keras.optimizers.RMSprop(learning_rate = 1e-4)
model2.compile(optimizer = rmsprop, loss = 'binary_crossentropy', metrics = ['accuracy'])

checkpoint_cb = keras.callbacks.ModelCheckpoint('best-LSTM_Dropout-model.h5', save_best_only = True)
earlystopping_cb = keras.callbacks.EarlyStopping(patience = 3, restore_best_weights = True)

History = model2.fit(train_seq, train_target, epochs = 100, batch_size = 64,
            validation_data = (val_seq, val_target),
            callbacks = [checkpoint_cb, earlystopping_cb])

WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.RMSprop` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.RMSprop`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.RMSprop`.
Epoch 1/100
313/313 [==============================] - 13s 30ms/step - loss: 0.6924 - accuracy: 0.5404 - val_loss: 0.6914 - val_accuracy: 0.5894
Epoch 2/100
  7/313 [..............................] - ETA: 8s - loss: 0.6918 - accuracy: 0.5379/opt/homebrew/lib/python3.11/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
313/313 [==============================] - 8s 27ms/step - loss: 0.6900 - accuracy: 0.5995 - val_loss: 0.6876 - val_accuracy: 0.6486
Epoch 3/100
313/313 [==============================] - 8s 26ms/step - loss: 0.6807 - accuracy: 0.6454 - val_loss: 0.6646 - val_accuracy: 0.6610
Epoch 4/100
313/313 [==============================] - 10s 31ms/step - loss: 0.6357 - accuracy: 0.6910 - val_loss: 0.6109 - val_accuracy: 0.7100
Epoch 5/100
313/313 [==============================] - 8s 27ms/step - loss: 0.6008 - accuracy: 0.7157 - val_loss: 0.5882 - val_accuracy: 0.7274
Epoch 6/100
313/313 [==============================] - 9s 29ms/step - loss: 0.5792 - accuracy: 0.7336 - val_loss: 0.5663 - val_accuracy: 0.7494
Epoch 7/100
313/313 [==============================] - 9s 29ms/step - loss: 0.5571 - accuracy: 0.7514 - val_loss: 0.5451 - val_accuracy: 0.7558
Epoch 8/100
313/313 [==============================] - 8s 26ms/step - loss: 0.5358 - accuracy: 0.7627 - val_loss: 0.5248 - val_accuracy: 0.7714
Epoch 9/100
313/313 [==============================] - 8s 26ms/step - loss: 0.5178 - accuracy: 0.7730 - val_loss: 0.5108 - val_accuracy: 0.7796
Epoch 10/100
313/313 [==============================] - 8s 27ms/step - loss: 0.5049 - accuracy: 0.7783 - val_loss: 0.5021 - val_accuracy: 0.7724
Epoch 11/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4959 - accuracy: 0.7793 - val_loss: 0.4918 - val_accuracy: 0.7824
Epoch 12/100
313/313 [==============================] - 12s 39ms/step - loss: 0.4859 - accuracy: 0.7846 - val_loss: 0.4844 - val_accuracy: 0.7854
Epoch 13/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4777 - accuracy: 0.7886 - val_loss: 0.4786 - val_accuracy: 0.7860
Epoch 14/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4708 - accuracy: 0.7909 - val_loss: 0.4718 - val_accuracy: 0.7892
Epoch 15/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4646 - accuracy: 0.7937 - val_loss: 0.4677 - val_accuracy: 0.7930
Epoch 16/100
313/313 [==============================] - 12s 39ms/step - loss: 0.4585 - accuracy: 0.7958 - val_loss: 0.4624 - val_accuracy: 0.7918
Epoch 17/100
313/313 [==============================] - 16s 50ms/step - loss: 0.4533 - accuracy: 0.7984 - val_loss: 0.4592 - val_accuracy: 0.7920
Epoch 18/100
313/313 [==============================] - 16s 50ms/step - loss: 0.4503 - accuracy: 0.7974 - val_loss: 0.4572 - val_accuracy: 0.7914
Epoch 19/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4459 - accuracy: 0.7997 - val_loss: 0.4527 - val_accuracy: 0.7956
Epoch 20/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4415 - accuracy: 0.8031 - val_loss: 0.4525 - val_accuracy: 0.7946
Epoch 21/100
313/313 [==============================] - 8s 27ms/step - loss: 0.4396 - accuracy: 0.8021 - val_loss: 0.4480 - val_accuracy: 0.7950
Epoch 22/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4379 - accuracy: 0.8005 - val_loss: 0.4459 - val_accuracy: 0.7960
Epoch 23/100
313/313 [==============================] - 7s 24ms/step - loss: 0.4351 - accuracy: 0.8039 - val_loss: 0.4454 - val_accuracy: 0.7964
Epoch 24/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4332 - accuracy: 0.8037 - val_loss: 0.4423 - val_accuracy: 0.7978
Epoch 25/100
313/313 [==============================] - 9s 28ms/step - loss: 0.4306 - accuracy: 0.8047 - val_loss: 0.4425 - val_accuracy: 0.7978
Epoch 26/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4298 - accuracy: 0.8038 - val_loss: 0.4394 - val_accuracy: 0.8012
Epoch 27/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4293 - accuracy: 0.8037 - val_loss: 0.4382 - val_accuracy: 0.8000
Epoch 28/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4266 - accuracy: 0.8061 - val_loss: 0.4378 - val_accuracy: 0.7976
Epoch 29/100
313/313 [==============================] - 8s 27ms/step - loss: 0.4247 - accuracy: 0.8089 - val_loss: 0.4388 - val_accuracy: 0.8000
Epoch 30/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4244 - accuracy: 0.8073 - val_loss: 0.4408 - val_accuracy: 0.7938
Epoch 31/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4226 - accuracy: 0.8080 - val_loss: 0.4350 - val_accuracy: 0.7984
Epoch 32/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4206 - accuracy: 0.8093 - val_loss: 0.4343 - val_accuracy: 0.8024
Epoch 33/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4205 - accuracy: 0.8099 - val_loss: 0.4374 - val_accuracy: 0.7956
Epoch 34/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4203 - accuracy: 0.8094 - val_loss: 0.4338 - val_accuracy: 0.8034
Epoch 35/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4179 - accuracy: 0.8115 - val_loss: 0.4329 - val_accuracy: 0.7976
Epoch 36/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4173 - accuracy: 0.8092 - val_loss: 0.4349 - val_accuracy: 0.7984
Epoch 37/100
313/313 [==============================] - 15s 46ms/step - loss: 0.4169 - accuracy: 0.8105 - val_loss: 0.4326 - val_accuracy: 0.7986
Epoch 38/100
313/313 [==============================] - 9s 30ms/step - loss: 0.4168 - accuracy: 0.8102 - val_loss: 0.4324 - val_accuracy: 0.8014
Epoch 39/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4152 - accuracy: 0.8122 - val_loss: 0.4320 - val_accuracy: 0.7994
Epoch 40/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4149 - accuracy: 0.8108 - val_loss: 0.4326 - val_accuracy: 0.8026
Epoch 41/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4151 - accuracy: 0.8104 - val_loss: 0.4303 - val_accuracy: 0.8022
Epoch 42/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4143 - accuracy: 0.8113 - val_loss: 0.4321 - val_accuracy: 0.7994
Epoch 43/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4126 - accuracy: 0.8117 - val_loss: 0.4297 - val_accuracy: 0.8010
Epoch 44/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4135 - accuracy: 0.8130 - val_loss: 0.4301 - val_accuracy: 0.8016
Epoch 45/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4114 - accuracy: 0.8127 - val_loss: 0.4299 - val_accuracy: 0.8010
Epoch 46/100
313/313 [==============================] - 8s 26ms/step - loss: 0.4131 - accuracy: 0.8100 - val_loss: 0.4296 - val_accuracy: 0.8044
Epoch 47/100
313/313 [==============================] - 9s 28ms/step - loss: 0.4117 - accuracy: 0.8105 - val_loss: 0.4291 - val_accuracy: 0.8036
Epoch 48/100
313/313 [==============================] - 9s 28ms/step - loss: 0.4108 - accuracy: 0.8141 - val_loss: 0.4351 - val_accuracy: 0.8040
Epoch 49/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4104 - accuracy: 0.8124 - val_loss: 0.4292 - val_accuracy: 0.8064
Epoch 50/100
313/313 [==============================] - 12s 40ms/step - loss: 0.4114 - accuracy: 0.8113 - val_loss: 0.4287 - val_accuracy: 0.8054
Epoch 51/100
313/313 [==============================] - 12s 39ms/step - loss: 0.4107 - accuracy: 0.8109 - val_loss: 0.4303 - val_accuracy: 0.8040
Epoch 52/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4109 - accuracy: 0.8130 - val_loss: 0.4303 - val_accuracy: 0.8010
Epoch 53/100
313/313 [==============================] - 10s 33ms/step - loss: 0.4094 - accuracy: 0.8128 - val_loss: 0.4281 - val_accuracy: 0.8036
Epoch 54/100
313/313 [==============================] - 12s 37ms/step - loss: 0.4094 - accuracy: 0.8122 - val_loss: 0.4287 - val_accuracy: 0.8022
Epoch 55/100
313/313 [==============================] - 10s 33ms/step - loss: 0.4065 - accuracy: 0.8149 - val_loss: 0.4282 - val_accuracy: 0.8054
Epoch 56/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4069 - accuracy: 0.8127 - val_loss: 0.4275 - val_accuracy: 0.8012
Epoch 57/100
313/313 [==============================] - 14s 44ms/step - loss: 0.4085 - accuracy: 0.8133 - val_loss: 0.4289 - val_accuracy: 0.8028
Epoch 58/100
313/313 [==============================] - 12s 37ms/step - loss: 0.4065 - accuracy: 0.8138 - val_loss: 0.4281 - val_accuracy: 0.8036
Epoch 59/100
313/313 [==============================] - 10s 32ms/step - loss: 0.4057 - accuracy: 0.8147 - val_loss: 0.4278 - val_accuracy: 0.8030

 

# 손실 그래프 그려보기

import matplotlib.pyplot as plt

plt.title('Loss by Epochs, Epochs = 100, Earlystopping, Optimizer = RMSprop')

plt.xlabel('Epochs', fontsize = 14)
plt.ylabel('Loss', fontsize = 14)

plt.grid(True, which = 'both', linestyle = '--', color = 'gray')

plt.plot(History.history['loss'], color = 'gray', linewidth = 2.5)
plt.plot(History.history['val_loss'], color = 'blue', linewidth = 2.5)

plt.legend(['Train set', 'Val set'])
plt.show()

 

 

2개의 RNN 연결하기

# 2개의 신경망 연결하기

model3 = keras.Sequential()
model3.add(keras.layers.Embedding(500, 16, input_length = 100))
model3.add(keras.layers.LSTM(8, dropout = 0.3, return_sequences = True))
model3.add(keras.layers.LSTM(8, dropout = 0.3))
model3.add(keras.layers.Dense(1, activation = 'sigmoid'))

model3.summary()

 

첫번째 LSTM 층은 모든 타임스텝(100개)의 은닉 상태를 출력하기 때문에 크기가 (None, 100, 8)이 출력

# 2개의 층과 Dropout을 적용해서 model 학습

rmsprop = keras.optimizers.RMSprop(learning_rate = 1e-4)

model3.compile(optimizer = rmsprop, loss = 'binary_crossentropy', metrics = ['accuracy'])

checkpoint_cb = keras.callbacks.ModelCheckpoint('best-LSTM_2nd-Dropout-model.h5', save_best_only = True)
earlystopping_cb = keras.callbacks.EarlyStopping(patience = 3, restore_best_weights = True)

History = model3.fit(train_seq, train_target, epochs = 100, batch_size = 64,
                    validation_data = (val_seq, val_target),
                    callbacks = [checkpoint_cb, earlystopping_cb])

WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.RMSprop` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.RMSprop`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.RMSprop`.
Epoch 1/100
313/313 [==============================] - 20s 50ms/step - loss: 0.5991 - accuracy: 0.7059 - val_loss: 0.5761 - val_accuracy: 0.7220
Epoch 2/100
  3/313 [..............................] - ETA: 12s - loss: 0.6085 - accuracy: 0.6719/opt/homebrew/lib/python3.11/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
313/313 [==============================] - 14s 46ms/step - loss: 0.5659 - accuracy: 0.7286 - val_loss: 0.5461 - val_accuracy: 0.7440
Epoch 3/100
313/313 [==============================] - 14s 46ms/step - loss: 0.5381 - accuracy: 0.7423 - val_loss: 0.5299 - val_accuracy: 0.7458
Epoch 4/100
313/313 [==============================] - 16s 50ms/step - loss: 0.5170 - accuracy: 0.7564 - val_loss: 0.5083 - val_accuracy: 0.7604
Epoch 5/100
313/313 [==============================] - 15s 47ms/step - loss: 0.5009 - accuracy: 0.7671 - val_loss: 0.4978 - val_accuracy: 0.7656
Epoch 6/100
313/313 [==============================] - 14s 45ms/step - loss: 0.4897 - accuracy: 0.7731 - val_loss: 0.4878 - val_accuracy: 0.7706
Epoch 7/100
313/313 [==============================] - 15s 49ms/step - loss: 0.4827 - accuracy: 0.7775 - val_loss: 0.4798 - val_accuracy: 0.7760
Epoch 8/100
313/313 [==============================] - 15s 47ms/step - loss: 0.4758 - accuracy: 0.7804 - val_loss: 0.4733 - val_accuracy: 0.7780
Epoch 9/100
313/313 [==============================] - 14s 46ms/step - loss: 0.4696 - accuracy: 0.7829 - val_loss: 0.4706 - val_accuracy: 0.7810
Epoch 10/100
313/313 [==============================] - 14s 46ms/step - loss: 0.4632 - accuracy: 0.7869 - val_loss: 0.4638 - val_accuracy: 0.7826
Epoch 11/100
313/313 [==============================] - 14s 45ms/step - loss: 0.4567 - accuracy: 0.7919 - val_loss: 0.4717 - val_accuracy: 0.7800
Epoch 12/100
313/313 [==============================] - 14s 46ms/step - loss: 0.4554 - accuracy: 0.7894 - val_loss: 0.4565 - val_accuracy: 0.7850
Epoch 13/100
313/313 [==============================] - 14s 46ms/step - loss: 0.4517 - accuracy: 0.7912 - val_loss: 0.4564 - val_accuracy: 0.7868
Epoch 14/100
313/313 [==============================] - 18s 57ms/step - loss: 0.4481 - accuracy: 0.7961 - val_loss: 0.4524 - val_accuracy: 0.7880
Epoch 15/100
313/313 [==============================] - 14s 43ms/step - loss: 0.4462 - accuracy: 0.7973 - val_loss: 0.4502 - val_accuracy: 0.7908
Epoch 16/100
313/313 [==============================] - 14s 46ms/step - loss: 0.4435 - accuracy: 0.7962 - val_loss: 0.4473 - val_accuracy: 0.7916
Epoch 17/100
313/313 [==============================] - 17s 55ms/step - loss: 0.4422 - accuracy: 0.7981 - val_loss: 0.4480 - val_accuracy: 0.7926
Epoch 18/100
313/313 [==============================] - 17s 54ms/step - loss: 0.4404 - accuracy: 0.7997 - val_loss: 0.4445 - val_accuracy: 0.7928
Epoch 19/100
313/313 [==============================] - 17s 54ms/step - loss: 0.4373 - accuracy: 0.7998 - val_loss: 0.4441 - val_accuracy: 0.7952
Epoch 20/100
313/313 [==============================] - 17s 55ms/step - loss: 0.4368 - accuracy: 0.7973 - val_loss: 0.4431 - val_accuracy: 0.7936
Epoch 21/100
313/313 [==============================] - 18s 57ms/step - loss: 0.4348 - accuracy: 0.8012 - val_loss: 0.4440 - val_accuracy: 0.7962
Epoch 22/100
313/313 [==============================] - 16s 52ms/step - loss: 0.4331 - accuracy: 0.8004 - val_loss: 0.4406 - val_accuracy: 0.7968
Epoch 23/100
313/313 [==============================] - 14s 43ms/step - loss: 0.4325 - accuracy: 0.8012 - val_loss: 0.4399 - val_accuracy: 0.7952
Epoch 24/100
313/313 [==============================] - 13s 43ms/step - loss: 0.4325 - accuracy: 0.8026 - val_loss: 0.4415 - val_accuracy: 0.7970
Epoch 25/100
313/313 [==============================] - 14s 44ms/step - loss: 0.4290 - accuracy: 0.8054 - val_loss: 0.4403 - val_accuracy: 0.7898
Epoch 26/100
313/313 [==============================] - 18s 57ms/step - loss: 0.4278 - accuracy: 0.8040 - val_loss: 0.4383 - val_accuracy: 0.7972
Epoch 27/100
313/313 [==============================] - 20s 65ms/step - loss: 0.4288 - accuracy: 0.8050 - val_loss: 0.4384 - val_accuracy: 0.7938
Epoch 28/100
313/313 [==============================] - 24s 78ms/step - loss: 0.4268 - accuracy: 0.8038 - val_loss: 0.4366 - val_accuracy: 0.7982
Epoch 29/100
313/313 [==============================] - 24s 76ms/step - loss: 0.4266 - accuracy: 0.8032 - val_loss: 0.4361 - val_accuracy: 0.7990
Epoch 30/100
313/313 [==============================] - 21s 67ms/step - loss: 0.4237 - accuracy: 0.8078 - val_loss: 0.4365 - val_accuracy: 0.7994
Epoch 31/100
313/313 [==============================] - 16s 51ms/step - loss: 0.4238 - accuracy: 0.8063 - val_loss: 0.4363 - val_accuracy: 0.7974
Epoch 32/100
313/313 [==============================] - 16s 50ms/step - loss: 0.4224 - accuracy: 0.8069 - val_loss: 0.4343 - val_accuracy: 0.8018
Epoch 33/100
313/313 [==============================] - 16s 50ms/step - loss: 0.4239 - accuracy: 0.8066 - val_loss: 0.4343 - val_accuracy: 0.7998
Epoch 34/100
313/313 [==============================] - 15s 48ms/step - loss: 0.4224 - accuracy: 0.8086 - val_loss: 0.4338 - val_accuracy: 0.8026
Epoch 35/100
313/313 [==============================] - 16s 51ms/step - loss: 0.4220 - accuracy: 0.8066 - val_loss: 0.4330 - val_accuracy: 0.8022
Epoch 36/100
313/313 [==============================] - 15s 49ms/step - loss: 0.4209 - accuracy: 0.8072 - val_loss: 0.4325 - val_accuracy: 0.8046
Epoch 37/100
313/313 [==============================] - 15s 49ms/step - loss: 0.4198 - accuracy: 0.8064 - val_loss: 0.4327 - val_accuracy: 0.8036
Epoch 38/100
313/313 [==============================] - 16s 50ms/step - loss: 0.4195 - accuracy: 0.8084 - val_loss: 0.4337 - val_accuracy: 0.8026
Epoch 39/100
313/313 [==============================] - 16s 51ms/step - loss: 0.4186 - accuracy: 0.8065 - val_loss: 0.4324 - val_accuracy: 0.8020
Epoch 40/100
313/313 [==============================] - 16s 52ms/step - loss: 0.4185 - accuracy: 0.8096 - val_loss: 0.4307 - val_accuracy: 0.8056
Epoch 41/100
313/313 [==============================] - 16s 50ms/step - loss: 0.4177 - accuracy: 0.8088 - val_loss: 0.4333 - val_accuracy: 0.7950
Epoch 42/100
313/313 [==============================] - 16s 51ms/step - loss: 0.4166 - accuracy: 0.8091 - val_loss: 0.4341 - val_accuracy: 0.8008
Epoch 43/100
313/313 [==============================] - 15s 48ms/step - loss: 0.4172 - accuracy: 0.8077 - val_loss: 0.4295 - val_accuracy: 0.8070
Epoch 44/100
313/313 [==============================] - 18s 56ms/step - loss: 0.4158 - accuracy: 0.8088 - val_loss: 0.4296 - val_accuracy: 0.8060
Epoch 45/100
313/313 [==============================] - 13s 40ms/step - loss: 0.4139 - accuracy: 0.8116 - val_loss: 0.4297 - val_accuracy: 0.8068
Epoch 46/100
313/313 [==============================] - 12s 39ms/step - loss: 0.4136 - accuracy: 0.8113 - val_loss: 0.4311 - val_accuracy: 0.8034

# 손실 그래프 그려보기

import matplotlib.pyplot as plt

plt.title('Loss by Epochs, Epochs = 100, Earlystopping, Optimizer = RMSprop')

plt.xlabel('Epochs', fontsize = 14)
plt.ylabel('Loss', fontsize = 14)

plt.grid(True, which = 'both', linestyle = '--', color = 'gray')

plt.plot(History.history['loss'], color = 'gray', linewidth = 2.5)
plt.plot(History.history['val_loss'], color = 'blue', linewidth = 2.5)

plt.legend(['Train set', 'Val set'])
plt.show()

손실이 개선 되었으며, 과대적합도 제어 되는것을 확인 할 수 있습니다.

GRU 셀 구현

# GRU 구현
model4 = keras.Sequential()
model4.add(keras.layers.Embedding(500, 16, input_length = 100))
model4.add(keras.layers.GRU(8))
model4.add(keras.layers.Dense(1, activation = 'sigmoid'))

model4.summary()

 

# GRU cell 훈련

rmsprop = keras.optimizers.RMSprop(learning_rate = 1e-4)

model4.compile(optimizer = rmsprop, loss = 'binary_crossentropy', metrics = ['accuracy'])

checkpoit_cb = keras.callbacks.ModelCheckpoint('best-GRU-model.h5', save_best_only = True)
earlystopping_cb = keras.callbacks.EarlyStopping(patience = 3, restore_best_weights = True)

History = model4.fit(train_seq, train_target, epochs = 100, batch_size = 64,
            validation_data = (val_seq, val_target),
            callbacks = [checkpoit_cb, earlystopping_cb])

WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.RMSprop` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.RMSprop`.
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.RMSprop`.
Epoch 1/100
313/313 [==============================] - 12s 33ms/step - loss: 0.6927 - accuracy: 0.5235 - val_loss: 0.6920 - val_accuracy: 0.5410
Epoch 2/100
  5/313 [..............................] - ETA: 10s - loss: 0.6921 - accuracy: 0.5625/opt/homebrew/lib/python3.11/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
313/313 [==============================] - 9s 30ms/step - loss: 0.6905 - accuracy: 0.5718 - val_loss: 0.6895 - val_accuracy: 0.5778
Epoch 3/100
313/313 [==============================] - 9s 28ms/step - loss: 0.6870 - accuracy: 0.6061 - val_loss: 0.6854 - val_accuracy: 0.6034
Epoch 4/100
313/313 [==============================] - 8s 26ms/step - loss: 0.6811 - accuracy: 0.6252 - val_loss: 0.6784 - val_accuracy: 0.6206
Epoch 5/100
313/313 [==============================] - 11s 35ms/step - loss: 0.6706 - accuracy: 0.6459 - val_loss: 0.6658 - val_accuracy: 0.6408
Epoch 6/100
313/313 [==============================] - 10s 32ms/step - loss: 0.6514 - accuracy: 0.6655 - val_loss: 0.6418 - val_accuracy: 0.6662
Epoch 7/100
313/313 [==============================] - 9s 28ms/step - loss: 0.6117 - accuracy: 0.6903 - val_loss: 0.5886 - val_accuracy: 0.7082
Epoch 8/100
313/313 [==============================] - 11s 34ms/step - loss: 0.5535 - accuracy: 0.7323 - val_loss: 0.5454 - val_accuracy: 0.7348
Epoch 9/100
313/313 [==============================] - 10s 31ms/step - loss: 0.5256 - accuracy: 0.7510 - val_loss: 0.5270 - val_accuracy: 0.7504
Epoch 10/100
313/313 [==============================] - 11s 35ms/step - loss: 0.5096 - accuracy: 0.7599 - val_loss: 0.5157 - val_accuracy: 0.7606
Epoch 11/100
313/313 [==============================] - 12s 39ms/step - loss: 0.4967 - accuracy: 0.7706 - val_loss: 0.5051 - val_accuracy: 0.7652
Epoch 12/100
313/313 [==============================] - 9s 30ms/step - loss: 0.4866 - accuracy: 0.7756 - val_loss: 0.4994 - val_accuracy: 0.7580
Epoch 13/100
313/313 [==============================] - 9s 27ms/step - loss: 0.4782 - accuracy: 0.7815 - val_loss: 0.4892 - val_accuracy: 0.7710
Epoch 14/100
313/313 [==============================] - 9s 30ms/step - loss: 0.4706 - accuracy: 0.7862 - val_loss: 0.4840 - val_accuracy: 0.7788
Epoch 15/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4636 - accuracy: 0.7889 - val_loss: 0.4777 - val_accuracy: 0.7826
Epoch 16/100
313/313 [==============================] - 9s 29ms/step - loss: 0.4578 - accuracy: 0.7934 - val_loss: 0.4735 - val_accuracy: 0.7836
Epoch 17/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4524 - accuracy: 0.7962 - val_loss: 0.4690 - val_accuracy: 0.7810
Epoch 18/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4479 - accuracy: 0.7983 - val_loss: 0.4699 - val_accuracy: 0.7752
Epoch 19/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4447 - accuracy: 0.8001 - val_loss: 0.4659 - val_accuracy: 0.7822
Epoch 20/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4414 - accuracy: 0.8029 - val_loss: 0.4615 - val_accuracy: 0.7814
Epoch 21/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4388 - accuracy: 0.8033 - val_loss: 0.4595 - val_accuracy: 0.7866
Epoch 22/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4355 - accuracy: 0.8055 - val_loss: 0.4578 - val_accuracy: 0.7904
Epoch 23/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4335 - accuracy: 0.8070 - val_loss: 0.4563 - val_accuracy: 0.7916
Epoch 24/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4314 - accuracy: 0.8084 - val_loss: 0.4555 - val_accuracy: 0.7910
Epoch 25/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4292 - accuracy: 0.8112 - val_loss: 0.4537 - val_accuracy: 0.7888
Epoch 26/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4270 - accuracy: 0.8105 - val_loss: 0.4553 - val_accuracy: 0.7906
Epoch 27/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4260 - accuracy: 0.8117 - val_loss: 0.4545 - val_accuracy: 0.7906
Epoch 28/100
313/313 [==============================] - 11s 34ms/step - loss: 0.4245 - accuracy: 0.8128 - val_loss: 0.4502 - val_accuracy: 0.7916
Epoch 29/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4233 - accuracy: 0.8124 - val_loss: 0.4497 - val_accuracy: 0.7906
Epoch 30/100
313/313 [==============================] - 7s 22ms/step - loss: 0.4219 - accuracy: 0.8141 - val_loss: 0.4519 - val_accuracy: 0.7874
Epoch 31/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4212 - accuracy: 0.8142 - val_loss: 0.4506 - val_accuracy: 0.7900
Epoch 32/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4200 - accuracy: 0.8143 - val_loss: 0.4605 - val_accuracy: 0.7810

 

plt.title('Loss by Epochs of GRU cell, Epochs = 100, Early stopping, Optimizer = RMSprop')

plt.xlabel('Epochs', fontsize = 14)
plt.ylabel('Loss', fontsize = 14)

plt.grid(True, which = 'both', linestyle = '--', color = 'gray')

plt.plot(History.history['loss'], color = 'gray', linewidth = 2.5)
plt.plot(History.history['val_loss'], color = 'blue', linewidth = 2.5)

plt.legend(['Train set', 'Val set'])

plt.show()

 

2개의 연결 RNN으로 된 LSTM model이 가장 좋은 결과를 나타냈기 때문에 이를 불러와서 test set로 evaluation 한 결과 val set와 유사한 test 결과를 얻었습니다.

# 테스트 세트 성능 확인

test_seq = pad_sequences(test_input, maxlen = 100)
rnn_model = keras.models.load_model('best-LSTM_2nd-Dropout-model.h5')
rnn_model.evaluate(test_seq, test_target)

782/782 [==============================] - 6s 6ms/step - loss: 0.4233 - accuracy: 0.8009
[0.4233225882053375, 0.8008800148963928]