Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/ko/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,8 @@
title: (λ²ˆμ—­μ€‘) Summary of the tokenizers
- local: in_translation
title: (λ²ˆμ—­μ€‘) Attention mechanisms
- local: in_translation
title: (λ²ˆμ—­μ€‘) Padding and truncation
- local: pad_truncation
title: νŒ¨λ”©κ³Ό μž˜λΌλ‚΄κΈ°
- local: in_translation
title: (λ²ˆμ—­μ€‘) BERTology
- local: in_translation
Expand Down
64 changes: 64 additions & 0 deletions docs/source/ko/pad_truncation.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
<!--Copyright 2022 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# νŒ¨λ”©κ³Ό μž˜λΌλ‚΄κΈ°[[padding-and-truncation]]

배치 μž…λ ₯은 길이가 λ‹€λ₯Έ κ²½μš°κ°€ λ§Žμ•„μ„œ κ³ μ • 크기 ν…μ„œλ‘œ λ³€ν™˜ν•  수 μ—†μŠ΅λ‹ˆλ‹€. νŒ¨λ”©κ³Ό μž˜λΌλ‚΄κΈ°λŠ” λ‹€μ–‘ν•œ 길이의 λ°°μΉ˜μ—μ„œ μ§μ‚¬κ°ν˜• ν…μ„œλ₯Ό 생성할 수 μžˆλ„λ‘ 이 문제λ₯Ό ν•΄κ²°ν•˜λŠ” μ „λž΅μž…λ‹ˆλ‹€. νŒ¨λ”©μ€ νŠΉμˆ˜ν•œ **νŒ¨λ”© 토큰**을 μΆ”κ°€ν•˜μ—¬ 짧은 μ‹œν€€μŠ€κ°€ λ°°μΉ˜μ—μ„œ κ°€μž₯ κΈ΄ μ‹œν€€μŠ€ λ˜λŠ” λͺ¨λΈμ—μ„œ ν—ˆμš©ν•˜λŠ” μ΅œλŒ€ 길이와 λ™μΌν•œ 길이λ₯Ό 갖도둝 ν•©λ‹ˆλ‹€. μž˜λΌλ‚΄κΈ°λŠ” κΈ΄ μ‹œν€€μŠ€λ₯Ό μž˜λΌλ‚΄μ–΄ νŒ¨λ”©κ³Ό λ‹€λ₯Έ λ°©μ‹μœΌλ‘œ μ‹œν€€μŠ€μ˜ 길이λ₯Ό λ™μΌν•˜κ²Œ ν•©λ‹ˆλ‹€.

λŒ€λΆ€λΆ„μ˜ 경우 λ°°μΉ˜μ— κ°€μž₯ κΈ΄ μ‹œν€€μŠ€μ˜ 길이둜 νŒ¨λ”©ν•˜κ³  λͺ¨λΈμ΄ ν—ˆμš©ν•  수 μžˆλŠ” μ΅œλŒ€ 길이둜 μž˜λΌλ‚΄λŠ” 것이 잘 μž‘λ™ν•©λ‹ˆλ‹€. κ·ΈλŸ¬λ‚˜ ν•„μš”ν•˜λ‹€λ©΄ APIκ°€ μ§€μ›ν•˜λŠ” 더 λ§Žμ€ μ „λž΅μ„ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€. ν•„μš”ν•œ μΈμˆ˜λŠ” `padding`, `truncation`, `max_length` μ„Έ κ°€μ§€μž…λ‹ˆλ‹€.

`padding` μΈμˆ˜λŠ” νŒ¨λ”©μ„ μ œμ–΄ν•©λ‹ˆλ‹€. λΆˆλ¦¬μ–Έ λ˜λŠ” λ¬Έμžμ—΄μΌ 수 μžˆμŠ΅λ‹ˆλ‹€:

- `True` λ˜λŠ” `'longest'`: λ°°μΉ˜μ—μ„œ κ°€μž₯ κΈ΄ μ‹œν€€μŠ€λ‘œ νŒ¨λ”©ν•©λ‹ˆλ‹€(단일 μ‹œν€€μŠ€λ§Œ μ œκ³΅ν•˜λŠ” 경우 νŒ¨λ”©μ΄ μ μš©λ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€).
- `'max_length'`: `max_length` μΈμˆ˜κ°€ μ§€μ •ν•œ 길이둜 νŒ¨λ”©ν•˜κ±°λ‚˜, `max_length`κ°€ μ œκ³΅λ˜μ§€ μ•Šμ€ 경우(`max_length=None`) λͺ¨λΈμ—μ„œ ν—ˆμš©λ˜λŠ” μ΅œλŒ€ 길이둜 νŒ¨λ”©ν•©λ‹ˆλ‹€. 단일 μ‹œν€€μŠ€λ§Œ μ œκ³΅ν•˜λŠ” κ²½μš°μ—λ„ νŒ¨λ”©μ΄ μ μš©λ©λ‹ˆλ‹€.
- `False` λ˜λŠ” `'do_not_pad'`: νŒ¨λ”©μ΄ μ μš©λ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. 이것이 κΈ°λ³Έ λ™μž‘μž…λ‹ˆλ‹€.

`truncation` μΈμˆ˜λŠ” μž˜λΌλ‚Ό 방법을 μ •ν•©λ‹ˆλ‹€. λΆˆλ¦¬μ–Έ λ˜λŠ” λ¬Έμžμ—΄μΌ 수 μžˆμŠ΅λ‹ˆλ‹€:

- `True` λ˜λŠ” `longest_first`: `max_length` μΈμˆ˜κ°€ μ§€μ •ν•œ μ΅œλŒ€ 길이둜 μž˜λΌλ‚΄κ±°λ‚˜,
`max_length`κ°€ μ œκ³΅λ˜μ§€ μ•Šμ€ 경우(`max_length=None`) λͺ¨λΈμ—μ„œ ν—ˆμš©λ˜λŠ” μ΅œλŒ€ 길이둜 μž˜λΌλƒ…λ‹ˆλ‹€.
μ‹œν€€μŠ€ μŒμ—μ„œ κ°€μž₯ κΈ΄ μ‹œν€€μŠ€μ˜ 토큰을 μ μ ˆν•œ 길이에 도달할 λ•ŒκΉŒμ§€ ν•˜λ‚˜μ”© μ œκ±°ν•©λ‹ˆλ‹€.
- `'only_second'`: `max_length` μΈμˆ˜κ°€ μ§€μ •ν•œ μ΅œλŒ€ 길이둜 μž˜λΌλ‚΄κ±°λ‚˜,
`max_length`κ°€ μ œκ³΅λ˜μ§€ μ•Šμ€ 경우(`max_length=None`) λͺ¨λΈμ—μ„œ ν—ˆμš©λ˜λŠ” μ΅œλŒ€ 길이둜 μž˜λΌλƒ…λ‹ˆλ‹€.
μ‹œν€€μŠ€ 쌍(λ˜λŠ” μ‹œν€€μŠ€ 쌍의 배치)κ°€ 제곡된 경우 쌍의 두 번째 λ¬Έμž₯만 μž˜λΌλƒ…λ‹ˆλ‹€.
- `'only_first'`: `max_length` μΈμˆ˜κ°€ μ§€μ •ν•œ μ΅œλŒ€ 길이둜 μž˜λΌλ‚΄κ±°λ‚˜,
`max_length`κ°€ μ œκ³΅λ˜μ§€ μ•Šμ€ 경우(`max_length=None`) λͺ¨λΈμ—μ„œ ν—ˆμš©λ˜λŠ” μ΅œλŒ€ 길이둜 μž˜λΌλƒ…λ‹ˆλ‹€.
μ‹œν€€μŠ€ 쌍(λ˜λŠ” μ‹œν€€μŠ€ 쌍의 배치)κ°€ 제곡된 경우 쌍의 첫 번째 λ¬Έμž₯만 μž˜λΌλƒ…λ‹ˆλ‹€.
- `False` λ˜λŠ” `'do_not_truncate'`: μž˜λΌλ‚΄κΈ°λ₯Ό μ μš©ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. 이것이 κΈ°λ³Έ λ™μž‘μž…λ‹ˆλ‹€.

`max_length` μΈμˆ˜λŠ” νŒ¨λ”© 및 μž˜λΌλ‚΄κΈ°λ₯Ό μ μš©ν•  길이λ₯Ό μ œμ–΄ν•©λ‹ˆλ‹€. 이 μΈμˆ˜λŠ” μ •μˆ˜ λ˜λŠ” `None`일 수 있으며, `None`일 경우 λͺ¨λΈμ΄ ν—ˆμš©ν•  수 μžˆλŠ” μ΅œλŒ€ 길이둜 기본값이 μ„€μ •λ©λ‹ˆλ‹€. λͺ¨λΈμ— νŠΉμ •ν•œ μ΅œλŒ€ μž…λ ₯ 길이가 μ—†λŠ” 경우 `max_length`에 λŒ€ν•œ μž˜λΌλ‚΄κΈ° λ˜λŠ” νŒ¨λ”©μ΄ λΉ„ν™œμ„±ν™”λ©λ‹ˆλ‹€.

λ‹€μŒ ν‘œμ—λŠ” νŒ¨λ”© 및 μž˜λΌλ‚΄κΈ°λ₯Ό μ„€μ •ν•˜λŠ” ꢌμž₯ 방법이 μš”μ•½λ˜μ–΄ μžˆμŠ΅λ‹ˆλ‹€.
μž…λ ₯으둜 μ‹œν€€μŠ€ μŒμ„ μ‚¬μš©ν•˜λŠ” 경우, λ‹€μŒ μ˜ˆμ œμ—μ„œ `truncation=True`λ₯Ό `['only_first', 'only_second', 'longest_first']`μ—μ„œ μ„ νƒν•œ `STRATEGY`, 즉 `truncation='only_second'` λ˜λŠ” `truncation='longest_first'`둜 λ°”κΎΈλ©΄ μ•žμ„œ μ„€λͺ…ν•œ λŒ€λ‘œ 쌍의 두 μ‹œν€€μŠ€κ°€ μž˜λ¦¬λŠ” 방식을 μ œμ–΄ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

| μž˜λΌλ‚΄κΈ° | νŒ¨λ”© | μ‚¬μš© 방법 |
|--------------------------------------|-----------------------------------|------------------------------------------------------------------------------------------|
| μž˜λΌλ‚΄κΈ° μ—†μŒ | νŒ¨λ”© μ—†μŒ | `tokenizer(batch_sentences)` |
| | 배치 λ‚΄ μ΅œλŒ€ 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding=True)` λ˜λŠ” |
| | | `tokenizer(batch_sentences, padding='longest')` |
| | λͺ¨λΈμ˜ μ΅œλŒ€ μž…λ ₯ 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding='max_length')` |
| | νŠΉμ • 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding='max_length', max_length=42)` |
| | λ‹€μ–‘ν•œ 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding=True, pad_to_multiple_of=8) |
| λͺ¨λΈμ˜ μ΅œλŒ€ μž…λ ₯ 길이둜 μž˜λΌλ‚΄κΈ° | νŒ¨λ”© μ—†μŒ | `tokenizer(batch_sentences, truncation=True)` λ˜λŠ” |
| | | `tokenizer(batch_sentences, truncation=STRATEGY)` |
| | 배치 λ‚΄ μ΅œλŒ€ 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding=True, truncation=True)` λ˜λŠ” |
| | | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY)` |
| | λͺ¨λΈμ˜ μ΅œλŒ€ μž…λ ₯ 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding='max_length', truncation=True)` λ˜λŠ” |
| | | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY)` |
| | νŠΉμ • 길이둜 νŒ¨λ”© | μ‚¬μš© λΆˆκ°€ |
| νŠΉμ • 길이둜 μž˜λΌλ‚΄κΈ° | νŒ¨λ”© μ—†μŒ | `tokenizer(batch_sentences, truncation=True, max_length=42)` λ˜λŠ” |
| | | `tokenizer(batch_sentences, truncation=STRATEGY, max_length=42)` |
| | 배치 λ‚΄ μ΅œλŒ€ 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding=True, truncation=True, max_length=42)` λ˜λŠ” |
| | | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY, max_length=42)` |
| | λͺ¨λΈμ˜ μ΅œλŒ€ μž…λ ₯ 길이둜 νŒ¨λ”© | μ‚¬μš© λΆˆκ°€ |
| | νŠΉμ • 길이둜 νŒ¨λ”© | `tokenizer(batch_sentences, padding='max_length', truncation=True, max_length=42)` λ˜λŠ” |
| | | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY, max_length=42)` |