[typing] Optional

Optional[int] = None

같은 코드는 뭘 의미하고 있을까..

typing module의먼저 Optional의 사용은 몇가지 type hinting과 default value에 관련되어있다.

1. Type Hinting : The Optional type is used to indicate that a variable can have either a specific type or None. In this case, Optional[int] means that the variable can either be an integer (int) or None. It's a shorthand for Union[int, None].

from typing import Optional

def foo(x: Optional[int]) -> None:
	if x is None:
    	print("x is None")
    else:
    	print(f"x is an integer : {x}")       

In the example above, x can be an int or None, and the function is designed to handle both cases.

2. Default Value: When you assign None to a variable with a type hint of Optional[int], you're providing a default value for the parameter. This is common in function signatures or class attributes where you want to give an optional parameter that defaults to None if no value is provided.

from typing import Optional

def foo(x: Optional[int] = None) -> None:
	if x is None:
    	print("x is None")
    else:
    	print(f"x is an integer : {x}")     
foo()		# Output : No value provided
foo(10)		# Output : Value provided: 10

Here, x is optional, and if the caller does not pass a value, x defaults to None.

Summary

• Optional[int]: This indicates that a variable can be either an integer (int) or None.
• = None: This assigns a default value of None to the variable, making the parameter optional in a function or class constructor.

This combination is very useful in scenarios where you want to allow an optional argument or value that can either be present (with a specificc type) or absent (represented by None). It's common in Python code where you want to handle cases where certain parameters or configuration options are optional.

@dataclass # decorator can be used to define a class that holds model configuration or hyperparameters.
class ModelArgs:
    dim: int = 4096 # dimension of QKV
    n_layers: int = 32 # # of layers
    n_heads: int = 32 # # of heads
    n_kv_heads: Optional[int] = None 
    vocab_size: int = -1
    multiple_of: int = 256 # make SwiGLU hidden layer size multiple of large power of 2
    ffn_dim_multiplier: Optional[float] = None
    norm_eps: float = 1e-5
    rope_theta: float = 500000

    max_batch_size: int = 32
    max_seq_len: int = 2048

The code snippet above is from the official LLaMA 3's ModelArgs class. In this class, the Optional type is used for the parameters n_kv_heads and ffn_dim_multiplier. I will explore the specific details of how these parameters are used layer.