[LangChain] Caching

캐싱(Caching)

동일한 질문을 반복해서 입력 받을 때, 매번 모델이 응답을 생성하지 않고 이전에 생성한 응답을 저장해두었다가 재사용하는 기능

👉 속도 향상: 같은 질문에 대해 모델이 다시 답변을 생성하지 않아 호출 속도 빨라짐
👉 비용 절감: OpenAI API를 호출할 때 발생하는 비용을 절약할 수 있음
👉 일관된 응답: 동일한 질문에 항상 같은 답변을 제공할 수 있음

from langchain_openai import ChatOpenAI

'''
캐싱을 사용하면 LM(언어모델)의 응답을 저장할 수 있음
예를 들어 챗봇이 항상 같은 질문을 받는다면 질문 받을 때마다 게속 답변을
생성하는 게 아니라 이미 답변한 답을 캐싱을 이용해 재사용할 수 있음
'''
from langchain.globals import set_llm_cache, set_debug
from langchain.cache import InMemoryCache, SQLiteCache

# 모든 response를 DB에 저장
set_llm_cache(SQLiteCache('cache.db'))

# 모든 response를 메모리에 저장
#set_llm_cache(InMemoryCache())

# 디버그 모드를 켜서 모델 호출 과정과 캐싱 작동 여부를 확인할 수 있음. 코드 실행 중 어떤 작업이 일어나고 있는지 로그를 확인할 수 있음
set_debug(True)

chat = ChatOpenAI(
    temperature=0.1
)

chat.predict('How to make Kimchi?')

[llm/start] [llm:ChatOpenAI] Entering LLM run with input:
{
  "prompts": [
    "Human: How to make Kimchi?"
  ]
}
[llm/end] [llm:ChatOpenAI] [2.40s] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "Ingredients:\n- 1 medium Napa cabbage\n- 1/4 cup sea salt\n- 4 cups water\n- 1 tablespoon grated ginger\n- 4 cloves garlic, minced\n- 1 tablespoon sugar\n- 3 tablespoons Korean red pepper flakes\n- 2 green onions, chopped\n- 1 small carrot, julienned\n- 1 tablespoon fish sauce (optional)\n\nInstructions:\n1. Cut the cabbage into quarters lengthwise and remove the core. Cut each quarter into 2-inch pieces.\n2. In a large bowl, dissolve the sea salt in water. Add the cabbage pieces and let them soak for 2 hours, turning occasionally.\n3. Drain the cabbage and rinse it under cold water. Squeeze out excess water and set aside.\n4. In a separate bowl, mix together the ginger, garlic, sugar, red pepper flakes, green onions, carrot, and fish sauce (if using).\n5. Add the cabbage to the mixture and toss until well combined.\n6. Pack the kimchi into a clean glass jar, pressing down to remove any air bubbles.\n7. Cover the jar loosely with a lid and let it ferment at room temperature for 1-5 days, depending on your preference for sourness.\n8. Once fermented, store the kimchi in the refrigerator. It will continue to ferment and develop flavor over time. Enjoy as a side dish or in recipes like kimchi fried rice or kimchi stew.",
        "generation_info": {
          "finish_reason": "stop",
          "logprobs": null
        },
        "type": "ChatGeneration",
        "message": {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain",
            "schema",
            "messages",
            "AIMessage"
          ],
          "kwargs": {
            "content": "Ingredients:\n- 1 medium Napa cabbage\n- 1/4 cup sea salt\n- 4 cups water\n- 1 tablespoon grated ginger\n- 4 cloves garlic, minced\n- 1 tablespoon sugar\n- 3 tablespoons Korean red pepper flakes\n- 2 green onions, chopped\n- 1 small carrot, julienned\n- 1 tablespoon fish sauce (optional)\n\nInstructions:\n1. Cut the cabbage into quarters lengthwise and remove the core. Cut each quarter into 2-inch pieces.\n2. In a large bowl, dissolve the sea salt in water. Add the cabbage pieces and let them soak for 2 hours, turning occasionally.\n3. Drain the cabbage and rinse it under cold water. Squeeze out excess water and set aside.\n4. In a separate bowl, mix together the ginger, garlic, sugar, red pepper flakes, green onions, carrot, and fish sauce (if using).\n5. Add the cabbage to the mixture and toss until well combined.\n6. Pack the kimchi into a clean glass jar, pressing down to remove any air bubbles.\n7. Cover the jar loosely with a lid and let it ferment at room temperature for 1-5 days, depending on your preference for sourness.\n8. Once fermented, store the kimchi in the refrigerator. It will continue to ferment and develop flavor over time. Enjoy as a side dish or in recipes like kimchi fried rice or kimchi stew.",
            "response_metadata": {
              "token_usage": {
                "completion_tokens": 298,
                "prompt_tokens": 13,
                "total_tokens": 311,
                "prompt_tokens_details": {
                  "cached_tokens": 0,
                  "audio_tokens": 0
                },
                "completion_tokens_details": {
                  "reasoning_tokens": 0,
                  "audio_tokens": 0,
                  "accepted_prediction_tokens": 0,
                  "rejected_prediction_tokens": 0
                }
              },
              "model_name": "gpt-3.5-turbo",
              "system_fingerprint": null,
              "finish_reason": "stop",
              "logprobs": null
            },
            "type": "ai",
            "id": "run-001f30dc-3961-423c-8c2a-1ca52dce5f3b-0",
            "tool_calls": [],
            "invalid_tool_calls": []
          }
        }
      }
    ]
  ],
  "llm_output": {
    "token_usage": {
      "completion_tokens": 298,
      "prompt_tokens": 13,
      "total_tokens": 311,
      "prompt_tokens_details": {
        "cached_tokens": 0,
        "audio_tokens": 0
      },
      "completion_tokens_details": {
        "reasoning_tokens": 0,
        "audio_tokens": 0,
        "accepted_prediction_tokens": 0,
        "rejected_prediction_tokens": 0
      }
    },
    "model_name": "gpt-3.5-turbo",
    "system_fingerprint": null
  },
  "run": null
}

chat.predict('How to make Kimchi?')

[llm/start] [llm:ChatOpenAI] Entering LLM run with input:
{
  "prompts": [
    "Human: How to make Kimchi?"
  ]
}
[llm/end] [llm:ChatOpenAI] [4ms] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "Ingredients:\n- 1 medium Napa cabbage\n- 1/4 cup sea salt\n- 4 cups water\n- 1 tablespoon grated ginger\n- 4 cloves garlic, minced\n- 1 tablespoon sugar\n- 3 tablespoons Korean red pepper flakes\n- 2 green onions, chopped\n- 1 small carrot, julienned\n- 1 tablespoon fish sauce (optional)\n\nInstructions:\n1. Cut the cabbage into quarters lengthwise and remove the core. Cut each quarter into 2-inch pieces.\n2. In a large bowl, dissolve the sea salt in water. Add the cabbage pieces and let them soak for 2 hours, turning occasionally.\n3. Drain the cabbage and rinse it under cold water. Squeeze out excess water and set aside.\n4. In a separate bowl, mix together the ginger, garlic, sugar, red pepper flakes, green onions, carrot, and fish sauce (if using).\n5. Add the cabbage to the mixture and toss until well combined.\n6. Pack the kimchi into a clean glass jar, pressing down to remove any air bubbles.\n7. Cover the jar loosely with a lid and let it ferment at room temperature for 1-5 days, depending on your preference for sourness.\n8. Once fermented, store the kimchi in the refrigerator. It will continue to ferment and develop flavor over time. Enjoy as a side dish or in recipes like kimchi fried rice or kimchi stew.",
        "generation_info": {
          "finish_reason": "stop",
          "logprobs": null
        },
        "type": "ChatGeneration",
        "message": {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain",
            "schema",
            "messages",
            "AIMessage"
          ],
          "kwargs": {
            "content": "Ingredients:\n- 1 medium Napa cabbage\n- 1/4 cup sea salt\n- 4 cups water\n- 1 tablespoon grated ginger\n- 4 cloves garlic, minced\n- 1 tablespoon sugar\n- 3 tablespoons Korean red pepper flakes\n- 2 green onions, chopped\n- 1 small carrot, julienned\n- 1 tablespoon fish sauce (optional)\n\nInstructions:\n1. Cut the cabbage into quarters lengthwise and remove the core. Cut each quarter into 2-inch pieces.\n2. In a large bowl, dissolve the sea salt in water. Add the cabbage pieces and let them soak for 2 hours, turning occasionally.\n3. Drain the cabbage and rinse it under cold water. Squeeze out excess water and set aside.\n4. In a separate bowl, mix together the ginger, garlic, sugar, red pepper flakes, green onions, carrot, and fish sauce (if using).\n5. Add the cabbage to the mixture and toss until well combined.\n6. Pack the kimchi into a clean glass jar, pressing down to remove any air bubbles.\n7. Cover the jar loosely with a lid and let it ferment at room temperature for 1-5 days, depending on your preference for sourness.\n8. Once fermented, store the kimchi in the refrigerator. It will continue to ferment and develop flavor over time. Enjoy as a side dish or in recipes like kimchi fried rice or kimchi stew.",
            "response_metadata": {
              "token_usage": {
                "completion_tokens": 298,
                "prompt_tokens": 13,
                "total_tokens": 311,
                "prompt_tokens_details": {
                  "cached_tokens": 0,
                  "audio_tokens": 0
                },
                "completion_tokens_details": {
                  "reasoning_tokens": 0,
                  "audio_tokens": 0,
                  "accepted_prediction_tokens": 0,
                  "rejected_prediction_tokens": 0
                }
              },
              "model_name": "gpt-3.5-turbo",
              "system_fingerprint": null,
              "finish_reason": "stop",
              "logprobs": null
            },
            "type": "ai",
            "id": "run-001f30dc-3961-423c-8c2a-1ca52dce5f3b-0",
            "tool_calls": [],
            "invalid_tool_calls": []
          }
        }
      }
    ]
  ],
  "llm_output": null,
  "run": null
}

❓ 캐싱은 꼭 동일한 입력에만 가능한가?

기본적으로 LangChain의 캐싱은 "동일한" 질문에만 캐싱된 결과를 반환. 즉, 입력 문자열이 정확히 같아야 캐싱된 응답을 가져올 수 있음. 비슷한 질문에 대해서도 캐싱 결과를 활용하려면 추가적인 설정이나 작업이 필요.

❓ 비슷한 질문에도 캐싱 결과 활용하는 방법

1. 입력 질문을 임베딩(Embedding)으로 변환

• 질문의 의미를 수치 벡터로 변환하여 유사도를 계산
• 새로운 질문 들어왔을 때 기존 질문들과 임베딩 기반 유사도를 비교해 가장 비슷한 질문 검색

2. 유사도 기반 검색

• 기존 질문과 새로운 질문의 유사도를 계산한 후, 유사도가 특정 임계값 이상인 경우 캐싱 결과 반환
• 이를 위해 추가적으로 벡터 데이터베이스(Vector Store) 활용 가능

👉 반복성 높은 질문이 많다면 임베딩 기반 검색이 장기적으로 비용을 절감할 가능성 높음
👉 질문이 항상 새롭다면 단순히 새로 응답을 생성하는 것이 더 경제적일 수 있음