[논문 리뷰] CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses

Introduction

• LLMs pose potential risks like generating unethical content
• Assessing LLMs' values helps expose misalignment
• Reference-free evaluators (fine-tuned LLMs or closed-source models like GPT-4) are used
• Evaluators face challenges in adaptability to human values and generalizability across scenarios
  • Adaptability Challenge: Values are diverse, cultural, and evolving, making it difficult for evaluators to align with changing human perspectives. (e.g. closed-source models like GPT-4)
  • Generalizability Challenge: Evaluators need to robustly identify values across varied expressions, scenarios, and domains. (e.g. fine-tuned LLMs)

• Open-ended value evaluation is preferred as it measures true value conformity

Methodology & CLAVE Framework

Overall

Two complementary LLMs that work together through fundamental "value concepts"

1. A large model (concept extractor) that leverages its extensive knowledge to extract high-level value concepts from diverse responses
   1. Value concepts (e.g., "advocating for personal choice in decisions" → “self-direction”) serve as generalized indicators
2. A small model (value recognizer) fine-tuned on these concepts to make value judgments

This approach enables calibration with arbitrary value systems using fewer than 100 human-labeled samples per value type.

Step 1: Value Concept Extraction (concept pool construction is more appropriate?)

• The large LLM analyzes training samples to identify highly generalized value concepts
• For each value dimension v, a concept set Cv is created from annotated training data
• Sample batches (extract value concept by batch) are constructed by clustering semantically similar examples
• Key criteria for concepts: they must be essential, generalized, and decoupled
• Concepts undergo hierarchical clustering to aggregate similar ideas and retain representative concepts

Step 2: Value Concept Mapping

• For test samples, the concept extractor identifies relevant concepts
• These newly extracted concepts are mapped to the existing concept pool created during training
• A similarity threshold θ determines if a concept should be mapped to existing concepts or used directly

Step 3: Value Recognition

• The small LLM takes the text, value definition, and mapped value concepts as input
• It's fine-tuned using LoRA and negative-log-likelihood loss to predict the value label
• The system outputs whether the response adheres to, opposes, or is unrelated to the value

ValEval Benchmark

Overall

• Contains 13,000+ annotated (text, value, label) samples across diverse domains
• Covers three major value systems:
  • Social Risk Taxonomy (safety issues like hate speech, violence, etc.)
    • original from BeaverTails (QA pairs with Alpaca-7B responses)
    • generalization set from Do-not-Answer dataset
  • Schwartz Theory of Basic Values (universal human values)
    • from ValueFulcra dataset (ValueFulcra includes ValueNet, Webis-arg-22)
    • generalization set from Do-not-Answer dataset
  • Moral Foundations Theory (moral decision-making foundations)
    • from DenEvil
    • generalization set from Moral Stories benchmark
• Three test sets for each value system: original, perturbed, and generalized
  • Original (i.i.d.): Primary split for main evaluation
  • Perturbation: Contains perturbed versions to evaluate robustness
  • Generalization: Out-of-distribution samples to evaluate cross-scenario generalizability (another dataset)
• Human annotators with psychology/sociology backgrounds ensured label quality

Data Preparation

• Preprocessing: Removed noisy and extreme samples (empty texts, special characters, outlier lengths)
• Annotation: Recruited qualified annotators with psychology/sociology backgrounds
• Quality Control: Each sample received annotations from 3 different annotators / employ majority voting
• Agreement Rates: 87.7% (Social Risks), 85.0% (Schwartz), 72.6% (Moral Foundation)

Perturbation Methods

• Two Types of Perturbations:
  • Modifying value-irrelevant text (synonyms, paraphrasing, repetition)
  • Minimal changes to value-relevant parts to flip labels
• Generation Process: Used Mistral-Large API to avoid evaluation leakage with GPT-4

Experimental Results

• Evaluators tested: 15+ LLM evaluators (prompt-based, tuning-based, and value-specific evaluators)
• Experimental settings: Vanilla prompts, few-shot, CoT, tuning small/large LLMs

• Key findings
  • Prompt-based methods excel on social risks but struggle with less common value theories
  • Fine-tuned models perform well across value systems but lack robustness to perturbations
  • CLAVE achieves optimal balance between adaptability and generalizability
  • CLAVE shows superior data efficiency (performs well with just 10-20 samples per value)
• Stronger large and small LLMs yield better performance.
• Large LLMs play a more pivotal role in enhancing performance in the CLAVE framework
• The optimal combination depends on the application scenarios.

Case Studies

• Large LLMs struggle with less popular value theories
• textual perturbations affect smaller models but not CLAVE
• value concepts enable generalization across different scenarios

Conclusion

• CLAVE achieves superior balance between adaptability and generalizability
• Value concepts enhance accuracy, generalization, and potential transparency
• The dual-model approach calibrates efficiently with minimal training data
• Future work could explore transparency aspects of value concepts and multilingual analysis

Takeaways

• Problem definition: Precisely identified generalizability and adaptability challenges in value evaluation
• Novel idea: Solved through complementary large-small model architecture