DailyBriefing8.27

탁가이버·2일 전

DailyBriefing

목록 보기
2/2

Daily Briefing — Thursday, August 27, 2026


🧠 Data Science Interview Practice

Topic Category: Feature Engineering & Data Preprocessing (rotation: day-of-month 27 mod 8 = 3)

Question:
You're building a churn prediction model for a subscription business. The dataset has a signup_date column, a last_login_date column, a plan_type categorical column with 40 unique values (many rare), and a monthly_spend column that's heavily right-skewed with some negative values from refunds. Walk me through how you'd engineer features from these columns and handle the data quality issues before modeling.

Model Answer:

  • Date columns → derived features, not raw dates. Convert signup_date and last_login_date into model-usable signals: tenure_days (today − signup_date), days_since_last_login (recency, often one of the strongest churn predictors), and calendar features like signup month/quarter if seasonality matters. Never feed raw datetimes into most models — they can't use them meaningfully.

  • High-cardinality categorical (plan_type, 40 levels) → don't one-hot blindly. One-hot encoding 40 rare-heavy levels blows up dimensionality and overfits on sparse categories. Better options: (1) group rare levels (e.g., <1% frequency) into an "Other" bucket, (2) target/mean encoding with proper cross-validation or smoothing to avoid leakage, or (3) frequency encoding if the model is tree-based and cardinality alone carries signal.

  • Negative values in monthly_spend → investigate before transforming. Negative values from refunds aren't noise — they're a valid signal (a refund could itself predict churn). Decide explicitly: keep as-is if the model can use the sign, or split into two features (gross_spend and refund_flag/refund_amount) so the model doesn't conflate "spent little" with "got refunded."

  • Right-skewed spend → transform for linear/distance-based models. Apply a log1p or Box-Cox transform (log1p handles zero/near-zero gracefully; Box-Cox needs positive values, so you'd have to shift first given the negatives). Tree-based models (XGBoost, random forest) are scale-invariant and don't strictly need this, so the choice depends on the downstream algorithm.

  • Leakage check. Make sure none of these engineered features accidentally encode the future — e.g., if last_login_date is captured after the churn event, days_since_last_login becomes a direct leak of the label. Time-box all features to a clear "as of" cutoff before the churn window.

  • Scaling. For linear models, SVMs, or KNN, standardize/normalize the numeric features after transformation. Not needed for tree ensembles.

import pandas as pd
import numpy as np

df["tenure_days"] = (pd.Timestamp.now() - df["signup_date"]).dt.days
df["days_since_last_login"] = (pd.Timestamp.now() - df["last_login_date"]).dt.days

# Bucket rare plan types
freq = df["plan_type"].value_counts(normalize=True)
rare = freq[freq < 0.01].index
df["plan_type_grouped"] = df["plan_type"].where(~df["plan_type"].isin(rare), "Other")

# Separate refund signal from spend magnitude
df["refund_flag"] = (df["monthly_spend"] < 0).astype(int)
df["gross_spend"] = df["monthly_spend"].clip(lower=0)
df["gross_spend_log"] = np.log1p(df["gross_spend"])

Likely Follow-up: "How would target encoding on plan_type leak information into your validation set, and how would you prevent it?"
(Expected answer: fitting the encoding on the full dataset before the train/val split lets validation rows influence their own encoded value; the fix is to compute encodings within each CV fold — or with an out-of-fold scheme — so no row's target ever contributes to its own encoded feature.)


🇰🇷 Korea Morning News

  1. Rescue Mission for Korean Workers Stranded in Nepal. Seoul is dispatching a team of Foreign Ministry officials, firefighters, and police to Nepal after nine Korean workers went unreachable, reportedly amid flooding/landslide disruption. (Source: Al Jazeera)

  2. Trump Scales Back Joint US–Korea Military Exercises. Donald Trump has ordered the US military to reduce the scope of joint drills with South Korea, citing his "very good relationship" with North Korea's Kim Jong Un — a move with significant implications for the alliance's deterrence posture. (Source: South China Morning Post)

  3. Record Jump in Births. South Korea's June births rose 15.6% year-on-year to 23,111 — the 24th straight month of growth and the sharpest percentage increase since record-keeping began in 1981, per Yonhap. The absolute increase was the largest since 1992. (Source: Yonhap News Agency, via The Shillong Times)

  4. Tech Exports Keep Climbing on AI Chip Demand. South Korea's ICT exports topped $50 billion for a second straight month, with July exports hitting the second-highest level on record, driven by semiconductors and eco-friendly vehicles; Samsung and SK Hynix featured in a late-July AI/chip summit. (Source: Korea Economic Daily / KED Global)

  5. Conservative Opposition Under Pressure to Reform. Korea's main conservative opposition party faces mounting pressure to broaden its appeal and rebuild public trust following recent political setbacks. (Source: South China Morning Post)


📈 US Stock Market Briefing

Overall sentiment: Mixed-to-flat. The S&P 500 closed essentially unchanged around 7,675.70; the Nasdaq Composite slipped 0.08% to 26,130.20; the Dow lost 113.52 points (−0.21%) to 53,463.88. Chip stocks were a drag on the Nasdaq/Dow even as the broader market held steady.

Notable movers/sectors:

  • Chipmakers weak: Nvidia and peers pulled back (Nvidia down as much as ~2.3% in intraday futures trade on semiconductor-sector weakness), despite Nvidia's own blockbuster earnings beat driven by continued AI infrastructure demand — a "sell the news" dynamic.
  • Meta +1%: Shares rose after Meta reached a settlement with state attorneys general over allegations its social apps harmed young users, removing a legal overhang.
  • Commodities diverging: Oil prices fell while gold surged, consistent with a modest risk-off/inflation-hedge tilt in parts of the market.

Macro factors: The July PCE price index — the Fed's preferred inflation gauge — came in slightly hotter than expected, though core PCE matched expectations. The mild upside surprise didn't meaningfully shift odds for the September FOMC meeting, but it reinforced that inflation remains sticky enough to keep the Fed cautious.


Sources:

쉽게 말하면, 이 문제는 “날짜·범주형·숫자 데이터가 각각 문제가 있는데, 모델이 잘 이해할 수 있는 형태로 어떻게 바꿀 것인가?”를 묻는 질문입니다.

1. 날짜 → “얼마나 오래됐나?”로 바꾼다

signup_date = 가입일
last_login_date = 마지막 로그인 날짜

모델에게 날짜 자체를 2024-03-15처럼 넣어주는 것보다 의미 있는 숫자로 바꾸는 게 좋습니다.

예를 들어:

  • tenure_days = 가입 후 지금까지 며칠?
  • days_since_last_login = 마지막 로그인 후 며칠?

예:

가입한 지 500일 + 마지막 로그인 30일 전 → 오래된 고객이고 최근 활동도 없음 → churn 가능성 ↑

특히 days_since_last_login은 churn 예측에서 매우 중요한 feature가 될 수 있습니다.

단, 중요한 것이 data leakage입니다.

만약 고객이 이미 churn한 뒤의 로그인 데이터를 사용했다면?

“churn한 고객은 로그인 안 했다”를 미리 알고 있는 셈

이므로 모델이 부정행위를 하는 것입니다.

따라서 “예측하는 시점”을 정하고 그 시점 이전의 데이터만 사용해야 합니다.


2. plan_type 40개 → 그대로 One-hot 하지 않는다

예를 들어 plan이:

Basic, Premium, Gold, Silver, A, B, C, ... 40개

라면 One-hot encoding을 하면 40개의 column이 생깁니다.

특히 이런 문제가 있습니다.

Plan A: 고객 10,000명
Plan B: 고객 8,000명
Plan Z: 고객 3명

Plan Z 같은 아주 희귀한 category는 모델이 우연한 패턴을 학습하기 쉽습니다.

그래서 보통:

방법 1 — 희귀 category 합치기

고객의 1% 미만인 plan → Other

즉,

A, B, C, ... , Z

A, B, C, ... , Other


방법 2 — Target Encoding

각 plan의 churn rate를 숫자로 바꾸는 방법입니다.

예:

Plan고객 수Churn rate
Basic10,00010%
Premium8,0005%
Gold5,0003%
Rare X1040%

Rare X가 40%라고 해서 그대로 믿으면 위험합니다. 고객이 10명밖에 없기 때문입니다.

그래서 smoothing + cross-validation을 사용해서 과도하게 학습하지 않도록 합니다.


3. monthly_spend의 음수 → 무조건 제거하면 안 된다

예를 들어:

  • 고객 A: $100
  • 고객 B: $50
  • 고객 C: -$30

-$30은 오류가 아니라 refund 때문이라고 했습니다.

따라서 단순히

"음수니까 이상치 → 삭제"

하면 안 됩니다.

오히려 환불을 받은 고객이 churn할 가능성이 높은지 알아볼 수 있습니다.

그래서 예를 들어:

  • gross_spend
  • refund_amount
  • refund_flag

처럼 분리할 수 있습니다.

예:

$100 사용 + $30 refund

→ 단순히 monthly_spend = $70이라고만 하면
“70달러를 쓴 고객”과 “100달러를 쓰고 30달러 환불받은 고객”을 구별할 수 없습니다.

이 차이가 churn prediction에 중요할 수 있습니다.


4. Spend가 오른쪽으로 치우침 → Log 변환을 생각한다

예를 들어 고객 대부분은:

$10, $20, $30, $50

인데 일부 고객은:

$1,000, $5,000, $20,000

이라면 데이터가 이렇게 한쪽으로 길게 늘어집니다.

이를 right-skewed라고 합니다.

이때 linear model에서는 log1p 같은 변환을 사용할 수 있습니다.

쉽게 말하면:

$10 → 조금 변함
$100 → 중간 정도
$10,000 → 너무 큰 숫자의 영향력을 줄임

그래서 모델이 극단적인 고액 고객 몇 명 때문에 지나치게 영향을 받는 것을 줄일 수 있습니다.

다만 XGBoost나 Random Forest 같은 tree model에서는 꼭 할 필요가 없습니다.


5. 모델 종류에 따라 Scaling도 다르다

예를 들어 feature가:

  • tenure = 500
  • days since login = 30
  • spending = 100

처럼 서로 scale이 다르면,

Logistic Regression, SVM, KNN에서는 standardization을 고려합니다.

예:

평균 0, 표준편차 1 정도로 변환

하지만 Random Forest / XGBoost 같은 tree model은 보통 scaling이 필요 없습니다.


결국 면접에서는 이렇게 생각하면 됩니다

이 문제를 만나면 5단계로 생각하면 아주 쉽습니다.

① 날짜

Raw date ❌ → tenure / recency 같은 숫자로 변환

② Category

40개 category를 무조건 One-hot ❌ → rare category 처리 + target/frequency encoding 고려

③ Spend

음수 = 오류라고 삭제 ❌ → refund라는 business signal인지 확인

④ Skew

심하게 치우친 숫자 → linear model이면 log transformation 고려

⑤ Leakage

가장 중요 ⭐
예측 시점 이후의 정보가 feature에 들어갔는지 확인

한 문장으로 답하면

“각 column을 단순히 변환하는 것이 아니라, business meaning을 살리면서 모델이 사용할 수 있는 feature로 만들고, 특히 rare category·refund·skewness·data leakage를 확인하겠습니다.”

면접에서는 사실 ⑤ Data Leakage를 제대로 언급하는 것이 상당히 중요합니다.

I’d turn the date fields into features like tenure and days since last login, group rare plan types, and handle monthly_spend negatives as potential refund signals rather than simply deleting them.
I’d also consider a log transformation for the skewed spend, use scaling when appropriate, and most importantly, make sure all features are based only on information available before the churn prediction date.

Target encoding can leak information if I calculate each plan’s churn rate using the entire dataset, including the validation set.
To prevent this, I’d calculate the encoding only from the training fold and apply that mapping to the validation fold, using cross-validation and smoothing when needed.

profile
더 나은 세상은 가능하다를 믿고 실천하는 활동가

0개의 댓글