hadoop로그
41, Adding block pool BP-185116165-<*>-1512025890710,['155.230.91.226']
43 Cached dfsUsed found for /home/james/hadoop-2.7.4/hdfs/data/current/BP-185116165-<>-1512025890710/current: <>,"['155.230.91.226', '12435390464']
46 Adding replicas to map for block pool BP-185116165-<*>-1512025890710 on volume /home/james/hadoop-2.7.4/hdfs/data/current...,['155.230.91.226']
51 Periodic Directory Tree Verification scan starting at <*> with interval of 21600000ms,['1514035486079ms']
52 Block pool <> (Datanode Uuid <> service to deimos26/<>:9000 <> <> <> <*>,"['BP-185116165-155.230.91.226-1512025890710', 'null)', '155.230.91.226', 'beginning', 'handshake with NN']"
15, Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter),[]
16, Added filter <> <> to context <*>,"['static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)', 'datanode']"
17, Added filter <> <> to context <*>,"['static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)', 'logs']"
18, Added filter <> <> to context <*>,"['static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)', 'static']"
154, Added filter <> <> to context <*>,"['static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)', 'secondary']"
155, Added filter <> <> to context <*>,"['static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)', 'static']"
156, Added filter <> <> to context <*>,"['static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)', 'logs']"
3385 Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter),[]
3653, Added filter <> <> to context <*>,"['RMAuthenticationFilter (class=org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter)', 'cluster']"
193, Registering class <> for class <>,"['org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType', 'org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher']"
문제점:
1. 명백히 port번호, ip_adress, node name 등 이더라도 서로 다른 토큰이 안나오면 무조건 static으로 취급한다. ex: 51, 52번과 같이 포트번호가 9000이외가 없으니..
2. 16, 17, 18번을 보면 filter 네임은 변수로 되어있지만 15번, 3385번은 static 취급한다. 왜냐하면 16, 17, 18은 15, 3385와 토큰 수가 달라 아예 다른 이벤트로 취급되기 때문이다. 다른 이벤트로 취급되기에 16, 17, 18은 filter name토큰이 달라 *로 취급하지만, 15, 3385내에서는 필터네임이 다 같기에 static으로 취급한다. 로그 의미가 이토록 비슷한데 토큰수가 달라서 다른 이벤트로 취급되는 것이 문제
Spark 로그
