관리 메뉴

data_lab

Hadoop HA μ‹€ν–‰ λ³Έλ¬Έ

BIGDATA/ν•˜λ‘‘μ—μ½”μ‹œμŠ€ν…œ

Hadoop HA μ‹€ν–‰

🐰히히 2022. 5. 25. 16:55

μ„œλ²„ ꡬ성도 μ˜ˆμ‹œ

  Zookeeper HDFS YARN MapReduce
Master quorumpeermain  journal node /Name Node
(active)
ResourceManager Job Histroyserver
Standby Master  quorumpeer journal node/
Name Node
(stand by)
ResourceManager  
Worker 1 quorumpeer journal node/
Data Node 1
NodeManager  
Worker 2   Data Node 2 NodeManager  
Worker 3   Data Node 3 NodeManager  

1.  zookeeper μ‹€ν–‰

- 각 μ„œλ²„ zkServer.sh start 

- quorumpeer, quorumpeermain μ‹€ν–‰ 됨

- leader, follower 선정은 λžœλ€μž„

2. journal node μ‹€ν–‰ - zkμ„œλ²„ μ‹€ν–‰ν•œ μ„œλ²„μ—μ„œ 싀행함 (Namenode HAλ₯Ό μœ„ν•¨)

- μ‹€ν–‰ν•΄μ•Ό 8485 포트 μ—΄λ¦Ό. μ‹€ν–‰μ•ˆν•˜λ©΄ λ„€μž„λ…Έλ“œ ν¬λ§·μ‹œ μ—λŸ¬λœΈ

3. active Namenode λ₯Ό μ‹€ν–‰ν•  μ„œλ²„μ—μ„œ μž‘μ—…

- Namenode resset

- Namenode start

4. standby Namenode μ‹€ν–‰ν•  μ„œλ²„μ—μ„œ μž‘μ—…

- active Namenode의 metadata 볡사

- hdfs namenode -bootstrapStandby

- Namenode start

5. namenode μ„œλ²„μ— zookeeper failcontroller μ‹€ν–‰

6. Worker 1-3 μ„œλ²„μ—μ„œ Datanode μ‹€ν–‰

7. RM, NM 각 μ„œλ²„μ—μ„œ μ‹€ν–‰

- Resource Manager 도 HA ꡬ성 κ°€λŠ₯함

8. Master μ„œλ²„μ—μ„œ MR job historyserver 데λͺ¬ μ‹€ν–‰

 

 

728x90
λ°˜μ‘ν˜•