λ°˜μ‘ν˜•

μ‹±κΈ€λ…Έλ“œμ—μ„œ ν•˜λ‘‘μ„ μ–΄λ–»κ²Œ μ„€μΉ˜ν•˜λŠ”μ§€ μž‘μ„±ν•˜λ €ν•œλ‹€.

ν•˜λ‘‘μ„ μ„€μΉ˜ν•˜κΈ° 전에 기본적으둜 osμ—μ„œ 섀정을 λ°”κΏ”μ•Ό ν•˜λŠ”κ²Œ μžˆλ‹€.

일단 기본적으둜 os 버전은 cent os7.x 이닀.

인프라λ₯Ό μš΄μ˜ν•  λ•Œ root 계정은 μ‚¬μš©ν•˜μ§€ μ•ŠλŠ”λ‹€.

 

λ‹€λ₯Έ 계정을 μƒμ„±ν•˜μ—¬ sudo κΆŒν•œμ„ λΆ€μ—¬λ°›κ³  μ΅œλŒ€ν•œ κ·Έ κ³„μ •μœΌλ‘œ μ„€μΉ˜λ₯Ό μ§„ν–‰ν•œλ‹€.

 

ν•˜λ‘‘μ„ μ„€μΉ˜ν•˜κΈ° 전에?

방화벽을 내리고 selinux disabled 그리고 μœ μ €κ³„μ •μ„ μΆ”κ°€ν•˜μ—¬ λ˜λ„λ‘ root계정을 μ‚¬μš©ν•˜μ§€ μ•Šκ³  μ§„ν–‰ν•˜λ €ν•œλ‹€.
μžλ°”λŠ” open jdk 1.8버전을 μ„€μΉ˜ν•˜μ—¬ ν™˜κ²½λ³€μˆ˜ μΆ”κ°€ν•œλ‹€. 

ν•΄λ‹Ή λ‚΄μš©μ€ 좔후에 μΆ”κ°€μ˜ˆμ •μ΄λ‹€.

ν•˜λ‘‘ μ„€μΉ˜

μ„€μΉ˜ 버전 : Apache Hadoop 3.1.1

Β 

Apache Hadoop 3.1.1 – Hadoop Cluster Setup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ sudo yum install openssh*
 
$ wget https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz
$ sudo tar xvzf hadoop-3.1.1.tar.gz -/home/hadoop
$ cd /home/hadoop
 
$ vi .bash_profile
#JAVA
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
ClASSPATH=$JAVA_HOME/lib/*:$CLASSPATH
 
#HADOOP
export HADOOP_HOME=/data/platform/hadoop-3.1.1
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PATH CLASSPATH 
 
$ source .bash_profile
 
cs

 

ν•˜λ‘‘ μ„€μΉ˜ ν›„ ν•˜λ‘‘ config μ„€μ • ν•˜κΈ° [μ‹±κΈ€λ…Έλ“œ κΈ°μ€€// ν΄λŸ¬μŠ€ν„°μΌ 경우 port, hostname λ“±λ“± κ³ λ €ν•˜μ—¬ 섀정해야함]

 

hadoop-env.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh
 
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
 
# To prevent accidents, shell commands be (superficially) locked
# to only allow certain users to execute certain subcommands.
# It uses the format of (command)_(subcommand)_USER.
#
# For example, to limit who can execute the namenode command,
# export HDFS_NAMENODE_USER=hdfs
 
# export HADOOP_CLASSPATH=
 
export HDFS_NAMENODE_USER="user_name"
export HDFS_DATANODE_USER="user_name"
export HDFS_SECONDARYNAMENODE_USER="user_name"
cs

yarn-env.sh

1
2
3
4
$ vi $HADOOP_HOME/etc/hadoop/yarn-env.sh
#YARN USER SETTING
export YARN_RESOURCEMANAGER_USER="user_name"
export YARN_NODAMANAGER_USER="user_name"
cs

core-site.xml

1
2
3
4
5
6
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
cs

 

hdfs-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<configuration>
    <property>
      <name>dfs.replication</name>
      <value>1</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-bind-host</name>
      <value>0.0.0.0</value>
    </property>
    <property>
      <name>dfs.namenode.servicerpc-bind-host</name>
      <value>0.0.0.0</value>
    </property>
    <property>
      <name>dfs.namenode.http-bind-host</name>
      <value>0.0.0.0</value>
    </property>
    <property>
      <name>dfs.namenode.https-bind-host</name>
      <value>0.0.0.0</value>
    </property>
    <property>
      <name>dfs.client.datanode-restart.timeout</name>
      <value>30</value>
    </property>
</configuration>
cs

mapred-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>1024</value>
  </property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>2560</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>hostname:10020</value>
 </property>
</configuration>
cs

 

yarn-site.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hostname:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hostname:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hostname:8031</value>
    </property>    
</configuration>
cs

 

μ—¬κΈ°μ„œ λ°μ΄ν„°λ…Έλ“œ, λ„€μž„λ…Έλ“œμ˜ ν¬νŠΈλ²ˆν˜ΈλŠ” μΆ”ν›„ λ‹€λ₯Έ ν•˜λ‘‘μ—μ½”μ‹œμŠ€ν…œ μ„€μΉ˜ν•  λ•Œ 도 λ™μΌν•˜κ²Œ μ„€μ •ν•΄μ€˜μ•Όν•˜λŠ” λΆ€λΆ„μ΄μžˆλ‹€.

 

 

* 1GB = 1024 MB

* 1GiB = 1000MB

 

728x90
λ°˜μ‘ν˜•

+ Recent posts