hadoop(八)集群namenode启动ssh免密登录(完全分布式五)|10

2023-01-20 21:50:11

启动namenode之前：

1. 先查看有无节点启动，执行jps查看，有的话停掉

[shaozhiqi@hadoop102 ~]$ jps

10566 Jps

[shaozhiqi@hadoop102 ~]$

2. 删除每个机器上之前运行的data和logs文件

启动namenode

格式化namenode
执行命令和执行成功的部分日志

[shaozhiqi@hadoop102 hadoop-3.1.2]$ bin/hdfs namenode –format

2019-06-25 06:07:55,932 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1212972932-192.168.1.102-1561468075913

2019-06-25 06:07:55,968 INFO common.Storage: Storage directory /opt/module/hadoop-3.1.2/data/tmp/dfs/name has been successfully formatted.

2019-06-25 06:07:55,988 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/module/hadoop-3.1.2/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

2019-06-25 06:07:56,207 INFO namenode.FSImageFormatProtobuf: Image file /opt/module/hadoop-3.1.2/data/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 396 bytes saved in 0 seconds

2019-06-25 06:07:56,237 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

2019-06-25 06:07:56,260 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at hadoop102/192.168.1.102

************************************************************/

[shaozhiqi@hadoop102 hadoop-3.1.2]$

启动102的namenode,启动成功

[shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start namenode

WARNING: Use of this script to start HDFS daemons is deprecated.

WARNING: Attempting to execute replacement **"hdfs --daemon start"** instead.

[shaozhiqi@hadoop102 hadoop-3.1.2]$ ll

[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps

10945 NameNode

11087 Jps

[shaozhiqi@hadoop102 hadoop-3.1.2]$

系统提示启动用hdfs --daemon start命令，可能是新版本更新了

启动102的datanode

[shaozhiqi@hadoop102 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start datanode

WARNING: Use of this script to start HDFS daemons is deprecated.

WARNING: Attempting to execute replacement "hdfs --daemon start" instead.

查看启动结果：

[shaozhiqi@hadoop102 hadoop-3.1.2]$ jps

10945 NameNode

11265 Jps

11205 DataNode

[shaozhiqi@hadoop102 hadoop-3.1.2]$

启动103.104的datanode

[shaozhiqi@hadoop103 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start datanode

WARNING: Use of this script to start HDFS daemons is deprecated.

WARNING: Attempting to execute replacement "hdfs --daemon start" instead.

WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.

[shaozhiqi@hadoop103 hadoop-3.1.2]$ jps

2816 DataNode

2833 Jps

[shaozhiqi@hadoop103 hadoop-3.1.2]$

启动 104的datanode

[shaozhiqi@hadoop104 module]$ cd hadoop-3.1.2/

[shaozhiqi@hadoop104 hadoop-3.1.2]$ sbin/hadoop-daemon.sh start datanode

WARNING: Use of this script to start HDFS daemons is deprecated.

WARNING: Attempting to execute replacement "hdfs --daemon start" instead.

WARNING: /opt/module/hadoop-3.1.2/logs does not exist. Creating.

[shaozhiqi@hadoop104 hadoop-3.1.2]$ jps

2819 Jps

2805 DataNode

[shaozhiqi@hadoop104 hadoop-3.1.2]$

目前为止整个集群的namenode和datanode都启动了

在浏览器查看是否可以访问我们的namenode web页面，可以访问启动成功

访问namenode web页面

访问namenodeweb

问题

假如我们的集群有100台机器每个都要这样起吗？

配置脚本ssh免密登录

文件解释
Know_hosts****记录ssh访问过的计算机的公钥
Id_ras****生成的私钥
Id_ras.pub****生成的公钥
Authorized_key****存放授权过的免密访问服务器的公钥

ssh命令可以直接登录到别的机器

[shaozhiqi@hadoop102 ~]$ ssh hadoop103

shaozhiqi@hadoop103's password:

Last failed login: Tue Jun 25 06:41:19 PDT 2019 from hadoop102 on ssh:notty

There were 2 failed login attempts since the last successful login.

Last login: Tue Jun 25 06:03:37 2019 from 192.168.1.1

[shaozhiqi@hadoop103 ~]$ hostname

hadoop103

[shaozhiqi@hadoop103 ~]$

免密原理

免密登录原理图

到102找到 .ssh

[shaozhiqi@hadoop102 ~]$ ls -al

total 32

drwx------. 7 shaozhiqi shaozhiqi 4096 Jun 25 06:07 .

drwxr-xr-x. 3 root root 22 Jun 18 05:44 ..

-rw-------. 1 shaozhiqi shaozhiqi 1075 Jun 24 09:06 .bash_history

-rw-r--r--. 1 shaozhiqi shaozhiqi 18 Dec 6 2016 .bash_logout

-rw-r--r--. 1 shaozhiqi shaozhiqi 193 Dec 6 2016 .bash_profile

-rw-r--r--. 1 shaozhiqi shaozhiqi 231 Dec 6 2016 .bashrc

drwxrwxr-x. 2 shaozhiqi shaozhiqi 64 Jun 24 08:36 bin

drwxrwxr-x. 3 shaozhiqi shaozhiqi 17 Jun 18 06:04 .cache

drwxrwxr-x. 3 shaozhiqi shaozhiqi 17 Jun 18 06:04 .config

drwxr-xr-x. 4 shaozhiqi shaozhiqi 37 May 4 2017 .mozilla

drwx-----. 2 shaozhiqi shaozhiqi 24 Jun 22 23:04 .ssh

-rw-------. 1 shaozhiqi shaozhiqi 6420 Jun 25 06:07 .viminfo

-rw-------. 1 shaozhiqi shaozhiqi 165 Jun 24 16:42 .Xauthority

[shaozhiqi@hadoop102 ~]$

进入.ssh生成公钥和私钥

[shaozhiqi@hadoop102 ~]$ cd .ssh/

[shaozhiqi@hadoop102 .ssh]$ ls

known_hosts

[shaozhiqi@hadoop102 .ssh]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/shaozhiqi/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/shaozhiqi/.ssh/id_rsa.

Your public key has been saved in /home/shaozhiqi/.ssh/id_rsa.pub.

The key fingerprint is:

76:b4:c5:20:b4:bb:ef:42:81:1f:60:98:1f:56:40:c1 shaozhiqi@hadoop102

The key's randomart image is:

+--[ RSA 2048]----+

| =+=+ . |

| o E o o |

| + +. . o |

| o oo o |

+-----------------+

[shaozhiqi@hadoop102 .ssh]$ ll

total 12

-rw-------. 1 shaozhiqi shaozhiqi 1675 Jun 25 06:52 id_rsa

-rw-r--r--. 1 shaozhiqi shaozhiqi 401 Jun 25 06:52 id_rsa.pub

-rw-r--r--. 1 shaozhiqi shaozhiqi 370 Jun 23 06:05 known_hosts

[shaozhiqi@hadoop102 .ssh]$

查看生成的公钥

[shaozhiqi@hadoop102 .ssh]$ cat id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102

[shaozhiqi@hadoop102 .ssh]$

将公钥copy到103

[shaozhiqi@hadoop102 .ssh]$ ssh-copy-id hadoop103

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

shaozhiqi@hadoop103's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'hadoop103'"

and check to make sure that only the key(s) you wanted were added.

[shaozhiqi@hadoop102 .ssh]$

查看103,验证是否copy成功

[shaozhiqi@hadoop103 ~]$ cd .ssh/

[shaozhiqi@hadoop103 .ssh]$ pwd

/home/shaozhiqi/.ssh

[shaozhiqi@hadoop103 .ssh]$ ls

authorized_keys known_hosts

[shaozhiqi@hadoop103 .ssh]$ cat authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102

[shaozhiqi@hadoop103 .ssh]$

同样将公钥copy到104

[shaozhiqi@hadoop102 .ssh]$ ssh-copy-id hadoop104

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

shaozhiqi@hadoop104's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'hadoop104'"

and check to make sure that only the key(s) you wanted were added.

[shaozhiqi@hadoop102 .ssh]$

这样以后在存102到103和104是可以免密跳转的。但是往回跳跳102还是需要输入密码

[shaozhiqi@hadoop102 .ssh]$ ssh hadoop103

Last login: Tue Jun 25 06:41:28 2019 from hadoop102

[shaozhiqi@hadoop103 ~]$ ssh hadoop102

shaozhiqi@hadoop102's password:

Last login: Tue Jun 25 06:43:20 2019 from hadoop103

[shaozhiqi@hadoop102 ~]$

往回跳可以用exit直接回去

[shaozhiqi@hadoop102 ~]$ ssh hadoop103

Last login: Tue Jun 25 07:04:23 2019 from hadoop102

[shaozhiqi@hadoop103 ~]$ exit

logout

Connection to hadoop103 closed.

[shaozhiqi@hadoop102 ~]$

后来发现访问自己也需要输入密码，所以啊，还得把公钥copy给自己一份

[shaozhiqi@hadoop102 .ssh]$ ssh-copy-id hadoop102

[shaozhiqi@hadoop102 .ssh]$ ll

total 16

-rw-------. 1 shaozhiqi shaozhiqi 401 Jun 25 07:11 authorized_keys

-rw-------. 1 shaozhiqi shaozhiqi 1675 Jun 25 06:52 id_rsa

-rw-r--r--. 1 shaozhiqi shaozhiqi 401 Jun 25 06:52 id_rsa.pub

-rw-r--r--. 1 shaozhiqi shaozhiqi 555 Jun 25 07:11 known_hosts

[shaozhiqi@hadoop102 .ssh]$

为啥

要让102可以免密访问其他机器呢？因为102的有我们的namenode他需要向其他机器分发和通信
同样103机器有我们的resoucemanager,也许要去控制访问其他节点的信息，所以在103上也要去访问别的机器哈哈。。。

生成103的公钥

[shaozhiqi@hadoop103 .ssh]$ cat authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102

[shaozhiqi@hadoop103 .ssh]$ ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/home/shaozhiqi/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/shaozhiqi/.ssh/id_rsa.

Your public key has been saved in /home/shaozhiqi/.ssh/id_rsa.pub.

The key fingerprint is:

6b:da:d0:d2:ab:8f:ff:48:bb:47:04:0a:8d:28:dc:c3 shaozhiqi@hadoop103

The key's randomart image is:

+--[ RSA 2048]----+

|. o. o |

|..Eo . . |

| . .. . . |

| O +. |

| ++O+. |

+-----------------+

[shaozhiqi@hadoop103 .ssh]$ ll

total 16

-rw-------. 1 shaozhiqi shaozhiqi 401 Jun 25 06:58 authorized_keys

-rw-------. 1 shaozhiqi shaozhiqi 1679 Jun 25 07:16 id_rsa

-rw-r--r--. 1 shaozhiqi shaozhiqi 401 Jun 25 07:16 id_rsa.pub

-rw-r--r--. 1 shaozhiqi shaozhiqi 370 Jun 22 23:03 known_hosts

[shaozhiqi@hadoop103 .ssh]$

将公钥同步到102,103,104

[shaozhiqi@hadoop103 .ssh]$ ssh-copy-id hadoop102

。。。

[shaozhiqi@hadoop103 .ssh]$ ssh-copy-id hadoop103

。。。

[shaozhiqi@hadoop103 .ssh]$ ssh-copy-id hadoop104

查看是否copy成功，可以看到103多了一个103的公钥

[shaozhiqi@hadoop103 .ssh]$ cat authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDyQ5kCo8iJVhJI6et0hBW/0XYBRUKXL9CBSFZ7FMhh9gkPO+HVlsJoEDqlTzBjNwjl9aWfrogvxa+9E6EoQT81qRG6KzN3gWsrgv6k8AXphhhwoU1OFhxL3cST0y9rsKQ5p0ptwKqJD1su6q8OtB1en+d3vab++V8BmZVpc1pjjTlZHG+HwYIlHpabL+ICdloTexChNFLJ91GjlLHGSFcbp81yqRXA0gPE03YGMFIVBhu+bPjuWn72kxdO7CLqZu7nIar6yP18MvtqnzrC7klE6E8ThgpbuRQerq5o0IfNR4GSH7NvvGG51CSMHqPvaR7lvVdOkJseAjkR6kDj6Ph shaozhiqi@hadoop102

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpI16PazeKVEHHmjRuHtkjT8jl3d+3Ra7+wzUA6Xtiky/BV3zQO64oLEOi7tSkGVPgupOWT5yGXhILJFYEKvCp91rP/pQWQFPNFnX+MbM6iTBDO8eL99Ma6owcipuVCGVGw5bPnX4F+4GZSOeCIsd21pJ4CU0OjdY64IneIdRokoMDmqvLzgT+/Hjb4c+HPzx4OUwcQyzN9UrWWsL557ahVfjwickdkLgR+33aTK5IOl8XCtkuMNJxqRzrsqgPKy01f+psdRBQkX0qCO6laL5laq/7zPmTdz9JZC4rekjYa0cTbvInX1C6wV+NTaedzRs2ewoGpu/VsU17eIWdDuLR shaozhiqi@hadoop103

[shaozhiqi@hadoop103 .ssh]$

同样的去查看其他两台机器都是copy成功

验证从103跳102,104都是ok

[shaozhiqi@hadoop103 .ssh]$ ssh hadoop102

Last login: Tue Jun 25 07:04:46 2019 from hadoop103

[shaozhiqi@hadoop102 ~]$

…

102上后续需要部分root权限操作我们hadoop基群，所以也要给root用户配置免密登录，当前只实现shaozhiqi用户

继续操作在102 切到root

[shaozhiqi@hadoop102 .ssh]$ su root

Password:

ABRT has detected 2 problem(s). For more info run: abrt-cli list --since 1561257992

[root@hadoop102 .ssh]#

发现我的102尽然没有.ssh

[root@hadoop102 ~]# cd .ssh/

bash: cd: .ssh/: No such file or directory

[root@hadoop102 ~]#

执行这个命令登录一次就有了

[root@hadoop102 ~]# ssh localhost

验证

[root@hadoop102 ~]# cd .ssh

[root@hadoop102 .ssh]# ll

total 4

-rw-r--r--. 1 root root 171 Jun 25 07:34 known_hosts

[root@hadoop102 .ssh]#

生成102 root的公钥私钥

[root@hadoop102 .ssh]# ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

95:24:14:30:a6:9d:3d:37:6d:46:19:08:35:45:72:1d root@hadoop102

The key's randomart image is:

+--[ RSA 2048]----+

| +o=+=+*+E. |

| + + o.B. . |

| . o o = + |

| |

+-----------------+

[root@hadoop102 .ssh]#

同步root公钥到102，103，104

[root@hadoop102 .ssh]# ssh-copy-id hadoop102

。。。

[root@hadoop102 .ssh]# ssh-copy-id hadoop103

。。。

[root@hadoop102 .ssh]# ssh-copy-id hadoop104

。。。

验证ok

[root@hadoop102 .ssh]# ssh hadoop103

Last login: Tue Jun 25 07:32:53 2019

[root@hadoop103 ~]#

总结下

102的shaozhiqi/root可以免密登录到102,103,104
103的shaozhiqi可以免密登录到102,103,104
这一节在102启动了namenode
在102,103,104启动了datanode

下一节计划启动集群 no time do more

码农公寓

启动namenode之前：

启动namenode

问题

配置脚本ssh免密登录

免密原理

为啥

总结下

相关文章