spark 1.6-HA-zookeeper 2.4.6安装

准备工作

  • zookeeper安装包zookeeper-3.4.6.tar.gz
  • s0,s1,s2 三台服务器配置zookeeper环境
  • 先在s0配置好，在把配置文件夹同步 $ZOOKEEPER_HOME/conf
  • 每台机器要单独启动，关闭

操作步骤

  • 解压压缩包

    tar -zxvf /opt/modules/bigdata/zookeeper/zookeeper-3.4.6.tar.gz -C /opt/modules/bigdata/zookeeper 即 $ZOOKEEPER_HOME=/opt/modules/bigdata/zookeeper/zookeeper-3.4.6

  • 复制文件 $ZOOKEEPER_HOME/conf/zoo_sample.cfg 为 $ZOOKEEPER_HOME/conf/zoo.cfg 增加内容

      dataDir=/opt/modules/bigdata/zookeeper/zookeeper-3.4.6/data
  dataLogDir=/opt/modules/bigdata/zookeeper/zookeeper-3.4.6/logs
  server.0=s0:2888:3888
  server.1=s1:2888:3888
  server.2=s2:2888:3888
  • 新建目录logs ==> $ZOOKEEPER_HOME/logs
  • 新建目录data ==> $ZOOKEEPER_HOME/data

  • 增加文件myid ==> $ZOOKEEPER_HOME/data/myid

    myid s0内容为：
0
myid s1内容为：
1
myid s2内容为：
2

    命令

  • 需要分别启动，停止每一台机器
  • ).启动 zookeeper： /opt/modules/bigdata/zookeeper/zookeeper-3.4.6/bin/zkServer.sh start
  • ).停止 zookeeper： /opt/modules/bigdata/zookeeper/zookeeper-3.4.6/bin/zkServer.sh stop
  • ).状态 zookeeper： /opt/modules/bigdata/zookeeper/zookeeper-3.4.6/bin/zkServer.sh status

jps 验证

s0,s1,s2 每台机器都有进程：QuorumPeerMain

修改spark配置

用zookeeper做spark的HA

  • 修改文件 $SPARK_HOME/conf/spark-env.sh

      # zookeeper 来维护状态信息
  #SPARK_MASTER_IP=s0
  export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=s0:2181,s1:2181,s2:2181 -Dspark.deploy.zookeeper.dir=/spark"

  • 同步所有节点配置文件

  • 主节点启动

      /opt/modules/bigdata/spark/spark-1.6.0-bin-hadoop2.6/sbin/start-all.sh

  • 从节点每台启动

      /opt/modules/bigdata/spark/spark-1.6.0-bin-hadoop2.6/sbin/start-master.sh
  • shell spark
      zookeeper master shell /opt/modules/bigdata/spark/spark-1.6.0-bin-hadoop2.6/bin/spark-shell  --master spark://s0:7077,s1:7077,s2:7077