Incrédulos Anónimos

Motivation 1 Description of the process step by step 2 Preliminars 2 Create folder to save the code/notebooks to interact with Spark 2 Run docker container 2 See your docker running container, stop them, delete them 2 See your docker running container, stop them, delete them 6 References 6 Annex A. sudo usermod mystery 6 Specific references 7 Annex B: Your containers 7 Specific references 8  Motivation I want to use Spark with Jupyter and Python in a simple way to start getting results very soon. I am a data scientist I want to do things with the data. I am going to use Docker containers the idea is that I do not want to create a configure a VM for Spark, Anaconda and so on. Also I want to learn more Docker. So I am going to follow some tutorials to create my cool work environment. IN ORDER TO TO DO THAT I COPY AND ADAPT this really awesome blog entrance: http://maxmelnick.com/2016/06/04/spark-docker.html . Also I am using this also cool...

Set up cluster manually: Spark 1.3.1 en Ubuntu 14.04 Introduction When the cluster is running let configure Spark 1.3.1 and assuming Anaconda install to provide numpy. Set up password-less SSH Contact each node: ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.127 ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.128 In the master ubuntu@master:~$ ssh-keygen -t rsa -P "" ubuntu@master:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys chmod 644 $HOME/.ssh/authorized_keys ubuntu@master:~$ ssh localhost with $HOME = /home/ubuntu On workers: copy ~/.ssh/id_dsa.pub from your master to the worker, then use: cat /home/ubuntu/.ssh/id_rsa.pub >> /home/ubuntu/.ssh/authorized_keys chmod 644 /home/ubuntu/.ssh/authorized_keys References: https://gist.github.com/samklr/75486c2d9e31c5998443 http://www.thecloudavenue.com/2012/01/how-to-setup-password-less-ssh-to.html http://why-not-...