Set up and initial commands of my Docker Spark Toy




Motivation

I want to use Spark with Jupyter and Python in a simple way to start getting results very soon. I am a data scientist I want to do things with the data.
I am going to use Docker containers the idea is that I do not want to create a configure a VM for Spark, Anaconda and so on. Also I want to learn more Docker. So I am going to follow some tutorials to create my cool work environment.
IN ORDER TO TO DO THAT I COPY AND ADAPT this really awesome blog entrance: http://maxmelnick.com/2016/06/04/spark-docker.html. Also I am using this also cool image: https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook where you can find a lot of information about possible uses and how to do them. This two pieces are amazin.

Description of the process step by step

Preliminars

I have ubuntu 16.04 and Docker installed (here if you want to install it: https://docs.docker.com/engine/installation/linux/ubuntu/#recommended-extra-packages-for-trusty-1404 ).
To avoid write many times sudo in front of every order to docker I create a Docker Group
sudo usermod -aG docker raf
In my ignorance I did not understand the previous line so I did some research which is in Annex A.

Create folder to save the code/notebooks to interact with Spark

cd /home/raf/Documents
mkdir spark-docker

Run docker container

Note: you should have the container!
For example we are going to work with this container: jupyter/pyspark-notebook which can be found in https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook. Then we can just write in the terminal: docker pull  jupyter/pyspark-notebook. For more detail see Annex B.

See your docker running container, stop them, delete them

This could be useful in case something goes wrong you can start quickly again.
To see the running containers: docker ps -a --no-trunc
To stop them docker stop ID-of-the -container/name-of-the -container
To delete them: docker rm ID-of-the -container/name-of-the -container


docker run -it -p 8888:8888 -v /home/raf/Documents/spark-docker --name spark_2   jupyter/pyspark-notebook



What’s going on when we run that command?
  • The -it runs the container in interactive mode..
  • The -p 8888:8888 makes the container’s port 8888 accessible to the host (i.e., your local computer) on port 8888. This will allow us to connect to the Jupyter Notebook server since it listens on port 8888.
  • The -v /home/raf/Documents/spark-docker allows us to map our spark-docker folder ( to the container’s /home/raf/Documents/spark-docker working directory (i.e., the directory the Jupyter notebook will run from). This makes it so notebooks we create are accessible in our spark-docker folder on our local computer. It also allows us to make additional files such as data sources (e.g., CSV, Excel) accessible to our Jupyter notebooks.
  • The --name spark2 gives the container the name spark, which allows us to refer to the container by name instead of ID in the future.
To stop them docker stop spark2
To delete them: docker rm spark2
  • The final part of the command, jupyter/pyspark-notebook tells Docker we want to run the container from the jupyter/pyspark-notebook image.
For more information about the docker run command, check out the https://docs.docker.com/engine/reference/run/#name---name.
As you can see you are provide with a url, you just have to go to your browser and start working. If you want to change the toker there is some clues in the github of the original:

Test

Just write some code:
import pyspark
sc = pyspark.SparkContext('local[*]')

# do something to prove it works
rdd = sc.parallelize(range(1000))
rdd.takeSample(False, 5)
Everything it is fine. Enjoy and good luck!

References

  2. The image with some documentation: https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook

Annex A. sudo usermod mystery

Apparently you should be very thoughtful when given access to particular capabilities, a way to do that in Ubuntu is through the creation of different groups.
 usermod - modify a user account
   -a, --append
           Add the user to the supplementary group(s). Use only with the -G
           option.
-G, --groups GROUP1[,GROUP2,...[,GROUPN]]]
           A list of supplementary groups which the user is also a member of.
           Each group is separated from the next by a comma, with no
           intervening whitespace. The groups are subject to the same
           restrictions as the group given with the -g option.

           If the user is currently a member of a group which is not listed,
           the user will be removed from the group. This behaviour can be
           changed via the -a option, which appends the user to the current
           supplementary group list.
Take a way: sudo usermod -aG docker raf add (a) the user raf (my user) to the group (G). sudo is because only a sudo administrator can make those kind of changes.

Specific references

Annex B: Your containers

You can see what do you have: docker images, if you do not have already what you want then you can search with: docker search key-word it will give you a list of images. You can also search for the images in google and you may see more details, such as the version of the software and some useful information and advice.
When you decide you like this image then, docker pull for example:
docker pull  jupyter/pyspark-notebook

Specific references


  1. http://blog.thoward37.me/articles/where-are-docker-images-stored/

Comentarios

Publicar un comentario

Entradas populares de este blog

Making MongoDB remotely available.

Imagen
Summary Here I show how to make your MongoDB accessible from a different server / computer/ instance, without using ssh. Motivation I have been using MongoDB to store and query some databases. This databases has been shared with other people and in most of the cases I have managed to use it as localhost or using ssh. However when using Docker it is easy to run into problems when using ssh. It is common to access other databases such as MySQL. For example in to use MySQL in Python you could have something like this: import pymysql.cursors # Connect to the database connection = pymysql.connect( host = ' localhost ' , user = ' user ' , password = ' passwd ' , db = ' db ' , charset = ' utf8mb4 ' , cursorclass = pymysql.cursors.DictCursor) In the same line you could query your MongoDB using
Leer más

Reflecting about SIR models and some examples

Imagen
Reflecting about SIR models  and some examples Feel free to send me you thoughts either in comments or here . This only reflect my personal opinion, none of my employees at any time. Summary SIR model trendy, but they also have some caveats Looks like in Europe and the States the peak could be reach between May and July, with different amount of population affected by country given different policies implemented (so the model need time to incorporate that effect). Potential limitations: Summer could reduce prevalence. Data limitations, brief discussion below. But we are learning about Covid19, we did not have enough tests and so on. Intro After massive disruption for Covid19 everyone is concerned about what is going on and how this could evolve in the future. Massive disruption has occurred and all of us are noticing the consequences (such as compulsory stay at home measures, terrible amounts of deaths and so on). All of us want to help a bit and trying to
Leer más

Adicction and Decision Making: a brainy view.

Imagen
  Why is this interesting? To answer questions such as how can we understand addictions within our brains? Why are people more affected than others? Is Decision Making relating to Drug addiction? In Bechara 2005 the following similarities between pathing with VMPC (see below, related to the decision making areas of the brain) damage and drug addicts are pointed out : often deny, or not aware, that they have a problem. When faced with a choice that brings immediate reward, even at the risk of incurring future negative outcomes, including loss of reputation, job, and family, they appear oblivious to the consequences of their actions. The above points out that there are similarities. How could define willpower in term of geography in the brain? Willpower as proposed on Bechara 2005: “... willpower emergers from the dynamic interaction of two separate, but interacting, neural systems: and impulsive system, in which the amygdala is a critical neural structure involved in triggering the affe
Leer más