Collect GDP forecast using OCDE API with python 3.4

Collect GDP forecast using OCDE API with python 2.7

Rafael Valero Fernandez 14/10/2016
Gather data using OCDE API (After writing this entrances I realice a possible easier way, by using padas data reader: https://pandas-datareader.readthedocs.io/en/latest/remote_data.html#oecd)
Summary
Step 2) Read the previous link with python
Step 3) get the json
Step 4) Parse json into dataframe pandas

In [60]:
request_URL = "http://stats.oecd.org/sdmx-json/data/EO/GBR.ET_ANNPCT.A+Q/all?detail=Full&dimensionAtObservation=AllDimensions&startPeriod=2016"

# libraries to obtain the data from web

import urllib
import json


html_text = urllib.urlopen(request_URL)
html_text
Out[60]:
<addinfourl at 146112136L whose fp = <socket._fileobject object at 0x000000000444EF48>>
In [61]:
data = json.load(html_text)
data
Out[61]:
{u'dataSets': [{u'action': u'Information',
   u'observations': {u'0:0:0:0': [1.1540489105753, 0, None, 0, 0, None],
    u'0:0:0:1': [0.581208615963739, 0, None, 0, 0, None],
    u'0:0:1:2': [0.750000000000886, 1, None, 0, 0, None],
    u'0:0:1:3': [0.650000000000173, 1, None, 0, 0, None],
    u'0:0:1:4': [0.619999999999355, 1, None, 0, 0, None],
    u'0:0:1:5': [0.600000000000531, 1, None, 0, 0, None],
    u'0:0:1:6': [0.579999999999496, 1, None, 0, 0, None],
    u'0:0:1:7': [0.56000000000028, 1, None, 0, 0, None],
    u'0:0:1:8': [0.540000000000551, 1, None, 0, 0, None],
    u'0:0:1:9': [0.529999999999949, 1, None, 0, 0, None]}}],
 u'header': {u'id': u'991b95e3-2946-4107-b472-e0fdaa9e6c27',
  u'links': [{u'href': u'http://stats.oecd.org:80/sdmx-json/data/EO/GBR.ET_ANNPCT.A+Q/all?detail=Full&dimensionAtObservation=AllDimensions&startPeriod=2016',
    u'rel': u'request'}],
  u'prepared': u'2016-10-14T08:40:25.356Z',
  u'sender': {u'id': u'OECD',
   u'name': u'Organisation for Economic Co-operation and Development'},
  u'test': False},
 u'structure': {u'annotations': [{u'text': u'',
    u'title': u'Copyright OECD - All rights reserved',
    u'uri': u''},
   {u'text': u'',
    u'title': u'Terms and Conditions',
    u'uri': u'http://www.oecd.org/termsandconditions/'},
   {u'text': u'',
    u'title': u'Privacy Policy',
    u'uri': u'http://www.oecd.org/privacy/'},
   {u'text': u'', u'title': u'MyOECD', u'uri': u'https://www.oecd.org/login'},
   {u'text': u'',
    u'title': u'Contact Us',
    u'uri': u'http://www.oecd.org/contact/'}],
  u'attributes': {u'dataSet': [],
   u'observation': [{u'id': u'TIME_FORMAT',
     u'name': u'Time Format',
     u'values': [{u'id': u'P1Y', u'name': u'Annual'},
      {u'id': u'P3M', u'name': u'Quarterly'}]},
    {u'id': u'OBS_STATUS', u'name': u'Observation Status', u'values': []},
    {u'id': u'UNIT',
     u'name': u'Unit',
     u'role': u'UNIT_MEASURE',
     u'values': [{u'id': u'PC', u'name': u'Percentage'}]},
    {u'default': u'0',
     u'id': u'POWERCODE',
     u'name': u'Unit multiplier',
     u'role': u'UNIT_MULT',
     u'values': [{u'id': u'0', u'name': u'Units'}]},
    {u'id': u'REFERENCEPERIOD',
     u'name': u'Reference period',
     u'role': u'BASE_PER',
     u'values': []}],
   u'series': []},
  u'description': u'Economic Outlook No 99 - June 2016',
  u'dimensions': {u'observation': [{u'id': u'LOCATION',
     u'keyPosition': 0,
     u'name': u'Country',
     u'role': u'REF_AREA',
     u'values': [{u'id': u'GBR', u'name': u'United Kingdom'}]},
    {u'id': u'VARIABLE',
     u'keyPosition': 1,
     u'name': u'Variable',
     u'values': [{u'id': u'ET_ANNPCT', u'name': u'Total employment, growth'}]},
    {u'id': u'FREQUENCY',
     u'keyPosition': 2,
     u'name': u'Frequency',
     u'role': u'FREQ',
     u'values': [{u'id': u'A', u'name': u'Annual'},
      {u'id': u'Q', u'name': u'Quarterly'}]},
    {u'id': u'TIME_PERIOD',
     u'name': u'Time',
     u'role': u'TIME_PERIOD',
     u'values': [{u'id': u'2016', u'name': u'2016'},
      {u'id': u'2017', u'name': u'2017'},
      {u'id': u'2016-Q1', u'name': u'Q1-2016'},
      {u'id': u'2016-Q2', u'name': u'Q2-2016'},
      {u'id': u'2016-Q3', u'name': u'Q3-2016'},
      {u'id': u'2016-Q4', u'name': u'Q4-2016'},
      {u'id': u'2017-Q1', u'name': u'Q1-2017'},
      {u'id': u'2017-Q2', u'name': u'Q2-2017'},
      {u'id': u'2017-Q3', u'name': u'Q3-2017'},
      {u'id': u'2017-Q4', u'name': u'Q4-2017'}]}]},
  u'links': [{u'href': u'http://stats.oecd.org/sdmx-json/dataflow/EO/all',
    u'rel': u'dataflow'}],
  u'name': u'Economic Outlook No 99 - June 2016'}}
In [54]:
type(data)
Out[54]:
dict
In [62]:
values_I_want=data['dataSets'][0]['observations']
values_I_want
Out[62]:
{u'0:0:0:0': [1.1540489105753, 0, None, 0, 0, None],
 u'0:0:0:1': [0.581208615963739, 0, None, 0, 0, None],
 u'0:0:1:2': [0.750000000000886, 1, None, 0, 0, None],
 u'0:0:1:3': [0.650000000000173, 1, None, 0, 0, None],
 u'0:0:1:4': [0.619999999999355, 1, None, 0, 0, None],
 u'0:0:1:5': [0.600000000000531, 1, None, 0, 0, None],
 u'0:0:1:6': [0.579999999999496, 1, None, 0, 0, None],
 u'0:0:1:7': [0.56000000000028, 1, None, 0, 0, None],
 u'0:0:1:8': [0.540000000000551, 1, None, 0, 0, None],
 u'0:0:1:9': [0.529999999999949, 1, None, 0, 0, None]}
In [63]:
for i in values_I_want:
    print(i)

0:0:1:4
0:0:1:8
0:0:1:9
0:0:1:5
0:0:0:1
0:0:0:0
0:0:1:6
0:0:1:7
0:0:1:2
0:0:1:3
In [70]:
import pandas as pd
import numpy as np

auxiliar_1= values_I_want.values()
auxiliar_2 = len(auxiliar_1)

data_values=np.empty([auxiliar_2,1])

for i in range(0,auxiliar_2):
     data_values[i] = auxiliar_1[i][0]
In [71]:
data_values
Out[71]:
array([[ 0.62      ],
       [ 0.54      ],
       [ 0.53      ],
       [ 0.6       ],
       [ 0.58120862],
       [ 1.15404891],
       [ 0.58      ],
       [ 0.56      ],
       [ 0.75      ],
       [ 0.65      ]])
In [85]:
data['structure']['dimensions']['observation'][3]['values'][1]['id']
Out[85]:
u'2017'
In [179]:
auxiliar_1= data['structure']['dimensions']['observation'][3]['values']
auxiliar_2 = len(auxiliar_1)



index_df=[]
for i in range(0,auxiliar_2):
     index_df.append(auxiliar_1[i]['id'].encode('ascii'))


index_df
Out[179]:
['2016',
 '2017',
 '2016-Q1',
 '2016-Q2',
 '2016-Q3',
 '2016-Q4',
 '2017-Q1',
 '2017-Q2',
 '2017-Q3',
 '2017-Q4']
In [185]:
df = pd.DataFrame(data_values,index=index_df)
df.columns=['gdp_projected']
df
Out[185]:

gdp_projected
20160.620000
20170.540000
2016-Q10.530000
2016-Q20.600000
2016-Q30.581209
2016-Q41.154049
2017-Q10.580000
2017-Q20.560000
2017-Q30.750000
2017-Q40.650000



Comentarios

Publicar un comentario

Entradas populares de este blog

Making MongoDB remotely available.

Imagen
Summary Here I show how to make your MongoDB accessible from a different server / computer/ instance, without using ssh. Motivation I have been using MongoDB to store and query some databases. This databases has been shared with other people and in most of the cases I have managed to use it as localhost or using ssh. However when using Docker it is easy to run into problems when using ssh. It is common to access other databases such as MySQL. For example in to use MySQL in Python you could have something like this: import pymysql.cursors # Connect to the database connection = pymysql.connect( host = ' localhost ' , user = ' user ' , password = ' passwd ' , db = ' db ' , charset = ' utf8mb4 ' , cursorclass = pymysql.cursors.DictCursor) In the same line you could query your MongoDB using ...
Leer más

How to create a function in python to be used from the terminal

Imagen
To write a python code able to read variables or inputs from the command line you only have to use sys.argv. Let's see a couple of examples. You can find the code here . File called example_v1.py: import sys def main(): print sys.argv if (__name__ == "__main__"): main(); Then you go to the folder where you have it and run this comman in the comman line: python example_v1.py Like here: As you can see the output is the name of the file. Let's try this: python example_v1.py hi hola ciao Basically the comand sys.argv is giving us a list of the commandline arguments received. Now I am going to create a program that is going to say hi and the name that you introduce after file name in the commandline. I called this script example_v2.py: import sys def main(): print 'Hi ' + sys.argv[1] + '!' if (__name__ == "__main__"): main(); So now we can write in the terminal: python ...
Leer más

Los mercados y las pensiones.

En estos tiempos muchos tratan a de demonizar a los mercados, en particular a los mercados de compra y venta de deuda pública, sin embargo, lo que no se dice es que estos mercados tienen en consideración y ponen a examen la economía y sostenibilidad de un país en su conjunto. Así que siendo realistas España tiene una serie de problemas  a corto, medio y largo plazo que también valoran los mercados. España tiene un sistema educativo en el que un alto porcentaje de alumnos no terminan la educación obligatoria, una universidad con poco prestigio  y proyección internacional. España dependen sobremanera del turismo (¿qué pasara cuando los países del norte de África vuelvan a ser relativamente seguros?). Nuestro inmenso paro, nuestras bajas tasas de innovación, nuestras altas tasas de pobreza relativa y creciente desarraigo. Pero no olvidemos algo mucho mayor, nuestro sistema de pensiones cuya sostenibilidad es bastante dudosa. Sobre el tema de las pensiones hay argumentos e...
Leer más