I need Data

Hi Guys,

In order to analyse and/or predict the trend in transactional price in the pair of BTC-USD I definetely need historical data. I don’t know yet what granularity is needed not mentioning time period needed to perform data mining.

Majority of google results were limited to last 2000 entries for free, where rest was paid. Data resolution wasn’t ideal too – 1 entry per day. I believe I need real-time data.

I gave it a try to SpreadStreet.io, where registration takes a second after which I was able to download my very first data set for Bitcoin historical data from GDax exchange. CSV files looks like that:


But I’m more interested in close to real-time, at a given time transactional data, like on the screenshot below:


which is default screen after login to former GDax exchange, currently rebranding to Coinbase Pro. Fortunately they provide free access to API, where my plan is to write a simple script and give it a go. Maybe after gathering some more significant amount of data I’ll create a service where everyone could easily access, fetch and crawl gathered data. My short initial research proved it’s not so easy (even doable?) to access that kind of data.

It’s time to write some code. After giving short thought I decided to use websocket data stream from GDax and I believe shortest code will be in NodeJS. Well.. even just discovered that there is official library for it:

npm -g install gdax

above command failed at first execution, so I figured out I need latest npm. Not sure why but I also needed to replace npm link from /usr/bin with /usr/local/bin/npm. After gdax library was installed  very short script was written which connects to the stream of live transactions and stores them in database.

Body of the scripts looks like that:

const GDAX = require("gdax");
const BTC_USD = 'BTC-USD';
const websocket = new GDAX.WebsocketClient( [BTC_USD],
key: 'suchkey',
secret: 'suchsecret',
passphrase: 'muchpassphrase',
channels: ['ticker']
} );

const websocketCallback = (data) => console.dir(data);
websocket.on('message', websocketCallback);

Well maybe it doesn’t connect to the database yet, but at least it flooded console with real-time trade data, which looks like that:

{ type: 'heartbeat',
last_trade_id: 46033447,
product_id: 'BTC-USD',
sequence: 6272004023,
time: '2018-07-07T21:01:23.160000Z' }
{ type: 'ticker',
sequence: 6272004036,
product_id: 'BTC-USD',
price: '6520.05000000',
open_24h: '6570.76000000',
volume_24h: '3141.35056307',
low_24h: '6512.00000000',
high_24h: '6620.00000000',
volume_30d: '247100.78541627',
best_bid: '6518.53',
best_ask: '6520.05',
side: 'buy',
time: '2018-07-07T21:01:23.442000Z',
trade_id: 46033448,
last_size: '0.04401512' }

That being said, tomorrow I’ll store data into database, so it could be followed by some analysis and predictions in near future.

Thank you for reading,

UPDATE 8th of July:

Actually I’m more interested in “matches” websocket channel from GDax, so a line in my script from :

channels: ['ticker']

changes to:

channels: ['matches']

Followed by new type of response:

{ type: 'match',
trade_id: 46073524,
maker_order_id: '08b111cf-2636-42e8-80c1-115b4c9f39c4',
taker_order_id: 'cdd8cf0c-8426-4b8d-8aa3-15ef191740a1',
side: 'buy',
size: '0.00415150',
price: '6746.74000000',
product_id: 'BTC-USD',
sequence: 6279790403,
time: '2018-07-08T19:28:52.390000Z' }

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s