submitted by jtimperio to DataHoarder [link] [comments]
Intro:In my exploration of the world of big data and I became curious about tick data. Unfortunately, market data is almost always behind a paywall or de-sampled to the point of uselessness. After discovering the Dukascopy API, I knew I wanted to make this data available for all in a more accessible format. Over the course of a few months, I downloaded, cleaned, parsed, and compressed over a decade of Forex tick data on 37 currency pairs and commodities. Today I am happy to finally release the final result of my work to the DataHoarder community!
Download Links:Warning: I have rented a seedbox for the next 3 months from seedbox.io but I have been having some issues. If you have any issues with the torrent please leave a comment. Also, PLEASE SEED when you are done. This is quite a large data set and I can only push so much data on my own.
Torrent File: https://drive.google.com/file/d/18ymZWeFLJK7FggK_iiWZ-TxgWIVdJVvv/view?usp=sharingCompanion Blog Post: https://www.driftinginrecursion.com/post/dukascopy_opensource_data/
About the Data:The data was collected from https://www.dukascopy.com/ via a public API that allows for the download of tick data on the hour level. These files come in the form of a .bi5 file. The data starts as early as 2004 all the way to 2019.
These files were decompressed, then merged into yearly CSV’s named in the following convention. “AUDCHF_tick_UTC+0_00_2011.csv” or ‘Pair_Resolution_Timezone_Year.csv’
These CSV’s are split into 3 categories “Majors”, “Crosses”, “Commodities”.
Majors, Crosses, and Commodities have had their timestamps modified so that they are in the official UTC ISO standard. This was originally done for a Postgresql database that quickly became obsolesced. Any files that have been modified are appended with a “-Parse”. These timestamps have been modified in the following format.
Millisecond timestamps to UTC +00:00 time [2017.01.01 22:37:08.014] -- [2017-01-01T22:37:08.014+00:00]
User Resources:For those looking to use this data in a live context or update it frequently, I have included a number of tools for both Windows and Linux that will be useful.
WindowsThe ~/dukascopy/resources/windows folder contains a third party tool written in java that can download and convert Dukascopy’s .bi5 files. I have also included the latest zstd binaries from Zstandard Github page.
LinuxLinux is my daily driver in 99% of cases, so I have developed all my scraping tools using Linux only tools. In the ~/dukascopy/resources/linux folder you will find a number of shell script and pyhton3 files that I used to collect this data. There are quite a few files in this directory but I will cover the core ones below.
download-day.py:This file is used to download a single symbol for a single day and then convert and merge all 24 .bi5 files into a single CSV.
download-year.pyThis file is used to download a single symbol for a full year and then convert and merge all .bi5 files into a single CSV.
dukascopy.pyThis file contains all the core logic for downloading and converting data from dukascopy.
utc-timestamp-convert.pyThis tad slow but works well enough. It requires the pandas project and parses timestamps into the UTC ISO standard. This is useful for those looking to maintain the format of new files with the those in this repo, or those looking to use this in a SQL database.
How to Download Dukascopy Tick Data and Purse it with Birt's PHP Scripts [Outdated] A Brief Guide On How To Download And Process The Free Forex Tick Data From Dukascopy Using Birt's PHP Scripts. The Available Data In Dukascopy Is Raw In The Form Of Files Spanning Only One Hour, So It's Obvious That Tools Will Be Needed To Download And Parse It dukascopy free download. MT4 JForex Clients Bridge MT4 JForex Clients Bridge - is simple plug-in for Dukascopy JForex platform. It allows to transfer t It also lets you download and use high-quality tick data from Dukascopy and TrueFX. 60 Forex pairs, gold, silver, oil and 12 stock indexes. The software gives you access to all Forex major pairs plus XAUUSD and XAGUSD. You can also run simulations on oil and main stock indexes. Choose your favourite instrument and trade it. Currency pairs ... Download Dukascopy apk 3.5.21 for Android. Dukascopy Connect 911 is a messenger with a payment system from a Swiss bank. dukascopy connect free download - Dukascopy Connect 911, Dukascopy TV, Dukascopy TV HD, and many more programs This free tool was originally created by Dukascopy Bank SA. The size of the latest downloadable installation package is 52.8 MB. This download was checked by our antivirus and was rated as virus free. The program lies within Business Tools, more precisely Investment-Tools. The most popular versions among the software users are 2.4 and 1.0. From the developer: JForex Platform is a realtime ... A Brief Guide On How To Download And Process The Free Forex Tick Data From Dukascopy Using Birt's PHP Scripts. The Available Data In Dukascopy Is Raw In The Form Of Files Spanning Only One Hour, So It's Obvious That Tools Will Be Needed To Download And Parse It 📈 Download historical market price data via Node.js and CLI Topics stock exchange-rates dukascopy ohlc historical-data forex-data candlestick-chart commodities cfd crypto tick-data trading stock-data stock-prices etf market-data forex Fixed Historical Data Feed bug, when the widget would not download data correctly for certain instruments. Release 0.41.4 February 4 Changed logic of input fields: invalid values are no longer coerced to a minimum value; a warning is shown instead. Add free customisable market widgets to your website and blog or support your trading with Forex, commodity and stock CFD, and other financial data from Dukascopy Bank.
[index]          
Check Out This Site: https://bit.ly/3f0iYB0 - How On-line Forex Chart - Forex Data & Tools by Dukascopy can Save You Time, Stress, and Money. NZ information ... Read More Here: https://bit.ly/39tawt8 - The On-line Forex Chart - Forex Data & Tools by Dukascopy Statements Nonetheless it's possible that your trade reach... Luktom Manual Trader JForex - to znakomite narzędzie usprawniające handel ręczny na platformie JForex. Znacznie przyspiesza wykonywanie wielu standardowych akcji, które normalnie zabierają ... Someone asked for this on my list newsletter list http://quantlabs.net/blog/2017/12/algo-forex-trading-strategy-tools-for-dukascopy-jforex-with-redis-nosql/ My first Java test strategy with Dukascopy JForex API for automated forex trading - Duration: 11:33. Bryan Downing 4,948 views Algo forex trading strategy tools for Dukascopy JForex with Redis NOSQL by Bryan Downing. 10:07. Future of algo forex trading and crypto currency by Bryan Downing. 20:20. Analyzing true forex ... Dec 20 Dukascopy forex trading update with latest tools overview Latest update with latest analysis with Redis NOQL message queue and backstore http://quantl...