Price History (and how to get it)

Historical price data is used in [Test] and [Train] mode, and can also be used in [Trade] mode for pre-loading the LookBack period. The needed data period can be determined with StartDate and EndDate. If historical prices from the desired period are missing, they can be downloaded with the assetHistory function or the Download script (see below). All downloaded prices are usually aligned to UTC time. The broker's time zone does not matter. Therefore it's normally no problem to use price data from different sources for the simulation and for trading.

Zorro stores historical price data in files in the History folder, together with optional  additional data such as spread and volume ("Level-1 data"), expiration dates, option chains, or order book content ("Level-2 data"). Each file name is composed from the asset name (with special characters such as '/' removed), an optional year number, and the extension. The format is given by the file extension:

The formats are very similar; they only differ in number of fields in a struct or record, indicated by the number after the "t". Price history can cover multiple years or a single year; in the latter case the file name has the year number attached. For instance, "EURUSD_2010.t6" contains historical data of the EUR/USD asset in 2010. For special cases it is also possible to import arbitrary user-specific formats in a dataset and retrieve historical data from it with the dataFind function. The .t6 price data files included with Zorro are 1-minute based, but any other time base can also be used as long as it's equal or less than the bar period used in the script.

The same asset can have different price histories, for instance EOD data for strategies with daily bars, M1 data for day trading and T1 data for scalping. The historical data files can be distinguished with a postfix character to the file name, f.i. "AAPLd.t6" for EOD data with no year number, "AAPL_2018m.t6" for M1 data split into years and "AAPL_2018.t1" for the tick data. The script can then select the right price history through the History variable, f.i. "History = d.t6" for selecting the EOD data, "History = m.t6" for the M1 data and "History = t1" for the tick data.

A .t1 or .t6 price data file is a simple list of T1 or T6 structs in descending order (newest price first).

typedef struct T1
DATE time; // timestamp of the tick in UTC, OLE date/time format
float fVal; // price data, positive for ask and negative for bid
} T1; typedef struct T6 { DATE time; // timestamp of the end of the tick in UTC, OLE date/time format float fHigh,fLow; float fOpen,fClose; float fVal,fVol; // additional data, like ask-bid spread, volume etc. } T6;

Both .t1 and .t6 data can contain the current ask-bid spread, which is returned by the marketVal function. For backtesting with variable spread, set Spread = marketVal() in the script.

 Historical data of options and futures are stored in a .t8 file that includes option specific parameters such as type, strike, and expiration date. It is a list of CONTRACT structs in descending order:

typedef struct CONTRACT
  DATE  time;   // time in UTC, OLE date/time format
  float fAsk,fBid; // premium without multiplier
  float fVal,fVol;  // additional data, like open interest, volume etc.
  float fUnl;   // underlying price
  float fStrike; // strike price
  long  Expiry; // YYYYMMDD format

Dependent on the History variable, historical price data can be taken either from a .t1, .t6, or .t8 file. The default is .t6. If a .t8 file contains different underlying prices for the same time stamp, which can happen with futures of different expiration, the average of the underlying prices is used for the historical price. Historical data should have the same format for all years and assets; mixing .t6 from 2016 and .t1 from 2015 would produce an error message. The exception are options that are always loaded from a .t8 file, while the underlying prices can come from the same .t8 or from a .t6 or .t1 file.

Historical order book data is stored in a .t2 file that contains the ask/bid quotes and the quoted volume:

typedef struct T2
  DATE time;  // timestamp in UTC, OLE date/time format
  float fVal; // price quote, positive for ask and negative for bid
  float fVol; // volume or size
} T2; 

For downloading and reading data files with a resolution of less than one minute (such as .t1 files), Zorro S is required. For converting price data from an external source to the Zorro format, separate it into years, convert it to a list of T6 or T1 structs, and store it in a binary file with the format and name described above. The structs are stored in reverse order, i.e. the most recent tick comes first. The OLE DATE format is a double float value, counting days since midnight 30 December 1899; hours, minutes, and seconds are represented as fractions of days. The timestamps are supposed to be the UTC time at the end of the tick, i.e. the time of the Close price. If you have price data based on local time, convert it to UTC (you can find some example code here). The exception is daily (D1) price history, where the timestamps are the local date with no time offset (otherwise a bar ending at 00:00 would have the date of the next day). If the original price data contains bid prices, you could convert them to ask prices by adding the spread; if the price data contains time stamps from the tick start, you could shift them to the tick end by adding the bar period when converting the file. However, both is normally not very relevant for backtest results.  

The fOpen, fClose, fHigh and fLow data streams can be separately accessed in the script. The fVal and fVol data can be used to store and access additional data, such as spread, quote frequency, trade volume, or rollover. Data in .t1 format can be used for additional information, such as external indicators or statistics such as the VIX (in that case set Detrend at 16 for disabling the automatic fixing of price data). All data streams are automatically synchronized to the streams of the first asset call in the script; so it does not matter if some data is only available in daily or weekly 'ticks'. The asset function can be used to select between different data files, while the price functions select between the four data streams inside the file.

Historical data in arbitrary CSV formats can be converted to .t1, .t6, or .t8 with the dataParse function. Exporting or converting historical data to and from other formats is also easy due to the simple format - in the Strategy folder you can find several small scripts for this purpose. Download.c (see below) downloads historical data in .t6 or .t1 format from Yahoo, IB, FXCM, or Oanda. CSVtoHistory converts data from arbitrary .csv formats, f.i. from HistData, Quandl, or Yahoo, to the Zorro .t6 format. Example code for automatically downloading .csv price data from the Internet or from a provider API can be found on the http page. A script for generating .t8 historical data for options from the yield rate, dividends, and underlying price history can be found here. Example scripts for converting history files from complex .csv or .nxc formats to the .t1 format can be found here.

For looking into .t8, .t6, .t2, or .t1 files and displaying their content in a panel, use the History script.

The Download script

History files for all example strategies are available on the Zorro Download page. For more price histories you can use Download.c, a small script for downloading prices and asset parameters from the Internet or from a broker or signal service. You can download either the history of a single asset, or of all assets of a given asset list. The script opens a control panel with toggle buttons (blue) and entry fields (yellow):

AAPL Asset name, or CSV asset list to download
2016-2017 Time period to download
M1 Data resolution (M1, Tick, EOD)
Ask/Bid Data type (Ask/Bid, Trades, Unadjusted, Assetlist)
Broker Data source (Broker, Stooq, AlphaVantage)
Download! Click here to start the download.

Asset or asset list

Enter here the symbol of the asset in the form needed by the broker or price source, f.i. "AAPL.US" for downloading AAPL data from Stooq. If the name contains the word "Assets" or ends with ".csv", historical data for all assets from the given list is downloaded (f.i. "AssetsFix").

Data resolution

Data type

Data source

Not all combinations make sense. For instance, you can normally not download M1 data or asset parameters from an Internet data source. Check with your broker which data types are available. If a price history file of the same asset already exists, the new prices are appended at the end. If the file was already updated on the same or previous day, no data is downloaded. After all is downloaded, click [Stop] to end the session.

See also:

Bars, file, Dataset, asset parameters, assetHistory

► latest version online