Removes, inverts, or randomizes trend in historic price data. Can be set up for removing bias from the parameters, the trades, the indicators, or the whole price curve.
Backtesting with detrended, inverted, or randomizied price curves is also a quick
"reality check" to verify the validity and robustness of a strategy. For removing trend, the curve or parts of it are "tilted" until the start and end prices have the same value.
Shuffled EUR/USD price curves
The following flags can be set or combined with '+' or '|':
||Detrend trade results. The trade return is corrected by the average price slope of the current WFO cycle. This removes trend bias from a
WFO test while keeping the properties of the price curve.
||Detrend trade results and price functions. The prices are corrected by the average price slope of the current WFO cycle. For detrending indicators and signals based on series generated with price() calls.
||Detrend historical price data on loading. The curve is tilted so that
the end and the start of the historical data are at the same level. This also affects the displayed price curve in the chart.
||Replace prices with their reciprocal values. This it can be used for
creating an artifical asset as the reciprocal of an existing forex or crypto
pair (f.i. convert BTC/USD to USD/BTC).
||Similar to RECIPROCAL, but keep the price range while
inverting the price curve. This reverses all trends in the curve, and can be used for a reality check of a system that is symmetric in long and short positions.
||Randomize the price curve by shuffling the price ticks. Keeps its overall trend
from start to end, but removes any short-term trends and correlations between the prices. Used for reality checks.
|Randomize the price curve as above, but generate a curve that does not
exceed the highest peak or/and the lowest valley of the original price
curve. Use this for keeping the original price range. Can be slow.
||Do not do not detect gaps or fix outliers on loading. For data files that contain no prices, but other types of data.
- Use setf or resf for setting
or resetting single flags, while keeping the other flags unchanged.
- If a trade strategy has a trend bias - f.i. when it buys more long than short positions, or vice versa - it is recommended to remove the trend from the price curve for generating strategy parameters. Otherwise the simulation will be too optimistic when the price curve has an overall upwards trend, or too pessimistic when it has a downwards trend. For symmetric strategies that buy long and short positions under similar conditions and with the same strategy parameters, detrending is normally not necessary.
- Setting Detrend at TRADES or PRICES detrends the prices from the end of the lookback period until the end of the simulation or WFO training cycle. This has the effect that all WFO cycles are separately detrended in [Train] mode. Detrend = CURVE however detrends the price data file, but keeps the short-term trends of the WFO cycles.
- SHUFFLE, CURVE, RECIPROCAL,
INVERT are applied to the ticks of the historical data with
no regard to bar period or start and end date.
- SHUFFLE and CURVE can produce negative
prices when the price range is large compared with the price amount. In that
case CURVE clips negative prices and produces a flat price
line close to zero. If negative prices in a shuffled curve are harmful for a
reality check, test the sign of the prices and skip curves with negative
- SHUFFLE shuffles the the bar ranges (high-low
difference) and changes (center price difference to the previous bar)
independently. This gives the shuffled curve the same magnitude and long
term trend as the
original curve, while destroying all bar correlations. If prices of the
shuffled curve are negative, shuffling is repeated until all prices are
greater than 0.
- For detrending only strategy parameters or rules, but not the test results, set Detrend only in [Train] mode: if(Train) Detrend = CURVE;.
- For detrending only the prices of certain assets, set Detrend before calling asset()
in the initial run, and set it to 0 afterwards.
- For detrending particular trades only, set Detrend = TRADES before entering the trade, and set Detrend = 0 afterwards. This
can be useful for testing portfolio systems that use both trend following and non-trend algorithms.
- For determining if a system's profit is caused by artifacts, temporarily set Detrend = SHUFFLE+CURVE and test. If the profit is caused by a real edge, it should then disappear.
- For determining if a system's profit is caused by randomness, temporarily set Detrend = INVERT and test. On symmetric assets such as currencies, the system should then still achieve profits.
- Detrended, inverted, or randomized price curves can be exported to a CSV file for further tests with R or similar statistical software.
- Reality checks using inverted or randomized price curves are mentioned on Financial Hacker | Development Process | Step 7: Reality Check.
Example (see also Detrend.c):
if(is(TRAINMODE)) Detrend = CURVE; // detrend the price curve for training
mode, WFO, DataSlope, randomize,
► latest version online