Added ability to filter by IS and OOS in each topic to see which IS/OOS should be used.

When adding the ability to filter by In-Sample (IS) and Out-of-Sample (OOS) performance in each metric or topic, it becomes critical to differentiate between the two phases of strategy testing. The IS data is used to develop and optimize the strategy, while the OOS data tests the strategy on unseen, untouched market conditions to assess its robustness.

Here's how to approach filtering each metric for IS and OOS:

IS (In-Sample) vs OOS (Out-of-Sample) Explanation

  • In-Sample (IS): The period used for building and optimizing the strategy. High IS performance may indicate overfitting if OOS performance is poor.
  • Out-of-Sample (OOS): The unseen data used to test the robustness of a strategy. Good OOS performance indicates that the strategy generalizes well to unseen market conditions.

Metric Prioritization and IS/OOS Expectations

  1. Profit Factor

    • IS Target: Should be > 1.5.
    • OOS Target: A good profit factor in OOS should remain above 1.3 or higher to show that the strategy is profitable without overfitting.
    • Purpose: This shows how well the strategy is balancing profits against losses both in-sample and out-of-sample.

  2. CAGR/Max DD%

    • IS Target: > 1 (This indicates returns are higher than the risk during optimization).
    • OOS Target: Should ideally remain above 0.8 in OOS. If the value dips significantly, it might indicate over-optimization in the IS phase.
    • Purpose: It helps compare growth to risk over the strategy’s life cycle. Stability in this metric across IS and OOS is important for long-term performance.

  3. Sharpe Ratio

    • IS Target: > 2 (This shows good risk-adjusted performance in-sample).
    • OOS Target: Should remain above 1.5 in OOS. Sharpe ratios dropping significantly in OOS may indicate risk or volatility problems in unseen data.
    • Purpose: This measures the strategy’s ability to handle risk consistently, and OOS results are crucial to validate its stability in live trading.

  4. Return/Drawdown Ratio (Ret/DD)

    • IS Target: > 2 (Shows solid returns in comparison to risk).
    • OOS Target: Ideally, you want this to stay above 1.5 to confirm stability.
    • Purpose: A high Ret/DD ratio in IS shows good returns relative to risk, but if the OOS version drops sharply, it indicates the strategy may not handle risk well in real market conditions.

  5. Stability

    • IS Target: > 0.8 (A smooth, consistent equity curve).
    • OOS Target: Stability should remain > 0.7 in OOS. A significant drop indicates the strategy’s performance may become inconsistent in live conditions.
    • Purpose: High stability across IS and OOS ensures a smooth equity curve, avoiding sharp performance drops in real trading.

  6. Open Drawdown %

    • IS Target: < 15% (Keeps risk exposure in check during optimization).
    • OOS Target: Should remain below 20% in OOS testing. If OOS drawdowns significantly exceed the IS values, the strategy may struggle in volatile or unseen markets.
    • Purpose: Managing open drawdowns is critical for risk control, and seeing how this behaves OOS ensures you're not taking on excessive risk in real trading.

  7. Max Drawdown %

    • IS Target: < 20% (Manages maximum peak-to-valley losses).
    • OOS Target: Should not exceed 25% in OOS. Drawdown stability between IS and OOS indicates the strategy is capable of handling stress.
    • Purpose: This metric helps manage risk, ensuring that no single market condition causes extreme losses.

  8. Consecutive Win/Loss Trades

    • IS Target: > 2 (Aims for win streaks without significant drawdowns).
    • OOS Target: Should be similar to IS results, or show a smooth transition. Long losing streaks in OOS can be a warning sign.
    • Purpose: This shows how often you can expect consecutive wins or losses. Stability between IS and OOS is a good indicator of robustness.

  9. Net Profit

    • IS Target: Positive net profit in IS with steady growth.
    • OOS Target: Should maintain a positive net profit. If OOS performance is significantly lower, this indicates overfitting.
    • Purpose: Maintaining positive net profit in both IS and OOS is crucial for real trading profitability.

  10. R-Squared (R²)

    • IS Target: > 0.8 (A high R² shows smooth, predictable growth).
    • OOS Target: Should remain above 0.7 to avoid overfitting. A significant drop in OOS means the strategy is inconsistent.
    • Purpose: This metric helps you find strategies that perform smoothly without high volatility in their returns.

  11. Sortino Ratio

    • IS Target: > 1.5 (Indicates good risk-adjusted returns considering only downside volatility).
    • OOS Target: Above 1.2 in OOS. A sudden drop in Sortino indicates the strategy is exposing itself to more downside risk than it did during optimization.
    • Purpose: This is a good alternative to Sharpe Ratio, focusing on negative risk rather than total risk. Stability in OOS is critical for downside protection.

How to Use These Filters in SQX:

  1. First pass with IS filters:

    • During the initial build/optimization phase, filter strategies based on high values for key metrics in the IS data. Use Profit Factor > 1.5, CAGR/Max DD% > 1, Sharpe Ratio > 2, Ret/DD > 2, and Stability > 0.8.

  2. Second pass with OOS filters:

    • After IS optimization, evaluate the strategies in the OOS phase. Set slightly lower but still acceptable values: Profit Factor > 1.3, Sharpe Ratio > 1.5, CAGR/Max DD% > 0.8, Ret/DD > 1.5, and Stability > 0.7.

  3. Compare IS and OOS performance:

    • Discrepancies between IS and OOS are normal, but they should not be drastic. If a strategy performs well in IS but poorly in OOS, it's likely overfitted. Favor strategies with consistent metrics across both phases.

By maintaining these IS/OOS targets in SQX, you can ensure your strategies are robust and able to handle live trading without deteriorating due to overfitting.

Related posts