Wei’s Project Details

Source: Notion | Last edited: 2025-03-19 | ID: 1bb2d2dc-3ef...

Applying Elnigma to Commodities Futures & Expanding ML Capabilities (2024 Summary)

Data Source Instability: Unlike the crypto market (24/7), commodities futures operate on a 24/5 schedule, leading to systematic missing data.
Data Variability: Different sources provided inconsistent data, introducing additional preprocessing challenges.
Non-Continuous Data Challenges: Addressing gaps in time-series data was critical for model reliability.
- Tested Approaches:
  - Interpolating Missing Data → Led to poor results due to artificial data generation.
  - Downsampling to Weekly Granularity → Ensured continuity but reduced available training data.
  - Time Features for Discontinuous Sequences → Showed promise by allowing models to adapt to market structure changes.
Findings: Models demonstrated profitability potential, but production deployment introduced significant risk considerations.

Stable Data but New Challenges: Unlike commodities, Chinese equities operate on an 8/5 schedule, leading to even greater discontinuities.
Feature Integrity Issues: Standard rolling features changed meaning due to non-continuous trading days.
Results:
- Profitability potential was present.
- Unknown risks remained, necessitating new feature engineering strategies.
Decision: Given the challenges, we refocused on our core strength: crypto markets.

Goal: Introduce correlation analysis to assess new models’ relationship with existing ones.
Risk Diversification: Lower correlation between active models reduces risk and enhances robustness.
Challenges:
- Existing architecture was rigid → Difficult to integrate external researchers’ expertise.
- Solution: Developed the Touchstone Service, enabling independent submission of:
  - Feature sets
  - Model architectures
- Automated Evaluation:
  - Determines correlation with existing models.
  - Assesses overall performance.

Problem: Once deployed, models remained in production until manually deemed outdated.
Issue: Some models, despite temporary underperformance, retained long-term value.
Solution: Introduced retraining capabilities for pre-trained models:
- Incorporates latest data → Adapts to market conditions.
- Optimizes specific weaknesses → E.g., mitigating max drawdown (MaxDD), optimizing Sortino ratio.
Outcome: Enhanced model adaptability and longevity, preserving profitable strategies.

Current State:
- Portfolio allocation was human-driven → Experience-based but prone to biases and delays.
Need for Automation:
- Real-time dynamic adjustments.
- Emotion-free decision-making.
Research Insights:
- RL-based portfolio optimization is an emerging field.
- Existing studies were not directly applicable:
  - Our allocation is based on Elnigma model performance, not just OHLCV price data.
Custom Design:
- Novel reward function tailored to our use case.
- Built a custom environment from scratch for PnL-driven feature engineering.
Results: Established foundational work for future RL-based trading strategies.

Elnigma’s expansion to new markets provided insights into data structure challenges and model generalization limits.
Correlation analysis & retraining improved our ability to integrate diverse models while extending their lifecycle.
Portfolio optimization via RL introduced a new avenue for automated, unbiased allocation strategies.
Next Steps:
- Further refining RL approaches for production deployment.
- Enhancing feature engineering to improve model adaptability.
- Expanding external model integration through Touchstone Service. Each of these advancements strengthens Eon Labs’ trading infrastructure, positioning us for greater resilience and innovation in algorithmic trading.