Touchstone Service (WIP)

Source: Notion | Last edited: 2024-06-05 | ID: 875338e6-bbf...

---

Background:

试金石service是用来检验被提交的feature_set是否优秀.

具体 feature_set 是如何被提交的，查看这个page Turing Design (WIP)

Architecture Draft

web提交代码和文件名

lambda 验证代码安全，储存到s3



server运行create_features，并使用这些feature 训练模型

可以修改整个feature file, 对于每个 feature group, 如果不specify某个type，则代表没有修改。

def create_features(*base_featureset_name, new_featureset_name, selectors, target_pairs=["binance_spot_BTC/USDT", "binance_spot_ETH/USDT"]*):
  # *target_pairs里面代表了用来训练，validate和test的数据。*比如*binance_spot_BTC-USDT，binance_spot_BTC-USDT_ETH-USDT*
  base_feature_helpers = import("*base_featureset_name.py"*)
  new_feature_helpers = import("*new_featureset_name.py"*)
  ohlcv = fetch_ohlcv(target_pairs) # 其中target_pairs决定那个 / 哪些pair来验证和测试模型，具体有哪些pair应该是predefined，由data engineering team来完成

  feature_list = []

  for selector in selectors:
    feature = new_feature_helpers.extract_by_type(
        feature_type,
        dates=dates,
        open_prices=open_prices,
        close_prices=close_prices,
        high_prices=high_prices,
        low_prices=low_prices,
        volumes=volumes,
    )

    # 如果new_feature中没有定义这个feature type，则使用base feature中定义的
    if len(feature) == 0:
      feature = base_feature_helpers.extract_by_type(
        feature_type,
        dates=dates,
        open_prices=open_prices,
        close_prices=close_prices,
        high_prices=high_prices,
        low_prices=low_prices,
        volumes=volumes,
      )

    if len(feature) != len(ohlcv):
      raise inputError(f"feature {selector} parse is incorrect: feature length ({len(feature)}) != ohlcv length ({len(ohlcv})")

    feature_list.append(feature)

  print_feature_distributions(feature_list)

  lstm_features = generate_lstm_features(feature_list, window_size) # 把feature_list 乘以 window_size

  return lstm_features

def train():
  lstm_features = create_features(*base_featureset_name, new_featureset_name, selectors, target_pairs*)
  # ... train model

ROCP MACD RSI VROCP BOLL MA VMA

PRICE_VOLUME

MIN_MAX

TIME VOLATILITY PATTERN STOCH