Exploring v3 Training Data¶
Training data v3.0 is the biggest data release since the challenge launched: prices are now Hyperliquid-native end to end (you train on exactly what you are scored on), the dataset regenerates daily, and the feature set expands from 80 to 180 columns with the original 80 carried over bit-identical.
TL;DR¶
- Don't ship a v2 model against v3 features. Un-retrained, you give up 13% of your 10d Spearman and 9% of your 30d on our walk-forward split, and up to 25% on other windows. Retraining costs one
fit()and restores all of it. - The new features are the bigger prize. The identical XGBoost gains +25% relative 10d Spearman (+18% on 30d) just by including the new columns, and an LSTM reading the lag axis gains more.
- Nothing else changes. Same ids, same targets, same file conventions; pipelines that select columns by name keep working. Retrain now, keep your current model submitting, deploy at the flip. The flip-day checklist at the bottom is all you strictly need.
Everything below is the evidence, measured in the challenge's own metrics: per-date Spearman and symmetric NDCG@40, computed with crowdcent_challenge.scoring.evaluate_hyperliquid_submission exactly like the leaderboard.
In [1]:
Copied!
from datetime import date
from pathlib import Path
import altair as alt
import numpy as np
import polars as pl
from xgboost import XGBRegressor
from crowdcent_challenge.scoring import evaluate_hyperliquid_submission
alt.data_transformers.disable_max_rows()
DATA = Path("data")
v2_path = DATA / "cc_train_v2.parquet"
v3_path = DATA / "cc_train_v3_preview.parquet"
if not (v2_path.exists() and v3_path.exists()):
import crowdcent_challenge as cc
client = cc.ChallengeClient("hyperliquid-ranking")
client.download_training_dataset("2.0", str(v2_path))
client.download_training_dataset("3.0", str(v3_path)) # "latest" after the flip
v2 = pl.read_parquet(v2_path).with_columns(pl.col("date").cast(pl.Date))
v3 = pl.read_parquet(v3_path).with_columns(pl.col("date").cast(pl.Date))
# Brand-dark chart theme: explicit background so charts stay readable on both
# the light and dark docs themes (Altair's default is transparent + dark text).
@alt.theme.register("crowdcent_dark", enable=True)
def crowdcent_dark():
return {
"config": {
"background": "#0d2230",
"view": {"stroke": "transparent"},
"axis": {
"labelColor": "#b7c7d1", "titleColor": "#d9e4ea",
"gridColor": "#1e3e4f", "domainColor": "#33566b",
"tickColor": "#33566b", "labelFontSize": 12, "titleFontSize": 13,
},
"legend": {"labelColor": "#d9e4ea", "titleColor": "#d9e4ea",
"labelFontSize": 12, "titleFontSize": 12,
"symbolType": "stroke", "symbolStrokeWidth": 3,
"symbolOpacity": 1},
"title": {"color": "#d9e4ea", "fontSize": 14},
"range": {"category": ["#62e4fb", "#ffb86b", "#ff6b81", "#b3b3b3", "#7ee787"]},
"line": {"strokeWidth": 2.5},
}
}
from datetime import date
from pathlib import Path
import altair as alt
import numpy as np
import polars as pl
from xgboost import XGBRegressor
from crowdcent_challenge.scoring import evaluate_hyperliquid_submission
alt.data_transformers.disable_max_rows()
DATA = Path("data")
v2_path = DATA / "cc_train_v2.parquet"
v3_path = DATA / "cc_train_v3_preview.parquet"
if not (v2_path.exists() and v3_path.exists()):
import crowdcent_challenge as cc
client = cc.ChallengeClient("hyperliquid-ranking")
client.download_training_dataset("2.0", str(v2_path))
client.download_training_dataset("3.0", str(v3_path)) # "latest" after the flip
v2 = pl.read_parquet(v2_path).with_columns(pl.col("date").cast(pl.Date))
v3 = pl.read_parquet(v3_path).with_columns(pl.col("date").cast(pl.Date))
# Brand-dark chart theme: explicit background so charts stay readable on both
# the light and dark docs themes (Altair's default is transparent + dark text).
@alt.theme.register("crowdcent_dark", enable=True)
def crowdcent_dark():
return {
"config": {
"background": "#0d2230",
"view": {"stroke": "transparent"},
"axis": {
"labelColor": "#b7c7d1", "titleColor": "#d9e4ea",
"gridColor": "#1e3e4f", "domainColor": "#33566b",
"tickColor": "#33566b", "labelFontSize": 12, "titleFontSize": 13,
},
"legend": {"labelColor": "#d9e4ea", "titleColor": "#d9e4ea",
"labelFontSize": 12, "titleFontSize": 12,
"symbolType": "stroke", "symbolStrokeWidth": 3,
"symbolOpacity": 1},
"title": {"color": "#d9e4ea", "fontSize": 14},
"range": {"category": ["#62e4fb", "#ffb86b", "#ff6b81", "#b3b3b3", "#7ee787"]},
"line": {"strokeWidth": 2.5},
}
}
In [2]:
Copied!
v2_feats = [c for c in v2.columns if c.startswith("feature_")]
v3_feats = [c for c in v3.columns if c.startswith("feature_")]
shared = [c for c in v3_feats if c in set(v2_feats)]
new = [c for c in v3_feats if c not in set(v2_feats)]
new_bases = sorted({c.rsplit("_lag", 1)[0] for c in new}, key=lambda s: int(s.split("_")[1]))
print(f"v2: {v2.shape}, {len(v2_feats)} features, {v2['date'].min()} -> {v2['date'].max()}")
print(f"v3: {v3.shape}, {len(v3_feats)} features, {v3['date'].min()} -> {v3['date'].max()}")
print(f"unchanged features: {len(shared)} | new: {len(new)} ({len(new_bases)} bases x 4 lags)")
print("new bases:", ", ".join(new_bases[:6]), "...", new_bases[-1])
v2_feats = [c for c in v2.columns if c.startswith("feature_")]
v3_feats = [c for c in v3.columns if c.startswith("feature_")]
shared = [c for c in v3_feats if c in set(v2_feats)]
new = [c for c in v3_feats if c not in set(v2_feats)]
new_bases = sorted({c.rsplit("_lag", 1)[0] for c in new}, key=lambda s: int(s.split("_")[1]))
print(f"v2: {v2.shape}, {len(v2_feats)} features, {v2['date'].min()} -> {v2['date'].max()}")
print(f"v3: {v3.shape}, {len(v3_feats)} features, {v3['date'].min()} -> {v3['date'].max()}")
print(f"unchanged features: {len(shared)} | new: {len(new)} ({len(new_bases)} bases x 4 lags)")
print("new bases:", ", ".join(new_bases[:6]), "...", new_bases[-1])
v2: (245930, 85), 80 features, 2019-07-26 -> 2026-01-01 v3: (187620, 185), 180 features, 2020-08-19 -> 2026-06-20 unchanged features: 80 | new: 100 (25 bases x 4 lags) new bases: feature_21, feature_22, feature_23, feature_24, feature_25, feature_26 ... feature_45
The new columns follow every convention the original 80 established: cross-sectional ranks in (0, 1], a neutral 0.5 fill during an asset's warm-up window, lags [0, 5, 10, 15], and the file's lag-major column order (lag-15 block first, oldest to newest). That last detail matters for sequence models: n_features_per_timestep = len(feature_cols) // len(lag_windows) still works, it is just 45 per timestep now instead of 20.
The experiment¶
One honest walk-forward split, mirroring how the challenge scores you:
- Train on everything dated on or before 2025-06-30 (from each dataset).
- Validate on v3 rows from 2025-08-01 onward; the 31+ day gap ensures no training target's forward window touches validation.
- All models are served v3 features at validation time, because that is what the platform serves after the flip. The only choice you control is what your model was trained on.
Three arms:
| arm | trained on | mimics |
|---|---|---|
| v2 model (no retrain) | v2.0, original 80 features | doing nothing on flip day |
| v3 retrained (original 80) | v3.0, original 80 features | a minimal retrain |
| v3 retrained (all 180) | v3.0, all 180 features | using what's new |
In [3]:
Copied!
TRAIN_END = date(2025, 6, 30)
VAL_START = date(2025, 8, 1)
TARGETS = ["target_10d", "target_30d"]
train_v2 = v2.filter(pl.col("date") <= TRAIN_END).drop_nulls(subset=v2_feats + TARGETS)
train_v3 = v3.filter(pl.col("date") <= TRAIN_END).drop_nulls(subset=v3_feats + TARGETS)
val = v3.filter(pl.col("date") >= VAL_START).drop_nulls(subset=TARGETS).sort(["date", "id"])
print(f"train v2: {train_v2.height} rows | train v3: {train_v3.height} rows | "
f"validation: {val.height} rows over {val['date'].n_unique()} dates")
ARMS = {
"v2 model (no retrain)": (train_v2, v2_feats),
"v3 retrained (original 80)": (train_v3, shared),
"v3 retrained (all 180)": (train_v3, v3_feats),
}
def fit_xgb(train_df, feats, target):
m = XGBRegressor(n_estimators=500, learning_rate=0.05, max_depth=6,
subsample=0.8, colsample_bytree=0.8, tree_method="hist",
random_state=42, n_jobs=-1)
m.fit(train_df.select(feats).to_numpy(), train_df[target].to_numpy())
return m
preds = {}
for arm, (train_df, feats) in ARMS.items():
for target in TARGETS:
model = fit_xgb(train_df, feats, target)
preds[(arm, target)] = model.predict(val.select(feats).to_numpy())
print(f"fitted: {arm}")
TRAIN_END = date(2025, 6, 30)
VAL_START = date(2025, 8, 1)
TARGETS = ["target_10d", "target_30d"]
train_v2 = v2.filter(pl.col("date") <= TRAIN_END).drop_nulls(subset=v2_feats + TARGETS)
train_v3 = v3.filter(pl.col("date") <= TRAIN_END).drop_nulls(subset=v3_feats + TARGETS)
val = v3.filter(pl.col("date") >= VAL_START).drop_nulls(subset=TARGETS).sort(["date", "id"])
print(f"train v2: {train_v2.height} rows | train v3: {train_v3.height} rows | "
f"validation: {val.height} rows over {val['date'].n_unique()} dates")
ARMS = {
"v2 model (no retrain)": (train_v2, v2_feats),
"v3 retrained (original 80)": (train_v3, shared),
"v3 retrained (all 180)": (train_v3, v3_feats),
}
def fit_xgb(train_df, feats, target):
m = XGBRegressor(n_estimators=500, learning_rate=0.05, max_depth=6,
subsample=0.8, colsample_bytree=0.8, tree_method="hist",
random_state=42, n_jobs=-1)
m.fit(train_df.select(feats).to_numpy(), train_df[target].to_numpy())
return m
preds = {}
for arm, (train_df, feats) in ARMS.items():
for target in TARGETS:
model = fit_xgb(train_df, feats, target)
preds[(arm, target)] = model.predict(val.select(feats).to_numpy())
print(f"fitted: {arm}")
train v2: 208894 rows | train v3: 124568 rows | validation: 57711 rows over 324 dates
fitted: v2 model (no retrain)
fitted: v3 retrained (original 80)
fitted: v3 retrained (all 180)
In [4]:
Copied!
def per_date_scores(val, preds, arms):
"""Score every validation date with the challenge's own metric function."""
dates_np = val["date"].to_numpy()
y10 = val["target_10d"].to_numpy()
y30 = val["target_30d"].to_numpy()
rows = []
for d in np.unique(dates_np):
m = dates_np == d
if m.sum() < 50:
continue
for arm in arms:
scores = evaluate_hyperliquid_submission(
y10[m], preds[(arm, "target_10d")][m],
y30[m], preds[(arm, "target_30d")][m],
)
rows.append({"date": str(d), "model": arm, **scores})
return pl.DataFrame(rows).with_columns(pl.col("date").str.to_date())
daily = per_date_scores(val, preds, ARMS)
summary = (
daily.group_by("model")
.agg(pl.col("spearman_10d", "spearman_30d", "ndcg@40_10d", "ndcg@40_30d").mean().round(4))
.sort("spearman_10d")
)
summary
def per_date_scores(val, preds, arms):
"""Score every validation date with the challenge's own metric function."""
dates_np = val["date"].to_numpy()
y10 = val["target_10d"].to_numpy()
y30 = val["target_30d"].to_numpy()
rows = []
for d in np.unique(dates_np):
m = dates_np == d
if m.sum() < 50:
continue
for arm in arms:
scores = evaluate_hyperliquid_submission(
y10[m], preds[(arm, "target_10d")][m],
y30[m], preds[(arm, "target_30d")][m],
)
rows.append({"date": str(d), "model": arm, **scores})
return pl.DataFrame(rows).with_columns(pl.col("date").str.to_date())
daily = per_date_scores(val, preds, ARMS)
summary = (
daily.group_by("model")
.agg(pl.col("spearman_10d", "spearman_30d", "ndcg@40_10d", "ndcg@40_30d").mean().round(4))
.sort("spearman_10d")
)
summary
Out[4]:
shape: (3, 5)
| model | spearman_10d | spearman_30d | ndcg@40_10d | ndcg@40_30d |
|---|---|---|---|---|
| str | f64 | f64 | f64 | f64 |
| "v2 model (no retrain)" | 0.0628 | 0.0866 | 0.5745 | 0.5864 |
| "v3 retrained (original 80)" | 0.0722 | 0.0956 | 0.583 | 0.5941 |
| "v3 retrained (all 180)" | 0.0904 | 0.1129 | 0.5958 | 0.6009 |
What the transition does to your scores¶
Four full-width panels, one story: faint points are single release dates, solid lines are 30-day rolling means, dashed rules are each arm's overall mean (gray at the no-skill anchor). The bottom two panels sum daily Spearman into a signal equity curve, which makes regime pockets and steady bleed much easier to see than daily noise. NDCG@40 moves the same way in every comparison; the table above has the levels. Drag any panel to zoom its x-axis.
In [5]:
Copied!
ROLL = 30
long = daily.unpivot(index=["date", "model"], variable_name="metric", value_name="value")
long = long.sort(["model", "metric", "date"]).with_columns(
pl.col("value").rolling_mean(ROLL, min_samples=10).over(["model", "metric"]).alias("ma"),
pl.col("value").cum_sum().over(["model", "metric"]).alias("cum"),
)
# hero arm in brand cyan, minimal retrain in amber, the stale model muted
ARM_SCALE = alt.Scale(domain=list(ARMS),
range=["#93a8b4", "#ffb86b", "#62e4fb"])
SHORT = {"v2 model (no retrain)": "no retrain",
"v3 retrained (original 80)": "retrained 80",
"v3 retrained (all 180)": "retrained 180"}
COLORS = alt.Color("model:N", scale=ARM_SCALE,
legend=alt.Legend(orient="bottom", columns=1, title=None))
brush = alt.selection_interval(bind="scales", encodings=["x"])
def panel(metric, title, anchor, y_field="ma", y_title="30d rolling mean"):
src = long.filter(pl.col("metric") == metric).to_pandas()
means = src.groupby("model")["value"].mean().reindex(list(ARMS))
pts = (alt.Chart(src).mark_point(opacity=0.15, size=8)
.encode(x=alt.X("date:T", title=None), y=alt.Y("value:Q", title=None), color=COLORS)
) if y_field == "ma" else None
line = (alt.Chart(src).mark_line(strokeWidth=2)
.encode(x=alt.X("date:T", title=None),
y=alt.Y(f"{y_field}:Q", title=y_title),
color=COLORS,
tooltip=["date:T", "model:N", alt.Tooltip(f"{y_field}:Q", format=".3f")]))
layers = ([pts] if pts is not None else []) + [line]
if y_field == "ma":
layers += [alt.Chart(src).mark_rule(strokeDash=[2, 2], color="gray").encode(y=alt.datum(anchor))]
mean_txt = " | ".join(f"{SHORT[m]} {v:.3f}" for m, v in means.items())
return (alt.layer(*layers).add_params(brush)
.properties(width=620, height=200,
title=alt.TitleParams(f"{title} ({mean_txt})", fontSize=12, anchor="start")))
chart = alt.vconcat(
panel("spearman_10d", "Spearman 10d", 0),
panel("spearman_30d", "Spearman 30d", 0),
panel("spearman_10d", "Cumulative daily Spearman 10d", 0, "cum", "cumulative"),
panel("spearman_30d", "Cumulative daily Spearman 30d", 0, "cum", "cumulative"),
).resolve_scale(color="shared")
chart
ROLL = 30
long = daily.unpivot(index=["date", "model"], variable_name="metric", value_name="value")
long = long.sort(["model", "metric", "date"]).with_columns(
pl.col("value").rolling_mean(ROLL, min_samples=10).over(["model", "metric"]).alias("ma"),
pl.col("value").cum_sum().over(["model", "metric"]).alias("cum"),
)
# hero arm in brand cyan, minimal retrain in amber, the stale model muted
ARM_SCALE = alt.Scale(domain=list(ARMS),
range=["#93a8b4", "#ffb86b", "#62e4fb"])
SHORT = {"v2 model (no retrain)": "no retrain",
"v3 retrained (original 80)": "retrained 80",
"v3 retrained (all 180)": "retrained 180"}
COLORS = alt.Color("model:N", scale=ARM_SCALE,
legend=alt.Legend(orient="bottom", columns=1, title=None))
brush = alt.selection_interval(bind="scales", encodings=["x"])
def panel(metric, title, anchor, y_field="ma", y_title="30d rolling mean"):
src = long.filter(pl.col("metric") == metric).to_pandas()
means = src.groupby("model")["value"].mean().reindex(list(ARMS))
pts = (alt.Chart(src).mark_point(opacity=0.15, size=8)
.encode(x=alt.X("date:T", title=None), y=alt.Y("value:Q", title=None), color=COLORS)
) if y_field == "ma" else None
line = (alt.Chart(src).mark_line(strokeWidth=2)
.encode(x=alt.X("date:T", title=None),
y=alt.Y(f"{y_field}:Q", title=y_title),
color=COLORS,
tooltip=["date:T", "model:N", alt.Tooltip(f"{y_field}:Q", format=".3f")]))
layers = ([pts] if pts is not None else []) + [line]
if y_field == "ma":
layers += [alt.Chart(src).mark_rule(strokeDash=[2, 2], color="gray").encode(y=alt.datum(anchor))]
mean_txt = " | ".join(f"{SHORT[m]} {v:.3f}" for m, v in means.items())
return (alt.layer(*layers).add_params(brush)
.properties(width=620, height=200,
title=alt.TitleParams(f"{title} ({mean_txt})", fontSize=12, anchor="start")))
chart = alt.vconcat(
panel("spearman_10d", "Spearman 10d", 0),
panel("spearman_30d", "Spearman 30d", 0),
panel("spearman_10d", "Cumulative daily Spearman 10d", 0, "cum", "cumulative"),
panel("spearman_30d", "Cumulative daily Spearman 30d", 0, "cum", "cumulative"),
).resolve_scale(color="shared")
chart
/home/exx/.cache/uv/archive-v0/7r6-INUKdc0t3SzCwRTTX/lib/python3.12/site-packages/IPython/core/interactiveshell.py:3701: UserWarning: Automatically deduplicated selection parameter with identical configuration. If you want independent parameters, explicitly name them differently (e.g., name='param1', name='param2'). See https://github.com/vega/altair/issues/3891 exec(code_obj, self.user_global_ns, self.user_ns)
Out[5]:
So what are the new columns?¶
They are obfuscated, but everything that matters for modeling is measurable. The table below looks at each base feature through three lenses, using the lag-0 columns (the other lags are the same series shifted):
- signal: mean per-date Spearman against each target;
- persistence: day-over-day autocorrelation of the rank series, separating fast signals from slow asset characteristics;
- novelty: the strongest absolute correlation with any original base, showing which new columns are genuinely new information rather than transformations of what you had.
In [6]:
Copied!
BASES_OLD = sorted({c.rsplit("_lag", 1)[0] for c in shared},
key=lambda s: int(s.split("_")[1]))
bases = BASES_OLD + new_bases
lag0 = [f"{b}_lag0" for b in bases]
n_old = len(BASES_OLD)
# signal: per-date Spearman of every base vs both targets, in one pass
ic_by_date = v3.group_by("date").agg(
pl.corr(pl.col(f"{b}_lag0"), pl.col(t), method="spearman").alias(f"{b}|{t}")
for b in bases
for t in ("target_10d", "target_30d")
).sort("date")
# dates where a feature is cross-sectionally constant (warm-up) give NaN: drop, don't propagate
ic_by_date = ic_by_date.with_columns(pl.exclude("date").fill_nan(None))
# persistence: mean per-asset day-over-day autocorrelation of the rank series
pers = (
v3.sort(["id", "date"])
.group_by("id")
.agg(pl.corr(pl.col(c), pl.col(c).shift(1)).alias(c) for c in lag0)
.select(lag0)
.mean()
)
# novelty: strongest |corr| of each new base against any original (recent 2y)
recent = v3.filter(pl.col("date") >= v3["date"].max() - pl.duration(days=730))
C = np.abs(np.corrcoef(recent.select(lag0).to_numpy(), rowvar=False))
novelty = [None] * n_old + [round(float(np.nanmax(C[i, :n_old])), 3)
for i in range(n_old, len(bases))]
summary = pl.DataFrame({
"base": bases,
"new": [i >= n_old for i in range(len(bases))],
"ic_10d": [ic_by_date[f"{b}|target_10d"].mean() for b in bases],
"ic_30d": [ic_by_date[f"{b}|target_30d"].mean() for b in bases],
"persistence": [pers[f"{b}_lag0"][0] for b in bases],
"max_corr_vs_originals": novelty,
}).with_columns(pl.col("ic_10d", "ic_30d", "persistence").round(3))
best_old = summary.filter(~pl.col("new"))["ic_30d"].abs().max()
stronger = summary.filter(pl.col("new") & (pl.col("ic_30d").abs() > best_old)).height
print(f"strongest ORIGINAL base, |mean per-date Spearman| (30d): {best_old:.3f}")
print(f"new bases individually stronger than that: {stronger}/25")
summary.sort(pl.col("ic_30d").abs(), descending=True).head(12)
BASES_OLD = sorted({c.rsplit("_lag", 1)[0] for c in shared},
key=lambda s: int(s.split("_")[1]))
bases = BASES_OLD + new_bases
lag0 = [f"{b}_lag0" for b in bases]
n_old = len(BASES_OLD)
# signal: per-date Spearman of every base vs both targets, in one pass
ic_by_date = v3.group_by("date").agg(
pl.corr(pl.col(f"{b}_lag0"), pl.col(t), method="spearman").alias(f"{b}|{t}")
for b in bases
for t in ("target_10d", "target_30d")
).sort("date")
# dates where a feature is cross-sectionally constant (warm-up) give NaN: drop, don't propagate
ic_by_date = ic_by_date.with_columns(pl.exclude("date").fill_nan(None))
# persistence: mean per-asset day-over-day autocorrelation of the rank series
pers = (
v3.sort(["id", "date"])
.group_by("id")
.agg(pl.corr(pl.col(c), pl.col(c).shift(1)).alias(c) for c in lag0)
.select(lag0)
.mean()
)
# novelty: strongest |corr| of each new base against any original (recent 2y)
recent = v3.filter(pl.col("date") >= v3["date"].max() - pl.duration(days=730))
C = np.abs(np.corrcoef(recent.select(lag0).to_numpy(), rowvar=False))
novelty = [None] * n_old + [round(float(np.nanmax(C[i, :n_old])), 3)
for i in range(n_old, len(bases))]
summary = pl.DataFrame({
"base": bases,
"new": [i >= n_old for i in range(len(bases))],
"ic_10d": [ic_by_date[f"{b}|target_10d"].mean() for b in bases],
"ic_30d": [ic_by_date[f"{b}|target_30d"].mean() for b in bases],
"persistence": [pers[f"{b}_lag0"][0] for b in bases],
"max_corr_vs_originals": novelty,
}).with_columns(pl.col("ic_10d", "ic_30d", "persistence").round(3))
best_old = summary.filter(~pl.col("new"))["ic_30d"].abs().max()
stronger = summary.filter(pl.col("new") & (pl.col("ic_30d").abs() > best_old)).height
print(f"strongest ORIGINAL base, |mean per-date Spearman| (30d): {best_old:.3f}")
print(f"new bases individually stronger than that: {stronger}/25")
summary.sort(pl.col("ic_30d").abs(), descending=True).head(12)
strongest ORIGINAL base, |mean per-date Spearman| (30d): 0.073 new bases individually stronger than that: 7/25
Out[6]:
shape: (12, 6)
| base | new | ic_10d | ic_30d | persistence | max_corr_vs_originals |
|---|---|---|---|---|---|
| str | bool | f64 | f64 | f64 | f64 |
| "feature_38" | true | -0.116 | -0.151 | 0.972 | 0.372 |
| "feature_42" | true | -0.114 | -0.148 | 0.98 | 0.397 |
| "feature_21" | true | 0.099 | 0.142 | 0.999 | 0.406 |
| "feature_41" | true | -0.106 | -0.14 | 0.957 | 0.323 |
| "feature_28" | true | -0.109 | -0.134 | 0.984 | 0.311 |
| … | … | … | … | … | … |
| "feature_3" | false | 0.058 | 0.073 | 0.966 | null |
| "feature_15" | false | 0.061 | 0.071 | 0.966 | null |
| "feature_7" | false | 0.062 | 0.07 | 0.967 | null |
| "feature_27" | true | 0.049 | 0.07 | 0.909 | 0.727 |
| "feature_2" | false | 0.051 | 0.065 | 0.937 | null |
In [7]:
Copied!
# regime honesty: rolling 60d mean of per-date Spearman, top new bases vs best original
top_new = (summary.filter(pl.col("new"))
.sort(pl.col("ic_30d").abs(), descending=True)["base"].head(3).to_list())
best_old_base = (summary.filter(~pl.col("new"))
.sort(pl.col("ic_30d").abs(), descending=True)["base"][0])
show = top_new + [best_old_base]
roll_ic = ic_by_date.select(
["date"] + [pl.col(f"{b}|target_30d").rolling_mean(60, min_samples=30).alias(b) for b in show]
).unpivot(index="date", variable_name="base", value_name="ic60")
alt.Chart(roll_ic.to_pandas()).mark_line().encode(
x=alt.X("date:T", title=None),
y=alt.Y("ic60:Q", title="60d rolling mean of per-date Spearman (30d target)"),
color=alt.Color("base:N", legend=alt.Legend(orient="bottom")),
tooltip=["date:T", "base:N", alt.Tooltip("ic60:Q", format=".3f")],
).properties(width=620, height=260) + alt.Chart(roll_ic.to_pandas()).mark_rule(
strokeDash=[2, 2], color="gray").encode(y=alt.datum(0))
# regime honesty: rolling 60d mean of per-date Spearman, top new bases vs best original
top_new = (summary.filter(pl.col("new"))
.sort(pl.col("ic_30d").abs(), descending=True)["base"].head(3).to_list())
best_old_base = (summary.filter(~pl.col("new"))
.sort(pl.col("ic_30d").abs(), descending=True)["base"][0])
show = top_new + [best_old_base]
roll_ic = ic_by_date.select(
["date"] + [pl.col(f"{b}|target_30d").rolling_mean(60, min_samples=30).alias(b) for b in show]
).unpivot(index="date", variable_name="base", value_name="ic60")
alt.Chart(roll_ic.to_pandas()).mark_line().encode(
x=alt.X("date:T", title=None),
y=alt.Y("ic60:Q", title="60d rolling mean of per-date Spearman (30d target)"),
color=alt.Color("base:N", legend=alt.Legend(orient="bottom")),
tooltip=["date:T", "base:N", alt.Tooltip("ic60:Q", format=".3f")],
).properties(width=620, height=260) + alt.Chart(roll_ic.to_pandas()).mark_rule(
strokeDash=[2, 2], color="gray").encode(y=alt.datum(0))
Out[7]:
How to read this before you train:
- Several new bases are individually stronger than anything in the original set. The top of the table is dominated by new columns, with mean per-date Spearman magnitudes the originals never reach. Signs differ (some rank against the target by construction); tree and neural models don't care, but check the sign if you build linear features.
- Persistence splits the new features into two species. Values near 1.0 are slow-moving asset characteristics that change composition over weeks; low values are fast signals. The slow columns behave like asset descriptors, useful for interactions and for sequence models reading the lag axis.
- The novelty column says the new information is real. Most new bases correlate only weakly with their closest original, so they are additions to the feature space, not transformations of it. Where a value is high the columns overlap; prune freely if you want a lean model.
- The rolling chart is the honesty panel. Signal strength varies by regime and every feature has weak stretches. Validate walk-forward (as below) rather than extrapolating any single window, and expect drawdowns in feature performance, not just portfolio performance.
The new features and sequence models¶
The 4-lag structure is not decoration, it is a sequence. centimators' SequenceEstimator family (LSTMRegressor, TransformerRegressor) reshapes the flat feature matrix into (samples, 4 timesteps, features_per_timestep) directly from the file's column order, so the expanded dataset feeds them with zero code changes beyond the feature count. (See the CV + LSTM tutorial for the full pattern.)
In our testing the new features are worth even more to sequence models than to trees: the risk features have temporal dynamics (volatility regimes, liquidity trends) that an LSTM can read across the lag axis but a flat snapshot cannot. Below, the same split, multi-target LSTM on the original 80 (4x20) vs all 180 (4x45).
In [8]:
Copied!
import os
os.environ["KERAS_BACKEND"] = "jax"
import keras
from centimators.model_estimators import LSTMRegressor
LAG_WINDOWS = [0, 5, 10, 15]
lstm_arms = {"LSTM (original 80)": shared, "LSTM (all 180)": v3_feats}
for arm, feats in lstm_arms.items():
keras.utils.set_random_seed(42)
model = LSTMRegressor(output_units=2, lag_windows=LAG_WINDOWS,
n_features_per_timestep=len(feats) // len(LAG_WINDOWS))
model.fit(train_v3.select(feats), train_v3.select(TARGETS).to_numpy(),
epochs=10, batch_size=1024, verbose=0)
pred = np.asarray(model.predict(val.select(feats), batch_size=4096, verbose=0))
preds[(arm, "target_10d")], preds[(arm, "target_30d")] = pred[:, 0], pred[:, 1]
print(f"fitted: {arm}")
daily_all = per_date_scores(val, preds, list(ARMS) + list(lstm_arms))
(daily_all.group_by("model")
.agg(pl.col("spearman_10d", "spearman_30d", "ndcg@40_10d", "ndcg@40_30d").mean().round(4))
.sort("spearman_30d"))
import os
os.environ["KERAS_BACKEND"] = "jax"
import keras
from centimators.model_estimators import LSTMRegressor
LAG_WINDOWS = [0, 5, 10, 15]
lstm_arms = {"LSTM (original 80)": shared, "LSTM (all 180)": v3_feats}
for arm, feats in lstm_arms.items():
keras.utils.set_random_seed(42)
model = LSTMRegressor(output_units=2, lag_windows=LAG_WINDOWS,
n_features_per_timestep=len(feats) // len(LAG_WINDOWS))
model.fit(train_v3.select(feats), train_v3.select(TARGETS).to_numpy(),
epochs=10, batch_size=1024, verbose=0)
pred = np.asarray(model.predict(val.select(feats), batch_size=4096, verbose=0))
preds[(arm, "target_10d")], preds[(arm, "target_30d")] = pred[:, 0], pred[:, 1]
print(f"fitted: {arm}")
daily_all = per_date_scores(val, preds, list(ARMS) + list(lstm_arms))
(daily_all.group_by("model")
.agg(pl.col("spearman_10d", "spearman_30d", "ndcg@40_10d", "ndcg@40_30d").mean().round(4))
.sort("spearman_30d"))
fitted: LSTM (original 80)
fitted: LSTM (all 180)
Out[8]:
shape: (5, 5)
| model | spearman_10d | spearman_30d | ndcg@40_10d | ndcg@40_30d |
|---|---|---|---|---|
| str | f64 | f64 | f64 | f64 |
| "v2 model (no retrain)" | 0.0628 | 0.0866 | 0.5745 | 0.5864 |
| "v3 retrained (original 80)" | 0.0722 | 0.0956 | 0.583 | 0.5941 |
| "v3 retrained (all 180)" | 0.0904 | 0.1129 | 0.5958 | 0.6009 |
| "LSTM (original 80)" | 0.0818 | 0.1151 | 0.5885 | 0.6032 |
| "LSTM (all 180)" | 0.1118 | 0.1521 | 0.6089 | 0.6306 |
Flip-day checklist¶
Retrain now, deploy at the flip. Download with
client.download_training_dataset("3.0", ...)today ("latest" after the flip) and retrain. But keep your current model submitting until the flip date: daily inference files carry v2 features until then, and a v3-trained model served v2 features suffers the same venue-volume mismatch in reverse.Switch on column count. The daily inference file grows from 83 to 183 columns at the flip, so an automated pipeline can pick the right model with no date logic:
feature_cols = [c for c in inf.columns if c.startswith("feature_")] model = model_v3 if len(feature_cols) >= 180 else model_v2
Nothing else changes: same ids, same targets, same file conventions. Column selection by name keeps working and the original 80 features are bit-identical. The dataset now refreshes daily, so retraining on a schedule is worth automating; see submission automation.