[SYSTEMDS-3863] Add PowerTransformer built-in functions by WenliangCao · Pull Request #2499 · apache/systemds

WenliangCao · 2026-06-21T21:54:37Z

Summary

This pull request adds PowerTransformer built-ins with separate fit and apply workflows. The implementation supports Yeo-Johnson and Box-Cox transformations, per-column parameter estimation, optional standardization, and reuse of fitted parameters on new data.

Yeo-Johnson is the default method and supports mixed-sign inputs. Box-Cox is available for strictly positive inputs.

API

[Y, lambdas, means, scales] = powerTransform(
    X,
    method="yeo-johnson",
    standardize=TRUE
)

Y = powerTransformApply(
    X,
    lambdas,
    means,
    scales,
    method="yeo-johnson"
)

powerTransform estimates one lambda per column and optionally standardizes the transformed columns. powerTransformApply reuses the fitted lambdas, means, and scales without re-estimation.

Implementation

Add powerTransform.dml for fitting and transforming feature matrices.
Add powerTransformApply.dml for applying fitted transformations.
Support Yeo-Johnson and Box-Cox transformations.
Keep Yeo-Johnson as the default method.
Support optional zero-mean, unit-variance standardization.
Estimate each lambda by minimizing the negative log-likelihood with Brent optimization.
Treat [-2, 2] as an initial bracket and expand it when necessary, allowing the optimum to lie outside the initial interval.
Validate method names, fitted-state dimensions, and Box-Cox input requirements.
Register both functions as SystemDS built-ins.
Add entries for both functions to the built-in reference documentation.

Validation

mvn -Dtest=BuiltinPowerTransformTest test

Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

The tests cover:

default Yeo-Johnson fitting with mixed-sign and constant columns in CP and Spark;
fitted Yeo-Johnson apply workflows in CP and Spark;
Box-Cox fit and apply workflows;
Yeo-Johnson and Box-Cox optima outside the initial [-2, 2] interval;
NaN preservation and finite-lambda fallback behavior;
rejection of non-positive Box-Cox input; and
standardization state reuse.

Fitted Yeo-Johnson and Box-Cox outputs are compared against the established R recipes package.

Documentation

User-facing signatures, arguments, return values, input requirements, and examples are documented in docs/site/builtins-reference.md.

Experiments

The reproducible evaluation compares no scaling, standard scaling, robust scaling, Yeo-Johnson, and Box-Cox on nine public datasets. Each experiment uses the fixed seeds 13, 37, 73, 101, and 149. All raw results, summaries, figures, the workbook, and the report were regenerated with the final implementation.

Task	Estimator	Primary metric
Regression	5-nearest-neighbors regressor	RMSE (lower is better)
Classification	5-nearest-neighbors classifier	Macro-F1 (higher is better)
Clustering	K-means (`n_init=20`)	ARI (higher is better)

Supervised tasks use 80/20 train/test splits; classification is stratified and Parkinsons Telemonitoring uses subject-level group splits. Each transformation is fitted on training data and applied to test data. Clustering uses the full dataset.

The table reports the mean primary metric over five runs. A dash indicates that Box-Cox is not applicable because the feature matrix contains non-positive values.

Task	Dataset	Metric	None	Standard	Robust	Yeo-Johnson	Box-Cox
Regression	California Housing	RMSE	1.0607	0.6486	0.6460	0.6114	—
Regression	Abalone	RMSE	2.2966	2.3575	2.3673	2.3547	2.3580
Regression	Parkinsons Telemonitoring	RMSE	12.8276	13.0932	12.6718	12.7222	—
Classification	Breast Cancer Wisconsin	Macro-F1	0.9303	0.9559	0.9560	0.9618	—
Classification	Wine	Macro-F1	0.6448	0.9734	0.9629	0.9683	0.9683
Classification	HTRU2	Macro-F1	0.9116	0.9312	0.9339	0.9327	—
Clustering	Iris	ARI	0.7302	0.6201	0.5803	0.6410	0.6180
Clustering	Seeds	ARI	0.7166	0.7733	0.7722	0.7759	0.7759
Clustering	Wholesale Customers	ARI	-0.0306	0.1873	-0.0024	0.5264	0.5330

Power transformations rank first or tie for first on four of the nine datasets. Representative results include:

California Housing: Yeo-Johnson reduces RMSE from 0.6486 with standard scaling to 0.6114.
Breast Cancer Wisconsin: Yeo-Johnson achieves the highest Macro-F1 at 0.9618.
Seeds: Yeo-Johnson and Box-Cox tie for the highest ARI at 0.7759.
Wholesale Customers: Box-Cox achieves an ARI of 0.5330, compared with 0.1873 for standard scaling.

Reproducible artifacts

Limitations

The downstream effect of preprocessing is dataset- and estimator-dependent. The evaluation uses fixed distance-based estimators, three datasets per task, and no hyperparameter search or statistical significance test. The results therefore demonstrate practical cases where power transformations help, rather than universal superiority over other preprocessing methods.

Related issue

SYSTEMDS-3863

codecov · 2026-06-23T13:53:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.65%. Comparing base (af500c8) to head (a9cde52).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2499      +/-   ##
============================================
+ Coverage     71.61%   71.65%   +0.04%     
- Complexity    50352    50490     +138     
============================================
  Files          1626     1628       +2     
  Lines        194664   195069     +405     
  Branches      38007    38042      +35     
============================================
+ Hits         139408   139782     +374     
- Misses        44328    44364      +36     
+ Partials      10928    10923       -5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add two DML builtins for the midterm PowerTransformer prototype: - powerTransform.dml estimates one Yeo-Johnson lambda per input column, uses lambda = 1 for constant columns, and applies the transform. - powerTransformApply.dml applies the Yeo-Johnson transform with supplied per-column lambdas. Register both builtins so they can be resolved by SystemDS. Add powerTransformSmokeTest.dml to verify: - powerTransformApply with lambda = 1 behaves as identity - powerTransform returns output dimensions matching the input - one lambda is returned per input column - constant columns use lambda = 1 - transformed output and lambdas do not contain NaN or Inf The smoke test passes with: ./bin/systemds src/test/scripts/functions/builtin/powerTransformSmokeTest.dml

Add a focused PowerTransformApply test that compares the DML builtin against an independent R reference implementation. The test adds: - powerTransformApply.dml as a small DML wrapper around the registered builtin - powerTransformApply.R as the reference Yeo-Johnson apply implementation - BuiltinPowerTransformTest.java to run the DML and R scripts and compare Y The test covers the key apply branches with fixed lambdas: - lambda = 0 for the positive log branch - lambda = 1 for the identity-style middle case - lambda = 2 for the negative log branch Also translate the remaining PowerTransformer comments from Chinese to English in the prototype DML files. Verified with: Rscript -e 'parse(file="src/test/scripts/functions/builtin/powerTransformApply.R"); cat("R syntax OK\n")' ./bin/systemds src/test/scripts/functions/builtin/powerTransformSmokeTest.dml mvn -Dtest=BuiltinPowerTransformTest test

gaturchenko

Hey @WenliangCao, thank you very much for the contribution and extensive experiments. I have left some comments, please address them so that we can merge your code into the main. The main issue however is a failing test, we cannot merge if the CI is red, so please address it first. Feel free to reach out via an email or right here in the PR thread if anything is unclear or if you disagree with some of my comments, we can absolutely discuss anything.

gaturchenko · 2026-08-03T14:13:16Z

+
+    # Apply one transform with this lambda and use the temporary y for scoring
+    emptyStats = matrix(0.0, rows=0, cols=0)
+    y = powerTransformApply(x, lambdaMatrix, emptyStats, emptyStats, method);


This has a fixed invocation overhead, and for matrices with, say, 1000 rows and 4 columns, this will dominate the runtime, so it should be inlined somehow

gaturchenko · 2026-08-03T14:18:42Z

+
+        # Jacobian term for the selected transformation
+        if (method == "box-cox") {
+            jacobian = (lambda - 1.0) * sum(log(x));


Only lambda changes here, x is always the same, so you can compute sum(log(x)) just once, which can be done with a separate function. Then you can add something like jacTerm as another input to ptNegLogLikelihood and get jacobian = (lambda - 1.0) * jacTerm;. This should improve performance substantially for large matrices

gaturchenko · 2026-08-03T14:19:24Z

+            jacobian = (lambda - 1.0) * sum(log(x));
+        }
+        else {
+            jacobian = (lambda - 1.0) * sum(sign(x) * log(abs(x) + 1.0));


Same logic as for the comment above

gaturchenko · 2026-08-03T14:30:59Z

+
+	private void runPowerTransformTest(
+		String method, boolean standardize, double[][] input, boolean shouldFail) {
+		ExecMode oldExecMode = setExecMode(ExecType.CP);


You only add CP coverage, would be good to have Spark covered as well. Let us know if there are particular issues with Spark for Power Transform however

Sign in to view

+	}
+
+	@Test
+	public void testPowerTransformYeoJohnsonLambdaAboveInitialInterval() {


Sign in to view

Add the required blank line after the Apache license headers in the PowerTransformer fit and apply built-ins. Remove the redundant -exec singlenode arguments from the fit and apply tests, which already select the execution mode through the test framework. Apply dev/format-changed.sh against upstream/main to format PR-edited Java lines with dev/CodeStyle_eclipse.xml and address the Java Format Check failure.

gaturchenko · 2026-08-04T11:09:38Z

In addition the format check fails @WenliangCao, please run dev/format-changed.sh from SystemDS root to fix

gaturchenko · 2026-08-11T08:41:27Z

+#   X            Input feature matrix of shape n-by-m
+#   method       Power transformation method: "yeo-johnson" (default) or "box-cox"
+#   standardize  Whether to normalize transformed columns to zero mean and unit variance


Please, remove 2 spaces after # such that you have 1 space in between # and the text that follows

gaturchenko · 2026-08-11T08:41:32Z

+#   Y        Power-transformed matrix of shape n-by-m
+#   lambdas  Estimated lambda parameters of shape 1-by-m, one per column
+#   means    Transformed column means of shape 1-by-m, or an empty matrix when not standardized
+#   scales   Transformed column scales of shape 1-by-m, or an empty matrix when not standardized


Please, remove 2 spaces after # such that you have 1 space in between # and the text that follows

gaturchenko · 2026-08-11T08:41:46Z

+#   X        Input feature matrix of shape n-by-m
+#   lambdas  Precomputed lambda parameters of shape 1-by-m, one per column
+#   means    Transformed column means of shape 1-by-m; empty to skip standardization
+#   scales   Transformed column scales of shape 1-by-m; empty to skip standardization
+#   method   Power transformation method: "yeo-johnson" (default) or "box-cox"


Please, remove 2 spaces after # such that you have 1 space in between # and the text that follows

gaturchenko · 2026-08-11T08:41:51Z

+#
+# OUTPUT:
+# -------------------------------------------------------------------------------------
+#   Y    Power-transformed matrix of shape n-by-m


Please, remove 2 spaces after # such that you have 1 space in between # and the text that follows

gaturchenko · 2026-08-11T08:50:52Z

+    if (sum(is.na(y)) > 0 | sum(is.infinite(y)) > 0 |
+        is.na(yVariance) | is.infinite(yVariance) | yVariance <= 0.0) {


Just if (is.na(yVariance) | is.infinite(yVariance) | yVariance <= 0.0) will suffice

gaturchenko · 2026-08-11T08:58:08Z

Hey @WenliangCao, thank you very much for addressing the initial review, you've done a very good job. There a few more minor things to resolve, I'd appreciate if you could look into it. Once it's done, we're good to merge your code.

WenliangCao · 2026-08-11T09:07:59Z

Hey @WenliangCao, thank you very much for addressing the initial review, you've done a very good job. There a few more minor things to resolve, I'd appreciate if you could look into it. Once it's done, we're good to merge your code.

Hi Grigorii, thank you very much for taking the time to review my code and for your helpful guidance. I have addressed the remaining comments and pushed the updates. I hope everything looks good now, and please let me know if any further adjustments are needed.

gaturchenko · 2026-08-11T09:21:51Z

Hey @WenliangCao, thank you very much for addressing the initial review, you've done a very good job. There a few more minor things to resolve, I'd appreciate if you could look into it. Once it's done, we're good to merge your code.

Hi Grigorii, thank you very much for taking the time to review my code and for your helpful guidance. I have addressed the remaining comments and pushed the updates. I hope everything looks good now, and please let me know if any further adjustments are needed.

Thank you for addressing these. Currently, the code style check is failing, please run dev/format-changed.sh from the root of the project to fix.

WenliangCao · 2026-08-11T09:39:18Z

Hey @WenliangCao, thank you very much for addressing the initial review, you've done a very good job. There a few more minor things to resolve, I'd appreciate if you could look into it. Once it's done, we're good to merge your code.

Hi Grigorii, thank you very much for taking the time to review my code and for your helpful guidance. I have addressed the remaining comments and pushed the updates. I hope everything looks good now, and please let me know if any further adjustments are needed.

Thank you for addressing these. Currently, the code style check is failing, please run dev/format-changed.sh from the root of the project to fix.

Hi Grigorii @gaturchenko , after checking the details of the Java Format Check, I found that it used base SHA 759d5b6 from August 4. The files reported as incorrectly formatted were therefore introduced by later changes on main, rather than by my PR. I tried synchronizing my branch with the current main tip, 9dccbc3 from August 8, and reran dev/format-changed.sh locally. The local format check now passes, and I hope the cloud CI will pass this time as well.

github-project-automation Bot added this to SystemDS PR Queue Jun 21, 2026

github-project-automation Bot moved this to In Progress in SystemDS PR Queue Jun 21, 2026

WenliangCao added 8 commits August 2, 2026 23:09

[SYSTEMDS-3863] Add PowerTransformer method and standardization

4a85849

[SYSTEMDS-3863] Add Box-Cox PowerTransformer support

9aaeddc

[SYSTEMDS-3863] Use Brent optimization for PowerTransformer

d1104d2

[SYSTEMDS-3863] Add PowerTransformer reference tests

826fe6a

Extend PowerTransform lambda search bracket

807fa87

[SYSTEMDS-3863] Document PowerTransformer built-ins

753c5a0

WenliangCao force-pushed the systemds-3863-power-transformer branch from 130c741 to 753c5a0 Compare August 2, 2026 21:16

[SYSTEMDS-3863] Stabilize PowerTransformer boundary test

a9cde52

gaturchenko suggested changes Aug 3, 2026

View reviewed changes

github-project-automation Bot moved this from In Progress to In Review in SystemDS PR Queue Aug 3, 2026

WenliangCao added 3 commits August 4, 2026 11:44

[SYSTEMDS-3863] Use recipes PowerTransformer references

416e1d7

[SYSTEMDS-3863] Optimize PowerTransformer and preserve NaNs

7821e6b

[SYSTEMDS-3863] Add PowerTransformer Spark tests

6f2a4d1

WenliangCao force-pushed the systemds-3863-power-transformer branch from a9cde52 to 5df732f Compare August 4, 2026 14:43

[SYSTEMDS-3863] Validate lambda bounds with recipes

280e878

WenliangCao force-pushed the systemds-3863-power-transformer branch from 5df732f to 280e878 Compare August 4, 2026 14:54

WenliangCao requested a review from gaturchenko August 5, 2026 15:46

gaturchenko suggested changes Aug 11, 2026

View reviewed changes

[SYSTEMDS-3863] Address follow-up review comments

d92a3b6

Merge upstream/main into systemds-3863-power-transformer

d15e602

		if (sum(is.na(y)) > 0 \| sum(is.infinite(y)) > 0 \|
		is.na(yVariance) \| is.infinite(yVariance) \| yVariance <= 0.0) {

Uh oh!

Conversation

WenliangCao commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

API

Implementation

Validation

Documentation

Experiments

Reproducible artifacts

Limitations

Related issue

Uh oh!

codecov Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gaturchenko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

gaturchenko commented Aug 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaturchenko commented Aug 11, 2026

Uh oh!

WenliangCao commented Aug 11, 2026

Uh oh!

gaturchenko commented Aug 11, 2026

Uh oh!

WenliangCao commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

WenliangCao commented Jun 21, 2026 •

edited

Loading

codecov Bot commented Jun 23, 2026 •

edited

Loading