Page MenuHomePhabricator

Bug 1858065 - Replace distutils' StrictVersion r=saschanaz!
ClosedPublic

Authored by sergesanspaille on Oct 10 2023, 7:37 AM.
Referenced Files
Unknown Object (File)
Wed, Oct 15, 11:32 PM
Unknown Object (File)
Wed, Oct 15, 7:16 AM
Unknown Object (File)
Tue, Oct 14, 3:38 AM
Unknown Object (File)
Jan 17 2025, 1:38 PM
Unknown Object (File)
Oct 11 2023, 12:12 PM

Details

Summary

distutils have been removed from Python 3.12, so replace it:

Sometimes using packaging's Version, sometimes providing our own in the
case of mozrelease/versions.py. Add more tests for the latter.

Test Plan

Diff Detail

Event Timeline

phab-bot changed the visibility from "Custom Policy" to "Public (No Login Required)".
phab-bot changed the edit policy from "Custom Policy" to "Restricted Project (Project)".
phab-bot removed a project: secure-revision.

Code analysis found 2 defects in diff 773526:

  • 1 defect found by py-black (Mozlint)
  • 1 defect found by py-ruff (Mozlint)
IMPORTANT: Found 2 defects (error level) that must be fixed before landing.

You can run this analysis locally with:

  • ./mach lint --warnings --outgoing

If you see a problem in this automated review, please report it here.

You can view these defects in the Diff Detail section of Phabricator diff 773526.

This is nice! Can you also cover tools/power/mach_commands.py too, because on Windows the build hits is_osx_10_10_or_greater and fails there. I locally copypasted your fix and it works nice there too!

But I accept this as this is nice as-is (assuming you are going to fix the lint errors)

This revision is now accepted and ready to land.Oct 10 2023, 11:43 AM

2 defects closed compared to the previous diff 773526.


If you see a problem in this automated review, please report it here.

sergesanspaille retitled this revision from Bug 1858065 - Use packaging's Version instead of distutils' StrictVersion r=saschanaz to Bug 1858065 - Use packaging's Version instead of distutils' StrictVersion r=saschanaz!.
This revision now requires review to proceed.Oct 16 2023, 11:25 AM
This revision is now accepted and ready to land.Oct 16 2023, 12:16 PM
This revision now requires review to proceed.Oct 16 2023, 12:16 PM
This revision is now accepted and ready to land.Oct 16 2023, 12:17 PM
This revision now requires review to proceed.Oct 16 2023, 12:17 PM

@sergesanspaille can you provide a try run with 1-2 raptor-browsertime linux pageload tests?

This revision was not accepted when it landed; it landed in state Needs Review.Oct 16 2023, 12:24 PM
This revision was automatically updated to reflect the committed changes.

This patch breaks ./mach browsertime locally:

sparky@sparky-ThinkPad-X1-Carbon-Gen-9:~/mozilla-source/mozilla-central$ ./mach browsertime --setup --clobber
 0:00.24 [INFO] This command should be used for browsertime setup only.
If you are looking to run performance tests on your patch, use `./mach raptor --browsertime` instead.

You can get visual-metrics by using the --browsertime-video and --browsertime-visualmetrics. Here is a sample command for raptor-browsertime: 
	`./mach raptor --browsertime -t amazon --browsertime-video --browsertime-visualmetrics`

See this wiki page for more information if needed: https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Browsertime


 0:05.26 Using artifact from local cache: /home/sparky/.mozbuild/cache/browsertime/2e82796d80fdc5f6-ffmpeg-4.4.1-i686-static.tar.xz
 0:05.26 Unpacking temporary location /home/sparky/.mozbuild/cache/browsertime/2e82796d80fdc5f6-ffmpeg-4.4.1-i686-static.tar.xz
Error running mach:

    mach browsertime --setup --clobber

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file browsertime`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'Version' object has no attribute 'version'

  File "/home/sparky/mozilla-source/mozilla-central/tools/browsertime/mach_commands.py", line 670, in browsertime
    return setup_browsertime(
  File "/home/sparky/mozilla-source/mozilla-central/tools/browsertime/mach_commands.py", line 317, in setup_browsertime
    if not setup_helper.check_node_executables_valid():
  File "/home/sparky/mozilla-source/mozilla-central/tools/lint/eslint/setup_helper.py", line 405, in check_node_executables_valid
    node_path, version = find_node_executable()
  File "/home/sparky/mozilla-source/mozilla-central/python/mozbuild/mozbuild/nodeutil.py", line 90, in find_node_executable
    return nodejs_exe, version.version
python/mozbuild/mozbuild/nodeutil.py
90

Oh right, this should be fixed too.

saschanaz requested changes to this revision.EditedOct 16 2023, 1:57 PM

(for the nodeutil change)

This revision now requires changes to proceed.Oct 16 2023, 1:57 PM
sergesanspaille updated this revision to Diff 776031.

Change done. @sparky could you check it doesn't disturb your workflow?

./mach browsertime --setup --clobber

works on my local setup.

Thank you for the fix! It works locally now, and I've triggered a couple browsertime tasks in the reviewbot run (the main one is already passing): https://treeherder.mozilla.org/jobs?repo=try&revision=26ded73d75d4cebe35c9a96fcbd234aece6ac2d9&searchStr=browsertime

I've also triggered some mozperftest unit tests since they weren't triggered by the bot, but I know we use the nodeutil file there: https://treeherder.mozilla.org/jobs?repo=try&revision=26ded73d75d4cebe35c9a96fcbd234aece6ac2d9&searchStr=perftest

saschanaz added inline comments.
python/mozrelease/mozrelease/versions.py
66
This revision now requires changes to proceed.Oct 16 2023, 2:40 PM
python/mozrelease/mozrelease/versions.py
66

./mach test python/mozrelease/test/test_versions.py

passes

Any hint about how to test https://treeherder.mozilla.org/jobs?repo=try&revision=e1bc0644d5e8bc86c9e0967cdf88f6f2a8e251d0 ?

sergesanspaille added inline comments.
python/mozrelease/mozrelease/versions.py
66

oh, and my first implementation here was faulty and triggered test_versions.py which it no longer does, which is a good hint toward "this work as expected"

Okay, a couple of things:

  1. _cmp() is still used by LooseVersion and should be kept. 👍
  2. This patch still changes the behavior:
> ./mach python
> from mozrelease.versions import MozillaVersion
> MozillaVersion("3.6.3plugin1") // This gives ModernMozillaVersion on m-c but LooseVersion now. Not covered by the test
> MozillaVersion("1.0.0.1") // This gives AncientMozillaVersion on m-c but Version now, because packaging.version.Version supports the x.x.x.x format while distutils does not. Not covered by the test

I wonder we still need ModernMozillaVersion and AncientMozillaVersion at all? The only real caller, update-verify-config-creator, downloads a list and parse it through MozillaVersion, so I tried it and got a URL: https://product-details.mozilla.org/1.0/firefox.json, but it did not include any special version string like 3.6.3plugin1.

I managed to run the script with ./mach python testing/mozharness/scripts/release/update-verify-config-creator.py --product firefox --platform win32 --archive-prefix https://archive.mozilla.org/pub --to-version 59.0b5 --last-watershed 57.0b10 --include-version ".+" --branch-prefix mozilla --app-name browser and it never failed to parse the versions, so I guess it's now fine to throw the special MozillaVersions away?

But the removal can be done in a separate patch, 👍 for now.

This revision is now accepted and ready to land.Oct 17 2023, 9:34 AM

Okay, a couple of things:

  1. _cmp() is still used by LooseVersion and should be kept. 👍
  2. This patch still changes the behavior:
> ./mach python
> from mozrelease.versions import MozillaVersion
> MozillaVersion("3.6.3plugin1") // This gives ModernMozillaVersion on m-c but LooseVersion now. Not covered by the test
> MozillaVersion("1.0.0.1") // This gives AncientMozillaVersion on m-c but Version now, because packaging.version.Version supports the x.x.x.x format while distutils does not. Not covered by the test

I wonder we still need ModernMozillaVersion and AncientMozillaVersion at all? The only real caller, update-verify-config-creator, downloads a list and parse it through MozillaVersion, so I tried it and got a URL: https://product-details.mozilla.org/1.0/firefox.json, but it did not include any special version string like 3.6.3plugin1.

I managed to run the script with ./mach python testing/mozharness/scripts/release/update-verify-config-creator.py --product firefox --platform win32 --archive-prefix https://archive.mozilla.org/pub --to-version 59.0b5 --last-watershed 57.0b10 --include-version ".+" --branch-prefix mozilla --app-name browser and it never failed to parse the versions, so I guess it's now fine to throw the special MozillaVersions away?

But the removal can be done in a separate patch, 👍 for now.

Thanks for the detailed answer. I Think I understand why we have this: version is not honoring version_re... I think I can improve this, let me have a look.

sergesanspaille retitled this revision from Bug 1858065 - Use packaging's Version instead of distutils' StrictVersion r=saschanaz! to Bug 1858065 - Replace distutils' StrictVersion r=saschanaz!.
sergesanspaille edited the summary of this revision. (Show Details)

@saschanaz : I ended up providing a limited implementation of StrictVersion that matches the behavior of distutils' because I was uncomfortable with the loss of validation against regular expression. Associated try build: https://treeherder.mozilla.org/jobs?repo=try&revision=b1d015f86285cf069bbc65bd269529758ff7f41c (I've fixed the formatting issue in the above since then, don't want to restart a build just for that)

python/mozrelease/test/test_versions.py
110

But packaging.version.Version covers both now, why do we need them? Every version string being tested here can be parsed by Version.

@saschanaz : I ended up providing a limited implementation of StrictVersion that matches the behavior of distutils' because I was uncomfortable with the loss of validation against regular expression. Associated try build: https://treeherder.mozilla.org/jobs?repo=try&revision=b1d015f86285cf069bbc65bd269529758ff7f41c (I've fixed the formatting issue in the above since then, don't want to restart a build just for that)

The point of the custom version classes were to loosen the rules and cover more version strings, but those strings are now either nonexistent or covered by Version, so I don't really see the point...

Code analysis found 1 defect in diff 776575:

  • 1 defect found by py-black (Mozlint)

3 defects closed compared to the previous diff 776574.

IMPORTANT: Found 1 defect (error level) that must be fixed before landing.

You can run this analysis locally with:

  • ./mach lint --warnings --outgoing

If you see a problem in this automated review, please report it here.

You can view these defects in the Diff Detail section of Phabricator diff 776575.

python/mozrelease/test/test_versions.py
110

They can be parsed by Version *but* they don't have the same version: MozillaVersion("1.2.0.4") == MozillaVersion("1.2.4") won't be valid with Version.

I'm all in for moving to Version, but I'd rather not do that in that patch we should "just" be a compatibility patch.

saschanaz added inline comments.
python/mozrelease/test/test_versions.py
110

Fair enough, we can have a separate patch for that.

1 defect closed compared to the previous diff 776575.


If you see a problem in this automated review, please report it here.