Page MenuHomePhabricator

Bug 1605283 - Improve support for invalidation debugging and testing r=gw
ClosedPublic

Authored by Bert on Jan 15 2020, 2:20 AM.

Details

Summary

Optionally serialize N frames into a circular memory buffer, and save
them as part of wr-capture into tilecache/.

The new tile_view utility reads the folder and converts the frames to
svg for easier visualization, with a few extra features:

  • color code each primitive based on which slice it is on;
  • highlight new or moving primitives in red (brute force diff);
  • print all invalidated tiles at the top and the invalidation reason;
  • overlay the tile quadtree to visualize splitting & merging;
  • HTML and JS wrappers for animation playback, timeline scrubbing;

Positioning of the tiles on the screen is a bit broken still; especially
during scrolling and "special" pages (like about:config).

Interning info is not used yet.

Diff Detail

Repository
rMOZILLACENTRAL mozilla-central
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
phab-bot requested review of this revision.Jan 15 2020, 2:20 AM
phab-bot removed a project: Restricted Project.

Hi Glenn, this looks huge but it's mostly plumbing and not very important :^)
There is a video here of what it looks like to help navigate the code:
https://drive.google.com/open?id=15jUgHyTFUEzh-llJHaDbZeWCIAui15GE

Scrolling is still broken and still no interning info, but I'd like to submit this as version 0.1, so you can play with it as well.

Bert added inline comments.Jan 15 2020, 2:40 AM
gfx/webrender_bindings/WebRenderAPI.cpp
623–625

Clarified this a bit to make it discoverable with grep.

gfx/wr/tileview/src/main.rs
51

Python code to write out HTML and SVG, except it's rust because the input is serde.

154

After the chat in Berlin about coordinate transforms, I will fix this :^)

gfx/wr/webrender/src/lib.rs
110

This was necessary to get things compiling, but it's highly suspicious I'm the only one who needs a pub here even though there's plenty pub use crate stuff below. If there's a proper way to do this please let me know!

gfx/wr/webrender/src/picture.rs
612

I tried implementing a manual serializer method to make this nicer, but couldn't make it work (nesting maps and sequences). So using derive and a lot of skip instead.

1471

frames is always allocated, but when the user pref is false, we never write anything to it. So that's probably ok and simplest.

1543

This gets added to the prims when visualizing, and that fixes everything being bunched up at (0,0), which makes the web content overlap with the UI chrome. However it breaks down under scrolling.

This origin ultimately comes from unclipped in take_context, see below.

Code analysis found 1 defect in the diff 218090:

  • 1 defect found by clang-format

You can run this analysis locally with:

  • ./mach clang-format -s -p gfx/thebes/gfxPlatform.cpp (C/C++)

For your convenience, here is a patch that fixes all the source-test-clang-format defects (use it in your repository with hg import or git apply -p0).

If you see a problem in this automated review, please report it here.

In our previous code coverage analysis run, we found some files which had no coverage and are being modified in this patch:

Should they have tests, or are they dead code ?

  • You can file a bug blocking Bug 1415824 for untested files that should be tested.
  • You can file a bug blocking Bug 1415819 for untested files that should be removed.

If you see a problem in this automated review, please report it here.

gw requested changes to this revision.Jan 15 2020, 3:41 AM

This generally looks good!

There's a heap of nits / style comments below, but they are mostly just minor suggestions for rust-style.

The main question I have is whether it makes sense in this case to manually serialize those data types instead of using serde. I mentioned it in one of the comments, but serde can cause quite a bit of extra compile time.

Another thought - since we're already deserializing into different structure types (understandably!) another option could be to _construct_ those tiny types in the capture function, and serialize those directly. The advantage of that is we avoid all the skip and serde annotations on most of the complex types, and constructing the types that we want to deserialize in the capture function probably makes the code a bit less fragile when fields get added to the tiny types? Thoughts?

gfx/wr/tileview/src/main.rs
13

This might be simplified by the manual serialization comment I added below?

43

nit: can probably be declared as: const BLAH: &str = "blah"; I think.

61

Might be able to use something like:

let mut s = String::new();
s.append( ... );
73

Pass these as &str

76

We could instead return a struct, or multiple values in a tuple.

82

I don't think the & is needed here.

113

& shouldn't be needed here

120

& shouldn't be needed here

166

:)

170

Could use iter().find(..) or iter().position(..)

203

slices could just be a borrow: &[Slice] so it can be any container type

204

Same here with returning a struct / multiple values

209

String::new()

gfx/wr/webrender/src/lib.rs
110

Yea, it doesn't seem like it should be needed. Could you paste the compile errors you get if you remove the pub?

gfx/wr/webrender/src/picture.rs
111

nit: we can remove these commented lines.

1471

Vec is guaranteed to not allocate if nothing gets written to it, so that seems fine.

1471

Maybe make this a struct instead of a tuple?

1491

s is normally passed as &str when it's a borrow

1543

If I understand what you're saying - it seems like we could probably pre-transform the coordinates that are written out by the spatial node transform state, and then everything would be in the correct position?

3486

Is this needed if we transform the prims per above comment?

4081

nit: can use DeviceRect::zero() here.

4084

if let Some(ref tile_cache) = self.tile_cache might be tidier here

4086

nit: remove this

4092

It might be possible to annotate the fn param with allow_unused attribute or similar - I think that's available in stable rust now.

5017–5020

Looking at the number of annotations for serde (and since serde can have quite a bad effect on compile times), I wonder if it might be better to serialize the data we need manually (e.g. something like https://docs.rs/json/0.12.1/json/#serialize-with-jsonstringifyvalue).

I'm unsure if that makes any sense here, but might be worth pondering.

This revision now requires changes to proceed.Jan 15 2020, 3:41 AM
Bert marked 17 inline comments as done.Jan 15 2020, 10:16 PM

Hi Glenn, thanks for reviewing this so quickly and also for going into such great detail. I really appreciate the nitpicking, it's a great help to improve my Rust along the way :)

I like the suggestion for moving the Tiny helper structs to picture.rs. I'll see if I can do that without too much trouble with moving/cloning/Rust-shenanigans. If that works and removes most of the skip spam, then perhaps we can consider sticking with Serde, at least for v0.1?

Thanks!

gfx/wr/tileview/src/main.rs
82

You're right, even though I only add these after the compiler complains that I need one. I should probably re-read the chapter on ownership :)

120

This one complains about "cannot move out of invalidation reason which is behind a shared reference".

gfx/wr/webrender/src/lib.rs
110

Sure, the problem is that tileview can't see the picture imports:

@DESKTOP-02TEENE /c/code/1605283_tilecache/gfx/wr/tileview
$ cargo build
   Compiling webrender v0.60.0 (C:\code\1605283_tilecache\gfx\wr\webrender)
   Compiling tileview v0.1.0 (C:\code\1605283_tilecache\gfx\wr\tileview)
error[E0603]: module `picture` is private
 --> tileview\src\main.rs:4:16
  |
4 | use webrender::picture::{TileDescriptor, TileId, TileNode, TileNodeKind, InvalidationReason};
  |                ^^^^^^^

But I imagine that wrench needs them too, so I'm wondering how wrench got access to those.

gfx/wr/webrender/src/picture.rs
1471

Good idea; I didn't do it at first because Rust doesn't have inner structures, but, it's not the end of the world to have another module-global type.

1543

Well, I'm trying to keep the prims in local space, and apply their local-to-world transform in tileview, just before writing out the SVG coordinates. That way, when scrolling, their local space coordinates don't change, only the transform does (shifting the parent picture around on the viewport). Thus diffing works as intended -- the primitives aren't moving inside the tile, the tile is not being invalidated, and so we don't want the primitives to flash bold red in the viewer.

That's the idea, the only flaw is that the local to world transform is not accurate enough, I'm hacking it with a 2D translation by unclipped.origin.

3486

Yes, see earlier note, ideally we keep the prims as is and export their mapping to viewspace along with them. That transform is what unclipped.origin is meant to represent.

4092

Sure, but it's a 700+ line function and most of it would be a bug if a var goes unused, so I think the extra cfg not is safer?

5017–5020

I actually started out doing a manual implementation, until I realized how much work that would be (plus some problems combining SerializeMap and SerializeStruct), and how awesome it is that Rust has Serde&Derive... so it's a bit sad if we can't use it :(

Let me move the ~Tiny serializer structures into picture.rs, and use those, and see what it looks like after that in terms of skip spam and compile slow downs?

Bert updated this revision to Diff 218516.Jan 15 2020, 11:49 PM
Bert marked 6 inline comments as done.

In our previous code coverage analysis run, we found some files which had no coverage and are being modified in this patch:

Should they have tests, or are they dead code ?

  • You can file a bug blocking Bug 1415824 for untested files that should be tested.
  • You can file a bug blocking Bug 1415819 for untested files that should be removed.

If you see a problem in this automated review, please report it here.

gw added inline comments.Jan 16 2020, 1:16 AM
gfx/wr/tileview/src/main.rs
120

Let's add #[derive(Copy, Clone] to the InvalidationReason enum. That means that type is treated as a value type that is copied, rather than moved. This often simplifies code and I usually mark any time < 16 bytes or so as Copy, not sure why I didn't here!

gfx/wr/webrender/src/lib.rs
110

The use line in tileview can be:

use webrender::{TileDescriptor, TileId, TileNode};

This means that it uses the types you publicly export in this lib.rs, whereas you're directly referencing a type inside the private picture module in that compile error.

gfx/wr/webrender/src/picture.rs
1543

Not quite sure I follow this bit - might be easier to discuss this bit on slack / matrix.

4092

Yup, sounds fair enough!

5017–5020

Yep, we should definitely keep serde if we can use those Tiny structs!

gw added a comment.Jan 16 2020, 1:17 AM

Hi Glenn, thanks for reviewing this so quickly and also for going into such great detail. I really appreciate the nitpicking, it's a great help to improve my Rust along the way :)

I like the suggestion for moving the Tiny helper structs to picture.rs. I'll see if I can do that without too much trouble with moving/cloning/Rust-shenanigans. If that works and removes most of the skip spam, then perhaps we can consider sticking with Serde, at least for v0.1?

Thanks!

Yup, if we can serialize the Tiny structs, we should definitely keep serde! I think that would give us the best of both worlds, if it works out :)

gw added a comment.Jan 16 2020, 3:33 AM

Yup, the change to use TileSerializer looks great!

Bert marked 13 inline comments as done.Jan 16 2020, 7:31 PM
Bert added inline comments.
gfx/wr/tileview/src/main.rs
76

I've tried and it felt like things got less readable. Not sure, maybe I was doing it wrong.

120

Ok, done!

204

Meh :)

gfx/wr/webrender/src/lib.rs
110

Ah, I see -- the pub use create::picture::Foo part in lib.rs acts as an export of some sorts. Thanks!

gfx/wr/webrender/src/picture.rs
1543

Yeah I might not be making a lot of sense here. Maybe we can set up a Zoom if you have 30 mins or so to cover what info you'd like to get out of the interning tables, and this coordinate transform business?
Code review wise I'll land it as-is for now, ie. "slightly buggy but ballpark".

Bert updated this revision to Diff 218875.Jan 16 2020, 7:36 PM
Bert marked 5 inline comments as done.

In our previous code coverage analysis run, we found some files which had no coverage and are being modified in this patch:

Should they have tests, or are they dead code ?

  • You can file a bug blocking Bug 1415824 for untested files that should be tested.
  • You can file a bug blocking Bug 1415819 for untested files that should be removed.

If you see a problem in this automated review, please report it here.

gw accepted this revision.Jan 16 2020, 8:15 PM

Sounds good!

This revision is now accepted and ready to land.Jan 16 2020, 8:15 PM
Bert updated this revision to Diff 218997.Jan 17 2020, 12:21 AM
nataliaCs reopened this revision.Jan 17 2020, 12:45 AM
This revision is now accepted and ready to land.Jan 17 2020, 12:45 AM
Bert updated this revision to Diff 219306.Jan 17 2020, 6:41 PM
aciure reopened this revision.Jan 17 2020, 7:00 PM
This revision is now accepted and ready to land.Jan 17 2020, 7:00 PM
Bert updated this revision to Diff 219509.Jan 18 2020, 8:22 AM
erahm retitled this revision from Bug 1605283 - Improve support for invalidation debugging and testing to Bug 1605283 - Improve support for invalidation debugging and testing r=gw.