This is a transcript of What's Up With That Episode 4, a 2022 video discussion between Sharon ([email protected]) and Stephen ([email protected]).
The transcript was automatically generated by speech-to-text software. It may contain minor errors.
Testing is important! What kinds of tests do we have in Chromium? What are they all about? Join in as Stephen, who led Chrome's involvement in web platform tests, tells us all about them.
Notes:
00:00 SHARON: Hello, everyone, and welcome to "What's Up With That," the series that demystifies all things Chrome. I'm your host, Sharon. And today we're talking testing. Within Chrome, there are so many types of tests. What are they all? What's the difference? What are the Chromium-specific quirks? Today's guest is Stephen. He previously led Chrome's involvement in web platform tests. Since then, he's worked on rendering, payments, and interoperability. As a fun aside, he's one of the first people I met who worked on Chrome and is maybe part of why I'm here today. So welcome, Stephen.
00:33 STEPHEN: Well, thank you very much for having me, Sharon, I'm excited to be here.
00:33 SHARON: Yeah, I'm excited to have you here. So today, we're in for maybe a longer episode. Testing is a huge topic, especially for something like Chrome. So grab a snack, grab a drink, and let's start. We'll start with what are all of the things that we have testing for in Chrome. What's the purpose of all these tests we have?
00:51 STEPHEN: Yeah. It's a great question. It's also an interesting one because I wanted to put one caveat on this whole episode, which is that there is no right answer in testing. Testing, even in the literature, never mind in Chromium itself, is not a solved problem. And so you'll hear a lot of different opinions. People will have different thoughts. And I'm sure that no matter how hard we try, by the end of this episode, our inbox will be filled with angry emails from people being like, no, you are wrong. So all of the stuff we're saying here today is my opinion, albeit I'll try and be as useful as possible. But yeah, so why do we test was the question, right? So there's a lot of different reasons that we write tests. Obviously, correctness is the big one. You're writing some code, you're creating a feature, you want it to be correct. Other reasons we write them, I mean, tests can be useful as a form of documentation in itself. If you're ever looking at a class and you're like, what does - why is this doing this, why is the code doing this, the test can help inform that. They're also useful - I think a topic of this podcast is sort of security. Tests can be very useful for security. Often when we have a security bug, we go back and we write what are called regression tests, so at least we try and never do that security failure again. And then there are other reasons. We have tests for performance. We have tests for - our launch process uses tests. There's lots and lots of reasons we have tests.
02:15 SHARON: Now that you've covered all of the different reasons why we test, how do we do each of these types of tests in Chromium? What are the test types we have?
02:27 STEPHEN: Yeah. So main test types we have in Chromium, unit tests, browser tests, what we call web tests, and then there's a bunch of more specialized ones, performance tests, testing on Android, and of course manual testing.
02:43 SHARON: We will get into each of these types now, I guess. The first type of test you mentioned is unit tests. Why don't you tell us a quick rundown of what unit tests are. I'm sure most people have encountered them or heard of them before. But just a quick refresher for those who might not.
02:55 STEPHEN: Yeah, absolutely. So as the name implies, a unit test is all about testing a unit of code. And what that is not very well defined. But you can usually think of it as just a class, a file, a small isolated component that doesn't have to talk to all the other bits of the code to work. Really, the goal is on writing something that's testing just the code under test - so that new method you've added or whatever. And it should be quick and easy to run.
03:22 SHARON: So on the screen now we have an example of a pretty typical unit
test we see in Chrome. So there's three parts here. Let's go through each of
them. So the first type - the first part of this is TEST_P. What is that
telling us?
03:38 STEPHEN: Yeah. So that is - in Chromium we use a unit testing framework
called Google test. It's very commonly used for C++. You'll see it all over the
place. You can go look up documentation. The test macros, that's what this is,
are essentially the hook into Google test to say, hey, the thing that's coming
here is a test. There's three types. There is just test, which it just says
here is a function. It is a test function. TEST_F says that you basically
have a wrapper class. It's often called a test fixture, which can do some
common setup across multiple different tests, common teardown, and that sort of
thing. And finally, TEST_P is what we call a parameterized test. And what
this means is that the test can take some input parameters, and it will run the
same test with each of those values. Very useful for things like when you want
to test a new flag. What happens if the flag is on or off?
04:34 SHARON: That's cool. And a lot of the things we're mentioning for unit
test also apply to browser test, which we'll cover next. But the
parameterization is an example of something that carries over to both. So
that's the first part. That's the TEST_P, the macro. What's the second part,
PendingBeaconHostTest? What is that?
04:54 STEPHEN: Yeah. So that is the fixture class, the test container class I was talking about. So in this case, we're assuming that in order to write a beacon test, whatever that is, they have some set up, some teardown they need to do. They might want to encapsulate some common functionality. So all you have to do to write one of these classes is, you declare a C++ class and you subclass from the Google test class name.
05:23 SHARON: So this is a TEST_P, but you mentioned that this is a fixture.
So are fixture tests a subset of parameterized tests?
05:35 STEPHEN: Parameterized tests are a subset of fixture test, is that the right way around to put it? All parameterized tests are fixtures tests. Yes.
05:41 SHARON: OK.
05:41 STEPHEN: You cannot have a parameterized test that does not have a
fixture class. And the reason for that is how Google test actually works under
the covers is it passes those parameters to your test class. You will have to
additionally extend from the testing::WithParamInterface. And that says, hey,
I'm going to take parameters.
06:04 SHARON: OK. But not all fixture tests are parameterized tests.
06:04 STEPHEN: Correct.
06:04 SHARON: OK. And the third part of this, SendOneOfBeacons. What is that?
06:10 STEPHEN: That is your test name. Whatever you want to call your test, whatever you're testing, put it here. Again, naming tests is as hard as naming anything. A lot of yak shaving, finding out what exactly you should call the test. I particularly enjoy when you see test names that themselves have underscores in them. It's great.
06:30 SHARON: Uh-huh. What do you mean by yak shaving?
06:35 STEPHEN: Oh, also known as painting a bike shed? Bike shed, is that the right word? Anyway, generally speaking -
06:40 SHARON: Yeah, I've heard -
06:40 STEPHEN: arguing about pointless things because at the end of the day, most of the time it doesn't matter what you call it.
06:46 SHARON: OK, yeah. So I've written this test. I've decided it's going to be parameterized. I've come up with a test fixture for it. I have finally named my test. How do I run my tests now?
06:57 STEPHEN: Yeah. So all of the tests in Chromium are built into different
test binaries. And these are usually named after the top level directory that
they're under. So we have components_unittests, content_unittests. I think
the Chrome one is just called unit_tests because it's special. We should
really rename that. But I'm going to assume a bunch of legacy things depend on
it. Once you have built whichever the appropriate binary is, you can just run
that from your out directory, so out/release/components_unittests, for
example. And then that, if you don't pass any flags, will run every single
components unit test. You probably don't want to do that. They're not that
slow, but they're not that fast. So there is a flag --gtest_filter, which
allows you to filter. And then it takes a test name after that. The format of
test names is always test class dot test name. So for example, here
PendingBeaconHostTest dot SendOneOfBeacons.
08:04 SHARON: Mm-hmm. And just a fun aside for that one, if you do have parameterized tests, it'll have an extra slash and a number at the end. So normally, whenever I use it, I just put a star before and after. And that generally does - covers the cases.
08:17 STEPHEN: Yeah, absolutely.
08:23 SHARON: Cool. So with the actual test names, you will often see them
prefixed with either MAYBE_ or DISABLED_, or before the test, there will be
an ifdef with usually a platform and then depending on the cases, it'll prefix
the test name with something. So I think it's pretty clear what these are
doing. Maybe is a bit less clear. Disabled pretty clear what that is. But can
you tell us a bit about these prefixes?
08:51 STEPHEN: Yeah, absolutely. So this is our way of trying to deal with that
dreaded thing in testing, flake. So when a test is flaky, when it doesn't
produce a consistent result, sometimes it fails. We have in Chromium a whole
continuous integration waterfall. That is a bunch of bots on different
platforms that are constantly building and running Chrome tests to make sure
that nothing breaks, that bad changes don't come in. And flaky tests make that
very hard. When something fails, was that a real failure? And so when a test is
particularly flaky and is causing sheriffs, the build sheriffs trouble, they
will come in and they will disable that test. Basically say, hey, sorry, but
this test is causing too much pain. Now, as you said, the DISABLED_ prefix,
that's pretty obvious. If you put that in front of a test, Google test knows
about it and it says, nope, will not run this test. It will be compiled, but it
will not be run. MAYBE_ doesn't actually mean anything. It has no meaning to
Google test. But that's where you'll see, as you said, you see these ifdefs.
And that's so that we can disable it on just one platform. So maybe your test
is flaky only on Mac OS, and you'll see basically, oh, if Mac OS, change the
name from maybe to disabled. Otherwise, define maybe as the normal test name.
10:14 SHARON: Makes sense. We'll cover flakiness a bit later. But yeah, that's a huge problem. And we'll talk about that for sure. So these prefixes, the parameterization and stuff, this applies to both unit and browser tests.
10:27 STEPHEN: Yeah.
10:27 SHARON: Right? OK. So what are browser tests? Chrome's a browser. Browser test, seems like there's a relation.
10:34 STEPHEN: Yeah. They test the browser. Isn't it obvious? Yeah. Browser tests are our version - our sort of version of an integration or a functional test depending on how you look at things. What that really means is they're testing larger chunks of the browser at once. They are integrating multiple components. And this is somewhere that I think Chrome's a bit weird because in many large projects, you can have an integration test that doesn't bring your entire product up and in order to run. Unfortunately, or fortunately, I guess it depends on your viewpoint, Chrome is so interconnected, it's so interdependent, that more or less we have to bring up a huge chunk of the browser in order to connect any components together. And so that's what browser tests are. When you run one of these, there's a massive amount of machinery in the background that goes ahead, and basically brings up the browser, and actually runs it for some definition of what a browser is. And then you can write a test that pokes at things within that running browser.
11:42 SHARON: Yeah. I think I've heard before multiple times is that browser tests launch the whole browser. And that's -
11:47 STEPHEN: More or less true. It's - yeah.
11:47 SHARON: Yes. OK. Does that also mean that because you're running all this stuff that all browser tests have fixtures? Is that the case?
11:59 STEPHEN: Yes, that is the case. Absolutely. So there is only - I think
it's - oh my goodness, probably on the screen here somewhere. But it's
IN_PROC_BROWSER_TEST_F and IN_PROC_BROWSER_TEST_P. There is no version that
doesn't have a fixture.
12:15 SHARON: And what does the in proc part of that macro mean?
12:15 STEPHEN: So that's, as far as I know - and I might get corrected on this. I'll be interested to learn. But it refers to the fact that we've run these in the same process. Normally, the whole Chromium is a multi-process architecture. For the case of testing, we put that aside and just run everything in the same process so that it doesn't leak, basically.
12:38 SHARON: Yeah. There's flags when you run them, like --single-process.
And then there's --single-process-test. And they do slightly different
things. But if you do run into that, probably you will be working with people
who can answer and explain the differences between those more. So something
that I've seen quite a bit in browser and unit tests, and only in these, are
run loops. Can you just briefly touch on what those are and what we use them
for in tests?
13:05 STEPHEN: Oh, yeah. That's a fun one. I think actually previous on an episode of this very program, you and Dana talked a little bit around the fact that Chrome is not a completely synchronous program, that we do we do task splitting. We have a task scheduler. And so run loops are part of that, basically. They're part of our stack for handling asynchronous tasks. And so this comes up in testing because sometimes you might be testing something that's not synchronous. It takes a callback, for example, rather than returning a value. And so if you just wrote your test as normal, you call the function, and you don't - you pass a callback, but then your test function ends. Your test function ends before that callback ever runs. Run loop gives you the ability to say, hey, put this callback into some controlled run loop. And then after that, you can basically say, hey, wait on this run loop. I think it's often called quit when idle, which basically says keep running until you have no more tasks to run, including our callback, and then finish. They're powerful. They're very useful, obviously, with asynchronous code. They're also a source of a lot of flake and pain. So handle with care.
14:24 SHARON: Yeah. Something a tip is maybe using the --gtest_repeat flag.
So that one lets you run your test however number of times you've had to do it.
14:30 STEPHEN: Yeah.
14:36 SHARON: And that can help with testing for flakiness or if you're trying
to debug something flaky. In tests, we have a variety of macros that we use. In
the unit test and the browser tests, you see a lot of macros, like EXPECT_EQ,
EXPECT_GT. These seem like they're part of maybe Google test. Is that true?
14:54 STEPHEN: Yeah. They come from Google test itself. So they're not
technically Chromium-specific. But they basically come in two flavors. There's
the EXPECT_SOMETHING macros. And there's the ASSERT_SOMETHING macros. And
the biggest thing to know about them is that expect doesn't actually cause - it
causes a test to fail, but it doesn't stop the test from executing. The test
will continue to execute the rest of the code. Assert actually throws an
exception and stops the test right there. And so this can be useful, for
example, if you want to line up a bunch of expects. And your code still makes
sense. You're like, OK, I expect to return object, and it's got these fields.
And I'm just going to expect each one of the fields. That's probably fine to
do. And it may be nice to have output that's like, no, actually, both of these
fields are wrong. Assert is used when you're like, OK, if this fails, the rest
of the test makes no sense. Very common thing you'll see. Call an API, get back
some sort of pointer, hopefully a smart pointer, hey. And you're going to be
like, assert that this pointer is non-null because if this pointer is null,
everything else is just going to be useless.
15:57 SHARON: I think we see a lot more expects than asserts in general anecdotally from looking at the test. Do you think, in your opinion, that people should be using asserts more generously rather than expects, or do we maybe want to see what happens - what does go wrong if things continue beyond a certain point?
16:15 STEPHEN: Yeah. I mean, general guidance would be just keep using expect. That's fine. It's also not a big deal if your test actually just crashes. It's a test. It can crash. It's OK. So use expects. Use an assert if, like I said, that the test doesn't make any sense. So most often if you're like, hey, is this pointer null or not and I'm going to go do something with this pointer, assert it there. That's probably the main time you'd use it.
16:45 SHARON: A lot of the browser test classes, like the fixture classes themselves, are subclass from other base classes.
16:53 STEPHEN: Mm-hmm.
16:53 SHARON: Can you tell us about that?
16:53 STEPHEN: Yeah. So basically, we have one base class for browser tests. I
think its BrowserTestBase, I think it's literally called, which sits at the
bottom and does a lot of the very low level setup of bringing up a browser. But
as folks know, there's more than one browser in the Chromium project. There is
Chrome, the Chrome browser that is the more full-fledged version. But there's
also content shell, which people might have seen. It's built out of content.
It's very simple browser. And then there are other things. We have a headless
mode. There is a headless Chrome you can build which doesn't show any UI. You
can run it entirely from the command line.
17:32 SHARON: What's the difference between headless and content shell?
17:39 STEPHEN: So content shell does have a UI. If you run content shell, you will actually see a little UI pop up. What content shell doesn't have is all of those features from Chrome that make Chrome Chrome, if you will. So I mean, everything from bookmarks, to integration with having an account profile, that sort of stuff is not there. I don't think content shell even supports tabs. I think it's just one page you get. It's almost entirely used for testing. But then, headless, sorry, as I was saying, it's just literally there is no UI rendered. It's just headless.
18:13 SHARON: That sounds like it would make -
18:13 STEPHEN: And so, yeah. And so - sorry.
18:13 SHARON: testing faster and easier. Go on.
18:18 STEPHEN: Yeah. That's a large part of the point, as well as when you want
to deploy a browser in an environment where you don't see the UI. So for
example, if you're running on a server or something like that. But yeah. So for
each of these, we then subclass that BrowserTestBase in order to provide
specific types. So there's content browser test. There's headless browser test.
And then of course, Chrome has to be special, and they called their version in
process browser test because it wasn't confusing enough. But again, it's sort
of straightforward. If you're in Chrome, /chrome, use
in_process_browser_test. If you're in /content, use content_browsertest.
It's pretty straightforward most of the time.
18:58 SHARON: That makes sense. Common functions you see overridden from those base classes are these set up functions. So they're set, set up on main thread, there seems to be a lot of different set up options. Is there anything we should know about any of those?
19:13 STEPHEN: I don't think that - I mean, most of it's fairly straightforward. I believe you should mostly be using setup on main thread. I can't say that for sure. But generally speaking, setup on main thread, teardown on main thread - or is it shutdown main thread? I can't remember - whichever the one is for afterwards, are what you should be usually using in a browser thread. You can also usually do most of your work in a constructor. That's something that people often don't know about testing. I think it's something that's changed over time. Even with unit tests, people use the setup function a lot. You can just do it in the constructor a lot of the time. Most of background initialization has already happened.
19:45 SHARON: I've definitely wondered that, especially when you have things in the constructor as well as in a setup method. It's one of those things where you just kind of think, I'm not going to touch this because eh, but -
19:57 STEPHEN: Yeah. There are some rough edges, I believe. Set up on main thread, some things have been initialized that aren't around when your class is being constructed. So it is fair. I'm not sure I have any great advice unless - other than you may need to dig in if it happens.
20:19 SHARON: One last thing there. Which one gets run first, the setup functions or the constructor?
20:19 STEPHEN: The constructor always happens first. You have to construct the object before you can use it.
20:25 SHARON: Makes sense. This doesn't specifically relate to a browser test or unit test, but it does seem like it's worth mentioning, which is the content public test API. So if you want to learn more about content and content public, check out episode three with John. But today we're talking about testing. So we're talking about content public test. What is in that directory? And how does that - how can people use what's in there?
20:48 STEPHEN: Yeah. It's basically just a bunch of useful helper functions and classes for when you are doing mostly browser tests. So for example, there are methods in there that will automatically handle navigating the browser to a URL and actually waiting till it's finished loading. There are other methods for essentially accessing the tab strip of a browser. So if you have multiple tabs and you're testing some cross tab thing, methods in there to do that. I think that's probably where the content browser test - like base class lives there as well. So take a look at it. If you're doing something that you're like, someone should write - it's the basic - it's the equivalent of base in many ways for testing. It's like, if you're like, someone should have written a library function for this, possibly someone has already. And you should take a look. And if they haven't, you should write one.
21:43 SHARON: Yeah. I've definitely heard people, code reviewers, say when you want to add something that seems a bit test only to content public, put that in content public test because that doesn't get compiled into the actual release binaries. So if things are a bit less than ideal there, it's a bit more forgiving for a place for that.
22:02 STEPHEN: Yeah, absolutely. I mean, one of the big things about all of our test code is that you can actually make it so that it's in many cases not compiled into the binary. And that is both useful for binary size as well as you said in case it's concerning. One thing you can do actually in test, by the way, for code that you cannot avoid putting into the binary - so let's say you've got a class, and for the reasons of testing it because you've not written your class properly to do a dependency injection, you need to access a member. You need to set a member. But you only want that to happen from test code. No real code should ever do this. You can actually name methods blah, blah, blah for test or for testing. And this doesn't have any - there's no code impact to this. But we have pre-submits that actually go ahead and check, hey, are you calling this from code that's not marked as test code? And it will then refuse to - it will fail to pre-submit upload if that happens. So it could be useful.
23:03 SHARON: And another thing that relates to that would be the friend test or friend something macro that you see in classes. Is that a gtest thing also?
23:15 STEPHEN: It's not a gtest thing. It's just a C++ thing. So C++ has the concept of friending another class. It's very cute. It basically just says, this other class and I, we can access each other's internal states. Don't worry, we're friends. Generally speaking, that's a bad idea. We write classes for a reason to have encapsulation. The entire goal of a class is to encapsulate behavior and to hide the implementation details that you don't want to be exposed. But obviously, again, when you're writing tests, sometimes it is the correct thing to do to poke a hole in the test and get at something. Very much in the schools of thought here, some people would be like, you should be doing dependency injection. Some people are like, no, just friend your class. It's OK. If folks want to look up more, go look up the difference between open box and closed box testing.
24:00 SHARON: For those of you who are like, oh, this sounds really cool, I will learn more.
24:00 STEPHEN: Yeah, for my test nerds out there.
24:06 SHARON: [LAUGHS] Yeah, Stephen's got a club. Feel free to join.
24:06 STEPHEN: Yeah. [LAUGHTER]
24:11 SHARON: You get a card. Moving on to our next type of test, which is your wheelhouse, which is web tests. This is something I don't know much about. So tell us all about it.
24:22 STEPHEN: [LAUGHS] Yeah. This is my - this is where hopefully I'll shine. It's the area I should know most about. But web tests are - they're an interesting one. So I would describe them is our version of an end-to-end test in that a web test really is just an HTML file, a JavaScript file that is when you run it, you literally bring up - you'll remember I said that browser tests are most of a whole browser. Web tests bring up a whole browser. It's just the same browser as content shell or Chrome. And it runs that whole browser. And the test does something, either in HTML or JavaScript, that then is asserted and checked. And the reason I say that I would call them this, I have heard people argue that they're technically unit tests, where the unit is the JavaScript file and the entire browser is just, like, an abstraction that you don't care about. I guess it's how you view them really. I view the browser as something that is big and flaky, and therefore these are end-to-end tests. Some people disagree.
25:22 SHARON: In our last episode, John touched on these tests and how that they're - the scope and that each test covers is very small. But how you run them is not. And I guess you can pick a side that you feel that you like more and go with that. So what are examples of things we test with these kind of tests?
25:49 STEPHEN: Yeah. So the two big categories of things that we test with web tests are basically web APIs, so JavaScript APIs, provided by the browser to do something. There are so many of those, everything from the fetch API for fetching stuff to the web serial API for talking to devices over serial ports. The web is huge. But anything you can talk to via JavaScript API, we call those JavaScript tests. It's nice and straightforward. The other thing that web tests usually encompass are what are called rendering tests or sometimes referred to as ref tests for reference tests. And these are checking the actual, as the first name implies, the rendering of some HTML, some CSS by the browser. The reason they're called reference tests is that usually the way you do this to check whether a rendering is correct is you set up your test, and then you compare it to some image or some other reference rendering that you're like, OK, this should look like that. If it does look like that, great. If it doesn't, I failed.
26:54 SHARON: Ah-ha. And are these the same as - so there's a few other test names that are all kind of similar. And as someone who doesn't work in them, they all kind of blur together. So I've also heard web platform tests. I've heard layout tests. I've heard Blink tests, all of which do - all of which are JavaScript HTML-like and have some level of images in them. So are these all the same thing? And if not, what's different?
27:19 STEPHEN: Yeah. So yes and no, I guess, is my answer. So a long time ago, there were layout tests basically. And that was something we inherited from the WebKit project when we forked there, when we forked Chromium from WebKit all those years ago. And they're exactly what I've described. They were both JavaScript-based tests and they were also HTML-based tests for just doing reference renderings. However, web platform test came up as an external project actually. Web platform test is not a Chromium project. It is external upstream. You can find them on GitHub. And their goal was to create a set of - a test suite shared between all browsers so that all browsers could test - run the same tests and we could actually tell, hey, is the web interoperable? Does it work the same way no matter what browser you're on? The answer is, no. But we're trying. And so inside of Chromium we said, that's great. We love this idea. And so what we did was we actually import web platform test into our layout tests. So web platform test now becomes a subdirectory of layout tests. OK?
28:30 SHARON: OK. [LAUGHS]
28:30 STEPHEN: To make things more confusing, we don't just import them, but we also export them. We run a continuous two-way sync. And this means that Chromium developers don't have to worry about that upstream web platform test project most of the time. They just land their code in Chromium, and a magic process happens, and it goes up into the GitHub project. So that's where we were for many years - layout tests, which are a whole bunch of legacy tests, and then also web platform tests. But fairly recently - and I say that knowing that COVID means that might be anything within the last three years because who knows where time went - we decided to rename layout test. And partly, the name we chose was web tests. So now you have web tests, of which web platform tests are a subset, or a - yeah, subset of web test. Easy.
29:20 SHARON: Cool.
29:20 STEPHEN: [LAUGHS]
29:20 SHARON: Cool. And what about Blink tests? Are those separate, or are those these altogether?
29:27 STEPHEN: I mean, if they're talking about the JavaScript and HTML, that's
going to just be another name for the web tests. I find that term confusing
because there is also the Blink tests target, which builds the infrastructure
that is used to run web tests. So that's probably what you're referring, like
blink_test. It is the target that you build to run these tests.
29:50 SHARON: I see. So blink_test is a target. These other ones, web test
and web platform tests, are actual test suites.
29:57 STEPHEN: Correct. Yes. That's exactly right.
30:02 SHARON: OK. All right.
30:02 STEPHEN: Simple.
30:02 SHARON: Yeah. So easy. So you mentioned that the web platform tests are cross-browser. But a lot of browsers are based on Chromium. Is it one of the things where it's open source and stuff but majority of people contributing to these and maintaining it are Chrome engineers?
30:23 STEPHEN: I must admit, I don't know what that stat is nowadays. Back when I was working on interoperability, we did measure this. And it was certainly the case that Chromium is a large project. There were a lot of tests being contributed by Chromium developers. But we also saw historically - I would like to recognize Mozilla, most of all, who were a huge contributor to the web platform test project over the years and are probably the reason that it succeeded. And we also - web platform test also has a fairly healthy community of completely outside developers. So people that just want to come along. And maybe they're not able to or willing to go into a browser, and actually build a browser, and muck with code. But they could write a test for something. They can find a broken behavior and be like, hey, there's a test here, Chrome and Firefox do different things.
31:08 SHARON: What are examples of the interoperability things that you're testing for in these cross-browser tests?
31:17 STEPHEN: Oh, wow, that's a big question. I mean, really everything and anything. So on the ref test side, the rendering test, it actually does matter that a web page renders the same in different browsers. And that is very hard to achieve. It's hard to make two completely different engines render some HTML and CSS exactly the same way. But it also matters. We often see bugs where you have a lovely - you've got a lovely website. It's got this beautiful header at the top and some content. And then on one browser, there's a two-pixel gap here, and you can see the background, and it's not a great experience for your users. So ref tests, for example, are used to try and track those down. And then, on the JavaScript side, I mean really, web platform APIs are complicated. They're very powerful. There's a reason they are in the browser and you cannot do them in JavaScript. And that is because they are so powerful. So for example, web USB to talk to USB devices, you can't just do that from JavaScript. But because they're so powerful, because they're so complicated, it's also fairly easy for two browsers to have slightly different behavior. And again, it comes down to what is the web developer's experience. When I try and use the web USB API, for example, am I going to have to write code that's like, if Chrome, call it this way, if Fire - we don't want that. That is what we do not want for the web. And so that's the goal.
32:46 SHARON: Yeah. What a team effort, making the whole web work is. All right. That's cool. So in your time working on these web platform tests, do you have any fun stories you'd like to share or any fun things that might be interesting to know?
33:02 STEPHEN: Oh, wow. [LAUGHS] One thing I like to bring up - I'm afraid it's not that fun, but I like to repeat it a lot of times because it's weird and people get tripped up by it - is that inside of Chromium, we don't run web platform tests using the Chrome browser. We run them using content shell. And this is partially historical. That's how layout tests run. We always ran them under content shell. And it's partially for I guess what I will call feasibility. As I talked about earlier, content shell is much simpler than Chrome. And that means that if you want to just run one test, it is faster, it is more stable, it is more reliable I guess I would say, than trying to bring up the behemoth that is Chrome and making sure everything goes correctly. And this often trips people up because in the upstream world of this web platform test project, they run the test using the proper Chrome binary. And so they're different. And different things do happen. Sometimes it's rendering differences. Sometimes it's because web APIs are not always implemented in both Chrome and content shell. So yeah, fun fact.
34:19 SHARON: Oh, boy. [LAUGHTER]
34:19 STEPHEN: Oh, yeah.
34:19 SHARON: And we wonder why flakiness is a problem. Ah. [LAUGHS]
34:19 STEPHEN: Yeah. It's a really sort of fun but also scary fact that even if we put aside web platform test and we just look at layout test, we don't test what we ship. Layout test running content shell, and then we turn around and we're like, here's a Chrome binary. Like uh, those are different. But, hey, we do the best we can.
34:43 SHARON: Yeah. We're out here trying our best. So that all sounds very cool. Let's move on to our next type of test, which is performance. You might have heard the term telemetry thrown around. Can you tell us what telemetry is and what these performance tests are?
34:54 STEPHEN: I mean, I can try. We've certainly gone straight from the thing I know a lot about into the thing I know very little about. But -
35:05 SHARON: I mean, to Stephen's credit, this is a very hard episode to find one single guest for. People who are working extensively usually in content aren't working a ton in performance or web platform stuff. And there's no one who is - just does testing and does every kind of testing. So we're trying our best. [INAUDIBLE]
35:24 STEPHEN: Yeah, absolutely. You just need to find someone arrogant enough that he's like, yeah, I'll talk about all of those. I don't need to know the details. It's fine. But yeah, performance test, I mean, the name is self explanatory. These are tests that are trying to ensure the performance of Chromium. And this goes back to the four S's when we first started Chrome as a project - speed, simplicity, security, and I've forgotten the fourth S now. Speed, simplicity, security - OK, let's not reference the four S's then. [LAUGHTER] You have the Comet. You tell me.
36:01 SHARON: Ah. Oh, I mean, I don't read it every day. Stability. Stability.
36:08 STEPHEN: Stability. God damn it. Let's literally what the rest of this is about. OK, where were we?
36:13 SHARON: We're leaving this in, don't worry. [LAUGHTER]
36:19 STEPHEN: Yeah. So the basic idea of performance test is to test performance because as much as you can view behavior as a correctness thing, in Chromium we also consider performance a correctness thing. It is not a good thing if a change lands and performance regresses. So obviously, testing performance is also hard to do absolutely. There's a lot of noise in any sort of performance testing. An so, we do it essentially heuristically, probabilistically. We run whatever the tests are, which I'll talk about in a second. And then we look at the results and we try and say, hey, OK, is there a statistically significant difference here? And there's actually a whole performance sheriffing rotation to try and track these down. But in terms of, yeah, you mentioned telemetry. That weird word. You're like, what is a telemetry test? Well, telemetry is the name of the framework that Chromium uses. It's part of the wider catapult project, which is all about different performance tools. And none of the names, as far as I know, mean anything. They're just like, hey, catapult, that's a cool name. I'm sure someone will explain to me now the entire history behind the name catapult and why it's absolutely vital. But anyway, so telemetry basically is a framework that when you give it some input, which I'll talk about in a second, it launches a browser, performs some actions on a web page, and records metrics about those actions. So the input, the test essentially, is basically a collection of go to this web page, do these actions, record these metrics. And I believe in telemetry that's called a story, the story of someone visiting a page, I guess, is the idea. One important thing to know is that because it's sort of insane to actually visit real websites, they keep doing things like changing - strange. We actually cache the websites. We download a version of the websites once and actually check that in. And when you go run a telemetry test, it's not running against literally the real Reddit.com or something. It's running against a version we saved at some point.
38:31 SHARON: And how often - so I haven't really heard of anyone who actually works on this and that we can't - you don't interact with everyone. But how - as new web features get added and things in the browser change, how often are these tests specifically getting updated to reflect that?
38:44 STEPHEN: I would have to plead some ignorance there. It's certainly also been my experience as a browser engineer who has worked on many web APIs that I've never written a telemetry test myself. I've never seen one added. My understanding is that they are - a lot of the use cases are fairly general with the hope that if you land some performance problematic feature, it will regress on some general test. And then we can be like, oh, you've regressed. Let's figure out why. Let's dig in and debug. But it certainly might be the case if you are working on some feature and you think that it might have performance implications that aren't captured by those tests, there is an entire team that works on the speed of Chromium. I cannot remember their email address right now. But hopefully we will get that and put that somewhere below. But you can certainly reach out to them and be like, hey, I think we should test the performance of this. How do I go about and do that?
39:41 SHARON: Yeah. That sounds useful. I've definitely gotten bugs filed against me for performance stuff. [LAUGHS] Cool. So that makes sense. Sounds like good stuff. And in talking to some people in preparation for this episode, I had a few people mention Android testing specifically. Not any of the other platforms, just Android. So do you want to tell us why that might be? What are they doing over there that warrants additional mention?
40:15 STEPHEN: Yeah. I mean, I think probably the answer would just be that Android is such a huge part of our code base. Chrome is a browser, a multi-platform browser, runs on multiple desktop platforms, but it also runs on Android. And it runs on iOS. And so I assume that iOS has its own testing framework. I must admit, I don't know much about that at all. But certainly on Android, we have a significant amount of testing framework built up around it. And so there's the option, the ability for you to test your Java code as well as your C++ code.
40:44 SHARON: That makes sense. And yeah, with iOS, because they don't use Blink, I guess there's - that reduces the amount of test that they might need to add, whereas on Android they're still using Blink. But there's a lot of differences because it is mobile, so they're just, OK, we actually can test those things. So let's go more general now. At almost every stage, you've mentioned flakiness. So let's briefly run down, what is flakiness in a test?
41:14 STEPHEN: Yes. So flakiness for a test is just - the definition is just that the test does not consistently produce the same output. When you're talking about flakiness, you actually don't care what the output is. A test that always fails, that's fine. It always fails. But a test that passes 90% of the time and fails 10%, that's not good. That test is not consistent. And it will cause problems.
41:46 SHARON: What are common causes of this?
41:46 STEPHEN: I mean, part of the cause is, as I've said, we write a lot of integration tests in Chromium. Whether those are browser tests, or whether those are web tests, we write these massive tests that span huge stacks. And what comes implicitly with that is timing. Timing is almost always the problem - timing and asynchronicity. Whether that is in the same thread or multiple threads, you write your test, you run it on your developer machine, and it works. And you're like, cool, my test works. But what you don't realize is that you're assuming that in some part of the browser, this function ran, then this function run. And that always happens in your developer machine because you have this CPU, and this much memory, and et cetera, et cetera. Then you commit your code, you land your code, and somewhere a bot runs. And that bot is slower than your machine. And on that bot, those two functions run in the opposite order, and something goes horribly wrong.
42:50 SHARON: What can the typical Chrome engineer writing these tests do in the face of this? What are some practices that you generally should avoid or generally should try to do more often that will keep this from happening in your test?
43:02 STEPHEN: Yeah. So first of all, write more unit tests, write less browser tests, please. Unit tests are - as I've talked about, they're small. They're compact. They focus just on the class that you're testing. And too often, in my opinion - again, I'm sure we'll get some nice emails stating I'm wrong - but too often, in my opinion people go straight to a browser test. And they bring up a whole browser just to test functionality in their class. This sometimes requires writing your class differently so that it can be tested by a unit test. That's worth doing. Beyond that, though, when you are writing a browser test or a web test, something that is more integration, more end to end, be aware of where timing might be creeping in. So to give an example, in a browser test, you often do things like start by loading some web contents. And then you will try and poke at those web contents. Well, so one thing that people often don't realize is that loading web contents, that's not a synchronous process. Actually knowing when a page is finished loading is slightly difficult. It's quite interesting. And so there are helper functions to try and let you wait for this to happen, sort of event waiters. And you should - unfortunately, the first part is you have to be aware of this, which is just hard to be. But the second part is, once you are aware of where these can creep in, make sure you're waiting for the right events. And make sure that once those events have happened, you are in a state where the next call makes sense.
44:28 SHARON: That makes sense. You mentioned rewriting your classes so they're more easily testable by a unit test. So what are common things you can do in terms of how you write or structure your classes that make them more testable? And just that seems like a general good software engineering practice to do.
44:50 STEPHEN: Yeah, absolutely. So one of the biggest ones I think we see in Chromium is to not use singleton accessors to get at state. And what I mean by that is, you'll see a lot of code in Chromium that just goes ahead and threw some mechanism that says, hey, get the current web contents. And as you, I think, you've talked about on this program before, web contents is this massive class with all these methods. And so if you just go ahead and get the current web contents and then go do stuff on that web contents, whatever, when it comes to running a test, well, it's like, hold on. That's trying to fetch a real web contents. But we're writing a unit test. What does that even look like? And so the way around this is to do what we call dependency injection. And I'm sure as I've said that word, a bunch of listeners or viewers have just recoiled in fear. But we don't lean heavily into dependency injection in Chromium. But it is useful for things like this. Instead of saying, go get the web contents, pass a web contents into your class. Make a web contents available as an input. And that means when you create the test, you can use a fake or a mock web contents. We can talk about difference between fakes and mocks as well. And then, instead of having it go do real things in real code, you can just be like, no, no, no. I'm testing my class. When you call it web contents do a thing, just return this value. I don't care about web contents. Someone else is going to test that.
46:19 SHARON: Something else I've either seen or been told in code review is to add delegates and whatnot.
46:25 STEPHEN: Mm-hmm.
46:25 SHARON: Is that a good general strategy for making things more testable?
46:25 STEPHEN: Yeah. It's similar to the idea of doing dependency injection by passing in your web contents. Instead of passing in your web contents, pass in a class that can provide things. And it's sort of a balance. It's a way to balance, if you have a lot of dependencies, do you really want to add 25 different inputs to your class? Probably not. But you define a delegate interface, and then you can mock out that delegate. You pass in that one delegate, and then when delegate dot get web content is called, you can mock that out. So very much the same goal, another way to do it.
47:04 SHARON: That sounds good. Yeah, I think in general, in terms of Chrome specifically, a lot of these testing best practices, making things testable, these aren't Chrome-specific. These are general software engineering-specific, C++-specific, and those you can look more into separately. Here we're mostly talking about what are the Chrome things. Right?
47:24 STEPHEN: Yeah.
47:24 SHARON: Things that you can't just find as easily on Stack Overflow and such. So you mentioned fakes and mocks just now. Do you want to tell us a bit about the difference there?
47:32 STEPHEN: I certainly can do it. Though I want to caveat that you can also just go look up those on Stack Overflow. But yeah. So just to go briefly into it, there is - in testing you'll often see the concept of a fake version of a class and also a mock version of a class. And the difference is just that a fake version of the class is a, what I'm going to call a real class that you write in C++. And you will probably write some code to be like, hey, when it calls this function, maybe you keep some state internally. But you're not using the real web contents, for example. You're using a fake. A mock is actually a thing out of the Google test support library. It's part of a - Google mock is the name of the sub-library, I guess, the sub-framework that provides this. And it is basically a bunch of magic that makes that fake stuff happen automatically. So you can basically say, hey, instead of a web contents, just mock that web contents out. And the nice part about mock is, you don't have to define behavior for any method you don't care about. So if there are, as we've discussed, 100 methods inside web contents, you don't have to implement them all. You can be like, OK, I only care about the do Foobar method. When that is called, do this.
48:51 SHARON: Makes sense. One last type of test, which we don't hear about that often in Chrome but does exist quite a bit in other areas, is manual testing. So do we actually have manual testing in Chrome? And if so, how does that work?
49:03 STEPHEN: Yeah, we actually do. We're slightly crossing the boundary here from the open Chromium into the product that is Google Chrome. But we do have manual tests. And they are useful. They are a thing. Most often, you will see this in two cases as a Chrome engineer. You basically work with the test team. As I said, all a little bit internal now. But you work with the test team to define a set of test cases for your feature. And these are almost always end-to-end tests. So go to this website, click on this button, you should see this flow, this should happen, et cetera. And sometimes we run these just as part of the launch process. So when you're first launching a new feature, you can be like, hey, I would love for some people to basically go through this and smoke test it, make sure that everything is correct. Some things we test every release. They're so important that we need to have them tested. We need to be sure they work. But obviously, all of the caveats about manual testing out there in the real world, they apply equally to Chromium or to Chrome. Manual testing is slow. It's expensive. We require people - specialized people that we have to pay and that they have to sit there, and click on things, and that sort of thing, and file bugs when it doesn't work. So wherever possible, please do not write manual tests. Please write automated testing. Test your code, please. But then, yeah, it can be used.
50:33 SHARON: In my limited experience working on Chrome, the only place that I've seen there actually be any level of dependency on manual test has been in accessibility stuff -
50:38 STEPHEN: Yeah.
50:38 SHARON: which kind of makes sense. A lot of that stuff is not necessarily - it is stuff that you would want to have a person check because, sure, we can think that the speaker is saying this, but we should make sure that that's the case.
50:57 STEPHEN: Exactly. I mean, that's really where manual test shines, where we can't integration test accessibility because you can't test the screen reader device or the speaker device. Whatever you're using, we can't test that part. So yes, you have to then have a manual test team that checks that things are actually working.
51:19 SHARON: That's about all of our written down points to cover. Do you have any general thoughts, things that you think people should know about tests, things that people maybe ask you about tests quite frequently, anything else you'd like to share with our lovely listeners?
51:30 STEPHEN: I mean, I think I've covered most of them. Please write tests. Write tests not just for code you're adding but for code you're modifying, for code that you wander into a directory and you say, how could this possibly work? Go write a test for it. Figure out how it could work or how it couldn't work. Writing tests is good.
51:50 SHARON: All right. And we like to shout-out a Slack channel of interest. Which one would be the - which one or ones would be a good Slack channel to post in if you have questions or want to get more into testing?
52:03 STEPHEN: Yeah. It's a great question. I mean, I always like to - I think it's been called out before, but the hashtag #halp channel is very useful for getting help in general. There is a hashtag #wpt channel. If you want to go ask about web platform tests, that's there. There's probably a hashtag #testing. But I'm going to admit, I'm not in it, so I don't know.
52:27 SHARON: Somewhat related is there's a hashtag #debugging channel.
52:27 STEPHEN: Oh.
52:27 SHARON: So if you want to learn about how to actually do debugging and not just do log print debugging.
52:34 STEPHEN: Oh, I was about to say, do you mean by printf'ing everywhere in your code?
52:41 SHARON: [LAUGHS] So there are a certain few people who like to do things in an actual debugger or enjoy doing that. And for a test, that can be a useful thing too - a tool to have. So that also might be something of interest. All right, yeah. And kind of generally, as you mentioned a lot of things are your opinion. And it seems like we currently don't have a style guide for tests or best practices kind of thing. So how can we -
53:13 STEPHEN: [LAUGHS] How can we get there? How do we achieve that?
53:19 SHARON: How do we get one?
53:19 STEPHEN: Yeah.
53:19 SHARON: How do we make that happen?
53:19 STEPHEN: It's a hard question. We do - there is documentation for
testing, but it's everywhere. I think there's /docs/testing, which has some
general information. But so often, there's just random READMEs around the code
base that are like, oh, hey, here's the content public test API surface. Here's
a bunch of useful information you might want to know. I hope you knew to look
in this location. Yeah, it's a good question. Should we have some sort of
process for - like you said, like a style guide but for testing? Yeah, I don't
know. Maybe we should enforce that people dependency inject their code.
54:04 SHARON: Yeah. Well, if any aspiring test nerds want to really get into it, let me know. I have people who are also interested in this and maybe can give you some tips to get started. But yeah, this is a hard problem and especially with so many types of tests everywhere. I mean, even just getting one for each type of test would be useful, let alone all of them together. So anyway - well, that takes us to the end of our testing episode. Thank you very much for being here, Stephen. I think this was very useful. I learned some stuff. So that's cool. So hopefully other people did too. And, yeah, thanks for sitting and answering all these questions.
54:45 STEPHEN: Yeah, absolutely. I mean, I learned some things too. And hopefully we don't have too many angry emails in our inbox now.
54:52 SHARON: Well, there is no email list, so people can't email in if they have issues. [LAUGHTER]
54:58 STEPHEN: If you have opinions, keep them to yourself -
54:58 SHARON: Yeah. [INAUDIBLE]
54:58 STEPHEN: until Sharon invites you on her show.
55:05 SHARON: Yeah, exactly. Yeah. Get on the show, and then you can air your grievances at that point. [LAUGHS] All right. Thank you.