Introduction to automated end-to-end testing
Video transcript
We utilized ChatGPT to enhance the grammar and syntax of the transcript.
Good morning, everyone. I’m glad that you’re all here, and I hope you’re caffeinated. If we haven’t had the opportunity to meet yet, my name is Paul. I used to work at the University of Missouri for a couple of decades, where I was in charge of building and maintaining their processes and workflows for managing their fleets of WordPress sites. So, I have a deep history in both higher education and WordPress. I’m now a Developer Relations Engineer at Platform.sh. If you’re not familiar with us, we are a secure, enterprise-grade Platform-as-a-Service provider. We remove the complexities of cloud infrastructure management so you and your teams can spend more time responding to the changing needs of your institution and supporting your institutional missions, and less time fighting cloud infrastructure. Whether it’s on-site or remote, whether it’s WordPress, Drupal, Django, Gatsby, or whatever you need to run, you can manage all of that from one central location. We can play a crucial role in your end-to-end testing.
Now, before I get started, I do want to give you a couple of warnings. The first is that I am not a testing expert. This presentation, like most of my presentations, came about because I was given a task. I was told to add end-to-end testing to some of the things that we support. I didn’t know much about it, so I had to learn. In that process, I thought maybe others could learn from it too. I asked the WordPress channel if anybody was interested, and I got a great response back. That’s how this presentation came about.
The second thing is that I’m going to do all of this live today, and you know how live demos go, so wish me luck. I’ll probably take up the full 45 minutes.
My goals for you when you leave here today are:
I want you to be able to define what end-to-end testing is.
I want you to be able to remember at least one advantage that end-to-end testing brings to your workflows as well as your organization.
I want you to understand what Cypress is, how to install it, how to configure it, and how to write at least your first test.
I want you to be familiar with the strategies and best practices and have all that foundational knowledge that you need in order to build the second, third, and fourth tests.
So, how many of you here are already doing some form of automated testing in your workflows? Wonderful. How many of you are doing end-to-end testing? A couple? Okay, so for everybody else, if you were given a task or you needed to write some new feature or new functionality, how would you go about verifying that what you’ve written is working the way you expect?
[Audience member response] "Visit the site."
Exactly, you visit the site and do it manually, right? Well, that’s all automated testing is. End-to-end testing mimics the real user and tries to replicate the steps that they would take and then verifies that the application has done exactly what we expected. That’s all end-to-end testing is—it’s just the automation of that manual process.
Now, it does bring some great benefits. In a lot of cases, we can’t surface problems with our software until it is working in that real-world scenario where it’s actually interacting with the database, has the user interface in front of it, and is connecting to all those third-party services. We really can’t surface those problems until we’ve tested it just like the user would. But that takes time, so that’s where automation benefits come in. Additionally, because we can catch those bugs before they occur—before we release them into production, it’s much less costly to fix them. Once we get our test passing, we then have a very high level of confidence that what we’re introducing or deploying to production is going to work exactly as we expected, and it’s not going to negatively affect the user experience, ultimately increasing our quality assurance. Lastly, if you’re working with departments where they’re giving you a set of features that they want, and you can write your test to verify those features, then you can ensure that you’re in alignment with those business requirements.
Now, how does end-to-end testing differ from other types of testing? Well, if you’re familiar with the automated testing strategy pyramid, we do unit tests at the bottom. We do a whole bunch of them, but they’re isolated, just testing individual pieces. Then we move into integrated testing, where we begin to test our code with external services. We do fewer of those because they’re a little harder to do. At the top, we do end-to-end testing, where we’re testing everything. But because we’re testing everything, it’s much more complex, so we do fewer of those, with the goal being that those manual tests are very few and far between.
Now, if you’re not familiar with the other two, unit testing is the idea that in an ideal world—and I know higher ed is not ideal—but in an ideal world, you actually write a test to test the smallest piece of functionality you can. Then you actually write the test before you write the functionality. So, in your programming language, that’s usually a function or a method. As you write that function or method, you continuously test it over and over again, but you isolate it from all its external requests so you’re only testing that functionality. That’s why we do it very often because it gives that immediate feedback. Now, once we start getting greens—we’re passing all those unit tests—then we move into integrated tests. So, we might take individual units that we’ve been testing and now group them together and begin to test them as a group. We might take our code and begin testing with external dependencies or external services with the goal of surfacing any issues in the communications between our code and those external dependencies. Then we come back to end-to-end testing again, where we’re mimicking everything from start to finish and trying to verify, exactly as a user would, that our features are working.
Now, with that said, the reason we have to do it far less often is because it’s more expensive. That can be both monetarily because we’re going to have to have an environment or server set up in order to run these tests, and also in terms of human resources—it takes time to write out all of these tests. Occasionally, I’ll mention brittleness. It can also be a bit hard to maintain. So, as an example, let’s say you write a feature where you’re going to add a menu item to the admin bar. Then you’re going to have the user go somewhere, they’re going to have a panel, fill in some information, and do something. We can write the test, verify it’s all working correctly, and then in the next iteration, someone decides to change the ID on that menu item. Well, now suddenly we can’t go to that menu item in our test like we did previously, and the test breaks even though the functionality is still there. So that’s just an example of a brittle test. We have to be aware that these can be a bit brittle.
So, Cypress. How many of you have heard of Cypress? Anybody played with it? A couple? Okay, so for everybody else, Cypress is an open-source automated testing framework whose main goal when they designed it or began to build it was to expedite the time between when you install it and, as a developer, start writing your tests. In previous times—the dark ages, I guess—there was a lot of configuration and setup involved in building automation of the browser. Their goal again was to get you in there and writing those tests as fast as possible.
Now, it does bring some specific advantages. It is open source; they have an optional SaaS product. It doesn’t use Selenium, it doesn’t use WebDrivers; it uses an Electron app and it embeds your application—your web application—inside that Electron, meaning it’s all running inside the browser, so it’s very, very fast. Now, they support all the browsers except Safari, which is on the roadmap. The other cool piece is time travel and test replay. So, when you run a test, you can go back and replay the test and watch your application state change as it moves through all the steps. Time travel then allows you to go back to an individual step and see your application state, how it’s performing, and look at exactly what’s happening at any step as you move backwards and forwards. Additionally, anything that you output to the console will be available in that step, and it provides a whole host of debugging tools for you to be able to debug your test and find out exactly why it’s failing.
Some other cool parts are auto-waiting. So, when you visit a site, it’ll automatically follow redirects until it gets to a 200. Then once it gets the page, it’ll wait for that page’s load event to fire before it continues on. If you’re asking for a DOM element, like in a dynamic application, it’ll automatically wait until that DOM element is available or until it times out. So, auto-waiting is a great feature for you to make sure that things have loaded in the way you want.
Network traffic control is also really cool. You can intercept any request—so any request to an endpoint or a location—you can intercept that, inspect it, change the data that it’s sending out, and when the payload is returned, you can catch the payload and manipulate that data, or you can catch it and return your own data, never allowing it to go back out. Lots of incredibly cool features.
Now, Cypress is not the only game in town. There are several alternatives to Cypress, including Playwright, Nightwatch, WebDriver, Codeception, etc. I’ll tell you that the Go team—and don’t quote me on the dates, but I think it was 2019—was on Cypress. Then sometime around 2021-2022, they moved to Puppeteer, which is backed by Google, and then as of February of this year, they’re now on Playwright, which is backed by Microsoft. Personally, coming out of my 20-plus years in higher ed, I have great respect and value for those tools that I can depend on for a long time, right? I never had the opportunity to change my tooling in 18 months; that was not an option. I needed something that I could depend on for 3 to 5 years, so I am sticking with Cypress because their documentation is excellent. They have not only thorough documents but also guides, tutorials, videos, and tons of example code, again with that goal of getting you to writing your test as quickly as possible.
That said, maybe next year I’ll be back and talking about Playwright, I’m not sure, but for now, it’s Cypress.
Alright, so installing Cypress, again, as fast as possible, right? It’s as simple as an npm install. And this is where we switch to the live code, so start wishing me luck that we have some internet here. So, I’m going to go ahead and start that. [Pause for action]
That is not clicking. There it goes. What’s going on? Oh, it’s back here. That’s interesting. Let’s try and clear npm install, and that is it. That’s all you have to do to get it installed. It automatically downloads the dependencies and builds the Electron app based on your operating system. It should actually be… there it is, it’s done.
From there, to configure Cypress, all we have to do is open it. So, I’ll open this up. It goes and builds that app. The first time it runs, it says, "Hey, you don’t have any configuration files. What are we doing today? Are we running end-to-end tests or component tests?" We’re going to focus on end-to-end. So, as soon as you click on that, it says, "Okay, I’ll configure things for you in the background." Now, once it updates, notice I’ve got a configuration file and a collection of supporting files, and that’s it. That’s all you have to do to configure it. From there, you just pick which browser you want to start testing in. I’m running my presentation in Chrome today, so I’m going to use Firefox. Go ahead and launch that. And then, just like with the configurations, as soon as it comes up, it says, "I don’t see any testing files." Testing files in Cypress are referred to as spec files. It says, "Do you want me to give you some scaffolding files, some examples so that you can start building your own, or would you like to go ahead and write your first test?" Let’s go ahead and write our first test. I’ll call this one "first" and then create that.
In the background, it has written that file in a directory called e2e. Can everybody see that code? Is that big enough? That’s big enough? Alright. What I’d like to do is, as we go through this, kind of break down all of these parts.
The very first part we have is called "describe." The describe block is simply an organizational tool for you. You can have multiple describe inside a single spec file. You can nest describe inside of a describe. It’s simply an organizational tool. It takes two parameters: a title that you want to have show up in the testing tool to identify your group of tests, and a callback function. We put our tests inside the callback function.
So then, notice the next piece down is "it." it blocks contain two things: it’s where we put our test, and it takes a label, again, to show up in the testing tool so we can identify the test we’re running, and then another callback function. That callback function is where we put our actions and our assertions, where we’re actually going to start doing some things.
So, if we’re testing a web application, what is the first thing you probably need to do? You’ve got to get to the application, right? So, usually, one of the first things you’re going to do or use is cy.visit. That is going to load a URL. It’s going to default to a GET request. It’s going to follow any 300 redirects until it gets to a 200, and once it gets that 200, it will wait until the page is loaded and fires. So, I’m going to go back into our code. I’m going to update this. It’s a little hard to see with all this other Zoom stuff, so if I misspell something, we’ll just go with it. [Pause for action]
It says, "Page has valid title." I don’t think I totally missed that. We’re going to go ahead and run this. Let’s run this. [Pause for action]
Oh, I forgot to mention, I am running DDEV. Anyone familiar with DDEV? A couple of you? DDEV is a local development environment. I use it a lot because we have an integration with it, which allows me to clone my production site, which I also forgot to tell you about. I have a production site. This is my production site. So, this allows me, from Platform.sh, to clone my production site locally into DDEV. I’m going to go grab that, so I’m going to say DDEV. That’s the right one, there we go. So, this is the URL assigned to my local instance. I’ll go back into my test. Let’s update this. Save that and run our test. [Pause for action]
Go. Now we can see that we have… hopefully, you can see that. There’s the homepage. That was the title I gave the describe, and then you name the test itself. The test body currently just has visiting the site.
Alright, once we have visited somewhere, what do we need to do next? In other words, I want to verify that this page is what I expect it to be.
[Audience response] "Check the page title."
Yes, check the page title, right? You need to get some DOM element and then verify that it contains data, has a class, or is visible. We need to, in some way, validate that the page is what we thought it was going to be. So, cy.get allows you to get one or more DOM elements from the page. It’s going to yield back that DOM element to the next thing that we chain to it. It’s always going to start—this is important to remember—it’s always going to start from cy.root, which is typically the document root. So, if I do cy.get("main") and then another chain with cy.get, it’s not going to limit the searches to that; it’s going to go back to the root. Now, there are other ways to do that, but just not with cy.get.
So, cy.get is how we’re going to get our elements, and just like the others, it’s going to automatically keep retrying to get that DOM element until it either exists or we reach some timeout. So, I’ll go back into our code, and I’ll add in cy.get. Let’s get the title—there it is. And then, what do we need to do? Tell it what it should be, right? So, the method we use is should. We use the should method on that DOM element, and that is what’s going to create our assertion or our validation. Now, it accepts a Chai-compatible string. Chai is a Behavior Driven Development (BDD) testing and Test-Driven Development (TDD) assertion library. It lets you use plain English words to describe the assertion or validation you’re trying to make. It’s always going to yield back the same subject it was given, so if I give it the title element and make some assertion, if it passes, it’ll yield it to the next, which means I can chain assertions. It’s automatically going to keep retrying to get a positive assertion until it either reaches a timeout or ultimately fails.
So, in this case, I’m going to break this the first time just so we can see it. I’m going to say should contain "WP Campus 2023." Let’s run that test, and notice it fails because this isn’t 2023, right? So, I’ve got a nice little red check mark— you might not be able to see the red check mark there—up next to the test, indicating the test failed. The assertion did not pass. Now, I’ll come back in and hopefully, I can fix that. There we go. Now it’s actually... it also watches, so it’s going to watch your test and automatically rerun it. It’s probably already rerun that test, and it did. So now I’ve got a nice green assertion. I’m also colorblind, so if that’s not green, somebody please tell me. I’m assuming it’s green. Okay, so we’ve got a green assertion and a green check mark, which means that you have now just completed your first test. There we go.
I know that’s a very simple test, but for me, the hardest part about learning some new thing is just getting the thing installed, getting it configured, and then doing something. So, if you can go back now and install it, configure it—which is basically just opening it—and then write just a simple test of "go to my homepage and make sure the title is right," you’ve now incorporated it into your workflow. The next one will be that much easier.
In fact, as you begin working towards writing your next set of tests, there are some things to keep in mind. One is that writing the hardest part about tests is knowing what to test. That’s the hardest piece. So, my suggestion would be that the next time you’re given a task, a feature that’s supposed to be incorporated into your site, write down all the steps that you take to manually verify it. Go in and say, "Okay, I clicked here, I clicked here, I clicked here, I clicked here," and at the end, "This is what should be on this page. This is how it should react." Keep breaking those down into smaller and smaller steps, and then convert those steps and actions into a test. Then keep running that test until you figure out—as long as the feature is working—how to make sure you’re writing it correctly. Make sure you’re testing both the happy paths and the unhappy paths.
What I mean by that is, if we were testing a login form, for example, a happy path would be: I put in a username and password that’s correct, I click submit, I get somewhere in the admin area, and in the upper right is my username. That’s the happy path. An unhappy path would be: I put in a username and password that aren’t correct, I click submit, I come back to the form, and now I have an error message. So, we want to make sure we’re testing the unhappy path to ensure the application is still acting and performing the way we expect it to, even if the user doesn’t follow the correct steps.
Now, a good test should cover three phases. One is setting up the application state—we’ll talk more about application state later. The next is to take those actions, again mimicking the user, performing the steps necessary to test this feature. And then the last is to make an assertion about that resulting state. Now, you may see this referred to as "arrange, act, and assert," or you might see it referred to as "given, when, then." So, if we think back to the first test, given I’m an anonymous user, when I visit the homepage, then the title of the page should be "WP Campus 2024." Make sense? So, be thinking in those terms: given some state of the application, when I do these steps, then the application should be this way.
The other piece to think about as you’re writing these tests is you should try to make them such that they are independent of each other. What I mean by that is, if I have four tests, I don’t want test three to be dependent on the state that test one created. Because one: test one may pass incorrectly and leave me with improper state; or two: it might fail and leave me with a state that is going to negatively affect test three. So, we want to make sure that any test can run independent of the others.
Are you ready to write the second test? That’s what I’m aiming for. Okay, so I’m going to come in here and create a second file. Get up there. We’ll do a new JavaScript file, and I’ll call this one auth.cy.js. Let’s say I’ll describe, and we are going to describe this as auth test. Then we have our callback. And then I’m going to do it can auth. So far, nothing too complicated. Callback function, lots of callbacks. And then, what’s the first thing I need to do if we want to test the login form? I need to visit the login form, right? So, I’m going to go cy.visit. I’m going to go back to DDEV, I’m going to grab my address again, come back in and say, “Go visit there with wp-login.php”. Now, can anybody foresee an issue that I have just created between auth.cy.js and first.cy.js, specifically in the visit?
Where is that URL right now? "Homepage." Homepage of where is it running? "Local." What if I need to run my end-to-end test on a shared development environment? What if I need to run them in a staging environment, or PR environment, or somewhere else? All right, so we need to talk briefly about configuration. One of the options in the configuration file is something called baseUrl. baseUrl allows us to define a domain, and then any calls to cy.visit or cy.request that are relative will have that prepended. But the really cool thing is that it can be overridden by other environment variables. So, Cypress does have a processing order for environment variables. And so, for that special baseUrl variable or any other environment variables we need to create, we can either put them in config, but we can also write an .env file that overrides those. Or, if the system where we’re running cypress run or cypress open has environment variables that are prepended with CYPRESS_, it’ll automatically parse all those and override any of the others. Or, at the time we run Cypress, we can also pass in overrides for the config or those environment variables. Or—it’s not even on the slide—in an individual test, you can override a config or those environment variables, giving us a lot of flexibility. So, that means we can write our tests such that we can run them anywhere and not have to worry about changing the test for any specific environment.
So, I am going to go in here and open up the configuration file, and I’ll add that baseUrl in it. I’m going to say maybe normally everybody else on the team uses https://local.site/, and I’m the oddball out. Alright, now if I run that right now—watch, Cypress closed out—come back and it says, "Hey, I can’t get there because there’s nothing there," right? So, if I come back in, close Cypress out first - close Cypress—I’m going to rerun this. Come back into my testing environment. Can you see down at the bottom? Hopefully. I’m going to do an export where I’m setting baseUrl equals that address I used earlier. Okay, I’m going to go ahead and do that. I’ll do npx open again. [Pause for action]
Oops, over too far. There we go. And then in my tests here, I will update those two tests. [Pause for action]
Update these tests. [Pause for action]
That one, and my auth test. There it is. And if we save those and go back out to Cypress, relaunch the testing environment. [Pause for action]
Go. And now if I run first.cy.js, we still go back to our local instance. And if I run auth.cy.js, we end up at the login. Now, I need to do a PR request, and I have a workflow that builds a copy of that environment. I don’t have to change my test; it’s all going to work. I’m going to be able to run those directly on that environment without having to change my test.
Now, if we’re testing the login form, what do we need to do next? Type the username? We need to type in a username and password. One of the cool things about this environment is if you notice this little button up next to the URL, it will give you an element suggested selector—a lot of words. So, I’m going to hit that, and as I hover around in here, it’s going to say, "Hey, here are all these different elements you can select." I click on it, then it gives me an option to copy it into my clipboard. So now, I can go back into my test and say, "Alright, we are going to go in here and get that element." Now, one thing I’ll suggest that I’ve learned is go ahead and do a clear() on the element. That just clears out the element in case there was placeholder text or anything else. And from there, I can type "bob." And what’s the next piece we need? Password. Sometimes it doesn’t rerun it; it doesn’t always redo the select. Let’s grab that again. Go in here, and we’ll drop you pasted. Here we go. And another clear(), and another type "123." Save that. Come over to our test. Now, we’ve got "bob" and "123." What’s the last thing we need to do?
[Audience response] "Enter."
I heard two things. I heard "enter" and I heard "click." You can do both. So, one thing you can do is simulate button presses or key presses using the curly braces and a short tag for that key. So, I’ll go ahead and do that. Or, if we want... I wiped that all out—oops—"123". Let me come back over and grab that button name. Come back in here, get that, and do a click. Wow, you’re a tough audience; you’re not laughing. I think it's pretty cool.
So, can anybody foresee a problem with what I’ve done here? "SSO." Oh, well, SSO might be one thing, which highlights… yes, you’re highlighting the problem. What is wrong about this? It’s hardcoded, right? We don’t want that. But we have the ability to start add environment variables, right? So, I can go back into my config, and I can say, "Hey, I want to add environment variables." Oops, that is not the right key. Let’s try that one. There we go. And I can say, "Hey, I want test_user, and I want test_user to default to bob." Oh, we’ll do "bob2" to change, and then I need test_userPass, and I want that default to "4567". Right? Now, if I update that and I come back into my test, I can get rid of those static values, and I can say, "Hey, I want to use Cypress.env and I want you to go retrieve test_user, and I want you to go here and I want you to go retrieve Cypress.env('test_user_pass)." And if we save that, assuming I didn’t make any typos, it’s using bob2 now. Which means then, in another environment, I can add those environment variables and it’ll automatically pick those up.
Alright, now there is one piece I want to mention real quick, and that is you do need to decide on a strategy for managing state. Now, this particular slide is about the state of a user since we’re testing a user, but this applies to the state of the application itself as well. So, one strategy is you can stub requests. Remember I mentioned the intercept? So, we could intercept a request to wp-login, catch that, and then return back static information about the cookies and the session information. It’s nice because it’s really fast, and I don’t have to have an environment—I don’t have to set up a database. But notice he’s grimacing because it requires fixtures. Fixtures are the static data that you want to return back. If the cookie information and the session information change in the app, I’ve got to remember to go back and update that static information I’ve saved. But it’s also not a true end-to-end test, right? In an end-to-end test, we really want to try to mimic as close to real-world as possible what the end user is experiencing.
So, the next strategy is a static user, where we go into our system and actually create a user that we’re going to use in our tests. It’s nice because then it is a real end-to-end session, right? We’ve got a database, we’ve got a real user. But that means we have to have an environment where we can put a database, and we’re going to have to set up the database—whether that’s exporting from some other system and then importing it there, or maybe automating the installation of WordPress and setting up that space. The downside, though, is that - particularly in WordPress - your user, as they do things, the state of what is associated with them changes and is stored in the database, right? So, we end up with dangling state or stacking state, which could affect reruns of the test or future tests because we’ve got that saved state with the user.
So, the last type is a dynamic user, where you’re going to delete and recreate the user before every test. This is a true end-to-end test because now we’re recreating exactly the experience that the user would have. We don’t have to worry about any of that state mutation or dangling state, but we do have to do the whole teardown and rebuild of that state with the database every time, which means it is a bit more slow, a bit more complex.
Well, for the purposes of today, I’m going to — if I can find the right keys — I’m going to have DDEV create a user for me, and I’ll have it create bob2. I’m going to take this password so we have a real live user. Copy that, come back over to Cypress, I’ll close out my test, close Cypress, and come back. And before I launch it, I’ll do an export of those values. Alright, so I’m doing an export of test_user, so now we’re going to use bob2 and that password. Now, let’s relaunch Cypress. Let’s relaunch our end-to-end testing.
[Pause for action]
Now, if we go to the auth test, TA-DA!, we have a real user. You can validate this worked because if you don’t put in the correct username and password, you go back to the login, but it does send a 200, which technically means it passes. So, how can we validate this worked? Grab the username. So, I’m going to go up here, I’ll use that picker, I’ll say, "Hey, give me this." I’ll grab that, go back into my test and say, "Okay, after you click it," it’s automatically going to wait until the next page loads, so then I can say, "Get that." I’m going to drop down because it’s getting a little long, and I’m going to say, should contain, and then I’ll go back and say Cypress.env('test_user'). If I spell it correctly. Now, if I save that, we run the test. Yay, there it is. There’s our assertion. It’s nice and green. We passed. Good? Everybody good so far?
Okay, so now let’s say we want to extend the test. We just did the writing of the second test, but we want to extend it. Don’t look at that; forget you saw that. I’m sorry, I went too far. So, we want to extend this test. Let’s say now we want to have a second test where we can add a custom post type post. Maybe they have a custom role—this user has a custom role—and we want to verify that they can create that post type and see it. One thing I have not mentioned is that when you run Cypress, when it runs each test, it’s just like it is using incognito mode or private mode for every test. This means it brings up an instance, it runs it, it finishes, it closes it, which means if I try to go here and do cy.visit and then maybe do something like wp-admin/edit.php, what’s going to happen? It’ll succeed in that it’ll get a 200, but it’s not going to go to the right place, right? So, should I copy all of this content and paste it into the next test?
[Audience response] "No."
Okay, the shaking of the head is the correct answer. Right, instead—remember I said it gave us a bunch of supporting files? One of those files is called "commands.js." That allows us to create our own custom commands. So, I can use Cypress.commands.add(). The first parameter is the name of the method, this new command you want to call, so I’ll say "wplogin." The second is—guess what the second one is? Callback function, right? Callback function. And then, inside of here, we can put in that code.
Now, do I still want to use these environment variables? Probably. Maybe? What if I want to test multiple users? Instead, I could say, "Alright, the callback function will accept a username and password." Now I can go in here and wipe this out. Then this— [Pause for action]
All of it, there we go. And say, "Username, username here." Oh, that was password, sorry—that’ll fail. Try "Password," here we go. And then, down here, do "Username." Then back in my auth test, what I can do is say cy.wplogin() and then hand it—wrong key there—Cypress.env('test_user') and Cypress.env('test_user_pass'). One downside of live demos is that I am not a fast typist. All right, now let’s go see if that worked. There’s our auth test. The first one said, "Yeah, look, it ran all the code, and we..." Oh, this is a good time to show you—remember that time travel? Here are all the different states. You can see it got the input for the username, typed it in, cleared the next one out, and the password was typed in. Click submit, follows, follows, follows, all the way there, and there’s our assertion.
Cool. Alright, so now let’s say I do want to make that next test. Should I copy this line and paste it in?
[Audience response] "No."
No. So, just like with WordPress, Cypress gives you hooks. One of those hooks is to run something before each test. Guess what it takes? Whoops, that’s too many—callback function. Yeah, everything’s a callback. And so now, I can take this line—actually, let’s do it this way; it’s running off the side of my screen, I can’t see it—cut it from there, put it up here. Now, if I run that, notice not only did it do it on the first one, it also did it on the second one. And now, at the end, it actually did the edit page.
Alright, but it is going to the form, getting the elements, and putting in the information every single time. Two tests, probably not a big deal. But what if we have 15 tests? Wouldn’t it be nice if, instead... this is what I wanted you to see earlier—if we could save the session data? Yeah. So, we have that ability in Cypress. It will take a session that we’ve created and store it based on some ID. Now, it takes four arguments. I bet you can guess what a couple of them are. The first one is an ID on how we want to identify that session. The next one is a callback where we run our setup code. The third option is another callback where, if we restore a session, we can validate the session and If validation fails it will rerun the setup callback. And the fourth is whether or not we want to share this saved session with additional spec files.
So, I'm going to add this to our code. I'll go back into the commands, grab all of this, cut it out, and say cy.session. I'll use the username as our ID, set the callback, and then paste our information back in. Now, if I save and run these tests, the first time it says, "Hey, I need to create the bobby2 session," and it runs through all those steps, just like earlier. But the second time it says, "I need the bobby2 session, but I'm going to restore it," so it doesn't have to repeat that process.
This is huge and really important because many of you will likely need to perform authenticated session tests. This was one of the hardest parts I came across, and the reason is that between version 12 and version 13, Cypress introduced a feature that made it much easier to save session information. If you search online for how to do this, you'll often find outdated articles and blog posts that no longer apply.
In the last few minutes, I want to give you some best practices to keep in mind as you start writing these tests. First, if you don’t need to test a login form, don’t bother bringing Chrome into the process. Instead, handle it programmatically. By that, I mean instead of visiting the page, do a POST to the wp-login.php page. Everything else remains the same, including the method name and session setup, but I'm posting that information programmatically, so I don't have to bring up Chrome.
The next point to consider is one of the trade-offs when using Cypress, especially if you're already a JavaScript developer, you can’t assign the return values from a Cypress command to a variable or constant; you can only access those returns via closures and aliases. Some people find this jarring, but it's the trade-off for speed. To quickly dive into Cypress, you have to work within their opinionated framework.
I mentioned this earlier, but try to set the state before each test, so that any test can run independently at any point. Also, try to avoid using brittle selectors. IDs and classes may change, and even the location of an element could change without being visibly noticeable. Cypress suggests using data attributes where possible, setting unique data attributes that don’t change, to ensure your tests are not brittle.
OK, pop quiz: who can define end-to-end testing? Anyone? This was one of my goals and I don't want to fail my goal! Somebody!
[Audience member]: test from end-to-end
Test all the parts, from beginning to end! We're going to try to mimic a user walking through the paths of our application and then validate from what they did, our application responded correctly. Can anybody tell me one benefit of end-to-end testing?
[Audience member]: save time!
Save time! right? Way faster. Basically doing manual testing but a lot faster, because the computer is way faster than you. All right, does everybody feel comfortable with what Cypress is, installing it, configuring it – which you're just opening it – and maybe writing your first test?
[Audience member]: I have a question
Sure.
[Audience member]: So in the code you wrote Cypress with a capital and then cy. What's the difference?
Cypress refers to the global application, and you use it for accessing environment variables and other global settings. For other commands, you use cy within your tests. You can check the docs for more details.
So does everyone feel comfortable writing that first test? Do you feel fairly comfortable with the best strategies on how to write your second and third tests?
Now, I want to show you a real-world scenario to give you an idea of how this all fits together. One of the reasons I love Cypress is that they have a well-documented and stable GitHub action. This means that in my GitHub workflows, when I create a pull request, Cypress can automatically run all my end-to-end tests. At Platform, we also have GitHub integration. When you create a pull request, it sends that code to us. We then clone your production environment into an isolated environment with that pull request code, bring up the cloned environment, generate an ephemeral URL, and send that URL back to GitHub.
When I run the Cypress action, I can pass in that URL of the cloned production environment. So, when I run the end-to-end tests, they're exactly what I would expect in production. This boosts confidence in the code before deployment.
I'll show you this in practice. I have a pull request open for my production site, and down at the bottom with my checks, I can see the Platform integration completed, and it deployed that PR environment. Opening it up, I can see a clone of my production environment assigned to the pull request. I also see that my end-to-end testing failed. Looking into the details, it says six tests passed and one failed. If you integrate your project with Cypress Cloud, you can replay the tests when something fails. You can rerun the test, see the steps it performed on the left-hand side, and view the state of the application on the right-hand side. In this case, the test failed because the trash can icon was missing, preventing a user from deleting their post. Now I can use the testing tools to debug the issue.
Alright, I think we have 15 minutes for questions. Does anyone have any?
Q&A
Please note: these are best-guess due to poor audio quality
[Audience member]: When you're deciding what to end-to-end test, is it always some sort of frontend functionality you're testing? or How do you decide broadly what to test?
[Paul]: This tool is for web based applications which most always will be inside some web interface. There are other tools that can do API testing directly. You could fake API testing if you really need to but this isn't meant for that. When you need to mimic a user's Behavior through the application and validate some feature or function, this is what you use.
[Audience member]: When you're working with WordPress you're often modifying a lot of core so is it helpful, or is there like a base package to actually test core before you go in and test your individual parts of the application? Is there a library you can pull to make sure regular WordPress works first and then test our layered-on functionality?
[Paul]: Core switched to Playwright so they have the Playwright tests; they abandoned the Cypress test years ago so "no" unfortunately. I would say if that is a crucial piece then I would look into Playwright and then that way you've got their test that you can run. It's not like Playwright is bad. I don't want anybody to think that, you shouldn't use it. It's just it's new; it's only a couple years old, the documentation is there but not as extensive. So if that's an important feature I'd say check out Playwright and then that way you've got their test that you can run.
[Audience member]: Well, I mean I don't know if you like actually need to test like everything that core is testing. You know, just overview. Like, oh WordPress is here. You know what I mean?
[Paul]: That's another example. We manage, when you do the one-click deploys, right? We manage those and so I have all the updates for those automated. And as part of the test, I do a quick check of core features to make sure the most important features are there so that none of our custom code has negatively affected any of the core features. So, yes. I mean I've done that and no reason you can't.
[Audience member]: Can you interact with that clone? If you want to do your own manual testing or something?
[Paul]: No, you can't. It's going to save the state of each one but you can't click and do additional things.
[Audience member]:
You can't log in as yourself and try stuff? You'd have to write a test for it?
[Different audience member]:
I think you're just asking: can you as a human visit the platform PR environment?
[Original audience member]:
Yeah.
[Paul]: Oh,absolutely! I thought you meant the screenshots of the testing environment that we saw that Cypress brought up to show us the test.
[Audience member]: No, no. I meant the clone using the new code. It'd be cool to be able to interact with that.
[Paul]: Yeah, you can absolutely do that. In fact, you can use that to help write your test too. When you go and check it, use that as where you're recording all the steps that you're taking. I'm sorry, I didn't understand. Anybody else?
[Audience member]: Do you have any visual regression in your stack that you do with these tools?
[Paul]: So that's a separate tool I use and I'm going to forget its name… It's got a monkey face… oh I forget the visual regression testing name… and I will remember about two seconds after you all leave what the name of that is… There is an open source - backstop! Backstop, we use backstop for visual regression testing to make sure there's no visual regressions. If you're not familiar with visual regression testing, what it does is: you give it a production site or a baseline site that should be correct and then you give it your testing environment, your development, staging, PR, whatever environment. It takes screenshots of locations that you identify and it diffs them. If the diff is greater than some percentage that you define, then it flags it as a false or that there has been some regression. So, yeah, absolutely you can add it in and I would encourage you to add in as many of these kind of automated tests as you can to reduce the number of man hours that you're having to put into this stuff, because I know in higher-ed you don't have the time; you just get more responsibilities, not more people. So anytime you can automate these kinds of things the better. Any other questions?
[Audience member]: Someone mentioned SSO. We tried behat like years ago and had trouble with logging into WordPress to do any of the inside stuff. We were able to do frontend, but not like log in, post, so…
[Paul]: Assuming you have a test user in the single sign-on system where you can have the password somewhere to actually use, you should be able to do it. I don't foresee any problems with SSO. I can't say I've tested it recently with SSO but it should work.
[Another audience member]: We haven't done that with Cypress but we've done it with Puppeteer with AWS. The same thing, and yeah… it actually works the right way.
[Paul]: That's a good point. Whether you do Cypress or you use Playwright the concepts are all basically the same. It's just the specifics of maybe how you write the test, right? So if you start playing with Cypress and you decide this isn't going to work for us, it's not like you really lost anything except maybe a little bit of time to learn the specifics. But you can just take that same testing ideology and move it over to the next product. Any other questions? Everybody ready for lunch?
[Room moderator]: There is one online. One question online.
[Paul]: Oh, so the tear down and setup. Yeah, so in the app state. So when I mentioned I test the users for WordPress core, what I have to do is I have functions that I've written in Cypress that will connect to that environment and create the dynamic user just like I did. Actually, first it deletes the user, deletes all the posts, deletes everything about the user. Then it creates the user, retrieves back the password, then sets that as a Cypress environment variable and then kicks off Cypress. So that's how I manage that and you'll have to do a similar thing. You can do it in the GitHub actions if you want, there's no reason you can't but since Cypress has those hooks you can do a before hook and say before I run any of these authenticated tests, run all this setup stuff or the beforeEach, where with GitHub actions it's going to have to be mainly before and after.
I believe that's all the questions online. Are there any other questions? Again I'll be here the rest of the time until Saturday. Feel free to grab me, ask me any more questions about anything. I love automation so if it's testing, DevOps, any of that stuff I love it.