Run automated tests on deployment urls

Edgio Q&A
1

We want to be able run automated tests on the deployment urls for each branch(or PR).
The url structure looks something like this
https: //team-site-branch-deployment-id.team.moovweb.io

Most of this is pretty standard but the deployment-id is something I’m having difficulties with.
Is there a way for me to easily obtain the deployment-id for the given deployment?

Alternatively is there a way for me to create an environment for that particular branch through the xdn-cli(preferably using a copy from another environment that already exists). So that I have a static url which I can run the automated tests on?

2

I noticed when you create an environment on the XDN there is a mutation called: createEnvironment

Would it be possible to expose this endpoint either via rest or graphql?

3

The only way to get this at the moment would be to parse the XDN CLI output:

***** Deployment Complete ********************************************
*                                                                    *
*  🖥  XDN Developer Console:                                        *
*  https://moovweb.app/my-team/my-site/env/default/builds/730        *
*                                                                    *
*  🌎 Website:                                                       *
*  https://my-site-default.moovweb-edge.io                           *
*                                                                    *
**********************************************************************

You cannot create an environment via the CLI or API. Perhaps it would be good to add a CLI command to output the URL in a more easily parsable format?

Something like:

xdn deploy --output-format=json

Which would yield

{ "url": "https://my-site-default.moovweb-edge.io" }

Would that be helpful?

4

@mark.brocato
Not really no.

We have multiple branches that need to run automated tests simultaneously. So if they are all deploying to the same environment it makes the tests worthless.

Ideally I would want a unique url for each branch/build that I can access in a simple manner.

Does this make sense to you?

5

Yes, perhaps my example was misleading… the URL that would be output would be the unique permalink for that deployment, so you could test an infinite number of branches concurrently. So, more precisely

{ "url": "https://my-team-my-app-2.free.moovweb.io" }

6

This is logged as XDN-9217. @siggnja can you confirm this proposal would resolve your issue?

7

@ianand It would definitely be a good start. I would honestly rather be able to dynamically create environments. This would mean we could create an environment that is as similar to production as possible when running the automatic test(for example with edge caching).

8

Can you help elaborate a bit on “environment that is as similar to production as possible”? The edge caching rules in the deployment would be essentially the same as CDN-as-JS and should match production if the CDN-as-JS code in routes.js is the same as production. The things that are different would be redirects, environment variables, domains, and split testing rules.

9

@ianand As I understand, the deployment urls do not have edge caching. Only when the deployments are the current live deployment on a particular environment they have edge caching.

Am I missing something here?

Although a subtle difference it is actually a key, for example if the automated tests are running Lighthouse or something similar to detect performance regressions.

10

Every time you deploy any branch to the XDN, it automatically deploys to the default edge environment. You could run your automated tests against that. You would just need to make sure that you pause any later deployments until the tests have finished. We do exactly this for our platform integration tests! We use GitHub actions for our CI and this handy plugin to ensure that only one deployment happens at a time:

- name: Wait for the previous integration tests to stop running
        uses: softprops/turnstyle@v0.1.3
        with:
          same-branch-only: false
          continue-after-seconds: 1200
          poll-interval-seconds: 30
11

@mark.brocato
I see how this could be a solution. However I’m not really a fan of it. We might have 5-10 builds running at the same time and each automated test run takes 5-10 minutes. This means that some builds might be in the queue for over an hour until they actually start running. This solution also doesn’t scale with more developers and more builds.

Is there no intention of exposing environment creation from an endpoint/cli ?

12

The fundamental problem is that the number of edge environments you can create is limited, while the number of deployments you can create is virtually unlimited. You could scale up a bit by creating a handful of environments and assigning each deployment to a free environment based on some kind of state machine. That would probably get you to a decently large scale, but not infinite.

13

@mark.brocato
Good point, I guess you would have to “clean up the environment after the automated tests”. I guess that is something that could be exposed as-well. This would mean that before the automated tests we create an environment and after the deployment has gone to production/is not needed anymore, we remove the environment.

Unless we are limited to something like 10-15 edge environments. I think this seems like a pretty scalable solution

14

Unfortunately there is a limit on the number environments. However, you can re-use the environments and that’s what we do for our integration tests (i.e. there’s a single environment we use for automated tests that just gets re-used).

Giving you the ability to spin up and delete environments may lead to timing issues. It would ultimately actually be faster for your tests to just re-use an environment (which should be fine since XDN environments are stateless).

15

@ianand
Could you elaborate on this part?

Giving you the ability to spin up and delete environments may lead to timing issues. It would ultimately actually be faster for your tests to just re-use an environment (which should be fine since XDN environments are stateless).

I don’t see how running a few extra queries/commands could be slower than creating some “automated tests queue” where builds might be stuck for over an hour or more.

This feels like a no brainer to me… If there is a hard limitation to the number of environments the endpoint/command would simply throw an error stating the environment limit has been reached.
It also goes back to what I said to @mark.brocato, a single environment for running automated tests is not a scalable solution for large companies with a lot of deployments and developers.

I feel like I’m missing something here :man_shrugging: , if this feels like a super difficult feature request or something that you feel like is something the platform shouldn’t support, please let me know. We will then need to look into alternative solutions to our testing pipeline.

1 Like