Running Cypress Tests in Parallel

Posted by Michael Herman Michael Herman Last updated February 19th, 2019

In the following tutorial, we walk you through configuring Cypress to run tests in parallel with CircleCI.

Want to see the final project in action? Check out the video.

Contents

Project Setup

Let's start by setting up a basic Cypress project:

$ mkdir cypress-parallel && cd cypress-parallel
$ npm init -y
$ npm install cypress --save-dev
$ ./node_modules/.bin/cypress open

This creates a new project folder, adds a package.json file, installs Cypress, opens the Cypress GUI, and scaffolds out the following files and folders:

├── cypress
│   ├── fixtures
│   │   └── example.json
│   ├── integration
│   │   └── examples
│   │       ├── actions.spec.js
│   │       ├── aliasing.spec.js
│   │       ├── assertions.spec.js
│   │       ├── connectors.spec.js
│   │       ├── cookies.spec.js
│   │       ├── cypress_api.spec.js
│   │       ├── files.spec.js
│   │       ├── local_storage.spec.js
│   │       ├── location.spec.js
│   │       ├── misc.spec.js
│   │       ├── navigation.spec.js
│   │       ├── network_requests.spec.js
│   │       ├── querying.spec.js
│   │       ├── spies_stubs_clocks.spec.js
│   │       ├── traversal.spec.js
│   │       ├── utilities.spec.js
│   │       ├── viewport.spec.js
│   │       ├── waiting.spec.js
│   │       └── window.spec.js
│   ├── plugins
│   │   └── index.js
│   └── support
│       ├── commands.js
│       └── index.js
└─── cypress.json

Close the Cypress GUI. Then, remove the "cypress/integration/examples" folder and add four sample spec files:

sample1.spec.js

describe('Cypress parallel run example - 1', () => {
  it('should display the title', () => {
    cy.visit(`https://mherman.org`);
    cy.get('a').contains('Michael Herman');
  });
});

sample2.spec.js

describe('Cypress parallel run example - 2', () => {
  it('should display the blog link', () => {
    cy.visit(`https://mherman.org`);
    cy.get('a').contains('Blog');
  });
});

sample3.spec.js

describe('Cypress parallel run example - 3', () => {
  it('should display the about link', () => {
    cy.visit(`https://mherman.org`);
    cy.get('a').contains('About');
  });
});

sample4.spec.js

describe('Cypress parallel run example - 4', () => {
  it('should display the rss link', () => {
    cy.visit(`https://mherman.org`);
    cy.get('a').contains('RSS');
  });
});

Your project should now have the following structure:

├── cypress
│   ├── fixtures
│   │   └── example.json
│   ├── integration
│   │   ├── sample1.spec.js
│   │   ├── sample2.spec.js
│   │   ├── sample3.spec.js
│   │   └── sample4.spec.js
│   ├── plugins
│   │   └── index.js
│   └── support
│       ├── commands.js
│       └── index.js
├── cypress.json
├── package-lock.json
└── package.json

Make sure the tests pass before moving on:

$ ./node_modules/.bin/cypress run

      Spec                                                Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔ sample1.spec.js                           00:02        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔ sample2.spec.js                           00:01        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔ sample3.spec.js                           00:02        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔ sample4.spec.js                           00:01        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    All specs passed!                           00:08        4        4        -        -        -

Once done, add a .gitignore file:

node_modules/
cypress/videos/
cypress/screenshots/

Create a new repository on GitHub called cypress-parallel, init a new git repo locally, and then commit and push your code up to GitHub.

CircleCI Setup

Sign up for a CircleCI account if you don't already have one. Then, add cypress-parallel as a new project on CircleCI.

Review the Getting Started guide for info on how to set up and work with projects on CircleCI.

Add a new file to the folder called ".circleci", and then add a new file to that folder called config.yml:

version: 2

jobs:
  build:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - checkout
      - run: pwd
      - run: ls
      - restore_cache:
          keys:
            - 'v2-deps-{{ .Branch }}-{{ checksum "package-lock.json" }}'
            - 'v2-deps-{{ .Branch }}-'
            - v2-deps-
      - run: npm ci
      - save_cache:
          key: 'v2-deps-{{ .Branch }}-{{ checksum "package-lock.json" }}'
          paths:
            - ~/.npm
            - ~/.cache
      - persist_to_workspace:
          root: ~/
          paths:
            - .cache
            - tmp
  test:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests
          command: $(npm bin)/cypress run
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots

workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      - test:
          requires:
            - build

Here, we configured two jobs, build and test. The build job installs Cypress, and the tests are run in the test job. Both jobs run inside Docker and extend from the cypress/base image.

For more on CircleCI configuration, review the Configuration Introduction guide.

Commit and push your code to trigger a new build. Make sure both jobs pass. You should be able to see the Cypress recorded videos within the "Artifacts" tab on the test job:

circleci dashboard

With that, let's look at how to split the tests up using the config file, so the Cypress tests can be run in parallel.

Parallelism

We'll start by manually splitting them up. Update the config file like so:

version: 2

jobs:
  build:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - checkout
      - run: pwd
      - run: ls
      - restore_cache:
          keys:
            - 'v2-deps-{{ .Branch }}-{{ checksum "package-lock.json" }}'
            - 'v2-deps-{{ .Branch }}-'
            - v2-deps-
      - run: npm ci
      - save_cache:
          key: 'v2-deps-{{ .Branch }}-{{ checksum "package-lock.json" }}'
          paths:
            - ~/.npm
            - ~/.cache
      - persist_to_workspace:
          root: ~/
          paths:
            - .cache
            - tmp
  test1:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 1
          command: $(npm bin)/cypress run --spec cypress/integration/sample1.spec.js
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots
  test2:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 2
          command: $(npm bin)/cypress run --spec cypress/integration/sample2.spec.js
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots
  test3:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 3
          command: $(npm bin)/cypress run --spec cypress/integration/sample3.spec.js
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots
  test4:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 4
          command: $(npm bin)/cypress run --spec cypress/integration/sample4.spec.js
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots

workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      - test1:
          requires:
            - build
      - test2:
          requires:
            - build
      - test3:
          requires:
            - build
      - test4:
          requires:
            - build

So, we created four test jobs, each will run a single spec file on a different machine on CircleCI. Commit your code and push it up to GitHub. This time, once the build job finishes, you should see each of the test jobs running at the same time:

circleci dashboard

Next, let's look at how to generate the config file dynamically.

Generate CircleCI Config

Create a "lib" folder in the project root, and then add the following files to that folder:

  1. circle.json
  2. generate-circle-config.js

Add the config for the build job to circle.json:

{
  "version": 2,
  "jobs": {
    "build": {
      "working_directory": "~/tmp",
      "docker": [
        {
          "image": "cypress/base:10",
          "environment": {
            "TERM": "xterm"
          }
        }
      ],
      "steps": [
        "checkout",
        {
          "run": "pwd"
        },
        {
          "run": "ls"
        },
        {
          "restore_cache": {
            "keys": [
              "v2-deps-{{ .Branch }}-{{ checksum \"package-lock.json\" }}",
              "v2-deps-{{ .Branch }}-",
              "v2-deps-"
            ]
          }
        },
        {
          "run": "npm ci"
        },
        {
          "save_cache": {
            "key": "v2-deps-{{ .Branch }}-{{ checksum \"package-lock.json\" }}",
            "paths": [
              "~/.npm",
              "~/.cache"
            ]
          }
        },
        {
          "persist_to_workspace": {
            "root": "~/",
            "paths": [
              ".cache",
              "tmp"
            ]
          }
        }
      ]
    }
  },
  "workflows": {
    "version": 2,
    "build_and_test": {
      "jobs": [
        "build"
      ]
    }
  }
}

Essentially, we'll use this config as the base, add the test jobs to it dynamically, and then save the final config file in YAML.

Add the code to generate-circle-config.js that:

  1. Gets the name of the spec files from the "cypress/integration" directory
  2. Reads the circle.json file as an object
  3. Adds the test jobs to the object
  4. Converts the object to YAML and writes it to disc as .circleci/config.yml

Code:

const path = require('path');
const fs = require('fs');

const yaml = require('write-yaml');


/*
  helpers
*/

function createJSON(fileArray, data) {
  for (const [index, value] of fileArray.entries()) {
    data.jobs[`test${index + 1}`] = {
      working_directory: '~/tmp',
      docker: [
        {
          image: 'cypress/base:10',
          environment: {
            TERM: 'xterm',
          },
        },
      ],
      steps: [
        {
          attach_workspace: {
            at: '~/',
          },
        },
        {
          run: 'ls -la cypress',
        },
        {
          run: 'ls -la cypress/integration',
        },
        {
          run: {
            name: `Running cypress tests ${index + 1}`,
            command: `$(npm bin)/cypress run --spec cypress/integration/${value}`,
          },
        },
        {
          store_artifacts: {
            path: 'cypress/videos',
          },
        },
        {
          store_artifacts: {
            path: 'cypress/screenshots',
          },
        },
      ],
    };
    data.workflows.build_and_test.jobs.push({
      [`test${index + 1}`]: {
        requires: [
          'build',
        ],
      },
    });
  }
  return data;
}

function writeFile(data) {
  yaml(path.join(__dirname, '..', '.circleci', 'config.yml'), data, (err) => {
    if (err) {
      console.log(err);
    } else {
      console.log('Success!');
    }
  });
}


/*
  main
*/

// get spec files as an array
const files = fs.readdirSync(path.join(__dirname, '..', 'cypress', 'integration')).filter(fn => fn.endsWith('.spec.js'));
// read circle.json
const circleConfigJSON = require(path.join(__dirname, 'circle.json'));
// add cypress specs to object as test jobs
const data = createJSON(files, circleConfigJSON);
// write file to disc
writeFile(data);

Review (and refactor) this on your own.

Install write-yaml and then generate the new config file:

$ npm install write-yaml --save-dev
$ node lib/generate-circle-config.js

Commit your code again and push it up to GitHub to trigger a new build. Again, four test jobs should run in parallel after the build job finishes.

Mochawesome

Moving along, let's add mochawesome as a Cypress custom reporter so we can generate a nice report after all test jobs finish running.

Install:

$ npm install mochawesome mocha --save-dev

Update the following run step in the createJSON function in generate-circle-config.js:

run: {
  name: `Running cypress tests ${index + 1}`,
  command: `$(npm bin)/cypress run --spec cypress/integration/${value} --reporter mochawesome --reporter-options "reportFilename=test${index + 1}"`,
},

Then, add a new step to store the generated report as an artifact to createJSON:

{
  store_artifacts: {
    path: 'mochawesome-report',
  },
},

createJSON should now look like:

function createJSON(fileArray, data) {
  for (const [index, value] of fileArray.entries()) {
    data.jobs[`test${index + 1}`] = {
      working_directory: '~/tmp',
      docker: [
        {
          image: 'cypress/base:10',
          environment: {
            TERM: 'xterm',
          },
        },
      ],
      steps: [
        {
          attach_workspace: {
            at: '~/',
          },
        },
        {
          run: 'ls -la cypress',
        },
        {
          run: 'ls -la cypress/integration',
        },
        {
          run: {
            name: `Running cypress tests ${index + 1}`,
            command: `$(npm bin)/cypress run --spec cypress/integration/${value} --reporter mochawesome --reporter-options "reportFilename=test${index + 1}"`,
          },
        },
        {
          store_artifacts: {
            path: 'cypress/videos',
          },
        },
        {
          store_artifacts: {
            path: 'cypress/screenshots',
          },
        },
        {
          store_artifacts: {
            path: 'mochawesome-report',
          },
        },
      ],
    };
    data.workflows.build_and_test.jobs.push({
      [`test${index + 1}`]: {
        requires: [
          'build',
        ],
      },
    });
  }
  return data;
}

Now, each test run will generate a mochawesome report with a unique name. Try it out. Generate the new config. Commit and push your code. Each test job should store a copy of the generated mochawesome report in the "Artifacts" tab:

circleci dashboard

The actual report should look something like:

mochawesome report

Combine Reports

The next step is to combine the separate reports into a single report. Start by adding a new step to store the generated report in a workspace to the createJSON function:

{
  persist_to_workspace: {
    root: 'mochawesome-report',
    paths: [
      `test${index + 1}.json`,
      `test${index + 1}.html`,
    ],
  },
},

Also, add a new job to lib/circle.json called combine_reports, which attaches the workspace and then runs an ls command to display the contents of the directory:

"combine_reports": {
  "working_directory": "~/tmp",
  "docker": [
    {
      "image": "cypress/base:10",
      "environment": {
        "TERM": "xterm"
      }
    }
  ],
  "steps": [
    {
      "attach_workspace": {
        "at": "/tmp/mochawesome-report"
      }
    },
    {
      "run": "ls /tmp/mochawesome-report"
    }
  ]
}

The purpose of the ls is to just make sure that we are persisting and attaching the workspace correctly. In other words, when run, you should see all the reports in the "/tmp/mochawesome-report" directory.

Since this job depends on the test jobs, update createJSON again, like so:

function createJSON(fileArray, data) {
  const jobs = [];
  for (const [index, value] of fileArray.entries()) {
    jobs.push(`test${index + 1}`);
    data.jobs[`test${index + 1}`] = {
      working_directory: '~/tmp',
      docker: [
        {
          image: 'cypress/base:10',
          environment: {
            TERM: 'xterm',
          },
        },
      ],
      steps: [
        {
          attach_workspace: {
            at: '~/',
          },
        },
        {
          run: 'ls -la cypress',
        },
        {
          run: 'ls -la cypress/integration',
        },
        {
          run: {
            name: `Running cypress tests ${index + 1}`,
            command: `$(npm bin)/cypress run --spec cypress/integration/${value} --reporter mochawesome --reporter-options "reportFilename=test${index + 1}"`,
          },
        },
        {
          store_artifacts: {
            path: 'cypress/videos',
          },
        },
        {
          store_artifacts: {
            path: 'cypress/screenshots',
          },
        },
        {
          store_artifacts: {
            path: 'mochawesome-report',
          },
        },
        {
          persist_to_workspace: {
            root: 'mochawesome-report',
            paths: [
              `test${index + 1}.json`,
              `test${index + 1}.html`,
            ],
          },
        },
      ],
    };
    data.workflows.build_and_test.jobs.push({
      [`test${index + 1}`]: {
        requires: [
          'build',
        ],
      },
    });
  }
  data.workflows.build_and_test.jobs.push({
    combine_reports: {
      'requires': jobs,
    },
  });
  return data;
}

Generate the config:

$ node lib/generate-circle-config.js

The config file should now look like:

version: 2
jobs:
  build:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - checkout
      - run: pwd
      - run: ls
      - restore_cache:
          keys:
            - 'v2-deps-{{ .Branch }}-{{ checksum "package-lock.json" }}'
            - 'v2-deps-{{ .Branch }}-'
            - v2-deps-
      - run: npm ci
      - save_cache:
          key: 'v2-deps-{{ .Branch }}-{{ checksum "package-lock.json" }}'
          paths:
            - ~/.npm
            - ~/.cache
      - persist_to_workspace:
          root: ~/
          paths:
            - .cache
            - tmp
  combine_reports:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: /tmp/mochawesome-report
      - run: ls /tmp/mochawesome-report
  test1:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 1
          command: >-
            $(npm bin)/cypress run --spec cypress/integration/sample1.spec.js
            --reporter mochawesome --reporter-options "reportFilename=test1"
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots
      - store_artifacts:
          path: mochawesome-report
      - persist_to_workspace:
          root: mochawesome-report
          paths:
            - test1.json
            - test1.html
  test2:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 2
          command: >-
            $(npm bin)/cypress run --spec cypress/integration/sample2.spec.js
            --reporter mochawesome --reporter-options "reportFilename=test2"
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots
      - store_artifacts:
          path: mochawesome-report
      - persist_to_workspace:
          root: mochawesome-report
          paths:
            - test2.json
            - test2.html
  test3:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 3
          command: >-
            $(npm bin)/cypress run --spec cypress/integration/sample3.spec.js
            --reporter mochawesome --reporter-options "reportFilename=test3"
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots
      - store_artifacts:
          path: mochawesome-report
      - persist_to_workspace:
          root: mochawesome-report
          paths:
            - test3.json
            - test3.html
  test4:
    working_directory: ~/tmp
    docker:
      - image: 'cypress/base:10'
        environment:
          TERM: xterm
    steps:
      - attach_workspace:
          at: ~/
      - run: ls -la cypress
      - run: ls -la cypress/integration
      - run:
          name: Running cypress tests 4
          command: >-
            $(npm bin)/cypress run --spec cypress/integration/sample4.spec.js
            --reporter mochawesome --reporter-options "reportFilename=test4"
      - store_artifacts:
          path: cypress/videos
      - store_artifacts:
          path: cypress/screenshots
      - store_artifacts:
          path: mochawesome-report
      - persist_to_workspace:
          root: mochawesome-report
          paths:
            - test4.json
            - test4.html
workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      - test1:
          requires:
            - build
      - test2:
          requires:
            - build
      - test3:
          requires:
            - build
      - test4:
          requires:
            - build
      - combine_reports:
          requires:
            - test1
            - test2
            - test3
            - test4

Commit and push to GitHub again. Make sure combine_reports runs at the end:

circleci dashboard

Next, add a script to combine the reports:

const fs = require('fs');
const path = require('path');

const shell = require('shelljs');
const uuidv1 = require('uuid/v1');


function getFiles(dir, ext, fileList = []) {
  const files = fs.readdirSync(dir);
  files.forEach((file) => {
    const filePath = `${dir}/${file}`;
    if (fs.statSync(filePath).isDirectory()) {
      getFiles(filePath, fileList);
    } else if (path.extname(file) === ext) {
      fileList.push(filePath);
    }
  });
  return fileList;
}

function traverseAndModifyTimedOut(target, deep) {
  if (target['tests'] && target['tests'].length) {
    target['tests'].forEach(test => {
      test.timedOut = false;
    });
  }
  if (target['suites']) {
    target['suites'].forEach(suite => {
      traverseAndModifyTimedOut(suite, deep + 1);
    })
  }
}

function combineMochaAwesomeReports() {
  const reportDir = path.join('/', 'tmp', 'mochawesome-report');
  const reports = getFiles(reportDir, '.json', []);
  const suites = [];
  let totalSuites = 0;
  let totalTests = 0;
  let totalPasses = 0;
  let totalFailures = 0;
  let totalPending = 0;
  let startTime;
  let endTime;
  let totalskipped = 0;
  reports.forEach((report, idx) => {
    const rawdata = fs.readFileSync(report);
    const parsedData = JSON.parse(rawdata);
    if (idx === 0) { startTime = parsedData.stats.start; }
    if (idx === (reports.length - 1)) { endTime = parsedData.stats.end; }
    totalSuites += parseInt(parsedData.stats.suites, 10);
    totalskipped += parseInt(parsedData.stats.skipped, 10);
    totalPasses += parseInt(parsedData.stats.passes, 10);
    totalFailures += parseInt(parsedData.stats.failures, 10);
    totalPending += parseInt(parsedData.stats.pending, 10);
    totalTests += parseInt(parsedData.stats.tests, 10);

    if (parsedData && parsedData.suites && parsedData.suites.suites) {
      parsedData.suites.suites.forEach(suite => {
        suites.push(suite)
      })
    }
  });
  return {
    totalSuites,
    totalTests,
    totalPasses,
    totalFailures,
    totalPending,
    startTime,
    endTime,
    totalskipped,
    suites,
  };
}

function getPercentClass(pct) {
  if (pct <= 50) {
    return 'danger';
  } else if (pct > 50 && pct < 80) {
    return 'warning';
  }
  return 'success';
}

function writeReport(obj, uuid) {
  const sampleFile = path.join(__dirname, 'sample.json');
  const outFile = path.join(__dirname, '..', `${uuid}.json`);
  fs.readFile(sampleFile, 'utf8', (err, data) => {
    if (err) throw err;
    const parsedSampleFile = JSON.parse(data);
    const stats = parsedSampleFile.stats;
    stats.suites = obj.totalSuites;
    stats.tests = obj.totalTests;
    stats.passes = obj.totalPasses;
    stats.failures = obj.totalFailures;
    stats.pending = obj.totalPending;
    stats.start = obj.startTime;
    stats.end = obj.endTime;
    stats.duration =  new Date(obj.endTime) - new Date(obj.startTime);
    stats.testsRegistered = obj.totalTests - obj.totalPending;
    stats.passPercent = Math.round((stats.passes / (stats.tests - stats.pending)) * 1000) / 10;
    stats.pendingPercent = Math.round((stats.pending / stats.testsRegistered) * 1000) /10;
    stats.skipped = obj.totalskipped;
    stats.hasSkipped = obj.totalskipped > 0;
    stats.passPercentClass = getPercentClass(stats.passPercent);
    stats.pendingPercentClass = getPercentClass(stats.pendingPercent);

    obj.suites.forEach(suit => {
      traverseAndModifyTimedOut(suit, 0);
    });

    parsedSampleFile.suites.suites = obj.suites;
    parsedSampleFile.suites.uuid = uuid;
    fs.writeFile(outFile, JSON.stringify(parsedSampleFile), { flag: 'wx' }, (error) => {
      if (error) throw error;
    });
  });
}

const data = combineMochaAwesomeReports();
const uuid = uuidv1();
writeReport(data, uuid);
shell.exec(`./node_modules/.bin/marge ${uuid}.json --reportDir mochareports --reportTitle ${uuid}`, (code, stdout, stderr) => {
  if (stderr) {
    console.log(stderr);
  } else {
    console.log('Success!');
  }
});

Save this as combine.js in "lib".

This script will gather up all the mochawesome JSON files (which contain the raw JSON output for each mochawesome report), combine them, and generate a new mochawesome report.

If interested, hop back to CircleCI to view one of the generated mochawesome JSON files in the "Artifacts" tab from one of the test jobs.

Install the dependencies:

$ npm install shelljs uuid --save-dev

Add sample.json to the "lib" directory:

{
  "stats": {
    "suites": 0,
    "tests": 0,
    "passes": 0,
    "pending": 0,
    "failures": 0,
    "start": "",
    "end": "",
    "duration": 0,
    "testsRegistered": 0,
    "passPercent": 0,
    "pendingPercent": 0,
    "other": 0,
    "hasOther": false,
    "skipped": 0,
    "hasSkipped": false,
    "passPercentClass": "success",
    "pendingPercentClass": "success"
  },
  "suites": {
    "uuid": "",
    "title": "",
    "fullFile": "",
    "file": "",
    "beforeHooks": [],
    "afterHooks": [],
    "tests": [],
    "suites": [],
    "passes": [],
    "failures": [],
    "pending": [],
    "skipped": [],
    "duration": 0,
    "root": true,
    "rootEmpty": true,
    "_timeout": 2000
  },
  "copyrightYear": 2019
}

Update combine_reports in circle.json to run the combine.js script and then save the new reports as an artifact:

"combine_reports": {
  "working_directory": "~/tmp",
  "docker": [
    {
      "image": "cypress/base:10",
      "environment": {
        "TERM": "xterm"
      }
    }
  ],
  "steps": [
    "checkout",
    {
      "attach_workspace": {
        "at": "~/"
      }
    },
    {
      "attach_workspace": {
        "at": "/tmp/mochawesome-report"
      }
    },
    {
      "run": "ls /tmp/mochawesome-report"
    },
    {
      "run": "node ./lib/combine.js"
    },
    {
      "store_artifacts": {
        "path": "mochareports"
      }
    }
  ]
}

To test, generate the new config, commit, and push your code. All jobs should pass and you should see the combined final report.

circleci dashboard

circleci dashboard

mochawesome report

Handle Test Failures

What happens if a test fails?

Change cy.get('a').contains('Blog'); to cy.get('a').contains('Not Real'); in sample2.spec.js:

describe('Cypress parallel run example - 2', () => {
  it('should display the blog link', () => {
    cy.visit(`https://mherman.org`);
    cy.get('a').contains('Not Real');
  });
});

Commit and push your code. Since the combine_reports job is dependent on the test jobs, if any one of those test jobs fail it won't run.

circleci dashboard

circleci dashboard

So, how do you get the combine_reports job to run even if a previous job in the workflow fails?

Unfortunately, this functionality is not currently supported by CircleCI. See this discussion for more info. Because we really only care about the mochawesome JSON report, you can get around this issue by suppressing the exit code for the test jobs. The test jobs will still run and generate the mochawesome report--they will just always pass regardless of whether the underlying tests pass or fail.

Update the following run again in createJSON:

run: {
  name: `Running cypress tests ${index + 1}`,
  command: `if $(npm bin)/cypress run --spec cypress/integration/${value} --reporter mochawesome --reporter-options "reportFilename=test${index + 1}"; then echo 'pass'; else echo 'fail'; fi`,
},

The single line bash if/else is a bit hard to read. Refactor this on your own.

Does it work? Generate the new config file, commit, and push your code. All test jobs should pass and the final mochawesome report should show the failing spec.

circleci dashboard

mochawesome report

One last thing: We should probably still fail the entire build if a job fails. The quickest way to implement this is within the shell.exec callback in combine.js:

shell.exec(`./node_modules/.bin/marge ${uuid}.json --reportDir mochareports --reportTitle ${uuid}`, (code, stdout, stderr) => {
  if (stderr) {
    console.log(stderr);
  } else {
    console.log('Success!');
    if (data.totalFailures > 0) {
      process.exit(1);
    } else {
      process.exit(0);
    }
  }
});

Test this out. Then, try testing a few other scenarios, like skipping a test or adding more than four spec files.

Conclusion

This tutorial looked at how to run Cypress tests in parallel, without using the Cypress record feature, on CircleCI. It's worth noting that you can implement the exact same workflow with any of the CI services that offer parallelism--like GitLab CI, Travis, and Semaphore, to name a few--as well as your own custom CI platform with Jenkins or Concourse. If your CI service does not offer parallelism, then you can use Docker to run jobs in parallel. Contact us for more details on this.

Looking for some challenges?

  1. Create a Slack bot that notifies a channel when the tests are done running and adds a link to the mochawesome report as well as any screenshots or videos of failed test specs
  2. Upload the final report to an S3 bucket (see cypress-mochawesome-s3)
  3. Track the number of failed tests over time by storing the test results in a database
  4. Run the entire test suite multiple times as a nightly job and then only indicate whether or not a test has failed if it fails X number of times--this will help surface flaky tests and eliminate unnecessary developer intervention

Grab the final code from the cypress-parallel repo. Cheers!

Featured Course

Developing RESTful APIs with Django REST Framework

Learn how to build RESTFul APIs with Django and Django REST Framework.

Buy Now $30 View Course
Featured Course

Developing RESTful APIs with Django REST Framework

Learn how to build RESTFul APIs with Django and Django REST Framework.

Buy Now $30 View Course