Working on the code

First, ensure you’ve followed our WSL2 setup instructions. This will prepare you a virtual linux environment on your windows machine that you can use to develop and run the apps, including cloning copies of the required proposals repos.

Visual Studio Code is the recommended editor for working on the proposal apps. To start developing in VS Code, you’ll need to install the “Remote - WSL” extension. This will allow you to interact directly with the code in the linux environment. Once it’s installed, open “Remote Explorer” in the sidebar and connect to your Ubuntu environment.

You will also want to install the ESLint VSCode extension for real-time linting of JavaScript code. With the Remote WSL extension installed, there are separate sections in the extensions sidebar for windows and WSL. Make sure it’s installed in WSL. The Docker extension may also be useful to manage containers using a GUI.

Then simply open the folder where user-office-core is checked out (in /home/<user>/repos).

For a step-by-step guide for creating or editing mutations or queries this a useful guide/checklist written by the ESS.

Please also read and follow the contribution guide also provided by the ESS. It is important to format your commit correctly as they are used to create release notes and the version number.

Building and running the apps

The goal of these steps is to get a version of the Proposals site running on your local machine (so you can tinker with it).

UOP dependencies

The UOP connects to the Authenticate site and User Office Web Service (UOWS) to manage logins and retrieve user details. If you haven’t installed Authenticate yet, see https://github.com/isisbusapps/ISISBusApps/wiki/Proposals-and-Outcomes-Developer-Environment. You should only need to deploy the Authenticate frontend using Visual Studio; the UOWS is now started automatically together with the proposals backend, so you just ned to clone the Users repo in your C:\programming folder. Once deployed, the Authenticate site starts automatically in IIS.

UOP

Firstly, run npm install in the user-office-core repo. This will install the repo's dependencies (which will appear in a node_modules folder).

Both the frontend and backend apps can be run as "thin" Docker containers, where the code from the host machine is mounted into a container which builds and runs it. The container only needs to be built once as it only includes the build tools, not the actual code or build files which remain on the host machine. Both the backend and frontend also support automatic recompilation when a source file is edited, so all you need to do is start the container once and Node will take care of recompiling the necessary parts when you edit and save a file.

To build the container image, run fba-compose build <app name> from any terminal (the integrated terminal in VS Code will work fine), where the app name is duo-backend for the backend or duo-frontend for the frontend. You can then start the container with fba-compose up -d <app name> (the -d flag will start the container in the background).

You will need to generate the proposal frontend SDK before you can run the site for the first time. To do so the ESS proposal backend is needed. First, build the backend with fba-compose build duo-backend, then start it with fba-compose up -d duo-backend. You may want to ensure you’re running the latest version of develop before generating the SDK; to do so, run git pull develop in the core and factory repos.

With the required service now running, you can then run SCHEMA_URL=http://localhost:8081/graphql npm run generate:local in the frontend directory (user-office-core/apps/frontend) to generate the SDK. After the SDK has been generated run fba-compose build duo-frontend and then fba-compose up -d duo-frontend in order to build and run the frontend.

You also need to run the Proposal launchpad every time you start the proposal system. After the frontend and backend are started and a ‘warming up’ period (up to a few minutes), the app should now be up and running. To connect, sign in on http://localhost:9003, and select the link to the new proposal system. If all has gone well, it should load a similar page to: https://devproposal.facilities.rl.ac.uk/.

You may also need to run clam-antivirus antivirus container this can be started by fba-compose up clam-antivirus

If so, congratulations! You’ve got the Proposals site running on your local machine!

If you’ve run into a problem during setup, try the suggestions in the Troubleshooting section.

If you want to check the console output of the program, run fba-compose logs <app name>; you can add the -f flag to "tail" the output (i.e. keep the container output open and display any new messages as they're written). When tailing the output, you can close the logs view with Ctrl+C. This will not stop the app, just disconnect from the container output. If you do want to stop the app, you can so with fba-compose stop <app name>.

Run fba-compose --help or visit the Docker Compose overview page for a full list of commands.

If you're using Visual Studio Code, the process can be automated by creating a new tasks.json file in the .vscode directory of a project. For example, the following snippet will build and start the backend (change duo-backend to duo-frontend to start the frontend instead). Since it's set as the default build task ("isDefault": true), you can run it by pressing Shift+Ctrl+B. See the VSCode docs for more details on how to edit the tasks file.

Starting the backend will automatically start the proposal DB and the reverse proxy in the background, so you will not need to start them manually. Starting the frontend will also automatically start the backend if it's not already started.

Using the app

The app is configured to run as an STFC instance, which means logging in with a username and password via the normal login page will not work, and the app instead needs to authenticate an STFC user with a valid session ID (external authentication).

To login as an STFC user, go to http://localhost:8081/external-auth?token=session-id, where session-id is a valid session ID for a logged in user. When the Proposal Launchpad is not running you can access the system by a valid session ID can be obtained by logging into the Proposal launchpad URL and copying the value after ?guid=. A new session ID may be required after a period of inactivity.

Debugging the backend

The backend is automatically started in debug mode, so all you need to do is attach to the debugger running on port 9229 using your IDE/debugger of choice. If you're using VSCode, you can save some time by copying the following bit of config as launch.json in the .vscode directory of the user-office-core project. You can then start the debugger by pressing F5 (or start the "Attach to Docker container" task from Run tab).

Testing

Generally, each change you make should include relevant frontend and backend tests, following the existing examples in the codebase.

All tests will run on the latest commit on any GitHub PR, and a PR cannot be merged without all tests passing. You can also run the tests locally.

Backend testing (unit and integration)

These tests are fairly quick to run. They include:

• Unit tests (files ending in .spec.ts). Run these with npm run test from the backend directory.

• Integration tests (files ending in .test.ts). Run these with npm run test:integration from the backend directory.

For extra safety, you may want to run the tests in the same Docker container that is building and running the app; this will ensure that the tests run on the same environment as dev and prod. To do so, run fba-compose exec duo-backend npm run test. Running the integration tests within the container may not work due to database connection issues. It is fine to run the integration tests outside of the container.

Frontend testing (E2E)

These tests are time consuming to run. Unless you have implemented a large feature or refactored at a large scale, it's generally not necessary to run the entire test suite locally. Instead, selectively run new tests you have written or tests that you have changed. You may also find it useful to debug persistent GitHub test failures locally.

Each test can run in either the ESS environment or the STFC environment, or both.

Running the frontend tests locally

Running the frontend tests in the ESS environment

To bring up duo-backend in test confguration simply run: fba-compose -f /mnt/c/programming/docker-orchestration/docker-compose.e2e.yml up -d duo-frontend, this uses docker-compose.e2e.yml to override some config values and start the backend in test mode.

You can now run the E2E tests with fba-compose run cypress-e2e.

Running the frontend tests in the STFC environment

All features of the app are enabled in this environment. The app will use external authentication provided by OAuth instead of the STFC User Office Web Service.

Most tests will run in this environment, except those explicitly marked as not being suitable for OAuth:

To run in this environment, you must change your backend configuration:

1. In cypress.config.ts change the value of DEV_AUTH_SERVER_URL to 'http://cypress-dev-auth-server:5000'. Also change baseUrl to 'http://localhost:8081'.

2. Rebuild the e2e image so that the updated config is copied into it: fba-compose build cypress-e2e.

3. Save the file and restart everything in stfc test mode: fba-compose down ; fba-compose -f /mnt/c/programming/docker-orchestration/docker-compose.e2e.stfc.yml up -d duo-frontend(note: docker-compose.e2e.stfc.yml is a different overide files with config for connecting to our mockserver)

4. Run the E2E tests with fba-compose run cypress-e2e.

Troubleshooting running STFC environment tests

• When you pull from the mock-uows, you might need to build the container with fba-compose build mockserver-client before running the E2E tests.

• Ensure you have the mock-uows repo cloned in WSL to your repos dir, and that the user-office-core, mock-uows and docker-orchestration repos are up to date.

• Some developers have reported the following error with starting the mockserver-client container: Service ‘mockserver-client’ failed to build. If this happens, open this file locally and change node to root. Then build the container again with fba-compose build mockserver-client and run the E2E tests again.

• If the MockServer client has started successfully you should see Created callback for GET request in the logs of the mockserver-client container.

• You may need to disconnect from the VPN if the Mockserver client container exits because it can’t download a certificate.

• Sometimes the MockServer server is not ready in time which causes the client setup to fail. If this happens, remove both mockserver and mockserver-client containers, then manually start them with fba-compose up -d mockServer followed by fba-compose up -d mockserver-client after 5 seconds.

Selectively running frontend tests

To only run tests in a single file, you can modify the command for the cypress-e2e service in the docker-compose.yml file by adding the --spec flag followed by the relative file path of the file you want to run. For example, --spec cypress/e2e/proposals.cy.ts.

To run or skip a single test (or block of tests), add the Cypress .only() or .skip() function. For example, it.only('Test name', () => {, or describe.skip('Test block', () => {. You may wish to use this in conjunction with running a single file.

To only run a subset of the tests you can also use the search function. (??)

Debugging frontend test failures

In general, E2E tests are not completely reliable and can sometimes initially fail but pass with retries.

Consider whether your changes may be the cause of a test failure, in which case the test may need updating. It may not be immediately obvious - for example, if you have changed the way a proposal is created, it may impact a test that creates a visit, where creating a proposal is a prerequisite.

A test will automatically fail if the frontend does not give a response within 15 seconds. Test failures are automatically retried once.

GitHub test failures

To see which tests have failed and why

Click "Details" next to the failed job in the PR. Each job run consists of multiple steps - usually setup steps, test execution, and teardown steps. Clicking "Details" will automatically take you to the end of the step that has failed, typically this will be "Run e2e tests stfc" or "Run e2e tests". Within either of these steps, you will be able to see a summary of which test files have run as part of this step, and how many tests have failed in each file. Scroll upwards to see error messages for individual tests.

Example screenshots

Open

The list of successful and failed jobs on a PR

Open

The failing “Run e2e tests” step showing which test files have failed

Open

An individual test failure error message

 

General advice

• Try re-running the tests. Click on one of the failing jobs, then at the top of the page, click "Re-run jobs" and then choose "Re-run failed jobs" from the dropdown. You may need to re-run jobs multiple times to get everything passing. Keep an eye on whether the same tests are persistently failing or if it varies with each run.

• Have a look at the Cypress screenshots by finding your job run in the list of test failures. Click on the title, then on the "Summary" page you can download the cypress-screenshots artifact.

• If tests persistently fail, you may be able to get more insight by running them locally.

Local test failures

• Have a look at the screenshots in the user-office-core/apps/e2e/cypress/screenshots directory. This directory is populated as tests fail, and the directory is wiped each time you start up a new test run.

• Use cy.log() to aid in debugging areas of tests. Note that the logs will not show up in the console, they will only be visible in screenshots or the GUI.

• Switch back to develop and try running the tests. If tests start passing, it can indicate that they're failing due to your changes.

• Replicate what the tests are doing in your browser.

  • For tests that run in the E2E dependency config (OAuth authentication), you will need to look at which user is being logged in for each test (e.g. cy.login('officer')) and map it to the user in the database test data. Take the email address of the user, and use it in the URL to login, e.g. http://localhost:8081/external-auth?token=Javon4@hotmail.com.

• Run the Cypress GUI. It may be easier to see what is happening by watching the GUI.

Running the Cypress GUI

The Cypress tests usually run in a headless Chrome window (i.e. the browser is hidden), but you can optionally launch the Cypress GUI. This may be useful in debugging tests, as you can see the browser being automated as it happens.

1. Ensure the user-office-core is cloned into windows 'C:\Programming\user-office-core'.

2. In docker-compose.yaml forcypress-dev-auth-server, add:
   ports:
- 5700:5000

3. Run fba-compose -f /mnt/c/programming/docker-orchestration/docker-compose.e2e.yml up -d duo-frontend

4. In ‘apps\e2e\cypress.config.ts’ update the DEV_AUTH_SERVER_URL to 'http://localhost:5700' and baseUrl to 'http://localhost:8081'

5. Replace the SDK in the frontend with the frontend in the linux environment (from'\\wsl.localhost\Ubuntu-20.04\home\<user_name>\repos\user-office-core\apps\frontend\src\generated' to 'C:\Programming\user-office-core\apps\frontend\src\generated').

6. Run npm install in user-office-core.

7. Remove and restart the backend container. (fba-compose rm duo-backend and fba-compose -f /mnt/c/programming/docker-orchestration/docker-compose.e2e.yml up -d duo-frontend).

8. Make sure Cypress is installed in Windows, using npm install cypress --save-dev.

9. Run npm run cy:open within e2e which will prompt the Chrome window to run the tests.

From the GUI you can select which specs to run and in which browser. Attempting to run all specs at the same time is not recommended due to the slow speed they are loaded into the Cypress GUI.

Writing STFC environment frontend tests

The e2e tests run based on features enabled. Therefore we need to get the features enabled before the tests run, the getAndStoreFeaturesEnabled() method does that.

• When creating a new file, in the beforeEach you will need to call the cy.getAndStoreFeaturesEnabled(); method. Example can be found here.

• If the entire file consists of tests that test a feature which is disabled in some environments, then create another beforeEach function underneath the one created with the cy.getAndStoreFeaturesEnabled(); method. Then create a condition that checks if the feature flag is present, if not, then skip. See an example here.

• If adding tests to existing files or writing new files where only some tests involve testing features behind feature flags, then within you can add the skip test condition inside the test. See an example here.

• If a part of the test involves a feature flag but the tests itself doesn't test that feature, you can add a condition around the code that involves the feature flag to filter out if the feature is disabled. See an example here.

• Any relevant container config changes to the docker-compose.yml file in docker-orchestration will need to be reflected in the docker-compose-stfc.e2e.yml file in the frontend. For example, an addition of a new environment variable.

Testing the full build of the apps

When running locally, the proposal apps are normally run in a simplified container which mounts the code and resources from your local drive. The build process also skips many compiling and optimisation steps to make the apps faster to start and easier to work with. When running on dev and prod, the full build process is used. This makes the resulting apps faster to use and produces a full container image ready for deployment, but takes significantly longer to run.

In some cases, you may want to test the full build locally, for example to test changes to the frontend Nginx config which isn’t normally used locally. To do so:

1. Open the docker-compose.yml file in the docker-orchestration repo

2. Find the service definition for the app you want to test (e.g. duo-frontend)

3. In the build section, change the dockerfile from Dockerfile.dev to just Dockerfile

4. Rebuild the app and restart it: fba-compose build <app name> && fba-compose up -d <app name>

Once you're done, follow these steps in reverse to go back to the usual local setup.

Using Postgres

1. Go here. Download

2. open pgADmin and if it asks for a port you should put any port except 5432 e.g. 5430.

3. Once you finish the setup it should take you to a website and you can press "Add new server".

4. Go to connection and put the host as 127.0.0.1, port to 5432, name to duouser and password to duopassword.

5. To access database tables go to Databases > Duo > Catalogs > Tables or Databases > Duo > Schemas > Public > Tables (These are the main ones that I have used).

6. When viewing the rows right click the table and press "View/edit data" then you can choose how many rows to edit.

Use database patches in the backend to add to the database. To run database patches you will have to restart the container as I don't think you can use psql without postgres as this will only give you pgAdmin.

Troubleshooting

General advice

• Ensure both repos are up to date and on the same branch.

• Ensure you have run npm install within both repos.

NOTE: Although npm ci is meant to re-install node_modules, we have encountered errors that were not resolved by running npm ci but were resolved by deleting the node_modules folder(s) and then running npm install. This is a good thing to try as a troubleshooting step. Once you’ve done that try starting the setup process from the start.

Errors

Backend fails to start with npm ERR! Maximum call stack size exceeded error

The backend container may log this error while still running, preventing the backend from actually starting up. To fix this:

1. Rename the backend/node_modules folder. This is a lot quicker than deleting it in Windows.

2. Run npm cache clean --f to clean the npm cache.

3. Run npm install.

This may also happen if the package-lock.json has been recreated in a way that's not compatible with Node 12. This is likely the case if you also see a warning message about incompatible lockfile versions. Try reverting the package-lock.json to the latest version from Git, then running the steps above to fix the error.

Backend fails to start with npm ERR! enoent ENOENT: no such file or directory

This may happen because necessary node modules aren't installed.
Try running npm install in the repo of the app you are trying to run, then run it again.

Error relating to JavaScript heap out of memory

Running the backend/frontend or tests fails with an error similar to:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Try restarting your computer.

'npm WARN EBADENGINE Unsupported engine' when running npm ci or npm install

This appears when the wrong version of Node.js is installed. Ensure you have Node.js version 12 installed.

I only have access to the User role

If you don't have access to other roles after logging in with your session ID, ensure your Facilities account is linked with your federal ID in the relevant database. Your federal ID will then need to be in the CLF_ISIS_UO_DEVELOPERS Active Directory group. This will put you in the BusApps User Officer role, which will then give you the ESS User Officer Role.