Unable To Access Deployed Demo Applications

I've deployed the demo admin and application storefronts to our cloud/non-local environment. The applications seem to have started without errors, but I am still unable to access it via a browser.

Overview

Broadleaf supports the idea of Tenancy and Applications. Tenant represents a distinct, independent operator of the Broadleaf services. Tenants share the same infrastructure (e.g., services) but do not share data (e.g., catalogs, products, assets). Thus, a single instance of Broadleaf could be shared by multiple companies (i.e., Tenants) without allowing their data to be mixed together. You can configure a URL to the back-office admin to manage data for that tenant.

Beneath a  Tenant reside its "Applications". A single Tenant can maintain multiple applications. An Application represents a storefront or commerce-facing frontend. All Applications have an identifier that defines part or all of a URL they can be resolved at. When resolution occurs, an Application can be identified by the domain, domain prefix, or a query parameter in the request’s URL.

For example:

• Domain: www.my-application.com
• Domain Prefix: my-application.my-company.com
• Query Parameter: www.my-company.com?application=my-application

Background

If you are having problems viewing the admin or storefront, here are a few troubleshooting tips and guides to verify your setup and installation.

1. Check that the "URL Resolution" data in the database is configured appropriately for your setup.

2. Based on the Flex Package you're deploying, verify that the Installation process has inserted the data completely and in the correct order.

IMPORTANT: In order for the demo admin and storefronts to be resolvable through a browser, there is a level of "data consistency" that need to be held across the schemas in the database for the various microservices.

Troubleshooting Step 1: Verify URL Resolution Data

Verify Tenant Data

Verify that the Tenant URL Data is as expected in the Tenant Microservice. Check the IDENTIFIER_VALUE and IDENTIFIER_TYPE

VERIFY: select * from tenant.blc_tenant

By default (if you have not created a Spring Profile/Liquibase changelog to explicitly update the seed data), the demo seed data comes pre-configured for things to work on a local setup out-of-box (i.e. everything running on a single host/local machine). For example:

As an example, if you were trying to deploy this to another machine or a Kubernetes cluster on the cloud, you will either want a Top Level Domain that points to the IP of an External Ingress that routes to the Admin Gateway or the IP Address of the host running the Admin Gateway. 

An example topology map can be found on this guide and tutorial: https://developer.broadleafcommerce.com/tutorials/devops/deploy-to-cloud#balanced_flex_package

You would then create a liquibase changelog that updates the seed data, or you can go directly into the database and change these values.
For example, it may look something like:

You need to also make sure that any Environment Variables being configured for the Tenant Service reflect any of the changes above. In particular, the properties documented here: https://developer.broadleafcommerce.com/services/tenant-services/configuration#url_resolver_properties. For example, the default looks like this:

broadleaf:
  tenant:
    urlresolver:
      admin:
        scheme: https
        domain: localhost
        port: 443

As an example, if you are trying to deploy this to a K8 cluster, you may want to pass in some override ENV properties like the following:

BROADLEAF_TENANT_URLRESOLVER_ADMIN_DOMAIN: admin.cloudexample.com (or something like 12.345.67.890)
BROADLEAF_TENANT_URLRESOLVER_ADMIN_PORT: 443 (or 8446 if you're going directly to the OOB gateway)
BROADLEAF_TENANT_URLRESOLVER_ADMIN_SCHEME: https

Next, you will want to verify that the AUTH Tenant Data for admin has the correct Redirect URLs.

VERIFY: select * from auth.blc_client_redirect_uris

ID	REDIRECT_URI
admin	https://admin.cloudexample.com
admin	https://admin.cloudexample.com/callback
admin	https://admin.cloudexample.com/silent-callback.html

VERIFY: select * from auth.blc_client

ID	DEFAULT_REDIRECT_URI
admin	https://admin.cloudexample.com

Verify Application Data

Verify that the Application URL Data is as expected in the Tenant Microservice. Check the IDENTIFIER_VALUE and IDENTIFIER_TYPE

VERIFY: select * from tenant.blc_tenant_application

By default (if you have not created a Spring Profile/Liquibase changelog to explicitly update the seed data), the demo seed data comes pre-configured for things to work on a local setup out-of-box (i.e. everything running on a single host/local machine). For example:

As an example, you may want to change this to be something like (depending on your setup):

Important Note: Application Data needs to be in sync across the different microservices. If you make a change to an application directly in the Database, it will NOT automatically propagate to the other schemas. We recommend any changes to applications be done directly in the Admin. So it is worth trying to get the Admin (Tenant) up and resolvable first.

If you make changes in the Admin, these changes will automatically propagate to all the necessary microservices.

For example, you can verify in the DB the data in the other schemas:

VERIFY: select * from adminuser.blc_user_application 
VERIFY: select * from asset.blc_application 
VERIFY: select * from auth.blc_application 
VERIFY: select * from campaign.blc_application 
VERIFY: select * from catalog.blc_application 

etc...

Troubleshooting Step 2: Verify Flex Package Installation Process

If you are deploying to a cloud environment and using the default demo seed data, you should be aware of a particular "sequence of events" that need to happen when you first deploy the services to another environment. With the way that the default "seed data initialization" is structured, it's important that certain services start (and initialize data) in a particular order - this becomes even more relevant if you try to deploy the services in a "granular" composition for the first time, as certain services need to start first as specific data will need to get inserted first (which may impact/insert data in other schemas)

We highly recommend reading this "Deploying to the Cloud" tutorial as well. It describes various other assumptions around the installation process when first deploying the stack to another environment: 

The following describes the order in which services should start (and why) when you're doing an initial installation of the full Broadleaf stack:

As an example, the following guides assumes that you will want to access the admin at "admin.cloudexample.com" and the storefronts at "heatclinic.cloudexample.com" and "aaahotsauces.cloudexample.com" - you'll want to configure your ENV with your appropriate Top Level Domain.

Verify Messaging Broker

1. First, before any of Broadleaf's core services (Flex Package Services) get deployed to your cluster, make sure you have the messaging broker set up (e.g. Kafka) and running, making sure it's ready to receive messages.

Install Auth

2. In general, it's best to install the AUTH service first. 
Before the first installation, make sure you have the following ENV properties set

BROADLEAF_AUTH_DATA_LOAD_BASEADMINHOST: admin.cloudexample.com
BROADLEAF_AUTH_DATA_LOAD_BASEAPPLICATIONHOST: cloudexample.com

• There are some "data initializers" inside the auth service that are responsible for pre-loading some initial permissions/roles/clients/redirect uri's into the auth schema (but not all of them just yet). The process will use these ENV properties above to configure the correct data in the DB on an initial installation. (This is why we recommend you have a Top Level Domain ready before you start the initial installation process).
• Important: if you started the auth service with a different admin host (or used the default) and then later change the ENV properties - once it has loaded the initial seed data, it will not update the data that has changed (as this is meant to be a one time initial load step). Any other updates after the first load will need to happen directly in the DB.

Verification Steps:
After the auth service has finished starting, you should be able to go into the database and verify the following: 

VERIFY: select * from auth.blc_client_redirect_uris

ID	REDIRECT_URI
admin	https://admin.cloudexample.com
admin	https://admin.cloudexample.com/callback
admin	https://admin.cloudexample.com/silent-callback.html

VERIFY: select * from auth.blc_client

ID	DEFAULT_REDIRECT_URI
admin	https://admin.cloudexample.com

Wait for AUTH to completely start before moving to the Flex Package service deployments.

Install the Flex Packages

If you are installing the Balanced Flex Package Composition:

3. Install the "Supporting" Flex Package first. Make sure you start up initially with the following ENV properties:

• SPRING_PROFILES_ACTIVE: default,cloudexample(or whatever spring profile has the correct SQL data for your environment)
  BROADLEAF_TENANT_URLRESOLVER_APPLICATION_SCHEME: https
  BROADLEAF_TENANT_URLRESOLVER_APPLICATION_DOMAIN: cloudexample.com
  BROADLEAF_TENANT_URLRESOLVER_APPLICATION_PORT: 443
  BROADLEAF_TENANT_URLRESOLVER_ADMIN_SCHEME: https
  BROADLEAF_TENANT_URLRESOLVER_ADMIN_DOMAIN: admin.cloudexample.com
  BROADLEAF_TENANT_URLRESOLVER_ADMIN_PORT: 443
  	

4. Install the "Browse", "Cart", and "Processing" Flex Packages next.

After Supporting has finished starting, you can verify data in "adminuser" and "auth"

VERIFY: select * from adminuser.blc_admin_user
VERIFY: select * from auth.blc_user

you should see the Global Admin user (master@test.com) in both

If you are installing the Granular Flex Package Composition:

3. Install Catalog, Cart, etc.... (everything else that you wish to deploy except for Admin User and Tenant as we will save this for the last step).

4. Install Admin User and wait for it to completely start up. During the startup process, several Admin Users will be inserted into the adminuser schema. Because of this new data, persistence messages will be published to the message broker and the "auth" service will listen to those events and update a few records in the auth schema.

after adminuser has finished starting, verify data in "adminuser" and "auth"

VERIFY: select * from adminuser.blc_admin_user 
VERIFY: select * from auth.blc_user 

you should see the Global Admin user (master@test.com) in both

5. Finally, install Tenant last and wait for it to completely start up. Make sure you start up initially with the following ENV properties:

• SPRING_PROFILES_ACTIVE: default,cloudexample(or whatever spring profile has the correct SQL data for your environment)
  BROADLEAF_TENANT_URLRESOLVER_APPLICATION_SCHEME: https
  BROADLEAF_TENANT_URLRESOLVER_APPLICATION_DOMAIN: cloudexample.com
  BROADLEAF_TENANT_URLRESOLVER_APPLICATION_PORT: 443
  BROADLEAF_TENANT_URLRESOLVER_ADMIN_SCHEME: https
  BROADLEAF_TENANT_URLRESOLVER_ADMIN_DOMAIN: admin.cloudexample.com
  BROADLEAF_TENANT_URLRESOLVER_ADMIN_PORT: 443
  	

Verify the Admin and Storefront URL's in the Database

After all the Flex Package services have finished starting, you should be able to go into the database and verify the following:

VERIFY: select * from tenant.blc_tenant >> this should have DOMAIN | admin.cloudexample.com
VERIFY: select * from tenant.blc_tenant_application >> this should have DOMAIN_PREFIX | heatclinic, etc....

Depending on Flex Package composition, new data can be inserted within various microservices - which will create persistence messages that are published to the message broker. Many of the Broadleaf services will listen to these events and update the appropriate tables.

The tables that are of particular interest are the " application" tables (which control multi-tenancy/multi-application data) in many of the schemas.
If all the services have started up correctly, all the other schemas should also have the same data in tenant.blc_tenant_application propagated to it.

For example, you can verify in the DB the data in the other schemas:

VERIFY: select * from adminuser.blc_user_application
VERIFY: select * from asset.blc_application
VERIFY: select * from auth.blc_application
VERIFY: select * from campaign.blc_application
VERIFY: select * from catalog.blc_application

etc...

NOTE: You will also want to verify the redirect urls in auth match what you intend:

VERIFY: select * from auth.blc_client_redirect_uris

Further Reading:

https://developer.broadleafcommerce.com/shared-concepts/multi-tenancy

https://developer.broadleafcommerce.com/tutorials/frontend-walkthroughs/tenant-resolution

https://developer.broadleafcommerce.com/services/tenant-services