Analytics : Metabase

Metabase is an open source business intelligence tool. It lets you ask questions about your data, and displays answers in formats that make sense, whether that’s a bar graph or a detailed table.

Your questions can be saved for later, making it easy to come back to them, or you can group questions into great looking dashboards. Metabase also makes it easy to share questions and dashboards with the rest of your team.

Install Prerequisites

These are the set of tools which are required in order to complete any build of the Metabase code. Follow the links to download and install them on your own before continuing.

1. Oracle JDK 8 (http://www.oracle.com/technetwork/java/javase/downloads/index.html)

2. Node.js (http://nodejs.org/)

3. Yarn package manager for Node.js

4. Leiningen (http://leiningen.org/)

If you are developing on Windows, make sure to use Ubuntu on Windows and follow instructions for Ubuntu/Linux instead of installing ordinary Windows versions.

Build Metabase

The entire Metabase application is compiled and assembled into a single .jar file which can run on any modern JVM. There is a script which will execute all steps in the process and output the final artifact for you.

./bin/build

After running the build script simply look in target/uberjar for the output .jar file and you are ready to go.

Building Metabase.app

See this guide.

Build Dashboard

So, you’ve gotten Metabase up and running and connected it to your data. It’s time to give you the lay of the land.

The home page

Fresh out of the box, Metabase will show you a few things on the home page:

• Some automatic explorations of your tables that you can look at and save as a dashboard if you like any of them.

• An area where things you or your teammates create will show up, along with a link to see all the dashboards, questions, and pulses you have.

• A list of the databases you’ve connected to Metabase.

Once you’ve created some dashboards, any of them that you pin in the main “Our analytics” collection will show up on the homepage for all of your teammates, so that when they log in to Metabase they’ll know right where to go.

Browse your data

If you connected your database to Metabase during setup, you’ll see it listed at the bottom of the homepage along with the sample dataset that Metabase comes with. Click on a database to see its contents. You can click on a table to see its rows, or you can also click on the bolt icon to x-ray a table and see an automatic exploration of it, or click on the book icon to go to the data reference view for that table to learn more about it.

Explore your analytics

As you and your team create dashboards and collections, they’ll start to show up on the homepage. Click on a collection in the “Our analytics” section to see its contents, or click “browse all items” to see everything you and your team have made. More about exploring

Ask a question or write a query

Click the Ask a question button in the top-right of Metabase to start a new simple exploration of one of your tables, ask a more detailed custom question using the notebook editor, or write a new SQL query if you want to really dig in.

Make a new dashboard or pulse

In Metabase, dashboards are made up of saved questions that you can arrange and resize as you please. They’re a great way to track important metrics and stats that you care about. Pulses are what regularly scheduled reports are called in Metabase. They can be sent out either via email, Slack, or both.

To make a dashboard or pulse, click the plus (+) icon in the top-right of the main navigation bar.

Use search to quickly find things

The search bar that’s always present at the top of the screen lets you search through your tables, dashboards, collections, saved questions, metrics, segments, and pulses in an instant. Just type part of the title of the thing you’re looking for and hit enter. You can activate the search bar from anywhere by pressing the / key.

Development Environment

If you plan to work on the Metabase code and make changes then you’ll need to understand a few more things.

Overview

The Metabase application has two basic compnents:

1. a backend written in Clojure which contains a REST API as well as all the relevant code for talking to databases and processing queries.

2. a frontend written as a Javascript single-page application which provides the web UI.

Both components are built and assembled together into a single jar file which runs the entire application.

3rd party dependencies

Metabase depends on lots of other 3rd party libraries to run, so as you are developing you’ll need to keep those up to date. Leiningen will automatically fetch Clojure dependencies when needed, but for JavaScript dependencies you’ll need to kick off the installation process manually when needed.

# javascript dependencies
$ yarn

Development server (quick start)

Run your backend development server with

lein ring server

Start the frontend build process with

yarn build-hot

Frontend development

We use these technologies for our FE build process to allow us to use modules, es6 syntax, and css variables.

• webpack

• babel

• cssnext

Frontend tasks are executed using yarn. All available tasks can be found in package.json under scripts.

To build the frontend client without watching for changes, you can use:

$ yarn build

If you’re working on the frontend directly, you’ll most likely want to reload changes on save, and in the case of React components, do so while maintaining state. To start a build with hot reloading, use:

$ yarn build-hot

Note that at this time if you change CSS variables, those changes will only be picked up when a build is restarted.

There is also an option to reload changes on save without hot reloading if you prefer that.

$ yarn build-watch

Some systems may have trouble detecting changes to frontend files. You can enable filesystem polling by uncommenting the watchOptions clause in webpack.config.js. If you do this it may be worth making git ignore changes to webpack config, using git update-index --assume-unchanged webpack.config.js

Frontend testing

All frontend tests are located in frontend/test directory. Run all frontend tests with

yarn test

which will run unit, end-to-end, and legacy Karma browser tests in sequence.

End-to-end tests

End-to-end tests simulate realistic sequences of user interactions. They render a complete DOM tree using Enzyme and use temporary backend instances for executing API calls.

End-to-end tests use an enforced file naming convention <test-suite-name>.e2e.spec.js to separate them from unit tests.

Useful commands:

lein run refresh-integration-test-db-metadata # Scan the sample dataset and re-run sync/classification/field values caching
yarn test-e2e-watch # Watches for file changes and runs the tests that have changed
yarn test-e2e-watch TestFileName # Watches the files in paths that match the given (regex) string

The way integration tests are written is a little unconventional so here is an example that hopefully helps in getting up to speed:

import {
    useSharedAdminLogin,
    createTestStore,
} from "__support__/e2e";
import {
    click
} from "__support__/enzyme"

import { mount } from "enzyme"

import { FETCH_DATABASES } from "metabase/redux/metadata";
import { INITIALIZE_QB } from "metabase/query_builder/actions";
import RunButton from "metabase/query_builder/components/RunButton";

describe("Query builder", () => {
    beforeAll(async () => {
        // Usually you want to test stuff where user is already logged in
        // so it is convenient to login before any test case.
        useSharedAdminLogin()
    })

    it("should let you run a new query", async () => {
        // Create a superpowered Redux store.
        // Remember `await` here!
        const store = await createTestStore()

        // Go to a desired path in the app. This is safest to do before mounting the app.
        store.pushPath('/question')

        // Get React container for the whole app and mount it using Enzyme
        const app = mount(store.getAppContainer())

        // Usually you want to wait until the page has completely loaded, and our way to do that is to
        // wait until the completion of specified Redux actions. `waitForActions` is also useful for verifying that
        // specific operations are properly executed after user interactions.
        // Remember `await` here!
        await store.waitForActions([FETCH_DATABASES, INITIALIZE_QB])

        // You can use `enzymeWrapper.debug()` to see what is the state of DOM tree at the moment
        console.log(app.debug())

        // You can use `testStore.debug()` method to see which Redux actions have been dispatched so far.
        // Note that as opposed to Enzyme's debugging method, you don't need to wrap the call to `console.log()`.
        store.debug();

        // For simulating user interactions like clicks and input events you should use methods defined
        // in `enzyme.js` as they abstract away some React/Redux complexities.
        click(app.find(RunButton))

        // Note: In pretty rare cases where rendering the whole app is problematic or slow, you can just render a single
        // React container instead with `testStore.connectContainer(container)`. In that case you are not able
        // to click links that lead to other router paths.
    });
})

You can also skim through __support__/e2e.js and __support__/enzyme.js to see all available methods.

Jest unit tests

Unit tests are focused around isolated parts of business logic.

Unit tests use an enforced file naming convention <test-suite-name>.unit.spec.js to separate them from end-to-end and integration tests.

yarn test-unit # Run all tests at once
yarn test-unit-watch # Watch for file changes

Karma browser tests

If you need to test code which uses browser APIs that are only available in real browsers, you can add a Karma test to frontend/test/legacy-karma directory.

yarn test-karma # Run all tests once
yarn test-karma-watch # Watch for file changes

Backend development

Leiningen and your REPL are the main development tools for the backend. There are some directions below on how to setup your REPL for easier development.

And of course your Jetty development server is available via

lein run

To automatically load backend namespaces when files are changed, you can instead run with

lein ring server

lein ring server takes significantly longer to launch than lein run, so if you aren’t working on backend code we’d recommend sticking to launching with lein run.

Building drivers

Most of the drivers Metabase uses to connect to external data warehouse databases are separate Leiningen projects under the modules/ subdirectory. When running Metabase via lein, you’ll need to build these drivers in order to have access to them. You can build drivers as follows:

# Build the 'mongo' driver
./bin/build-driver.sh mongo

(or)

# Build all drivers
./bin/build-drivers.sh

The first time you build a driver, it will be a bit slow, because Metabase needs to build the core project a couple of times so the driver can use it as a dependency; you can take comfort in the fact that you won’t need to build the driver again after that. Alternatively, running Metabase 1.0+ from the uberjar will unpack all of the pre-built drivers into your plugins directory; you can do this instead if you already have a Metabase uberjar (just make sure plugins is in the root directory of the Metabase source, i.e. the same directory as project.clj).

Including driver source paths for development or other Leiningen tasks

For development when running various Leiningen tasks you can add the include-all-drivers profile to merge the drivers’ dependencies and source paths into the Metabase project:

# Install dependencies
lein with-profiles +include-all-drivers deps

This profile is added by default when running lein repl, tests, and linters.

Unit Tests / Linting

Run unit tests with

lein test

or a specific test with

lein test metabase.api.session-test

By default, the tests only run against the h2 driver. You can specify which drivers to run tests against with the env var DRIVERS:

DRIVERS=h2,postgres,mysql,mongo lein test

Some drivers require additional environment variables when testing since they are impossible to run locally (such as Redshift and Bigquery). The tests will fail on launch and let you know what parameters to supply if needed.

Run the linters:

lein eastwood && lein bikeshed && lein docstring-checker && lein check-namespace-decls && ./bin/reflection-linter

Developing with Emacs

.dir-locals.el contains some Emacs Lisp that tells clojure-mode how to indent Metabase macros and which arguments are docstrings. Whenever this file is updated, Emacs will ask you if the code is safe to load. You can answer ! to save it as safe.

By default, Emacs will insert this code as a customization at the bottom of your init.el. You’ll probably want to tell Emacs to store customizations in a different file. Add the following to your init.el:

(setq custom-file (concat user-emacs-directory ".custom.el")) ; tell Customize to save customizations to ~/.emacs.d/.custom.el
(ignore-errors                                                ; load customizations from ~/.emacs.d/.custom.el
  (load-file custom-file))

Documentation

Instant Cheatsheet

Start up an instant cheatsheet for the project + dependencies by running

lein instant-cheatsheet

Internationalization

We are an application with lots of users all over the world. To help them use Metabase in their own language, we mark all of our strings as i18n.

Adding new strings:

If you need to add new strings (try to be judicious about adding copy) do the following:

1. Tag strings in the frontend using t and jt ES6 template literals (see more details in https://ttag.js.org/):

const someString = t`Hello ${name}!`;
const someJSX = <div>{jt`Hello ${name}`}</div>;

and in the backend using trs and related macros (see more details in https://github.com/puppetlabs/clj-i18n):

(trs "Hello {0}!" name)

Translation errors or missing strings

If you see incorrect or missing strings for your langauge, please visit our POEditor project and submit your fixes there.