A Business Intelligence Developer Guide
Nowadays, Business Intelligence (BI) is becoming a fundamental piece of the management process of a company. BI supports and, in most of the cases, possibly improves decision-making process. This can be done supplying an organization with right data, right timing, and right form of presenting the data in the timing. To be clear, BI doesn’t make decisions in an organization, but support them. This can have a highly impact on the success of an organization via decision-making process. So, it is important to consider a set of characteristics and conditions to develop BI in a right way.
The term of Business Intelligence was first adopted in 1958 by Hans Peter Luhn in article written for an IBM publication. He defined BI as “the ability to apprehend the interrelationships of presented facts in such a way as to guide action toward a desired goal”. So, since the beginning of the term usage, BI was considered as a part of supporting and guiding decisions inside an organization to promote a better position with what we know in facts. Based on that, we can precise the definition of Business Intelligence by Boris Evelson of Forrester Research:
“Business intelligence is a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision making.”
Based on the definition of BI, different bibliography, and my experience I considerate BI process as a connection between two items to data-driven decision making:
· Data: Data Sources
· Facts: Reporting Environment & BI Tools
For the connection of this items, I will share a series of step we need to follow to develop BI in a better way.
BI Process
Business Intelligence workflow is composed by 4 main stages:
· Data Sources: we need to identify sources, which data we’ll need and how we’ll need it.
· Data Cleansing: How is going to be done the ETL Process, which tools are going to be used, which data we need and how is going to be integrated as a part of a whole.
· Data Integration: How is structured the data, what are the departments needs and who can access to the data, if there is a data governance policy and at what levels do we need the data.
· BI Results: how do we need the information as a report, dashboard, visuals, etc. In which environment is going to be the consumption of data.
Based on what we have described, in the next image we can see a simple and basic architecture of the BI workflow process we need to follow to develop BI.
Guide steps to develop BI
Having in mind this workflow and based on my experience, I want to share a series of steps and concepts considerations that I founded useful and necessary to do BI the right way:
Step 1: Identifying KPI, Objectives and requirements
We need to have an aligned vision about BI in the organization before starting this step. The first big step is to define what problem or set of problems there are going to be solve as an organization and/or as department with the help of BI. Setting up the objectives will help to determine high levels of BI answering these questions:
o What sources of data will be used: CRM, ERP, Databases, External Sources?
o What of data we need to source: sales, reports, website traffic, finance?
o Who needs access to this data: Head management, Marketing Department, Finance, Human Resources?
o Who is the audience: Directors, Managers, Analysts?
o What types of report do we need and how must be presented: Interactive dashboards, reports, spreadsheets, diagrams, websites?
o How would progress be measured?
o Which is the best BI environment to share the data and visualizations to the audience?
Knowing the answer to this question we can start thinking in possible KPIs and evaluation metrics to see how tasks are accomplished. These KPIs can be financial restraints, sales, performance indicators, etc.
Step 2: Choose BI Environment
For small-medium companies, the BI market offers a great number of tools are available as embedded versions and cloud-based technologies. Some examples of BI tools of public or low cost access for small-medium companies might be options like Power BI and Tableau. For large companies it’s possible to custom or build a BI ecosystem(s), this option can be considered for several reasons:
o Companies may not entrust their valuable data to a third party.
o There might be no BI Tools or vendor on the market that provides services for the industry of the company.
o The volume of data can play a particular reason to custom BI Development and there are needs higher flexibility in terms of choosing cloud infrastructure provider.
Step 3: Define your KPI
Drawing a sketch of the dashboard or the KPIs might help to build the pipeline and the ETL process properly. In most of the cases we need data from different sources, so is necessary to build the ETL process with the idea to standardize the data. Having the idea or knowing precisely of which KPIs we need or are going to be build can lead us to know how the pipelines and the standardization has to be done, so at the end we can have a clean and clear dashboard.
Step 4: Build or have a Team for BI projects in the organization
There are different roles for BI projects, this can lead us to build a development process and make architectural, technical, and strategic decisions. There are three main roles to consider in BI projects:
o Head of BI: this person must be armed with theoretical, practical and technical knowledge to support the implementation of the strategy and tools. This person can be an executive with business intelligence knowledge, knowledge of the business and industry, and access to data sources. This person might take the decisions to drive the implementation.
o BI Engineer: is a technical member that specializes in building, implementing and setting BI systems. This position has a software development and/or databases configuration background. They also must be well-versed in data integration methods and techniques.
o Data Analyst: Provide the team with expertise in data validation, data processing and data visualizations.
Step 5: Documentation of the BI strategy
Knowing the KPIs, Data Sources and the BI team, we can start building a strategy required to solve a problem. We can draw the BI strategy with product roadmaps. Is recommended that the BI Strategy considerate these components:
· Data Sources: the documentation of the chosen data sources channels. These channels might be from stakeholders, analytics of the industry and/or information of the employees and/or departments. Examples: Google Analytics, CRM, ERP, databases, etc.
· Custom KPIs: We can use BI tools to track KPIs of the industry or specific one the draw the full picture of the business growth and performance.
· Reporting standards: this section may include data types we want to deal with. For this part we need to standardize the reports visuals and text that need to be shown on the dashboard.
Step 6: Set up of data integration tools
This stage requires a lot of time and work by the IT department. One of the core elements of the BI is data warehousing. Data warehouse is a database that allow us to access and keeps information in a predefined format, usually structured, classified, and clean without errors. Since is possible that we can’t connect the data warehouse directly with the source of information, we must use ETL (Extract, Transform and Load) process tools or Data Integration tools.
For this types of tools we might consider if we have budget to build pipelines or if there are particular needs of programing languages or softwares to achieve this process. For example, there are free open sources like python and we can use Luigi to build pipelines in this language. But there are powerful tools that need budget, there is the existence of softwares that allow us to build easier and practical pipelines to feed the Data warehouse. An example of this case is Data Synchronization Studio and Simego that help us to do an automated pipelines in a ease way between the databases and the Data warehouse.
Set 7: Data Warehouse configuration and architectural approach based on needs
Working with high volumes of that may represent a cost in performance on queries and also with money. So, for this there are various types of solutions to present to data analyst small portions of a warehouse. The most used are OLAPs (Online Analytical Processing) and Data Marts. These options can offer us quick reporting and easy access to required data. We must considerate that these options are suggested if we have high volumes of data, because we can incur on extra costs of time and money. If we have small amount of data or we don’t handle high volume of data, SQL queries would be enough.
o OLAPs Cubes: Is a typical option to the portion of data we need from the data warehouse. This option applies to all size companies that require data storage and complex multidimensional analysis of the information. If we don’t want or need to build so many queries, we can considerate OLAP architecture approach.
o Data Marts: is a subject oriented piece of the data warehouse that gathers thematically familiar information dedicated to a specific department. This option can help us to avoid setting up permissions for the end users, and data marts help us to dedicate the analysis to a single sphere of the business.
We might need to choose the appropriate architectural approach for the needs of the organization.
Step 8: Choose the BI environment
We must select the BI environment that satisfies the need of the organization. There a lot of BI Tools to share our reports and dashboards. Also there is the possibility to build a new BI environment. There are options as Power BI, Tableau, Looker, Qlik, SAP, Excel , Sisense, Zoho Analytics, etc. But for this we might consider that for some of this environments we need to pay a membership or fee based on the size of the company and the needs. So, we need to consider the BI environment based on the budget, size of the company, amount of data, connections with data warehouse and data sources, and, maybe, how we want to deliver data and visualizations.
Here is an article that discuss features of 10 differentes BI tools environments: https://data-flair.training/blogs/business-intelligence-tools/
BI Bonus Tip
In this space I want to share an article that my boss shared with me the first time I started working with him. He is a Head of BI and Director of IT of the company where I work. All the meetings, since we started working together, always remembered me to read this article. When I read it, I found ideas and steps that are fundamental to build BI the right way. This article is name it Doing Power BI the Right way from Paul Turley.
