General Data Science Life Cycle
- [Data Science Lifecycle - Microsoft]
- Overview
Data science is a buzzword right now because of its success in many applications. From the oil industry to the retail industry, everyone is benefiting from data science.
A careful understanding of the data science lifecycle and implementing these steps correctly can help business growth. There are many tools available to extract insights from data which can then be used to improve the business
The data science lifecycle represents the iterative steps taken to build, deliver, and maintain any data science product. All data science projects are structured differently, so their lifecycles also vary. Still, we can picture a general lifecycle that includes some of the most common data science steps.
A general data science lifecycle process includes the use of machine learning algorithms and statistical practices that result in better predictive models. Some of the most common data science steps involved in the overall process are data extraction, preparation, cleaning, modeling and evaluation, etc. The data science community refers to this general process as "the cross-industry standard process for data mining."
- Who Is Involved In The Project?
- Domain Experts: Data science projects are applied in real life in different domains or industries such as banking, healthcare, oil, etc. A domain expert is someone who has experience working in a particular field and knows both inside and outside of that field.
- Business Analyst: A business analyst needs to understand the business needs in the identified domain. This person can guide the design of the correct solution and timeline.
- Data Scientist: A data scientist is an expert in data science projects, has experience working with data, and can formulate solutions based on the data needed to generate the desired solution.
- Machine Learning Engineer: A Machine Learning Engineer can suggest which model to apply to get the desired output and devise a solution to produce the correct and desired output.
- Data Engineers and Architects: Data architects and data engineers are experts in data modeling. They are responsible for data visualization for better understanding as well as storing and retrieving data efficiently.
- Problem Identification
This is a critical step in any data science project. The first is to understand in what ways data science is useful in the domain under consideration, and to identify appropriate tasks that are useful for this. Domain experts and data scientists are key players in problem identification. Domain experts have in-depth knowledge of the application domain and the problem to be solved. Data scientists understand the domain and help identify problems and possible ways to solve them.
- Business Understanding
Understanding what customers want from a business perspective is business understanding. Whether the client wants to make forecasts or wants to increase sales or minimize losses or optimize any particular process etc. constitutes a business objective. In the business understanding process, two important steps are followed:
- KPIs (Key Performance Indicators)
For any data science project, key performance indicators define the performance or success of the project. There needs to be an agreement between the client and the data science project team on business-relevant metrics and related data science project goals. Design business metrics based on business needs, and then the data science project team decides on goals and metrics accordingly. To understand this better, let's look at an example. Assuming the business requirement is to optimize the company's overall spend, the goal of data science would be to manage twice as many customers with existing resources. Defining key performance indicators is very important for any data science project because the cost of solutions will vary for different goals.
- SLA (Service Level Agreement)
Once performance metrics are set, it's important to finalize service level agreements. Determine service level agreement terms based on business goals. For example, any airline reservation system needs to handle 1000 users simultaneously. Then the product must meet this service requirement as part of the service level agreement.
Once performance metrics are agreed upon and service level agreements finalized, the project moves on to the next important step.
- Data Collection
Data collection is an important step as it forms a vital foundation for achieving the targeted business goals. Data flows into the system in a variety of ways.
Basic data collection can be accomplished using surveys. Often, data collected through surveys provides important insights. A lot of data is collected from various processes followed in a business. At various steps, data is recorded in various software systems used in the enterprise, which is very important for understanding the process from product development to deployment and delivery.
Historical data obtained through archives is also important for better understanding the business. Transaction data also plays a vital role as it is collected on a daily basis. Many statistical methods are applied to data to extract important information relevant to the business. In a data science project, data plays a major role, so proper data collection methods are important.
- Preprocessing Data
Big data is collected from archives, day-to-day transactions, and intermediate records. Data is provided in various formats and in various forms. Some data may also be available in hard copy format. Data is scattered in various places on various servers. All this data is extracted and converted into a single format, which is then processed. Typically, data warehouses are built where extract, transform, and load (ETL) processes or operations are performed. In data science projects, this ETL operation is critical. Data architect role is important at this stage, he decides the structure of the data warehouse and performs the steps of ETL operations.
- Analyze Data
Now that the data is available and prepared in the desired format, the next important step is to gain insight into the data. This understanding comes from the analysis of the data using the various statistical tools available. Data engineers play a vital role in data analysis. This step is also known as exploratory data analysis (EDA). Here, the data are examined by formulating various statistical functions and the dependent and independent variables or characteristics are determined. Careful analysis of the data reveals which data or characteristics are important and how well the data is disseminated. Various charts are used to visualize the data for better understanding. Tools like Tableau, PowerBI, etc. are known for performing exploratory data analysis and visualization. Knowledge of data science using Python and R is important for performing EDA on any type of data.
- Data Modeling
Once data has been analyzed and visualized, data modeling is an important next step. Important components are retained in the dataset, further refining the data. The important thing now is to decide how to model the data? Which tasks are suitable for modeling? Suitable tasks, such as classification or regression, depend on the desired business value. There are also many modeling approaches available for these tasks. Machine learning engineers apply various algorithms to data and generate output. When modeling data multiple times, the model is first tested on dummy data that resembles real data.
- Model Evaluation/Monitoring
Since there are many ways to model data, it is important to determine which one works. For this model evaluation and monitoring phase is very critical and important. The model is now tested with real data. Data may be minimal, in which case the output is monitored for improvement. While evaluating or testing a model, the data may change, and the output will vary dramatically based on the change in the data. Therefore, when evaluating a model, the following two phases are important:
- Data Drift Analysis: Variations in the input data are known as data drift. Data drift is a common phenomenon in data science because, depending on the situation, the data changes. The analysis of this variation is called data drift analysis. A model's accuracy depends on its ability to handle this data drift. Changes in the data are mainly due to changes in the statistical characteristics of the data.
- Model Drift Analysis: To spot data drift, machine learning techniques can be used. Also, more complex methods like Adaptive Window, Page Hinkley, etc. can be used. Modeling drift analysis is important because we all know that change is constant. Incremental learning can also be used effectively in situations where the model is exposed to new data incrementally.
- Model Training
Once the tasks and models are finalized, as is the data drift analysis modeling, an important step is to train the models. Important parameters can be further fine-tuned to obtain the desired accurate output, which can be done during the training phase. The model is exposed to real data and monitors the output during the production phase.
- Model Deployment
Once the model is trained with real data and parameters are fine-tuned, then the model is deployed. Models are now exposed to real-time data flowing into the system and generating output. The model can be deployed as a web service or as an embedded application in an edge or mobile application. This is a very important step, since the model is now exposed to the real world.
- Drive Insights and Generate BI Reports
After deploying the model in the real world, the next step is to find out how the model behaves in real-world scenarios. The model is used to gain insights that help in strategic business-related decisions. Business goals are tied to these insights. Generate various reports to understand how your business is doing. These reports help determine whether key process indicators are being achieved.
- Make Decisions Based On Insight
In order for data science to work wonders, each step mentioned above has to be done with great care and precision. If the steps are followed correctly, the reports generated in the above steps help in making critical decisions for the organization. The resulting insights help in making strategic decisions, for example, organizations can predict in advance that raw materials will be needed. Data science can greatly assist in making many important decisions related to business growth and better revenue generation.
[More to come ...]
