Abstract
In today's fast-developing world of information systems, companies use big data engineering platforms that operate with massive amounts of information. The implementation of DevOps practices in these infrastructures has then raised as a strategic approach those targets to maximize efficiency, scalability, and reliability. In this paper, the author focuses on the effects of DevOps in large-scale data engineering and discusses the pros, cons, and results achieved in different sectors. Also, the incorporation of DevOps improves the efficiency and productivity of organizations by enhancing their workflows, automating their work process, and increasing the pace of deployment, all of which are paramount to sustaining a competitive edge. This research work makes use of both qualitative and quantitative research methods. It collects data to examine the case of organizations that have tried to adopt DevOps in data engineering teams. Hattin and Hooper’s research findings point to positive developments such as faster deployment, increased system dependability, and greater adeptness in processing large amounts of information; however, they also indicate unresolved issues, for example, organizational culture change and technical integration. The findings of this research add to the literature in the following ways: This research established the benefits and shortcomings of DevOps in giant data sets contexts. To these intents and purposes, the study offers guidelines for organizations adopting DevOps or contemplating it by pointing out that DevOps is and should be a continuous improvement process. Organizations should, therefore, devise strategies that address specific challenges as they implement the DevOps model.

Keywords: DevOps, Data Engineering, Large-Scale Infrastructure, Continuous Integration, Continuous Deployment, Scalability, Automation, Case Study

1. Introduction

Data engineering has been evolving for the recent few years due to the need for complex and cost-efficient approaches to manage large datasets. Earlier, data engineering mainly looked at four basic operations: extraction, transformation, loading, and analysis, where data was extracted from sources and transformed into a form suitable for analysis before being loaded into a data warehouse¹. Although these processes are typically effective, they pose operational problems because they are static, time-consuming, and incapable of relating to data volume, variety, and velocity. Due to the constantly increasing volume of data produced and consumed by businesses, there was a definite predisposition to more flexible and adaptive data infrastructures. This has created the need for modern data engineering practices such as real-time data processing, distributed computing, and cloud-based data platforms. Consequently, to meet these new demands, organizations have looked to DevOps as a tool to improve the approaches taken to managing and improving the data engineering environments. The DevOps process, which is based on automation, cooperation, and improvement, is well-tailored to tackle the issues derived from the modern data landscapes effectively.

Role of DevOps in Data Engineering

Even though DevOps practices were initially deployed in software development to enhance the speed of the software development process and their deployment, the concept has been seen to be of immense value when applied to data engineering¹. CI/CD is the foundational concept of DevOps, which enables the integration and delivery of updates at a fast and secure rate. In data engineering, CI/CD allows for improving the speed and quality of data processing pipelines’ development, testing, and production. The last major factor of DevOps is automation, which makes it possible to eliminate much of the time and effort that is otherwise spent on managing data processes and preventing mistakes. There is no need to spend much time on data validation, transformation, and deployment since these automation tools guarantee that the data pipelines are correct. In addition, the DevOps discipline emphasizes cooperation between the development and operation teams, eradicating the non-existent barriers and improving cooperation in handling challenges. That is why such collaboration is especially effective in data engineering when the effectiveness of the data pipeline highly depends on teams. Deploying DevOps can enable establishments to make rapid, credible, and progressive tendencies in the data handling channels, hence improving organizational performance.

Scope and Objectives

This study's case focus is on how the DevOps concept can be applied within big data engineering environments and the associated issues and techniques. Consequently, the objectives of this study are multifold. First, it explores the main issues organizations experience when adopting DevOps in data engineering. Such challenges may include technical issues, organizational hindrances, and possibly a lack of skills. Second, the study intends to establish the nature of solutions that organizations have implemented to redress the above-listed challenges. This includes a look at some of the specific and effective tools and the frameworks and processes that have been useful in getting organizations up to speed with DevOps. Finally, the research will investigate DevOps's effects on the scalability, availability, and flexibility of data pipelines. Thus, through an analysis of practical cases, the study will determine DevOps's role in enhancing data processing and organization efficiency. Last, this analysis will conclude with practical recommendations for various organizations that may be practicing or planning to practice DevOps in their data engineering activities. Based on the case studies, these recommendations will be given so that this paper can provide guidelines on overcoming the challenges faced in adopting DevOps for data engineering returns.

2. Literature Survey

Role of Enterprise DevOps Solutions

Figure 1: Role of Enterprise DevOps Solutions.

The Role of Enterprise DevOps solutions reflects the idea of the ‘DevOps cycle,’ which is unbroken and repeated. This diagram is in the form of an infinity⁶ loop because DevOps is a continuous cycle, without start and end, where the development (Dev) and operations (Ops) of software are integrated to deliver it quickly and stably.

Plan: Before anything else, planning is done, unlike in the traditional waterfall model, where planning is not an essential stage of developing and deploying software. Teams describe objectives, time frames, and resources used in the project. This phase is a bit more involved, and it usually comprises stakeholders in the development process as they seek to make sure that the business needs are understood so that they can work on the development processes with a shared understanding of what is expected of the project.

Code: In this phase, developers code the application or the service that is writing the application or service code. Coding is the code's definition, construction, and examination for compliance with the set-down specifications. An example of a code repository is Git, which helps with version control and allows many developers to work on the same project.

Build: The create phase converts the source code into executable files. It involves library relations, file bundling, and preparing the software and its delivery. CI tools like Jenkins or Travis CI do this automatically, allowing developers to identify and correct errors in the initial phases of the SDLC.

Test: In the case of automated testing, testing is performed to verify that the code is running correctly and complies with quality standards. It is carried out to capture as many bugs, errors, and issues in the software as possible. The verification phase of DevOps is an essential compartment as it includes testing, an ongoing process that helps provide immediate feedback to minimize the chances of introducing a defective version in the production system.

Deploy:After the software has gone through the testing phase, it is released to a staging or production server. This phase focuses on remotely deploying the code on different servers so that the software runs as desired. Continuous Delivery (CD) helps to achieve an understanding that updates are ready to be released at any time.

Operate: During the operating phase, the software is run as a live system, perhaps the most critical stage of its lifecycle. This comprises overseeing the application's functioning, ensuring it is always up, and managing the infrastructure. Support teams are responsible for the efficient functioning and optimization of the developed software, the identification of problems, and their immediate resolution.

Monitor: If you intend to develop healthy software that can perform to the maximum, then monitoring is key. This phase consists of monitoring the key indicators, including uptimes, response times, and error rates, using monitoring tools like Prometheus or Grafana. Monitoring is a process of constantly observing the system so that the teams can diagnose the problem before causing inconvenience to the users.

Review: The last process in the DevOps phase is evaluation and monitoring, in which the results of the deployment and monitoring states are considered. It involves examining the performance data, following up with feedback, and improving subsequent compilations. This one focuses on evolution, which is one of the core values of the DevOps framework.

Overall Integration: These phases should be incorporated into a continuous loop that underlines the cooperative aspect of DevOps. Development and operation are integrated through the lifecycle, wherein individuals use automation practices to improve the duration and quality of software delivery. Such an approach implemented in the DevOps practice creates never-ending feedback. It allows the software to alter dynamically to changing requirements and environmental factors, which makes this practice crucial for modern software development.

DevOps in Software Development vs Data Engineering

DevOps practice has become an essential part of software development, but its incorporation in the data engineering domain comes with its issues and challenges. DevOps features are pretty simple in software development, particularly when combined with build, testing, and deployment. However, the ecosystem in data engineering is relatively more complex than data science because it deals with ETL processes that involve extracting, transforming, and loading big data from diverse sources^7,8. Such pipelines are usually cascaded, and each stage is generally dependent on the other in that they should work together in harmony to ensure that data is correct and consistent. Also, the type and amount of data being analyzed vary greatly, from database records to logs and social media posts, for which there is a need for specific tools and frameworks to accommodate the dimensions and volumes of data. The need for real-time processing further makes it a challenge since it has low latency requirements and data needs to be analyzed in near real-time. Therefore, even though DevOps practices apply here in one way or another, data engineering is somewhat different in terms of what is involved and will need some unique practices to address.

DevOps engineer responsibilities

DevOps engineers can be illustrated, which shows the various activities they are involved in⁹ within the DevOps setting. Every icon in the image corresponds to elements of the DevOps engineer’s functional responsibilities because the role encompasses various fields of Signification.

Writing Specifications and Documentation

Some of the activities this duty encompasses include documentation development and management of various systems, procedures, and organizational activities. DevOps engineers are responsible for writing specifications for developing, deploying, and running software systems. Documents are used both as a source of information for the team members and to achieve the goal of making the development and operation teams more aligned with helping the organization achieve its goals.

 Infrastructure Management: It is the engineers or specialists in DevOps who are charged with the task of dealing with the underlying framework of a software application. This covers setting up, managing, and provisioning servers, networks, and storage to meet the organization's needs. The other aspect of infrastructure management includes the performance analysis of the systems and their security, and in general, it entails the management of the entire environment that can be utilized for the development and implementation of the software.

 
Figure 2: DevOps Engineer Responsibilities.

Performance Assessment and Monitoring

Another crucial responsibility of a DevOps engineer is measuring and evaluating the performance of systems and applications. This encompasses the establishment of tools for monitoring various KPIs or observation of system behavior with a view to identifying any potential problem that may affect users. This makes it possible to maintain applications on an ongoing basis to support high availability and best performance.

Cloud Deployment and Management

DevOps engineers depend on cloud deployers for their work, as it falls under their purview to deploy and manage applications in the cloud. This consists of fitting applications to the public or private cloud by using AWS, Azure, or Google Cloud, refreshing resources to accommodate an organization's needs, and managing databases. Cloud deployment also includes making applications remain continuously available, protected, and able to scale up and down as needed by adopting cloud-native features.

CI/CD Management and Automation

CI/CD is a key DevOps activity, and supervising these pipelines is among the significant tasks of the DevOps engineers. This means they take over the handling of code changes, execute tests, and thereby deploy updates to production. This automation assists in minimizing errors that might be occasioned by a human hand, expediting delivery cycles and releases, and guaranteeing that new features or fixes are touched on for the users.

Avanade DevOps Culture Adoption Support

They also support creating a DevOps culture within the organizations since they are involved in the implementation process of the DevOps culture. This consists of ensuring cooperation between development and operations and supporting the use of effective practices and the use of the framework by the teams. Dealing with such a cultural change, DevOps engineers thus contribute to eliminating barriers and optimizing performance within the company.

All of these roles highlight the DevOps engineer as the key figure linking development and operations to ensure that software is delivered to end users as quickly, efficiently, and safely as possible.

Primary Obstacles to DevOps for Data Engineering

Implementing DevOps in data engineering has its limitations, and it is not an easy task because of the advancement in data environments^10-12. The most critical issue that needs to be mentioned is that data management is becoming increasingly complicated. Data from various sources, with different formats, qualities, and the frequency of arriving at the system, can quadruple the scale at which the data needs to be governed and stored. Also, data pipelines require pipeline dependency management where, in the different stages of a pipeline, the completion of one stage depends on the other stage. This interdependence could cause some challenges in the deployment process, especially in cases of frequent changes or updates of the pipeline.

The last but not least important challenge is processing real-time data. Real-time data processing and analysis are essential factors in DevOps since they provide immediate insights into an organization. However, doing it at scale is not easy, and specific tools and architectures that allow efficient processing of high-rate data are needed.

Table 1: DevOps Toolchain for Data Engineering.

The final hurdle in DevOps for data engineering is the issue of toolchain integration. The DevOps toolchain-soon to be familiar to anyone involved in application development involving version control, continuous integration, continuous deployment, and monitoring—must interoperate with data engineering platforms and tools. This integration may be challenging as the data engineering tools can be unique and strict in allowing integration and communication between the toolchain constituents.

Four Current Frameworks and Strategies

To this end, multiple strategies and models have appeared to help solve the issues aforesaid and promote the use of DevOps in data engineering. One of the many approaches explicitly targets the data analytics area, which is called DataOps. DataOps uses CI/CD practices to integrate, deploy, and automate the data analytics pipeline so that the data analysts’ processes are as responsive as the software developers’ processes. Data integration also involves working with data scientists, engineers, and operations, which allows everyone responsible for data quality and pipeline quality.

Another related practice is known as MLops – essentially, applying DevOps concepts to the processes related to machine learning. MLops is concerned with the problems of deploying and operating machine learning models at scale. It entails the continuous practice automation of education, assessment, and implementation and the supervision of the models to guarantee that their precision is not compromised in the future. Data engineering process flow is also incorporated in the MLops to enhance the delivery of data insights in machine learning workflow.

Besides these frameworks, there is CI/CD for DP, which consists of tools and practices aimed at automating the deployment and management of DP. Standard processes within CI/CD pipelines are stages for data validation and transformation and for data engineering deployment to ensure that changes to the pipeline are tested and deployed. This approach contributes to increasing the speed of deployment and reducing the likelihood of errors, with the subsequent enhancement of the reliability of data processing pipelines.

Altogether, these approaches guide organizations willing to introduce DevOps in their data engineering work, pointing out tools and best-practice frameworks tailored to this field’s peculiarities. DataOps, as well as MLops and CI/CD for data pipelines, could help organizations increase the speed, flexibility, and robustness of data pipelines, providing a better foundation for decision-making and business value.

3. Methodology

Attributes of Selected Cases

The choice of cases for this research was guided by specific criteria intended to guarantee the analysis’s focus and depth in identifying the adoption of DevOps in large-scale infrastructures for data engineering. First, the significance of organization size was established with a primary interest in large organizations. They are usually the most data-intensive organizations that must deal with massive amounts of data across several platforms and procedures. It allows for studying the specificities of implementing and adapting DevOps practices at such a large scale and the particularities of the difficulties encountered^13-15. Secondly, the maturity of DevOps was a concern. From the list, chosen organizations had to meet the criteria for implementing DevOps in the software development cycle. Such maturity in the DevOps field makes it possible to have case studies drawn from organizations past the primary stages of DevOps implementation and have fine-tuned these practices for data engineering. Finally, there is the criterion of data engineering complexity of the evaluative metrics, the importance of which is at the highest level. It is particularly pertinent that the case studies selected center on architectures where large data flows go through highly intertwined and correlated pipelines. This complexity needs to be addressed, and it is necessary to seek more advanced ideas that would help in the further development of DevOps, which is capable of handling the integration, processing, and organization of the data at large. Thus, the selection of organizations meeting these criteria will enable the research to consider the learners and related barriers and the approaches and implications of DevOps implementation in large-scale data engineering settings.

Data Collection Methods

Qualitative and quantitative data-gathering techniques were employed for this research, enabling an all-rounded understanding of DevOps in Data Engineering. The study participants comprised DevOps engineers, data engineers, and IT managers to get a feel of how DevOps is being implemented and its effects. The semi-structured interviews offered good qualitative information that described the processes and the role by providing experiences and the difficulties and triumphs of actual participants. These interviews served the purpose of knowing the finer things about how and why DevOps was being implemented in the organizations. Aside from interviews, a documentation review was conducted, and interviews were complemented with a documentation review. This involved a review of internal reports on DevOps, gameplay, references, and technical reports submitted by the organizations. At the same time, the documentation review gave an understanding of the work done, including the processes, tools, and frameworks of DevOps implemented by the organization. It also served to clarify the data from interviews and allowed us to observe the organization’s approaches to DevOps in data engineering in a more ordered manner.

Moreover, a tool and technology review was conducted to ascertain the particular DevOps toolchains used in data engineering. This paper concentrated on the tools chosen and integrated to automate and monitor data pipelines and their efficiency. Using the data collection methods mentioned above, the research was able to construct the practicum of DevOps in large-scale data engineering infrastructures.

Analysis Framework

Guided by the research questions, the collected data was analyzed and arranged based on several dimensions to ensure we got a general view of the state or level of DevOps in data engineering. The first is a process that analyses the alterations DevOps causes in data engineering processes. This is, for example, how DevOps practices have changed the process of building, testing, and deploying data pipelines and the degree of enhancement they have brought. The second dimension is dedicated to the presence of tools. Here, elements cover how automation and monitoring facilities assist the DevOps approach²⁰. It evaluates the effectiveness of these tools within the data engineering duty and the level of automation in the process. The third and last dimension we have is collaboration, where the changes introduced by DevOps are assessed based on communication between the development team and the operational team. This part of the evaluation is essential to analyze how DevOps practices impacted the organization’s culture and the performance of the collaboration of cross-functional teams regarding data engineering responsibilities. Last but not least, it also applies performance evaluation of the system in terms of common performance indicators, which include, in this case, deployment frequency, data processing speed, and error percentages. These metrics give tangible measures of the virtue DevOps can bring to enhancing the rates, productivity, and quality of data engineering. Dividing the analysis by these dimensions, the research intends to provide a comprehensive and qualitative picture of what contributes to successful DevOps in large-scale data engineering infrastructures.

Table 2: Performance Metrics Before and After DevOps Adoption.

4. Case Studies

Case Study 1: Global Financial Institution

This paper focuses on the example of one of the largest international financial companies handling petabytes of financial information daily. The institution's big data team was under tremendous pressure mainly because of the slow deployment of pipelines and high failure rates. These problems not only hampered the provision of timely analytical information to the team but also worsened the work’s reliability and cost. To overcome these challenges, the following strategies were Implemented to enhance DevOps in the institution: They provided CI/CD pipelines where they automated the testing and deploying of data processing jobs, thus cutting on the number of workforce interventions done on the jobs, and also helped hasten the deployment of employment. Furthermore, the team extended the use of infrastructure as a code approach, utilizing Terraform and Ansible to handle their data infrastructure. This approach enabled them to know and configure new architectural parts by code, thus eliminating the mistakes that are normal when physically building something. To increase the stability of data pipelines, the institution continued the usage of Prometheus and Grafana in monitoring and alerting pipelines. These tools gave him the live status of his data infrastructure and helped the team to address the problems on time. The results of this DevOps adoption were notable: when the software was released monthly, there were 40% more pipeline failures; Weekly release reduced pipeline failures, and organizations reported increased cross-functional collaboration and overall better cohesiveness.

Case Study 2: E-commerce Giant

In this case, an e-commerce giant's executive management struggled to obtain accurate real-time analytics of their customers’ activities to respond to changes in traffic patterns, for example, during sales seasons. For such traffic increases, the company's data infrastructure had to be ready to scale while still performing as expected. To this end, the organization adopted DevOps, centered on using containers with Docker and Kubernetes. This helps them control and optimize large data processing tasks and guarantees that implementing their architecture would accommodate huge workload fluctuations. Also, they introduced mechanisms of data versioning to deal with changes in the datasets and safeguard the data integrity, whether it is developed or undergoing a change. To address these issues, the company used automatic scaling technologies such as AWS Lambda and auto-scaling groups, which allowed for scaling their infrastructure depending on the traffic, hence using resources to their optimum. Implementing these DevOps practices led to several key outcomes: the infrastructure structure expanded in proportion to the traffic. Thus, the company could analyze data with minimal delays. Also, the applications and services that use their cloud infrastructure were more cost-efficient, as the costs were reduced by 30 percent because of the improved scaling mechanism. These results confirm that DevOps supports large-scale data processing in real-time for large companies with constantly changing and high burdening environments.

Table 3: Overview of Case Studies.

5. Results and Discussion

Impact of DevOps on Data Engineering

DevOps has revolutionized data engineering by bringing simplicity and scalability to data infrastructure. For this article, the requirement of flexibility or scalability was defined as the most important, as numerous organizations pointed out that their data storage should be ready to handle various types of workloads. For instance, in the case of the e-commerce giant, the ability to use automated scaling, thanks to the AWS Lambda and the Auto-Scaling Groups, helped the infrastructure answer during the critical load period. This led to the optimization of the infrastructure cost cutting on the cloud infrastructure by 30%. Similarly, the organization enhanced its reliability as a global financial institution. Thanks to implementing automated testing tools and monitoring tools such as Prometheus and Grafana, the efficiency rose 40% regarding pipeline failure. This reliability ensured that any raw and critical financial data to be processed was done with close to zero incidences of operational risks. In addition, there was a significant improvement in the flexibility of data engineering work streams partly due to continuous integration and deployment practices. For instance, the financial institution switched from monthly deployment cycles to weekly cycles to deliver new data processing jobs and adapt quickly to the changes within the organization.

Main Challenges with Relevant Hedging Strategies

Nevertheless, several issues relating to the implementation of DevOps for data engineering remained challenging for the organizations, including the following. Toolchain complexity was essential because different tools tended to be challenging to integrate into a DevOps toolchain. Several organizations were able to reduce the impact of this issue by following the toolchain concept and using different tools in isolation or going for platform solutions where different tools had a more integrated form. For instance, the organization achieved its objective of adopting a financial institution that uses infrastructure such as Code, Terraform, and Ansible. The other problem area was skills. The lack of engineers who can implement both DevOps and data engineering approaches caused a barrier to practical implementation. To overcome this, organizations turned to cross-training or employing multi-skilled personnel who performed DevOps and data engineering functions. This helped build up the required skilled personnel and provided a mechanism through which other teams understood each other and improved collaboration. Data security thus had issues since data moves very fast in a DevOps environment, and this would lead to security issues. Organizations incorporated enhanced security measures to reduce such risks, including encryption, access authorization, and perpetual security vigilance. Such measures kept data safe while the speed and frequency of the deployments rose.

Comparison of case studies

A comparison of the two cases reveals how the two organizations have different importance and strategies for implementing DevOps. The deployment frequency was a significant concern of the global financial institution, and it was equally important because the organization was able to release new data processing jobs more frequently than it used to do, such as changing from a monthly to a weekly basis. Facilitating this frequency increase within the institution enabled addressing evolving business dynamics and sustaining a strong position. For the web giant, the major concerns were scalability and cost control. The ability to provide containerized designs for marked scaling also helped them to address the inundation of traffic typical of the peaks of holidays or other critical shopping times devoid of aggressive costs or performance degradation. As for the technology stack, both organizations applied containerization technologies, but the tools they used were different according to the organization’s requirements. The particular financial institution utilized Kubernetes to orchestrate containerized workloads as it helped manage the containers very efficiently. In contrast, the e-commerce giant utilized AWS Lambda to handle event-driven workloads. It is a very efficient system model that requires no maintenance and is scalable and cost-efficient. Due to these differences, DevOps must be suited to the needs and goals of the company to create harmony in the implementation domains.

6. Conclusion

Summary of Findings

From the literature, the evidence shows how DevOps has supported agility, dependability, and scalability in large, complex data engineering infrastructures. The case studies gave real-life examples of how various organizations have adopted DevOps to deal with the challenges of fulfilling their data processing objectives. The global financial institution and the e-commerce giant received notable enhancements in deployment frequency, scalability, and operational efficiency. They asserted DevOps' ability to work in various aspects of data engineering.

Recommendations for Practice

For organizations looking to implement DevOps in their data engineering practices, several key recommendations emerge from the case studies: For organizations looking to implement DevOps in their data engineering practices, several key recommendations emerge from the case studies:

Start Small: Pilot projects to prove to leadership that DevOps principles are important in data engineering. This war gaming approach enables the tweaking to be rolled out across the organization.

Invest in Automation: Automating software testing, deployment, and monitoring is crucial to minimize human interference. Automation does not just fasten processes; it reduces the chances of error occurrences and guarantees standardization.

Foster Collaboration: Promote coordination of development, operation, and data groups to implement the DevOps technique properly. Collaboration between different functions results in a greater ability to address challenges and concurrently develop solutions for the organisation’s data processing.

Future Research Directions

Although this research has offered insights into the effects of DevOps in data engineering, many questions have not been fully expounded. A study should be carried out on the longevity of devOps in data engineering and its further applicability in fast-growing domains such as machine learning, big data analytics, and real-time processing. Furthermore, the contribution of new trends in technology, such as AI automation, to the improvement of DevOps may help look at future innovation advancements in this field. While organizations press on with their DevOps implementation initiatives, identifying the new tendencies in this area will provide essential insights to stay ahead in the era of data-driven business.

7. References

1.             Humble J, Farley D. Continuous delivery: reliable software releases through build, test, and deployment automation. In: Pearson Education, 2010.

2.             Kim G, Humble J, Debois P, et al. The DevOps handbook: How to create world-class agility, reliability, & security in technology organizations. In: It Revolution, 2021.

3.             Bass L, Weber I, Zhu L. DevOps: A software architect’s perspective.  In: Addison-Wesley Professional, 2015.

4.             Kreps J, Narkhede N, Rao J. Kafka: A distributed messaging system for log processing. In: Proceedings of the NetDB, 2011; 11: 1-7.

5.             Guo Y,  ACM Digital Library. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. In: ACM, 2018.

6.             https://www.orangemantra.com/blog/the-roadmap-to-devop-solutions-adoption-at-enterprise-levels/

7.             Zaharia M, Chowdhury M, Das T, et al. Resilient distributed datasets: A {Fault-Tolerant} abstraction for {In-Memory} cluster computing. In: 9th USENIX symposium on networked systems design and implementation (NSDI 12), 2012; 15-28.

8.             D'Angeac GD.  Big data: The management revolution-Harvard business review. Harv. Bus. Rev, 2012;  90: 60-66.

9.             https://www.samsungdevcon.com/making-devops-your-most-trusted-ally/

10.          Qumer Gill A, Loumish A, Riyat I, Han S. DevOps for information management systems. VINE Journal of Information and Knowledge Management Systems, 2018; 48: 122-139.

11.          M.S.K.A.W.K.F.K.M.A, W.T.K. Khan. Critical challenges to adopt DevOps culture in software organizations: A systematic review. Ieee Access, 2022; 10: 14339-14349.

12.          https://appinventiv.com/blog/devops-adoption-and-implementation/

13.          https://atlan.com/devops-vs-data-engineer/

14.          Perera P, Silva R, Perera I. Improve software quality through practicing DevOps. In: 2017 seventeenth international conference on advances in ICT for emerging regions (ICTer). IEEE, 2017: 1-6.

15.          Hasselbring W, Henning S, Latte B, et al. Industrial devops. In 2019 IEEE International Conference on Software Architecture Companion (ICSA-C).  IEEE, 2019; 123-126.

16.          Wang C, Liu C. Adopting DevOps in Agile: Challenges and Solutions. 2018.

17.          Qumer Gill A, Loumish A, Riyat I, et al. DevOps for information management systems. VINE Journal of Information and Knowledge Management Systems, 2018; 48: 122-139.

18.          https://www.xenonstack.com/insights/devops-best-practices-for-data-engineers https://medium.com/@mudrapatel17/data-engineering-concepts-part-7-devops-dataops-and-mlops-afc6f432473c

19.          Silva MA, Faustino JP, Pereira R, et al. Productivity gains of DevOps adoption in an IT team: A case study. 2018.