Azure Data Factory

What is Azure Data Factory?

Azure Data Factory (ADF) is Microsoft's managed service for building data pipelines. Its job is to connect the operational systems of the business to the data warehouse: pull data from many sources, clean and transform it, and load it into the target in an orderly, scheduled way. It is an infrastructure component - users never see it, but without it the reports simply do not refresh in the morning.

What is a pipeline made of?

A typical process looks like this: a trigger starts the pipeline at a fixed hour, the pipeline pulls data from the ERP and from files through linked services, processes it in a data flow, writes it to the staging layer of the data warehouse - and finally refreshes the Power BI model. If something fails you get an alert, and can re-run only the step that broke.

When do we choose Azure Data Factory?

• Data arrives from several systems - ERP, CRM, Excel files, websites and APIs
• The warehouse and the reports need a daily or more frequent refresh
• The organisation runs on Azure and wants a managed service that fits into it
• There are on-premise systems that must be read securely
• Every load needs to be tracked and documented - what ran, when, and how many rows

ADF, the data warehouse and Power BI

ADF is the first link in the chain: it feeds the data warehouse - whether Azure SQL, Snowflake or Microsoft Fabric - and the semantic model and reports are built on top of it in Power BI. When the pipeline is built properly, refresh becomes invisible: the data is simply up to date when people arrive in the morning.

What do we offer?

• Scoping and building end-to-end pipelines - from your source systems to the reports
• Connecting core systems: Priority, SAP, Dynamics, Hashavshevet and bespoke systems
• Replacing manual processes or legacy scripts with a managed, monitored pipeline
• Building monitoring, alerting and failure handling - including incremental loads
• Improving BI performance and shortening refresh times for heavy reports