Get data
Objective coverage
- Create a data connection
- Discover data by using OneLake catalog and Real-Time hub
- Ingest or access data as needed
- Choose between different data stores
- Implement OneLake integration for Eventhouse and semantic models
Access is an architectural choice
Before choosing an ingestion button, describe where the data lives, who controls it, how current it must be, and which engine will consume it. Copying, referencing, mirroring, and streaming solve different problems. A copy provides a separately managed representation. A shortcut provides a reference with its own access and availability dependencies. Mirroring replicates supported source changes. Event ingestion supports continuous operational data. A connection describes how a service reaches a source; it does not by itself move or authorize every row.
Northstar combines order records from an operational database, product files from a supplier, and delivery events from vehicles. Finance needs reconciled daily totals. Dispatch needs recent event analysis. Analysts need reusable tables and a governed semantic model. One tool does not have to perform all three tasks. A defensible design makes the contracts between stages visible: source freshness, transformation grain, security identity, and consumer expectations.
Create a data connection
A connection binds a supported connector to endpoint information, authentication, and other required settings. It may be a cloud connection or use a gateway when the source is not directly reachable by the service. Distinguish the human creating the connection, the identity stored or selected for source access, and the identity consuming downstream data. They can differ. A successful connection test establishes a particular connectivity and authentication check; it does not establish that every intended table, query, or scheduled refresh will succeed.
Identify the connector, source address, database or path, required authentication type, and network route. Confirm whether a gateway is needed and who administers it. Create or select an approved connection using the supported Fabric experience. Grant connection usage only to the relevant authors or items. Test the exact table or query, then bind the pipeline, dataflow, or model that will use it. Validate the scheduled or service execution path, not only a local Desktop preview. Document credential rotation and ownership so that an employee departure does not silently break refresh.
An analyst can query an internal SQL server from their laptop because it is on the corporate network. A service refresh fails because the cloud service cannot reach that server through the same route. Northstar configures the appropriate gateway and an authorized connection, maps the model to it, and tests refresh using the service path. If the failure changes from network timeout to table permission denied, the gateway solved connectivity but source authorization still needs work. Adding the analyst as workspace Admin would not correct the gateway route or database grant.
Diagnose in layers: name resolution and reachability, connector support, authentication, source permissions, gateway or cloud binding, then query behavior. A preview may run under a different credential from a scheduled job. A connection can also remain valid while an upstream schema change breaks the consuming transformation. Keep credential material out of notebooks and public repository examples. On the exam, a connection, a gateway, and a data-access permission are complementary requirements. None should be presented as a universal substitute for the others.
Scenario: Desktop refresh works on a corporate laptop, while the same published model times out before reading an internal source. What should you verify first?
Answer: Verify the service's network route and required gateway configuration, then its connection binding and credentials. Reproduce the refresh through the service identity. A successful local preview does not demonstrate cloud reachability, and granting broader workspace rights does not create a network path.
Recall: Which identities participate in a service refresh? What evidence distinguishes a source permission failure from a network timeout?
Discover data by using OneLake catalog and Real-Time hub
Discovery tools help people find existing assets before creating another copy. OneLake catalog organizes discoverable Fabric data and its governance context. Real-Time hub focuses on discovering and connecting to streaming and event-oriented data. Discovery is not authorization: seeing an item or its metadata does not necessarily permit reading its contents. Evaluate purpose, owner, endorsement, freshness, and access requirements before adopting an asset. An asset with a familiar name can still use a different business definition or grain.
Begin with the analytical question and required latency. Search the relevant discovery surface, inspect the item's type and owner, and examine its description and governance signals. Check whether it represents raw events, curated tables, or a reusable semantic model. Confirm the available connection or consumption path and request the appropriate access through the organization's process. Inspect sample schema and known freshness evidence after authorization. Prefer an existing governed asset when its semantics fit; do not choose it solely because it is easy to find.
Finance finds Sales Curated in OneLake catalog and Dispatch finds Delivery Events in Real-Time hub. Sales Curated has one row per posted order line and refreshes after reconciliation. Delivery Events has multiple status records per shipment and arrives continuously. Joining the two without reducing events to the required grain could multiply finance totals. Discovery solved the finding problem, not the modeling problem. The analyst records the owner, grain, timestamp meaning, and intended use of each asset before designing the combined dataset.
If an expected item is absent, check tenant and workspace context, supported item types, permissions, and discovery configuration. If it appears but cannot be read, inspect the requested consumption path and data permissions. A certification badge indicates a governed trust process, not universal fitness for every question. A real-time source may still have end-to-end delay through ingestion, transformation, and serving. The exam distinction is between finding an asset, establishing access, and deciding that its content meets the analytical requirement.
Scenario: An analyst finds a certified model but cannot create a report against it. Should the team copy its tables into a new lakehouse immediately?
Answer: First inspect the model's intended audience and Build permission, and confirm that its measures and grain fit the task. Resolve authorized reuse before creating another copy. A catalog entry and certification do not automatically grant authoring access or prove that a physical copy is necessary.
Recall: When is Real-Time hub the more relevant starting point? Why should grain and timestamp semantics be recorded during discovery?
Ingest or access data as needed
Data can be moved into a managed destination or accessed through a supported reference. Copy activities and related ingestion tools create or update target data. Shortcuts expose supported external or internal data through OneLake without requiring an independently scheduled full copy for every consumer. Mirroring replicates changes from supported sources. Streaming tools handle continuous event flows. Compare ownership, retention, latency, transformation needs, source load, and failure recovery before selecting one. “No copy” does not mean “no security or availability dependency.”
Record whether consumers need a stable snapshot, continuously updated replica, or live access to an owned source. Verify connector and source support. Establish the least-privilege connection, destination, schema, and update contract. For a copy, decide append versus replacement or keyed update and how retries behave. For a shortcut, validate both the target and source access paths and current limitations. For mirroring, inspect initial synchronization, supported changes, and lag. For events, define event time, arrival time, duplicate handling, and the downstream serving path.
Supplier product attributes are published as governed Delta data that Northstar may reference, while the order system must be relieved of repeated analytical scans. A supported shortcut can expose the supplier data, and a supported replication or incremental ingestion path can maintain analytical orders. Delivery events flow to an event-oriented store. Each choice has a separate acceptance check: supplier permissions and schema, order reconciliation and change lag, and event latency and duplicates. If supplier access is revoked, an existing shortcut does not become an independently owned backup.
Avoid choosing a mechanism from its marketing description alone. A mirror has supported source and object constraints. A shortcut may preserve a dependency that violates an offline or retention requirement. A copy can drift if updates and deletes are ignored. A stream can deliver duplicate or late events. Troubleshoot the specific stage: source availability, authentication, transfer, target write, or consumer refresh. An exam question asking for minimal duplication may favor a reference; one requiring an independently retained historical snapshot may require a managed copy and retention design.
Scenario: A team needs historical supplier snapshots even after the supplier overwrites its current table. Is a shortcut alone sufficient?
Answer: No. A shortcut exposes the supported referenced data; it does not by itself create Northstar's independently retained history. Design an authorized snapshot or change-retention process with explicit keys, timestamps, and validation. A shortcut may still be useful as an access route into that process.
Recall: Which source changes can a simple append-only copy miss? What makes a retry safe when an ingestion job partially completed?
Choose between different data stores
A lakehouse supports lake-oriented engineering and open table access across engines. A warehouse serves relational analytics and SQL-centered dimensional workloads. An Eventhouse supports event and time-series analytics using KQL. These are workload choices rather than rankings. Consider the consuming languages, transaction requirements, update patterns, data shapes, operational skills, latency, and governance. A SQL analytics endpoint associated with a lakehouse does not turn every lakehouse operation into a warehouse operation; supported write and query surfaces differ.
Write down representative queries and data-change operations before selecting the store. Identify whether the team mainly prepares mixed files with Spark, maintains relational analytical tables with T-SQL, or explores high-volume events with time-window queries. Check connector support, security requirements, and semantic-model integration. Prototype the critical path with representative data and concurrency. Validate correctness and freshness first, then measure cost and response time. Multiple stores can cooperate, but every extra representation needs a clear owner and consistency contract.
Finance maintains conformed dimensions and sales facts with a SQL-skilled team, making a warehouse a strong candidate. Data science prepares supplier files with Spark and benefits from a lakehouse. Dispatch analyzes device events using KQL and benefits from an Eventhouse. The semantic model can present governed business measures without pretending these underlying workloads are identical. If Finance later requires a Spark transformation, that does not automatically invalidate the warehouse choice; an upstream lakehouse preparation stage may be appropriate if the data contract is explicit.
Language familiarity is evidence about team fit, but it is not the only requirement. A SQL preference should not obscure a strict event-latency need, and a Spark preference should not obscure a relational serving contract. Inspect the actual Fabric surface area rather than assuming all SQL Server or Spark features are available everywhere. Diagnose wrong-store symptoms by measuring the critical operations: ingestion latency, transformation complexity, transaction behavior, concurrency, and query shape. On the exam, select from stated workload constraints instead of always choosing the most familiar item.
Scenario: A dispatch team needs rapid KQL exploration of high-volume device events, while Finance needs reconciled dimensional reporting. Must both use the same store?
Answer: No. An Eventhouse can serve event analysis and a warehouse or lakehouse can supply curated financial data, with an explicit transformation and freshness contract between them where needed. Choose each serving path from its workload and avoid unnecessary copies without sacrificing correctness.
Recall: How does a lakehouse SQL analytics endpoint differ from a warehouse's writable SQL surface? Which measurements would show that a store choice meets the requirement?
Implement OneLake integration for Eventhouse and semantic models
OneLake integration can expose data from other Fabric experiences as Delta tables for reuse. Eventhouse OneLake availability makes supported KQL database data available through a Delta representation to other engines. Semantic-model OneLake integration exports supported Import table data to OneLake. These flows have different producers and timing. Neither should be confused with Direct Lake, where a semantic model reads prepared Delta data from OneLake. Draw the direction of data movement before reasoning about refresh, ownership, or security.
For Eventhouse, inspect the database's supported OneLake availability settings and the tables to expose; verify materialization status and end-to-end availability before attaching another consumer. For a semantic model, confirm capacity, tenant settings, and model permissions, enable supported OneLake integration, and run a model refresh so eligible Import data is written out. Inspect integration refresh history and the resulting tables. Then test the downstream engine's access and schema. Review whether security controls from the producing experience apply to the exported representation; do not infer inherited model RLS at the file level.
Northstar's operations model imports a small classification table maintained by an approved source. OneLake integration exposes that imported table for a separate Fabric workflow after the required refresh. Meanwhile, Eventhouse exposes delivery records through OneLake availability for lake-based analysis. A new Direct Lake model could read suitable Delta tables, but that is a third, consuming step. Calling all three “Direct Lake refresh” would obscure which producer must run and which permissions govern the downstream read. The acceptance test records a known new row at each boundary.
A feature switch being enabled does not prove that data is already materialized or current. Inspect refresh history, availability status, schema, and the identity reading the files. If KQL shows a new event while the Delta consumer does not, investigate the export or availability boundary before changing DAX. If an exported Import table exists but is stale, inspect the semantic-model refresh and integration history. Distinguish object availability, data freshness, and permission to read. Security and model calculations are not automatically equivalent to a raw Delta representation.
Scenario: An owner enables semantic-model OneLake integration but cannot find a newly added imported row in the exported Delta table. What lifecycle event and evidence should be checked?
Answer: Check that the model completed the required refresh after the row became available, then inspect OneLake integration history and the exported table with an authorized identity. Enabling the setting alone is not proof of a completed export. Separate an export failure from a downstream permission failure.
Recall: Which direction does semantic-model OneLake integration move data? How does that differ from a Direct Lake model reading a lakehouse table?
A source-to-consumer acceptance exercise
Create a paper design for Northstar with three source cards. The first card represents the order database and says that updates and cancellations are possible. The second represents a supplier Delta table whose owner can replace files. The third represents delivery events that can arrive late. For each card, choose a supported ingestion or access mechanism and write one failure that your design must detect. Do not use “refresh succeeded” as the only acceptance statement.
For orders, a useful check reconciles business keys and net totals after an update and a cancellation. For supplier data, a useful check verifies current authorization and documents whether historical snapshots are required. For events, a useful check compares event time with arrival time and tests how a delayed event affects an analysis window. Now add the semantic model and write the data freshness that a report consumer can actually expect. This final freshness is the combined result of all required stages, not the latency claimed by the first ingestion mechanism.
Suppose the order pipeline finishes at 08:00, the transformation finishes at 08:10, and an Import model last refreshed at 07:45. The report is not current to 08:10 merely because the warehouse is. A refresh or other supported consumption lifecycle must complete after the relevant data is available. Conversely, a report that reads a current source through a suitable live query path may have different freshness and load characteristics. The design should specify which behavior is intended and how it will be observed.
Finally, change one requirement: the supplier contract requires Northstar to stop reading data immediately when access is revoked. A design that silently retains unrestricted snapshots might violate that contract even if it improves availability. A design that depends on a shortcut should communicate the resulting availability dependency. Architecture choices are not isolated technical preferences; they encode ownership and business constraints. In exam questions, those constraints often determine the correct choice among several technically possible mechanisms.
Shared Fabric foundations
Choose an appropriate data store
| Store | Best fit | Key clue |
|---|---|---|
| Lakehouse | Open files/Delta, Spark engineering, mixed structured data | Multiple engines need the same OneLake data |
| Warehouse | Governed relational analytics and T-SQL | Dimensional SQL serving and transactions dominate |
| Eventhouse | High-ingest, time-series/log analytics with KQL | Low-latency event exploration and time windows dominate |
| Operational database | Application transactions | Point writes and application consistency, not analytical scans |
Do not choose from language preference alone. Consider concurrency, update pattern, governance, latency, file/table format, and consumers.
Choose between Dataflows Gen2, notebooks, KQL, and T-SQL for data transformation
Dataflow Gen2 is Power Query-based, visual, and approachable for reusable tabular shaping. Notebooks provide PySpark/Python/SQL flexibility and distributed control. T-SQL fits set-based relational transformations close to a warehouse. KQL fits telemetry, logs, semi-structured events, and time-series analysis in Eventhouse.
Dataflow Gen2 in depth
A useful Dataflow Gen2 review follows the data:
- Connect and authenticate to a source.
- Assign correct data types early; type drives comparison, arithmetic, and folding behavior.
- Select/rename columns and filter rows to reduce unnecessary data.
- Clean values: replace errors, trim/standardize text, handle nulls, and remove duplicates using an explicit key.
- Combine data: merge performs a join; append stacks compatible rows.
- Reshape: split/merge columns, pivot/unpivot, group, aggregate, and add derived columns.
- Define destination and write behavior, then monitor refresh.
Query folding pushes supported transformations back to the source. Keep foldable filters and projections early, inspect folding indicators/query plans when available, and avoid assuming every connector or step folds. A nonfolding custom operation can force much more data into the mashup engine.
Schema strategy matters. Fixed destination schema gives predictable downstream contracts; automatic or dynamic handling eases evolution but can surprise consumers. Treat added columns, removed columns, type changes, and renamed columns as different events with explicit policy. Dataflow destinations and update methods vary, so verify whether append, replace, or another supported behavior matches the target.
Use Dataflow Gen2 for analyst-friendly transformations and managed destinations; use a notebook when logic needs libraries, tests, complex algorithms, or distributed tuning. Use SQL/KQL when the data already lives in the corresponding engine and set-based work can remain close to storage.
Create and manage OneLake shortcuts
A shortcut is an independent OneLake object pointing to a target path. It avoids an extra copy and exposes supported external or internal data through the OneLake namespace. Deleting the shortcut does not delete the target; moving or deleting the target can break the shortcut.
Create table shortcuts in the lakehouse Tables area for supported table formats and file shortcuts in Files. Plan the cloud connection and credentials, source permissions, shortcut security, cache behavior, and schema evolution. The shortcut is read-through access, so source availability and network latency remain relevant.
Implement mirroring
Mirroring adds a source database or catalog to Fabric. Database mirroring continuously replicates supported operational data into OneLake Delta tables. Metadata mirroring synchronizes catalog metadata and uses shortcuts to open-format data in place. Open mirroring accepts change files written to a Fabric landing zone according to the published specification.
Choose a shortcut for selected open-format tables/folders, mirroring for a database/catalog as a unit, and pipeline/copy when custom cadence, complex shaping, or a non-OneLake destination is needed. Mirrored tables are not general-purpose writable replicas; change the source and let synchronization propagate.
Ingest data by using pipelines
Pipeline Copy Activity supports controlled movement across connectors and destinations. Parameterize dataset/path/table names, choose mappings deliberately, configure parallelism from measurements, and preserve source metadata. Use staging only when required by the connector or performance design.
Capture rows read/written/skipped, duration, throughput, run ID, watermark, and rejected records. Retry transient throttling or network faults with backoff; do not repeatedly retry invalid credentials or deterministic schema errors.
Data-store decision workshop
Prerequisites include the right Fabric capacity, workspace/item permissions, source connectivity, and a retention/security design. Procedure: quantify data volume and velocity, latency objective, update/delete frequency, transaction needs, query language, concurrency, file-format interoperability, and consumers; score each store; prototype the highest-risk query and load. A lakehouse is not automatically fastest for every SQL dashboard, and a warehouse is not the right landing zone for arbitrary binary files.
Scenario. A team receives Parquet telemetry plus curated sales dimensions. Land and engineer open telemetry in a lakehouse, serve governed star-schema BI from a warehouse when T-SQL concurrency dominates, and use Eventhouse when subsecond KQL exploration over incoming events is required. Do not force one store merely to avoid architecture decisions. Troubleshoot a poor fit by measuring ingestion delay, scan volume, concurrency waits, update complexity, and duplicated copies. Mini-lab: rank all four stores for (1) images plus JSON, (2) a finance star schema, (3) live device logs, and (4) order entry, explaining which requirement eliminated each alternative.
Shortcuts and mirroring
Failure lab. Create a test shortcut, query a known row count, revoke source read, and observe the error without changing the target. Restore permission, then rename or move a test target and diagnose the reference. Check shortcut path, connection identity, source ACL, supported format, Delta log, schema refresh/synchronization, and any cache or acceleration layer. A user may have access to the shortcut item but lack source data authorization through the chosen credential path.
Choose mirroring for low-friction replication or catalog-wide access when the source fits; choose shortcut for selected in-place open data; choose pipeline copy for scheduled/custom mappings and transformations. Do not write directly to a mirrored target as if it were the operational source. For stale data, inspect source change capture/retention, replication status, rejected tables or types, permissions, capacity, and lag. Mini-lab: insert, update, and delete one source row, record when each appears, pause or break connectivity, and prove the monitoring evidence identifies the last synchronized point.
Pipeline ingestion
Tune from measurements: source query selectivity, partitioned reads, parallel copies, ForEach concurrency, staging, sink batch/commit behavior, network, and capacity. More parallelism can throttle a source or create small files. Retry transient network or throttling faults with backoff; repair authentication, mapping, constraint, and type errors before rerun. Mini-lab: copy two date partitions through one parameterized child pipeline, force one deterministic mapping error, verify only that partition fails, correct it, and rerun without duplicating the successful partition.
Design and implement full and incremental data loads
A full load replaces or rebuilds the target from the complete selected source. It is simple and self-reconciling but can be expensive and disruptive. An incremental load selects only changes since a trusted position, reducing work while increasing state-management complexity.
| Change signal | Strength | Main risk |
|---|---|---|
| Monotonic ID | Simple | Updates and deletes may be invisible |
| Modified timestamp | Widely available | Clock precision, ties, and late commits |
| Source CDC/version | Captures inserts, updates, and often deletes | Retention and ordering must be managed |
| File/path partition | Efficient for append-oriented sources | Rewritten or late partitions need reconciliation |
Persist a high-water mark only after the target transaction or durable write succeeds. Read an overlap window when timestamps can collide or arrive late, then deduplicate by business key and source version. A safe retry either overwrites a deterministic partition or performs an idempotent upsert.
MERGE dbo.Customer AS target
USING #CustomerStage AS source
ON target.CustomerBusinessKey = source.CustomerBusinessKey
WHEN MATCHED AND source.ModifiedAt > target.ModifiedAt THEN
UPDATE SET CustomerName = source.CustomerName,
ModifiedAt = source.ModifiedAt
WHEN NOT MATCHED THEN
INSERT (CustomerBusinessKey, CustomerName, ModifiedAt)
VALUES (source.CustomerBusinessKey, source.CustomerName, source.ModifiedAt);
MERGE is not automatically correct: the staged source must contain at most one winning row per target key, delete semantics must be explicit, and concurrency must be tested. Periodic full or bounded reconciliation detects changes missed by the incremental signal.
Full and incremental load field guide
Sources and provenance
The measured objectives follow the DP-600 blueprint effective July 21, 2026. Product references were retrieved for this edition; the book manifest records the actual retrieval time and SHA-256 of each downloaded HTML response. Examples and scenarios are original teaching material. Shared field guides reuse this project’s original DP-700 writing where the underlying Fabric concepts apply. Microsoft Learn remains authoritative, especially for current feature support and capacity limits.
- Study guide for Exam DP-600: Implementing Analytics Solutions Using Microsoft Fabric
- Data source management - Microsoft Fabric
- OneLake catalog overview - Microsoft Fabric
- Introduction to Microsoft Fabric Real-Time hub - Microsoft Fabric
- Turn on OneLake availability for an eventhouse - Microsoft Fabric
- Learn about Microsoft OneLake Delta table integration in Power BI and Microsoft Fabric - Microsoft Fabric
- OneLake, the unified data lake - Microsoft Fabric
- Comparing Copy Job, Mirroring, Copy Activity, and Eventstreams in Microsoft Fabric - Microsoft Fabric
- Load Tables in a Dimensional Model - Microsoft Fabric
- Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric
- Unify Data Sources With OneLake Shortcuts - Microsoft Fabric
- Mirroring - Microsoft Fabric
- Stateful processing with Structured Streaming - Microsoft Fabric