Using CIM Incrementals¶

In power system operations and planning, network models are constantly evolving. Equipment is added, removed, or modified; switch states change; and operational parameters are updated. Rather than exchanging complete network models every time a change occurs, CIM supports incremental updates - transmitting only what has changed.

CIMantic Graphs automatically tracks all model changes in an incrementals dictionary, enabling efficient model synchronization, change auditing, and state management.

What Are Incrementals?¶

Incrementals (also called "difference models" or "delta models") represent changes to a CIM model without requiring the entire model to be re-transmitted. They are essential for:

Use Cases:¶

1. Real-Time Operations - SCADA systems update switch positions, measurements change, and topology evolves
2. Model Synchronization - Multiple applications need to stay synchronized with a master model
3. Change Tracking - Audit trails of who changed what and when
4. Efficient Communication - Minimize bandwidth by sending only changes, not full models
5. Conflict Resolution - Detect and resolve conflicting changes from multiple sources
6. Undo/Redo - Maintain change history for rollback capabilities

Types of Incrementals:¶

CIMantic Graphs tracks two types of incremental changes:

1. Forward Differences - Changes to apply to transform model from state A to state B

   • New objects created
   • Attribute values changed
   • Objects deleted

2. Reverse Differences - Changes to apply to transform model from state B back to state A

   • Inverse of forward differences
   • Enables undo operations
   • Supports conflict detection

In [1]:

import os
os.environ['CIMG_CIM_PROFILE'] = 'cimhub_2023'
import cimgraph.data_profile.cimhub_2023 as cim

import os os.environ['CIMG_CIM_PROFILE'] = 'cimhub_2023' import cimgraph.data_profile.cimhub_2023 as cim

Example: Tracking Changes with Incrementals¶

Let's load an existing model and create some new objects to see how incrementals are automatically tracked.

First, set up the environment and load a model:

In [2]:

from cimgraph.databases import XMLFile
file = XMLFile(filename='../../sample_models/ieee13.xml')

from cimgraph.databases import XMLFile file = XMLFile(filename='../../sample_models/ieee13.xml')

Now load a CIM XML file into a database connection:

In [3]:

from cimgraph.models import FeederModel
network = FeederModel(container=cim.Feeder(), connection=file)

from cimgraph.models import FeederModel network = FeederModel(container=cim.Feeder(), connection=file)

Create a FeederModel that will automatically track all changes:

In [4]:

line = network.create(cim.ACLineSegment, name='new_line')
line.pprint()

line = network.create(cim.ACLineSegment, name='new_line') line.pprint()

{
    "@id": "4d29f925-8390-4e1d-a877-fdcf5ac624b1",
    "@type": "ACLineSegment",
    "name": "new_line"
}

Creating New Objects¶

When you use the network.create() method, the object is automatically added to the forward differences:

In [6]:

network.incrementals['forwardDifferences'][cim.ACLineSegment][line.uri()]

network.incrementals['forwardDifferences'][cim.ACLineSegment][line.uri()]

Out[6]:

{'mRID': '4d29f925-8390-4e1d-a877-fdcf5ac624b1', 'name': 'new_line'}

Summary¶

Incrementals are a powerful feature of CIM that enable efficient model synchronization and change management:

Key Concepts:¶

1. Automatic Tracking - CIMantic Graphs automatically tracks all model changes
2. Forward Differences - Changes to apply (old state → new state)
3. Reverse Differences - Changes to undo (new state → old state)
4. Efficient Communication - Send only what changed, not entire models
5. Change Audit - Complete history of what changed and when

Incremental Types:¶

• Create - New objects appear in forward differences with all attributes
• Modify - Changed attributes in forward, old values in reverse
• Delete - None in forward, full object in reverse for restoration

Common Use Cases:¶

• Real-time SCADA updates
• Model synchronization across applications
• Conflict detection and resolution
• Undo/redo functionality
• Change auditing and logging
• Efficient network bandwidth usage

Next Steps:¶

The final notebook in this series covers Units and Quantities - how CIM handles physical measurements with automatic unit conversion.

With incrementals, you now have the tools to build sophisticated, real-time power system applications that efficiently manage model state!

Best Practices for Incrementals¶

1. Use Incrementals for Real-Time Updates¶

Instead of re-sending entire network models, send only what changed:

# After SCADA updates switch position
switch.open = False
network.update(switch, open=False)

# Send just the incremental
send_to_subscribers(network.incrementals['forwardDifferences'])

# Clear after sending
network.incrementals['forwardDifferences'].clear()

2. Timestamp Your Incrementals¶

Add metadata to track when changes occurred:

from datetime import datetime

incremental_message = {
    'timestamp': datetime.utcnow().isoformat(),
    'source': 'SCADA_System_A',
    'changes': network.incrementals['forwardDifferences']
}

3. Validate Before Applying¶

Validate received incrementals before applying to your model:

def validate_incremental(incremental, profile):
    """Validate incremental matches profile schema"""
    for cim_class, objects in incremental.items():
        # Check class exists in profile
        if not hasattr(profile, cim_class.__name__):
            raise ValueError(f"Unknown class: {cim_class}")
        
        # Validate attributes
        for uri, attributes in objects.items():
            if attributes is not None:
                # Check all attributes are valid for this class
                valid_attrs = {f.name for f in fields(cim_class)}
                for attr in attributes.keys():
                    if attr not in valid_attrs:
                        raise ValueError(f"Invalid attribute {attr} for {cim_class}")

4. Handle Conflicts¶

When multiple sources update the same object, detect and resolve conflicts:

def detect_conflicts(local_incremental, received_incremental):
    """Detect if same objects modified in both incrementals"""
    conflicts = []
    
    for cim_class in received_incremental:
        if cim_class in local_incremental:
            # Check for overlapping URIs
            local_uris = set(local_incremental[cim_class].keys())
            received_uris = set(received_incremental[cim_class].keys())
            
            conflicts.extend(local_uris & received_uris)
    
    return conflicts

5. Use Reverse Differences for Undo¶

Implement undo functionality using reverse differences:

def undo_last_change(network):
    """Undo the last set of changes"""
    reverse = network.incrementals['reverseDifferences']
    
    # Apply reverse differences
    for cim_class, objects in reverse.items():
        for uri, attributes in objects.items():
            if attributes is None:
                # Was a creation, now delete
                network.delete_by_uri(uri)
            else:
                # Restore old values
                network.update_by_uri(uri, **attributes)
    
    # Swap forward and reverse
    network.incrementals['forwardDifferences'], \
    network.incrementals['reverseDifferences'] = \
    network.incrementals['reverseDifferences'], \
    network.incrementals['forwardDifferences']

Working with Incrementals¶

Clearing Incrementals¶

After processing or exporting incrementals, clear them to start tracking a new set of changes:

# Clear all tracked changes
network.incrementals['forwardDifferences'].clear()
network.incrementals['reverseDifferences'].clear()

Exporting Incrementals¶

Export incrementals to XML or JSON for transmission to other systems:

# Export forward differences to XML
from cimgraph.databases import XMLFile

incremental_file = XMLFile(filename='model_changes.xml')
incremental_file.export_incrementals(
    incrementals=network.incrementals['forwardDifferences'],
    profile=cim
)

Applying Incrementals¶

Receive incrementals from another system and apply them to your model:

# Load incrementals from file
received_changes = incremental_file.import_incrementals()

# Apply changes to network
for cim_class, objects in received_changes.items():
    for uri, attributes in objects.items():
        if attributes is None:
            # Deletion
            network.delete_by_uri(uri)
        else:
            # Create or update
            network.update_by_uri(uri, **attributes)

In [ ]:

# Delete the line
network.delete(line)

# Forward difference shows None (deletion marker)
print("Forward difference (deletion):")
fwd_diff = network.incrementals['forwardDifferences'][cim.ACLineSegment].get(line.uri())
print(f"  {line.uri()}: {fwd_diff}")

# Reverse difference contains full object state for restoration
print("\\nReverse difference (full object for restoration):")
rev_diff = network.incrementals['reverseDifferences'][cim.ACLineSegment].get(line.uri())
if rev_diff:
    print(f"  Can restore: {rev_diff}")

# Delete the line network.delete(line) # Forward difference shows None (deletion marker) print("Forward difference (deletion):") fwd_diff = network.incrementals['forwardDifferences'][cim.ACLineSegment].get(line.uri()) print(f" {line.uri()}: {fwd_diff}") # Reverse difference contains full object state for restoration print("\\nReverse difference (full object for restoration):") rev_diff = network.incrementals['reverseDifferences'][cim.ACLineSegment].get(line.uri()) if rev_diff: print(f" Can restore: {rev_diff}")

Deleting Objects¶

When you delete an object, the forward difference shows the deletion, and the reverse difference contains the full object state for restoration:

In [ ]:

# Modify the line we just created
network.update(line, name='modified_line_name', description='Updated description')

# Check forward differences (new values)
print("Forward difference (new value):")
print(network.incrementals['forwardDifferences'][cim.ACLineSegment][line.uri()])

# Check reverse differences (old values for undo)
print("\\nReverse difference (old value for undo):")
print(network.incrementals['reverseDifferences'][cim.ACLineSegment][line.uri()])

# Modify the line we just created network.update(line, name='modified_line_name', description='Updated description') # Check forward differences (new values) print("Forward difference (new value):") print(network.incrementals['forwardDifferences'][cim.ACLineSegment][line.uri()]) # Check reverse differences (old values for undo) print("\\nReverse difference (old value for undo):") print(network.incrementals['reverseDifferences'][cim.ACLineSegment][line.uri()])

Additional Examples¶

Modifying Existing Objects¶

When you modify an existing object, both forward and reverse differences are tracked:

Incrementals Dictionary Structure¶

The network.incrementals dictionary has the following structure:

{
    'forwardDifferences': {
        <CIM_Class>: {
            <object_uri>: {<attribute>: <value>, ...},
            ...
        },
        ...
    },
    'reverseDifferences': {
        <CIM_Class>: {
            <object_uri>: {<attribute>: <value>, ...},
            ...
        },
        ...
    }
}

• forwardDifferences: Changes to apply to get from old state → new state
• reverseDifferences: Changes to apply to get from new state → old state (undo)
• Objects are organized by class type
• Each object is identified by its URI
• Only changed attributes are included (not the full object)

Inspecting the Incrementals Dictionary¶

The network.incrementals dictionary tracks all changes. Let's examine what was recorded: