Registry / data / maggma

maggma

JSON →
library0.72.1pypypi✓ verified 89d ago

Maggma is a framework to build scientific data processing pipelines, handling data from diverse sources like databases, Azure Blobs, and local files, up to REST APIs. It provides core abstractions, `Store` and `Builder`, for modular ETL-like operations. The `Store` interface often mimics PyMongo syntax, enabling consistent data access across different backends. Actively developed by the Materials Project, it is currently at version 0.72.1 and requires Python 3.9+.

pip install maggma
INSTALL
IMPORT
SIG · MAGGMA
M
maggma
datapythonv0.72.1
Install
17.4s avg
Import
2160ms
Disk
259MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.9–3.13
musl
3.9–3.13
Install & Compatibility
Where this runs
tested against v0.72.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 2.230s · 256MB
glibc
py 3.10–3.920 runs
installs and imports cleanly · install 17.4s · import 2.089s · 246MB
259MB installed
● package 259MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

MemoryStore
✓ from maggma.stores import MemoryStore
MongoStore
✓ from maggma.stores import MongoStore
Builder
✓ from maggma.builders import Builder
Store
✓ from maggma.core import Store

This quickstart demonstrates the core concepts of Maggma: defining data as a list of dictionaries, creating a `Store` (using `MemoryStore` for simplicity), connecting to it, adding data using the `update` method, and querying data. It highlights the use of a `key` field for unique document identification. A commented-out example for `MongoStore` is included to illustrate persistent storage.

import os from maggma.stores import MemoryStore # Sample data turtles = [ {"name": "Leonardo", "color": "blue", "tool": "sword"}, {"name": "Donatello", "color": "purple", "tool": "staff"}, {"name": "Michelangelo", "color": "orange", "tool": "nunchuks"}, {"name": "Raphael", "color": "red", "tool": "sai"} ] # Create a MemoryStore (in-memory, data not persistent) # 'key' argument specifies the unique identifier for documents store = MemoryStore(key="name") # Connect to the store (for MemoryStore, this just initializes it) store.connect() # Add data to the store using update # upsert=True means insert if not found, update if found store.update(turtles, key_field='name', upsert=True) # Query the store print(f"Total documents: {store.count()}") print(f"Blue turtle: {store.query(criteria={'color': 'blue'}).current()}") # Find distinct values print(f"Distinct colors: {list(store.distinct(field='color'))}") # Close the store connection (important for persistent stores) store.close() # Example of using a persistent store (e.g., MongoStore) # Requires a MongoDB instance running and pymongo installed. # uri = os.environ.get('MONGO_URI', 'mongodb://localhost:27017/test_db') # from maggma.stores import MongoStore # mongo_store = MongoStore(collection_name='my_collection', database_name='test_db', host=uri, key='name') # try: # mongo_store.connect() # mongo_store.update(turtles, key_field='name', upsert=True) # print(f"MongoStore count: {mongo_store.count()}") # finally: # mongo_store.close()
maggma --version
Debug
Known issues
breakingThe `maggma.api` module has been deprecated and will be migrated. This could significantly impact projects relying on Maggma's built-in API functionalities.
fix
Review the Changelog and documentation for `v0.72.0` for migration details. Projects should update their API implementations to align with the new recommended patterns.
affects: v0.72.0 and later
gotchaMaggma's `Store` classes provide a unified interface that resembles PyMongo. However, not all `Store` implementations (e.g., FileStore, S3Store) support the full breadth of PyMongo's query capabilities or advanced features like aggregation pipelines. Over-reliance on PyMongo-specific syntax with non-Mongo backends can lead to unexpected behavior or unsupported operations.
fix
Consult the specific `Store` class documentation for its supported query features. Stick to basic `query`, `count`, `distinct` operations for maximum compatibility across different `Store` types. For advanced queries, consider processing data after retrieval or using a `MongoStore`.
affects: All versions
gotchaUsing `MemoryStore` is suitable for testing and quick examples, but it is not persistent. Any data added to a `MemoryStore` will be lost when the Python interpreter closes or the `Store` object is garbage collected.
fix
For persistent storage, use a dedicated `Store` implementation like `MongoStore`, `FileStore`, `GridFSStore`, or `S3Store`. Ensure proper connection and disconnection for persistent stores.
affects: All versions
gotchaDocuments added to a `Store` must have a unique identifier, specified by the `key` argument during `Store` initialization (defaulting to `task_id`). If duplicates are inserted with the same key and `upsert=True`, the old document will be overwritten. If `upsert=False`, it may lead to errors depending on the store implementation.
fix
Always ensure your data has a robust, unique identifier for the `key` field. When performing `update` operations, be mindful of the `key_field` and `upsert` parameters to avoid unintended data overwrites or errors.
affects: All versions
breakingMaggma, particularly components like `OpenDataStore`, has reported compatibility issues with `numpy` version 2.0. This can lead to unexpected errors or broken functionality.
fix
Pin your `numpy` version to `<2.0` (e.g., `numpy<2.0`) in your project's dependencies until official `maggma` compatibility with `numpy` 2.0 is confirmed and released.
affects: Reported with `numpy` 2.0 (maggma v0.72.1).
Errors
Common errors & fixes
StoreError: No field 'last_updated' in store document.
A Store is configured with a `last_updated_field` (defaulting to 'last_updated') but the documents being processed do not contain this field, which is essential for incremental building and tracking updates.
fix
Ensure all documents in your source Store have a field named 'last_updated' (or the custom `last_updated_field` you've specified) containing a datetime object, or set `store.last_updated_field = None` if incremental updates based on time are not needed.
AttributeError: 'MongoStore' object has no attribute 'find_one'
The `maggma.Store` interface, while mimicking PyMongo syntax, provides its own methods like `query_one` for querying single documents, rather than directly exposing the `find_one` method from the underlying PyMongo collection.
fix
Replace `find_one` with `query_one` when attempting to retrieve a single document from a `maggma.Store` object.
pymongo.errors.ConfigurationError: Server at localhost:27017 reports wire version X, but this version of PyMongo requires at least Y (MongoDB Z.0).
The version of PyMongo installed (which `maggma` uses for `MongoStore`) is incompatible with the version of the MongoDB server you are trying to connect to.
fix
Upgrade your MongoDB server to a version compatible with your PyMongo client (e.g., MongoDB Z.0 or newer) or downgrade your PyMongo library to a version that supports your MongoDB server.
ModuleNotFoundError: No module named 'maggma.stores.some_non_existent_module'
The user is attempting to import a specific `Store` or `Builder` class from an incorrect or non-existent module path within the `maggma` library.
fix
Consult the `maggma` documentation to find the correct import path for the desired class, e.g., `from maggma.stores import MongoStore` or `from maggma.builders import MapBuilder`.
Upgrade
Version history
0.72.1latest on PyPI · released Feb 11, 2026
Audit
Dependencies
pydanticrequiredData validation and settings management.
pymongorequiredPrimary MongoDB interaction, often used as a backend for 'Store' classes.
montyrequiredUtility functions for materials science, a common dependency in the Materials Project ecosystem.
pandasrequiredData manipulation and analysis, used in some 'Store' and 'Builder' implementations.
numpyrequiredNumerical operations, a fundamental data science library.
boto3optionalAWS SDK for Python, enabling S3 and Azure Blob 'Store' functionality.
sshtunneloptionalSSH tunneling capabilities, often used for secure database connections.
Agent activity
13 hits · last 30 days
node
12
OpenAI (training)
1
Resources
maggma — pip install maggma · libregistry