Registry / aws / emr-notebooks-magics

emr-notebooks-magics

library0.2.4pypypi✓ verified 29d ago

EMR Notebooks Magics is a Python library providing Jupyter magics specifically for Amazon EMR Notebooks. These magics enhance the interactive experience by allowing direct interaction with EMR cluster resources, such as mounting S3 workspaces and executing other notebooks. The library is currently at version 0.2.4 and is actively maintained by AWS Labs, with updates tied to EMR Notebooks and EMR Studio feature development.

aws

pip install emr-notebooks-magics

package ↗

INSTALL

IMPORT

SIG · EMR-NOTEBOOKS-MAGI

emr-notebooks-magics

awspythonv0.2.4

Install

4.8s avg

Import

—

Disk

51MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.2.4 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 52.9MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 4.8s · import 0.000s · 53MB

51MB installed

● package 51MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

%mount_workspace_dir

✓ %mount_workspace_dir .

✗ from emr_notebooks_magics import mount_workspace_dir

This library provides IPython magics, not traditional Python modules for direct import. Magics are used directly in Jupyter cells with a '%' prefix after installation and kernel restart.

%generate_s3_download_url

✓ %generate_s3_download_url s3://my_bucket/path/to/object

✗ import emr_notebooks_magics; emr_notebooks_magics.generate_s3_download_url(...)

Magics are invoked directly in Jupyter cells; they are not imported as standard Python functions.

This quickstart demonstrates the core functionalities of EMR Notebooks Magics: mounting a workspace directory, generating a presigned S3 download URL, and executing another notebook. These commands are run directly in a Jupyter cell within an EMR Notebook environment.

# After installing the package and restarting the kernel in an EMR Notebook. # Mount the entire Workspace directory to the EMR cluster instance %mount_workspace_dir . # Generate a presigned URL for an S3 object # Replace with your actual S3 bucket and object key %generate_s3_download_url s3://my-example-bucket/path/to/my-file.csv # Execute another notebook in the background # Make sure 'another_notebook.ipynb' exists in your workspace %execute_notebook another_notebook.ipynb

Debug

Known issues

gotchaAfter installing `emr-notebooks-magics` (e.g., via `pip install`), you must restart the Jupyter kernel before the magics become available. Installing via bootstrap actions is not supported.

fix

Restart the Jupyter kernel (`Kernel -> Restart Kernel`) after installation. Do not attempt to install it via EMR bootstrap actions as the notebook environment is installed later.

affects: All versions

breakingThe `%mount_workspace_dir` magic primarily functions with Python 3 kernels. Spark executors do not have access to the mounted directory when used with a Python 3 kernel.

fix

Be aware of this limitation. If Spark executors need access to data, ensure data is directly accessible via S3 paths or other cluster-wide storage, rather than relying solely on a Python kernel's mounted workspace.

affects: All versions

gotchaEnabling write access with `%mount_workspace_dir` is irreversible and applies changes directly to your S3 Workspace. By default, mounts are read-only.

fix

Always enable S3 versioning on your S3 Workspace bucket as a precaution if you enable write access. Strongly consider keeping mounts read-only unless write access is absolutely necessary.

affects: All versions

gotchaUsing magics like `%execute_notebook` requires specific IAM permissions for the EMR-EC2 instance role. Lack of proper S3 read access for the EC2 instance profile will lead to failures.

fix

Ensure that your EMR cluster's EC2 instance profile (e.g., `EMR_EC2_DefaultRole`) has `AmazonElasticMapReduceforEC2Role` managed policy or equivalent S3 read permissions. For `%execute_notebook`, additional permissions might be needed.

affects: All versions

breakingEMR Serverless and Amazon EMR on EKS clusters do not support EMR Notebooks magics, including those from `emr-notebooks-magics`.

fix

These magics are only compatible with traditional EMR clusters. For EMR Serverless, manage Spark configurations directly via the `SparkContext` object as a workaround for some functionalities.

affects: All versions

Upgrade

Version history

0.2.4latest on PyPI · released Jun 30, 2023

Audit

Dependencies

s3fs-fuseoptionalRequired for mounting S3 workspaces via %mount_workspace_dir.

goofysoptionalAlternative FUSE-based filesystem for S3, can be used with %mount_workspace_dir.

Agent activity

11 hits · last 30 days

node

Resources

packagehttps://pypi.org/project/emr-notebooks-magics/ ↗