Migrate content between two sites#
In this tutorial, we will write a script that copies news items and their images from a source to a target. We will rehearse it with a dry run, watch it fail part way and recover, and verify the result by counting. At the end, you will have a migration script that you can run again safely, because it skips what is already there.
To keep the tutorial light on your computer, we will run one Plone site, with the folders /source and /target standing for the two sites.
The script has two clients, so moving to two real sites means changing two URLs.
You need uv and Docker, and you should have followed Your first script.
Start a Plone site#
Get the Compose file, and start the site.
curl -O https://raw.githubusercontent.com/plone/plone-client/main/docker-compose.yml
docker compose up --wait
Open http://localhost:3000 to see it, with the user admin and the password admin.
These credentials belong to the throwaway Compose site; never reuse them on a real site.
Run a Plone site for testing explains what it has.
Prepare the source#
Create a project.
uv init --bare migration
cd migration
uv add plone-client
Create a file named prepare.py, which creates the two folders and three news items, each with a lead image stored inside it.
from pathlib import Path
from plone_client import blocks
from plone_client import operations
from plone_client import PloneClient
import base64
PIXEL = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGNgaPgPAAIDAYBB0fBwAAAAAElFTkSuQmCC"
NEWS = [
("spring", "Spring meeting", "We met in April."),
("summer", "Summer sprint", "We wrote code in July."),
("autumn", "Autumn release", "The release came out in October."),
]
Path("photo.png").write_bytes(base64.b64decode(PIXEL))
with PloneClient("http://localhost:8080/Plone", username="admin", password="admin") as site:
for name in ("source", "target"):
site.content.create("/", "Document", id=name, title=name.title())
site.workflow.transition(f"/{name}", "publish")
for item_id, title, description in NEWS:
operations.create_with_files(
site,
"/source",
"News Item",
["photo.png"],
id=item_id,
title=title,
description=description,
publish="publish",
)
print("prepared")
uv run python prepare.py
prepared
Read the source#
Create a file named migrate.py.
The source client is read-only, so nothing in this script can change the source by mistake.
records reads what a news item needs: its fields, and the images inside it.
from pathlib import Path
from plone_client import operations
from plone_client import PloneClient
from plone_client import PloneOperationError
import sys
import tempfile
URL = "http://localhost:8080/Plone"
AUTH = {"username": "admin", "password": "admin"}
FIELDS = ("title", "description", "subjects", "blocks", "blocks_layout")
TRANSITION = "publish"
SOURCE = "/source"
TARGET = "/target"
def records(source):
for entry in source.search.iter(SOURCE, portal_type="News Item", sort_on="sortable_title"):
item = source.content.get(entry["@id"], include_items=False)
images = list(source.content.children(entry["@id"], portal_type="Image"))
yield item, images
def copy_item(source, target, item, images, folder):
files = []
for image in images:
path = Path(folder) / image["@id"].rsplit("/", 1)[-1]
path.write_bytes(source.files.download(image["@id"], field="image"))
files.append(path)
data = {name: item[name] for name in FIELDS if item.get(name)}
return operations.create_with_files(
target, TARGET, "News Item", files, data=data, id=item["id"], publish=TRANSITION
)
def main():
apply = "--apply" in sys.argv
with (
PloneClient(URL, read_only=True, **AUTH) as source,
PloneClient(URL, dry_run=not apply, **AUTH) as target,
tempfile.TemporaryDirectory() as folder,
):
for item, images in records(source):
if target.exists(f"{TARGET}/{item['id']}"):
print("skip", item["id"])
continue
try:
result = copy_item(source, target, item, images, folder)
except PloneOperationError as error:
print("FAILED", item["id"], error.result.error["message"])
for created in reversed(error.result.created):
print(" left behind:", created["@id"])
return 1
print("copy", item["id"], len(result.created), "items,", len(result.requests), "requests")
return 0
if __name__ == "__main__":
sys.exit(main())
Notice that the script uses create_with_files() on the target: it finds where this site keeps the lead image, which may differ from the source.
Note
Internal links inside blocks point to the source.
Rewrite them if the two sites have different paths: iter_links() finds them.
Rehearse with a dry run#
Run it without --apply.
The target client records its changes and sends none.
uv run python migrate.py
copy autumn 2 items, 9 requests
copy spring 2 items, 9 requests
copy summer 2 items, 9 requests
Each item takes one GET of its type and four changes: the item, the image, the lead image, and the transition.
Check that the target is still empty.
from plone_client import PloneClient
with PloneClient("http://localhost:8080/Plone", username="admin", password="admin") as site:
for name in ("/source", "/target"):
news = site.search.count(name, portal_type="News Item")
images = site.search.count(name, portal_type="Image")
published = site.search.count(name, portal_type="News Item", review_state="published")
print(name, news, "news items,", images, "images,", published, "published")
uv run python count.py
/source 3 news items, 3 images, 3 published
/target 0 news items, 0 images, 0 published
Watch it fail#
Real migrations stop part way. Make this one stop with a typing mistake in the name of the transition.
sed -i.bak 's/TRANSITION = "publish"/TRANSITION = "publsh"/' migrate.py
uv run python migrate.py --apply
FAILED autumn No workflow provides the '${action_id}' action.
left behind: http://localhost:8080/Plone/target/autumn/photo.png
left behind: http://localhost:8080/Plone/target/autumn
The site refused the transition.
By then, the item and its image existed, and PloneOperationError carries a result that says so, so the script can name what it left behind.
Recover#
Delete what the failure left, and correct the mistake. Deleting the news item also deletes the image inside it.
from plone_client import PloneClient
with PloneClient("http://localhost:8080/Plone", username="admin", password="admin") as site:
site.content.delete("/target/autumn")
print("cleaned")
uv run python clean.py
sed -i.bak 's/TRANSITION = "publsh"/TRANSITION = "publish"/' migrate.py
cleaned
Migrate#
Run it for real.
uv run python migrate.py --apply
copy autumn 2 items, 6 requests
copy spring 2 items, 6 requests
copy summer 2 items, 6 requests
Run it again.
uv run python migrate.py --apply
skip autumn
skip spring
skip summer
Notice that the second run skipped everything, which is what lets you run a migration again after any failure.
Verify by counting#
uv run python count.py
/source 3 news items, 3 images, 3 published
/target 3 news items, 3 images, 3 published
The two lines match. Check one lead image as well.
from plone_client import PloneClient
with PloneClient("http://localhost:8080/Plone", username="admin", password="admin") as site:
item = site.content.get("/target/summer")
print(item["review_state"], item["preview_image_link"]["@id"])
uv run python lead.py
published http://localhost:8080/Plone/target/summer/photo.png
The link points to the copy inside the target, not to the source.
You have rehearsed a migration, recovered from a failure, and proved the result, with a script that is safe to run again.
Move to two real sites#
Change the URL and AUTH of each client, and the paths of SOURCE and TARGET.
Give each site a profile, and use PloneClient.from_profile("source", read_only=True) to keep passwords out of the script.
Clean up#
docker compose down -v
If a step failed, docker compose down -v still resets the site; then start again from the top.
Next steps#
Work with large sites keeps a migration fast on large sites.
Operations explains the result of an operation.
Troubleshoot helps when something does not work.