Replace URLs

Replaces URLs in content in the database with the correct cloud storage URL. Please read this whole article before using.

wp mediacloud:storage replace [--dry-run] [--local] [--imgix] [--imgix-domain=<string>] [--imgix-key=<string>] [--cdn=<string>] [--doc-cdn=<string>] [--batch-size=<number>] [--sleep=<number>]

Argument	Type	Optional	Description
`dry-run`	switch	yes	Simulate the search and replace.
`local`	switch	yes	Revert to local URLs regardless of current cloud storage settings.
`imgix`	switch	yes	Generate imgix URLs, use this if you are trying to switch back from imgix. To use this switch, you should have an imgix domain and/or signing key saved in imgix settings, otherwise use the `--imgix-domain` and `--imgix-key` arguments.
`imgix-domain`	string	yes	The imgix domain to use, if not using what is saved in the settings.
`imgix-key`	string	yes	The imgix signing key to use, if not using what is saved in the settings.
`cdn`	string	yes	If you are trying to rollback from a setup that used a CDN, specify the CDN here, including the https:// part.
`doc-cdn`	string	yes	If you are trying to rollback from a setup that used a doc CDN, specify the doc CDN here, including the https:// part.
`batch-size`	number	yes	The number of attachments to process in a batch.
`sleep`	number	yes	The amount of time, in milliseconds, to sleep between replacements. Will slow down processing, but reduce database CPU usage. Default is 250, use 0 to disable.

Before You Use This

This command will update the content in your database with the correct URLs for items in your media library. There are a lot of caveats to using it and it’s not as straight forward as it may seem for a lot of different technical reasons.

In the most basic terms, the way the command works is that it iterates through all the items in your media library, figures out the local URL for the item and then maps that to the current cloud storage URL for that item. It then searches and replaces through your WordPress database, replacing the old URL with the new one.

As to why you would want to run this command to begin with depends on how you want Media Cloud to work. By default, Media Cloud does realtime URL replacement. This means as users view your content, Media Cloud is filtering the content and replacing URLs where appropriate. There may be the case that this behavior isn’t what you want. For some sites with huge (we are talking multi-gigabyte) posts and post meta tables, Media Cloud can be slow. Or, more specifically, the database query it uses can be slow. This is actually a pretty rare circumstance, even with mammoth databases, and has more to do with how the HTML that Media Cloud is filtering is constructed. Either way, if this is something you are experiencing, or you are unreasonably paranoid about how many database queries Media Cloud is making, you can disable this behavior in Cloud Storage settings. But, disabling this setting requires that you run this search and replace command so that your content has the correct URLs.

Note that if you are using Imgix, disabling the realtime URL replacement may lead to weird behavior.

Important: This command only works with items in your Media Library, it will not replace URLs in
third party plugins that do not store things there.

Handling the Edge Cases

You can, generally speaking, run this command every time that you make a change to your cloud storage settings, but there are some changes that the command won’t be able to account for:

If you are using a CDN and decide to change it later
If you change the Imgix domain and/or signing key
You enable transfer acceleration for Amazon S3
You have been using imgix but have since disabled it

In those cases you will have to provide some hints to the command via the --cdn, --doc-cdn, --imgix, --imgix-domain and --imgix-key arguments.

For example, let’s say we have decided to change our CDN. We would then use the command:

wp mediacloud:storage replace --cdn=oldcdn.mydomain.com

Doing this, the command will replace both the local (non-cloud) URL and the specified oldcdn.mydomain.com URLs with the new current URL for the media item.

For imgix, in the case that we were using imgix but have since disabled it, we must specify the --imgix switch. This switch, however, will only work if the imgix domain and signing key (if used) are still saved in the cloud storage settings (the imgix feature is turned off). If not, we need to specify the domain and key (if used) with the appropriate arguments:

wp mediacloud:storage replace --imgix-domain=mydomain.imgix.net --imgix-key=12341234

Going Back to Local URLs

If you want to revert all of your URLs back to your local, non-cloud, URLs, you can simply specify the --local switch. This basically makes the command work in reverse.

Note that the above edge cases still apply.

Command Performance

Running this command can cause your database to break a serious sweat and use a lot of CPU. By default the command will sleep for 250 milliseconds between URL replacements, which drops CPU usage but makes the command slower. You can disable this sleep by passing --sleep=0 but just be aware that it may tie up your server. If the default isn’t enough, you can specify a higher value to lower CPU usage.

Additionally, the command can consume quite a bit of memory. By default, the command will break up all of the media items you are processing into batches of 400. In our testing, on a 64-bit system, this uses about 350MB of memory for the entire run. That value scales almost linearly, so 4000 items, in our testing, used about 3.5GB of memory and was eventually killed by the system. You can change this default batch size by specifying --batch=<number> when running the command.