Greg Benz Photography

How to shrink your RAW files by 90% or more

Imagine if you could store 10 times as many RAW files on your laptop? Or if you could email a RAW file to someone else that was only 1-5 MB instead of 50+?

Lightroom Classic (and ACR) recently added support for a new DNG format which enables the ability to create RAW files which are 92% smaller with no visible loss of quality! This is made possible by using a new “lossy” image format based on the new JPEG XL (aka JXL) file format in DNG v1.7. Lossy means that the new DNG is not 100% identical to your original RAW file. However, in my testing, the results are extremely good and would be indistinguishable from the original in nearly any real scenario. The loss of quality is nearly undetectable for the vast majority of RAW files. Even the most discerning photographer would be hard pressed to see a difference in any realistic scenario (you’ll find a difference if you enlarge well beyond reasonable limits).

And if you really want to push things, you can also reduce resolution to create RAW files which are easily >98% smaller than the original. That will obviously cause a loss of quality compared to full resolution, but may be a great format for sharing a source image over email if the end use won’t be printed. You could even use it as a proxy and copy your RAW adjustments back to the original later.

CAUTION: By using the information on this page, you understand you are taking significant risks and agree that you are solely responsible for your own actions and  agree not to hold Greg Benz responsible for your use of the information. When you convert, you will no longer have mosaic data and may later be unable to use software such as Adobe AI Denoise on the lossy version of the image. When you export files, you risk overwrite existing ones if not careful. If you delete your original files, you risk loss of data if you did not convert properly. You could accidentally convert at reduced resolution. Furthermore, is possible that there are bugs in any software, that software behaviors may vary by platform or change in future releases, or that there errors / omissions / confusing information in this tutorial.  Please be sure to back up your images and catalog, and validate your results before you delete any original files. Bulk conversion elevate all of these risks substantially, and should be avoided unless you are an expert user with high confidence in your own ability to do that safely.

 

What impact does “lossy” compression have on the RAW?

If you zoom in extremely close, you may find some artifacts in the detail. You’ll have to zoom in to levels which are absurdly close for any realistic print to notice, but they are definitely in there. For example:

• I’ve seen some discoloration and haloing along the edges where building structures touch a clear sky. And I’ve seen some skin textures look a little strange.
• It’s more about how the noise relates to the content, such as smooth, fair skin shot at ISO 400 on a 50mm lens. In that case, using a higher or lower ISO, longer or shorter lens, or different model probably would have avoided it. So it’s a little like the risk of moire in that it’s very specific, and you’d need significant enlargement to see it even when it does occur.

In my experience, you’d typically need to enlarge the image >10x to see a difference at normal viewing distances (ie the size of a very large wall when shooting a modern 40+ megapixel camera). If you’re a serious pixel peeper and will view the image much closer than normal viewing distance, I’d say a 4x enlargement is probably your upper limit for more extreme cases (such as building details against a clean sky). I find the likelihood of seeing a quality difference is most likely with a low quality source (such as images from a smart phone or from an old low resolution camera). In general, I’d say you should hold off on images which you might print very large until you’ve done some testing to make sure you are comfortable with the limits.

I’ve also seen that the compression can slightly alter the way some intelligent sliders (such as shadows) work when pushed to the extreme. For example, I had to tweak a shadows slider from 95 to 100 to get a better match to the original. I haven’t seen it often, and it has been easily correctable when I have. But if it does occur, it can affect large areas of the image (not just fine detail).

The relatively new Adobe AI noise reduction shows some interesting interaction here. First, you can only apply it to the RAW before conversion to lossy. You can use both, but you must do the noise reduction before compressing the file. Secondly, while noise reduction makes the original file larger, the image with noise reduction will actually be smaller than the compressed version without noise reduction. Noise is very detailed, so cleaning up the image has the added benefit of shrinking your final file a further 15% or so.

 

When should you use lossy DNG?

As you’ll see below, there are a lot of caveats if you want to use this on a broad range of files. I don’t know that anyone should use this on all their files (I’m personally skipping my most important RAW files). But I do think think this could be valuable for those of you who would benefit from saving significant space. Here are a few scenarios where this may be worth the effort:

• To avoid having to buy more storage. This is my case, as compressing a large number of my images has helped me avoid the need to replace six hard drives on a nearly full RAID array.
• To archive large batches of files without having to cull them first. If you shoot lots of brackets, weddings, sports, etc – you may have a lot of images you’re keeping but unsure if you’ll ever need. This can give you a way to keep more or everything, just in case.
• To make the most of limited internal SSD storage on a laptop.

Perhaps in the future we’ll see an option in LR to import RAW images as lossy DNG (ideally with an option to apply noise reduction). Or perhaps someone will create a plugin for LR which helps facilitate this whole process. Either would be far simpler and more convenient for compressing images going forward than this manual process.

 

When should you NOT use lossy DNG?

There are also several scenarios where it is probably not worth it:

• If you don’t feel very comfortable with the steps below. Don’t risk your work if you are unsure how ensure the safety of your images.
• I’d probably skip your most important files. The risk isn’t worth the benefit on a small number of files.
• When you might want the originals in case of future improvements in RAW processing which require the original mosaic data. *
• You might skip existing images with edits (virtual copies, old process versions, or anything else which is more complicated or higher risk). You might simply choose to do this going forward on newly imported images before you make any edits to them.
• If you’re sending the image to someone on an old version of LR / ACR (raw v15.3 or later supports the new compression).

* Why can tools like AI Denoise work on the original

 

How to convert to lossy DNGs?

Before we discuss a specific workflow, here are a few general things to know / consider when using lossy compression:

• Lightroom does not offer a way to create lossy DNGs at the time of import. You’ll need to use the export feature to compress the images.
• If you want to use AI Denoise, do it before the lossy compression. You cannot use Adobe’s noise reduction after conversion to the lossy format.
• Use your export preset to compress any RAW images which are low risk (haven’t been adjusted heavily, won’t be printed extremely large, etc).
• After the export (and with only the same source files selected), add a unique keyword to the source images such as “duplicateRawToDelete” to make it easy to confirm later that the file is safe to delete because you’ve created a lossy DNG. You might also mark those images as rejected (“X”) to ensure they are queued for deletion, as you’ll only benefit once you’ve deleted the large originals.
• Review a few images after compression before you delete anything to make sure you are comfortable with the results.
  • Double-check that your lossy DNG file names say “Enhanced-NR” if your intention was to remove noise. You will not be able to use this type of noise reduction on the files after compressing to the lossy format.
  • You might consider marking the source as rejected rather deleting immediately if you want to be caution.
  • You might confirm everything is converted by checking that your total number of lossy images in a given folder is exactly half the total number of RAWs in that folder (obviously, this only works if you have appropriate filters and aren’t mixing other similar files in the same folder).

Select your RAW image(s) to process and then use the export dialog in LR. It can help significantly to filter just to original RAW images (as noted below).

Or if you are using ACR, you can open multiple RAW files directly into ACR, shift-click in the filmstrip to select all of them, and then click the save icon near the top-right (just left of the gear icon). Note that there will be no DNG option if you open an image which is not RAW.

Export settings:

• export location:
  • If your goal is to save space on the computer: you may export to “same folder as original folder” and check “add to this catalog“
  • If your goal is to share small RAW files with someone else: then export to a specific folder outside your catalog.
• in the file naming section:
  • use “document name” and then append something like “-lossyDNG” so that you can easily find and differentiate both the original and newly compressed output.
• in the file settings section:
  • select Digital negative (this requests DNG and allows you to see the rest of the options)
  • select compatibility = Camera Raw 15.3 and later for maximum savings (v6.6+ offers an older version lossy compression which is twice as big).
  • JPEG preview = medium size
  • enable “embed fast load data“
  • enable “use lossy compression” (and choose “limit size to 2560 pixels” or limit pixel count set for 8.0MP or more).
  • leave “embed original Raw file” unchecked, or you’ll be making a larger image.
• Image sizing:
  • turn off “resize to fit” if you want to preserve quality.
  • You can get significantly greater file size reduction by using resizing, but this will obviously reduce quality (as you will no longer have full resolution).

 

Existing images and bulk conversions

I recommend you limit your use of compressed DNG to new images you shoot, at least for a while. There is greater complexity with existing edits, virtual copies, and especially with bulk conversions across multiple folders.

I have developed my own workflow to help manage bulk conversions where use custom filters and keywords to help tag the image and move through each stage of conversion (AI denoise, compress DNG, delete the original). It’s complicated and I’m going to skip it here for now. But if there is significant interest in a follow-up article on bulk conversion, I may share it later (please comment below if you’d like to see such a tutorial).

There is a utility built into Lightroom meant to help do conversions. Go to Library / Convert Photo to DNG and use the following settings:

• check “only convert Raw files“
• you may wish to check “delete originals after successful conversion”
• compatibility = camera raw v15.3 or later (to ensure smallest files)
• check “use lossy compression“
• use medium JPG previews
• check “embed fast load data”
• do not use “embed original raw file”, as this will result in files which are larger, not smaller.

 

How filter for original vs compressed images in LR?

There are a few helpful search filters:

• Set metadata “file type” to both “raw” (originals which are not DNG) and “digital negative / lossless” (uncompressed DNG) for potential sources that have yet to be converted.
• Set metadata “file type” to “digital negative / lossy” to find converted images.
• Set metadata “edit” to “unedited” if you want to avoid any images you’ve processed. This isn’t always helpful as even very trivial changes are marked as edited (such as if your default import always enables remove chromatic aberration). So this is only useful in some cases.
• Use a keyword or file name search if you’ve tagged your converted source images (with something like “-lossyDNG”). You can search by “does not contain” to exclude these files.
• If you failed to update the name on your lossy conversions or to keyword the source, you can filter by date to help select select images that way (since the converted images will have a newer date).

While you cannot search on other aspects of the DNG type, you can change the metadata tab in the library to show “DNG” and will find the following information about the active image:

• DNG version will be at least 1.7 if the image was compressed.
• Lossy compression will say “yes” if the image was compressed.
• Mosaic data will say “no” if the image was compressed or converted with some other tool.
• Bits per sample will say 16 if the image was compressed (or in the unusual case that your camera captures 16-bit RAW).

 

Does this compress RAW Smart Objects?

Yes, but only modestly. If I open a compressed and original RAW as smart objects (in separate documents) and save as a compressed TIF, the result is that the image based on the compress smart object is about 43MB smaller.

So there is a real gain here too. However, the overhead in either TIF is enormous, so the savings in percentage terms are modest (about 8%). My 7MB lossy DNG becomes a 560MB TIF, while a 49MB original becomes a 603 MB TIF. So in both cases, there is an increase of about 550MB over the source file. And even though a TIF only has a single compatibility layer, the difference doesn’t substantially change when saving a file with numerous smart objects.

 

Caveats: edge cases to be aware of:

• There are some caveats when processing images which use RAW process version 1 or 2 (these would likely have been imported in 2012 or earlier).
  • I have found that conversion of process version 1 or 2 can create some unexpected results when converted to a new DNG (either for noise reduction or a lossy DNG). This resolved itself simply by turning B&W off and then back on. I haven’t found serious issues I couldn’t resolve by updating, but I’ve only tested a few files.
  • I found that updating the process version (found in the calibration tab) to v6 avoided the issues, but can’t guarantee that will always be the case – and this update will change the appearance as the sliders change (in many cases this may be an improvement for tonal adjustments and clarity, but it will definitely be different). My solution was simply to select all RAW/DNG files up through through 2012 which had no star rating and update to PV6 (I could change or revise the edit later if needed and I felt comfortable enough to make the change in bulk – but this definitely altered some old photos of mine).
  • If you want to search your catalog for particular process versions, you may use the Any Filter plugins (or Data Explorer or Search Replace Transfer). I have not personally used Any Filter, but have heard very good things about it. Lightroom’s only search tool (Library > Find Previous Process Photos) will match all prior versions, and it’s really just v1 or v2 which might cause a conflict.
• Be careful if your target images are virtual copies or are referenced by virtual copies.
  • Virtual copies are deleted when you delete the original they point to. If you convert the original file and not virtual copies, you could delete your virtual edits (the file would be safe, but you’d lose your editing work).
  • You could copy and paste the virtual copy settings, but that’s a tedious manual process and prone to error. The best approach is probably either to skip such files with virtual copies or just covert the virtual copies themselves (less efficient, but still saves space almost every time since they are so much smaller than the original).
  • You may also use the Any Filter plugin (linked above) to exclude any source images which are referenced by 1 or more virtual copies.
• There may be others, these are just the problems I know of…

Generative Remove in Lightroom / ACR

Adobe just released an awesome new AI-based feature for your RAW files: “generative remove“. This allows you to do much more advanced cleanup right with much less work. In the past, you could easily remove dust spots or do simple cloning – but even moderately more complicated jobs often required Photoshop. Not only does that add extra work and time, but it forces you to work on rasterized data. That meant you couldn’t update the RAW later without having to redo any cloning work.

You can now tackle much more complex jobs right in the RAW, like removing the branches as shown in this tutorial video.

Workflow for Generative Remove:

• Activate the remove tool (Q in LR, B in ACR)
• Set the mode to “remove” (pencil eraser icon)
• Check “generative AI“
• Brush over the target area to fix.
  • You can brush multiple times to refine results or add areas which aren’t connected.
  • Hold <alt/option> when brushing to remove red target areas (there’s a button, but the keyboard shortcut is much faster).
• Click “apply” when your red target area is ready.
• If you don’t like the initial results, click the “variations” arrow for other choices. If none of the three are ideal, click “refresh”. If you need to go back after refreshing, you can use history.

 

Tips for working with Generative Remove:

• You may optionally check “object aware” if you want it to help refine your target area. This concept is great, but I find the results are a bit mixed and tend to leave it off. I expect this will keep improving and would be very interested in using it after further enhancement.
• Set the tool overlay to “auto”
  • the UI for repaired areas will hide when you move the cursor out of the image
  • you’ll see an outline of the selected move pin, and can easily select others as needed for review or deletion (to undo).
• Click the eyeball icon at bottom left of the remove panel to see before / after.
• Be sure to combine with the healing brush (band aid icon), sometimes that tool works better for small areas and these tools work great together.
• Once you apply, there is no way to refine the targeted shape, but you can move it or change the variation. Just delete and redo if your target area isn’t working.
• Note that generative remove is like generative fill, but without a way for you to provide text input. You’re effectively deleting part of the image and asking the AI to fill it with something it would expect there. This has a couple implications:
  • Margins matter. For example: if you want to delete a license plate, paint it out to the edges. But if you want to simply remove the current license plate number, don’t go to the edges.
  • You can use this like generative fill. If you remove the face of an animal, the AI will likely see the body and generate a new face. I’ve found this works well in some cases, but not others (faces, hands, and feet for people tend to not work well with this initial release).
• Leave opacity at 100% (anything less will just cause ghosting or poor results)

 

What is the best “order of operations” for LR / ACR?

The order in which you make some changes matters. For example, the remove tool affects any AI masks you might create later (and existing ones are not updated automatically).

So I recommend you consider the following order for your RAW editing workflow to avoid unexpected results:

1. Denoise first:
   • This will affect both remove and any AI masks.
   • The impact is minor, so I wouldn’t sweat if you need to remove noise later – but you might want to check and see if you need to update anything else.
2. Generative Remove:
   1. (at least be aware that it will use data outside your visible crop, and that it can affect lens blur)
3. You can generally safely proceed with other edits (but remember that the active state of the image can affect some tools like masks)
4. Use last: point color (it’s very dependent on other edits)

Generative Remove is supported in Lightroom Classic v13.3, LR Desktop v7.3, and Adobe Camera RAW16.3