Accessing PMC Article Datasets Using Amazon Web Services

As part of our Cloud Service, PMC makes the datasets described below freely accessible on Amazon Web Services (AWS), without charge, through either an HTTPS or S3 URL, and without any log-in requirement for retrieval. The National Library of Medicine works with the AWS Open Data Sponsorship Program to provide this access. Read on to learn why and how you may access these datasets from our AWS Cloud Service.

Description and Location of PMC Article Datasets on AWS

Add --no-sign-request to any AWS CLI command to access the bucket anonymously.

The README.txt file in the S3 bucket is kept up to date with technical information such as:

• Structure of the Datasets
• Inventory
• Versioning and Update Frequency
• S3 Bucket Schematic Overview
• AWS CLI Data Access
• Using an eSearch-S3 Pipeline

FAQs

Q: How does versioning work and which version should I use?

A: Most articles in the PMC Article Datasets have only one version, version 1. When multiple versions do exist, it is for one of two reasons:

1. The journal or preprint server uses versions. If the journal or preprint server versions their articles, PMC maintains those versions.
2. PMC holds both an author manuscript and a final published version. You can identify author manuscripts using the is_manuscript property in the article's JSON metadata ojbect (set to "yes"). See Author Manuscripts in PMC for more information.

The version number reflects the sequence in which PMC processed the deposited versions.

PMC does not create a new version when changes are made to an article's content or metadata. When an article is updated, the LastModifiedDate in the inventory and the MD5 values in the JSON metadata object will reflect the update, but no new version prefix is created. See About the Content in PMC for an overview of the types of content in PMC.

A higher version number does not mean the version is more recent or should be preferred over a lower-numbered version, except for case 1 above. Version numbers are assigned sequentially as PMC processes file sets, but the version number alone does not indicate whether a version is the author manuscript or is the final published version. It does not indicate which version is preferred for a given use case. To determine what a given version is, inspect its JSON metadata object, which indicates whether it is an author manuscript (is_manuscript). The inventory documentation in the README.txt file describes the fields available in the JSON metadata objects.

If a version is removed from the S3 bucket, its prefix and all associated files are deleted from the bucket with no explicit notification. Inventory reconciliation is the only supported way to detect removals. See Accessing the Inventory in the README.txt file for details on working with inventory snapshots.

Q: Why is there a version 2 but no version 1 in the dataset?

A: In some cases you may find that a prefix such as PMC12805672.2 exists in the bucket while PMC12805672.1 does not. This commonly occurs when version 1 does not have an appropriate license and therefore is not part of the PMC Article Datasets. Do not assume that a version 1 always exists for every PMCID in the S3 bucket.

Q: How current is the inventory? Will articles added or updated today appear in it?

A: The S3 Inventory is generated once per day. The inventory always lags the current state of the bucket — articles added, updated, or removed since the most recent snapshot was generated will not be reflected in it. See Accessing the Inventory in the README.txt file for more information on the inventory structure and how to access the latest snapshot.

Q: What does LastModifiedDate in the inventory CSV represent?

A: LastModifiedDate is the object creation date or the last modified date of the JSON metadata object (documented in the README.txt file) for that article version, whichever is the latest. It reflects the last time the JSON metadata object for that version was written to the S3 bucket, which can happen for reasons unrelated to content changes (e.g., metadata enrichment, reprocessing runs, or schema updates).

Q: Can the same PMCID appear in both the "open access"[filter] and "author manuscript"[filter] search filters?

A: Yes. The most common reason is that an author manuscript has a Creative Commons license — in that case, it qualifies as open access and will be returned by both search filters even if there is only one version of the article in the dataset. A PMCID may also appear in both search filters if it has multiple versions, one of which is an open access published article and another of which is an author manuscript. When building workflows that depend on specific version types, use the JSON metadata for each version to determine its type and license rather than relying solely on search filter results. See the PMC User Guide for documentation on available search filters.

Q: The legacy Cloud Service AND FTP Service separated articles into oa_comm and oa_noncomm directories. How do I retrieve only articles that allow commercial reuse?

A: The updated PMC Cloud Service does not separate articles by license category at the directory level, however, there are several different approaches you can use to identify articles by license category. Here are a few suggestions:

First method: License information is available per article version in the JSON metadata object. You can read each article's JSON metadata to check its license and retrieve only the versions whose licenses meet your requirements. JSON metadata fields are described in the inventory documentation in the README.txt file. This method might benefit from using some of the advanced tools available for working with AWS inventory and bucket files.

Second method: PMC search filters for license types allow you to find articles by a specific Creative Commons license type or set of types. These searches are available in the NCBI eUtilities that can be used in automated scripting.

To find articles available for commercial reuse that have files available for retrieval from our Cloud Service, use a variation of the following query:

((cc0 license[filter] OR cc by license[filter] OR cc by-sa license[filter] OR cc by-nd license[filter]) OR author manuscript[sb]) NOT pmc embargo[filter]

This returns articles with a commercial Creative Commons license, plus all distributed author manuscripts (which are available for commercial reuse regardless of license) and excludes currently embargoed articles. These results can then be combined into an eSearch-S3 pipeline (documented in the README.txt file) to retrieve the corresponding files. See the PMC User Guide section on searching by license for a full list of available license filters.

Third method: The PMC OAI-PMH API output includes a machine-readable license in the <dc:rights> element in the compact metadata record when you use the metadataPrefix=oai_dc format. You could retrieve all records in the set=pmc-open and then parse the records to find the PMCIDs with the license values you want.

Q: The legacy Cloud Service had separate locations for author manuscripts and the Historical OCR Dataset. How do I identify articles of these types in the updated structure?

A: In the updated PMC Cloud Service, all article versions are stored together in a single bucket regardless of type. The JSON metadata object for each article version contains the information needed to identify its type, including whether it is an author manuscript or part of the Historical OCR Dataset. A full description of JSON metadata fields is available in the inventory documentation in the README.txt file.

Q: The ESearch API only returns the first 10,000 results. How do I retrieve more than 10,000 records?

A: ESearch is capped at 10,000 returned records per query. To work around this:

• Use EDirect (esearch + efetch in the NCBI EDirect toolkit), which handles pagination automatically and can retrieve more than 10,000 records.
• Subdivide your query using date filters (e.g., pmcrdat), license filters ("cc by license"[filter], etc.), or other fields documented in the PMC User Guide to keep each sub-query under 10,000 results and then combine the results.
• Use the daily inventory CSV (documented in the README.txt file as your source of PMCIDs rather than ESearch — the inventory is a complete snapshot and has no record limit.

Q: Is there a way to get all the article PDFs easily?

A: You can search for all articles with PDFs that are part of the PMC Open Access Subset, and build that search into a pipeline (such as the eUtilities eSearch-S3 pipeline (documented in the README.txt file) limited to PDF file retrieval) to retrieve all the PDFs.

Find all articles added August 17, 2026 with PDFs that are available as part of the PMC Article Datasets

open_access[filter] AND has_pdf[filter] NOT pmc embargo[filter] AND 2026/08/17[pmcrdat]

Q. Where do I find the baseline and incremental files?

A. There are no longer baseline bulk file packages that contain the XML or TXT for millions of articles.

As there are no baseline files, there are also no incremental files. Instead, each individual file will have a timestamp and ETag that will be updated if the file changes. The bucket inventory (see documentation in the README.txt file) includes S3 modification dates as well as ETag data.

If you want to retrieve all the articles that have been added on a specific date or in a specific date range, you can use these queries with eSearch combined into an eSearch-S3 pipeline (documented in the README.txt file).

Find articles added on a specific day

2026/08/19[pmcrdat] AND (open_access[filter] OR author_manuscript[filter])

This search query will find all articles added on 2026/08/19 that are available for automated retrieval.

Find articles added during a specific date range

2026/08/13:2026/08/15[pmcrdat] AND (open_access[filter] OR author_manuscript[filter])

This search query will find all articles added from 2026/08/13 through 2026/08/15 that are available for automated retrieval.

Q. Do I need an account to retrieve the files from the AWS Cloud?

A. No account is necessary to retrieve files from the AWS Cloud to your local storage. You will need to add –no-sign-request to all of your command line statements while using the AWS CLI in order to do this. Access is also available without login using a URL in a browser or in a Curl command. The URLs all start with https://pmc-oa-opendata.s3.amazonaws.com/.

Engage

NCBI wants your feedback on accessing PMC Article Datasets using AWS. Write to the PMC Help Desk.