Metrics Onboarding

Prime's Benefit Usage Tracking allows 3P Partners to share benefit usage metrics with Prime by uploading files to a Prime-owned Amazon S3 bucket. Prime's data processing then ingests these files on the agreed cadence.

This page provides the generic Standard Operating Procedure (SOP) for sharing metrics. The exact set of metrics, cadences, schemas, file formats, and S3 prefixes are specific to your partnership and will be provided by your Prime point of contact (POC).

Concepts

• Metric batch — a single upload for one metric and one dataset date. A batch consists of one or more chunk files followed by a single completion marker file (_SUCCESS).
• Chunk file — a data file containing a portion of the batch. A batch may be split across multiple chunks.
• Completion marker (_SUCCESS) — an empty file that signals to Prime that all chunk files for the batch have finished uploading and the batch is ready to be ingested.
• Dataset date — the date that identifies the batch of data, formatted as YYYYMMDD. The dataset date is the latest date for which you pulled data in that batch.

info

S3 buckets for metrics ingestion are only in us-east-1. Upload all files to us-east-1.

One-Time Setup

Because you upload files to S3, you first need to create an IAM user or role that can write to the Prime-owned S3 bucket, and have Prime allow-list it. You can create as many users/roles as you need.

Prerequisites

• An AWS account. (See create an AWS account if you don't have one.)
• The S3 bucket name from Prime. Reach out to your POC if you need the bucket name.

Steps

1. Create an IAM Role/User in your AWS account. (See AWS documentation to create an IAM Role or User.)

2. Add the following permissions to the Role/User (replace <s3BucketName> with the bucket name provided by Prime):

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "s3:ListBucket",
         "Resource": "arn:aws:s3:::<s3BucketName>"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:Abort*",
           "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::<s3BucketName>/*"
       }
     ]
   }

3. Provide the IAM Role/User ARN to the Prime team so they can allow-list it on the S3 bucket.

S3 File Structure

Prime will provide the S3 bucket name and, for each metric, a prefix and cadence. Each batch is uploaded to a location that combines the metric prefix with the dataset date, for example:

s3://<s3BucketName>/<metricPrefix>/datasetDate=<YYYYMMDD>/v1/

For example, given a metric with prefix prod/US/Weekly-Aggregated-Feed/v1 and dataset date 20250107, the batch location is:

s3://<s3BucketName>/prod/US/Weekly-Aggregated-Feed/v1/datasetDate=20250107/v1/

All chunk files and the _SUCCESS completion marker for a batch must be uploaded to this exact same location.

Upload a Batch of Metrics

Uploading a batch of metrics requires uploading one or more chunk files, and then uploading the completion marker file. Make sure you are using one of the IAM roles/users you created in One-Time Setup.

danger

Only upload the _SUCCESS completion marker file after all chunk files have successfully uploaded. Prime begins ingesting the batch as soon as the marker appears.

1. Generate one or more batch chunk files.
2. Determine the dataset date (YYYYMMDD) — the latest date for which you pulled data in this batch.
3. Determine the S3 location to upload the files (see S3 File Structure).
4. Upload all chunk files to the exact same S3 location.
   • Chunk files can be named anything, as long as the file extension matches the agreed file format.
5. Once all chunk files are uploaded, upload the completion marker file to the same location. This file:
   • must be empty;
   • must be named exactly _SUCCESS;
   • must be uploaded to the exact same location as the chunk files.

Chunk sizing

A good rule of thumb is to keep chunk files between 100–200 MB.

Examples

Assume Prime has provided the S3 bucket test-bucket and requested the following batches:

Metric Name	Cadence	Prefix
Aggregated Metrics	Weekly	prod/US/Weekly-Aggregated-Feed/v1
Daily Customer Metrics	Daily	prod/US/Daily-Customer-Feed/v1

Scenario 1 — Weekly Aggregated Metrics. You run the job on Jan 08, 2025, pulling the previous week's data from Jan 01, 2025 to Jan 07, 2025.

1. Determine the dataset date: since the latest date you pulled data for is Jan 07, 2025, the dataset date is 20250107.
2. Your job created one chunk named chunk1.csv. Upload it to: s3://test-bucket/prod/US/Weekly-Aggregated-Feed/v1/datasetDate=20250107/v1/chunk1.csv
3. Once the chunk is uploaded, upload the completion marker at: s3://test-bucket/prod/US/Weekly-Aggregated-Feed/v1/datasetDate=20250107/v1/_SUCCESS

Scenario 2 — Daily Customer Metrics. You run the job on Jan 08, 2025, pulling the previous day's data from Jan 07, 2025.

1. Determine the dataset date: since the latest date you pulled data for is Jan 07, 2025, the dataset date is 20250107.
2. Your job created two chunks named chunk1.csv and chunk2.csv. Upload them to:
   • s3://test-bucket/prod/US/Daily-Customer-Feed/v1/datasetDate=20250107/v1/chunk1.csv
   • s3://test-bucket/prod/US/Daily-Customer-Feed/v1/datasetDate=20250107/v1/chunk2.csv
3. Once the chunks are uploaded, upload the completion marker at: s3://test-bucket/prod/US/Daily-Customer-Feed/v1/datasetDate=20250107/v1/_SUCCESS

Troubleshooting

If you run into an issue while onboarding:

• Check that you are using an IAM role/user you have provided to the Prime team.
• Check that the IAM credentials have not expired.
• Check that the IAM role/user has the required IAM policies/permissions.

If you are still unable to resolve your problem, reach out to your Prime team POC for help.

FAQs

How can I amend a previous batch of data? Reach out to the Prime team to let them know you need to update previously uploaded data. They will work with you to determine how to upload the corrected data, typically by giving you a new location in the same S3 bucket where you will upload all of the data.

What happens if I use the wrong schema or file format? Prime's data processing has likely already identified the issue, but it is still best to reach out to your POC. Prime will request that you re-upload the data using the correct schema and format.

How can I make updates to the file schema? If you need to change the schema (adding a column, changing a type, etc.), reach out to your POC. They will work with you to define the updated schema. They may give you a new object prefix for the new schema — make sure to use that new prefix when you start uploading with the updated schema.

How many chunks should I use? A good rule of thumb is to stick to 100–200 MB chunks.

How can I add a totally new set of metrics? Work with your POC to define the new schema (including data types and file formats) and the upload cadence. Prime will give you a new object prefix for this new set of metrics.