Skip to content

index.md

Latest commit

 

History

History
104 lines (75 loc) · 2.75 KB

index.md

File metadata and controls

104 lines (75 loc) · 2.75 KB
Raw

Serverless Data Processing Pipeline

Overview

This project implements a serverless data processing pipeline that handles daily CSV file uploads and transforms them into structured JSON messages for downstream consumption.

Problem Statement

An external client uploads four CSV files to an S3 bucket daily. The files follow a naming convention of {TYPE}_{DATE}.csv (e.g., clients_20200130.csv).

Input Files

The system processes four types of CSV files:

  1. clients: List of all clients. A client can have multiple portfolios.
  2. portfolios: List of all portfolios managed for clients.
  3. accounts: List of all bank accounts. Each portfolio has exactly one bank account.
  4. transactions: List of all transactions on managed portfolios.

Processing Requirements

Once all four files are uploaded for a given day, the system automatically processes them and generates JSON messages that are sent to a message queue.

Expected Output

The system generates the following message types:

  1. Client messages: Exactly one message per client
    • Aggregates taxes_paid across all accounts held by the client
  2. Portfolio messages: Exactly one message per portfolio
    • Contains portfolio-level financial information
  3. Error messages: Generated for any unexpected inputs or validation failures

Data Examples

Input CSV Files

1. Clients CSV

record_id,first_name,last_name,client_reference,tax_free_allowance
1,Frida,Muller,9e40659b-8b9f-4fc4-814b-5a7b5a23b64,801
2,Fritz,Maier,f4a0cc2c-d0b4-4f14-b202-c8a5e45e90e7,0

2. Portfolios CSV

record_id,account_number,portfolio_reference,client_reference,agent_code
1,12345678,90755e32-7438-4354-ad37-ad900e297844,9e40659b-8b9f-4fc4-814b-5a7b5a23b64,EREZBT
2,123456789,439695b4-508d-4562-8576-670e70024627,f4a0cc2c-d0b4-4f14-b202-c8a5e45e90e7,SFOJFK

3. Accounts CSV

record_id,account_number,cash_balance,currency,taxes_paid
1,12345678,15000.00,EUR,0.00
2,123456789,-56.00,EUR,789.56

4. Transactions CSV

record_id,account_number,transaction_reference,amount,keyword
1,12345678,14e56786,5000,DEPOSIT
2,123456789,dfeb5fe13cd4,-786.56,TAX

Output Message Formats

Client Message

{
  "type": "client_message",
  "client_reference": "9e40659b-8b9f-4fc4-814b-5a7b5a23b64",
  "tax_free_allowance": 801,
  "taxes_paid": 0
}

Portfolio Message

{
  "type": "portfolio_message",
  "portfolio_reference": "90755e32-7438-4354-ad37-ad900e297844",
  "cash_balance": 15000,
  "number_of_transactions": 15,
  "sum_of_deposits": 5000
}

Error Message

{
  "type": "error_message",
  "client_reference": null,
  "portfolio_reference": "90755e32-7438-4354-ad37-ad900e297844",
  "message": "Something went wrong!"
}