feat(portfolio): Read data and calculate mean-variance for returns

[SaurabhJamadagni]

The PR is in reference to issue:

• Implement Mean-Variance Optimization #8

PR makes the following changes:

• Creates a Portfolio struct which holds:
  • tickers
  • mean returns vector for assets
  • covariance_matrix
  • risk-free rate for the market to use
  • expected return for the portfolio
  • method of calculating returns (log or simple)
  • weights for the assets
  • returns calculated (internal)
• In it's current state, the struct takes a path to a .csv file which contains price data.
• fn new() will read from the data file and perform return calculations and produce a covariance matrix.
• ndarray and 'ndarray_stats` are used to store records from the csv and perform operations.

Example output:

Next steps:

• Perform portfolio optimization using legrange multipliers and optimization crates

@carlobortolan please let me know your feedback on this. I haven't written tests for this yet. Was planning to include them when I add the optimization part. Can add something before as well if you would like that before merging. Let me know :)

[codecov]

Codecov Report

❌ Patch coverage is 0% with 108 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.76%. Comparing base (5407b79) to head (6f1fcb3).

Files with missing lines	Patch %	Lines
src/portfolio/mean_variance.rs	0.00%	108 Missing ⚠️

❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #81      +/-   ##
==========================================
- Coverage   93.28%   89.76%   -3.53%     
==========================================
  Files          29       30       +1     
  Lines        2754     2862     +108     
==========================================
  Hits         2569     2569              
- Misses        185      293     +108     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[SaurabhJamadagni]

In the changes following this, I would also like to separate out the functions that calculate returns as they aren't really struct functions. Could move them to a more general utilities module or something that the whole package can make use of.

[carlobortolan]

Thanks for putting this together @SaurabhJamadagni; already looks really solid!
Just FYI: I'll be a bit busy with travelling & moving for the next 2.5 weeks (after that I'll also finally have time to regularly contribute to quantrs again and review/test new PRs thoroughly)

Two quick notes:

• expect is used for csv reading/parsing; returning a Result might handle errors more gracefully

• As you suggested, the return calculation functions could be moved to a utils module since they aren't strictly tied to the struct

Appreciate all the work so far!

P.S. Just merged #65 🚀

[SaurabhJamadagni]

No worries on the delay @carlobortolan. Hope you have a stress free move!

expect is used for csv reading/parsing; returning a Result might handle errors more gracefully

Noted. I'll give it a look.

P.S. Just merged #65 🚀

Thanks on this one! Appreciate you pushing the final few commits to get it merged. Do we wanna keep this PR open till the whole optimization is implemented or are you looking to merge it after some of the above changes? No rush of course, I just wanted to clarify where you stand on this.

[carlobortolan]

Hey @SaurabhJamadagni

Sorry for the long delay - finally finished moving (and all the bureaucracy that comes with it 🙄), so I'll have some time to review it over the weekend.

Do we wanna keep this PR open till the whole optimization is implemented or are you looking to merge it after some of the above changes?

This PR is already quite well-sized with ~300 added LOC, so it's fine to leave it as is. (However, I'd say that it would be better to have the whole optimization merged as one PR, so that master doesn't contain too many placeholders / TODOs, but I'll let you know once I've tested this PR, as this might depend on how it's implemented.)

[SaurabhJamadagni]

Sorry for the long delay

No worries @carlobortolan! I know the headache that comes with moving. I hope everything went well.

However, I'd say that it would be better to have the whole optimization merged as one PR, so that master doesn't contain too many placeholders / TODOs

I agree with this and hence was curious. I also wanted to check if you would consider having dev and release branch instead of merging with main. Incomplete features could stay on dev and after a certain amount of features are added or after a certain period of time you could merge dev into main as a release which could be documented through the current CHANGELOG or something. Do you think there's a benefit to such a separation?

[carlobortolan]

I also wanted to check if you would consider having dev and release branch instead of merging with main. Incomplete features could stay on dev and after a certain amount of features are added or after a certain period of time you could merge dev into main as a release which could be documented through the current CHANGELOG or something. Do you think there's a benefit to such a separation?

@SaurabhJamadagni: Yes, having a dev branch could make things more organized, especially for tracking unreleased features. That said, I think that setup usually benefits larger projects more. If we had master, dev and feature branches, it might introduce redundancy since I’d need to review code before merging into dev, and then again when merging dev into master 😅

However, what we could do is keep master as the release branch and handle ongoing work through feature branches. For example, in this case we could have portfolio_optimization_structs and portfolio_optimization_mean_variance as sub-branches merged into the main feature branch portfolio_optimization.

This way, features that are less modular / need multiple PRs can be split into smaller branches& PRs, while others can still be merged in a single PR. Does this approach sound good?

(P.S.: I took the liberty of fixing the merge conflicts here, since I just updated some dependencies in another PR which might cause some issues with our Cargo.lock.MSRV otherwise)

[carlobortolan]

Thanks for the PR and again sorry for the long review-time: The PR looks mostly good to me; just added a few very small comments 👍

edit: another thing I noticed: The weights field is Option<Vec<f64>> but currently never populated. Is this intended? It might be a good idea to add a method stub/placeholder for computing optimal weights, so we don't forget to implement this later.

[carlobortolan]

Use std::env::args() or a relative path to an example file (e.g., examples/data/ETFprices.csv) to avoid hard coded paths.

[carlobortolan]

calculate_simple_returns and calculate_log_returns currently create new arrays using .to_owned(). Consider in-place operations or preallocating arrays for large datasets.

[carlobortolan]

I think the latest version is 0.17.0 - is there any reason for using 0.16.x? If not, we should probably upgrade to the newer version

[SaurabhJamadagni]

Not really. I think I just copied what was on crates.io but I might be remembering it incorrectly. I'll update it.

[carlobortolan]

see comment above

[SaurabhJamadagni]

Hey @carlobortolan! Apologies for the giant break in between! I was wrapping up my final semester and graduated couple of weeks back. Had to go through the thread again a bit cause I lost touch with previous commit.

It might be a good idea to add a method stub/placeholder for computing optimal weights

Agree with this, what I was thinking before is that I would immediately implement the function. I did not anticipate such a large break in between.

I'll go ahead and complete the implementation for the optimization that's left for this feature.

[carlobortolan]

Hey there @SaurabhJamadagni , no worries at all and congratulations on graduating!

I'll go ahead and complete the implementation for the optimization that's left for this feature.

Sounds good 👍

[SaurabhJamadagni]

Hey @carlobortolan! Just pushed some code to resolve :

calculate_simple_returns and calculate_log_returns currently create new arrays using .to_owned(). Consider in-place operations or preallocating arrays for large datasets.

Wanted to check if this is what you meant by in-place operations. I was thinking I would move the functions to utils once I am done with the optimization code first. After coming back to the code after some time, I was trying to remember some of the rust syntax as well as work with merge conflicts and that really took some time. I think I want to focus on getting the implementation done before I refactor code if that's okay.

I wanted to ask, how do you deal with merge conflicts on the Cargo.lock or Cargo.lock.MSRV files? Do you manually pick between incoming and current changes or is there a more efficient way to do it as the Cargo.lock file is auto-generated?

Thank you for your patience on this one. I really appreciate it!

[carlobortolan]

Hey @SaurabhJamadagni thanks for your update and sorry for the late reply! (I somehow missed the notification for your message)

I'm currently on vacation, but I'll review the code once I've access to a laptop/PC again later this week.

I think I want to focus on getting the implementation done before I refactor code if that's okay.

👍

I wanted to ask, how do you deal with merge conflicts on the Cargo.lock or Cargo.lock.MSRV files?

What I usually do is first delete the current lock files and then:

# remove conflicted file
rm Cargo.lock

# generate MSRV lockfile
rustup install 1.83.0
rustup override set 1.83.0
cargo generate-lockfile
mv Cargo.lock Cargo.lock.MSRV

# generate stable lockfile
rustup override unset
cargo generate-lockfile

Edit: Also took the liberty of cleaning up the merge conflicts on your branch, since I just released a new version -hope that's alright.

[carlobortolan]

Wanted to check if this is what you meant by in-place operations

Yes exactly! This looks perfect 👍

[carlobortolan]

Just noticed an edge case: If there is a missing or malformed value in the CSV, it currently defaults to 0.0, but this would cause issues afterwards:

• In calculate_simple_returns, if prv is 0.0, the calculation (nxt - prv) / prv will result in inf or NaN (division by zero)
• In calculate_log_returns, 0.0.ln() will result in -inf

If these inf or NaN values feed into the cov matrix, they will most likely ruin the portfolio optimization results. I think we should handle bad data differently, e.g., by filtering out bad rows entirely or carrying forward the previous day's price (or maybe you also have any other ideas?).