Spike: Fold rex-mime into rapid7/metasploit-framework

[smcintyre-r7]

A long long time ago in an internet far far away, Metasploit was split into many gems. The intention at the time was to facilitate a Ruby based community of projects. Over the years, this never came to pass and as it stands now, is an extra layer of complexity and effort that we the core maintainers and open source contributors have to subject ourselves to while making trivial changes.

Currently if 1 gem needs to be updated:

1. The developer clones rapid7/metasploit-framework
2. The developer clones rapid7/rex-whatever
3. The developer makes necessary code changes
4. The developer makes a PR to rapid7/rex-whatever
5. The developer updates their Gemfile with the incantation to load their PR'ed changes
6. The developer makes a PR to rapid7/metasploit-framework
7. Changes are reviewed and iterated on until ready to be merged
8. A maintainer merges the changes to rapid7/rex-whatever, cutting a release
9. The developer reverts the changes from step 4 and adds a commit to bump the gem
10. A maintainer merges the changes to rapid7/metasploit-framework

The intention of this ticket is to spike our ability to bring some of these gems into the framework tree to allow changes to be made to their source code directly, in parallel / the same repo. That would ideally eliminate steps 2, 4, 5, 8, and 9. Of course not all of these steps are the same level of effort, but that is half the number of steps. Ideally, Metasploit would load the local gem changes, and any bumps would be handled by Jenkins automatically on merge. If possible, we should continue to publish the gems as we go to make testing and distribution easier, as well as maintain the git history.

Based on the dependency graph today, rex-mime seems like a reasonable candidate to use as a test subject.

graph BT
  classDef terminal fill:#fff8c5,stroke:#333,stroke-width:2px
  classDef msfOnly fill:#b6f3c2,stroke:#333
  classDef disqualified fill:#ffffff,stroke:#888

  rex_arch["rex-arch"]:::disqualified
  rex_bin_tools["rex-bin_tools"]:::msfOnly
  rex_core["rex-core"]:::disqualified
  rex_encoder["rex-encoder"]:::msfOnly
  rex_exploitation["rex-exploitation"]:::msfOnly
  rex_java["rex-java"]:::msfOnly
  rex_mime["rex-mime"]:::msfOnly
  rex_nop["rex-nop"]:::msfOnly
  rex_ole["rex-ole"]:::msfOnly
  rex_powershell["rex-powershell"]:::msfOnly
  rex_random_identifier["rex-random_identifier"]:::msfOnly
  rex_registry["rex-registry"]:::msfOnly
  rex_rop_builder["rex-rop_builder"]:::msfOnly
  rex_socket["rex-socket<br/>ext: lacewing"]:::disqualified
  rex_sslscan["rex-sslscan"]:::msfOnly
  rex_struct2["rex-struct2"]:::msfOnly
  rex_text["rex-text<br/>ext: lacewing, rise-cli"]:::disqualified
  rex_zip["rex-zip"]:::msfOnly
  metasploit_framework["metasploit-framework"]:::terminal
  metasploit_credential["metasploit-credential"]:::msfOnly
  metasploit_data_models["metasploit_data_models"]:::msfOnly
  metasploit_concern["metasploit-concern<br/>ext: wg-metasploit_data_models"]:::disqualified
  metasploit_model["metasploit-model<br/>ext: wg-metasploit_data_models"]:::disqualified
  metasploit_payloads["metasploit-payloads"]:::msfOnly
  metasploit_payloads_mettle["metasploit_payloads-mettle"]:::msfOnly
  metasploit_erd["metasploit-erd"]:::disqualified
  metasploit_yard["metasploit-yard<br/>ext: wg-metasploit_data_models"]:::disqualified
  metasploit_aggregator["metasploit-aggregator"]:::disqualified
  metasploit_version["metasploit-version"]:::disqualified
  yard_metasploit_erd["yard-metasploit-erd<br/>ext: wg-metasploit_data_models"]:::disqualified

  rex_arch --> rex_text
  rex_bin_tools --> rex_arch
  rex_bin_tools --> rex_core
  rex_bin_tools --> rex_struct2
  rex_bin_tools --> rex_text
  rex_encoder --> rex_arch
  rex_encoder --> rex_text
  rex_exploitation --> rex_arch
  rex_exploitation --> rex_encoder
  rex_exploitation --> rex_text
  rex_mime --> rex_text
  rex_nop --> rex_arch
  rex_ole --> rex_text
  rex_powershell --> rex_random_identifier
  rex_powershell --> rex_text
  rex_random_identifier --> rex_text
  rex_rop_builder --> rex_core
  rex_rop_builder --> rex_text
  rex_socket --> rex_core
  rex_sslscan --> rex_core
  rex_sslscan --> rex_socket
  rex_sslscan --> rex_text
  rex_zip --> rex_text
  metasploit_framework --> metasploit_concern
  metasploit_framework --> metasploit_credential
  metasploit_framework --> metasploit_data_models
  metasploit_framework --> metasploit_model
  metasploit_framework -->|"= 2.0.34"| metasploit_payloads
  metasploit_framework -->|"= 1.0.6"| metasploit_payloads_mettle
  metasploit_framework --> rex_arch
  metasploit_framework --> rex_bin_tools
  metasploit_framework --> rex_core
  metasploit_framework --> rex_encoder
  metasploit_framework --> rex_exploitation
  metasploit_framework --> rex_java
  metasploit_framework --> rex_mime
  metasploit_framework --> rex_nop
  metasploit_framework --> rex_ole
  metasploit_framework --> rex_powershell
  metasploit_framework --> rex_random_identifier
  metasploit_framework --> rex_registry
  metasploit_framework --> rex_rop_builder
  metasploit_framework --> rex_socket
  metasploit_framework --> rex_sslscan
  metasploit_framework --> rex_struct2
  metasploit_framework --> rex_text
  metasploit_framework --> rex_zip
  metasploit_credential --> metasploit_concern
  metasploit_credential -->|">= 5.0.0"| metasploit_data_models
  metasploit_credential --> metasploit_model
  metasploit_credential --> rex_socket
  metasploit_data_models --> metasploit_concern
  metasploit_data_models -->|">= 5.0.4"| metasploit_model
  metasploit_aggregator --> rex_arch
  yard_metasploit_erd --> metasploit_erd

Loading

[smcintyre-r7]

On second thought, instead of targeting rex-mime just to have something to target, rex-powershell might be a better starting point because it'd unblock rex-random_identifier. If we could fold rex-random_identifer into the framework, we could be more aggressive in our recommendations to contributors that should add new languages to obfuscate code. rex-socket and metasploit-credential are also likely to see additional activity in the relatively near future.

[github-actions]

Hi!

This issue has been left open with no activity for a while now.

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 30 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request.

[karandesai2005]

Hey @smcintyre-r7 saw your note about rex-socket potentially seeing more activity soon. I had PR #80 merged there a while back, so I'm already familiar with its structure, dependents, and the gem release flow.
I'd like to take a shot at spiking the fold-in using rex-socket as the test subject, covering:

Moving the gem source into the framework tree (preserving git history via git subtree/filtered merge)
Gemfile/gemspec changes so the framework loads it locally instead of from a published gem
A plan for Jenkins to keep auto-bumping/publishing the gem version on merge, per your point about not breaking distribution/testing

Will report back with findings (and tradeoffs vs. rex-powershell/rex-random_identifier if relevant) before opening any PR. Let me know if rex-socket still makes sense as the target or if you'd rather I look at one of the others.

[smcintyre-r7]

I appreciate the offer but I don't think we can start with rex-socket. That one has downstream dependencies. I think starting with rex-powershell or rex-mime would be better because it wouldn't affect anything else. Also FWIW the architecture of the gem shouldn't be a large factor.

[karandesai2005]

Got it, makes sense re: rex-socket's downstream deps — appreciate the steer.

I'll go with rex-mime then, sticking close to your original plan:

• Moving the gem source into the framework tree (preserving git history via git subtree / filtered merge)
• Gemfile/gemspec changes so the framework loads it locally instead of from the published gem
• A plan for Jenkins to auto-bump/publish on merge so we don't break distribution/testing

Will report back with findings before opening a PR.