Server implementation

[silver-volt4]

Resolves this forum thread

Changes:

Adds an implementation for Swapdoodle. This uses Jon's unfinished DataStore rework (PretendoNetwork/nex-protocols-common-go#53).

The server currently doesn't work out of the box due to the incomplete state of the DataStore rework. There are some quirks I discovered that I wanted to discuss here, instead of polluting the DataStore PR with possibly Swapdoodle-specific info. This would be:

• object_enabled.go:27: Swapdoodle seems to be uploading doodles with the needUploadCompletion flag set to false, which in turn causes uploadCompleted to be set to true, and so CompletePostObjectV1 fails. Additionally, since ObjectEnabled also checks for state == 0, the file is actually never accepted because it's always initialized to 0 but that's likely to be fixed anyway
• This callback is actually never run, because CompletePostObjectV1 does not call it. This likely won't be needed in the future when notification support is added to the package, I'm currently creating notifications manually for the sake of maintaining functionality Notifications are now supported, so this is no longer relevant

With the two quirks patched out locally via a Go workspace, I have confirmed the server to correctly relay notes.

---

• I have read and agreed to the Code of Conduct.
• I have read and complied with the contributing guidelines.
• What I'm implementing was an approved issue. (it theoretically wasn't?)
• I have tested all of my changes.

[silver-volt4]

how i feel about this pr (lol):

[jonbarrow]

object_enabled.go:27: Swapdoodle seems to be uploading doodles with the needUploadCompletion flag set to false, which in turn causes uploadCompleted to be set to true, and so CompletePostObjectV1 fails

Do you happen to have dumps of this? This sounds like something Swapdoodle-specific. In my testing, the CompletePostObject and CompletePostObjectV1 methods were only able to be used on data that was uploaded with DATA_FLAG_NEED_COMPLETION, resulting in an error if the data was already accessible, and objects could not be accessed on the server if this flag was set until one of those functions was called. The client shouldn't even be calling those methods if the flag is not set? Or maybe this is a version difference? Though that also sounds odd, since according to this page @DaniElectra made Swapdoodle uses NEX 3.10.1 which is very new, but CompletePostObjectV1 is a NEX 2 and below function?

Additionally, since ObjectEnabled also checks for state == 0, the file is actually never accepted because it's always initialized to 0 but that's likely to be fixed anyway

An objects status is only set to 0 if DATA_FLAG_NEED_REVIEW is not set during upload. The status of an object just indicates its visibility status to the recipients. An object can be either DATA_STATUS_PENDING when under review or DATA_STATUS_REJECTED when the object is (presumably) rejected after review. If an object has either of these states, then only the owner of the object can view/update the objects data no matter what the permission settings are. Otherwise the status is DATA_STATUS_NONE, indicating that normal permission checks should be done

Compared to the objects upload status, which treats the object as if it doesn't even exist until the upload is "completed". This is distinctly different from visibility status

That's why the checks are done this way. If an object is not visible or not "completed", then it's treated as not "enabled"

This callback is actually never run, because CompletePostObjectV1 does not call it. This likely won't be needed in the future when notification support is added to the package, I'm currently creating notifications manually for the sake of maintaining functionality

The callback system is being largely ignored right now (you can see that most methods have no callbacks). The callback you're linking to is only intended for CompletePostObject, not CompletePostObjectV1. I haven't added the callbacks to the rework PR (outside of the existing ones) because I'm not sure if I want to keep them or not. Callbacks are a design pattern that's not super used in Go, as it favors more explicit control flow. I've been thinking about instead just removing the callbacks and exporting the default method handlers in the common lib, and then if people need to do something before or after a NEX method is called they would just define their own handler for said method which calls the exported common method internally. Such as:

// CompletePostObjectV1 is a custom handler for DataStore::CompletePostObjectV1
// defined by the game server
func (commonProtocol *CommonProtocol) CompletePostObjectV1(err error, packet nex.PacketInterface, callID uint32, param datastore_types.DataStoreCompletePostParamV1) (*nex.RMCMessage, *nex.Error) {
	fmt.Println("Do something before method is handled")

	// Call the common implementation internally
	message, errCode := common_datastore.CompletePostObjectV1(err, packet, callID, param)

	fmt.Println("Do something after method is handled")

	return message, errCode
}

[silver-volt4]

Interesting... Thanks for the info, I haven't really dived into NEX much outside of Swapdoodle. Now I understand the upload process a bit more clearly.

Here is a mitmproxy capture of my 3DS uploading a note that I have just taken. One thing to note right away is the version header on the HPP requests, it is set to NEX_2_4_1_S25. Unless NEX/HPP uses some exotic versioning, I'd say that doesn't look like NEX 3.10.1 at all 😀 So at least that can be explained.

FYI, Mitmproxy makes it a bit tedious to extract the raw NEX packet from the HTTP request. I use this bash script (courtesy of @CenTdemeern1) to extract the packet from a hex dump of the multipart data (which is easy to view in mitm):

xxd -r - | python3 -c 'import sys;s = sys.stdin.read();i = s.find("\r\n"*2)+4;sys.stdout.write(s[i:s.find("----------", i)-2]);sys.exit(0);' | xxd -p

Though honestly, I'm still not convinced that checking for ObjectEnabled during CompletePostObject is correct - unless all files uploaded have the DATA_FLAG_NEED_REVIEW flag set? Or maybe I'm just overlooking something obvious

[jonbarrow]

Sorry, I sent that message at 2am and misunderstood something you had said

Though honestly, I'm still not convinced that checking for ObjectEnabled during CompletePostObject is correct - unless all files uploaded have the DATA_FLAG_NEED_REVIEW flag set? Or maybe I'm just overlooking something obvious

No, you're correct. My apologies. The object status is being incorrectly used here. EnableObject also sets the status to DATA_STATUS_NONE which shouldn't be happening as that would mean that an object which needs review would suddenly no longer be in the pending state after being uploaded. The upload status and the visibility status are 2 different things, so you're right that CompletePostObjectV1/CompletePostObject should not be using the visibility status at all

the version header on the HPP requests, it is set to NEX_2_4_1_S25. Unless NEX/HPP uses some exotic versioning, I'd say that doesn't look like NEX 3.10.1 at all 😀 So at least that can be explained.

Let me ping @DaniElectra on this one. The page he made was generated using the titles reported library versions iirc, so if it's reporting 3.10.1 then it was at least compiled with that version. But I've never seen a case where a title is using a different version than what it was compiled with. That's very odd. The _S25 at the end does make me think it's doing something custom, though, as typically a title will only have that sort of "extra" version data if it made changes to the protocols (for example Super Mario Maker uses 3.8.12-AMA on Wii U and 3.8.9-AMA on 3DS, where AMA is the games product ID)

That being said, since the issue seems to just be that CompletePostObjectV1/CompletePostObject are rejecting objects that don't have DATA_FLAG_NEED_COMPLETION set, we can probably just be inaccurate and ignore this check in those methods. We would be inaccurate, which I'd rather not do in most cases, but this one seems harmless enough that it should be fine

FYI, Mitmproxy makes it a bit tedious to extract the raw NEX packet from the HTTP request. I use this bash script (courtesy of @CenTdemeern1) to extract the packet from a hex dump of the multipart data (which is easy to view in mitm):

Thank you for that! I might add support for flows.txt dumps into the NEX Viewer rewrite tbh

[DaniElectra]

HPP connections are made by the BOSS sysmodule, which bundles NEX by itself, so the version between that the games have and the version used on the HPP connection will be different. As the console sends on the request, HPP on the 3DS uses NEX 2.4.1 NEX_2_4_1_S25 because that's the version that the BOSS sysmodule bundles, but Swapdoodle is bundled with NEX 3.10.1 within the game.

The same also applies on the Wii U, where the NEX version that is bundled in BOSS is 3.5.2, while Smash 4 has NEX version 3.6, so the HPP requests that the game does through BOSS will use version 3.5.2

[silver-volt4]

There's one more thing to Swapdoodle that I probably should have mentioned sooner. I apologise if it causes any trouble.

In my testing, I've found that Swapdoodle refuses to upload notes if it can't download a file called dstsetting from BOSS. Returning a 304 is enough to make it work, even on a completely empty save file (when no If-Modified-Since HTTP header is sent), which is what I ultimately ended up locally patching into the BOSS server to make Swapdoodle work when hosted on my computer.

The likely only possible way to get this file is by digging it out of the 3DS somehow (I'm not that experienced with the 3DS's internals to know if/how is that possible), since nearly all packet dumps will just have a 304 from the original BOSS server.

[Duplicake-fyi]

Any updates?

[silver-volt4]

Any updates?

Please read this post by Jon.

[CLAassistant]

All committers have signed the CLA.

[silver-volt4]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

This looks like a cheap phishing attempt, but the repo it belongs to has 1.5K stars. What the hell?

ETA: https://forum.pretendo.network/t/github-is-this-a-phishing-attempt/26759

[jonbarrow]

It's a legitimate bot we use. It was just seemingly broken for a while and no one noticed, it was only fixed yesterday

The CLA is just an acknowledgment that you own the rights to propose the changes in the PR. We started using it after a few people tried to PR in information from Nintendo SDK leaks, as an effort to officially take a stance against that practice

After the bot got fixed it sent out a ton of messages from the backlog, so it looks a little spammy

[silver-volt4]

It's a legitimate bot we use. It was just seemingly broken for a while and no one noticed, it was only fixed yesterday

Ah, understood. I signed it just now.

[Duplicake-fyi]

seems like this pr is the last one left to be merged! PretendoNetwork/nex-protocols-common-go#53

[silver-volt4]

seems like this pr is the last one left to be merged! PretendoNetwork/nex-protocols-common-go#53

woo! 🎉 really excited to see that 😊 I haven't checked on the progress in a while, I think I'll just pull the PR branch and test the app with it, to see if the problems I've encountered have been addressed

[jonbarrow]

seems like this pr is the last one left to be merged! PretendoNetwork/nex-protocols-common-go#53

woo! 🎉 really excited to see that 😊 I haven't checked on the progress in a while, I think I'll just pull the PR branch and test the app with it, to see if the problems I've encountered have been addressed

Sorry for the delays. We had other things we were focusing on until that PR merged, and I plan to finish the DataStore rework soon too. I wanted to spend some time and actually finish the documentation for the protocol though, and get it posted on the new Nintendo wiki and I only just finished it today https://nintendo.wiki/wiki/Online/Nintendo_Network/NEX/Protocols/DataStoreProtocol

[jonbarrow]

• Swapdoodle seems to be uploading doodles with the needUploadCompletion flag set to false, which in turn causes uploadCompleted to be set to true, and so CompletePostObjectV1 fails

I have confirmed this behavior, also. The errors I was seeing before were likely from the fact that I had tried to "complete" an object multiple times, which results in DataStore::Unknown

All objects, regardless of the completion flag, need to be "completed". I tested this earlier today on the official servers by uploading an object with no flags set, and I was unable to access it until calling CompleteObject. I'm not sure why this flag exists, possibly something from older versions, who knows

I'll likely add a flag to the common protocols called something like "respect completion flag", which is disabled by default, so that all objects are required to be manually completed. Then if it is some weird version thing, and we end up seeing some clients trying to access objects without marking them as completed and without the completion flag set, we can selectively enable this for those games

[Duplicake-fyi]

So excited for it to be done! I hope I can start using swapdoodle in 2027

[silver-volt4]

Hi there, do we need to take any action to support opaque tokens?
accounts.go is the only file that works with accounts and it only takes PIDs and Usernames, so I think all I need to do is update the used libraries, but I figured it wouldn't hurt to ask.

[DaniElectra]

Swapdoodle, since it uses HPP, won't see any tokens, so you don't need to do anything other than the other updates like the accounts.go one yeah