In all of the test files (test/graph_tests.cpp, test/algorithm_tests.cpp) you will see at the top:
auto const memserver_grpc_addr = MEMADDR;
auto const ipoib_addr = "192.168.12.2";
auto const ipoib_port = "35287";
The first assignment is reading a value from a configuration file (test/constants.hpp.in)
#ifndef _CONSTANTS_H_
#define _CONSTANTS_H_
#cmakedefine MEMADDR "@MEMADDR@"
#cmakedefine MMAP_TEST1 "@MMAP_TEST1@"
#cmakedefine FGIDX_TEST1 "@FGIDX_TEST1@"
#cmakedefine TEST_GRAPH_DIR "@TEST_GRAPH_DIR@"
#cmakedefine LARGE_TEST_GRAPH_DIR "@LARGE_TEST_GRAPH_DIR@"
#endif
And these constants are in turn defined in test/CMakeLists.txt
set(MEMADDR "0.0.0.0:50051" CACHE STRING "IP of Memserver Ethernet Interface")
set(MMAP_TEST1 "${CMAKE_CURRENT_SOURCE_DIR}/files/mmap_test1" CACHE STRING "test file for remote mmap")
set(FGIDX_TEST1 "${CMAKE_CURRENT_SOURCE_DIR}/files/fgidxtest1.idx" CACHE STRING "test file for .idx")
set(TEST_GRAPH_DIR "${CMAKE_CURRENT_SOURCE_DIR}/test_graphs" CACHE STRING "Graph Inputs")
set(LARGE_TEST_GRAPH_DIR "/netscratch/fam-graph" CACHE STRING "Large Graph Inputs")
configure_file("constants.hpp.in" "${CMAKE_CURRENT_BINARY_DIR}/constants.hpp")Where the last line takes the test/constants.hpp.in file and generates a file ${CMAKE_CURRENT_BINARY_DIR}/constants.hpp. We want to take these two lines from the first code block:
auto const ipoib_addr = "192.168.12.2";
auto const ipoib_port = "35287";
And make them look like:
auto const ipoib_addr = "MEMSERVER_IPOIB";
auto const ipoib_port = "MEMSERVER_RDMA_PORT";
To do this you will need to update test/constants.hpp.in and test/CMakeLists.txt accordingly. Make the current hard coded values the defaults. Test this out by doing a deployment on aries1 (right now these variables are hard coded for aries2). Pass aries1's IPoIB address to the build using cmake .. -DMEMSERVER_IPOIB="192.168.12.1"
The above should solve the hard-coding on the client side, but we will also need to change the server to match up. The server should take the rdma port as a command line parameter. The server already takes the GRPC IP/port as a command line argument. From src/server.cpp:
po::options_description desc{ "Options" };
desc.add_options()("help,h", "Help screen")("server-addr,a",
po::value<std::string>()->default_value("0.0.0.0"),
"Server's IPoIB addr")(
"port,p", po::value<std::string>()->default_value("50051"), "server port");
po::variables_map vm;
po::store(po::parse_command_line(argc, argv, desc), vm);
po::notify(vm);The commandline parser is generated by boost program options. You can make an option for the rdma port and set its default value to 35287. The place where the RDMA port is ultimately set is in FAM_rdma.hpp:
void inline bind_addr(rdma_cm_id *const id)
{
sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_port = htons(static_cast<uint16_t>(35287));
auto const err =
rdma_bind_addr(id, reinterpret_cast<struct sockaddr *>(&addr));
if (err) throw std::runtime_error("rdma_bind_addr() failed");
}The hard-coded value should be replaced with the new commandline option. On the server side you don't need to pass in the IPoIB address, as the rdma_cm library automatically finds ib0.
To test this out you should be able to configure two different cmake builds using two sets of GRPC and RDMA IP/port combos, then run two server processes simultaneously and run the test suites from both builds.
In all of the test files (test/graph_tests.cpp, test/algorithm_tests.cpp) you will see at the top:
The first assignment is reading a value from a configuration file (test/constants.hpp.in)
And these constants are in turn defined in test/CMakeLists.txt
Where the last line takes the test/constants.hpp.in file and generates a file ${CMAKE_CURRENT_BINARY_DIR}/constants.hpp. We want to take these two lines from the first code block:
And make them look like:
To do this you will need to update test/constants.hpp.in and test/CMakeLists.txt accordingly. Make the current hard coded values the defaults. Test this out by doing a deployment on aries1 (right now these variables are hard coded for aries2). Pass aries1's IPoIB address to the build using
cmake .. -DMEMSERVER_IPOIB="192.168.12.1"The above should solve the hard-coding on the client side, but we will also need to change the server to match up. The server should take the rdma port as a command line parameter. The server already takes the GRPC IP/port as a command line argument. From src/server.cpp:
po::options_description desc{ "Options" }; desc.add_options()("help,h", "Help screen")("server-addr,a", po::value<std::string>()->default_value("0.0.0.0"), "Server's IPoIB addr")( "port,p", po::value<std::string>()->default_value("50051"), "server port"); po::variables_map vm; po::store(po::parse_command_line(argc, argv, desc), vm); po::notify(vm);The commandline parser is generated by boost program options. You can make an option for the rdma port and set its default value to 35287. The place where the RDMA port is ultimately set is in FAM_rdma.hpp:
The hard-coded value should be replaced with the new commandline option. On the server side you don't need to pass in the IPoIB address, as the rdma_cm library automatically finds ib0.
To test this out you should be able to configure two different cmake builds using two sets of GRPC and RDMA IP/port combos, then run two server processes simultaneously and run the test suites from both builds.