go get fails

[psankar]

➜  ~  go get github.com/narula/ddtxn/...
package github.com/narula/ddtxn/...
        imports github.com/narula/ddtxn
        imports ddtxn/dlog: unrecognized import path "ddtxn/dlog"
package github.com/narula/ddtxn/...
        imports github.com/narula/ddtxn
        imports ddtxn/spinlock: unrecognized import path "ddtxn/spinlock"
package github.com/narula/ddtxn/...
        imports github.com/narula/ddtxn
        imports ddtxn/wfmutex: unrecognized import path "ddtxn/wfmutex"
package github.com/narula/ddtxn/...
        imports github.com/narula/ddtxn
        imports gotomic: unrecognized import path "gotomic"
package github.com/narula/ddtxn/...
        imports github.com/narula/ddtxn/apps
        imports ddtxn: unrecognized import path "ddtxn"
package github.com/narula/ddtxn/...
        imports github.com/narula/ddtxn/benchmarks
        imports ddtxn/apps: unrecognized import path "ddtxn/apps"
package github.com/narula/ddtxn/...
        imports github.com/narula/ddtxn/benchmarks
        imports ddtxn/prof: unrecognized import path "ddtxn/prof"
➜  ~  go version
go version go1.4 linux/amd64

[psankar]

The above error is caused because the imports are referring to "ddtxn/prof" etc. instead of the absolute path starting from the github url.

Even after fixing that, the go get command fails because of the following "redeclared" errors due to multiple files in the same directory sharing the same variable names.

➜  ddtxn git:(master) ✗ go get github.com/narula/ddtxn/...
# github.com/narula/ddtxn/benchmarks
benchmarks/big.go:18: nprocs redeclared in this block
        previous declaration at benchmarks/bid.go:21
benchmarks/big.go:19: nsec redeclared in this block
        previous declaration at benchmarks/bid.go:22
benchmarks/big.go:20: clientGoRoutines redeclared in this block
        previous declaration at benchmarks/bid.go:23
benchmarks/big.go:21: nworkers redeclared in this block
        previous declaration at benchmarks/bid.go:24
benchmarks/big.go:22: doValidate redeclared in this block
        previous declaration at benchmarks/bid.go:25
benchmarks/big.go:24: contention redeclared in this block
        previous declaration at benchmarks/bid.go:27
benchmarks/big.go:25: nbidders redeclared in this block
        previous declaration at benchmarks/bid.go:28
benchmarks/big.go:26: readrate redeclared in this block
        previous declaration at benchmarks/bid.go:29
benchmarks/big.go:27: notcontended_readrate redeclared in this block
        previous declaration at benchmarks/bid.go:30
benchmarks/big.go:29: dataFile redeclared in this block
        previous declaration at benchmarks/bid.go:31
benchmarks/big.go:29: too many errors

I have fixed both of them. Patch is to follow shortly.

[psankar]

Is this project no longer actively maintained ?

[narula]

I'm traveling till 1/21. Will get to it after that.
On Jan 12, 2015 12:05 PM, "Sankar சங்கர்" notifications@github.com wrote:

Is this project no longer actively maintained ?

—
Reply to this email directly or view it on GitHub
#3 (comment).

[psankar]

ah okay. Sorry for the noise.

[narula]

No worries. Thanks for the interest!
On Jan 12, 2015 2:43 PM, "Sankar சங்கர்" notifications@github.com wrote:

ah okay. Sorry for the noise.

—
Reply to this email directly or view it on GitHub
#3 (comment).

[narula]

Should work now. Mind testing?

[psankar]

go get github.com/narula/ddtxn/ works. But when we do a go install ./single from the benchmarks folder it complains of a missing prof package.

So I did a go get github.com/narula/ddtxn/... to install all the subdirs and their dependencies too. But that lead to a build error:

➜  narula  go get github.com/narula/ddtxn/...
# github.com/narula/ddtxn/benchmarks/time-single
ddtxn/benchmarks/time-single/time-single.go:197: assignment count mismatch: 3 = 7
# github.com/narula/ddtxn/benchmarks/bid
ddtxn/benchmarks/bid/bid.go:195: assignment count mismatch: 3 = 7
# github.com/narula/ddtxn/benchmarks/big
ddtxn/benchmarks/big/big.go:61: cannot use coord.Workers[0].E (type ddtxn.ETransaction) as type *ddtxn.ETransaction in argument to big_app.Populate:
        *ddtxn.ETransaction is pointer to interface, not interface
ddtxn/benchmarks/big/big.go:82: undefined: apps.Latency
ddtxn/benchmarks/big/big.go:94: undefined: apps.Latency
ddtxn/benchmarks/big/big.go:95: big_app.Time undefined (type *apps.Big has no field or method Time)
ddtxn/benchmarks/big/big.go:115: assignment count mismatch: 3 = 7
ddtxn/benchmarks/big/big.go:132: big_app.LatencyString undefined (type *apps.Big has no field or method LatencyString)

If we give all the four go get statements in the project homepage, then the go install ./single is successful (but it fails at runtime due to a different error (missing list-cpus) which I am trying to debug to understand if it is any issue at my end).

However, the issue of "redeclared" seem to be fixed.

[narula]

Pull and try again. I left some old packages around and I hadn't updated
all the benchmarks to use the new stats code. Also make sure you do:

go get github.com/narula/prof
go get github.com/narula/wfmutex
go get -u github.com/narula/ddtxn

Thanks!

On Thu, Feb 12, 2015 at 11:48 PM, Sankar சங்கர் notifications@github.com
wrote:

go get github.com/narula/ddtxn/
works. But when we do a go install ./single from the benchmarks folder it
complains of a missing prof package.

So I did a go get github.com/narula/ddtxn/... to install all the subdirs
and their dependencies too. But that lead to a build error:

➜ narula go get github.com/narula/ddtxn/...

github.com/narula/ddtxn/benchmarks/time-single

ddtxn/benchmarks/time-single/time-single.go:197 http://github.com/narula/ddtxn/benchmarks/time-singleddtxn/benchmarks/time-single/time-single.go:197: assignment count mismatch: 3 = 7

github.com/narula/ddtxn/benchmarks/bid

ddtxn/benchmarks/bid/bid.go:195 http://github.com/narula/ddtxn/benchmarks/bidddtxn/benchmarks/bid/bid.go:195: assignment count mismatch: 3 = 7

github.com/narula/ddtxn/benchmarks/big

ddtxn/benchmarks/big/big.go:61 http://github.com/narula/ddtxn/benchmarks/bigddtxn/benchmarks/big/big.go:61: cannot use coord.Workers[0].E (type ddtxn.ETransaction) as type *ddtxn.ETransaction in argument to big_app.Populate:
*ddtxn.ETransaction is pointer to interface, not interface
ddtxn/benchmarks/big/big.go:82: undefined: apps.Latency
ddtxn/benchmarks/big/big.go:94: undefined: apps.Latency
ddtxn/benchmarks/big/big.go:95: big_app.Time undefined (type *apps.Big has no field or method Time)
ddtxn/benchmarks/big/big.go:115: assignment count mismatch: 3 = 7
ddtxn/benchmarks/big/big.go:132: big_app.LatencyString undefined (type *apps.Big has no field or method LatencyString)

If we give all the four go get statements in the project homepage, then
the go install ./single is successful (but it fails at runtime due to a
different error (missing list-cpus) which I am trying to debug to
understand if it is any issue at my end).

However, the issue of "redeclared" seem to be fixed.

—
Reply to this email directly or view it on GitHub
#3 (comment).

[psankar]

There still seem to be a failure.

➜  narula  go get -u github.com/narula/ddtxn/...
# github.com/narula/ddtxn/benchmarks/big
ddtxn/benchmarks/big/big.go:61: cannot use &coord.Workers[0].E (type *ddtxn.ETransaction) as type ddtxn.ETransaction in argument to big_app.Populate:
        *ddtxn.ETransaction is pointer to interface, not interface
➜  narula

If we do at the top-level, it will succeed though as the subdirs will not be installed.

There is another issue related to a missing list-cpus command. That leads to a runtime failure. Should I report that also ? Where do I get this from ? The log for that in case you want:

➜  benchmarks git:(f0330d3) go install ./single              
➜  benchmarks git:(f0330d3) python bm.py --exp=single --rlock --ncores=3
GOGC=off numactl -C `list-cpus seq -n 3 ` $GOPATH/bin/single -nprocs=3 -ngo=3 -nw=3 -nsec=10 -contention=0 -rr=0 -allocate=False -sys=0 -rlock=False -wr=2.0 -phase=20 -sr=500 -atomic=False -zipf=-1 -out=data.out -ncrr=0.0 -cw=2.00 -rw=0.50 -split=False -latency=False -v=0 -partition=False
Bad status 256 sh: list-cpus: command not found
libnuma: Warning: unparseable cpu description `/home/psankar/go/bin/single'

usage: numactl [--all | -a] [--interleave= | -i <nodes>] [--preferred= | -p <node>]
               [--physcpubind= | -C <cpus>] [--cpunodebind= | -N <nodes>]
               [--membind= | -m <nodes>] [--localalloc | -l] command args ...
       numactl [--show | -s]
       numactl [--hardware | -H]
       numactl [--length | -l <length>] [--offset | -o <offset>] [--shmmode | -M <shmmode>]
               [--strict | -t]
               [--shmid | -I <id>] --shm | -S <shmkeyfile>
               [--shmid | -I <id>] --file | -f <tmpfsfile>
               [--huge | -u] [--touch | -T] 
               memory policy | --dump | -d | --dump-nodes | -D

memory policy is --interleave | -i, --preferred | -p, --membind | -m, --localalloc | -l
<nodes> is a comma delimited list of node numbers or A-B ranges or all.
Instead of a number a node can also be:
  netdev:DEV the node connected to network device DEV
  file:PATH  the node the block device of path is connected to
  ip:HOST    the node of the network device host routes through
  block:PATH the node of block device path
  pci:[seg:]bus:dev[:func] The node of a PCI device
<cpus> is a comma delimited list of cpu numbers or A-B ranges or all
all ranges can be inverted with !
all numbers and ranges can be made cpuset-relative with +
the old --cpubind argument is deprecated.
use --cpunodebind or --physcpubind instead
<length> can have g (GB), m (MB) or k (KB) suffixes

[narula]

Aha. I think I fixed that bug.

I added list-cpus to a github repo. Get this:

git clone git@github.com:narula/list-cpus.git

And add it to your path.

Thank you for bearing with me!

On Fri, Feb 13, 2015 at 11:08 AM, Sankar சங்கர்
notifications@github.com wrote:

There still seem to be a failure.

➜ narula go get -u github.com/narula/ddtxn/...

github.com/narula/ddtxn/benchmarks/big

ddtxn/benchmarks/big/big.go:61: cannot use &coord.Workers[0].E (type
*ddtxn.ETransaction) as type ddtxn.ETransaction in argument to
big_app.Populate:
*ddtxn.ETransaction is pointer to interface, not interface
➜ narula

If we do at the top-level, it will succeed though as the subdirs will not be
installed.

There is another issue related to a missing list-cpus command. That leads to
a runtime failure. Should I report that also ? Where do I get this from ?
The log for that in case you want:

➜ benchmarks git:(f0330d3) go install ./single
➜ benchmarks git:(f0330d3) python bm.py --exp=single --rlock --ncores=3
GOGC=off numactl -C list-cpus seq -n 3 $GOPATH/bin/single -nprocs=3
-ngo=3 -nw=3 -nsec=10 -contention=0 -rr=0 -allocate=False -sys=0
-rlock=False -wr=2.0 -phase=20 -sr=500 -atomic=False -zipf=-1 -out=data.out
-ncrr=0.0 -cw=2.00 -rw=0.50 -split=False -latency=False -v=0
-partition=False
Bad status 256 sh: list-cpus: command not found
libnuma: Warning: unparseable cpu description `/home/psankar/go/bin/single'

usage: numactl [--all | -a] [--interleave= | -i ] [--preferred= | -p
]
[--physcpubind= | -C ] [--cpunodebind= | -N ]
[--membind= | -m ] [--localalloc | -l] command args
...
numactl [--show | -s]
numactl [--hardware | -H]
numactl [--length | -l ] [--offset | -o ] [--shmmode
| -M ]
[--strict | -t]
[--shmid | -I ] --shm | -S
[--shmid | -I ] --file | -f
[--huge | -u] [--touch | -T]
memory policy | --dump | -d | --dump-nodes | -D

memory policy is --interleave | -i, --preferred | -p, --membind | -m,
--localalloc | -l
is a comma delimited list of node numbers or A-B ranges or all.
Instead of a number a node can also be:
netdev:DEV the node connected to network device DEV
file:PATH the node the block device of path is connected to
ip:HOST the node of the network device host routes through
block:PATH the node of block device path
pci:[seg:]bus:dev[:func] The node of a PCI device
is a comma delimited list of cpu numbers or A-B ranges or all
all ranges can be inverted with !
all numbers and ranges can be made cpuset-relative with +
the old --cpubind argument is deprecated.
use --cpunodebind or --physcpubind instead
can have g (GB), m (MB) or k (KB) suffixes

—
Reply to this email directly or view it on GitHub.

[psankar]

Adding the contents of that git repository fixed the runtime error. May be it is a good idea to document it in the README or even better package these also in this same git project ?

Also, do you think it could be a good idea to move the individual benchmarks to the individual folders so that a single go get -u <narula/ddtxn>/... will install all the benchmarks ?

I should indeed thank you as it is not very easy to work on interesting projects for newbies :)

[narula]

I added the list-cpus stuff to the README. It might prove useful for
another project so I left it in a different repo.

What do you mean when you say move the individual benchmarks to individual
folders? You mean so that bid, big, buy, rubis, single, and time-single
are in the top-level directory instead of /benchmarks?

On Fri, Feb 13, 2015 at 12:02 PM, Sankar சங்கர் notifications@github.com
wrote:

Adding the contents of that git repository fixed the runtime error. May be
it is a good idea to document it in the README or even better package these
also in this same git project ?

Also, do you think it could be a good idea to move the individual
benchmarks to the individual folders so that a single go get -u
<narula/ddtxn>/... will install all the benchmarks ?

I should indeed thank you as it is not very easy to work on interesting
projects for newbies :)

—
Reply to this email directly or view it on GitHub
#3 (comment).

[psankar]

What do you mean when you say move the individual benchmarks to individual
folders?

Sorry. Ignore my comment. I was somehow confused and thought that the big, bid, simple etc. are files in benchmarks folder at one-level and not in their own individual subfolders. I was wrong. Everything is fine now. go get -u /... also works perfectly. Thanks. The issue can be closed.

[psankar]

https://github.com/jteeuwen/go-bindata helps to package non-go files also via go get it seems. I have not tried it yet but heard about it from another issue in another project hound-search/hound#7

I will see if I could use the go-bindata to package the python files (list-cpu etc.).