MacOS headless usage

[Stef16Robbe]

Hi there,

First of all, thanks for making this package, it works perfectly for the protected site(s) I want to scrape!
My issue is that headless mode does not work on Mac. You already mentioned this in your README:

The headless option only works on linux, and requires Xvfb to be installed. Could theoretically work on Mac OS with xquartz but I don't have a Mac to test with, so feel free to PR.

I did install xquartz and your example only works by removing the headless option.

You mentioned you can't test it on here so I'm willing to do so but I would need some pointers on where to start if possible.

Thanks!

[Davincible]

Hey, @Stef16Robbe nice to hear! I am not too sure myself as well. I mentioned it in the readme after I did some googling when creating this package, but I'm not up to date on what is required to make it work with xquartz. Xvfb is used as a virtual display, to support headless mode, and xquartz is supposed to replace that.

If you have the time I would suggest to look up some examples of xquartz usage and how we can use that as a virtual display.

If you have anything specific I can help brainstorm

[Stef16Robbe]

So I've taken a look already and xquartz does not seem to be the real/first problem here. I installed it and with it came the xvfb cli util, which I'm assuming will function similarly.

The issue I'm tackling right now is that the Darwin build of Go does not support syscall.SysProcAttr.Pdeathsig (used here)

• Darwin build
• Linux build

So I'll have to find a way to manually kill it on exit. Working on that first.

[Davincible]

Yeah windows has the same problem, that's why its separated now. But asides from killing it, does xvfb seem to work then?

[Stef16Robbe]

So just plain copy-pasting the linux build to darwin and running the tests does not work, I get an EOF error on these lines in display_darwin.go

bufr.ReadString('\n') is returning the EOF error, stated in it's docs:

If ReadString encounters an error before finding a delimiter, it returns the data read before the error and the error itself (often io.EOF). ReadString returns err != nil if and only if the returned data does not end in delim.

...
	go func() {
		bufr := bufio.NewReader(pipeReader)
		s, err := bufr.ReadString('\n')
		ch <- resp{s, err}
	}()

	var display string
	select {
	case resp := <-ch:
		if resp.err != nil {
			log.Println("newFrameBuffer() error: ", resp.err)

			return nil, resp.err
		}
...

➜  chromedp-undetected (master) go test -v -count=1 . 
=== RUN   TestRunCommand
    chrome_test.go:39: Attempt 1/3
2023/06/04 19:51:42 newFrameBuffer() error:  EOF
    chrome_test.go:39: Attempt 2/3
2023/06/04 19:51:42 newFrameBuffer() error:  EOF
    chrome_test.go:39: Attempt 3/3
2023/06/04 19:51:42 newFrameBuffer() error:  EOF
    chrome_test.go:71: 3 errors occurred:
        	* create context: EOF
        	* create context: EOF
        	* create context: EOF
...

I'm new to Go and unfamiliar with Goroutines so I'm not entirely sure how data is getting fed into the Reader but I'll take a deeper dive later.

[Stef16Robbe]

So, some testing later I managed to get rid of the error above, except I'm not entirely sure how. I ran the tests with sudo a couple of times because of what I read here, and I believe I ran the xauth command manually once which might have generated some files...?

I get multiple running instances of xvfb because they aren't killed like we discussed above:

scraping/chromedp-undetected [master●] » ps -ef | grep -i 'xvfb'   
    0 12823     1   0  8:25PM ttys000    0:00.05 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12827     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12831     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12835     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12839     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12843     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12847     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12851     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12855     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12859     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12863     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24
    0 12867     1   0  8:25PM ttys000    0:00.04 Xvfb -displayfd 3 -nolisten tcp -screen 0 1920x1080x24

And then errors from xauth saying it can't access the Security extension (tried with & without sudo)?:

scraping/chromedp-undetected [master●] » sudo go test -v -count=1 .
=== RUN   TestRunCommand
    chrome_test.go:39: Attempt 1/3
/opt/X11/bin/xauth generate :0 . trusted
xauth: (argv):1:  couldn't query Security extension on display ":0"
    chrome_test.go:39: Attempt 2/3
/opt/X11/bin/xauth generate :0 . trusted
xauth: (argv):1:  couldn't query Security extension on display ":0"
    chrome_test.go:39: Attempt 3/3
/opt/X11/bin/xauth generate :0 . trusted
xauth: (argv):1:  couldn't query Security extension on display ":0"
    chrome_test.go:71: 3 errors occurred:
        	* create context: exit status 1
        	* create context: exit status 1
        	* create context: exit status 1
...

I have never worked with xauth / xvfb before so this is a little out of my league ATM. Let me know if you have some things I can try. In the meantime I'll just be using this module without headless mode which is working just fine for now 😊