Waveform query channel expansion now not supported

[calum-chamberlain]

This kind of follows on from #129 - previously (yesterday morning) channel codes were handled so that multiple channel codes to be queried at the same time. Now this returns a 204 code.

Example Python/obspy code:

from obspy import UTCDateTime
from obspy.clients.fdsn import Client

stations = ['OGWZ', 'TUWZ', 'BSWZ', 'MSWZ', 'CAW', 'DUWZ', 'MRZ', 'KIW', 'CMWZ', 'WEL']
client = Client("GEONET")
t1 = UTCDateTime(2016, 11, 14)
t2 = t1 + 86400

bulk = []
for station in stations:
    bulk.append(('NZ', station, '10', '[EH]H?', t1, t2))
data = client.get_waveforms_bulk(bulk=bulk)

Yesterday morning this worked, but as of yesterday afternoon, not so much:

FDSNNoDataException                       Traceback (most recent call last)
<ipython-input-219-3630116c6e60> in <module>()
----> 1 data = client.get_waveforms_bulk(bulk=bulk)

~/my_programs/Building/obspy/obspy/clients/fdsn/client.py in get_waveforms_bulk(self, bulk, quality, minimumlength, longestonly, filename, attach_response, **kwargs)
   1007 
   1008         data_stream = self._download(url,
-> 1009                                      data=bulk)
   1010         data_stream.seek(0, 0)
   1011         if filename:

~/my_programs/Building/obspy/obspy/clients/fdsn/client.py in _download(self, url, return_string, data, use_gzip)
   1380             debug=self.debug, return_string=return_string, data=data,
   1381             timeout=self.timeout, use_gzip=use_gzip)
-> 1382         raise_on_error(code, data)
   1383         return data
   1384 

~/my_programs/Building/obspy/obspy/clients/fdsn/client.py in raise_on_error(code, data)
   1704     if code == 204:
   1705         raise FDSNNoDataException("No data available for request.",
-> 1706                                   server_info)
   1707     elif code == 400:
   1708         msg = ("Bad request. If you think your request was valid "

FDSNNoDataException: No data available for request.
Detailed response of server:

Might I suggest that the FDSN system has a regression test for this behaviour?

[calum-chamberlain]

I can't actually see any tests in this repository, are there any tests run by ci before changes are made?
Apologies, tests are obviously run by travis: https://github.com/GeoNet/fdsn/blob/master/.travis.yml#L36

[junghao]

The support to regex query has been removed yesterday since it's not in the spec and might cause some confusion (of level of regular expression supporting).

https://github.com/GeoNet/fdsn/blob/master/internal/fdsn/dataselect_test.go#L63
(Starts from line 91 is the way we treat regex special chars)

[calum-chamberlain]

Could we then have a better error message? Also, FDSN spec states that 204 error code is for:

Request was properly formatted and submitted but no data matches the selection
It sounds like if you have moved regex out of spec this shouldn't be the error code returned (perhaps 400?).

Agree on the causing confusion!

[junghao]

The query parameter here "[EH]H?" is not an invalid query. Our service just not taking it as regular expression. You're telling our service to look for stations which has a left bracket then "E" then "H" then right bracket then "H" then an any char -> which doesn't exist.

[calum-chamberlain]

Fair, I would expect 6-character channel names to be invalid (following SEED spec)?

[nbalfour]

The specification suggests using wildcards and lists to get multiple channels. I hope that helps.

From the spec (https://www.fdsn.org/webservices/FDSN-WS-Specifications-1.1.pdf):

Wildcards and lists in channel constraints parameters
The channel constraint parameters (network, station, location and channel) may include two specific wildcard characters with the following meaning:
* – Matches zero to many characters
? – Matches any single character
The channel constraint parameters may also be submitted as comma-separated lists in order to
select two or more values with a single request. For example, the channel parameter may be used
to specify multiple channels:
channel=LHE,LHN,LHZ,BHZ (the individual values may also include wildcards)

[junghao]

Though FDSN spec says:
"String type parameters may include ASCII characters, including the asterisk (*) and question mark (?) that have special meaning as wildcards for specific parameters." (most common special chars are ASCII as well)

But I agree that if we return 400 error for queries have regex chars, the user will know that we don't like regex. They won't misunderstood that "we've searched by regex but no result". (I think IRIS does this way)

[calum-chamberlain]

FDSN spec also says:

channel | Required | Select one or more SEED channel codes. Multiple codes are comma-separated.

And SEED says:

The SEED format uses three letters to name seismic channels, and three letters to name weather or environmental channels.

But yup, agree, 400 for regex chars would be good in general (obviously apart from * and ?) - an error message alongside that which says that regex except * and ? is not supported would be really helpful too.