NetCDF  4.6.1
auth.md
1 netCDF Authorization Support
2 ======================================
3 <!-- double header is needed to workaround doxygen bug -->
4 
5 # netCDF Authorization Support {#Header}
6 
7 __Author__: Dennis Heimbigner<br>
8 __Initial Version__: 11/21/2014<br>
9 __Last Revised__: 08/24/2017
10 
11 [TOC]
12 
13 ## Introduction {#Introduction}
14 
15 netCDF can support user authorization using the facilities provided by the curl
16 library. This includes basic password authentication as well as
17 certificate-based authorization.
18 
19 At the moment, this document only applies to DAP2 and DAP4 access
20 because they are (for now) the only parts of the netCDF-C library
21 that uses libcurl.
22 
23 With some exceptions (e.g. see the section on <a href="#REDIR">redirection</a>)
24 The libcurl authorization mechanisms can be accessed in two ways
25 
26 1. Inserting the username and password into the url, or
27 2. Accessing information from a so-called _rc_ file named either
28  `.daprc` or `.dodsrc`
29 
30 ## URL-Based Authentication {#URLAUTH}
31 
32 For simple password based authentication, it is possible to
33 directly insert the username and the password into a url in this form.
34 
35  http://username:password@host/...
36 
37 This username and password will be used if the server asks for
38 authentication. Note that only simple password authentication
39 is supported in this format.
40 
41 Specifically note that [redirection-based](#REDIR)
42 authorization may work with this but it is a security risk.
43 This is because the username and password
44 may be sent to each server in the redirection chain.
45 
46 Note also that the `user:password` form may contain characters that must be
47 escaped. See the <a href="#USERPWDESCAPE">password escaping</a> section to see
48 how to properly escape the user and password.
49 
50 ## RC File Authentication {#DODSRC}
51 The netcdf library supports an _rc_ file mechanism to allow the passing
52 of a number of parameters to libnetcdf and libcurl.
53 Locating the _rc_ file is a multi-step process.
54 
55 ### Search Order
56 
57 The file must be called one of the following names:
58 ".daprc" or ".dodsrc".
59 If both ".daprc" and ".dodsrc" exist, then
60 the ".daprc" file will take precedence.
61 
62 It is strongly suggested that you pick one of the two names
63 and use it always. Otherwise you may observe unexpected results
64 when the netcdf-c library finds one that you did not intend.
65 
66 The search for an _rc_ file looks in the following places in this order.
67 
68 1. Check for the environment variable named _DAPRCFILE_.
69  This will specify the full path for the _rc_ file
70  (not just the containing directory).
71 2. Search the current working directory (`./`) looking
72  for (in order) .daprc or .dodsrc.
73 3. Search the HOME directory (`$HOME`) looking
74  for (in order) .daprc or .dodsrc. The HOME environment
75  variable is used to define the directory in which to search.
76 
77 It is strongly suggested that you pick a uniform location
78 and use it always. Otherwise you may observe unexpected results
79 when the netcdf-c library get an rc file you did not expect.
80 
81 ### RC File Format
82 
83 The rc file format is a series of lines of the general form:
84 
85  [<host:port>]<key>=<value>
86 
87 where the bracket-enclosed host:port is optional.
88 
89 ### URL Constrained RC File Entries
90 
91 Each line of the rc file can begin with
92 a host+port enclosed in square brackets.
93 The form is "host:port".
94 If the port is not specified
95 then the form is just "host".
96 The reason that more of the url is not used is that
97 libcurl's authorization grain is not any finer than host level.
98 
99 Examples.
100 
101  [remotetest.unidata.ucar.edu]HTTP.VERBOSE=1
102 
103 or
104 
105  [fake.ucar.edu:9090]HTTP.VERBOSE=0
106 
107 If the url request from, say, the _netcdf_open_ method
108 has a host+port matching one of the prefixes in the rc file, then
109 the corresponding entry will be used, otherwise ignored.
110 This means that an entry with a matching host+port will take
111 precedence over an entry without a host+port.
112 
113 For example, the URL
114 
115  http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc
116 
117 will have HTTP.VERBOSE set to 1 because its host matches the example above.
118 
119 Similarly,
120 
121  http://fake.ucar.edu:9090/dts/test.01
122 
123 will have HTTP.VERBOSE set to 0 because its host+port matches the example above.
124 
125 ## Authorization-Related Keys {#AUTHKEYS}
126 
127 The currently defined set of authorization-related keys are as follows.
128 The second column is the affected curl_easy_setopt option(s), if any
129 (see reference #1).
130 <table>
131 <tr><th>Key</th><th>Affected curl_easy_setopt Options</th><th>Notes</th>
132 <tr><td>HTTP.COOKIEJAR</td><td>CURLOPT_COOKIEJAR</td>
133 <tr><td>HTTP.COOKIEFILE</td><td>CURLOPT_COOKIEJAR</td><td>Alias for CURLOPT_COOKIEJAR</td>
134 <tr><td>HTTP.PROXY.SERVER</td><td>CURLOPT_PROXY, CURLOPT_PROXYPORT, CURLOPT_PROXYUSERPWD</td>
135 <tr><td>HTTP.PROXY_SERVER</td><td>CURLOPT_PROXY, CURLOPT_PROXYPORT, CURLOPT_PROXYUSERPWD</td><td>Decprecated: use HTTP.PROXY.SERVER</td>
136 <tr><td>HTTP.SSL.CERTIFICATE</td><td>CURLOPT_SSLCERT</td>
137 <tr><td>HTTP.SSL.KEY</td><td>CURLOPT_SSLKEY</td>
138 <tr><td>HTTP.SSL.KEYPASSWORD</td><td>CURLOPT_KEYPASSWORD</td>
139 <tr><td>HTTP.SSL.CAINFO</td><td>CURLOPT_CAINFO</td>
140 <tr><td>HTTP.SSL.CAPATH</td><td>CURLOPT_CAPATH</td>
141 <tr><td>HTTP.SSL.VERIFYPEER</td><td>CURLOPT_SSL_VERIFYPEER</td>
142 <tr><td>HTTP.SSL.VALIDATE</td><td>CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST</td>
143 <tr><td>HTTP.CREDENTIALS.USERPASSWORD</td><td>CURLOPT_USERPASSWORD</td>
144 <tr><td>HTTP.CREDENTIALS.USERNAME</td><td>CURLOPT_USERNAME</td>
145 <tr><td>HTTP.CREDENTIALS.PASSWORD</td><td>CURLOPT_PASSWORD</td>
146 <tr><td>HTTP.NETRC</td><td>N.A.</td><td>Specify path of the .netrc file</td>
147 </table>
148 
149 ### Password Authentication
150 
151 The key
152 HTTP.CREDENTIALS.USERPASSWORD
153 can be used to set the simple password authentication.
154 This is an alternative to setting it in the url.
155 The value must be of the form "username:password".
156 See the <a href="#USERPWDESCAPE">password escaping</a> section
157 to see how this value must escape certain characters.
158 Also see <a href="#REDIR">redirection authorization</a>
159 for important additional information.
160 
161 The pair of keys
162 HTTP.CREDENTIALS.USERNAME and HTTP.CREDENTIALS.PASSWORD
163 can be used as an alternative to HTTP.CREDENTIALS.USERPASSWORD
164 to set the simple password authentication.
165 If present, they take precedence over HTTP.CREDENTIALS.USERPASSWORD.
166 The values do not need to be escaped.
167 See <a href="#REDIR">redirection authorization</a>
168 for important additional information.
169 
170 ### Cookie Jar
171 
172 The HTTP.COOKIEJAR key
173 specifies the name of file from which
174 to read cookies (CURLOPT_COOKIEJAR) and also
175 the file into which to store cookies (CURLOPT_COOKIEFILE).
176 The same value is used for both CURLOPT values.
177 It defaults to in-memory storage.
178 See [redirection authorization](#REDIR)
179 for important additional information.
180 
181 ### Certificate Authentication
182 
183 HTTP.SSL.CERTIFICATE
184 specifies a file path for a file containing a PEM cerficate.
185 This is typically used for client-side authentication.
186 
187 HTTP.SSL.KEY is essentially the same as HTTP.SSL.CERTIFICATE
188 and should always have the same value.
189 
190 HTTP.SSL.KEYPASSWORD
191 specifies the password for accessing the HTTP.SSL.CERTIFICAT/HTTP.SSL.key file.
192 
193 HTTP.SSL.CAPATH
194 specifies the path to a directory containing
195 trusted certificates for validating server sertificates.
196 See reference #2 for more info.
197 
198 HTTP.SSL.VALIDATE
199 is a boolean (1/0) value that if true (1)
200 specifies that the client should verify the server's presented certificate.
201 
202 HTTP.PROXY.SERVER
203 specifies the url for accessing the proxy:
204 e.g. *http://[username:password@]host[:port]*
205 
206 HTTP.PROXY_SERVER
207 deprecated; use HTTP.PROXY.SERVER
208 
209 HTTP.NETRC
210 specifies the absolute path of the .netrc file.
211 See [redirection authorization](#REDIR)
212 for information about using .netrc.
213 
214 ## Password Escaping {#USERPWDESCAPE}
215 
216 With current password rules, it is is not unlikely that the password
217 will contain characters that need to be escaped. Similarly, the user
218 may contain characters such as '@' that need to be escaped. To support this,
219 it is assumed that all occurrences of `user:password` use URL (i.e. %%XX)
220 escaping for at least the characters in the table below.
221 
222 The minimum set of characters that must be escaped depends on the location.
223 If the user+pwd is embedded in the URL, then '@' and ':' __must__ be escaped.
224 If the user+pwd is the value for
225 the HTTP.CREDENTIALS.USERPASSWORD key in the _rc_ file, then
226 ':' __must__ be escaped.
227 Escaping should __not__ be used in the `.netrc` file nor in
228 HTTP.CREDENTIALS.USERNAME or HTTPCREDENTIALS.PASSWORD.
229 
230 The relevant escape codes are as follows.
231 <table>
232 <tr><th>Character</th><th>Escaped Form</th>
233 <tr><td>'@'</td><td>%40</td>
234 <tr><td>':'</td><td>%3a</td>
235 </table>
236 Additional characters can be escaped if desired.
237 
238 ## Redirection-Based Authentication {#REDIR}
239 
240 Some sites provide authentication by using a third party site
241 to do the authentication. Examples include ESG, URS, RDA, and most oauth2-based
242 systems.
243 
244 The process is usually as follows.
245 
246 1. The client contacts the server of interest (SOI), the actual data provider
247 using, typically _http_ protocol.
248 2. The SOI sends a redirect to the client to connect to the e.g. URS system
249 using the _https_ protocol (note the use of _https_ instead of _http_).
250 3. The client authenticates with URS.
251 4. URS sends a redirect (with authorization information) to send
252 the client back to the SOI to actually obtain the data.
253 
254 It turns out that libcurl, by default, uses the password in the
255 `.daprc` file (or from the url) for all connections that request
256 a password. This causes problems because only the the specific
257 redirected connection is the one that actually requires the password.
258 This is where the `.netrc` file comes in. Libcurl will use `.netrc`
259 for the redirected connection. It is possible to cause libcurl
260 to use the `.daprc` password always, but this introduces a
261 security hole because it may send the initial user+pwd to every
262 server in the redirection chain.
263 In summary, if you are using redirection, then you are
264 ''strongly'' encouraged to create a `.netrc` file to hold the
265 password for the site to which the redirection is sent.
266 
267 The format of this `.netrc` file will contain lines that
268 typically look like this.
269 
270  machine mmmmmm login xxxxxx password yyyyyy
271 
272 where the machine, mmmmmm, is the hostname of the machine to
273 which the client is redirected for authorization, and the
274 login and password are those needed to authenticate on that machine.
275 
276 The location of the `.netrc` file can be specified by
277 putting the following line in your `.daprc`/`.dodsrc` file.
278 
279  HTTP.NETRC=<path to netrc file>
280 
281 If not specified, then libcurl will look first in the current
282 directory, and then in the HOME directory.
283 
284 One final note. In using this, you MUST
285 to specify a real file in the file system to act as the
286 cookie jar file (HTTP.COOKIEJAR) so that the
287 redirect site can properly pass back authorization information.
288 
289 ## Client-Side Certificates {#CLIENTCERTS}
290 
291 Some systems, notably ESG (Earth System Grid), requires
292 the use of client-side certificates, as well as being
293 [re-direction based](#REDIR).
294 This requires setting the following entries:
295 
296 - HTTP.COOKIEJAR &mdash; a file path for storing cookies across re-direction.
297 - HTTP.NETRC &mdash; the path to the netrc file.
298 - HTTP.SSL.CERTIFICATE &mdash; the file path for the client side certificate file.
299 - HTTP.SSL.KEY &mdash; this should have the same value as HTTP.SSL.CERTIFICATE.
300 - HTTP.SSL.CAPATH &mdash; the path to a "certificates" directory.
301 - HTTP.SSL.VALIDATE &mdash; force validation of the server certificate.
302 
303 Note that the first two are there to support re-direction based authentication.
304 
305 ## References
306 
307 1. https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
308 2. https://curl.haxx.se/docs/ssl-compared.html
309 
310 ## Appendix A. All RC-File Keys {#allkeys}
311 
312 For completeness, this is the list of all rc-file keys.
313 If this documentation is out of date with respect to the actual code,
314 the code is definitive.
315 <table>
316 <tr><th>Key</th><th>curl_easy_setopt Option</th>
317 <tr valign="top"><td>HTTP.DEFLATE</td><td>CUROPT_DEFLATE<br>with value "deflate,gzip"</td>
318 <tr><td>HTTP.VERBOSE</td><td>CUROPT_VERBOSE</td>
319 <tr><td>HTTP.TIMEOUT</td><td>CUROPT_TIMEOUT</td>
320 <tr><td>HTTP.USERAGENT</td><td>CUROPT_USERAGENT</td>
321 <tr><td>HTTP.COOKIEJAR</td><td>CUROPT_COOKIEJAR</td>
322 <tr><td>HTTP.COOKIE_JAR</td><td>CUROPT_COOKIEJAR</td>
323 <tr valign="top"><td>HTTP.PROXY.SERVER</td><td>CURLOPT_PROXY,<br>CURLOPT_PROXYPORT,<br>CURLOPT_PROXYUSERPWD</td>
324 <tr valign="top"><td>HTTP.PROXY_SERVER</td><td>CURLOPT_PROXY,<br>CURLOPT_PROXYPORT,<br>CURLOPT_PROXYUSERPWD</td>
325 <tr><td>HTTP.SSL.CERTIFICATE</td><td>CUROPT_SSLCERT</td>
326 <tr><td>HTTP.SSL.KEY</td><td>CUROPT_SSLKEY</td>
327 <tr><td>HTTP.SSL.KEYPASSWORD</td><td>CUROPT_KEYPASSWORD</td>
328 <tr><td>HTTP.SSL.CAINFO</td><td>CUROPT_CAINFO</td>
329 <tr><td>HTTP.SSL.CAPATH</td><td>CUROPT_CAPATH</td>
330 <tr><td>HTTP.SSL.VERIFYPEER</td><td>CUROPT_SSL_VERIFYPEER</td>
331 <tr><td>HTTP.CREDENTIALS.USERPASSWORD</td><td>CUROPT_USERPASSWORD</td>
332 <tr><td>HTTP.CREDENTIALS.USERNAME</td><td>CUROPT_USERNAME</td>
333 <tr><td>HTTP.CREDENTIALS.PASSWORD</td><td>CUROPT_PASSWORD</td>
334 <tr><td>HTTP.NETRC</td><td>CURLOPT_NETRC,CURLOPT_NETRC_FILE</td>
335 </table>
336 
337 ## Appendix B. URS Access in Detail {#URSDETAIL}
338 
339 It is possible to use the NASA Earthdata Login System (URS)
340 with netcdf by using using the process specified in the
341 [redirection based authorization section](#REDIR).
342 In order to access URS controlled datasets, however, it is necessary to
343 register as a user with NASA at this website (subject to change):
344 
345  https://uat.urs.earthdata.nasa.gov/
346 
347 ## Appendix C. ESG Access in Detail {#ESGDETAIL}
348 
349 It is possible to access Earth Systems Grid (ESG) datasets
350 from ESG servers through the netCDF API using the techniques
351 described in the section on [Client-Side Certificates](#CLIENTCERTS).
352 
353 In order to access ESG datasets, however, it is necessary to
354 register as a user with ESG and to setup your environment
355 so that proper authentication is established between an netcdf
356 client program and the ESG data server. Specifically, it
357 is necessary to use what is called "client-side keys" to
358 enable this authentication. Normally, when a client accesses
359 a server in a secure fashion (using "https"), the server
360 provides an authentication certificate to the client.
361 With client-side keys, the client must also provide a
362 certificate to the server so that the server can know with
363 whom it is communicating. Note that this section is subject
364 to change as ESG changes its procedures.
365 
366 The netcdf library uses the _curl_ library and it is that
367 underlying library that must be properly configured.
368 
369 ### Terminology
370 
371 The key elements for client-side keys requires the constructions of
372 two "stores" on the client side.
373 
374 * Keystore - a repository to hold the client side key.
375 * Truststore - a repository to hold a chain of certificates
376 that can be used to validate the certificate
377 sent by the server to the client.
378 
379 The server actually has a similar set of stores, but the client
380 need not be concerned with those.
381 
382 ### Initial Steps
383 
384 The first step is to obtain authorization from ESG.
385 Note that this information may evolve over time, and
386 may be out of date.
387 This discussion is in terms of BADC and NCSA. You will need
388 to substitute as necessary.
389 
390 1. Register at http://badc.nerc.ac.uk/register
391  to obtain access to badc and to obtain an openid,
392  which will looks something like:
393  <pre>https://ceda.ac.uk/openid/Firstname.Lastname</pre>
394 
395 2. Ask BADC for access to whatever datasets are of interest.
396 
397 3. Obtain short term credentials at
398  _http://grid.ncsa.illinois.edu/myproxy/MyProxyLogon/_
399  You will need to download and run the MyProxyLogon program.
400  This will create a keyfile in, typically, the directory ".globus".
401  The keyfile will have a name similar to this: "x509up_u13615"
402  The other elements in ".globus" are certificates to use in
403  validating the certificate your client gets from the server.
404 
405 4. Obtain the program source ImportKey.java
406  from this location: _http://www.agentbob.info/agentbob/79-AB.html_
407  (read the whole page, it will help you understand the remaining steps).
408 
409 ### Building the KeyStore
410 
411 You will have to modify the keyfile in the previous step
412 and then create a keystore and install the key and a certificate.
413 The commands are these:
414 
415  openssl pkcs8 -topk8 -nocrypt -in x509up_u13615 -inform PEM -out key.der -outform DER
416  openssl x509 -in x509up_u13615 -inform PEM -out cert.der -outform DER
417  java -classpath <path to ImportKey.class> -Dkeypassword="<password>" -Dkeystore=./<keystorefilename> key.der cert.der
418 
419 Note, the file names "key.der" and "cert.der" can be whatever you choose.
420 It is probably best to leave the .der extension, though.
421 
422 ### Building the TrustStore
423 
424 Building the truststore is a bit tricky because as provided, the
425 certificates in ".globus" need some massaging. See the script below
426 for the details. The primary command is this, which is executed for every
427 certificate, c, in globus. It sticks the certificate into the file
428 named "truststore"
429 
430  keytool -trustcacerts -storepass "password" -v -keystore "truststore" -importcert -file "${c}"
431 
432 ### Running the C Client
433 
434 Refer to the section on [Client-Side Certificates](#CLIENTCERTS).
435 The keys specified there must be set in the rc file to support ESG access.
436 
437 - HTTP.COOKIEJAR=~/.dods_cookies
438 - HTTP.NETRC=~/.netrc
439 - HTTP.SSL.CERTIFICATE=~/esgkeystore
440 - HTTP.SSL.KEY=~/esgkeystore
441 - HTTP.SSL.CAPATH=~/.globus
442 - HTTP.SSL.VALIDATE=1
443 
444 Of course, the file paths above are suggestions only;
445 you can modify as needed.
446 The HTTP.SSL.CERTIFICATE and HTTP.SSL.KEY
447 entries should have same value, which is the file path for the
448 certificate produced by MyProxyLogon. The HTTP.SSL.CAPATH entry
449 should be the path to the "certificates" directory produced by
450 MyProxyLogon.
451 
452 As noted, ESG also uses re-direction based authentication.
453 So, when it receives an initial connection from a client, it
454 redirects to a separate authentication server. When that
455 server has authenticated the client, it redirects back to
456 the original url to complete the request.
457 
458 ### Script for creating Stores
459 
460 The following script shows in detail how to actually construct the key
461 and trust stores. It is specific to the format of the globus file
462 as it was when ESG support was first added. It may have changed
463 since then, in which case, you will need to seek some help
464 in fixing this script. It would help if you communicated
465 what you changed to the author so this document can be updated.
466 
467  #!/bin/sh -x
468  KEYSTORE="esgkeystore"
469  TRUSTSTORE="esgtruststore"
470  GLOBUS="globus"
471  TRUSTROOT="certificates"
472  CERT="x509up_u13615"
473  TRUSTROOTPATH="$GLOBUS/$TRUSTROOT"
474  CERTFILE="$GLOBUS/$CERT"
475  PWD="password"
476 
477  D="-Dglobus=$GLOBUS"
478  CCP="bcprov-jdk16-145.jar"
479  CP="./build:${CCP}"
480  JAR="myproxy.jar"
481 
482  # Initialize needed directories
483  rm -fr build
484  mkdir build
485  rm -fr $GLOBUS
486  mkdir $GLOBUS
487  rm -f $KEYSTORE
488  rm -f $TRUSTSTORE
489 
490  # Compile MyProxyCmd and ImportKey
491  javac -d ./build -classpath "$CCP" *.java
492  javac -d ./build ImportKey.java
493 
494  # Execute MyProxyCmd
495  java -cp "$CP myproxy.MyProxyCmd
496 
497  # Build the keystore
498  openssl pkcs8 -topk8 -nocrypt -in $CERTFILE -inform PEM -out key.der -outform DER
499  openssl x509 -in $CERTFILE -inform PEM -out cert.der -outform DER
500  java -Dkeypassword=$PWD -Dkeystore=./${KEYSTORE} -cp ./build ImportKey key.der cert.der
501 
502  # Clean up the certificates in the globus directory
503  for c in ${TRUSTROOTPATH}/*.0 ; do
504  alias=`basename $c .0`
505  sed -e '0,/---/d' <$c >/tmp/${alias}
506  echo "-----BEGIN CERTIFICATE-----" >$c
507  cat /tmp/${alias} >>$c
508  done
509 
510  # Build the truststore
511  for c in ${TRUSTROOTPATH}/*.0 ; do
512  alias=`basename $c .0`
513  echo "adding: $TRUSTROOTPATH/${c}"
514  echo "alias: $alias"
515  yes | keytool -trustcacerts -storepass "$PWD" -v -keystore ./$TRUSTSTORE -alias $alias -importcert -file "${c}"
516  done
517  exit

Return to the Main Unidata NetCDF page.
Generated on Wed Aug 1 2018 05:36:48 for NetCDF. NetCDF is a Unidata library.